Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2019-08-21T12:47:54ZPulp
Planio RPM Support - Test #5320 (CLOSED - WONTFIX): Module Streams not copying correctly with recursive ...https://pulp.plan.io/issues/53202019-08-21T12:47:54Zbherring
<ol>
<li>
<p>Create and sync the following yum repo (Source) -> <a href="https://partha.fedorapeople.org/test-repos/pteradactyl/" class="external">https://partha.fedorapeople.org/test-repos/pteradactyl/</a></p>
</li>
<li>
<p>Create another repo Dest which will serve as the destination repo</p>
</li>
<li>
<p>Go to mongo and pick up a uuid for the pteradactly:2 module stream. This stream will be copied from source to dest .</p>
</li>
<li>
<p>run the following command</p>
<pre><code>https://<fqdn>/pulp/api/v2/repositories/Dest/actions/associate/: {"source_repo_id":"Source","criteria":{"type_ids":["modulemd"],"filters":{"association":{"unit_id":{"$in":[<$MODULE UUID>]}}}},"override_config":{"recursive":true}}: {"content_type"=>"application/json", "accept"=>"application/json"}
</code></pre>
</li>
<li>
<p>pulp-admin rpm repo list. Check for the number of module mds copied over by the above call.</p>
</li>
<li>
<p>notice that with recursive set to true all the pteradactyl module streams gets copied over, instead of just pteradactly:2 and packages belonging to that</p>
</li>
<li>
<p>Behavior is similar for recursive conservative</p>
</li>
</ol> Docker Support - Test #5181 (CLOSED - COMPLETE): Removing docker manifests from a docker reposito...https://pulp.plan.io/issues/51812019-07-24T12:58:57Zbherring
<a name="Original-Dev-Problem"></a>
<h3 >Original Dev Problem<a href="#Original-Dev-Problem" class="wiki-anchor">¶</a></h3>
<p>Removing all docker manifests from a large docker repo seems to take a long time:</p>
<p>~300 manifests takes ~2 minutes<br>
~2000 manifests takes ~30-40 minutes</p>
<a name="Reproducer"></a>
<h3 >Reproducer:<a href="#Reproducer" class="wiki-anchor">¶</a></h3>
<p>1. Create and sync a docker repo such as: <a href="https://quay.io" class="external">https://quay.io</a> datawire/ambassador<br>
2. Remove all docker manifests from the repo: pulp-admin docker repo remove manifest --repo-id=1-docker-dev-7915f7d0-7a98-4131-9c41-1be7b578d442 --not id=foo</p>
<a name="Discussion-with-asmacado"></a>
<h3 >Discussion with asmacado<a href="#Discussion-with-asmacado" class="wiki-anchor">¶</a></h3>
<ul>
<li>A lot of the delay were caused by the sequential removals</li>
<li>If there was an outage while the task was on-going, the removal was left in a non-deterministic state</li>
<li>The timing model for removal for general use cases grew incredibly fast</li>
</ul>
<a name="Solutions-with-the-new-refactor"></a>
<h3 >Solutions with the new refactor:<a href="#Solutions-with-the-new-refactor" class="wiki-anchor">¶</a></h3>
<ul>
<li>All BLOBS and MANIFESTS are all removed at once</li>
<li>Higher level units are all removed second</li>
<li>Timing shrank from 300 ~=1hr to 300 ~=a few seconds</li>
</ul>
<a name="Concerns"></a>
<h3 >Concerns:<a href="#Concerns" class="wiki-anchor">¶</a></h3>
<ul>
<li>This refactor would affect removals and syncs</li>
<li>The removal, if interrupted, would leave Orphans which could affect upstream verification</li>
</ul>
<a name="Actions"></a>
<h3 >Actions:<a href="#Actions" class="wiki-anchor">¶</a></h3>
<a name="Upstream"></a>
<h4 >Upstream<a href="#Upstream" class="wiki-anchor">¶</a></h4>
<ul>
<li>Need an upstream verification that ensures there is no change in functionality with remove units and sync units</li>
</ul>
<a name="Downstream"></a>
<h4 >Downstream<a href="#Downstream" class="wiki-anchor">¶</a></h4>
<ul>
<li>Need to verify downstream verification are not affected by this change/patch</li>
<li>BZ verification will just be the removal of a large number of repos takes a short amount of time</li>
</ul>
<a name="Test-Permutations"></a>
<h3 >Test Permutations<a href="#Test-Permutations" class="wiki-anchor">¶</a></h3>
<a name="Remove"></a>
<h4 >Remove<a href="#Remove" class="wiki-anchor">¶</a></h4>
<p>Variables:</p>
<ul>
<li>(Permutations) Shared or Not Shared [manifest_lists, manifests]</li>
<li>type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob']</li>
</ul>
<p>Verification Scenarios:</p>
<ul>
<li>For Not Shared, remove $REMOVAL_TYPES and verify all units have been removed</li>
<li>For Shared, remove $REMOVAL_TYPES and verify only non-shared units have been removed. Shared units should remain.</li>
</ul>
<p>Note:</p>
<ul>
<li>Removal Criteria short-cut for relevant type_id being removed.</li>
</ul>
<pre><code> criteria = {
"type_ids": ['docker_tag', 'docker_manifest_list', 'docker_manifest', 'docker_blob'],
"filters": {"unit": {"_id": {"$in": all_units}}},
}
</code></pre>
<ul>
<li>There are already tests covering tag removal</li>
</ul> RPM Support - Test #5055 (CLOSED - DUPLICATE): [EPIC] Ursine RPM Copy dependencies on modular RPM...https://pulp.plan.io/issues/50552019-06-27T15:17:53Zbherring
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2217":<a href="https://github.com/pulp/pulp_rpm/issues/2217" class="external">https://github.com/pulp/pulp_rpm/issues/2217</a></p>
<hr>
<a name="History"></a>
<h2 >History<a href="#History" class="wiki-anchor">¶</a></h2>
<p>There is currently a 'policy decision' that modules in the base repo (of rhel) don't rely on modular rpms, but that it's not an engineering limitation, and that fedora and rhel will likely do so at some point in the future. This is already a consideration for fedora and fedora-updates repos.</p>
<p>At least in Fedora, you can expect that there will be content in the updates-testing repo depending on content in updates-modular and/or fedora-modular.</p>
<p>This means Pulp will <strong>eventually</strong> need to support multiple source repos for copy.</p>
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>As pulp is delivered today, Ursine RPM deps on modular RPMs are NOT covered by test.</p>
<p>Normal, non-modular (ursine) [0] RPMs can depend on module packages.</p>
<p>As an example, the postgres language RPMs depend on the default stream of the postgresql module.</p>
<p>When this is the case, the default module and all artifacts related to that module and the URSINE RPM are ALL copied to the target repo.</p>
<p>At this time, to account for this scenario, a hybrid repo containing URSINE and MODULAR RPMs with modules will be required.</p>
<a name="Example"></a>
<h2 >Example<a href="#Example" class="wiki-anchor">¶</a></h2>
<a name="Recursive"></a>
<h3 >Recursive<a href="#Recursive" class="wiki-anchor">¶</a></h3>
<a name="Before"></a>
<h4 >Before<a href="#Before" class="wiki-anchor">¶</a></h4>
<ul>
<li>Copy Ursine RPM "zebra-1.0.rpm" from Repo A to B</li>
<li>Zebra requires modular RPM bar-1.0.rpm and has no other dependencies</li>
</ul>
<pre><code>
default module: module-FOO: [foo-1.0.rpm, bar-1.0.rpm, baz-1.0.rpm]
repo A
|
|---- chicken-1.1.rpm
|---- zebra-1.0.rpm
|----module-FOO
|----foo-1.0.rpm
|----bar-1.0.rpm
|----baz-1.0.rpm
repo B
|
|----bar-0.7.rpm
</code></pre>
<a name="After-Copy"></a>
<h4 >After Copy<a href="#After-Copy" class="wiki-anchor">¶</a></h4>
<p>Result of copying ursine RPM zebra-1.0.rpm from repo A to repo B:</p>
<pre><code>repo B
|
|---- zebra-1.0.rpm
|----module-FOO
|----foo-1.0.rpm
|----bar-1.0.rpm
|----baz-1.0.rpm
|----bar-0.7.rpm
All available artifacts are copied, always. There is no way to copy just module on its own,
if any of its artifacts are present in a source repo (repo A).
</code></pre>
<a name="Solution"></a>
<h2 >Solution<a href="#Solution" class="wiki-anchor">¶</a></h2>
<ul>
<li>Add Ursine RPM permutations that exercise this case for recursive and non_recursive Ursine RPM copies in mixed repos with modular RPMs where default streams <strong>are</strong> and <strong>are not</strong> defined.</li>
</ul>
<a name="Note"></a>
<h2 >Note:<a href="#Note" class="wiki-anchor">¶</a></h2>
<ul>
<li>the entire module should be treated as a single content unit to the greatest extent possible. it should copy the module that provides both A and B, all if its artifacts, and, the module default</li>
<li>a module RPM should never be treated as just a normal RPM</li>
<li>If module has multiple streams, ONLY the artifacts associated with the default module and the default module are copied, correct?
<ul>
<li>only the artifacts associated with the default module stream should ever be considered in the first place. if there is no default stream for a module, then RPMs should not be able to depend on it</li>
</ul>
</li>
</ul>
<a name="References"></a>
<h2 >References<a href="#References" class="wiki-anchor">¶</a></h2>
<p>[0] - <a href="https://docs.fedoraproject.org/en-US/modularity/architecture/consuming/dnf-behavior/" class="external">https://docs.fedoraproject.org/en-US/modularity/architecture/consuming/dnf-behavior/</a><br>
[1] - <a href="https://docs.pulpproject.org/plugins/pulp_rpm/user-guide/features.html?highlight=modularity" class="external">https://docs.pulpproject.org/plugins/pulp_rpm/user-guide/features.html?highlight=modularity</a></p> Pulp - Test #5034 (CLOSED - COMPLETE): PULP_MANIFEST does not update correctly to publish files w...https://pulp.plan.io/issues/50342019-06-25T19:24:19Zbherring
<p>PULP_MANIFEST does not update correctly when delete an unit</p>
<p>PULP_MANIFEST does not update correctly to publish files with iso_distributor via fast-forward (refer to the issue[1]) when deleting an unit.</p>
<p>Reproducing steps:<br>
1. Create an iso repo<br>
2. Upload an iso to the repo<br>
3. Delete the iso and check the iso and PULP_MANIFEST, the iso file was removed but metadata is still in PULP_MANIFEST</p>
<p>[1] <a href="https://pulp.plan.io/issues/4708" class="external">https://pulp.plan.io/issues/4708</a>.</p> RPM Support - Test #4952 (CLOSED - COMPLETE): Missing rpms in erratum pkglist when an erratum app...https://pulp.plan.io/issues/49522019-06-11T12:15:26Zbherring
<p>Description of problem:<br>
Pulp will returns wrong 'pkglist' if an erratum exists in multiple repositories and user syncs multiple of these repositories</p>
<p>For example:<br>
Erratum 'RHSA-2019:1235' exists in both 'rhel-7-server-rpms' repo and 'rhel-7-server-optional-debug-rpms' repo with the following 'pkglist' respectively</p>
<p>rhel-7-server-rpms:<br>
"ruby-libs-2.0.0.648-35.el7_6.x86_64.rpm",<br>
"ruby-libs-2.0.0.648-35.el7_6.i686.rpm",<br>
"ruby-2.0.0.648-35.el7_6.x86_64.rpm",<br>
"rubygem-bigdecimal-1.2.0-35.el7_6.x86_64.rpm",<br>
"rubygem-psych-2.0.0-35.el7_6.x86_64.rpm",<br>
"rubygem-rdoc-4.0.0-35.el7_6.noarch.rpm",<br>
"ruby-irb-2.0.0.648-35.el7_6.noarch.rpm",<br>
"rubygem-io-console-0.4.2-35.el7_6.x86_64.rpm",<br>
"rubygems-2.0.14.1-35.el7_6.noarch.rpm",<br>
"rubygem-json-1.7.7-35.el7_6.x86_64.rpm"</p>
<p>rhel-7-server-optional-debug-rpms:<br>
"ruby-debuginfo-2.0.0.648-35.el7_6.x86_64.rpm",</p>
<p>After syncing these 2 repos to the Satellite, Pulp will only list the 'ruby-debuginfo rpm' in the erratum 'pkglist'. Pulp will calculate the wrong applicability for the consumer because the erratum is listing only 1 rpm.</p>
<p>Test in "foreman-rake console":</p>
<ol>
<li>Before enabling debug repo, the erratum shows 10 packages:<br>
pp Katello::pulp_server.resources.repository.unit_search("046aa6b4-6369-4ef7-a3f7-3959250bd86f", {:type_ids => ['erratum'], filters: {unit: {"errata_id": "RHSA-2019:1235"}}})[0]['metadata']['pkglist'][0]['packages'].size<br>
10</li>
</ol>
<ol>
<li>After enabling and sync debug repo. the erratum shows only 1 package.<br>
pp Katello::pulp_server.resources.repository.unit_search("046aa6b4-6369-4ef7-a3f7-3959250bd86f", {:type_ids => ['erratum'], filters: {unit: {"errata_id": "RHSA-2019:1235"}}})[0]['metadata']['pkglist'][0]['packages'].size<br>
1</li>
</ol>
<p>pp Katello::pulp_server.resources.repository.unit_search("046aa6b4-6369-4ef7-a3f7-3959250bd86f", {:type_ids => ['erratum'], filters: {unit: {"errata_id": "RHSA-2019:1235"}}})[0]['metadata']['pkglist'][0]['packages'][0]['filename']<br>
"ruby-debuginfo-2.0.0.648-35.el7_6.x86_64.rpm"</p>
<p>exit</p>
<p>Steps to Reproduce:<br>
1. Ensure you have a RHEL 7 client with older ruby version. Lets say Client 'A'.<br>
2. Enable and sync 'rhel-7-server-rpms' repo<br>
3. Go to Web UI -> Hosts -> Content Hosts -> Client 'A' -> Errata tab -> Filter "id = RHSA-2019:1235" should show this erratum is applicable/installable.<br>
4. Enable and sync 'rhel-7-server-optional-debug-rpms' repo<br>
5. Now need to do something so that we can FORCE recalculate the applicability for Client 'A'</p>
<p>On client 'A' do:<br>
subscription-manager repos --enable="rhel-7-server-optional-debug-rpms"<br>
katello-enabled-repos-upload -f</p>
<p>6. Wait for the regenerate applicability task to finish<br>
7. Go to Web UI -> Hosts -> Content Hosts -> Client 'A' -> Errata tab -> Filter "id = RHSA-2019:1235" shows empty result.</p>
<p>Actual results:<br>
Errata RHSA-2019:1235 is not applicable to Client A</p>
<p>Expected results:<br>
Errata RHSA-2019:1235 is applicable to the Client A</p> Pulp - Test #4942 (CLOSED - COMPLETE): [EPIC] - Pulp2 - 2.20 https://pulp.plan.io/issues/49422019-06-10T12:33:32Zbherring
<a name="Information"></a>
<h2 >Information<a href="#Information" class="wiki-anchor">¶</a></h2>
<p>As new issues are added for release candidate to P2 for 2.20, compiling all of those issues as related issues to this parent issue.</p>
<p>This ticket allows the release shepherd to keep up with the status of each ticket as QE is heading to Dev Freeze to attempt to stay ahead.</p>
<a name="Current-Topics"></a>
<h3 >Current Topics<a href="#Current-Topics" class="wiki-anchor">¶</a></h3>
<p>These topics will be a subset of Dev release notes.</p>
<p>Just here for quick FYI on scanning:</p>
<ul>
<li>Modularity</li>
<li>Unit Copy Duplication</li>
<li>Issue with libsolv even in building: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1718015" class="external">https://bugzilla.redhat.com/show_bug.cgi?id=1718015</a>
</li>
</ul> Docker Support - Test #4941 (CLOSED - COMPLETE): Pulp 2 Nightly Regression: Dockerv1 Interface de...https://pulp.plan.io/issues/49412019-06-10T11:36:24Zbherring
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<ul>
<li>Nightly CI is now failing for docker v1</li>
<li>Started: June 7, 2019</li>
</ul>
<a name="Solution"></a>
<h2 >Solution<a href="#Solution" class="wiki-anchor">¶</a></h2>
<ul>
<li>Remove deprecated/unsupported DockerV1 testing from Pulp-2-Testing</li>
</ul> RPM Support - Test #4922 (CLOSED - WONTFIX): Module Streams not copying correctly with recursive ...https://pulp.plan.io/issues/49222019-06-05T18:17:27Zbherring
<ol>
<li>
<p>Create and sync the following yum repo (Source) -> <a href="https://partha.fedorapeople.org/test-repos/pteradactyl/" class="external">https://partha.fedorapeople.org/test-repos/pteradactyl/</a></p>
</li>
<li>
<p>Create another repo Dest which will serve as the destination repo</p>
</li>
<li>
<p>Go to mongo and pick up a uuid for the pteradactly:2 module stream. This stream will be copied from source to dest .</p>
</li>
<li>
<p>run the following command</p>
<pre><code>https://<fqdn>/pulp/api/v2/repositories/Dest/actions/associate/: {"source_repo_id":"Source","criteria":{"type_ids":["modulemd"],"filters":{"association":{"unit_id":{"$in":[<$MODULE UUID>]}}}},"override_config":{"recursive":true}}: {"content_type"=>"application/json", "accept"=>"application/json"}
</code></pre>
</li>
<li>
<p>pulp-admin rpm repo list. Check for the number of module mds copied over by the above call.</p>
</li>
<li>
<p>notice that with recursive set to true all the pteradactyl module streams gets copied over, instead of just pteradactly:2 and packages belonging to that</p>
</li>
<li>
<p>Behavior is similar for recursive conservative</p>
</li>
</ol> RPM Support - Test #4824 (CLOSED - DUPLICATE): Pulp does not resync yum metadata files on changehttps://pulp.plan.io/issues/48242019-05-15T17:01:53Zbherring
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2215":<a href="https://github.com/pulp/pulp_rpm/issues/2215" class="external">https://github.com/pulp/pulp_rpm/issues/2215</a></p>
<hr>
<p>Pulp uses revision numbers in repomd.xml to determine if contents need to be updated on sync. However "modifyrepo" does not generate new revision numbers for non rpm data.</p>
<p>Steps:<br>
1) Setup the following repo</p>
<pre><code>$ mkdir /tmp/my-data
$ cd /tmp/my-data
$ wget https://partha.fedorapeople.org/test-repos/rpm-with-productid/elephant-0.3-0.8.noarch.rpm
$ createrepo .
$ echo "100000" >> productid
$ modifyrepo --mdtype=productid productid repodata
$ grep revision repodata/repomd.xml
<revision>1554217257</revision>
</code></pre>
<p>2) Sync this repo<br>
3) Now update the repo</p>
<pre><code>$ cd /tmp/my-data
$ echo "100001" >> productid
$ modifyrepo --mdtype=productid productid repodata
$ grep revision repodata/repomd.xml
<revision>1554217257</revision>
</code></pre>
<p>Notice that the revision number did not change even though a metadata file got updated. Try resyncing this change and notice that the productid change will get ignored.</p> Pulp - Test #4823 (CLOSED - DUPLICATE): Modulemd profiles not getting removed from the consumerhttps://pulp.plan.io/issues/48232019-05-15T15:57:41Zbherring
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/1827":<a href="https://github.com/pulp/pulpcore/issues/1827" class="external">https://github.com/pulp/pulpcore/issues/1827</a></p>
<hr>
<p>I have consumer "integration_test_consumer_support" registered to pulp. It has 1 module enabled. When I try to delete all modulemd profiles belonging to this consumer, it simple gets ignored.</p>
<pre><code># before profile delete
$ curl -k -u admin:admin https://localhost/pulp/api/v2/consumers/integration_test_consumer_support/profiles/
[{"profile": [{"context": "deadbeef", "version": "20180730223407", "arch": "noarch", "name": "kangaroo", "stream": "0"}], "_href": "/pulp/api/v2/consumers/integration_test_consumer_support/profiles/modulemd/", "_ns": "consumer_unit_profiles", "profile_hash": "f69375f21b302f40025ff8a2128004e81436407554972029cbfbe445d0a2b563", "consumer_id": "integration_test_consumer_support", "content_type": "modulemd", "_id": {"$oid": "5cab7b49db284e2341a2c0ac"}, "id": "5cab7b49db284e2341a2c0ac"}]
# Delete profile
curl -X "DELETE" -k -u admin:admin https://localhost/pulp/api/v2/consumers/integration_test_consumer_support/profiles/modulemd/
# after profile delete
curl -k -u admin:admin https://localhost/pulp/api/v2/consumers/integration_test_consumer_support/profiles/
[{"profile": [{"context": "deadbeef", "version": "20180730223407", "arch": "noarch", "name": "kangaroo", "stream": "0"}], "_href": "/pulp/api/v2/consumers/integration_test_consumer_support/profiles/modulemd/", "_ns": "consumer_unit_profiles", "profile_hash": "f69375f21b302f40025ff8a2128004e81436407554972029cbfbe445d0a2b563", "consumer_id": "integration_test_consumer_support", "content_type": "modulemd", "_id": {"$oid": "5cab7b49db284e2341a2c0ac"}, "id": "5cab7b49db284e2341a2c0ac"}]
</code></pre> Python Support - Test #4748 (CLOSED - DUPLICATE): Improve Publications functional testshttps://pulp.plan.io/issues/47482019-04-30T12:13:07Zamacdona@redhat.comaustin@redhat.com
<p>The tests written during the feature change are a stopgap, and have some problems:<br>
<a href="https://github.com/pulp/pulp_python/pull/242/" class="external">https://github.com/pulp/pulp_python/pull/242/</a></p>
<p>The tests need to be broken up. Currently they are fragile and fully dependent on each other to pass. Ideally, the preparation should be moved to the setup of each class, but this could cause slow tests since a lot of steps need to be in place before a publication can be created. Please have a look at the utilities, they probably need to be refactored.</p> Pulp - Test #4659 (CLOSED - DUPLICATE): Add RHEL to the ansible-pulp molecule CIhttps://pulp.plan.io/issues/46592019-04-09T15:14:09Zamacdona@redhat.comaustin@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/1822":<a href="https://github.com/pulp/pulpcore/issues/1822" class="external">https://github.com/pulp/pulpcore/issues/1822</a></p>
<hr>
<p>Unlike the other OSes that are currently tested, RHEL needs some additional steps, notably, enabling the subscription.</p>
<p>This test will be complete when RHEL is included in the test matrix for the ansible installer.</p> Docker Support - Test #4129 (CLOSED - COMPLETE): Test sync of a repository that returns a 403 res...https://pulp.plan.io/issues/41292018-11-06T19:56:22Zamacdona@redhat.comaustin@redhat.com
<p>This test will require the creation of a new fixture for pulp_docker that is a corrupted repository. This fixture is primarily intended to test <a href="https://pulp.plan.io/issues/2966" class="external">https://pulp.plan.io/issues/2966</a> (pulp 2) but could also be useful for testing pulp 3.</p>
<p>From 2966, it appears that a 403 can be caused by a missing symlink in the published repository.</p> Docker Support - Test #4128 (CLOSED - COMPLETE): Test sync of a repository that is missing blobshttps://pulp.plan.io/issues/41282018-11-06T19:52:42Zamacdona@redhat.comaustin@redhat.com
<p>This test will require the creation of a new fixture for pulp_docker that is a corrupted repository. This fixture is primarily intended to test <a href="https://pulp.plan.io/issues/2849" class="external">https://pulp.plan.io/issues/2849</a> (pulp 2) but could also be useful for testing pulp 3.</p>
<p>The "corrupted repository" can be a fully published non-corrupt repository that is simply missing one or more Images referenced by or or more of the Image Manifests.</p> Pulp - Test #3634 (CLOSED - COMPLETE): Write functional tests for task delete and cancel codehttps://pulp.plan.io/issues/36342018-04-30T20:59:51Zamacdona@redhat.comaustin@redhat.com
<p>Tests should successfully delete tasks in final states, and should not be able to delete tasks in waiting or running states.</p>
<p>Tests should successfully cancel tasks in waiting and running states, but should not be able to cancel tasks in final states.</p>