Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2022-01-12T17:32:38ZPulp
Planio Pulp - Story #9670 (CLOSED - DUPLICATE): In an access policy for reposiroty versions repository p...https://pulp.plan.io/issues/96702022-01-12T17:32:38Zmdellweg
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/2076":<a href="https://github.com/pulp/pulpcore/issues/2076" class="external">https://github.com/pulp/pulpcore/issues/2076</a></p> Pulp - Story #9635 (CLOSED - DUPLICATE): As a user, I can specify the desired maximum amount of m...https://pulp.plan.io/issues/96352021-12-13T16:23:46Zbmbouterbmbouter@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/2069":<a href="https://github.com/pulp/pulpcore/issues/2069" class="external">https://github.com/pulp/pulpcore/issues/2069</a></p>
<hr>
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>It would be nice if users could specify a desired maximum amount of RAM to be used during sync. For example, a user can say I only want 1500 MB of RAM to be used max.</p>
<a name="What-is-already-in-place"></a>
<h2 >What is already in place<a href="#What-is-already-in-place" class="wiki-anchor">¶</a></h2>
<p>The stages pipeline restricts memory usage by only allowing 1000 declarative content objects between each stage (so for 8-9 stages that's 8000-9000 declarative content objects. This happens <a href="https://github.com/pulp/pulpcore/blob/main/pulpcore/plugin/stages/api.py#L217" class="external">here</a>.</p>
<p>Interestingly the docstring says this defaults to 100, but it seems to actually be 1000!</p>
<p>Also the stages perform batching, so they will only taking in a limited number of items (the batch size). That happens <a href="https://github.com/pulp/pulpcore/blob/main/pulpcore/plugin/stages/api.py#L84" class="external">with minsize</a>.</p>
<a name="Why-this-isnt-enough"></a>
<h2 >Why this isn't enough<a href="#Why-this-isnt-enough" class="wiki-anchor">¶</a></h2>
<p>These are count-based mechnisms and don't correspond to actual MB or GB of memory used. Some content units vary a lot in how much memory each DeclarativeContent objects take up.</p>
<p>Another lesser problem is that it doesn't help plugin writers restrict their usage of memory in FirstStage.</p>
<a name="Idea"></a>
<h2 >Idea<a href="#Idea" class="wiki-anchor">¶</a></h2>
<p>Add a new param called <code>max_mb</code> to base Remote, which defaults to None. If specified, the user will be specifying the desired maximum MB used by process syncing.</p>
<p>Have the queues between the stages, and the bather implementation, both check the total memory the current process is using and asyncio.sleep() polling until it goes down. This should keep the maximum amount used by all objects roughly to that number.</p>
<a name="Details"></a>
<h2 >Details<a href="#Details" class="wiki-anchor">¶</a></h2>
<p>Introduce a new <code>MBSizeQueue</code> which is a wrapper around <code>asyncio.Queue</code> used today. It will have the same <code>put()</code> call, only wait if the amount of memory in use is greater than the remote is configured for.</p>
<p>Then introduce the same memory checking feature in the batcher. I'm not completely sure this second part is needed though.</p>
<p>We have to be very careful not to deadlock with this feature. For example, we have to account for the base case where even a single item is larger than the memory desired. Repos in pulp_rpm have had a single unit use more than 1.2G if I remember right, so if someone was syncing with 800 MB and we weren't careful to allow that unit to still flow through the pipeline we'd deadlock.....</p> RPM Support - Task #9633 (CLOSED - DUPLICATE): Use repo priorities in the dependency solverhttps://pulp.plan.io/issues/96332021-12-12T06:09:47Zdalleydalley@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2309":<a href="https://github.com/pulp/pulp_rpm/issues/2309" class="external">https://github.com/pulp/pulp_rpm/issues/2309</a></p>
<hr>
<p>We ought to be setting repo priorities such that for every set of copies, any matching RPMs present in the same repository are prioritzed over ones in other repos.</p> RPM Support - Test #9626 (CLOSED - DUPLICATE): Add tests for SHA repo to test_synchttps://pulp.plan.io/issues/96262021-12-08T21:43:04Zggainey
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulp_rpm/2320":<a href="https://github.com/pulp/pulp_rpm/issues/2320" class="external">https://github.com/pulp/pulp_rpm/issues/2320</a></p>
<hr>
<p>We have been bitten a few times now by repos that use 'sha' (instead of 'sha1') checksums. Build a test for same that does <strong>not</strong> require sync'ing all of RHEL6.6...</p> Pulp - Refactor #9623 (CLOSED - DUPLICATE): As a developer, I have a pytest fixture that allows m...https://pulp.plan.io/issues/96232021-12-08T20:31:21Zbmbouterbmbouter@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/2086":<a href="https://github.com/pulp/pulpcore/issues/2086" class="external">https://github.com/pulp/pulpcore/issues/2086</a></p>
<hr>
<p>As a basic capability, it would be nice to use the aiohttp pytest plugin to serve fixture data to Pulp's tests. This should rewrite one test to demonstrate its feasibility also.</p> RPM Support - Test #9622 (MODIFIED): Add a repo signed using 'sha' as alias for 'sha1'https://pulp.plan.io/issues/96222021-12-08T19:00:00Zggainey
<p>'sha' support exists in the wild, is the same as 'sha1', and has broken us several times now, Let's make it possible to write tests for it.</p> Pulp - Story #9621 (CLOSED - CURRENTRELEASE): As a user I can pass environment variables to the s...https://pulp.plan.io/issues/96212021-12-08T18:14:33Zipanova@redhat.comipanova@redhat.comContainer Support - Task #9618 (CLOSED - CURRENTRELEASE): Adjust code to work wiht recent group p...https://pulp.plan.io/issues/96182021-12-08T16:34:14ZmdellwegPulp - Story #9615 (CLOSED - CURRENTRELEASE): Add async sign method for SigningServicehttps://pulp.plan.io/issues/96152021-12-07T20:32:27ZgerrodPulp - Story #9614 (CLOSED - DUPLICATE): As a developer, I can mark a Model as RBAC enabled and h...https://pulp.plan.io/issues/96142021-12-07T19:03:50Zbmbouterbmbouter@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/2067":<a href="https://github.com/pulp/pulpcore/issues/2067" class="external">https://github.com/pulp/pulpcore/issues/2067</a></p>
<hr>
<p>This is build on <a href="https://pulp.plan.io/issues/9613" class="external">the introduction of <code>with_perm</code></a>.</p>
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Everytime a queryset is constructed that deals with an RBAC enabled object, we need to ensure that only those objects that user has permissions to operate on are available in the queryset results. For example, if I have the <code>core.delete_task</code> permission on some objects, but not others, I can't just run <code>Task.objects.all().delete()</code>.</p>
<p>We deal with querysets in so many places, it would be great to have a safer way to be told if I've filtered each queryset at least in some way by permissions.</p>
<a name="Proposal"></a>
<h2 >Proposal<a href="#Proposal" class="wiki-anchor">¶</a></h2>
<p>Add an attribute on all models called <code>RBAC_PROTECTED = False</code> and have models opt-in to using this safety feature by setting it to <code>True</code> on their model definition.</p>
<p>Then modify the querset evaluation to raise an exception if that queryset never had a <code>with_perm</code> call occur. This would be an opt-in, model-by-model safety feature.</p>
<p>There are some situations when you are supposed to not need a <code>with_perm</code> call. For example if the viewset queries for all objects, and then passes the list of pks to the task in the backend to handle, the backend queryset construction already handled permissions but there is no call to <code>with_perm</code> there.</p>
<p>Let's add a queryset method called <code>qs.with_no_perms()</code>. With this I could call <code>Task.objects.with_no_perms().all()</code> and I would not receive the exception even without a call to <code>with_perm</code>.</p>
<a name="Special-considerations"></a>
<h2 >Special considerations<a href="#Special-considerations" class="wiki-anchor">¶</a></h2>
<p>There could be situations where a new querset is made as a new object, e.g. boolean or set operations. Let's get examples of these kinds of situations right:</p>
<ul>
<li>
<code>qs.all() | qs.with_perm("core.task_show")</code> -> unsafe</li>
<li>
<code>qs.none() | qs.with_perm("core.task_show")</code> -> safe</li>
<li>
<code>qs.none() & qs.with_perm("core.task_show")</code> ??</li>
<li>
<code>qs.all() & qs.with_perm("core.task_show")</code> -> safe</li>
</ul> Pulp - Story #9613 (CLOSED - DUPLICATE): As a developer, I can make permission object filtering c...https://pulp.plan.io/issues/96132021-12-07T18:46:13Zbmbouterbmbouter@redhat.com
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/2066":<a href="https://github.com/pulp/pulpcore/issues/2066" class="external">https://github.com/pulp/pulpcore/issues/2066</a></p>
<hr>
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>As a developer with the new Roles facilities in pulpcore==3.17, you likely will want to filter by permissions with something like this example taken from <a href="https://github.com/pulp/pulpcore/pull/1721/files" class="external">this PR</a>.</p>
<pre><code>current_user = get_current_authenticated_user()
qs = Task.objects.filter(finished_at__lt=finished_before, state__in=states)
units_deleted, details = get_objects_for_user(current_user, "core.delete_task", qs=qs).delete()
</code></pre>
<p>As you can see, this needs to determine who the current user is, and you can't build the queryset in one go by using chaining.</p>
<a name="Proposal"></a>
<h2 >Proposal<a href="#Proposal" class="wiki-anchor">¶</a></h2>
<p>Introduce a <code>with_perm</code> chainable call on all querysets for Pulp objects. It could be used like this:</p>
<ul>
<li><code>qs.with_perm("core.task_delete")</code></li>
<li><code>qs.with_perm("core.task_delete", "core.task_view")</code></li>
<li><code>qs.with_perms(["core.task_delete", "core.task_view"])</code></li>
<li><code>qs.with_perm("core.task_delete").with_perm( "core.task_view")</code></li>
</ul> Container Support - Story #9607 (CLOSED - CURRENTRELEASE): Enable rate_limit option during contai...https://pulp.plan.io/issues/96072021-12-06T18:03:54Zipanova@redhat.comipanova@redhat.com
<p>pulp-container plugin subclassed <code>HttpDownloader</code> and has overridden <code>_run</code> method which enables Throttler</p>
<p><a href="https://github.com/pulp/pulp_container/blob/main/pulp_container/app/downloaders.py#L60" class="external">https://github.com/pulp/pulp_container/blob/main/pulp_container/app/downloaders.py#L60</a>
<a href="https://github.com/pulp/pulpcore/blob/main/pulpcore/download/http.py#L269" class="external">https://github.com/pulp/pulpcore/blob/main/pulpcore/download/http.py#L269</a></p> Pulp - Story #9606 (CLOSED - CURRENTRELEASE): As a user who manages permissions, I can reset an a...https://pulp.plan.io/issues/96062021-12-06T15:40:13Zbmbouterbmbouter@redhat.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Users can modify access policies, but sometimes they may want to reset it back to the shipped default.</p>
<a name="Design"></a>
<h2 >Design<a href="#Design" class="wiki-anchor">¶</a></h2>
<p>The detail view of an AccessPolicy should have a <code><path_to_AccessPolicy_instance>/reset/</code> endpoint that accepts a POST. Upon posting it should:</p>
<ol>
<li>Restore the default access policy</li>
<li>Ensure that the customized flag is false</li>
</ol> Pulp - Task #9604 (CLOSED - CURRENTRELEASE): As a developer, I can easily add add/remove/list Rol...https://pulp.plan.io/issues/96042021-12-03T17:33:08Zbmbouterbmbouter@redhat.com
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>Now that pulpcore knows about Roles, and users can define their own, we need to allow users to manage the role assignments to specific objects and "model level" permissions.</p>
<a name="Design"></a>
<h2 >Design<a href="#Design" class="wiki-anchor">¶</a></h2>
<p>Create the following API calls that would be nested under any given viewset, e.g. TaskViewset.</p>
<ul>
<li>
<p><code> add_role</code> - If on a detail view, add the role the user specifies to the group or groups and/or user or users the user specifies to the specific object. If not on a detail view, add the role the user specifies to the group or gorups and/or user or users the user specifies as a model level role. The role is required. At least one group or user must be specified. If the Role does not have a permission applicable to this object type an error is expected.</p>
</li>
<li>
<p><code>remove_role</code> - If on a detail view, remove the role the user specifies from the group or groups and/or user or users the user specifies to the specific object. If not on a detail view, remove the role the user specifies from the group or gorups and/or user or users the user specifies as a model level role. The role is required. At least one group or user must be specified. If the Role does not have a permission applicable to this object type an error is expected. If no users or groups had that role no error is expected.</p>
</li>
<li>
<p><code>list_roles</code> - List the roles that could have at least one permission that is meaningful for this object type.</p>
</li>
<li>
<p><code>my_permissions</code> - If on a detail view, lists the effective object-level permissions a user has through both direct and group-based membership. If not on a detail view, lists the effective model level permissions a user has through both direct and group-based membership.</p>
</li>
</ul>
<p>Create a <code>RoleMixin</code> that allows developers to add ^ endpoint to any Viewset easily.</p>
<a name="Authorization-details"></a>
<h2 >Authorization details<a href="#Authorization-details" class="wiki-anchor">¶</a></h2>
<ul>
<li>
<p>The developer is expected to define a new "manage permissions" permission that is specific to that object type. For example, <code>core.manage_roles_task</code> would be a reasonable name for managing the permissions of a <code>Task</code>.</p>
</li>
<li>
<p>The developer needs to add to their access policy the specific calls to use that new permission to authorize only users who have these calls to make the calls to <code>list_roles</code>, <code>add_roles</code>, and <code>remove_role</code>. For example for <code>core.manage_roles_task</code> that would look like:</p>
</li>
</ul>
<pre><code> {
"action": ["list_roles", "add_role", "remove_role"],
"principal": "authenticated",
"effect": "allow",
"condition": "has_model_or_obj_perms:core.manage_roles_task",
},
</code></pre>
<p>It is expected the drf-access-policy would allow any authenticated user to list <code>my_permissions</code>.</p> Pulp - Story #9603 (CLOSED - DUPLICATE): Reclaim disk space without providing a list of repositorieshttps://pulp.plan.io/issues/96032021-12-03T17:22:17Ziballou
<p><strong>Ticket moved to GitHub</strong>: "pulp/pulpcore/2065":<a href="https://github.com/pulp/pulpcore/issues/2065" class="external">https://github.com/pulp/pulpcore/issues/2065</a></p>
<hr>
<p>Related: <a href="https://pulp.plan.io/issues/8459" class="external">https://pulp.plan.io/issues/8459</a></p>
<p>Katello would like to be able to clean out all repositories for a given Pulp installation. This would be useful for smart proxies, since we don't index the repository hrefs. As a work around, we have to query the repositories API to get all of the repository hrefs.</p>
<p>Perhaps this could be done by passing in an empty array for the repository hrefs.</p>