Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-05-05T12:59:40ZPulp
Planio Pulp - Story #8701 (NEW): As a pulp_installer user, I can use the full logic to add repos to the ...https://pulp.plan.io/issues/87012021-05-05T12:59:40Zmdepaulo@redhat.com
<p>As mentioned in <a class="issue tracker-1 status-11 priority-6 priority-default closed" title="Issue: pulp_installer fails to install redis due to no EPEL7 (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/7773">#7773</a> , we should refactor our logic to add repos to the system (in a robust & configurable manner) into another role like <code>pulp_repos</code>.</p>
<p>I propose the following design:</p>
<ol>
<li>This is a dependency role. pulp_common, pulp_redis, pulp_database, will all depend on it.</li>
<li>When a role like pulp_common depends on it, it passes variables like <code>__pulp_repos_epel: true</code> to denote which repos the role needs. It passes variables via roles/pulp_common/meta/main.yml : <code>dependencies:</code>
</li>
<li>If a user wants to disable the logic to add the repo (if they added it manually), they'll pass a variable like <code>pulp_repos_epel: false</code> to disable it.</li>
<li>Existing variables for configuring how we add the repos to the system, like <code>epel_release_packages</code>, should still used.</li>
</ol>
<p>This logic is found in:</p>
<ul>
<li>roles/pulp_common/tasks/ambiguously-named-repo.yml</li>
<li>roles/pulp_common/tasks/repos.yml</li>
</ul> Pulp - Task #7638 (NEW): Fix ansible_python_interpreter issues in pulp_installerhttps://pulp.plan.io/issues/76382020-10-01T18:03:57Zmdepaulo@redhat.com
<p>There are 3 minor / potential issues pertaining to this.</p> Pulp - Task #6798 (NEW): Document the new guidelines for plugin installation logichttps://pulp.plan.io/issues/67982020-05-21T18:47:54Zmdepaulo@redhat.com
<p>There are 3 places they could be:</p>
<ol>
<li>A role in a separate git repo and on galaxy.</li>
<li>A separate role in the pulp_installer repo (pulp_rpm will be this.)</li>
<li>Conditional logic within the pulp_installer's other roles.</li>
</ol> Pulp - Story #6797 (ASSIGNED): [epic] As a user, I can consume all the plugin prereq roles in the...https://pulp.plan.io/issues/67972020-05-21T18:45:22Zmdepaulo@redhat.com
<p>pulp_rpm_prerequisites exists because the installer has had a plugin neutral policy.</p>
<p>This policy was for very long misunderstood: It's not about avoiding favoritism to any plugins, it's about not tying the installer (which is tied to pulpcore releases) to plugin releases. So that say pulpcore 3.3 logic would be in pulp_installer 3.3 release, and so that pulp_cardboardbox 0.7 logic would be in the pulp_cardboardbox_prerequisites 0.7 role.</p>
<p>The team now agrees that this policy is counter-productive because:</p>
<ol>
<li>Having a role in a separate repo (not part of the pulp_installer collection) is extra work for developers, and for users.</li>
<li>The only plugin that currently needs a prereq role, pulp_rpm, has version numbers and releases that correspond to pulpcore releases. pulp_rpm 3.3.z needs pulpcore 3.3.z, etc. So the pulp_rpm specific installation logic can be safely bundled in pulp_installer 99% of the time.</li>
</ol> Python Support - Story #4396 (MODIFIED): Add one-shot upload supporthttps://pulp.plan.io/issues/43962019-02-07T16:42:44Zdalleydalley@redhat.com
<p>The model viewset is not really intended to be used the way we're using it. Instead of allowing a "create" action on /content/python/python/, we should instead create a separate API endpoint for uploading new content, and get rid of what we currently have.</p>
<p>Example:<br>
<a href="https://github.com/pulp/pulp_rpm/pull/1246" class="external">https://github.com/pulp/pulp_rpm/pull/1246</a></p> Python Support - Story #3626 (MODIFIED): Python Viewset Filtershttps://pulp.plan.io/issues/36262018-04-29T18:10:09Zdalleydalley@redhat.com
<p>PythonPackageContent<br>
------------------------------------</p>
<ul>
<li>Inherit from the base Content (currently just "type" which isn't very helpful)</li>
<li>name (exact, in)</li>
<li>author (exact, in)</li>
<li>packagetype (exact, in)</li>
<li>filename (exact, in)</li>
</ul>
<p>PythonRemote<br>
----------------------</p>
<ul>
<li>Inherit from the base Remote - currently (name, last_updated)</li>
<li>Projects (exact, in) - see which remotes are set to download certain projects</li>
</ul>
<p>PythonPublisher<br>
-------------------------</p>
<ul>
<li>No extra filters, just the ones on base Publisher - currently (name, last_updated)</li>
</ul> Python Support - Story #3624 (MODIFIED): As a user, I can as for only a reduced set of fields for...https://pulp.plan.io/issues/36242018-04-27T21:10:24Zdalleydalley@redhat.com
<p>When retreiving a list of PythonPackageContent from /content/python/packages/, the user should not be subjected to the full details of every content unit.</p>
<p>Which fields should we show? I propose</p>
<ul>
<li>filename</li>
<li>name</li>
<li>packagetype</li>
<li>version</li>
</ul>
<p>In addition to the fields</p>
<ul>
<li>_href</li>
<li>created</li>
<li>notes</li>
<li>artifacts</li>
</ul>
<p>Which get pulled in by the base Content serializer which we inherit from.</p>
<p>This looks like the following</p>
<pre><code>{
"_href": "http://localhost:8000/pulp/api/v3/content/python/packages/0d2f11f5-a8f6-44c9-9053-d1f3047adf3c/",
"artifacts": {
"shelf-reader-0.1.tar.gz": "http://localhost:8000/pulp/api/v3/artifacts/e6d61484-0d16-4296-9f80-5f6559dcbc2f/"
},
"created": "2018-04-27T21:27:06.925524Z",
"filename": "shelf-reader-0.1.tar.gz",
"name": "shelf-reader",
"notes": {},
"packagetype": "sdist",
"type": "python",
"version": "0.1"
}
</code></pre> Python Support - Refactor #3597 (MODIFIED): Use a serializer to validate sync and publish parametershttps://pulp.plan.io/issues/35972018-04-24T13:54:18Zamacdona@redhat.comaustin@redhat.com
<p>To be included in the schema (autodoc and bindings), we need to create a serializer for sync and publish</p>
<p><a href="https://github.com/pulp/pulp_file/blob/master/pulp_file/app/viewsets.py#L32-L52" class="external">https://github.com/pulp/pulp_file/blob/master/pulp_file/app/viewsets.py#L32-L52</a></p>
<p>This serializer should use HyperlinkedRelatedFields to validate that the repository and repository version exist.</p> Packaging - Task #3533 (MODIFIED): Prepare for pypi pulp3 beta releasehttps://pulp.plan.io/issues/35332018-03-28T17:43:28Zbizhangbizhang@redhat.com
<p>There's some pypi packaging related work necessary for the pulp3 beta release. There's bits and pieces of this work filed under multiple redmine issues, but they all should be done before beta. This is the master issue to do those and some other necessary updates in order to release the beta.</p>
<a name="Dependecies"></a>
<h2 >Dependecies<a href="#Dependecies" class="wiki-anchor">¶</a></h2>
<ul>
<li>We should switch our psycopg2 dependency to psycopg2-binary. There is a task for that filed here: <a href="https://pulp.plan.io/issues/3375" class="external">https://pulp.plan.io/issues/3375</a>
</li>
</ul>
<ul>
<li>Django should be >=1.11<br>
There is also an issue filed that we should revisit before pulp3 rc is released. <a href="https://pulp.plan.io/issues/2896" class="external">https://pulp.plan.io/issues/2896</a>
</li>
</ul>
<ul>
<li>celery should be >=4</li>
</ul>
<a name="Versions"></a>
<h2 >Versions<a href="#Versions" class="wiki-anchor">¶</a></h2>
<p>versions for pulpcore[0] and pulpcore-common [1] and pulp_python should be set to 3.0.0b1<br>
versions for pulpcore-plugin [2] , pulp_file, and should be set to 0.0.0b1</p>
<p>[0] <a href="https://github.com/pulp/pulp/blob/3.0-dev/pulpcore/setup.py" class="external">https://github.com/pulp/pulp/blob/3.0-dev/pulpcore/setup.py</a><br>
[1] <a href="https://github.com/pulp/pulp/blob/3.0-dev/common/setup.py" class="external">https://github.com/pulp/pulp/blob/3.0-dev/common/setup.py</a><br>
[2] <a href="https://github.com/pulp/pulp/blob/3.0-dev/plugin/setup.py" class="external">https://github.com/pulp/pulp/blob/3.0-dev/plugin/setup.py</a><br>
[3] <a href="https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies" class="external">https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies</a></p> Python Support - Story #3197 (MODIFIED): As a User, I can create a PythonPackageContent from an A...https://pulp.plan.io/issues/31972017-12-13T20:12:38Zamacdona@redhat.comaustin@redhat.com
<p>The upload workflow in Pulp 3 is:</p>
<ol>
<li>Use pulpcore REST API to upload bits and create an Artifact</li>
<li>POST to a Detail ContentUnit (api/v3/content/python/) to create a PythonPackageContent from a specified Artifact</li>
</ol>
<p>This story is to make sure that (2) works. (1) Does not require anything from the plugin.</p>
<p>In Pulp 2, we create Uploaded Python ContentUnits here [0] by using twine, but we should move away from doing it that way. [1] We only use a small part of twine (which isn't really supposed to be used as a library), so we will need to implement that functionality. [2]</p>
<p>Extra credit: Put at least 1 of each upload type in <a href="https://github.com/PulpQE/pulp-fixtures" class="external">https://github.com/PulpQE/pulp-fixtures</a> (half extra credit, provide a link to 1 of each here)</p>
<p>[0]: <a href="https://github.com/pulp/pulp_python/blob/master/plugins/pulp_python/plugins/models.py#L125-L150" class="external">https://github.com/pulp/pulp_python/blob/master/plugins/pulp_python/plugins/models.py#L125-L150</a><br>
[1]: <a href="https://pulp.plan.io/issues/3157" class="external">https://pulp.plan.io/issues/3157</a><br>
[2]: <a href="https://github.com/pypa/twine/blob/3efd3b5d2715f537096b95cb2e6853e02990b059/twine/package.py" class="external">https://github.com/pypa/twine/blob/3efd3b5d2715f537096b95cb2e6853e02990b059/twine/package.py</a></p> Python Support - Story #2885 (MODIFIED): As a User I can Publish Python packages in a way that is...https://pulp.plan.io/issues/28852017-07-10T21:50:50Zamacdona@redhat.comaustin@redhat.com
<p>This story is to extend the Publisher model and serializer with all Python specific fields and to implement publish.</p>
<p>Publisher.publish() should only be concerned with implementing the the simple API (which is consumed by pip). When this story is complete, the user will be able to create a Publisher, use it to publish a Repository with PythonPackageContent in it, creating a Publication. The user will be able to associate that Publication to a Distribution, and then pip will be able to install packages from the distribution.base_path.</p>
<p><a href="https://wiki.python.org/moin/PyPISimple" class="external">https://wiki.python.org/moin/PyPISimple</a><br>
<a href="https://www.python.org/dev/peps/pep-0301/" class="external">https://www.python.org/dev/peps/pep-0301/</a></p>
<p>For every unique project in a repository pulp should distribute a HTML page located at: { distribution.base_path.}/simple/{project-name}/</p>
<p>This HTML page should contain every distribution package that shares the same project name and should look like:</p>
<pre><code><!DOCTYPE html>
<html>
<head>
<title>Links for pulpcore</title>
</head>
<body><h1>Links for pulpcore</h1>
<a href="{link-to-content}#md5={md5_hash}" rel="internal">{PythonPackageContentUnit.filename}</a><br/>
<a href="../../packages/02/ee/3ceee60a9694b9462c62eee3df0d2b3e525be82750bc3f9e8b4d21a0cfb7/pulpcore-3.0.0a1.dev5-py3-none-any.whl#md5=16f188762c54527815557af8a7067234" rel="internal">pulpcore-3.0.0a1.dev5-py3-none-any.whl</a><br/>
<a href="../../packages/03/6c/e48eb03fa930a36838b3ff92e4367d901bb09e725859e0030b3bb40dff77/pulpcore-3.0.0a1.dev4-py3-none-any.whl#md5=3c8c966327231e8a3f48b70866f3c56a" rel="internal">pulpcore-3.0.0a1.dev4-py3-none-any.whl</a><br/>
</body></html>
</code></pre>
<p>The json API (which would be consumed by other Pulps) is <strong>not</strong> a part of this story, and will be implemented in <a href="https://pulp.plan.io/issues/2886" class="external">https://pulp.plan.io/issues/2886</a></p> Python Support - Story #2884 (MODIFIED): As a user I can sync from PyPIhttps://pulp.plan.io/issues/28842017-07-10T21:47:15Zamacdona@redhat.comaustin@redhat.com
<p>As a user, I can sync a list of projects from PyPI</p>
<p>This story is complete when:</p>
<ul>
<li>I can initiate a sync from PyPI</li>
<li>The sync completes without error</li>
<li>I can see that the expected content was added to the repo</li>
</ul>
<blockquote>
<ul>
<li>Syncing a project includes all releases</li>
<li>Syncing a release includes all distribution packages</li>
</ul>
</blockquote> Python Support - Story #2041 (MODIFIED): As a user, I can whitelist packages to sync with standar...https://pulp.plan.io/issues/20412016-06-29T14:48:04Zamacdona@redhat.comaustin@redhat.com
<p>This story is to use the syntax from python requirements[0] files to specify which packages should be synced. This story does NOT include directly uploading a requirements.txt (though that feature could be discussed in another issue)</p>
<p>Note:<br>
It doesn't make sense for Pulp to support all of the possible syntaxes in a requirements file (like specifying a local file).</p>
<a name="Background"></a>
<h3 >Background:<a href="#Background" class="wiki-anchor">¶</a></h3>
<p>At the time of writing, pulp-python only supports a whitelist of project names, but this whitelist should become more granular and flexible.</p>
<a name="Specifiers-12"></a>
<h3 >Specifiers [1][2]<a href="#Specifiers-12" class="wiki-anchor">¶</a></h3>
<p>It would be ideal to support multiple levels of filtering:</p>
<ul>
<li>project name</li>
<li>version specifiers (including gt, lt, range)</li>
<li>specific python distributions (specified by hash) [3]</li>
</ul>
<p>Allowing users to specify python distributions by hashes [3] will significantly improve 2 of our use cases:</p>
<ul>
<li>reproducible, deterministic builds</li>
<li>improved security</li>
</ul>
<a name="Related-Ideas"></a>
<h3 >Related Ideas:<a href="#Related-Ideas" class="wiki-anchor">¶</a></h3>
<p>These ideas are related to the implementation of this story, but if they are accepted, they should be filed separately.</p>
<ol>
<li>Create a whitelist from a requirements.txt</li>
<li>Create a whitelist from a Pipfile (pipenv)</li>
<li>Create a whitelist from a Pipfile.lock (pipenv)</li>
<li>Create a whitelist from a python toml file</li>
</ol>
<p>[0]: <a href="https://pip.pypa.io/en/stable/user_guide/#requirements-files" class="external">https://pip.pypa.io/en/stable/user_guide/#requirements-files</a><br>
[1]: <a href="https://www.python.org/dev/peps/pep-0440/" class="external">https://www.python.org/dev/peps/pep-0440/</a><br>
[2]: <a href="https://www.python.org/dev/peps/pep-0508/" class="external">https://www.python.org/dev/peps/pep-0508/</a><br>
[3]: <a href="https://pip-python3.readthedocs.io/en/latest/reference/pip_install.html#hash-checking-mode" class="external">https://pip-python3.readthedocs.io/en/latest/reference/pip_install.html#hash-checking-mode</a></p> Python Support - Story #1884 (MODIFIED): As a user, I can lazily sync python packageshttps://pulp.plan.io/issues/18842016-04-29T21:55:04Zamacdona@redhat.comaustin@redhat.com
<p>With the previous work done in for python packages, it is now possible to create objects using only the metadata provided by the json API of PyPI. This metadata is also now provided by pulp publish.</p>
<p>Completion of this story should allow the user to set the download policy to on_demand and background.</p> Python Support - Story #1132 (MODIFIED): As an API user, I can provide the package_names paramete...https://pulp.plan.io/issues/11322015-07-09T19:03:56Zehelms@redhat.comehelms@redhat.com
<p>Currently the package_names parameter requires a comma separated string as input where a JSON list might be more intuitive to API users.</p>