Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2021-10-11T14:56:13ZPulp
Planio Pulp - Task #9499 (MODIFIED): As a developer, I have azurite on dev environmenthttps://pulp.plan.io/issues/94992021-10-11T14:56:13Zfao89Pulp - Story #8702 (NEW): As a user, the example-use playbook is not cluttered with object storag...https://pulp.plan.io/issues/87022021-05-05T13:31:24Zmdepaulo@redhat.com
<p>We should move the object storage checks from the the example-use playbook to the pulp_common role to solve this.</p>
<p>It will provide a better user experience. (Making the example playbook as small as possible.)</p>
<p>It will also enforce the checks for users that do not use the example-use playbook.</p>
<p><a href="https://github.com/pulp/pulp_installer/blob/master/playbooks/example-use/playbook.yml" class="external">https://github.com/pulp/pulp_installer/blob/master/playbooks/example-use/playbook.yml</a></p>
<p><a href="https://github.com/pulp/pulp_installer/blob/master/roles/pulp_common/tasks/main.yml#L16" class="external">https://github.com/pulp/pulp_installer/blob/master/roles/pulp_common/tasks/main.yml#L16</a></p> Pulp - Story #8701 (NEW): As a pulp_installer user, I can use the full logic to add repos to the ...https://pulp.plan.io/issues/87012021-05-05T12:59:40Zmdepaulo@redhat.com
<p>As mentioned in <a class="issue tracker-1 status-11 priority-6 priority-default closed" title="Issue: pulp_installer fails to install redis due to no EPEL7 (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/7773">#7773</a> , we should refactor our logic to add repos to the system (in a robust & configurable manner) into another role like <code>pulp_repos</code>.</p>
<p>I propose the following design:</p>
<ol>
<li>This is a dependency role. pulp_common, pulp_redis, pulp_database, will all depend on it.</li>
<li>When a role like pulp_common depends on it, it passes variables like <code>__pulp_repos_epel: true</code> to denote which repos the role needs. It passes variables via roles/pulp_common/meta/main.yml : <code>dependencies:</code>
</li>
<li>If a user wants to disable the logic to add the repo (if they added it manually), they'll pass a variable like <code>pulp_repos_epel: false</code> to disable it.</li>
<li>Existing variables for configuring how we add the repos to the system, like <code>epel_release_packages</code>, should still used.</li>
</ol>
<p>This logic is found in:</p>
<ul>
<li>roles/pulp_common/tasks/ambiguously-named-repo.yml</li>
<li>roles/pulp_common/tasks/repos.yml</li>
</ul> Pulp - Story #6797 (ASSIGNED): [epic] As a user, I can consume all the plugin prereq roles in the...https://pulp.plan.io/issues/67972020-05-21T18:45:22Zmdepaulo@redhat.com
<p>pulp_rpm_prerequisites exists because the installer has had a plugin neutral policy.</p>
<p>This policy was for very long misunderstood: It's not about avoiding favoritism to any plugins, it's about not tying the installer (which is tied to pulpcore releases) to plugin releases. So that say pulpcore 3.3 logic would be in pulp_installer 3.3 release, and so that pulp_cardboardbox 0.7 logic would be in the pulp_cardboardbox_prerequisites 0.7 role.</p>
<p>The team now agrees that this policy is counter-productive because:</p>
<ol>
<li>Having a role in a separate repo (not part of the pulp_installer collection) is extra work for developers, and for users.</li>
<li>The only plugin that currently needs a prereq role, pulp_rpm, has version numbers and releases that correspond to pulpcore releases. pulp_rpm 3.3.z needs pulpcore 3.3.z, etc. So the pulp_rpm specific installation logic can be safely bundled in pulp_installer 99% of the time.</li>
</ol> Python Support - Story #4396 (MODIFIED): Add one-shot upload supporthttps://pulp.plan.io/issues/43962019-02-07T16:42:44Zdalleydalley@redhat.com
<p>The model viewset is not really intended to be used the way we're using it. Instead of allowing a "create" action on /content/python/python/, we should instead create a separate API endpoint for uploading new content, and get rid of what we currently have.</p>
<p>Example:<br>
<a href="https://github.com/pulp/pulp_rpm/pull/1246" class="external">https://github.com/pulp/pulp_rpm/pull/1246</a></p> Python Support - Story #3624 (MODIFIED): As a user, I can as for only a reduced set of fields for...https://pulp.plan.io/issues/36242018-04-27T21:10:24Zdalleydalley@redhat.com
<p>When retreiving a list of PythonPackageContent from /content/python/packages/, the user should not be subjected to the full details of every content unit.</p>
<p>Which fields should we show? I propose</p>
<ul>
<li>filename</li>
<li>name</li>
<li>packagetype</li>
<li>version</li>
</ul>
<p>In addition to the fields</p>
<ul>
<li>_href</li>
<li>created</li>
<li>notes</li>
<li>artifacts</li>
</ul>
<p>Which get pulled in by the base Content serializer which we inherit from.</p>
<p>This looks like the following</p>
<pre><code>{
"_href": "http://localhost:8000/pulp/api/v3/content/python/packages/0d2f11f5-a8f6-44c9-9053-d1f3047adf3c/",
"artifacts": {
"shelf-reader-0.1.tar.gz": "http://localhost:8000/pulp/api/v3/artifacts/e6d61484-0d16-4296-9f80-5f6559dcbc2f/"
},
"created": "2018-04-27T21:27:06.925524Z",
"filename": "shelf-reader-0.1.tar.gz",
"name": "shelf-reader",
"notes": {},
"packagetype": "sdist",
"type": "python",
"version": "0.1"
}
</code></pre> Python Support - Story #3197 (MODIFIED): As a User, I can create a PythonPackageContent from an A...https://pulp.plan.io/issues/31972017-12-13T20:12:38Zamacdona@redhat.comaustin@redhat.com
<p>The upload workflow in Pulp 3 is:</p>
<ol>
<li>Use pulpcore REST API to upload bits and create an Artifact</li>
<li>POST to a Detail ContentUnit (api/v3/content/python/) to create a PythonPackageContent from a specified Artifact</li>
</ol>
<p>This story is to make sure that (2) works. (1) Does not require anything from the plugin.</p>
<p>In Pulp 2, we create Uploaded Python ContentUnits here [0] by using twine, but we should move away from doing it that way. [1] We only use a small part of twine (which isn't really supposed to be used as a library), so we will need to implement that functionality. [2]</p>
<p>Extra credit: Put at least 1 of each upload type in <a href="https://github.com/PulpQE/pulp-fixtures" class="external">https://github.com/PulpQE/pulp-fixtures</a> (half extra credit, provide a link to 1 of each here)</p>
<p>[0]: <a href="https://github.com/pulp/pulp_python/blob/master/plugins/pulp_python/plugins/models.py#L125-L150" class="external">https://github.com/pulp/pulp_python/blob/master/plugins/pulp_python/plugins/models.py#L125-L150</a><br>
[1]: <a href="https://pulp.plan.io/issues/3157" class="external">https://pulp.plan.io/issues/3157</a><br>
[2]: <a href="https://github.com/pypa/twine/blob/3efd3b5d2715f537096b95cb2e6853e02990b059/twine/package.py" class="external">https://github.com/pypa/twine/blob/3efd3b5d2715f537096b95cb2e6853e02990b059/twine/package.py</a></p> Python Support - Task #2936 (MODIFIED): Add pep8speakshttps://pulp.plan.io/issues/29362017-07-24T13:42:09ZdaviddavisPython Support - Story #2885 (MODIFIED): As a User I can Publish Python packages in a way that is...https://pulp.plan.io/issues/28852017-07-10T21:50:50Zamacdona@redhat.comaustin@redhat.com
<p>This story is to extend the Publisher model and serializer with all Python specific fields and to implement publish.</p>
<p>Publisher.publish() should only be concerned with implementing the the simple API (which is consumed by pip). When this story is complete, the user will be able to create a Publisher, use it to publish a Repository with PythonPackageContent in it, creating a Publication. The user will be able to associate that Publication to a Distribution, and then pip will be able to install packages from the distribution.base_path.</p>
<p><a href="https://wiki.python.org/moin/PyPISimple" class="external">https://wiki.python.org/moin/PyPISimple</a><br>
<a href="https://www.python.org/dev/peps/pep-0301/" class="external">https://www.python.org/dev/peps/pep-0301/</a></p>
<p>For every unique project in a repository pulp should distribute a HTML page located at: { distribution.base_path.}/simple/{project-name}/</p>
<p>This HTML page should contain every distribution package that shares the same project name and should look like:</p>
<pre><code><!DOCTYPE html>
<html>
<head>
<title>Links for pulpcore</title>
</head>
<body><h1>Links for pulpcore</h1>
<a href="{link-to-content}#md5={md5_hash}" rel="internal">{PythonPackageContentUnit.filename}</a><br/>
<a href="../../packages/02/ee/3ceee60a9694b9462c62eee3df0d2b3e525be82750bc3f9e8b4d21a0cfb7/pulpcore-3.0.0a1.dev5-py3-none-any.whl#md5=16f188762c54527815557af8a7067234" rel="internal">pulpcore-3.0.0a1.dev5-py3-none-any.whl</a><br/>
<a href="../../packages/03/6c/e48eb03fa930a36838b3ff92e4367d901bb09e725859e0030b3bb40dff77/pulpcore-3.0.0a1.dev4-py3-none-any.whl#md5=3c8c966327231e8a3f48b70866f3c56a" rel="internal">pulpcore-3.0.0a1.dev4-py3-none-any.whl</a><br/>
</body></html>
</code></pre>
<p>The json API (which would be consumed by other Pulps) is <strong>not</strong> a part of this story, and will be implemented in <a href="https://pulp.plan.io/issues/2886" class="external">https://pulp.plan.io/issues/2886</a></p> Python Support - Story #2884 (MODIFIED): As a user I can sync from PyPIhttps://pulp.plan.io/issues/28842017-07-10T21:47:15Zamacdona@redhat.comaustin@redhat.com
<p>As a user, I can sync a list of projects from PyPI</p>
<p>This story is complete when:</p>
<ul>
<li>I can initiate a sync from PyPI</li>
<li>The sync completes without error</li>
<li>I can see that the expected content was added to the repo</li>
</ul>
<blockquote>
<ul>
<li>Syncing a project includes all releases</li>
<li>Syncing a release includes all distribution packages</li>
</ul>
</blockquote> Python Support - Task #2883 (MODIFIED): Create model(s) for Python's Releaseshttps://pulp.plan.io/issues/28832017-07-10T21:44:08Zamacdona@redhat.comaustin@redhat.com
<p>A content model, content serializer and content viewset have been already created by <a href="https://pulp.plan.io/issues/2882" class="external">https://pulp.plan.io/issues/2882</a></p>
<p>This task is to finish those classes, adding any Python specific fields.</p>
<p>This task will be complete when a django shell user can CRUD full representations of Python Package "releases". A REST API user should be able to read a list of all Python units `/v3/content/python/` as well as retrieve data on a specific unit (url is not yet decided).</p>
<p>All unit metadata is provided by the shell user at this point. It is not expected that the plugin extract the metadata from a package or scrape it from upstream.</p>
<p>After discussion we will go with the Python "distribution package" as content unit model.</p>
<p>The PythonPackageContent (because it's not really a PythonContent, and DistributionContent would overload the term 'distribution' too much) would contain the following fields:</p>
<a name="Pulp-related"></a>
<h4 >Pulp-related<a href="#Pulp-related" class="wiki-anchor">¶</a></h4>
<table>
<thead>
<tr>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td>packagetype</td>
</tr>
<tr>
<td>path</td>
</tr>
<tr>
<td>filename (primary key)</td>
</tr>
</tbody>
</table>
<a name="Python-related"></a>
<h4 >Python-related<a href="#Python-related" class="wiki-anchor">¶</a></h4>
<table>
<thead>
<tr>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td>name</td>
</tr>
<tr>
<td>version</td>
</tr>
<tr>
<td>metadata_version</td>
</tr>
<tr>
<td>summary</td>
</tr>
<tr>
<td>description</td>
</tr>
<tr>
<td>keywords</td>
</tr>
<tr>
<td>home_page</td>
</tr>
<tr>
<td>download_url</td>
</tr>
<tr>
<td>author</td>
</tr>
<tr>
<td>author_email</td>
</tr>
<tr>
<td>maintainer</td>
</tr>
<tr>
<td>maintainer_email</td>
</tr>
<tr>
<td>license</td>
</tr>
<tr>
<td>classifier</td>
</tr>
<tr>
<td>requires_python</td>
</tr>
<tr>
<td>project_url</td>
</tr>
<tr>
<td>platform</td>
</tr>
<tr>
<td>supported_platform</td>
</tr>
<tr>
<td>requires_dist</td>
</tr>
<tr>
<td>provides_dist</td>
</tr>
<tr>
<td>obsoletes_dist</td>
</tr>
<tr>
<td>requires_external</td>
</tr>
</tbody>
</table>
<p>This is they way Pulp2 is modeled currently. Each content unit would contain one artifact corresponding to the filename distribution package on PyPI.</p>
<a name="Disadvantages"></a>
<h3 >Disadvantages<a href="#Disadvantages" class="wiki-anchor">¶</a></h3>
<p>The disadvantage of modeling a Python distribution package as a content unit is that this is something the user would not care as much about. We would have multiple content units for the same release, but for different systems:<br>
eg.<br>
scipy-0.9.0-cp26-cp26mu-manylinux1_x86_64.whl<br>
scipy-0.9.0-cp27-cp27m-manylinux1_x86_64.whl<br>
scipy-0.9.0-cp27-cp27mu-manylinux1_x86_64.whl<br>
scipy-0.9.0.tar.gz<br>
scipy-0.9.0.zip</p>
<p>As a user I do not want to view all these distribution packages when I query a repository. The only thing I would care about is the release, and I will let pip take care of which distribution package to install. PyPI in particular makes the release a first class citizen instead of the distribution packages.</p>
<p>Metadata that belongs to a release (i.e. additional metadata) would be repeated across content units. PyPI stores these metadata fields as a part of the release [0], and these fields could be updated in PyPI outside of a release. The metadata we store would be the metadata in a distribution package, which is immutable, so if a user updates metadata in PyPI, we would not sync the metadata updates.</p>
<a name="Glossary"></a>
<h3 >Glossary<a href="#Glossary" class="wiki-anchor">¶</a></h3>
<p><span><strong>Release</strong></span><br>
A snapshot of a Project at a particular point in time, denoted by a version identifier.<br>
Making a release may entail the publishing of multiple "distribution packages". For example, if version 1.0 of a project was released, it could be available in both a source distribution format and a Windows installer distribution format.</p>
<p><span><strong>Distribution Package</strong></span><br>
A versioned archive file that contains Python packages, modules, and other resource files that are used to distribute a Release. The archive file is what an end-user will download from the internet and install. A project may contain many releases, and releases may contain many distribution packages. Can be type sdist, bdist, etc. "Distribution package" is used instead of "package" to avoid confusion with "import packages" or linux "distributions".</p>
<p>[0] <a href="https://warehouse.pypa.io/api-reference/xml-rpc/" class="external">https://warehouse.pypa.io/api-reference/xml-rpc/</a></p> Python Support - Task #2882 (MODIFIED): bootstrap pulp_python for Pulp 3https://pulp.plan.io/issues/28822017-07-10T21:37:16Zamacdona@redhat.comaustin@redhat.com
<p>This task is to create the basic plugin structure for the python plugin. This task is complete when:</p>
<p>1. The plugin is discoverable by Pulp<br>
2. CRUD for ContentUnits, Importers, Publishers (vanilla, without python specific fields)<br>
3. sync/publish remain NotImplemented</p> Python Support - Story #2041 (MODIFIED): As a user, I can whitelist packages to sync with standar...https://pulp.plan.io/issues/20412016-06-29T14:48:04Zamacdona@redhat.comaustin@redhat.com
<p>This story is to use the syntax from python requirements[0] files to specify which packages should be synced. This story does NOT include directly uploading a requirements.txt (though that feature could be discussed in another issue)</p>
<p>Note:<br>
It doesn't make sense for Pulp to support all of the possible syntaxes in a requirements file (like specifying a local file).</p>
<a name="Background"></a>
<h3 >Background:<a href="#Background" class="wiki-anchor">¶</a></h3>
<p>At the time of writing, pulp-python only supports a whitelist of project names, but this whitelist should become more granular and flexible.</p>
<a name="Specifiers-12"></a>
<h3 >Specifiers [1][2]<a href="#Specifiers-12" class="wiki-anchor">¶</a></h3>
<p>It would be ideal to support multiple levels of filtering:</p>
<ul>
<li>project name</li>
<li>version specifiers (including gt, lt, range)</li>
<li>specific python distributions (specified by hash) [3]</li>
</ul>
<p>Allowing users to specify python distributions by hashes [3] will significantly improve 2 of our use cases:</p>
<ul>
<li>reproducible, deterministic builds</li>
<li>improved security</li>
</ul>
<a name="Related-Ideas"></a>
<h3 >Related Ideas:<a href="#Related-Ideas" class="wiki-anchor">¶</a></h3>
<p>These ideas are related to the implementation of this story, but if they are accepted, they should be filed separately.</p>
<ol>
<li>Create a whitelist from a requirements.txt</li>
<li>Create a whitelist from a Pipfile (pipenv)</li>
<li>Create a whitelist from a Pipfile.lock (pipenv)</li>
<li>Create a whitelist from a python toml file</li>
</ol>
<p>[0]: <a href="https://pip.pypa.io/en/stable/user_guide/#requirements-files" class="external">https://pip.pypa.io/en/stable/user_guide/#requirements-files</a><br>
[1]: <a href="https://www.python.org/dev/peps/pep-0440/" class="external">https://www.python.org/dev/peps/pep-0440/</a><br>
[2]: <a href="https://www.python.org/dev/peps/pep-0508/" class="external">https://www.python.org/dev/peps/pep-0508/</a><br>
[3]: <a href="https://pip-python3.readthedocs.io/en/latest/reference/pip_install.html#hash-checking-mode" class="external">https://pip-python3.readthedocs.io/en/latest/reference/pip_install.html#hash-checking-mode</a></p> Python Support - Story #1884 (MODIFIED): As a user, I can lazily sync python packageshttps://pulp.plan.io/issues/18842016-04-29T21:55:04Zamacdona@redhat.comaustin@redhat.com
<p>With the previous work done in for python packages, it is now possible to create objects using only the metadata provided by the json API of PyPI. This metadata is also now provided by pulp publish.</p>
<p>Completion of this story should allow the user to set the download policy to on_demand and background.</p> Python Support - Story #1132 (MODIFIED): As an API user, I can provide the package_names paramete...https://pulp.plan.io/issues/11322015-07-09T19:03:56Zehelms@redhat.comehelms@redhat.com
<p>Currently the package_names parameter requires a comma separated string as input where a JSON list might be more intuitive to API users.</p>