Pulp: Issueshttps://pulp.plan.io/https://pulp.plan.io/favicon.ico2020-10-01T18:03:57ZPulp
Planio Pulp - Task #7638 (NEW): Fix ansible_python_interpreter issues in pulp_installerhttps://pulp.plan.io/issues/76382020-10-01T18:03:57Zmdepaulo@redhat.com
<p>There are 3 minor / potential issues pertaining to this.</p> Pulp - Task #6798 (NEW): Document the new guidelines for plugin installation logichttps://pulp.plan.io/issues/67982020-05-21T18:47:54Zmdepaulo@redhat.com
<p>There are 3 places they could be:</p>
<ol>
<li>A role in a separate git repo and on galaxy.</li>
<li>A separate role in the pulp_installer repo (pulp_rpm will be this.)</li>
<li>Conditional logic within the pulp_installer's other roles.</li>
</ol> OSTree Support - Task #4642 (MODIFIED): Delete the 'master' branch and make 2-master to main branchhttps://pulp.plan.io/issues/46422019-04-05T13:40:27Zbmbouterbmbouter@redhat.com
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>The OStree repository is confusing users because the master branch shows 'this is the Pulp3 plugin for OSTree....". This is really just a stub and the plugin is in no way functional.</p>
<a name="Solution"></a>
<h2 >Solution<a href="#Solution" class="wiki-anchor">¶</a></h2>
<p>Delete the 'master' branch and make the default branch '2-master'. This is similar to what was done to the pulp/pulp repository which is also a Pulp2 only repository.</p> Puppet Support - Task #4641 (MODIFIED): Delete the 'master' branch and make 2-master to main branchhttps://pulp.plan.io/issues/46412019-04-05T13:40:24Zbmbouterbmbouter@redhat.com
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>The Puppet repository is confusing users because the master branch shows 'this is the Pulp3 plugin for Puppet....". This is really just a stub and the plugin is in no way functional.</p>
<a name="Solution"></a>
<h2 >Solution<a href="#Solution" class="wiki-anchor">¶</a></h2>
<p>Delete the 'master' branch and make the default branch '2-master'. This is similar to what was done to the pulp/pulp repository which is also a Pulp2 only repository.</p> Python Support - Story #3626 (MODIFIED): Python Viewset Filtershttps://pulp.plan.io/issues/36262018-04-29T18:10:09Zdalleydalley@redhat.com
<p>PythonPackageContent<br>
------------------------------------</p>
<ul>
<li>Inherit from the base Content (currently just "type" which isn't very helpful)</li>
<li>name (exact, in)</li>
<li>author (exact, in)</li>
<li>packagetype (exact, in)</li>
<li>filename (exact, in)</li>
</ul>
<p>PythonRemote<br>
----------------------</p>
<ul>
<li>Inherit from the base Remote - currently (name, last_updated)</li>
<li>Projects (exact, in) - see which remotes are set to download certain projects</li>
</ul>
<p>PythonPublisher<br>
-------------------------</p>
<ul>
<li>No extra filters, just the ones on base Publisher - currently (name, last_updated)</li>
</ul> Python Support - Story #3624 (MODIFIED): As a user, I can as for only a reduced set of fields for...https://pulp.plan.io/issues/36242018-04-27T21:10:24Zdalleydalley@redhat.com
<p>When retreiving a list of PythonPackageContent from /content/python/packages/, the user should not be subjected to the full details of every content unit.</p>
<p>Which fields should we show? I propose</p>
<ul>
<li>filename</li>
<li>name</li>
<li>packagetype</li>
<li>version</li>
</ul>
<p>In addition to the fields</p>
<ul>
<li>_href</li>
<li>created</li>
<li>notes</li>
<li>artifacts</li>
</ul>
<p>Which get pulled in by the base Content serializer which we inherit from.</p>
<p>This looks like the following</p>
<pre><code>{
"_href": "http://localhost:8000/pulp/api/v3/content/python/packages/0d2f11f5-a8f6-44c9-9053-d1f3047adf3c/",
"artifacts": {
"shelf-reader-0.1.tar.gz": "http://localhost:8000/pulp/api/v3/artifacts/e6d61484-0d16-4296-9f80-5f6559dcbc2f/"
},
"created": "2018-04-27T21:27:06.925524Z",
"filename": "shelf-reader-0.1.tar.gz",
"name": "shelf-reader",
"notes": {},
"packagetype": "sdist",
"type": "python",
"version": "0.1"
}
</code></pre> Python Support - Refactor #3597 (MODIFIED): Use a serializer to validate sync and publish parametershttps://pulp.plan.io/issues/35972018-04-24T13:54:18Zamacdona@redhat.comaustin@redhat.com
<p>To be included in the schema (autodoc and bindings), we need to create a serializer for sync and publish</p>
<p><a href="https://github.com/pulp/pulp_file/blob/master/pulp_file/app/viewsets.py#L32-L52" class="external">https://github.com/pulp/pulp_file/blob/master/pulp_file/app/viewsets.py#L32-L52</a></p>
<p>This serializer should use HyperlinkedRelatedFields to validate that the repository and repository version exist.</p> Packaging - Task #3533 (MODIFIED): Prepare for pypi pulp3 beta releasehttps://pulp.plan.io/issues/35332018-03-28T17:43:28Zbizhangbizhang@redhat.com
<p>There's some pypi packaging related work necessary for the pulp3 beta release. There's bits and pieces of this work filed under multiple redmine issues, but they all should be done before beta. This is the master issue to do those and some other necessary updates in order to release the beta.</p>
<a name="Dependecies"></a>
<h2 >Dependecies<a href="#Dependecies" class="wiki-anchor">¶</a></h2>
<ul>
<li>We should switch our psycopg2 dependency to psycopg2-binary. There is a task for that filed here: <a href="https://pulp.plan.io/issues/3375" class="external">https://pulp.plan.io/issues/3375</a>
</li>
</ul>
<ul>
<li>Django should be >=1.11<br>
There is also an issue filed that we should revisit before pulp3 rc is released. <a href="https://pulp.plan.io/issues/2896" class="external">https://pulp.plan.io/issues/2896</a>
</li>
</ul>
<ul>
<li>celery should be >=4</li>
</ul>
<a name="Versions"></a>
<h2 >Versions<a href="#Versions" class="wiki-anchor">¶</a></h2>
<p>versions for pulpcore[0] and pulpcore-common [1] and pulp_python should be set to 3.0.0b1<br>
versions for pulpcore-plugin [2] , pulp_file, and should be set to 0.0.0b1</p>
<p>[0] <a href="https://github.com/pulp/pulp/blob/3.0-dev/pulpcore/setup.py" class="external">https://github.com/pulp/pulp/blob/3.0-dev/pulpcore/setup.py</a><br>
[1] <a href="https://github.com/pulp/pulp/blob/3.0-dev/common/setup.py" class="external">https://github.com/pulp/pulp/blob/3.0-dev/common/setup.py</a><br>
[2] <a href="https://github.com/pulp/pulp/blob/3.0-dev/plugin/setup.py" class="external">https://github.com/pulp/pulp/blob/3.0-dev/plugin/setup.py</a><br>
[3] <a href="https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies" class="external">https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies</a></p> Python Support - Story #3197 (MODIFIED): As a User, I can create a PythonPackageContent from an A...https://pulp.plan.io/issues/31972017-12-13T20:12:38Zamacdona@redhat.comaustin@redhat.com
<p>The upload workflow in Pulp 3 is:</p>
<ol>
<li>Use pulpcore REST API to upload bits and create an Artifact</li>
<li>POST to a Detail ContentUnit (api/v3/content/python/) to create a PythonPackageContent from a specified Artifact</li>
</ol>
<p>This story is to make sure that (2) works. (1) Does not require anything from the plugin.</p>
<p>In Pulp 2, we create Uploaded Python ContentUnits here [0] by using twine, but we should move away from doing it that way. [1] We only use a small part of twine (which isn't really supposed to be used as a library), so we will need to implement that functionality. [2]</p>
<p>Extra credit: Put at least 1 of each upload type in <a href="https://github.com/PulpQE/pulp-fixtures" class="external">https://github.com/PulpQE/pulp-fixtures</a> (half extra credit, provide a link to 1 of each here)</p>
<p>[0]: <a href="https://github.com/pulp/pulp_python/blob/master/plugins/pulp_python/plugins/models.py#L125-L150" class="external">https://github.com/pulp/pulp_python/blob/master/plugins/pulp_python/plugins/models.py#L125-L150</a><br>
[1]: <a href="https://pulp.plan.io/issues/3157" class="external">https://pulp.plan.io/issues/3157</a><br>
[2]: <a href="https://github.com/pypa/twine/blob/3efd3b5d2715f537096b95cb2e6853e02990b059/twine/package.py" class="external">https://github.com/pypa/twine/blob/3efd3b5d2715f537096b95cb2e6853e02990b059/twine/package.py</a></p> Python Support - Story #2884 (MODIFIED): As a user I can sync from PyPIhttps://pulp.plan.io/issues/28842017-07-10T21:47:15Zamacdona@redhat.comaustin@redhat.com
<p>As a user, I can sync a list of projects from PyPI</p>
<p>This story is complete when:</p>
<ul>
<li>I can initiate a sync from PyPI</li>
<li>The sync completes without error</li>
<li>I can see that the expected content was added to the repo</li>
</ul>
<blockquote>
<ul>
<li>Syncing a project includes all releases</li>
<li>Syncing a release includes all distribution packages</li>
</ul>
</blockquote> Python Support - Task #2883 (MODIFIED): Create model(s) for Python's Releaseshttps://pulp.plan.io/issues/28832017-07-10T21:44:08Zamacdona@redhat.comaustin@redhat.com
<p>A content model, content serializer and content viewset have been already created by <a href="https://pulp.plan.io/issues/2882" class="external">https://pulp.plan.io/issues/2882</a></p>
<p>This task is to finish those classes, adding any Python specific fields.</p>
<p>This task will be complete when a django shell user can CRUD full representations of Python Package "releases". A REST API user should be able to read a list of all Python units `/v3/content/python/` as well as retrieve data on a specific unit (url is not yet decided).</p>
<p>All unit metadata is provided by the shell user at this point. It is not expected that the plugin extract the metadata from a package or scrape it from upstream.</p>
<p>After discussion we will go with the Python "distribution package" as content unit model.</p>
<p>The PythonPackageContent (because it's not really a PythonContent, and DistributionContent would overload the term 'distribution' too much) would contain the following fields:</p>
<a name="Pulp-related"></a>
<h4 >Pulp-related<a href="#Pulp-related" class="wiki-anchor">¶</a></h4>
<table>
<thead>
<tr>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td>packagetype</td>
</tr>
<tr>
<td>path</td>
</tr>
<tr>
<td>filename (primary key)</td>
</tr>
</tbody>
</table>
<a name="Python-related"></a>
<h4 >Python-related<a href="#Python-related" class="wiki-anchor">¶</a></h4>
<table>
<thead>
<tr>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td>name</td>
</tr>
<tr>
<td>version</td>
</tr>
<tr>
<td>metadata_version</td>
</tr>
<tr>
<td>summary</td>
</tr>
<tr>
<td>description</td>
</tr>
<tr>
<td>keywords</td>
</tr>
<tr>
<td>home_page</td>
</tr>
<tr>
<td>download_url</td>
</tr>
<tr>
<td>author</td>
</tr>
<tr>
<td>author_email</td>
</tr>
<tr>
<td>maintainer</td>
</tr>
<tr>
<td>maintainer_email</td>
</tr>
<tr>
<td>license</td>
</tr>
<tr>
<td>classifier</td>
</tr>
<tr>
<td>requires_python</td>
</tr>
<tr>
<td>project_url</td>
</tr>
<tr>
<td>platform</td>
</tr>
<tr>
<td>supported_platform</td>
</tr>
<tr>
<td>requires_dist</td>
</tr>
<tr>
<td>provides_dist</td>
</tr>
<tr>
<td>obsoletes_dist</td>
</tr>
<tr>
<td>requires_external</td>
</tr>
</tbody>
</table>
<p>This is they way Pulp2 is modeled currently. Each content unit would contain one artifact corresponding to the filename distribution package on PyPI.</p>
<a name="Disadvantages"></a>
<h3 >Disadvantages<a href="#Disadvantages" class="wiki-anchor">¶</a></h3>
<p>The disadvantage of modeling a Python distribution package as a content unit is that this is something the user would not care as much about. We would have multiple content units for the same release, but for different systems:<br>
eg.<br>
scipy-0.9.0-cp26-cp26mu-manylinux1_x86_64.whl<br>
scipy-0.9.0-cp27-cp27m-manylinux1_x86_64.whl<br>
scipy-0.9.0-cp27-cp27mu-manylinux1_x86_64.whl<br>
scipy-0.9.0.tar.gz<br>
scipy-0.9.0.zip</p>
<p>As a user I do not want to view all these distribution packages when I query a repository. The only thing I would care about is the release, and I will let pip take care of which distribution package to install. PyPI in particular makes the release a first class citizen instead of the distribution packages.</p>
<p>Metadata that belongs to a release (i.e. additional metadata) would be repeated across content units. PyPI stores these metadata fields as a part of the release [0], and these fields could be updated in PyPI outside of a release. The metadata we store would be the metadata in a distribution package, which is immutable, so if a user updates metadata in PyPI, we would not sync the metadata updates.</p>
<a name="Glossary"></a>
<h3 >Glossary<a href="#Glossary" class="wiki-anchor">¶</a></h3>
<p><span><strong>Release</strong></span><br>
A snapshot of a Project at a particular point in time, denoted by a version identifier.<br>
Making a release may entail the publishing of multiple "distribution packages". For example, if version 1.0 of a project was released, it could be available in both a source distribution format and a Windows installer distribution format.</p>
<p><span><strong>Distribution Package</strong></span><br>
A versioned archive file that contains Python packages, modules, and other resource files that are used to distribute a Release. The archive file is what an end-user will download from the internet and install. A project may contain many releases, and releases may contain many distribution packages. Can be type sdist, bdist, etc. "Distribution package" is used instead of "package" to avoid confusion with "import packages" or linux "distributions".</p>
<p>[0] <a href="https://warehouse.pypa.io/api-reference/xml-rpc/" class="external">https://warehouse.pypa.io/api-reference/xml-rpc/</a></p> Python Support - Task #2882 (MODIFIED): bootstrap pulp_python for Pulp 3https://pulp.plan.io/issues/28822017-07-10T21:37:16Zamacdona@redhat.comaustin@redhat.com
<p>This task is to create the basic plugin structure for the python plugin. This task is complete when:</p>
<p>1. The plugin is discoverable by Pulp<br>
2. CRUD for ContentUnits, Importers, Publishers (vanilla, without python specific fields)<br>
3. sync/publish remain NotImplemented</p> Python Support - Story #2041 (MODIFIED): As a user, I can whitelist packages to sync with standar...https://pulp.plan.io/issues/20412016-06-29T14:48:04Zamacdona@redhat.comaustin@redhat.com
<p>This story is to use the syntax from python requirements[0] files to specify which packages should be synced. This story does NOT include directly uploading a requirements.txt (though that feature could be discussed in another issue)</p>
<p>Note:<br>
It doesn't make sense for Pulp to support all of the possible syntaxes in a requirements file (like specifying a local file).</p>
<a name="Background"></a>
<h3 >Background:<a href="#Background" class="wiki-anchor">¶</a></h3>
<p>At the time of writing, pulp-python only supports a whitelist of project names, but this whitelist should become more granular and flexible.</p>
<a name="Specifiers-12"></a>
<h3 >Specifiers [1][2]<a href="#Specifiers-12" class="wiki-anchor">¶</a></h3>
<p>It would be ideal to support multiple levels of filtering:</p>
<ul>
<li>project name</li>
<li>version specifiers (including gt, lt, range)</li>
<li>specific python distributions (specified by hash) [3]</li>
</ul>
<p>Allowing users to specify python distributions by hashes [3] will significantly improve 2 of our use cases:</p>
<ul>
<li>reproducible, deterministic builds</li>
<li>improved security</li>
</ul>
<a name="Related-Ideas"></a>
<h3 >Related Ideas:<a href="#Related-Ideas" class="wiki-anchor">¶</a></h3>
<p>These ideas are related to the implementation of this story, but if they are accepted, they should be filed separately.</p>
<ol>
<li>Create a whitelist from a requirements.txt</li>
<li>Create a whitelist from a Pipfile (pipenv)</li>
<li>Create a whitelist from a Pipfile.lock (pipenv)</li>
<li>Create a whitelist from a python toml file</li>
</ol>
<p>[0]: <a href="https://pip.pypa.io/en/stable/user_guide/#requirements-files" class="external">https://pip.pypa.io/en/stable/user_guide/#requirements-files</a><br>
[1]: <a href="https://www.python.org/dev/peps/pep-0440/" class="external">https://www.python.org/dev/peps/pep-0440/</a><br>
[2]: <a href="https://www.python.org/dev/peps/pep-0508/" class="external">https://www.python.org/dev/peps/pep-0508/</a><br>
[3]: <a href="https://pip-python3.readthedocs.io/en/latest/reference/pip_install.html#hash-checking-mode" class="external">https://pip-python3.readthedocs.io/en/latest/reference/pip_install.html#hash-checking-mode</a></p> Python Support - Story #1884 (MODIFIED): As a user, I can lazily sync python packageshttps://pulp.plan.io/issues/18842016-04-29T21:55:04Zamacdona@redhat.comaustin@redhat.com
<p>With the previous work done in for python packages, it is now possible to create objects using only the metadata provided by the json API of PyPI. This metadata is also now provided by pulp publish.</p>
<p>Completion of this story should allow the user to set the download policy to on_demand and background.</p> Python Support - Story #1132 (MODIFIED): As an API user, I can provide the package_names paramete...https://pulp.plan.io/issues/11322015-07-09T19:03:56Zehelms@redhat.comehelms@redhat.com
<p>Currently the package_names parameter requires a comma separated string as input where a JSON list might be more intuitive to API users.</p>