https://pulp.plan.io/https://pulp.plan.io/favicon.ico2019-10-08T16:51:26ZPulpPulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=483042019-10-08T16:51:26Zbmbouterbmbouter@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/48304/diff?detail_id=48941">diff</a>)</li></ul> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=483062019-10-08T17:07:12Zbmbouterbmbouter@redhat.com
<ul></ul><p>I believe webservers have the same issues, users can request a url like <code>/foo/</code> which would be a directory on the filesystem <code>foo</code> and expect to receive data. They do this with the 'index' convention where the webserver typically looks for a file called <code>index</code> with some type of file extension.</p>
<p>One option is to leave Pulp as-is and just have the Exporter export with ^ convention. At export time though it would be more complicated to perform.</p>
<p>Another option is to find some way to prevent Pulp from making a Publication that can't be exported. Some sort of validation that ensures you don't have <code>/foo</code> and <code>/foo/bar</code> both in a Publication.</p>
<p>What sort of adjustment to <code>PublishedArtifact</code> and <code>PublishedMetadata</code> would improve this situation? Maybe the DB should distinguish directories? Or is it better to leave as-is, and handle at export time?</p> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=483072019-10-08T17:15:11Zbmbouterbmbouter@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-3 status-11 priority-6 priority-default closed" href="/issues/5378">Story #5378</a>: Index pages on content server</i> added</li></ul> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=483092019-10-08T17:15:18Zbmbouterbmbouter@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-3 status-11 priority-6 priority-default closed" href="/issues/5086">Story #5086</a>: As an user, I have an exporter that I can ship it on a disc or on a "dumb" webserver</i> added</li></ul> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=483112019-10-08T17:17:13Zbmbouterbmbouter@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-3 status-11 priority-6 priority-default closed" href="/issues/5397">Story #5397</a>: As a user, I can see an index page at :24816/<CONTENT_PREFIX>/</i> added</li></ul> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=483132019-10-08T18:09:33Zdaviddavis
<ul></ul><p>Another case to consider is plugins that don't have publications (as plugins have the option to distribute repo versions directly like we do in pulp_ansible). Perhaps one option is to have plugins validate repo versions.</p> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=483152019-10-08T18:17:03Zbmbouterbmbouter@redhat.com
<ul></ul><p>daviddavis wrote:</p>
<blockquote>
<p>Another case to consider is plugins that don't have publications (as plugins have the option to distribute repo versions directly like we do in pulp_ansible). Perhaps one option is to have plugins validate repo versions.</p>
</blockquote>
<p>That's a good point...</p> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=483982019-10-10T19:18:15Zdaviddavis
<ul><li><strong>Sprint/Milestone</strong> set to <i>3.0.0</i></li></ul> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=484192019-10-11T12:10:54Zdaviddavis
<ul><li><strong>Tracker</strong> changed from <i>Issue</i> to <i>Story</i></li><li><strong>% Done</strong> set to <i>0</i></li></ul> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=484292019-10-11T19:12:10Zmdepaulo@redhat.com
<ul></ul><p>I agree with bmbouter's approach.</p>
<p>I will offer some other suggestions/ideas:</p>
<p>1. A filesystem is a hierarchical database. The Windows registry (also a hierarchical database) encountered a kind-of-similar issue decades ago. Their approach applied to this situation would be to create a file named like:</p>
<p>foo/.default</p>
<p>That gets created whenever foo is both a file and a directory. The registry is only accessed via their APIs, so they could do that easily though.</p>
<p>However, that would prevent package managers from accessing the filesystem directly at the original paths and reading foo, correct? Or any 3rd party webservers (hosting for package managers) would have to interpret the filepath.</p>
<p>2. You could also store the real path to the "default" file in a filesystem's folder's extended attribute (user_xattr), but that would still have the same problem.</p> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=484372019-10-12T16:17:47Zgmbnomis
<ul></ul><p>I think we are trying to make a rather "esoteric" use case work if we map file/directory name conflicts to <code>index</code> or <code>.default</code> or similar.</p>
<p>What should happen if there is already a file/directory named <code>index</code> or <code>.default</code>? We are just shifting the problem one directory layer down. (Of course we could use another mapping in this case)</p>
<p>Additionally, we might want to enable directory listings in the content app in the future. Theoretically, <code>http:/content.app/foo/</code> (returning the directory listing) could behave differently than <code>http:/content.app/foo</code> (returning the foo content). But that would be very surprising (usually web servers deliver a redirection foo -> foo/ and that's what I would expect the content app to do as well).</p>
<p>I think that we should regard this use case as invalid. We have three options where to do this:<br>
- repo version (as proposed by @daviddavis, i.e. it could be part of the upcoming repo version validation)<br>
- publication<br>
- export</p>
<p>IMHO, we should let publication fail given the strange URL scheme such a publication exhibits when being distributed.<br>
But what is the point of allowing a repo version that can't be published? That's why I tend to already let validation fail. And, as noted above by @daviddavis, this also solves the problem for plugins that have no publication.</p> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=487382019-10-29T19:34:55Zdaviddavis
<ul></ul><p>Last week we discussed this problem and came to the conclusion that we need to add this validation to two places:</p>
<p>1. Repo version creation<br>
2. Publication</p>
<p>In order to handle the case where the plugin doesn't have publications, we need to check at repo version creation time. We also need to check at publication time though because published artifacts may clash with content relative paths. For example, suppose I create a file content with relative path PULP_MANIFEST/test.iso. That will create problems for file publications.</p>
<p>I think we should also just fail if a overlap is detected. And I think plugins control both of these places though so we'll probably need to create a method or some util that they can reuse.</p>
<p>If this all sounds right, I'll update the description and get this issue groomed by next week.</p> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=488722019-11-04T20:18:16Zbmbouterbmbouter@redhat.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Low</i></li></ul><p>Can be delivered in after 3.0 branching and before GA as a bugfix.</p> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=489492019-11-08T11:53:14Zdaviddavis
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li><li><strong>Assignee</strong> set to <i>daviddavis</i></li></ul> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=496392019-11-27T16:39:53Zdaviddavis
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>POST</i></li></ul><p><a href="https://github.com/pulp/pulpcore/pull/420" class="external">https://github.com/pulp/pulpcore/pull/420</a></p> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=497072019-12-02T17:40:03Zdaviddavis
<ul><li><strong>Related to</strong> <i><a class="issue tracker-3 status-11 priority-6 priority-default closed" href="/issues/5827">Story #5827</a>: As a plugin writer, I have a finalize_new_publication() method I can hook into</i> added</li></ul> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=497382019-12-03T15:59:07Zdaviddavis
<ul><li><strong>Status</strong> changed from <i>POST</i> to <i>MODIFIED</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset <a class="changeset" title="Add validation to check for file path overlaps fixes #5559 https://pulp.plan.io/issues/5559" href="https://pulp.plan.io/projects/pulp/repository/pulpcore/revisions/e915cf7663d113a7e12f1ca7b7507821815a70ad">pulpcore|e915cf7663d113a7e12f1ca7b7507821815a70ad</a>.</p> Pulp - Story #5559: As a plugin writer, I cannot export my Publication to POSIX filesystemshttps://pulp.plan.io/issues/5559?journal_id=511512019-12-13T17:31:01Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>MODIFIED</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul>