https://pulp.plan.io/https://pulp.plan.io/favicon.ico2020-11-04T17:33:04ZPulpPulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=646202020-11-04T17:33:04Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Copied from</strong> <i><a class="issue tracker-3 status-12 priority-6 priority-default closed" href="/issues/7790">Story #7790</a>: As a user I can re-upload artifacts if the file has gone missing or corrupted</i> added</li></ul> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=655892020-12-09T17:50:08Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/65589/diff?detail_id=65706">diff</a>)</li></ul> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=657792020-12-16T13:10:54Zttereshcttereshc@redhat.com
<ul></ul><blockquote>
<ol>
<li>If a file is missing it is impossible to upload a new one
when saving artifact add try/except, look for existing one
verify whether storage_path is an existing location if not update it with the newly uploaded bits</li>
</ol>
</blockquote>
<p>I would explicitly mention here that the checksums should match, the one in the DB and the checksum of newly uploaded file (checksum in the DB should not be updated, just used for comparison)</p>
<blockquote>
<ol start="2">
<li>If a file is corrupted it is impossible to re-upload and replace it with a valid one</li>
</ol>
</blockquote>
<p>how about option3 (a more explicit version of option2): introduce a flag which will be specified at upload time
E.g. --repair, or --force, or --validate-checksum.<br>
It will replace broken bits if checksum of the newly uploaded file matches a checksum in the DB.<br>
The recalculation of a checksum will happen on_demand this way and not for every upload attempt.</p> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=657852020-12-16T14:22:14Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>ttereshc wrote:</p>
<blockquote>
<blockquote>
<ol>
<li>If a file is missing it is impossible to upload a new one
when saving artifact add try/except, look for existing one
verify whether storage_path is an existing location if not update it with the newly uploaded bits</li>
</ol>
</blockquote>
<p>I would explicitly mention here that the checksums should match, the one in the DB and the checksum of newly uploaded file (checksum in the DB should not be updated, just used for comparison)</p>
</blockquote>
<p>I was imagining the artifat._init_and_validate would calculate all the checksums of the upload in course and when saving it in case of existing artifact we would handle the IntegrityError</p>
<blockquote>
<blockquote>
<ol start="2">
<li>If a file is corrupted it is impossible to re-upload and replace it with a valid one</li>
</ol>
</blockquote>
<p>how about option3 (a more explicit version of option2): introduce a flag which will be specified at upload time
E.g. --repair, or --force, or --validate-checksum.<br>
It will replace broken bits if checksum of the newly uploaded file matches a checksum in the DB.<br>
The recalculation of a checksum will happen on_demand this way and not for every upload attempt.</p>
</blockquote>
<p>This is a good compromise for the user experience improvement.</p> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=657862020-12-16T14:26:17Zttereshcttereshc@redhat.com
<ul></ul><p><a href="mailto:ipanova@redhat.com" class="email">ipanova@redhat.com</a> wrote:</p>
<blockquote>
<p>ttereshc wrote:</p>
<blockquote>
<blockquote>
<ol>
<li>If a file is missing it is impossible to upload a new one
when saving artifact add try/except, look for existing one
verify whether storage_path is an existing location if not update it with the newly uploaded bits</li>
</ol>
</blockquote>
<p>I would explicitly mention here that the checksums should match, the one in the DB and the checksum of newly uploaded file (checksum in the DB should not be updated, just used for comparison)</p>
</blockquote>
<p>I was imagining the artifat._init_and_validate would calculate all the checksums of the upload in course and when saving it in case of existing artifact we would handle the IntegrityError</p>
<blockquote>
</blockquote>
</blockquote>
<p>Right, it's a part of the try/except you mention, ok. Disregard my comment then.</p> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=658042020-12-16T15:47:10Zttereshcttereshc@redhat.com
<ul></ul><p>After some offline discussion, it turned out that this ticket implies that the upload of existing artifact (which is good and NOT corrupted or missing) will also return the existing artifact and won't fail.<br>
This directly relates to <a class="issue tracker-3 status-12 priority-6 priority-default closed" title="Story: Improve Artifact upload experience (CLOSED - DUPLICATE)" href="https://pulp.plan.io/issues/7114">#7114</a>.<br>
If it's done this way, we ought to be consistent across all resources. At least, content upload needs to be adjusted as well.</p>
<p>We also need to be sure that it's only affecting upload workflows and not the artifact creation during sync.</p> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=658062020-12-16T15:48:04Zttereshcttereshc@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-3 status-12 priority-6 priority-default closed" href="/issues/7114">Story #7114</a>: Improve Artifact upload experience</i> added</li></ul> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=659252020-12-23T13:49:28Zdaviddavis
<ul></ul><p>ttereshc wrote:</p>
<blockquote>
<p>After some offline discussion, it turned out that this ticket implies that the upload of existing artifact (which is good and NOT corrupted or missing) will also return the existing artifact and won't fail.<br>
This directly relates to <a class="issue tracker-3 status-12 priority-6 priority-default closed" title="Story: Improve Artifact upload experience (CLOSED - DUPLICATE)" href="https://pulp.plan.io/issues/7114">#7114</a>.<br>
If it's done this way, we ought to be consistent across all resources. At least, content upload needs to be adjusted as well.</p>
<p>We also need to be sure that it's only affecting upload workflows and not the artifact creation during sync.</p>
</blockquote>
<p><a class="issue tracker-3 status-12 priority-6 priority-default closed" title="Story: Improve Artifact upload experience (CLOSED - DUPLICATE)" href="https://pulp.plan.io/issues/7114">#7114</a> asks the pulp href be returned if the artifact already exists. It does not ask that the end point should succeed and no longer fail.</p>
<p>Also, IMO this proposal would break semantic versioning. Changing the response when a user uploads the exact same artifact twice would be a backwards incompatible change.</p> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=659282020-12-23T15:55:00Zdaviddavis
<ul></ul><blockquote>
<p>how about option3 (a more explicit version of option2): introduce a flag which will be specified at upload time E.g. --repair, or --force, or --validate-checksum.</p>
</blockquote>
<p>To me this makes the most sense. It's backwards compatible and is clear to the user what's happening. Fixing an artifact by calling the create endpoint without a special param seems a bit too strange/magical/clever IMO.</p> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=659522021-01-04T11:58:48Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/65952/diff?detail_id=66079">diff</a>)</li></ul> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=659532021-01-04T12:05:16Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>daviddavis wrote:</p>
<blockquote>
<blockquote>
<p>how about option3 (a more explicit version of option2): introduce a flag which will be specified at upload time E.g. --repair, or --force, or --validate-checksum.</p>
</blockquote>
<p>To me this makes the most sense. It's backwards compatible and is clear to the user what's happening. Fixing an artifact by calling the create endpoint without a special param seems a bit too strange/magical/clever IMO.</p>
</blockquote>
<p>The caveat of this option is if a client is trying get content into pulp, like podman, there is no way to specify this option on the client side, since the pulp upload layer is in the middle of the process. I am not advocating to adjust the whole proposal based on this specific usecase, but we'd need to whether hardcode the 'force' flag in the plugin code during the upload or come up with something else to enable container plugin to deal with the corrupted content.</p> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=659582021-01-04T14:00:28Zdaviddavis
<ul></ul><p><a href="mailto:ipanova@redhat.com" class="email">ipanova@redhat.com</a> wrote:</p>
<blockquote>
<p>The caveat of this option is if a client is trying get content into pulp, like podman, there is no way to specify this option on the client side, since the pulp upload layer is in the middle of the process. I am not advocating to adjust the whole proposal based on this specific usecase, but we'd need to whether hardcode the 'force' flag in the plugin code during the upload or come up with something else to enable container plugin to deal with the corrupted content.</p>
</blockquote>
<p>That's a good point. What about introducing a new endpoint (smart_update?)?</p>
<p>Also, just to call it out: another option is to bump the pulpcore major version. We have some other changes in our queue that would benefit from being able to make some backwards incompatible changes to the API (e.g. <a href="https://pulp.plan.io/issues/7762" class="external">https://pulp.plan.io/issues/7762</a>)</p> Pulp - Story #7791: As a user I can re-upload artifacts if the file has gone missing or corruptedhttps://pulp.plan.io/issues/7791?journal_id=784592022-01-17T15:12:52Zpulpbot
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/78459/diff?detail_id=79290">diff</a>)</li><li><strong>Status</strong> changed from <i>NEW</i> to <i>CLOSED - DUPLICATE</i></li></ul>