https://pulp.plan.io/https://pulp.plan.io/favicon.ico2020-11-12T21:08:45ZPulpPulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=647892020-11-12T21:08:45Zbmbouterbmbouter@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/64789/diff?detail_id=64910">diff</a>)</li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=651382020-11-26T13:41:26Zttereshcttereshc@redhat.com
<ul></ul><p>Thanks a lot for the background and use cases, it helps to efficiently refresh how this feature worked in Pulp 2.</p>
<p>I think it's a good idea to use remotes for ACS and create RemoteArtifacts as a refresh operation.
Few questions to get my head around the proposal:</p>
<ol>
<li>What will be on the ACS model apart from presumably a FK to a Remote?</li>
<li>Was <code>expired</code> configuration used in any way? I can imagine that it could be valuable for a regular task which refreshes outdated ACS.</li>
<li>Was it useful to have one base url and a list of paths in Pulp 2 from the user point of view? Or was it more of a downside that to refresh one path, I would refresh all for its specific ACS?<br>
My concern is if the common workflow is to configure and/or refresh ACSs per base_url, then with the current proposal in Pulp 3 it will be a lot of work for a user to do so.</li>
<li>Similar concern when I want to remove an ACS. I'm switching off my local server and I want to make sure it's no longer configured as an ACS. If we have one Remote per ACS, so in pulp 2 terms, it will be base_url + a path, how user can identify all ACSs for the server they want to shut down?</li>
<li>What happens when I configure Remotes and ACSs for the first time. I haven't run sync yet, so I have no content in Pulp which Remotes refer to. If we need to create RemoteArtifact, then we'll need to create Content and ContenArtifacts for all the content possible.</li>
<li>RemoteArtifact creation will cover Content with Artifacts. What happens with other content? Is it always synced from the authoritative source since all the logic which source to choose happens in the downloader for artifacts only?</li>
<li>
<code>It's like an on-demand sync in the sense that when called it indexes the remote metadata and creates remote artifacts.</code> What is implied by the indexing of remote metadata? I read the proposal as the refresh action creates RemoteArtifacts and that's it.</li>
</ol> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=651392020-11-26T13:46:02Zttereshcttereshc@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/65139/diff?detail_id=65262">diff</a>)</li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=652102020-12-01T21:02:32Zbmbouterbmbouter@redhat.com
<ul></ul><p>ttereshc wrote:</p>
<blockquote>
<p>Thanks a lot for the background and use cases, it helps to efficiently refresh how this feature worked in Pulp 2.</p>
<p>I think it's a good idea to use remotes for ACS and create RemoteArtifacts as a refresh operation.
Few questions to get my head around the proposal:</p>
<ol>
<li>What will be on the ACS model apart from presumably a FK to a Remote?</li>
</ol>
</blockquote>
<ul>
<li>Yes a FK to a Remote.</li>
<li>Also the list of base path fragments (expected to be appended to the remote.url).</li>
<li>a name</li>
</ul>
<blockquote>
<ol>
<li>Was <code>expired</code> configuration used in any way? I can imagine that it could be valuable for a regular task which refreshes outdated ACS.</li>
</ol>
</blockquote>
<p>No it was not, and the plan was for us to not implement it.</p>
<blockquote>
<ol>
<li>Was it useful to have one base url and a list of paths in Pulp 2 from the user point of view? Or was it more of a downside that to refresh one path, I would refresh all for its specific ACS?<br>
My concern is if the common workflow is to configure and/or refresh ACSs per base_url, then with the current proposal in Pulp 3 it will be a lot of work for a user to do so.</li>
</ol>
</blockquote>
<p>Agreed. It was useful. I'm modifying the ticket to have a list of "base path fragments"</p>
<blockquote>
<ol>
<li>Similar concern when I want to remove an ACS. I'm switching off my local server and I want to make sure it's no longer configured as an ACS. If we have one Remote per ACS, so in pulp 2 terms, it will be base_url + a path, how user can identify all ACSs for the server they want to shut down?</li>
</ol>
</blockquote>
<p>Yes let's use one ACS with multiple paths.</p>
<blockquote>
<ol>
<li>What happens when I configure Remotes and ACSs for the first time. I haven't run sync yet, so I have no content in Pulp which Remotes refer to. If we need to create RemoteArtifact, then we'll need to create Content and ContenArtifacts for all the content possible.</li>
</ol>
</blockquote>
<p>I was hoping to have the ACS refresh only create RemoteArtifacts without content, and @dkliban gave suggested the downloaders could match the RemoteArtifact by the checksum of the thing being downloaded. The issue with this approach is that RemoteArtifact requires ContentArtifact which in turn requires Content. I need to think this part over some more. What do you think?</p>
<blockquote>
<ol>
<li>RemoteArtifact creation will cover Content with Artifacts. What happens with other content? Is it always synced from the authoritative source since all the logic which source to choose happens in the downloader for artifacts only?</li>
</ol>
</blockquote>
<p>Yes I think this can only apply to artifacts because that is the only "binary data" there is, all the other data I think falls into the metadata category at which point the ACS should not be more authoritative than the remote url. What do you think?</p>
<blockquote>
<ol>
<li>
<code>It's like an on-demand sync in the sense that when called it indexes the remote metadata and creates remote artifacts.</code> What is implied by the indexing of remote metadata? I read the proposal as the refresh action creates RemoteArtifacts and that's it.</li>
</ol>
</blockquote>
<p>That's the idea, is the refresh only generates RemoteArtifacts, and then when subsequent sync's occur these RemoteArtifacts are "preferred".</p> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=653942020-12-07T12:53:14Zipanova@redhat.comipanova@redhat.com
<ul></ul><p>bmbouter wrote:</p>
<blockquote>
<p>ttereshc wrote:</p>
<blockquote>
<p>Thanks a lot for the background and use cases, it helps to efficiently refresh how this feature worked in Pulp 2.</p>
<p>I think it's a good idea to use remotes for ACS and create RemoteArtifacts as a refresh operation.
Few questions to get my head around the proposal:</p>
<ol>
<li>What will be on the ACS model apart from presumably a FK to a Remote?</li>
</ol>
</blockquote>
<ul>
<li>Yes a FK to a Remote.</li>
<li>Also the list of base path fragments (expected to be appended to the remote.url).</li>
<li>a name</li>
</ul>
</blockquote>
<p>And <code>headers</code>. These are needed for the case when RHUI is set as ACS content provider <a href="https://pulp.plan.io/issues/1282#note-15" class="external">https://pulp.plan.io/issues/1282#note-15</a></p>
<blockquote>
<blockquote>
<ol>
<li>Was <code>expired</code> configuration used in any way? I can imagine that it could be valuable for a regular task which refreshes outdated ACS.
No it was not, and the plan was for us to not implement it.</li>
</ol>
</blockquote>
<blockquote>
<ol>
<li>Was it useful to have one base url and a list of paths in Pulp 2 from the user point of view? Or was it more of a downside that to refresh one path, I would refresh all for its specific ACS?<br>
My concern is if the common workflow is to configure and/or refresh ACSs per base_url, then with the current proposal in Pulp 3 it will be a lot of work for a user to do so.
Agreed. It was useful. I'm modifying the ticket to have a list of "base path fragments"</li>
</ol>
</blockquote>
</blockquote>
<p>Will <code>refresh</code> command deal with the outdated content as well? We need a way to also be able to stay up-to-date with the available content and purge the one which is not longer available.</p>
<blockquote>
<blockquote>
<ol>
<li>Similar concern when I want to remove an ACS. I'm switching off my local server and I want to make sure it's no longer configured as an ACS. If we have one Remote per ACS, so in pulp 2 terms, it will be base_url + a path, how user can identify all ACSs for the server they want to shut down?
Yes let's use one ACS with multiple paths.</li>
</ol>
</blockquote>
<blockquote>
<ol>
<li>What happens when I configure Remotes and ACSs for the first time. I haven't run sync yet, so I have no content in Pulp which Remotes refer to. If we need to create RemoteArtifact, then we'll need to create Content and ContenArtifacts for all the content possible.
I was hoping to have the ACS refresh only create RemoteArtifacts without content, and @dkliban gave suggested the downloaders could match the RemoteArtifact by the checksum of the thing being downloaded. The issue with this approach is that RemoteArtifact requires ContentArtifact which in turn requires Content. I need to think this part over some more. What do you think?</li>
</ol>
</blockquote>
</blockquote>
<p>I think one of the primary use cases of setting up an ACS is to quickly populate Pulp instance with the content available locally, so I would imagine that in most of the cases a user will have setup an ACS and not yet synced repos. I think we will need to find a path to not only create RA but also Content and CA.</p>
<blockquote>
<blockquote>
<ol>
<li>RemoteArtifact creation will cover Content with Artifacts. What happens with other content? Is it always synced from the authoritative source since all the logic which source to choose happens in the downloader for artifacts only?
Yes I think this can only apply to artifacts because that is the only "binary data" there is, all the other data I think falls into the metadata category at which point the ACS should not be more authoritative than the remote url. What do you think?</li>
</ol>
</blockquote>
<blockquote>
<ol>
<li>
<code>It's like an on-demand sync in the sense that when called it indexes the remote metadata and creates remote artifacts.</code> What is implied by the indexing of remote metadata? I read the proposal as the refresh action creates RemoteArtifacts and that's it.
That's the idea, is the refresh only generates RemoteArtifacts, and then when subsequent sync's occur these RemoteArtifacts are "preferred".</li>
</ol>
</blockquote>
</blockquote>
<p>I think we should also add logic to what what happens when:</p>
<ol>
<li>ACS is deleted --> remove RAs</li>
<li>A remote that is used in ACS was deleted, make sure we don't blow up with 500</li>
<li>A remote that is used in ACS was updated ( for example the remote.url). Next <code>refresh</code> should create new RAs but also remove outdated ones.</li>
<li>Will we support update to an existing ACS, for example add/remove base_path fragments? In that case will <code>refresh</code> be automatically trigged or it will require a manual <code>refresh</code>
</li>
</ol> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=692512021-03-30T20:06:55Zbmbouterbmbouter@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/69251/diff?detail_id=69438">diff</a>)</li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=692522021-03-30T20:12:27Zbmbouterbmbouter@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/69252/diff?detail_id=69439">diff</a>)</li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=692532021-03-30T20:16:10Zbmbouterbmbouter@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/69253/diff?detail_id=69440">diff</a>)</li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=692642021-03-31T13:49:26Zbmbouterbmbouter@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/69264/diff?detail_id=69448">diff</a>)</li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=692712021-03-31T15:51:28Zbmbouterbmbouter@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/69271/diff?detail_id=69455">diff</a>)</li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=703712021-05-04T01:15:35Zjsherril@redhat.comjsherril@redhat.com
<ul></ul><p>I didn't see the main content updated for 'base path fragments'. Is that intentional? was that updated elsewhere?</p> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=706462021-05-11T15:13:53Zbmbouterbmbouter@redhat.com
<ul><li><strong>Subject</strong> changed from <i>As a user, I have Alternate Content Sources</i> to <i>[EPIC] As a user, I have Alternate Content Sources</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=738852021-07-28T15:37:53Zdalleydalley@redhat.com
<ul><li><strong>Sprint</strong> set to <i>Sprint 101</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=740762021-08-02T17:44:16Zipanova@redhat.comipanova@redhat.com
<ul><li><strong>Sprint</strong> changed from <i>Sprint 101</i> to <i>Sprint 102</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=744532021-08-12T15:23:39Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 102</i> to <i>Sprint 103</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=750052021-08-27T15:08:47Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 103</i> to <i>Sprint 104</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=754012021-09-09T22:28:07Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 104</i> to <i>Sprint 105</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=759052021-09-23T21:54:38Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 105</i> to <i>Sprint 106</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=764282021-10-08T13:16:02Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 106</i> to <i>Sprint 107</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=767132021-10-21T16:35:36Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 107</i> to <i>Sprint 108</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=769952021-11-04T21:21:46Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 108</i> to <i>Sprint 109</i></li></ul> Pulp - Story #7832: [EPIC] As a user, I have Alternate Content Sourceshttps://pulp.plan.io/issues/7832?journal_id=770722021-11-10T07:22:15Zppicka
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>CLOSED - CURRENTRELEASE</i></li></ul>