https://pulp.plan.io/https://pulp.plan.io/favicon.ico2017-11-27T22:00:27ZPulpInfrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=229592017-11-27T22:00:27Zbmbouterbmbouter@redhat.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/22959/diff?detail_id=23447">diff</a>)</li></ul> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=229602017-11-27T22:01:30Zdkliban@redhat.com
<ul><li><strong>Groomed</strong> changed from <i>No</i> to <i>Yes</i></li></ul> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=229942017-11-28T22:14:51Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li><li><strong>Assignee</strong> set to <i>bmbouter</i></li></ul><p>I wrote a plan in the form of checklist items.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=229952017-11-28T22:25:10Zbmbouterbmbouter@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/13762">duck</a> I have some questions.</p>
<p>I had generated this public key before, did it get used for anything? Is this the key we will use to push the docs with?</p>
<pre><code>ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNfEAXdhT7mIf4d+XOWvE4hw5uduVyqc1N3ttqHV/5SM37XWHvln9D0IzMtDllXrmvlFuUA8sUCnc3zw3s+zIFh0ln+nhEQAo3StvchmlzmYJD1VyS1e8Pfd9xnD8DOBQNdy3aq4ZVrLtCAYndfX1Y5sfqlgN35OQ+7frjKaJXQWVHuLDGTxWR/y82EOT5JyXPyZd6B9VsQRPmB5oK1eSqMNQikuXJITDRxK6olfIdLzze85gV0aRJYo6GD5Aj68SciIsR95y//afwOQZr8m7OjygTsxLTlzsoRVR2N4uTyOe2i2szqy9kxUnvTiTVDULWicQZLBmv7S1pjaytKJnj bmbouter@localhost.localdomain
</code></pre>
<p>What username should be used to push these docs?</p>
<p>Are we are going to host these docs on the same host as pulpproject.org ( 8.43.85.236 )?</p>
<p>The docs are built on Jenkins which in on the RH network w/ a 10.x.y.z address, can it reach 8.43.85.236? Do we need a jumphost of some kind?</p>
<p>I can paste the various kinds of commands the rsync does when it publishes so that we can restrict the commands that are rsync commands the host will accept. How do you all plan to configure that restriction?</p>
<p>Also if we could have similar http -> https rollover and also TLS as we do for docs.pulpproject.org that would be cool.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=230092017-11-29T03:54:29Zduck
<ul></ul><p>This key in installed at the end of 'playbooks/deploy.yml' and allowed for running rsync only for security reasons. The user is defined in the same block of rules and called 'doc_builder'.</p>
<p>As it is the same kind of content. static website, and as I think the VM can handle both traffic well enough, I initially decided to host both at the same place. 8.43.85.236 is already configured for the doc vhost.</p>
<p>8.43.85.236 is a directly reachable public address. In the future we would like to move web machine behind a reverse proxy whenever possible (to save IPs and for security), but this is not yet built.</p>
<p>TLS and redirection is already prepared, (commented in the same block of rules) We can only activate it during the migration when the DNS is switched for Let's Encrypt to validate the vhost.</p>
<p>So the next step would be to publish to the new vhost, which is already ready. You can modify your local '/etc/hosts' locally to check the resulting website, similarly as we did for the main website.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=230242017-11-29T18:51:12Zrchan
<ul><li><strong>Sprint/Milestone</strong> set to <i>48</i></li></ul> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=231942017-12-05T21:28:49Zbmbouterbmbouter@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/13762">duck</a>, thanks. I want to check my understanding so I can get the rsync commands working. I see that the key sets command to run:</p>
<pre><code>rsync --server -vlogtrze.isf --delete-after --omit-dir-times . /var/www/docs.pulpproject.org
</code></pre>
<p>I can ssh using the corresponding private key with:</p>
<pre><code>ssh -i ~/.ssh/pulp-infra doc_builder@8.43.85.236
</code></pre>
<p>When I run that in the home directory of where the content lives I think it's going to upload everything from the current directory <code>.</code> to <code>/var/www.docs.pulpproject.org</code> for serving.</p>
<p>The docs builder only builds a part of the site each time, and it needs to be able to run several types of rsync commands. It runs these things:</p>
<pre><code>rsync -avzh --delete --exclude en /some/local/path /var/www/docs.pulpproject.org/some/deeper/path
rsync -avzh --rsync-path path_option_arg='mkdir -p /var/www/docs.pulpproject.org/en/nightly/ && rsync' --delete /some/local/path /var/www/docs.pulpproject.org/some/deeper/path
</code></pre>
<p>What are our options to have the ansible playbook be updated to allow those types of commands? Maybe we could use sudo to allow any command that starts with <code>rsync *</code> or similar?</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=232272017-12-11T12:07:07Zduck
<ul></ul><p>Well, this is not a limitation on Ansible's side. This is nore what kind of setup is possible with rsync and SSH. Hardcoding the options is better for security; so if the build is compromised, only the content of the website and no other data can be, which might lead to compromission too.</p>
<p>So either we allow to pass an option, less secure, or we add a different SSH key with a different setup for syncing the different parts. We might also use a wrapper command checking its options carefully.</p>
<p>Misc, any opinion?</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=232322017-12-11T20:58:02Zbmbouterbmbouter@redhat.com
<ul></ul><p>What about using a new, dedicated user to have the content pushed as to achieve that kind of isolation. This would discontinue the use of the <code>command</code> ssh option, and instead use POSIX permissions. The POSIX user level perms would allow this content user to write to /var/www/docs.pulpproject.org/ and its home directory only, so the rsync options won't provide much additional security.</p>
<p>The explicitly coded use of <code>command</code> is more secure in the sense that you can't run a kernel exploit against it, but that type of restriction impacts the schedule and flexibility some. The old site is being turned off on (dec 31), so we need to do something soon. Also there are some needs of docs maintainers.</p>
<p>These options may change at times as part of normal maintenance/development. Having an exact, specific whitelist commands probably won't give the flexibility that the maintainers of the build scripts (a group of 4 devs) probably practically desire. With an explicit whitelist of commands approach, we need to get the infra folks to modify and push new ssh rules with normal maintenance changes which is undesirable.</p>
<p>Given all ^ what should we do?</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=232482017-12-12T14:05:25Zduck
<ul></ul><p>Leaving full shell access would really not be secure, but that's a possibility if we cannot be on time. In this case we could restrict the source IP.</p>
<p>In the meanwhile we have a little bit more time, so I remembered the RSSH tool could help. I made the changes (not pushed in the repo yet) and deployed on the server. You should be able to use rsync with any options. Could you test with the new configuration?</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=235162017-12-19T16:18:07Zrchan
<ul><li><strong>Sprint/Milestone</strong> changed from <i>48</i> to <i>52</i></li></ul> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=235222017-12-19T16:31:50Zbmbouterbmbouter@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/13762">duck</a> Sorry for the long delay. Thanks for updating the config. I tested it, but now I get a:</p>
<pre><code>/usr/bin/rssh: Permission denied
</code></pre>
<p>I get this when I run with a local copy of all of the website content at <code>/home/bmbouter/Documents/site_backup_12_5_2017/diy/</code></p>
<pre><code>rsync -avzh /home/bmbouter/Documents/site_backup_12_5_2017/diy/ doc_builder@8.43.85.236:/var/www/docs.pulpproject.org/
</code></pre>
<p>I also get the rssh denied if I try regular ssh with:</p>
<pre><code>ssh -i ~/path/the/correct/private/key doc_builder@8.43.85.236
</code></pre>
<p>Is there a permissions issue? Thanks for whatever you can do to resolve it!</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=235582017-12-20T06:15:03Zduck
<ul></ul><p>On RedHat systems the user needs to be in a special group to be allowed to use RSSH, additionally to configuration. I updated the Ansible rules, tested with my key and was able to send a test file. It should work now.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=235592017-12-20T11:19:09Zbmbouterbmbouter@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/13762">duck</a>, Thanks! Yes this is letting me push files now, and I pushed a basic backup of the site. I just proofed the site and it looks good. My next step is to update the docs builders on Jenkins so they push to this new environment. I'm going to do that now.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=235622017-12-20T16:31:59Zbmbouterbmbouter@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/13762">duck</a>, I got the Jenkins builders pushing to the new environment. Here are the two new things I'm blocked on. Thank you so much for everything you've done to help so far!</p>
<a name="Symlink-Woes"></a>
<h2 >Symlink Woes<a href="#Symlink-Woes" class="wiki-anchor">¶</a></h2>
<p>The last step publishes with a symlink and "rssh" prevents that from happening. We use symlinks because we want to lay out all of the content indexed by the x.y version and have it stored that way permanently. Meanwhile the 'latest' version needs to point to one or another and be changed over time. That motivates the symlink design. Currently <a href="https://github.com/pulp/pulp-ci/blob/master/ci/docs-builder.py#L179" class="external">this line</a> fails with:</p>
<pre><code>This account is restricted by rssh.
Allowed commands: rsync
If you believe this is in error, please contact your system administrator.
</code></pre>
<p>Maybe we could add <code>ln</code> as a whitelisted command? Or whatever resolution you think would be good.</p>
<a name="Permission-problem"></a>
<h2 >Permission problem<a href="#Permission-problem" class="wiki-anchor">¶</a></h2>
<p>Rsync fails with as error: <code>rsync: failed to set times on "/var/www/docs.pulpproject.org/.": Operation not permitted (1)</code></p>
<p>I applied a workaround to my rsync commands to add a <code>--omit-dir-times</code>, but that also means that rsync can't set the times for directories inside of the directory structure.</p>
<p>Does it make any sense tchange the POSIX permission on <code>/var/lib/docs.pulpproject.org/</code> to allow that timestamp to be written.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=235632017-12-20T17:03:33Zduck
<ul></ul><p>It's not possible to add commands to RSSH. It is made to know specific commands and filter parameters to avoid security problems, so it would need a serious patch of RSSH itself, not just config.</p>
<p>I would suggest this you create a new local directory with the proper relative symlink (which does not need to point to anything exiting locally) and then rsync it to push the symlink to the right destination directory.</p>
<p>The rsync error is not a permission problem on this directory:</p>
<pre><code>drwxrwxr-x. 8 root doc_builder 4096 Dec 20 11:00 /var/www/docs.pulpproject.org
</code></pre>
<p>To be able to change this timestamp on the top directory you would need to have permissions over the parent directory (/var/www), meaning you could destroy the underlying directory structure by mistake (and Apache would not be very happy). Unfortunately there is no option in rsync to skip setting the timestamp only to the top directory.</p>
<p>--omit-dir-times only affects directories, so is this really needed? Who is gonna use these timestamps?<br>
If really needed then I see no other option than changing the permissions on /var/www.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=235952017-12-30T04:32:50Zbmbouterbmbouter@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/13762">duck</a>, thanks for the ideas. I wrote a workaround which uses rsync instead of symlinks and also rsync instead of mkdir. Also thanks for reminding me how POSIX dir write perms work. :) Yes, let's make no changes and I'll continue to carry the <code>--omit-dir-times</code> where necessary.</p>
<p>I tested the builders, and everything looks like it's ready to go live. The only issue is that after we update the DNS of docs.pulpproject.org to be 8.43.85.236, we will need to have it's letsencrypt cert renewed. Currently I'm getting an error when I try to fetch content from it. Specifically I get a:</p>
<pre><code>The owner of docs.pulpproject.org has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.
This site uses HTTP Strict Transport Security (HSTS) to specify that Firefox may only connect to it securely. As a result, it is not possible to add an exception for this certificate.
</code></pre>
<p>If you are able to take the site live as soon as possible that would be ideal.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=235962017-12-30T20:13:15Zbmbouterbmbouter@redhat.com
<ul></ul><p>I think we're all set to migrate. I just handled some final things. I just took a backup of the existing site and stored it safely offline. I also announced that we will migrate soon via pulp-dev <a href="https://www.redhat.com/archives/pulp-dev/2017-December/msg00084.html" class="external">here</a> I also mirrored the entire site to the new environment so that all existing content is ready to go.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=237582018-01-08T21:27:11Zrchan
<ul><li><strong>Sprint/Milestone</strong> changed from <i>52</i> to <i>53</i></li></ul> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=237682018-01-09T08:44:57Zduck
<ul></ul><p>I migrated the docs website and activated let's encrypt.</p>
<p>I always discovered a redirection for a old blog vhost in the DNS, which was broken since we activated HSTS (done by default in the 'httpd' role for security) and redirect.redhat.com does not hold certificates for Pulp vhosts. So I created a new vhost on the web VM to do the redirect and switched the DNS accordingly.</p>
<p>Everything seems to work fine.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=237972018-01-09T18:46:00Zbmbouterbmbouter@redhat.com
<ul></ul><p><a class="user active" href="https://pulp.plan.io/users/13762">duck</a>, thank you so much! Indeed the site is working excellently. The blog.pulpproject.org is our old site, so it's great that continues to redirect to pulpproject.org.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=238662018-01-10T17:34:20Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>CLOSED - COMPLETE</i></li></ul><p>I archived and deleted the old site, and I'm sending the announcement now. This site is done!</p>
<p><a class="user active" href="https://pulp.plan.io/users/13762">duck</a> thank you so much! We really, really appreciate the help and support.</p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=238672018-01-10T17:37:14Zbmbouterbmbouter@redhat.com
<ul></ul><p>Announcement is here: <a href="https://www.redhat.com/archives/pulp-dev/2018-January/msg00031.html" class="external">https://www.redhat.com/archives/pulp-dev/2018-January/msg00031.html</a></p> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=263472018-03-08T23:30:32Zbmbouterbmbouter@redhat.com
<ul><li><strong>Sprint</strong> set to <i>Sprint 31</i></li></ul> Infrastructure - Task #3153: Migrate the docs.pulpproject.org site from Openshift V2 to an OSCI environmenthttps://pulp.plan.io/issues/3153?journal_id=263722018-03-08T23:31:03Zbmbouterbmbouter@redhat.com
<ul><li><strong>Sprint/Milestone</strong> deleted (<del><i>53</i></del>)</li></ul>