Infrastructure

duck I have some questions.

I had generated this public key before, did it get used for anything? Is this the key we will use to push the docs with?

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNfEAXdhT7mIf4d+XOWvE4hw5uduVyqc1N3ttqHV/5SM37XWHvln9D0IzMtDllXrmvlFuUA8sUCnc3zw3s+zIFh0ln+nhEQAo3StvchmlzmYJD1VyS1e8Pfd9xnD8DOBQNdy3aq4ZVrLtCAYndfX1Y5sfqlgN35OQ+7frjKaJXQWVHuLDGTxWR/y82EOT5JyXPyZd6B9VsQRPmB5oK1eSqMNQikuXJITDRxK6olfIdLzze85gV0aRJYo6GD5Aj68SciIsR95y//afwOQZr8m7OjygTsxLTlzsoRVR2N4uTyOe2i2szqy9kxUnvTiTVDULWicQZLBmv7S1pjaytKJnj bmbouter@localhost.localdomain

What username should be used to push these docs?

Are we are going to host these docs on the same host as pulpproject.org ( 8.43.85.236 )?

The docs are built on Jenkins which in on the RH network w/ a 10.x.y.z address, can it reach 8.43.85.236? Do we need a jumphost of some kind?

I can paste the various kinds of commands the rsync does when it publishes so that we can restrict the commands that are rsync commands the host will accept. How do you all plan to configure that restriction?

Also if we could have similar http -> https rollover and also TLS as we do for docs.pulpproject.org that would be cool.

This key in installed at the end of 'playbooks/deploy.yml' and allowed for running rsync only for security reasons. The user is defined in the same block of rules and called 'doc_builder'.

As it is the same kind of content. static website, and as I think the VM can handle both traffic well enough, I initially decided to host both at the same place. 8.43.85.236 is already configured for the doc vhost.

8.43.85.236 is a directly reachable public address. In the future we would like to move web machine behind a reverse proxy whenever possible (to save IPs and for security), but this is not yet built.

TLS and redirection is already prepared, (commented in the same block of rules) We can only activate it during the migration when the DNS is switched for Let's Encrypt to validate the vhost.

So the next step would be to publish to the new vhost, which is already ready. You can modify your local '/etc/hosts' locally to check the resulting website, similarly as we did for the main website.

duck, thanks. I want to check my understanding so I can get the rsync commands working. I see that the key sets command to run:

rsync --server -vlogtrze.isf --delete-after --omit-dir-times . /var/www/docs.pulpproject.org

I can ssh using the corresponding private key with:

ssh -i ~/.ssh/pulp-infra doc_builder@8.43.85.236

When I run that in the home directory of where the content lives I think it's going to upload everything from the current directory . to /var/www.docs.pulpproject.org for serving.

The docs builder only builds a part of the site each time, and it needs to be able to run several types of rsync commands. It runs these things:

rsync -avzh --delete --exclude en /some/local/path /var/www/docs.pulpproject.org/some/deeper/path
rsync -avzh --rsync-path path_option_arg='mkdir -p /var/www/docs.pulpproject.org/en/nightly/ && rsync' --delete /some/local/path /var/www/docs.pulpproject.org/some/deeper/path

What are our options to have the ansible playbook be updated to allow those types of commands? Maybe we could use sudo to allow any command that starts with rsync * or similar?

Well, this is not a limitation on Ansible's side. This is nore what kind of setup is possible with rsync and SSH. Hardcoding the options is better for security; so if the build is compromised, only the content of the website and no other data can be, which might lead to compromission too.

So either we allow to pass an option, less secure, or we add a different SSH key with a different setup for syncing the different parts. We might also use a wrapper command checking its options carefully.

Misc, any opinion?

What about using a new, dedicated user to have the content pushed as to achieve that kind of isolation. This would discontinue the use of the command ssh option, and instead use POSIX permissions. The POSIX user level perms would allow this content user to write to /var/www/docs.pulpproject.org/ and its home directory only, so the rsync options won't provide much additional security.

The explicitly coded use of command is more secure in the sense that you can't run a kernel exploit against it, but that type of restriction impacts the schedule and flexibility some. The old site is being turned off on (dec 31), so we need to do something soon. Also there are some needs of docs maintainers.

These options may change at times as part of normal maintenance/development. Having an exact, specific whitelist commands probably won't give the flexibility that the maintainers of the build scripts (a group of 4 devs) probably practically desire. With an explicit whitelist of commands approach, we need to get the infra folks to modify and push new ssh rules with normal maintenance changes which is undesirable.

Given all ^ what should we do?

Leaving full shell access would really not be secure, but that's a possibility if we cannot be on time. In this case we could restrict the source IP.

In the meanwhile we have a little bit more time, so I remembered the RSSH tool could help. I made the changes (not pushed in the repo yet) and deployed on the server. You should be able to use rsync with any options. Could you test with the new configuration?

duck Sorry for the long delay. Thanks for updating the config. I tested it, but now I get a:

/usr/bin/rssh: Permission denied

I get this when I run with a local copy of all of the website content at /home/bmbouter/Documents/site_backup_12_5_2017/diy/

rsync -avzh /home/bmbouter/Documents/site_backup_12_5_2017/diy/ doc_builder@8.43.85.236:/var/www/docs.pulpproject.org/

I also get the rssh denied if I try regular ssh with:

ssh -i ~/path/the/correct/private/key doc_builder@8.43.85.236

Is there a permissions issue? Thanks for whatever you can do to resolve it!

On RedHat systems the user needs to be in a special group to be allowed to use RSSH, additionally to configuration. I updated the Ansible rules, tested with my key and was able to send a test file. It should work now.

duck, Thanks! Yes this is letting me push files now, and I pushed a basic backup of the site. I just proofed the site and it looks good. My next step is to update the docs builders on Jenkins so they push to this new environment. I'm going to do that now.

duck, I got the Jenkins builders pushing to the new environment. Here are the two new things I'm blocked on. Thank you so much for everything you've done to help so far!

Symlink Woes¶

The last step publishes with a symlink and "rssh" prevents that from happening. We use symlinks because we want to lay out all of the content indexed by the x.y version and have it stored that way permanently. Meanwhile the 'latest' version needs to point to one or another and be changed over time. That motivates the symlink design. Currently this line fails with:

This account is restricted by rssh.
Allowed commands: rsync

If you believe this is in error, please contact your system administrator.

Maybe we could add ln as a whitelisted command? Or whatever resolution you think would be good.

Permission problem¶

Rsync fails with as error: rsync: failed to set times on "/var/www/docs.pulpproject.org/.": Operation not permitted (1)

I applied a workaround to my rsync commands to add a --omit-dir-times, but that also means that rsync can't set the times for directories inside of the directory structure.

Does it make any sense tchange the POSIX permission on /var/lib/docs.pulpproject.org/ to allow that timestamp to be written.

It's not possible to add commands to RSSH. It is made to know specific commands and filter parameters to avoid security problems, so it would need a serious patch of RSSH itself, not just config.

I would suggest this you create a new local directory with the proper relative symlink (which does not need to point to anything exiting locally) and then rsync it to push the symlink to the right destination directory.

The rsync error is not a permission problem on this directory:

drwxrwxr-x. 8 root doc_builder 4096 Dec 20 11:00 /var/www/docs.pulpproject.org

To be able to change this timestamp on the top directory you would need to have permissions over the parent directory (/var/www), meaning you could destroy the underlying directory structure by mistake (and Apache would not be very happy). Unfortunately there is no option in rsync to skip setting the timestamp only to the top directory.

--omit-dir-times only affects directories, so is this really needed? Who is gonna use these timestamps?
If really needed then I see no other option than changing the permissions on /var/www.

duck, thanks for the ideas. I wrote a workaround which uses rsync instead of symlinks and also rsync instead of mkdir. Also thanks for reminding me how POSIX dir write perms work. :) Yes, let's make no changes and I'll continue to carry the --omit-dir-times where necessary.

I tested the builders, and everything looks like it's ready to go live. The only issue is that after we update the DNS of docs.pulpproject.org to be 8.43.85.236, we will need to have it's letsencrypt cert renewed. Currently I'm getting an error when I try to fetch content from it. Specifically I get a:

The owner of docs.pulpproject.org has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.

This site uses HTTP Strict Transport Security (HSTS) to specify that Firefox may only connect to it securely. As a result, it is not possible to add an exception for this certificate.

If you are able to take the site live as soon as possible that would be ideal.

I think we're all set to migrate. I just handled some final things. I just took a backup of the existing site and stored it safely offline. I also announced that we will migrate soon via pulp-dev here I also mirrored the entire site to the new environment so that all existing content is ready to go.

I migrated the docs website and activated let's encrypt.

I always discovered a redirection for a old blog vhost in the DNS, which was broken since we activated HSTS (done by default in the 'httpd' role for security) and redirect.redhat.com does not hold certificates for Pulp vhosts. So I created a new vhost on the web VM to do the redirect and switched the DNS accordingly.

Everything seems to work fine.

duck, thank you so much! Indeed the site is working excellently. The blog.pulpproject.org is our old site, so it's great that continues to redirect to pulpproject.org.

Announcement is here: https://www.redhat.com/archives/pulp-dev/2018-January/msg00031.html