Pulp

Tried installing qpid in FIPS enabled centos7 and qpid service is running but qpid service fails when its installed with katello, I guess there might be an integration issue.

@amit, what error are you seeing? We've not run into any qpid problems in our pulp dev environment.

When pulp is integrated with katello while installing Katello on FIPS enabled machine following are the observations:

'qpid-config --ssl-certificate /etc/pki/katello/certs/centos7-fips.ak.example.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/centos7-fips.ak.example.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' returned 1 instead of one of [0]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:164:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:257:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:171:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:156:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:186:in `block (2 levels) in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:369:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:237:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:211:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:354:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:661:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

I don't think this has anything to do with Katello. This is probably related to using SSL connections in a FIPS enabled environment.

Pulp QE is currently working on getting Pulp tested with an SSL connection to Qpid. I'll report here when they have some results.

From dalley:

There may or may not be an error in the certificate generation steps. It seems like the Qpid certificates mentioned in the logs from the previous post are successfully created, but creating the Katello CA does not work, and that also happens in the "Certs::Qpid" stage.

It could be unrelated but should be looked into as a possible reason.

[ WARN 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Qpid/Certs::Keypair[qpid]/Privkey[/etc/pki/katello/private/centos7-fips-katello-nightly.localhost.example.com-qpid-broker.key]/ensure: created
[ WARN 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Qpid/Certs::Keypair[qpid]/Pubkey[/etc/pki/katello/certs/centos7-fips-katello-nightly.localhost.example.com-qpid-broker.crt]/ensure: created
[ INFO 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Qpid/Certs::Keypair[qpid]/Pubkey[/etc/pki/katello/certs/centos7-fips-katello-nightly.localhost.example.com-qpid-broker.crt]: Scheduling refresh of Certs::Ssltools::Certutil[broker]
[ WARN 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Qpid/Certs::Keypair[qpid]/File[/etc/pki/katello/private/centos7-fips-katello-nightly.localhost.example.com-qpid-broker.key]/group: group changed 'root' to 'qpidd'
[ WARN 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Qpid/Certs::Keypair[qpid]/File[/etc/pki/katello/private/centos7-fips-katello-nightly.localhost.example.com-qpid-broker.key]/mode: mode changed '0400' to '0440'
[ WARN 2018-06-14T14:32:18 verbose]  /File[/etc/pki/katello/private/centos7-fips-katello-nightly.localhost.example.com-qpid-broker.key]/seluser: seluser changed 'unconfined_u' to 'system_u'
[ INFO 2018-06-14T14:32:18 verbose]  Certs::Keypair[qpid]: Scheduling refresh of Class[Certs::Ssltools::Nssdb]
[ INFO 2018-06-14T14:32:18 verbose]  Class[Certs::Ssltools::Nssdb]: Scheduling refresh of Package[openssl]
[ INFO 2018-06-14T14:32:18 verbose]  Class[Certs::Ssltools::Nssdb]: Scheduling refresh of Package[nss-tools]
[ INFO 2018-06-14T14:32:18 verbose]  Class[Certs::Ssltools::Nssdb]: Scheduling refresh of Exec[generate-nss-password]
[ INFO 2018-06-14T14:32:18 verbose]  Class[Certs::Ssltools::Nssdb]: Scheduling refresh of Exec[create-nss-db]
[ WARN 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/Package[openssl]: Triggered 'refresh' from 1 event
[ WARN 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/Package[nss-tools]: Triggered 'refresh' from 1 event
[ WARN 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb]/ensure: created
[ WARN 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/Exec[generate-nss-password]/returns: executed successfully
[ WARN 2018-06-14T14:32:18 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/Exec[generate-nss-password]: Triggered 'refresh' from 1 event
[ WARN 2018-06-14T14:32:18 verbose]  /File[/etc/pki/katello/nssdb/nss_db_password-file]/seluser: seluser changed 'unconfined_u' to 'system_u'
[ WARN 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/Exec[create-nss-db]/returns: executed successfully
[ WARN 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/Exec[create-nss-db]: Triggered 'refresh' from 1 event
[ WARN 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb/cert8.db]/mode: mode changed '0600' to '0640'
[ WARN 2018-06-14T14:32:19 verbose]  /File[/etc/pki/katello/nssdb/cert8.db]/seluser: seluser changed 'unconfined_u' to 'system_u'
[ WARN 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb/key3.db]/mode: mode changed '0600' to '0640'
[ WARN 2018-06-14T14:32:19 verbose]  /File[/etc/pki/katello/nssdb/key3.db]/seluser: seluser changed 'unconfined_u' to 'system_u'
[ WARN 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Ssltools::Nssdb/File[/etc/pki/katello/nssdb/secmod.db]/mode: mode changed '0600' to '0640'
[ WARN 2018-06-14T14:32:19 verbose]  /File[/etc/pki/katello/nssdb/secmod.db]/seluser: seluser changed 'unconfined_u' to 'system_u'
[ INFO 2018-06-14T14:32:19 verbose]  Class[Certs::Ssltools::Nssdb]: Scheduling refresh of Certs::Ssltools::Certutil[ca]
[ INFO 2018-06-14T14:32:19 verbose]  Certs::Ssltools::Certutil[ca]: Scheduling refresh of Exec[delete ca]
[ INFO 2018-06-14T14:32:19 verbose]  Certs::Ssltools::Certutil[ca]: Scheduling refresh of Exec[ca]
[ WARN 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[delete ca]: Triggered 'refresh' from 1 event
[ WARN 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]/returns: Notice: Trust flag u is set automatically if the private key is present.
[ WARN 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]/returns: Error opening input terminal for read
[ WARN 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]/returns: certutil: could not authenticate to token NSS FIPS 140-2 Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect.
[ERROR 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]: Failed to call refresh: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'ca' -t 'TCu,Cu,Tuw' -a -i '/etc/pki/katello/certs/katello-default-ca.crt'' returned 255 instead of one of [0]
[ERROR 2018-06-14T14:32:19 verbose]  /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'ca' -t 'TCu,Cu,Tuw' -a -i '/etc/pki/katello/certs/katello-default-ca.crt'' returned 255 instead of one of [0]
[ERROR 2018-06-14T14:32:19 verbose] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'

Hello,

Your observations are right. creating the Katello CA does not work and Qpid fails to respond in FIPS generated SSL environment. I am a bit layman in working with SSL issues, it would be nice if any workaround suggested for this meanwhile the issue gets fixed.

Thanks.

@amitkarsale, I think you're running into this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1552159

Looks like there's a fix in the bug that involves using --empty-password.

@daviddavis I have already modified the `--empty-password` attribute that made me move ahead of the mentioned bug from the BZ but stuck to the current `qpid-config --ssl-certtificate` issue. : /

What issue are you seeing with `qpid-config --ssl-certtificate`?

comment #3 (https://pulp.plan.io/issues/3645#note-3) has the stack-trace where executing `qpid-config--ssl-certtificate` fails.