Debian Support

It is possible that the bionic apt-client requires the presence of InRelease files by default. (While the Xenial apt-client is more forgiving.)

If that is the case you have two options: Either configure the apt-clients on your Bionic machines to not require InRelease files, or configure pulp_deb to publish InRelease files. How to achieve this is documented here: https://github.com/pulp/pulp_deb/blob/2-master/README.md

(See the "InRelease file signing" heading).

I don't believe so. If you read above, a workaround we have in place is using nginx on a different port to serve the same repository structure that pulp is publishing. This of course does not have an InRelease file and works fine from Bionic. So there is some difference at play.

I am a bit stumped. The "Connection failed" and 404 errors suggest some kind of networking issue perhaps?
Can you try configuring your setup to publish InRelease files, and see if it works then?

See the bottom of: https://github.com/pulp/pulp_deb/blob/2-master/README.md

Yeah it is quite strange especially since Xenial has no issues at all. I am going to carve out some more time after the US holiday to dig into this a bit more and test the InRelease config. I will update with my additional findings afterward. Thanks for keeping this issue open in the meantime!

I think I have a simillar or the same problem.

My repository is on the right place and is accessible over the webbrowser:

root@lxpulp# pulp-admin deb repo list --repo-id=ubuntu-bionic-main --details
+----------------------------------------------------------------------+
                          Debian Repositories
+----------------------------------------------------------------------+

Id:                   ubuntu-bionic-main
Display Name:         None
Description:          None
Content Unit Counts:  
  Deb:           6391
  Deb Component: 1
  Deb Release:   1
Notes:                
Scratchpad:           
Importers:            
  Config:               
    Architectures:  amd64
    Components:     main
    Feed:           https://ftp.gwdg.de/pub/linux/debian/ubuntu
    Releases:       bionic
    SSL Validation: False
  Id:                   deb_importer
  Importer Type Id:     deb_importer
  Last Override Config: 
  Last Sync:            2019-07-26T15:29:49Z
  Last Updated:         2019-07-26T13:21:23Z
  Repo Id:              ubuntu-bionic-main
  Scratchpad:           None
Distributors:         
  Auto Publish:         True
  Config:               
    Http:         False
    Https:        True
    Relative URL: ubuntu
  Distributor Type Id:  deb_distributor
  Id:                   deb_distributor
  Last Override Config: 
  Last Publish:         2019-07-26T15:47:35Z
  Last Updated:         2019-07-26T13:21:23Z
  Repo Id:              ubuntu-bionic-main

The Problem is the following:

root@linux:~# apt-get update
Err:1 https://lxpulp.MYDOMAIN.de/pulp/deb/ubuntu bionic InRelease
  Connection failed [IP: 172.16.11.150 443]
Reading package lists... Done
W: Failed to fetch https://lxpulp.MYDOMAIN.de/pulp/deb/ubuntu/dists/bionic/InRelease  Connection failed [IP: 172.16.11.150 443]
W: Some index files failed to download. They have been ignored, or old ones used instead.

Normally the repo hasn't an InRelease file as mentioned above and I created one as described here:
https://github.com/pulp/pulp_deb/blob/2-master/README.md

Unfortunately this doesn't help, same error as before.

Then I noticed some strange behaviour:

When I try this

root@linux:~# apt-get update

I see in the apache logs on the pulp server

172.16.11.88 - - [29/Jul/2019:14:23:13 +0200] "GET /pulp/deb/ubuntu/dists/bionic/InRelease HTTP/1.1" 403 272

When I try it with wget it works:

root@linux:~# wget https://lxpulp.MYDOMAIN.de/pulp/deb/ubuntu/dists/bionic/InRelease
Connecting to lxpulp.MYDOMAIN.de (lxpulp.MYDOMAIN.de)|172.16.11.150|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2362 (2.3K) [application/octet-stream]
Saving to: 'InRelease'

InRelease                              100%[===============================================================================>]   2.31K  --.-KB/s    in 0s      

2019-07-29 13:48:03 (188 MB/s) - 'InRelease' saved [2362/2362]

and get in the logs:

172.16.11.88 - - [29/Jul/2019:15:48:03 +0200] "GET /pulp/deb/ubuntu/dists/bionic/InRelease HTTP/1.1" 200 2362

Any suggestions?

Thanks!

I want provide additional infos.

Everything works when I use http connections.
Access via https fails with the following apache messages:

[Tue Jul 30 11:14:04.987106 2019] [ssl:debug] [pid 5123] ssl_engine_kernel.c(583): [client 172.16.11.88:40152] AH02255: Changed client verification type will force renegotiation
[Tue Jul 30 11:14:04.987135 2019] [ssl:info] [pid 5123] [client 172.16.11.88:40152] AH02221: Requesting connection re-negotiation
[Tue Jul 30 11:14:04.987164 2019] [ssl:debug] [pid 5123] ssl_engine_kernel.c(783): [client 172.16.11.88:40152] AH02260: Performing full renegotiation: complete handshake protocol (client does support secure renegotiation)
[Tue Jul 30 11:14:04.987272 2019] [ssl:info] [pid 5123] [client 172.16.11.88:40152] AH02226: Awaiting re-negotiation handshake
[Tue Jul 30 11:14:24.989430 2019] [reqtimeout:info] [pid 5123] [client 172.16.11.88:40152] AH01382: Request body read timeout
[Tue Jul 30 11:14:24.989523 2019] [ssl:error] [pid 5123] [client 172.16.11.88:40152] AH02261: Re-negotiation handshake failed

In my apt.conf I have the following without any effect

Acquire::https {
        Verify-Peer "false";
        Verify-Host "false";
}

with curl I get a similar output and it works fine:

[Tue Jul 30 11:31:19.803862 2019] [ssl:debug] [pid 5123] ssl_engine_kernel.c(583): [client 172.16.11.88:40238] AH02255: Changed client verification type will force renegotiation
[Tue Jul 30 11:31:19.803891 2019] [ssl:info] [pid 5123] [client 172.16.11.88:40238] AH02221: Requesting connection re-negotiation
[Tue Jul 30 11:31:19.803911 2019] [ssl:debug] [pid 5123] ssl_engine_kernel.c(783): [client 172.16.11.88:40238] AH02260: Performing full renegotiation: complete handshake protocol (client does support secure renegotiation)
[Tue Jul 30 11:31:19.803974 2019] [ssl:info] [pid 5123] [client 172.16.11.88:40238] AH02226: Awaiting re-negotiation handshake
[Tue Jul 30 11:31:19.804444 2019] [ssl:debug] [pid 5123] ssl_engine_kernel.c(1891): [client 172.16.11.88:40238] AH02043: SSL virtual host for servername lxpulp.localdomain found

Are there any configuration options for the apt-client which handle this case?

I do not see, where there should be a difference with curl.

Can you have a look, at the permissions of that file compared to the Release file next to it? Do you have selinux enabled?
Can you access the Release file via https without curl?

The permissions of the files are fine, selinux is disabled.

I can download all release files and .deb packages within the webbrowser, wget, curl and httpie, no problem.

With apt-get or aptitude the ssl re-negotiation handshake always failed with:

[Wed Jul 31 12:51:20.658550 2019] [ssl:debug] [pid 7780] ssl_engine_kernel.c(583): [client 172.16.11.88:51436] AH02255: Changed client verification type will force renegotiation
[Wed Jul 31 12:51:20.658550 2019] [ssl:info] [pid 7780] [client 172.16.11.88:51436] AH02221: Requesting connection re-negotiation
[Wed Jul 31 12:51:20.658599 2019] [ssl:debug] [pid 7780] ssl_engine_kernel.c(783): [client 172.16.11.88:51436] AH02260: Performing full renegotiation: complete handshake protocol (client does support secure renegotiation)
[Wed Jul 31 12:51:20.658657 2019] [ssl:info] [pid 7780] [client 172.16.11.88:51436] AH02226: Awaiting re-negotiation handshake
[Wed Jul 31 12:51:40.672500 2019] [reqtimeout:info] [pid 7780] [client 172.16.11.88:51436] AH01382: Request body read timeout
[Wed Jul 31 12:51:40.672621 2019] [ssl:error] [pid 7780] [client 172.16.11.88:51436] AH02261: Re-negotiation handshake failed
[Wed Jul 31 12:51:40.672728 2019] [ssl:debug] [pid 7780] ssl_engine_io.c(1202): (70014)End of file found: [client 172.16.11.88:51436] AH02007: SSL handshake interrupted by system [Hint: Stop button pressed in browser?!]
[Wed Jul 31 12:51:40.672753 2019] [ssl:info] [pid 7780] [client 172.16.11.88:51436] AH01998: Connection closed to child 4 with abortive shutdown (server lxpulp.localdomain:443)

I think it's not a pulp or apache related problem because rpm based repositorys via https are also working fine.
The only suspect here is the reqtimeout and could be apache related:
https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1833039
But I use a CentOS 7.6 with other openssl versions.

I guess it must be something with the ssl support in the apt-client or self-signed ssl certifications ( unlikely ).
Unfortunately I could get no proper debug messages with apt-get or aptitude.

For now, because it's in our internal network and only public packages are used, I will use http instead.

Maybe I will test it with a Debian in the future.

Thanks!

Behavior recreated against bionic-to-centos76. Pointing bionic at a 'standard' public Ubuntu repo that supports HTTPS (e.g. mirrors.bloomu.edu ) works. Noting that bionic added TLS1.3 support, which CentOS7.6 (and Xenial) openssl does not have - but investigating the handshake to the 'standard' repo shows it successfully falling back to New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384 .

As noted in #c7 and #c9, the AH02261: Re-negotiation handshake failed is the core of the problem - apache is asking apt-get to renegotiate the protocol/cipher to use for SSL, and it appears the client fails to respond and apache closes the connection. Note the consistent 20-sec pause between AH02226: Awaiting re-negotiation handshake and AH01382: Request body read timeout in the log-snippet in #c9, I see the exact same pause-before-failure on my reproducer.

Will see if I can figure out exactly what's going on here, or at least a workaround in apache's settings; investigation continues.

(NB: useful commands, for anyone playing at home and for future-me:

• openssl ciphers -v shows all the ciphers/protocols available to openssl on a machine.
• openssl s_client -debug -connect <pulp-host>:<port> shows the client-ssl-conversation.
• nmap --script ssl-enum-ciphers -p 443 <pulp-host> shows ciphers the pulp Apache is willing to talk)

I have done a lot of experimentation the last day or two in httpd-conf, openssl-conf, and apt-client-conf, without being able to figure out exactly what's going on here or a way to work around the issue. To reiterate the experiences of various folks in the comments:

• apt-get between xenial and pulp2-on-centos-7.6 works fine
• curl, wget, and openssl s_client between bionic or disco, and pulp2-on-centos, all work fine
• dnf, curl, wget, and openssl s_client between fedora30 and pulp2-on-centos all work fine
• apt-get between bionic or disco, and pulp2-on-centos, fails
• presence/absence of InRelease is irrelevant - the failure happens 'inside' apache, long before it asks pulp for anything
• my comments RE TLS1.3 are irrelevant - F28 doesn't have TLS1.3 support either, and that works

On F28, we have the following RPMs for httpd and openssl:

• httpd-2.4.39-1.1.fc28.x86_64
• openssl-1.1.0i-1.fc28.x86_64

On Centos7.6, the following:

• httpd-2.4.6-89.el7.centos.1.x86_64
• openssl-1.0.2k-16.el7_6.1.x86_64

As noted in #c9, ubuntu issue 1833039 def appears to be relevant. Given that everything other than apt-get works between bionic/disco, and httpd-2.4.6-89, it feels to me like a combination of something in the depths of apt-get's SSL handling, and apache's behavior as described in '3039, is...suboptimal.

There's no code-change pulp can make here - we're not even getting to pulp-code in this failure state. I've tried to come up some combination of httpd/openssl/apt config options that gets us around the issue, without success.

I am currently out of ideas for getting us past whatever this problem is. I'll leave this issue open in case anyone has a brilliant insight.