Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #9213

closed

DNF gets wrong rpm download size, until package downloaded with curl, then works

Added by optiz0r over 3 years ago. Updated almost 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
dkliban@redhat.com
Category:
-
Sprint/Milestone:
3.16.0
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 104
Quarter:

Description

I have setup Katello with https://rpm.releases.hashicorp.com/RHEL/8/x86_64/stable/ as an on-demand RPM repo. A content host is subscribed to this via the Library. Attempting to install any package results in an error, that the downloaded RPM size did not match the expected size. This is repeatable, persists across dnf clean all, and happens on multiple hosts.

Downloading the package with wget via the http repo url may result in an error the first time, but subsequent downloads will fetch the entire file successfully. As soon as the file has been downloaded successfully once via wget, dnf will then happily install the package. The behaviour is the same for all packages within the repo.

I have done multiple "Complete syncs", "Verify content checksum", and "Regenerate repository metadata" operations, none of which have had any impact.

Two examples, with some output snipped for brevity:

[root@alma-canary ~]# yum install waypoint -y
Stable (EL8)                                                                                                         36 kB/s | 1.7 kB     00:00    
Dependencies resolved.
====================================================================================================================================================
 Package                        Architecture                 Version                        Repository                                         Size
====================================================================================================================================================
Installing:
 waypoint                       x86_64                       0.4.2-1                        sihnon_hashicorp_stable-el8                        46 M

Transaction Summary
====================================================================================================================================================
Install  1 Package

Total download size: 46 M
Installed size: 100 M
Downloading Packages:
[MIRROR] waypoint-0.4.2-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 47747976 but expected size is: 47746472       
[MIRROR] waypoint-0.4.2-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 47747976 but expected size is: 47746472       
[MIRROR] waypoint-0.4.2-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 47747976 but expected size is: 47746472       
[MIRROR] waypoint-0.4.2-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 47747976 but expected size is: 47746472       
[FAILED] waypoint-0.4.2-1.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success                                      

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
  Cannot download Packages/w/waypoint-0.4.2-1.x86_64.rpm: All mirrors were tried

[root@alma-canary ~]# yum install waypoint -y
Stable (EL8)                                                                                                         36 kB/s | 1.7 kB     00:00    
Dependencies resolved.
====================================================================================================================================================
 Package                        Architecture                 Version                        Repository                                         Size
====================================================================================================================================================
Installing:
 waypoint                       x86_64                       0.4.2-1                        sihnon_hashicorp_stable-el8                        46 M

Transaction Summary
====================================================================================================================================================
Install  1 Package

Total download size: 46 M
Installed size: 100 M
Downloading Packages:
[MIRROR] waypoint-0.4.2-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 47747976 but expected size is: 47746472       
[MIRROR] waypoint-0.4.2-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 47747976 but expected size is: 47746472       
[MIRROR] waypoint-0.4.2-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 47747976 but expected size is: 47746472       
[MIRROR] waypoint-0.4.2-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 47747976 but expected size is: 47746472       
[FAILED] waypoint-0.4.2-1.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success                                      

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
  Cannot download Packages/w/waypoint-0.4.2-1.x86_64.rpm: All mirrors were tried

https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/w/waypoint-0.4.2-1.x86_64.rpm
--2021-08-04 21:13:13--  https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/w/waypoint-0.4.2-1.x86_64.rpm
Resolving whitefall.example.com (whitefall.example.com)... 81.xx.xx.141
Connecting to whitefall.example.com (whitefall.example.com)|81.xx.xx.141|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 47747976 (46M) [binary/octet-stream]
Saving to: ‘waypoint-0.4.2-1.x86_64.rpm’

waypoint-0.4.2-1.x86_64.rpm           99%[=====================================================================> ]  45.53M  11.2MB/s    in 3.9s    

2021-08-04 21:13:18 (11.8 MB/s) - Connection closed at byte 47746472. Retrying.

--2021-08-04 21:13:19--  (try: 2)  https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/w/waypoint-0.4.2-1.x86_64.rpm
Connecting to whitefall.example.com (whitefall.example.com)|81.xx.xx.141|:443... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

    The file is already fully retrieved; nothing to do.

[root@alma-canary ~]# wget --no-check-certificate https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/w/waypoint-0.4.2-1.x86_64.rpm
--2021-08-04 21:13:35--  https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/w/waypoint-0.4.2-1.x86_64.rpm
Resolving whitefall.example.com (whitefall.example.com)... 81.xx.xx.141
Connecting to whitefall.example.com (whitefall.example.com)|81.xx.xx.141|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 47746472 (46M) [application/x-rpm]
Saving to: ‘waypoint-0.4.2-1.x86_64.rpm.1’

waypoint-0.4.2-1.x86_64.rpm.1        100%[======================================================================>]  45.53M   107MB/s    in 0.4s    

2021-08-04 21:13:36 (107 MB/s) - ‘waypoint-0.4.2-1.x86_64.rpm.1’ saved [47746472/47746472]

[root@alma-canary ~]# yum install waypoint -y
Stable (EL8)                                                                                                         32 kB/s | 1.7 kB     00:00    
Dependencies resolved.
====================================================================================================================================================
 Package                        Architecture                 Version                        Repository                                         Size
====================================================================================================================================================
Installing:
 waypoint                       x86_64                       0.4.2-1                        sihnon_hashicorp_stable-el8                        46 M

Transaction Summary
====================================================================================================================================================
Install  1 Package

Total download size: 46 M
Installed size: 100 M
Downloading Packages:
waypoint-0.4.2-1.x86_64.rpm                                                                                          63 MB/s |  46 MB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                63 MB/s |  46 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                            1/1 
  Installing       : waypoint-0.4.2-1.x86_64                                                                                                    1/1 
  Verifying        : waypoint-0.4.2-1.x86_64                                                                                                    1/1 
Installed products updated.

Installed:
  waypoint-0.4.2-1.x86_64                                                                                                                           

Complete!
[root@alma-canary ~]# yum install boundary -y
Stable (EL8)                                                                                                         49 kB/s | 1.7 kB     00:00    
Dependencies resolved.
====================================================================================================================================================
 Package                        Architecture                 Version                        Repository                                         Size
====================================================================================================================================================
Installing:
 boundary                       x86_64                       0.4.0-1                        sihnon_hashicorp_stable-el8                        18 M

Transaction Summary
====================================================================================================================================================
Install  1 Package

Total download size: 18 M
Installed size: 57 M
Downloading Packages:
[MIRROR] boundary-0.4.0-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 18515599 but expected size is: 18519432       
[MIRROR] boundary-0.4.0-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 18515599 but expected size is: 18519432       
[MIRROR] boundary-0.4.0-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 18515599 but expected size is: 18519432       
[MIRROR] boundary-0.4.0-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 18515599 but expected size is: 18519432       
[FAILED] boundary-0.4.0-1.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success                                      

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
  Cannot download Packages/b/boundary-0.4.0-1.x86_64.rpm: All mirrors were tried
[root@alma-canary ~]# yum install boundary -y
Stable (EL8)                                                                                                        6.8 kB/s | 1.7 kB     00:00    
Dependencies resolved.
====================================================================================================================================================
 Package                        Architecture                 Version                        Repository                                         Size
====================================================================================================================================================
Installing:
 boundary                       x86_64                       0.4.0-1                        sihnon_hashicorp_stable-el8                        18 M

Transaction Summary
====================================================================================================================================================
Install  1 Package

Total download size: 18 M
Installed size: 57 M
Downloading Packages:
[MIRROR] boundary-0.4.0-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 18515599 but expected size is: 18519432       
[MIRROR] boundary-0.4.0-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 18515599 but expected size is: 18519432       
[MIRROR] boundary-0.4.0-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 18515599 but expected size is: 18519432       
[MIRROR] boundary-0.4.0-1.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 18515599 but expected size is: 18519432       
[FAILED] boundary-0.4.0-1.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success                                      

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
  Cannot download Packages/b/boundary-0.4.0-1.x86_64.rpm: All mirrors were tried
[root@alma-canary ~]# wget --no-check-certificate https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/b/boundary-0.4.0-1.x86_64.rpm
--2021-08-04 21:15:36--  https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/b/boundary-0.4.0-1.x86_64.rpm
Resolving whitefall.example.com(whitefall.example.com)... 81.xx.xx.141
Connecting to whitefall.example.com (whitefall.example.com)|81.xx.xx.141|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18515599 (18M) [binary/octet-stream]
Saving to: ‘boundary-0.4.0-1.x86_64.rpm’

boundary-0.4.0-1.x86_64.rpm          100%[======================================================================>]  17.66M  12.1MB/s    in 1.5s    

2021-08-04 21:15:37 (12.1 MB/s) - ‘boundary-0.4.0-1.x86_64.rpm’ saved [18515599/18515599]

[root@alma-canary ~]# yum install boundary -y
Stable (EL8)                                                                                                        6.8 kB/s | 1.7 kB     00:00    
Dependencies resolved.
====================================================================================================================================================
 Package                        Architecture                 Version                        Repository                                         Size
====================================================================================================================================================
Installing:
 boundary                       x86_64                       0.4.0-1                        sihnon_hashicorp_stable-el8                        18 M

Transaction Summary
====================================================================================================================================================
Install  1 Package

Total download size: 18 M
Installed size: 57 M
Downloading Packages:
boundary-0.4.0-1.x86_64.rpm                                                                                          59 MB/s |  18 MB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                59 MB/s |  18 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                            1/1 
  Installing       : boundary-0.4.0-1.x86_64                                                                                                    1/1 
  Verifying        : boundary-0.4.0-1.x86_64                                                                                                    1/1 
Installed products updated.

Installed:
  boundary-0.4.0-1.x86_64                                                                                                                           

Complete!

Attmpting to download the file using chrome yields a "Failed - Network Error", until the file has been downloaded successfully via wget, and then the browser download completes successfully.

Another example, downloading vault on a PC with a different OS and more recent version of wget in quick succession yields different sizes on the two downloads.

$ wget --no-check-certificate https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/v/vault-1.8.0-1.x86_64.rpm
--2021-08-04 21:26:12--  https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/v/vault-1.8.0-1.x86_64.rpm
HTTP request sent, awaiting response... 200 OK
Length: 52581251 (50M) [binary/octet-stream]
Saving to: ‘vault-1.8.0-1.x86_64.rpm’

vault-1.8.0-1.x86_64.rpm             100%[======================================================================>]  50.14M  11.3MB/s    in 4.3s    

2021-08-04 21:26:16 (11.8 MB/s) - ‘vault-1.8.0-1.x86_64.rpm’ saved [52581251/52581251]

$ wget --no-check-certificate https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/v/vault-1.8.0-1.x86_64.rpm
--2021-08-04 21:26:30--  https://whitefall.example.com/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/v/vault-1.8.0-1.x86_64.rpm
HTTP request sent, awaiting response... 200 OK
Length: 52588104 (50M) [application/x-rpm]
Saving to: ‘vault-1.8.0-1.x86_64.rpm.1’

vault-1.8.0-1.x86_64.rpm.1           100%[======================================================================>]  50.15M  12.6MB/s    in 4.1s    

2021-08-04 21:26:35 (12.3 MB/s) - ‘vault-1.8.0-1.x86_64.rpm.1’ saved [52588104/52588104]

Versions:

[root@whitefall ~]# rpm -qa '*pulp*'
rubygem-pulp_rpm_client-3.13.3-1.el8.noarch
python3-pulp-file-1.8.1-1.el8.noarch
python3-pulp-rpm-3.14.0-1.el8.noarch
rubygem-pulp_container_client-2.7.0-1.el8.noarch
python3-pulp-cli-0.10.1-1.el8.noarch
rubygem-pulp_deb_client-2.13.0-1.el8.noarch
python3-pulp-container-2.7.0-1.el8.noarch
python3-pulpcore-3.14.3-1.el8.noarch
pulpcore-selinux-1.2.4-1.el8.x86_64
rubygem-pulp_file_client-1.8.1-1.el8.noarch
rubygem-pulp_ansible_client-0.8.0-1.el8.noarch
python3-pulp-certguard-1.4.0-1.el8.noarch
pulp-client-1.0-1.noarch
rubygem-pulp_certguard_client-1.4.0-1.el8.noarch
python3-pulp-ansible-0.8.0-1.el8.noarch
rubygem-pulpcore_client-3.14.1-1.el8.noarch
rubygem-pulp_python_client-3.4.0-1.el8.noarch
rubygem-smart_proxy_pulp-3.1.0-1.fm2_6.el8.noarch
python3-pulp-deb-2.13.0-1.el8.noarch

[root@whitefall ~]# rpm -qa '*katello*'
katello-4.2.0-0.1.master.el8.noarch
katello-debug-4.2.0-0.1.master.el8.noarch
rubygem-katello-4.2.0-0.4.pre.master.20210723175601git5f23a60.el8.noarch
katello-repos-4.2-0.2.nightly.el8.noarch
katello-server-ca-1.0-1.noarch
katello-ca-consumer-whitefall.example.com-1.0-1.noarch
katello-common-4.2.0-0.1.master.el8.noarch
katello-certs-tools-2.8.0-1.el8.noarch
foreman-installer-katello-3.0.0-0.1.develop.20210723110556gitc4bb060.el8.noarch
katello-selinux-4.0.2-1.el8.noarch
rubygem-hammer_cli_katello-1.1.2-1.el8.noarch
katello-client-bootstrap-1.7.6-1.el8.noarch
katello-default-ca-1.0-1.noarch

Related issues

Copied to Pulp - Backport #9325: Backport #9213 "DNF gets wrong rpm download size, until package downloaded with curl, then works" to 3.14.zCLOSED - CURRENTRELEASEbmbouter

Actions
Actions
Send by e-mail Copy link

Also available in: Atom PDF