Project

Profile

Help

Issue #9631

Updated by fao89 6 months ago

 

 **Ticket moved to GitHub**: "pulp/pulp_rpm/2308":https://github.com/pulp/pulp_rpm/issues/2308 




 ---- 


 Hi All 

 Pulp 3.16.1 on docker. 

 The Sync from remote never completes. 
 If you could please assist. 

 We are seeing Ansible error: 

 ~~~ 
 An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Exception: Task failed to complete. (failed; A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/src/kernel-source-4.12.14-122.98.1.src.rpm failed validation due to checksum.) 
 failed: [localhost] (item={'name': 'SLES12-SP5-Updates', 'backend': 'https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/', 'base': 'SLE/SLES12/SP5/updates'}) => {"ansible_index_var": "s_idx", "ansible_loop_var": "item", "changed": false, "item": {"backend": "https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/", "base": "SLE/SLES12/SP5/updates", "name": "SLES12-SP5-Updates"}, "msg": "Task failed to complete. (failed; A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/src/kernel-source-4.12.14-122.98.1.src.rpm failed validation due to checksum.)", "s_idx": 3} 


 ~~~ 
 On remote: rmt.abcgroup.net 
 # sysctl -a|grep aio 
 fs.aio-max-nr = 65536 
 fs.aio-nr = 2080 

 On pulp server 
 # sysctl -a|grep aio 
 fs.aio-max-nr = 65536 
 fs.aio-nr = 2 


 ================ 
 Attached is the Docker pulp log. 
 docker logs pulp  

 aio reported errors for various syncs(tries) of the same repo. 

 The sha256 for the RPMs are correct on the remote server's repodata. 

 ~~~ 
 Backing off download_wrapper(...) for 0.7s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [6efc07543b0c4821922029cf5cda3e90]: backoff:INFO: Backing off download_wrapper(...) for 0.7s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 

 Backing off download_wrapper(...) for 3.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [6efc07543b0c4821922029cf5cda3e90]: backoff:INFO: Backing off download_wrapper(...) for 3.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 0.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 1.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 3.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 3.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 1.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.1s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 1.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 1.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 2.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [896ae2851414447db9e177f69c677185]: backoff:INFO: Backing off download_wrapper(...) for 2.5s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.8s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 0.4s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 3.8s (aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected) 
 pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 3.8s (aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected) 
 Backing off download_wrapper(...) for 1.2s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 1.2s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 6.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 6.6s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Backing off download_wrapper(...) for 6.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:INFO: Backing off download_wrapper(...) for 6.3s (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 pulp [5f8f6f6814aa494da5817dc65d65dca2]: backoff:ERROR: Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed) 
 ~~~ 

 -------- 


 The failures reported soon after the reported aio errors: (see attached log file) 


 ~~~ 
 pulp [2fa84b10f57c41f39635eddffc89ee0a]: pulpcore.tasking.pulpcore_worker:INFO: Task 086ebba6-1f81-471c-9f60-47eda47969a6 failed () 
 pulp [2fa84b10f57c41f39635eddffc89ee0a]: pulpcore.tasking.pulpcore_worker:INFO:     File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task 
 -- 
 pulp [6efc07543b0c4821922029cf5cda3e90]: pulpcore.tasking.pulpcore_worker:INFO: Task c375f81f-7b64-4356-983b-b8729048094f failed (A file located at the url https://rmt.abcgroup.net/repo/SUSE/Updates/SLE-SERVER/12-SP5/x86_64/update/noarch/kernel-source-azure-4.12.14-16.76.2.noarch.rpm failed validation due to checksum.) 
 pulp [6efc07543b0c4821922029cf5cda3e90]: pulpcore.tasking.pulpcore_worker:INFO:     File "/usr/local/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task 
 ~~~ 
 -- 
 ------------ 

 Installation information: 

 ~~~ 
 pulp rpm remote show --name "remote_SLES12-SP5-Updates" 
 "download_concurrency": 2,    (tried 1, 4, 10 - same result) 
 ~~~ 

 pulp --version 
 pulp3 command line interface, version 0.12.0 

 pip freeze 
 appdirs==1.4.3 
 asn1crypto==0.24.0 
 Babel==2.8.0 
 beautifulsoup4==4.10.0 
 bs4==0.0.1 
 certifi==2021.5.30 
 cffi==1.13.2 
 chardet==4.0.0 
 click==8.0.1 
 colorama==0.4.4 
 commonmark==0.9.1 
 contextlib2==21.6.0 
 cryptography==2.8 
 dataclasses==0.8 
 httpie==1.0.3 
 idna==2.10 
 importlib-metadata==4.6.1 
 iotop==0.6 
 isc==2.0 
 Jinja2==2.10.1 
 MarkupSafe==1.0 
 packaging==20.3 
 pip-search==0.0.7 
 ply==3.10 
 pulp-cli==0.12.0 
 py==1.8.1 
 pyasn1==0.4.2 
 pycparser==2.17 
 Pygments==2.6.1 
 pyOpenSSL==19.0.0 
 pyparsing==2.4.7 
 pytz==2020.5 
 PyYAML==5.4.1 
 requests==2.25.1 
 rich==10.15.2 
 schema==0.7.4 
 six==1.14.0 
 soupsieve==2.3.1 
 toml==0.10.2 
 typing-extensions==3.10.0.0 
 urllib3==1.26.6 
 zipp==3.5.0 


 pulp status 
   "versions": [ 
     { 
       "component": "core", 
       "version": "3.16.1" 
     }, 
     { 
       "component": "rpm", 
       "version": "3.16.1" 
     }, 
     { 
       "component": "python", 
       "version": "3.5.2" 
     }, 
     { 
       "component": "file", 
       "version": "1.10.1" 
     }, 
     { 
       "component": "deb", 
       "version": "2.16.0" 
     }, 
     { 
       "component": "container", 
       "version": "2.9.1" 
     }, 
     { 
       "component": "certguard", 
       "version": "1.5.1" 
     }, 
     { 
       "component": "ansible", 
       "version": "0.10.1" 
     } 

 Docker server 
 ~~~ 
 cat /etc/os-release 
 NAME="SLES" 
 VERSION="15-SP3" 
 VERSION_ID="15.3" 
 PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3" 
 ~~~ 

 Thanks so much for the support. 
 Jan 

Back