Task #1333
closed
Document that syncs give up if network fails
0%
Description
As shown below, if a network error occurs during a sync and several requests fail, the sync will skip the rest of its downloads. This is good, because trying to make thousands more requests to a struggling or missing remote service could take a lot of time, could make that remove service's problems worse, and is unlikely to succeed.
It would be helpful to document that behavior. I think this logic is in nectar, so most importers will behave this way. Here is an example of that behavior "in the wild":
[root@ibm-x3550m3-06 ~]# rpm -qa pulp-server
pulp-server-2.7.0-0.7.beta.el7.noarch
[root@ibm-x3550m3-06 ~]#
[root@ibm-x3550m3-06 ~]# pulp-admin node sync run --node-id node1
This command may be exited via ctrl+c without affecting the request.
[-]
Accepted...
(1/1) Repository: centos
[==== ] 9%
Error occurred during synchronization, check the child node logs for details
+----------------------------------------------------------------------+
Child Node Synchronization
+----------------------------------------------------------------------+
Repository:
Action: Merged
Content Sources:
Downloads:
Source Id: Parent Node
Total Failed: 4611
Total Succeeded: 490
Total Sources: 0
Id: centos
Units:
Added: 490
Removed: 0
Updated: 0
+----------------------------------------------------------------------+
The following [4611] errors were reported
+----------------------------------------------------------------------+
- 01: Received error [A connection error occurred] while downloading a unit file
at URL
[https://ibm-x3550m3-06.lab.eng.brq.redhat.com/pulp/nodes/https/repos/centos/con
tent/rpm/evolution-mapi-devel/3.8.5/6.el7/i686/ec7af664520b926516f473cf086388373
14ad9d5bc2a03af79d945228d49f7ca/evolution-mapi-devel-3.8.5-6.el7.i686.rpm] for
repository [centos]. The cause could be that the repository has not been
published.
- 02: Received error [Download skipped] while downloading a unit file at URL
[https://ibm-x3550m3-06.lab.eng.brq.redhat.com/pulp/nodes/https/repos/centos/con
tent/rpm/OpenEXR/1.7.1/7.el7/x86_64/4abca5307be39571089dd481eeb307bcf29fbb473629
005cae5721079108546a/OpenEXR-1.7.1-7.el7.x86_64.rpm] for repository [centos].
The cause could be that the repository has not been published.
- 03: Received error [Download skipped] while downloading a unit file at URL
[https://ibm-x3550m3-06.lab.eng.brq.redhat.com/pulp/nodes/https/repos/centos/con
tent/rpm/texlive-charter-doc/svn15878.0/32.el7/noarch/e068aee3536f1e724610ebe09b
d9c077fa82275d147df9a45e55abead8e18ddd/texlive-charter-doc-svn15878.0-32.el7.noa
rch.rpm] for repository [centos]. The cause could be that the repository has not
been published.
- 04: Received error [Download skipped] while downloading a unit file at URL
[https://ibm-x3550m3-06.lab.eng.brq.redhat.com/pulp/nodes/https/repos/centos/con
tent/rpm/mythes-uk/1.6.5/6.el7/noarch/753a0d54ce20196ac1cc61914258f38f8f3cd13f9d
a280fc562f59c9b503062a/mythes-uk-1.6.5-6.el7.noarch.rpm] for repository
[centos]. The cause could be that the repository has not been published.
- 05: Received error [Download skipped] while downloading a unit file at URL
[https://ibm-x3550m3-06.lab.eng.brq.redhat.com/pulp/nodes/https/repos/centos/con
tent/rpm/libbluray-devel/0.2.3/5.el7/x86_64/424fbab8446ab76bab7454ddac4b98daa6aa
b71f461ec672cdaf0de84c8c99a5/libbluray-devel-0.2.3-5.el7.x86_64.rpm] for
repository [centos]. The cause could be that the repository has not been
published.
- 06: Received error [Download skipped] while downloading a unit file at URL
[https://ibm-x3550m3-06.lab.eng.brq.redhat.com/pulp/nodes/https/repos/centos/con
tent/rpm/libgnome/2.32.1/9.el7/i686/59c0d5006cc901129acecaee16f7cfdda57873d8c520
ace99c63ac6036124203/libgnome-2.32.1-9.el7.i686.rpm] for repository [centos].
The cause could be that the repository has not been published.
- 07: Received error [Download skipped] while downloading a unit file at URL
[https://ibm-x3550m3-06.lab.eng.brq.redhat.com/pulp/nodes/https/repos/centos/con
tent/rpm/alsa-plugins-vdownmix/1.0.27/3.el7/i686/ffd0f28a6540ec51b9abc6843caa1f3
a461554097b5bf49044f0f55bb8fe18ca/alsa-plugins-vdownmix-1.0.27-3.el7.i686.rpm]
for repository [centos]. The cause could be that the repository has not been
published.
Updated by pthomas@redhat.com over 8 years ago
Steps to reproduce.
1. Create & Sync a centos repo
pulp-admin rpm repo create --repo-id centos --feed http://mirror.mojohost.com/centos/7/os/x86_64/
2. Install & configure node parent & node child
3. Run node sync
Updated by dkliban@redhat.com over 8 years ago
- Tags Documentation added
I logged in to this machine and determined that the problem seems to be caused by network problem. We should document in the troubleshooting guide that a node sync will stop syncing at the first sign of network connectivity problems. However, when the node sync is run again, the sync resumes where it left off.
Updated by mhrivnak over 8 years ago
- Tracker changed from Issue to Task
- Subject changed from node sync errors when syncing big repos (centos) to Document that syncs give up if network fails
- Description updated (diff)
- Groomed set to No
- Sprint Candidate set to Yes
- Tags Easy Fix added
Updated by bmbouter about 5 years ago
- Status changed from NEW to CLOSED - WONTFIX
Updated by bmbouter about 5 years ago
Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.
.