Issue #998
closed
mongo cursor times out during regenerate_applicability_for_repos
Description
The regenerate_applicability_for_repos task throws OperationFailure("cursor id '179967754771123979' not valid at server",) when multiple repos and thousands of consumers are involved (3000). This is a symptom of the cursor timing out during the operation.
Updated by dkliban@redhat.com almost 9 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to dkliban@redhat.com
- Platform Release set to 2.6.3
- Triaged changed from No to Yes
Updated by dkliban@redhat.com almost 9 years ago
- Status changed from ASSIGNED to POST
So far I have only been able to reproduce by setting the batch size to 2 and adding a sleep of 310 seconds into the loop. This way the second fetch occurs 620 seconds after the first one. This cause the OperationFailure exception to be raised.
I am now generating initial applicability for 3000 machines. Once that is done I will run the task of regenerating applicability for repos. I expect the failure to occur then.
Updated by dkliban@redhat.com almost 9 years ago
Updated by dkliban@redhat.com almost 9 years ago
After running some experiments, I have determined that it takes roughly 12 to 13 seconds to perform an applicability calculation for each profile related to a repo. The first batch that the cursor retrieves consists of 91 documents. This behavior is consistent with the MongoDB documentation [0]. As a result the second fetch occurs more than 18 minutes later. Reducing the batch size to 25 allows calculations to take up to 24 seconds each.
[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches
Added by dkliban@redhat.com almost 9 years ago
Added by dkliban@redhat.com almost 9 years ago
Revision f8644708 | View on GitHub
Adds batch size to cursor used to iterate repo profile applicabilities
According to mongo documentation [0] the cursor will initially return about 101 documents or slightly more than 1 megabyte of data. The subsequent fetches return 4 times as much data. The default timeout of mongo cursor (version 2.4) is 600 seconds. This timeout cannot be adjusted until mongodb 2.6. If 400 applicability calculations need to be performed in 600 seconds, each calculation cannot take any longer than 1.5 seconds. In my testing I found the calculations to take 12 to 13 seconds. Limiting the batch size to 25 ensures that calculations can take up to 24 seconds each before the cursor times out.
[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches
Updated by dkliban@redhat.com almost 9 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|f8644708e1ed15dc2d4b04f4edd77eb7bc873963.
Added by dkliban@redhat.com almost 9 years ago
Revision 06ea7e92 | View on GitHub
Adds batch size to cursor used to iterate repo profile applicabilities
According to mongo documentation [0] the cursor will initially return about 101 documents or slightly more than 1 megabyte of data. The subsequent fetches return 4 times as much data. The default timeout of mongo cursor (version 2.4) is 600 seconds. This timeout cannot be adjusted until mongodb 2.6. If 400 applicability calculations need to be performed in 600 seconds, each calculation cannot take any longer than 1.5 seconds. In my testing I found the calculations to take 12 to 13 seconds. Limiting the batch size to 25 ensures that calculations can take up to 24 seconds each before the cursor times out.
[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches
Added by dkliban@redhat.com almost 9 years ago
Revision 06ea7e92 | View on GitHub
Adds batch size to cursor used to iterate repo profile applicabilities
According to mongo documentation [0] the cursor will initially return about 101 documents or slightly more than 1 megabyte of data. The subsequent fetches return 4 times as much data. The default timeout of mongo cursor (version 2.4) is 600 seconds. This timeout cannot be adjusted until mongodb 2.6. If 400 applicability calculations need to be performed in 600 seconds, each calculation cannot take any longer than 1.5 seconds. In my testing I found the calculations to take 12 to 13 seconds. Limiting the batch size to 25 ensures that calculations can take up to 24 seconds each before the cursor times out.
[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches
Updated by dkliban@redhat.com almost 9 years ago
Applied in changeset pulp|06ea7e921a437b24211ed77f972eb3b2e36e51c5.
Added by dkliban@redhat.com almost 9 years ago
Revision 46ac0e54 | View on GitHub
This reduces the batch size to 5
25 was still causing timeouts for some users. 10 worked, so 5 seems like a safe bet.
Added by dkliban@redhat.com almost 9 years ago
Revision 46ac0e54 | View on GitHub
This reduces the batch size to 5
25 was still causing timeouts for some users. 10 worked, so 5 seems like a safe bet.
Updated by dkliban@redhat.com almost 9 years ago
Applied in changeset pulp|46ac0e5486ea698d6100f5aa9d8ffc107bab909a.
Updated by dkliban@redhat.com almost 9 years ago
- Status changed from MODIFIED to 5
Updated by dkliban@redhat.com over 8 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
.
Adds batch size to cursor used to iterate repo profile applicabilities
According to mongo documentation [0] the cursor will initially return about 101 documents or slightly more than 1 megabyte of data. The subsequent fetches return 4 times as much data. The default timeout of mongo cursor (version 2.4) is 600 seconds. This timeout cannot be adjusted until mongodb 2.6. If 400 applicability calculations need to be performed in 600 seconds, each calculation cannot take any longer than 1.5 seconds. In my testing I found the calculations to take 12 to 13 seconds. Limiting the batch size to 25 ensures that calculations can take up to 24 seconds each before the cursor times out.
[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches
https://pulp.plan.io/issues/998 fixes #998