Project

Profile

Help

Issue #998

mongo cursor times out during regenerate_applicability_for_repos

Added by dkliban@redhat.com over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.4.3
Platform Release:
2.6.3
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

The regenerate_applicability_for_repos task throws OperationFailure("cursor id '179967754771123979' not valid at server",) when multiple repos and thousands of consumers are involved (3000). This is a symptom of the cursor timing out during the operation.

Associated revisions

Revision f8644708 View on GitHub
Added by dkliban@redhat.com over 6 years ago

Adds batch size to cursor used to iterate repo profile applicabilities

According to mongo documentation [0] the cursor will initially return about 101 documents or slightly more than 1 megabyte of data. The subsequent fetches return 4 times as much data. The default timeout of mongo cursor (version 2.4) is 600 seconds. This timeout cannot be adjusted until mongodb 2.6. If 400 applicability calculations need to be performed in 600 seconds, each calculation cannot take any longer than 1.5 seconds. In my testing I found the calculations to take 12 to 13 seconds. Limiting the batch size to 25 ensures that calculations can take up to 24 seconds each before the cursor times out.

[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches

https://pulp.plan.io/issues/998 fixes #998

Revision f8644708 View on GitHub
Added by dkliban@redhat.com over 6 years ago

Adds batch size to cursor used to iterate repo profile applicabilities

According to mongo documentation [0] the cursor will initially return about 101 documents or slightly more than 1 megabyte of data. The subsequent fetches return 4 times as much data. The default timeout of mongo cursor (version 2.4) is 600 seconds. This timeout cannot be adjusted until mongodb 2.6. If 400 applicability calculations need to be performed in 600 seconds, each calculation cannot take any longer than 1.5 seconds. In my testing I found the calculations to take 12 to 13 seconds. Limiting the batch size to 25 ensures that calculations can take up to 24 seconds each before the cursor times out.

[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches

https://pulp.plan.io/issues/998 fixes #998

Revision 06ea7e92 View on GitHub
Added by dkliban@redhat.com over 6 years ago

Adds batch size to cursor used to iterate repo profile applicabilities

According to mongo documentation [0] the cursor will initially return about 101 documents or slightly more than 1 megabyte of data. The subsequent fetches return 4 times as much data. The default timeout of mongo cursor (version 2.4) is 600 seconds. This timeout cannot be adjusted until mongodb 2.6. If 400 applicability calculations need to be performed in 600 seconds, each calculation cannot take any longer than 1.5 seconds. In my testing I found the calculations to take 12 to 13 seconds. Limiting the batch size to 25 ensures that calculations can take up to 24 seconds each before the cursor times out.

[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches

https://pulp.plan.io/issues/998 fixes #998

Revision 06ea7e92 View on GitHub
Added by dkliban@redhat.com over 6 years ago

Adds batch size to cursor used to iterate repo profile applicabilities

According to mongo documentation [0] the cursor will initially return about 101 documents or slightly more than 1 megabyte of data. The subsequent fetches return 4 times as much data. The default timeout of mongo cursor (version 2.4) is 600 seconds. This timeout cannot be adjusted until mongodb 2.6. If 400 applicability calculations need to be performed in 600 seconds, each calculation cannot take any longer than 1.5 seconds. In my testing I found the calculations to take 12 to 13 seconds. Limiting the batch size to 25 ensures that calculations can take up to 24 seconds each before the cursor times out.

[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches

https://pulp.plan.io/issues/998 fixes #998

Revision 46ac0e54 View on GitHub
Added by dkliban@redhat.com over 6 years ago

This reduces the batch size to 5

25 was still causing timeouts for some users. 10 worked, so 5 seems like a safe bet.

https://pulp.plan.io/issues/998 fixes #998

Revision 46ac0e54 View on GitHub
Added by dkliban@redhat.com over 6 years ago

This reduces the batch size to 5

25 was still causing timeouts for some users. 10 worked, so 5 seems like a safe bet.

https://pulp.plan.io/issues/998 fixes #998

History

#1 Updated by dkliban@redhat.com over 6 years ago

  • Description updated (diff)

#3 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dkliban@redhat.com
  • Platform Release set to 2.6.3
  • Triaged changed from No to Yes

#4 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from ASSIGNED to POST

So far I have only been able to reproduce by setting the batch size to 2 and adding a sleep of 310 seconds into the loop. This way the second fetch occurs 620 seconds after the first one. This cause the OperationFailure exception to be raised.

I am now generating initial applicability for 3000 machines. Once that is done I will run the task of regenerating applicability for repos. I expect the failure to occur then.

#6 Updated by dkliban@redhat.com over 6 years ago

After running some experiments, I have determined that it takes roughly 12 to 13 seconds to perform an applicability calculation for each profile related to a repo. The first batch that the cursor retrieves consists of 91 documents. This behavior is consistent with the MongoDB documentation [0]. As a result the second fetch occurs more than 18 minutes later. Reducing the batch size to 25 allows calculations to take up to 24 seconds each.

[0] http://docs.mongodb.org/v2.4/core/cursors/#cursor-batches

#7 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#10 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from MODIFIED to 5

#11 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from 5 to 6

#12 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#15 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF