Project

Profile

Help

Issue #4263

closed

Tasks Stuck in Running

Added by quantumfoam about 4 years ago. Updated almost 4 years ago.

Status:
CLOSED - DUPLICATE
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Master
Platform Release:
OS:
CentOS 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Quarter:

Description

My infrastructure has three independent Pulp servers, one of which is the "Master" and is where new packages are uploaded and promoted, as well as the point where external repositories are synced down from the internet. The two other Pulp servers reside in different data centers, and sync nightly from this master server. All three servers are virtual machines running CentOS 7.2 on KVM hosts. They have 8 vCPUs and 32GB of memory each, and they serve both mirrors of external repositories (e.g. CentOS, EPEL, etc) and internal-only repositories for our custom developed packages. Each data center is pointed to its appropriate Pulp server for Yum using DNS.

I recently upgraded all three servers to Pulp 2.18. The Master server is operating normally, but the two "Slave" servers seem to be unable to complete Sync tasks. The tasks will queue up, but they never complete. As a result, the Pulp server enters what I would call a death spiral where tasks build up endlessly and system resources are consumed by MongoDB.

The tasks that are stuck in a Running state typically look like the following:

[root@app01.pulp.prod.sea02 jrogers] # pulp-admin tasks details --task-id 17979b7c-eb2c-4779-9d51-9429239408eb
--------------------------------------------------------------------
Task Details
--------------------------------------------------------------------

Operations: sync
Resources: signalsciences-6-unstable (repository)
State: Running
Start Time: 2018-12-11T17:00:17Z
Finish Time: Incomplete
Result: Incomplete
Task Id: 17979b7c-eb2c-4779-9d51-9429239408eb
Worker Name:
Progress Report:
Yum Importer:
Comps:
State: NOT_STARTED
Content:
Details:
Drpm Done: 0
Drpm Total: 0
Rpm Done: 0
Rpm Total: 0
Error Details:
Items Left: 0
Items Total: 0
Size Left: 0
Size Total: 0
State: FINISHED
Distribution:
Error Details:
Items Left: 0
Items Total: 0
State: FINISHED
Errata:
State: NOT_STARTED
Metadata:
State: FINISHED
Modules:
State: IN_PROGRESS
Purge Duplicates:
State: NOT_STARTED

Here is some output from /var/log/messages (all Pulp logging is set up to go to syslog) regarding the above task:

Dec 11 16:54:54 app01 pulp: celery.app.trace:INFO: [03c5521c] Task pulp.server.tasks.repository.sync_with_auto_publish[03c5521c-7365-40e6-a8fc-108da065e1fe] succeeded in 0.0105378520093s: <AsyncResult: 17979b7c-eb2c-4779-9d51-9429239408eb>
Dec 11 17:00:17 app01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[17979b7c-eb2c-4779-9d51-9429239408eb]
Dec 11 17:00:17 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Downloading metadata from http://yum.rsglab.com//signalsciences/6/x86_64/unstable/.
Dec 11 17:00:18 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Parsing metadata.
Dec 11 17:00:18 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Downloading metadata from http://yum.rsglab.com//signalsciences/6/x86_64/unstable/.
Dec 11 17:00:19 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Parsing metadata.
Dec 11 17:00:19 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Downloading metadata from http://yum.rsglab.com//signalsciences/6/x86_64/unstable/.
Dec 11 17:00:20 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Parsing metadata.
Dec 11 17:00:20 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Downloading metadata from http://yum.rsglab.com//signalsciences/6/x86_64/unstable/.
Dec 11 17:00:21 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Parsing metadata.
Dec 11 17:00:21 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Downloading metadata files.
Dec 11 17:00:22 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Generating metadata databases.
Dec 11 17:00:22 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Determining which units need to be downloaded.
Dec 11 17:00:27 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Downloading 0 RPMs.
Dec 11 17:00:27 app01 pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: [17979b7c] The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: http://yum.rsglab.com//signalsciences/6/x86_64/unstable/
Dec 11 17:00:27 app01 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [17979b7c] Removing missing units.
Dec 11 17:00:28 app01 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [17979b7c] Downloading additional units.
Dec 11 17:00:31 app01 pulp: pulp.server.async.tasks:INFO: [e5e8b4ce] Task failed : [17979b7c-eb2c-4779-9d51-9429239408eb] : Worker terminated abnormally while processing task 17979b7c-eb2c-4779-9d51-9429239408eb. Check the logs for details
Dec 11 20:48:22 app01 pulp: pulp.server.async.tasks:INFO: Task canceled: 17979b7c-eb2c-4779-9d51-9429239408eb.

Note that in this case, there was no change to the upstream repository (yum.rsglab.com points to the Master pulp server), and so there was nothing to do but the task failed anyway. Also of note is that although the log says the task was canceled, it was still in the queue.


Related issues

Has duplicate RPM Support - Issue #4333: repo syncs fail causing worker to crash with /lib64/libmodulemd.so.1: undefined symbol: g_log_structured_standardCLOSED - CURRENTRELEASEActions

Also available in: Atom PDF