Additional observations:
[root@bumblebee ~]# pulp-admin tasks list | grep Waiting | wc -l
5
[root@bumblebee ~]# pulp-admin tasks list | grep Running | wc -l
0
[root@bumblebee ~]# pulp-admin tasks list | less
--------------------------------------------------------------------------
Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: 19832c86-bff5-4bba-a2c8-900678c612f6
Operations: sync
Resources: nux-dextop-el7-x86_64 (repository)
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: b621280e-75b5-42eb-9e2c-c2e3f8a23cfa
Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: e3cdff0d-0cf8-4924-92ae-20a6b5bdf5d0
Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: fb368d3f-4de1-4dc8-ac57-074947ee6af6
Operations: reaper
Resources:
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: c3bdcb56-01b7-45f4-97f1-da7849a118b6
--------------------------------------------------------------------------
What are all those reapers waiting for?
What happens if I cancel them?
--------------------------------------------------------------------------
[root@bumblebee ~]# pulp-admin tasks cancel --task-id=19832c86-bff5-4bba-a2c8-900678c612f6
Task cancel is successfully initiated.
[root@bumblebee ~]# pulp-admin tasks cancel --task-id=e3cdff0d-0cf8-4924-92ae-20a6b5bdf5d0
Task cancel is successfully initiated.
[root@bumblebee ~]# pulp-admin tasks cancel --task-id=fb368d3f-4de1-4dc8-ac57-074947ee6af6
Task cancel is successfully initiated.
[root@bumblebee ~]# pulp-admin tasks cancel --task-id=c3bdcb56-01b7-45f4-97f1-7849a118b6
Task cancel is successfully initiated.
[root@bumblebee ~]# pulp-admin tasks list | grep Waiting | wc -l
1
[root@bumblebee ~]# pulp-admin tasks list | grep Running | wc -l
0
[root@bumblebee ~]# pulp-admin tasks list
Operations: sync
Resources: nux-dextop-el7-x86_64 (repository)
State: Waiting
Start Time: Unstarted
Finish Time: Incomplete
Task Id: b621280e-75b5-42eb-9e2c-c2e3f8a23cfa
--------------------------------------------------------------------------
So now I know that there is only the sync tasks waiting, and that nothing else is running.
Does re-starting pulp have any effect?
pulp-stop <- this is taking a very long time to complete.
pulp-start
The pulp-stop and pulp-start commands are scripts as follows:
--------------------------- pulp-stop ---------------------------------------
[root@bumblebee ~]# cat /usr/local/bin/pulp-stop
#!/bin/bash
- pulp-stop
- stops the pulp daemons
- Complete stop/start also includes httpd (between pulp_worksrs and mongod)
- This is not included here as it interrupts customer service.
SERVICES=(pulp_resource_manager pulp_celerybeat pulp_workers mongod)
for ((i=0; i<${#SERVICES[@]}; ++i ))
do
systemctl stop ${SERVICES[i]}
done
-----------------------------------------------------------------------------
--------------------------- pulp-start --------------------------------------
[root@bumblebee ~]# cat /usr/local/bin/pulp-start
#!/bin/bash
- pulp-start
- starts up the pulp daemons
- Complete stop/start also includes httpd (between pulp_worksrs and mongod)
- This is not included here as it interrupts customer service.
SERVICES=(pulp_resource_manager pulp_celerybeat pulp_workers mongod)
for ((i=${#SERVICES[@]}-1; i>=0; --i ))
do
systemctl start ${SERVICES[i]} || exit 1
done
-----------------------------------------------------------------------------
Hmmm... Now things are busy
[root@bumblebee ~]# pulp-admin tasks list | grep Running | wc -l
3
[root@bumblebee ~]# pulp-admin tasks list | grep Waiting | wc -l
525
It looks like it has decided to sync the whole world after I re-started pulp. That was unexpected.
Lets stop that so that we can test the repo under investigation.
[root@bumblebee ~]# while true; do sleep 5; date; CancelAllRunningTasks; done
This calls a bash function
-----------------------------------------------------------------------------
function CancelAllRunningTasks() {
local AllTaskIds=( $(pulp-admin tasks list | perl 0777 -pe 's/Operations:[^\n]*\nResources:[^\n]*\nState: \nStart Time: +[^ \n]\nFinish Time: [^ \n]\nTask Id: ([^ \n])/\2/igs' | grep -e "^[a-f0-9\]\+\$") )
local i
for ((i=0; i<${#AllTaskIds[@]}; ++i ))
do
pulp-admin tasks cancel --task-id ${AllTaskIds[i]}
done
}
-----------------------------------------------------------------------------
When this finally finishes killing tasks, we are ready to proceed.
Now attempt to sync the problem repo.
-----------------------------------------------------------------------------
[root@bumblebee ~]# pulp-admin rpm repo sync run --repo-id=nux-dextop-el7-x86_64
--------------------------------------------------------------------
Synchronizing Repository [nux-dextop-el7-x86_64]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[-]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 117/117 items
Delta RPMs: 0/0 items
... completed
Individual package errors encountered during sync:
Package:
http://mirror.li.nux.ro/li.nux.ro/nux//dextop/el7/x86_64/mythtv-debuginfo-0.27.1
-3.el7.nux.x86_64.rpm
Error:
An unexpected error has occurred. More information can be found in the client
log file ~/.pulp/admin.log.
[root@bumblebee ~]# less ~/.pulp/admin.log
2015-06-02 19:38:49,172 - ERROR - Client-side exception occurred
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/pulp/client/extensions/core.py", line 478, in run
exit_code = Cli.run(self, args)
File "/usr/lib/python2.7/site-packages/okaara/cli.py", line 974, in run
exit_code = command_or_section.execute(self.prompt, remaining_args)
File "/usr/lib/python2.7/site-packages/pulp/client/extensions/extensions.py", line 224, in execute
return self.method(*arg_list, **clean_kwargs)
File "/usr/lib/python2.7/site-packages/pulp/client/commands/repo/sync_publish.py", line 126, in run
self.poll([sync_task], kwargs)
File "/usr/lib/python2.7/site-packages/pulp/client/commands/polling.py", line 120, in poll
task = self._poll_task(task)
File "/usr/lib/python2.7/site-packages/pulp/client/commands/polling.py", line 211, in _poll_task
self.progress(task, running_spinner)
File "/usr/lib/python2.7/site-packages/pulp/client/commands/repo/sync_publish.py", line 71, in progress
self.renderer.display_report(task.progress_report)
File "/usr/lib/python2.7/site-packages/pulp_rpm/extensions/admin/status.py", line 72, in display_report
self.render_download_step(progress_report)
File "/usr/lib/python2.7/site-packages/pulp_rpm/extensions/admin/status.py", line 249, in render_download_step
'name': error[constants.NAME],
KeyError: 'name'
-----------------------------------------------------------------------------
The sync has run without getting stuck, but something else has gone wrong as well
Observations:
1) Something is going wrong with pulp such that synchs can get stuck. Re-starting pulp seems to fix it.
2) Observed a KeyError .
So it seems that something in pulp's internal state is getting upset, and re-setting pulp fixes it.