Project

Profile

Help

Issue #1722

OSTree sync test fails

Added by dkliban@redhat.com almost 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - OSTree:
Platform Release:
Target Release - OSTree:
1.1.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

While running pulp-smash tests for OSTree, I see the following output. As

pulp.server.async.tasks:INFO: Task failed : [52eee922-6b24-497e-b473-27d9197a0bf4]
celery.worker.job:ERROR: (13311-23488) Task pulp.server.managers.repo.sync.sync[52eee922-6b24-497e-b473-27d9197a0bf4] raised unexpected: LibError("GLib.Error('Failed to parse url 'd02d8fc9-28cf-4ace-bdc3-424b746ba01f'', 'g-io-error-quark', 0)",)
celery.worker.job:ERROR: (13311-23488) Traceback (most recent call last): 
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
celery.worker.job:ERROR: (13311-23488)     R = retval = fun(*args, **kwargs)
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
celery.worker.job:ERROR: (13311-23488)     return super(Task, self).__call__(*args, **kwargs)
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
celery.worker.job:ERROR: (13311-23488)     return super(PulpTask, self).__call__(*args, **kwargs)
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
celery.worker.job:ERROR: (13311-23488)     return self.run(*args, **kwargs)
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 757, in sync
celery.worker.job:ERROR: (13311-23488)     sync_report = sync_repo(transfer_repo, conduit, call_config)
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
celery.worker.job:ERROR: (13311-23488)     return f(*args, **kwargs)
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/web.py", line 91, in sync_repo
celery.worker.job:ERROR: (13311-23488)     report = step.process_lifecycle()
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
celery.worker.job:ERROR: (13311-23488)     super(PluginStep, self).process_lifecycle() 
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
celery.worker.job:ERROR: (13311-23488)     step.process()
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 249, in process
celery.worker.job:ERROR: (13311-23488)     self._process_block()
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 293, in _process_block
celery.worker.job:ERROR: (13311-23488)     self.process_main()
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/steps.py", line 116, in process_main
celery.worker.job:ERROR: (13311-23488)     refs = [r.dict() for r in remote.list_refs()]
celery.worker.job:ERROR: (13311-23488)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/lib.py", line 30, in _fn
celery.worker.job:ERROR: (13311-23488)     raise LibError(repr(ge))
celery.worker.job:ERROR: (13311-23488) LibError: GLib.Error('Failed to parse url 'd02d8fc9-28cf-4ace-bdc3-424b746ba01f'', 'g-io-error-quark', 0)
celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[406afdba-e6ba-4290-a63c-49763aa660e2] succeeded in 0.014632966002s: None

Pulp-smash produces the following traceback:

Traceback (most recent call last):
  File "/home/jenkins/workspace/pulp-smash-runner/pulp-smash/pulp_smash/tests/ostree/api_v2/test_sync_publish.py", line 201, in setUpClass
    repo_href, cls.report, cls.tasks = _create_sync_repo(cls.cfg, body)
  File "/home/jenkins/workspace/pulp-smash-runner/pulp-smash/pulp_smash/tests/ostree/api_v2/test_sync_publish.py", line 89, in _create_sync_repo
    tasks = tuple(api.poll_spawned_tasks(server_config, response.json()))
  File "/home/jenkins/workspace/pulp-smash-runner/pulp-smash/pulp_smash/api.py", line 316, in poll_spawned_tasks
    for final_task_state in poll_task(server_config, href):
  File "/home/jenkins/workspace/pulp-smash-runner/pulp-smash/pulp_smash/api.py", line 353, in poll_task
    'Task {} is ongoing after {} polls.'.format(href, poll_limit)
TaskTimedOutError: Task /pulp/api/v2/tasks/903f697c-7e59-4348-9a44-9a99041b0541/ is ongoing after 24 polls.

Associated revisions

Revision b83c33d6 View on GitHub
Added by jortel@redhat.com almost 6 years ago

Better handling of GLib.Error raised when fetching the summary. closes #1722

History

#1 Updated by dkliban@redhat.com almost 6 years ago

I don't like that the task status does not go into a final state after the exception is raised. That's why the test times out checking the status of the task. We should look into this also.

#2 Updated by mhrivnak almost 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com
  • Triaged changed from No to Yes

#4 Updated by jortel@redhat.com almost 6 years ago

There are two problems here. The GLib.Error is fixed by the PR. The second is a problem in pulp-smash. the api.poll_task() has a hard coded poll_limit of 24 (120 seconds). Recommend not hard coding the poll_limit and update the test to sync a smaller ostree repository. I would be glad to compose a small one but not sure where to put it for the smash tests to use. Looks like poll_limit needs to be around 96 for this test to work on my laptop.

#5 Updated by bmbouter almost 6 years ago

I propose we create a separate fedorapeople URL to house all test and demo repos used by Pulp. We never should have dual-purposed the usage of our production bits we distribute with with testing/demo repos.

#6 Updated by jortel@redhat.com almost 6 years ago

agreed. Thinking that storing the test repositories in pulp-smash might be better. That way pulp-smash can be more self contained.

#7 Updated by jortel@redhat.com almost 6 years ago

  • Status changed from ASSIGNED to MODIFIED
  • % Done changed from 0 to 100

#8 Updated by rbarlow almost 6 years ago

On Mon, Feb 29, 2016 at 07:12:17PM +0100, Brian wrote:

I propose we create a separate fedorapeople URL to house all test and demo repos used by Pulp. We never should have dual-purposed the usage of our production bits we distribute with with testing/demo repos.

We may be able to use facls to grant access to the test data
area of our repo to the team without compromising the production area.

#9 Updated by jortel@redhat.com almost 6 years ago

An option for OSTree tests is to generate the upstream repo on demand. I know that sounds heavy but with OSTree it's not. With a few shell commands we can create a repository containing several trees in < 50 ms.

#10 Updated by dkliban@redhat.com almost 6 years ago

@jortel, would we be able to guarantee that they look exactly like a real tree served by Fedora?

#11 Updated by jortel@redhat.com almost 6 years ago

wrote:

@jortel, would we be able to guarantee that they look exactly like a real tree served by Fedora?

Yes.

#12 Updated by jortel@redhat.com almost 6 years ago

Decided to post a small test repo in fedorapeople and use that for the test.

https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/test-ostree-small/

pending: https://github.com/PulpQE/pulp-smash/pull/147

#13 Updated by dkliban@redhat.com almost 6 years ago

  • Status changed from MODIFIED to 5

#14 Updated by semyers over 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#15 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF