Actions
Issue #3586
closed
Filtering workers is indeterministic
Added by dkliban@redhat.com about 6 years ago. Updated over 4 years ago.
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 38
Quarter:
Description
pulp-smash test to filter workers is failing on python 3.5 from time to time.
It looks like when you filter by either 'online' or 'missing' AND 'name' you can get 2 results when only 1 worker should be returned when filtering by name.
Updated by dalley about 6 years ago
It fails like 90% of the time, but it does work like 10% of the time on Python 3.5.
It works (seemingly) 100% of the time on Python 3.6
Updated by daviddavis about 6 years ago
- Triaged changed from No to Yes
- Sprint set to Sprint 36
I think the next step is to try to reproduce this locally and debug. I tried with python 3.5.4 and couldn't reproduce though. I think Travis is using python 3.5.5 so maybe that's why.
Updated by jortel@redhat.com almost 6 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to jortel@redhat.com
Updated by jortel@redhat.com almost 6 years ago
Unable to reproduce so far using:
- django-crispy-forms (1.7.2)
- django-extensions (2.0.7)
- django-filter (1.1.0)
- djangorestframework (3.8.2)
- djangorestframework-jwt (1.11.0)
Updated by dalley almost 6 years ago
- Assignee changed from jortel@redhat.com to dalley
- Tags Pulp 3 added
Updated by dalley almost 6 years ago
Reproduced:
CPython/3.5.2
=============
pip list | grep -i django
-------------------------
Django 2.0.5
django-crispy-forms 1.7.2
django-extensions 2.0.7
django-filter 1.1.0
djangorestframework 3.8.2
(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/"
{
"next": null,
"previous": null,
"results": [
{
"_href": "http://localhost:8000/pulp/api/v3/workers/267e7528-43c4-4e98-aa43-5e39e72a94c3/",
"id": "267e7528-43c4-4e98-aa43-5e39e72a94c3",
"last_heartbeat": "2018-05-15T22:23:19.825483Z",
"missing": false,
"name": "reserved_resource_worker_2@pulp3.dev",
"online": false
},
{
"_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
"id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
"last_heartbeat": "2018-05-15T22:34:23.902196Z",
"missing": false,
"name": "reserved_resource_worker_1@pulp3.dev",
"online": true
},
{
"_href": "http://localhost:8000/pulp/api/v3/workers/d5096dd8-8126-4bb6-917d-7a4ffe9932d9/",
"id": "d5096dd8-8126-4bb6-917d-7a4ffe9932d9",
"last_heartbeat": "2018-05-15T22:23:19.829143Z",
"missing": false,
"name": "resource_manager@pulp3.dev",
"online": false
}
]
}
(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?online=True"
{
"next": null,
"previous": null,
"results": [
{
"_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
"id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
"last_heartbeat": "2018-05-15T22:23:20.987260Z",
"missing": false,
"name": "reserved_resource_worker_1@pulp3.dev",
"online": true
}
]
}
(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev"
{
"next": null,
"previous": null,
"results": [
{
"_href": "http://localhost:8000/pulp/api/v3/workers/267e7528-43c4-4e98-aa43-5e39e72a94c3/",
"id": "267e7528-43c4-4e98-aa43-5e39e72a94c3",
"last_heartbeat": "2018-05-15T22:23:19.825483Z",
"missing": false,
"name": "reserved_resource_worker_2@pulp3.dev",
"online": false
}
]
}
(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev&online=True"
{
"next": null,
"previous": null,
"results": [
{
"_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
"id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
"last_heartbeat": "2018-05-15T22:23:51.110808Z",
"missing": false,
"name": "reserved_resource_worker_1@pulp3.dev",
"online": true
}
]
}
CPython/3.6.5
=============
pip list | grep -i django
-------------------------
Django 2.0.5
django-crispy-forms 1.7.2
django-extensions 2.0.7
django-filter 1.1.0
djangorestframework 3.8.2
(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/"
{
"next": null,
"previous": null,
"results": [
{
"_href": "http://localhost:8000/pulp/api/v3/workers/4be920ab-2458-4955-a54c-bfe673f94d1a/",
"id": "4be920ab-2458-4955-a54c-bfe673f94d1a",
"last_heartbeat": "2018-05-15T23:43:48.026657Z",
"missing": false,
"name": "reserved_resource_worker_2@pulp3.dev",
"online": false
},
{
"_href": "http://localhost:8000/pulp/api/v3/workers/bbc7abd1-7c5e-49fa-9231-2e9a42d5bf39/",
"id": "bbc7abd1-7c5e-49fa-9231-2e9a42d5bf39",
"last_heartbeat": "2018-05-15T23:43:48.028704Z",
"missing": false,
"name": "resource_manager@pulp3.dev",
"online": false
},
{
"_href": "http://localhost:8000/pulp/api/v3/workers/db161f9d-b67a-444e-8280-f8dfd5a0dc97/",
"id": "db161f9d-b67a-444e-8280-f8dfd5a0dc97",
"last_heartbeat": "2018-05-15T23:44:12.789671Z",
"missing": false,
"name": "reserved_resource_worker_1@pulp3.dev",
"online": true
}
]
}
(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?online=True"
{
"next": null,
"previous": null,
"results": [
{
"_href": "http://localhost:8000/pulp/api/v3/workers/db161f9d-b67a-444e-8280-f8dfd5a0dc97/",
"id": "db161f9d-b67a-444e-8280-f8dfd5a0dc97",
"last_heartbeat": "2018-05-15T23:46:28.543678Z",
"missing": false,
"name": "reserved_resource_worker_1@pulp3.dev",
"online": true
}
]
}
(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev"
{
"next": null,
"previous": null,
"results": [
{
"_href": "http://localhost:8000/pulp/api/v3/workers/4be920ab-2458-4955-a54c-bfe673f94d1a/",
"id": "4be920ab-2458-4955-a54c-bfe673f94d1a",
"last_heartbeat": "2018-05-15T23:43:48.026657Z",
"missing": false,
"name": "reserved_resource_worker_2@pulp3.dev",
"online": false
}
]
}
(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev&online=True"
{
"next": null,
"previous": null,
"results": []
}
Added by dalley almost 6 years ago
Added by dalley almost 6 years ago
Revision d62bf5a2 | View on GitHub
Fix indeterministic worker filtering
Updated by dalley almost 6 years ago
- Status changed from ASSIGNED to POST
PR: https://github.com/pulp/pulp/pull/3500
Intersection limits what you can do in further queries. If the intersection happens first, the name filter doesn't get applied.
System check identified no issues (0 silenced).
May 22, 2018 - 15:53:24
Django version 2.0.5, using settings 'pulpcore.app.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
(SELECT "pulp_app_worker"."id",
"pulp_app_worker"."created",
"pulp_app_worker"."last_updated",
"pulp_app_worker"."name",
"pulp_app_worker"."last_heartbeat",
"pulp_app_worker"."gracefully_stopped",
"pulp_app_worker"."cleaned_up"
FROM "pulp_app_worker"
WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev) INTERSECT
(SELECT "pulp_app_worker"."id",
"pulp_app_worker"."created",
"pulp_app_worker"."last_updated",
"pulp_app_worker"."name",
"pulp_app_worker"."last_heartbeat",
"pulp_app_worker"."gracefully_stopped",
"pulp_app_worker"."cleaned_up"
FROM "pulp_app_worker"
WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:52:57.894782+00:00))
[22/May/2018 15:53:27] "GET /pulp/api/v3/workers/?name=resource_manager@pulp3.dev&online=True HTTP/1.1" 200 290
^C(pulp) [vagrant@pulp3 pulp]$ python manage.py runserver 0.0.0.0:8000
/home/vagrant/.virtualenvs/pulp/lib64/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
/home/vagrant/.virtualenvs/pulp/lib64/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Performing system checks...
System check identified no issues (0 silenced).
May 22, 2018 - 15:53:35
Django version 2.0.5, using settings 'pulpcore.app.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
(SELECT "pulp_app_worker"."id",
"pulp_app_worker"."created",
"pulp_app_worker"."last_updated",
"pulp_app_worker"."name",
"pulp_app_worker"."last_heartbeat",
"pulp_app_worker"."gracefully_stopped",
"pulp_app_worker"."cleaned_up"
FROM "pulp_app_worker") INTERSECT
(SELECT "pulp_app_worker"."id",
"pulp_app_worker"."created",
"pulp_app_worker"."last_updated",
"pulp_app_worker"."name",
"pulp_app_worker"."last_heartbeat",
"pulp_app_worker"."gracefully_stopped",
"pulp_app_worker"."cleaned_up"
FROM "pulp_app_worker"
WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:53:19.456311+00:00))
[22/May/2018 15:53:49] "GET /pulp/api/v3/workers/?name=resource_manager@pulp3.dev&online=True HTTP/1.1" 200 808
In [11]: fmt(Worker.objects.filter(name='resource_manager@pulp3.dev').intersection(Worker.objects.online_workers()))
(SELECT "pulp_app_worker"."id",
"pulp_app_worker"."created",
"pulp_app_worker"."last_updated",
"pulp_app_worker"."name",
"pulp_app_worker"."last_heartbeat",
"pulp_app_worker"."gracefully_stopped",
"pulp_app_worker"."cleaned_up"
FROM "pulp_app_worker"
WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev) INTERSECT
(SELECT "pulp_app_worker"."id",
"pulp_app_worker"."created",
"pulp_app_worker"."last_updated",
"pulp_app_worker"."name",
"pulp_app_worker"."last_heartbeat",
"pulp_app_worker"."gracefully_stopped",
"pulp_app_worker"."cleaned_up"
FROM "pulp_app_worker"
WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:41:50.722037+00:00))
In [12]: fmt(Worker.objects.online_workers().intersection(Worker.objects.filter(name='resource_manager@pulp3.dev')))
(SELECT "pulp_app_worker"."id",
"pulp_app_worker"."created",
"pulp_app_worker"."last_updated",
"pulp_app_worker"."name",
"pulp_app_worker"."last_heartbeat",
"pulp_app_worker"."gracefully_stopped",
"pulp_app_worker"."cleaned_up"
FROM "pulp_app_worker"
WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:42:25.810284+00:00)) INTERSECT
(SELECT "pulp_app_worker"."id",
"pulp_app_worker"."created",
"pulp_app_worker"."last_updated",
"pulp_app_worker"."name",
"pulp_app_worker"."last_heartbeat",
"pulp_app_worker"."gracefully_stopped",
"pulp_app_worker"."cleaned_up"
FROM "pulp_app_worker"
WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev)
Updated by dalley almost 6 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulp|d62bf5a292bd552d509491f6e7013ac180a26b8b.
Updated by bizhang almost 6 years ago
- Status changed from MODIFIED to ASSIGNED
It looks like this issue is still popping up in the current travis tests
Updated by Ichimonji10 almost 6 years ago
- Subject changed from Filtering workers is indetermanistic to Filtering workers is indeterministic
Updated by dalley almost 6 years ago
- Status changed from ASSIGNED to MODIFIED
I can't reproduce this anymore locally, despite an awful lot of trying. Going to close it out and re-enable the tests.
Updated by bmbouter over 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
.
Fix indeterministic worker filtering
closes #3586 https://pulp.plan.io/issues/3586