Project

Profile

Help

Issue #3586

Filtering workers is indeterministic

Added by dkliban@redhat.com over 1 year ago. Updated 6 months ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 38

Description

pulp-smash test to filter workers is failing on python 3.5 from time to time.

It looks like when you filter by either 'online' or 'missing' AND 'name' you can get 2 results when only 1 worker should be returned when filtering by name.

Associated revisions

Revision d62bf5a2 View on GitHub
Added by dalley over 1 year ago

Fix indeterministic worker filtering

closes #3586
https://pulp.plan.io/issues/3586

Revision d62bf5a2 View on GitHub
Added by dalley over 1 year ago

Fix indeterministic worker filtering

closes #3586
https://pulp.plan.io/issues/3586

Revision d62bf5a2 View on GitHub
Added by dalley over 1 year ago

Fix indeterministic worker filtering

closes #3586
https://pulp.plan.io/issues/3586

History

#1 Updated by dkliban@redhat.com over 1 year ago

  • Description updated (diff)

#2 Updated by dalley over 1 year ago

It fails like 90% of the time, but it does work like 10% of the time on Python 3.5.

It works (seemingly) 100% of the time on Python 3.6

#3 Updated by daviddavis over 1 year ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 36

I think the next step is to try to reproduce this locally and debug. I tried with python 3.5.4 and couldn't reproduce though. I think Travis is using python 3.5.5 so maybe that's why.

#4 Updated by jortel@redhat.com over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#6 Updated by rchan over 1 year ago

  • Sprint changed from Sprint 36 to Sprint 37

#7 Updated by jortel@redhat.com over 1 year ago

Unable to reproduce so far using:

  • django-crispy-forms (1.7.2)
  • django-extensions (2.0.7)
  • django-filter (1.1.0)
  • djangorestframework (3.8.2)
  • djangorestframework-jwt (1.11.0)

#8 Updated by dalley over 1 year ago

  • Assignee changed from jortel@redhat.com to dalley
  • Tags Pulp 3 added

#9 Updated by dalley over 1 year ago

Reproduced:

CPython/3.5.2
=============

pip list | grep -i django
-------------------------
Django                   2.0.5
django-crispy-forms      1.7.2
django-extensions        2.0.7
django-filter            1.1.0
djangorestframework      3.8.2

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/" 

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/267e7528-43c4-4e98-aa43-5e39e72a94c3/",
            "id": "267e7528-43c4-4e98-aa43-5e39e72a94c3",
            "last_heartbeat": "2018-05-15T22:23:19.825483Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
            "id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
            "last_heartbeat": "2018-05-15T22:34:23.902196Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/d5096dd8-8126-4bb6-917d-7a4ffe9932d9/",
            "id": "d5096dd8-8126-4bb6-917d-7a4ffe9932d9",
            "last_heartbeat": "2018-05-15T22:23:19.829143Z",
            "missing": false,
            "name": "resource_manager@pulp3.dev",
            "online": false
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?online=True" 

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
            "id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
            "last_heartbeat": "2018-05-15T22:23:20.987260Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev" 

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/267e7528-43c4-4e98-aa43-5e39e72a94c3/",
            "id": "267e7528-43c4-4e98-aa43-5e39e72a94c3",
            "last_heartbeat": "2018-05-15T22:23:19.825483Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev&online=True" 

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
            "id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
            "last_heartbeat": "2018-05-15T22:23:51.110808Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

CPython/3.6.5
=============

pip list | grep -i django
-------------------------
Django                   2.0.5
django-crispy-forms      1.7.2
django-extensions        2.0.7
django-filter            1.1.0
djangorestframework      3.8.2

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/" 

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/4be920ab-2458-4955-a54c-bfe673f94d1a/",
            "id": "4be920ab-2458-4955-a54c-bfe673f94d1a",
            "last_heartbeat": "2018-05-15T23:43:48.026657Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/bbc7abd1-7c5e-49fa-9231-2e9a42d5bf39/",
            "id": "bbc7abd1-7c5e-49fa-9231-2e9a42d5bf39",
            "last_heartbeat": "2018-05-15T23:43:48.028704Z",
            "missing": false,
            "name": "resource_manager@pulp3.dev",
            "online": false
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/db161f9d-b67a-444e-8280-f8dfd5a0dc97/",
            "id": "db161f9d-b67a-444e-8280-f8dfd5a0dc97",
            "last_heartbeat": "2018-05-15T23:44:12.789671Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?online=True" 

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/db161f9d-b67a-444e-8280-f8dfd5a0dc97/",
            "id": "db161f9d-b67a-444e-8280-f8dfd5a0dc97",
            "last_heartbeat": "2018-05-15T23:46:28.543678Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev" 

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/4be920ab-2458-4955-a54c-bfe673f94d1a/",
            "id": "4be920ab-2458-4955-a54c-bfe673f94d1a",
            "last_heartbeat": "2018-05-15T23:43:48.026657Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev&online=True" 

{
    "next": null,
    "previous": null,
    "results": []
}

#10 Updated by dalley over 1 year ago

  • Status changed from ASSIGNED to POST

PR: https://github.com/pulp/pulp/pull/3500

Intersection limits what you can do in further queries. If the intersection happens first, the name filter doesn't get applied.

System check identified no issues (0 silenced).
May 22, 2018 - 15:53:24
Django version 2.0.5, using settings 'pulpcore.app.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up" 
   FROM "pulp_app_worker" 
   WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev) INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up" 
   FROM "pulp_app_worker" 
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:52:57.894782+00:00))
[22/May/2018 15:53:27] "GET /pulp/api/v3/workers/?name=resource_manager@pulp3.dev&online=True HTTP/1.1" 200 290
^C(pulp) [vagrant@pulp3 pulp]$ python manage.py runserver 0.0.0.0:8000
/home/vagrant/.virtualenvs/pulp/lib64/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
/home/vagrant/.virtualenvs/pulp/lib64/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Performing system checks...

System check identified no issues (0 silenced).
May 22, 2018 - 15:53:35
Django version 2.0.5, using settings 'pulpcore.app.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up" 
   FROM "pulp_app_worker") INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up" 
   FROM "pulp_app_worker" 
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:53:19.456311+00:00))
[22/May/2018 15:53:49] "GET /pulp/api/v3/workers/?name=resource_manager@pulp3.dev&online=True HTTP/1.1" 200 808

In [11]: fmt(Worker.objects.filter(name='resource_manager@pulp3.dev').intersection(Worker.objects.online_workers()))

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up" 
   FROM "pulp_app_worker" 
   WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev) INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up" 
   FROM "pulp_app_worker" 
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:41:50.722037+00:00))

In [12]: fmt(Worker.objects.online_workers().intersection(Worker.objects.filter(name='resource_manager@pulp3.dev')))

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up" 
   FROM "pulp_app_worker" 
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:42:25.810284+00:00)) INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up" 
   FROM "pulp_app_worker" 
   WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev)

#11 Updated by dalley over 1 year ago

  • Status changed from POST to MODIFIED

#12 Updated by bizhang over 1 year ago

  • Status changed from MODIFIED to ASSIGNED

It looks like this issue is still popping up in the current travis tests

#13 Updated by Ichimonji10 over 1 year ago

  • Subject changed from Filtering workers is indetermanistic to Filtering workers is indeterministic

#14 Updated by rchan over 1 year ago

  • Sprint changed from Sprint 37 to Sprint 38

#15 Updated by dalley over 1 year ago

  • Status changed from ASSIGNED to MODIFIED

I can't reproduce this anymore locally, despite an awful lot of trying. Going to close it out and re-enable the tests.

#16 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF