Project

Profile

Help

Issue #4795

closed

Pulp2 task details information returns incorrect values when polled too quickly

Added by bherring almost 4 years ago. Updated over 2 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Problem

When attempting to sync a feed for a second time in automation/CI (not manually), the sync will return incorrect/invalid information for the task details failing the test "often".

Problem Breakout

There are two discrete issues here:

  • `pulp-smash` is now/recently updated polling at a 0.3s rate that. For a test including a second repo sync, this will occasionally returning a task "COMPLETE" but the values for the task details had yet to be reset. The resulting return of non-zero values causes test failures. It appears that the ACTUAL change is none, however the task_details are reporting incorrectly. This was supposedly discovered long ago, but the legacy issue could not be found.
  • Due to the above, the values of 'added_count', 'removed_count', 'updated_count' are being permanently updated to values in the db that are incorrect. Meaning, running `pulp-admin tasks details --task-id` of the automated task will result in the invalid values still being populated in the historic record. This appears to be related to legacy issue #4428

Recreation

The RCA seems to be from updating pulp-smash polling to a much quicker rate:
https://github.com/PulpQE/pulp-smash/blob/e385e6685a82709fd167b5374d19c4dfcfdb78c5/pulp_smash/api.py#L687

To test this, locally have pulp-smash installed and set to a VERY low polling rate. At the time of this writing, anything around `0.3` would fail 50-80% of the time. Adjusting to an even smaller value should increase the probabiltiy of failure on a non-patched system.

Note: This MUST be done in an automated fashion with pulp-smash due to the api polling. Testing this by hand or manually will NOT result in issue recreation.

Run this test through pytest with the following syntax in an appropriately prepared virtenv:

(pulp2) [herring@redherring api_v2]$ pwd
/home/herring/git/Pulp-2-Tests/pulp_2_tests/tests/rpm/api_v2
(pulp2) [herring@redherring api_v2]$ count=1; while [ $count -le 10 ]; do echo -e "Iteration: $count\n"; pytest -svv test_sync_publish.py::SyncRpmRepoTestCase::test_no_change_in_second_sync --disable-warnings; ((count++)); done

A loop value of 10 was chosen to ensure math was easy.

It is expected there will be a 100% pass rate. At this time, even with larger values around 2 seconds, there is still ~10% chance of failure.

Also, running the following on a passing Task ID should NOT result in incorrect values for the task:

[root@rhel76 ~]# pulp-admin tasks details --task-id d0140034-e1ca-4d1e-a34f-7fdf1d17a697

Where the `--task-id` is appropriate for the jobs being ran.

QE Workaround in place

For the time being, the values for pulp-smash when used on Pulp2 polling can be increase to keep these failures from happening.

Once this task issues are resolved, pulp-smash can have the values restored to lower polling values.

References

  • Additional Reference needed here from @dkliban about how the values are calculated
    https://pulp.plan.io/issues/4428#note-17
  • Original investigation information moved down in other Notes to retain investigation history.

Related issues

Related to Pulp - Issue #4428: Upsert query raises NotUniqueErrorCLOSED - CURRENTRELEASEdaviddavisActions

Also available in: Atom PDF