Issue #4795
Updated by bherring over 5 years ago
h2. Problem
When attempting to sync a feed for a second time in automation/CI (not manually), the sync will return incorrect/invalid information for the task details failing the test "often".
h2. Problem Breakout
There are two discrete issues here:
* `pulp-smash` is now/recently updated polling at a 0.3s rate that. For a test including a second repo sync, this will occasionally returning a task "COMPLETE" but the values for the task details had yet to be reset. The resulting return of non-zero values causes test failures. It appears that the ACTUAL change is none, however the task_details are reporting incorrectly. This was supposedly discovered long ago, but the legacy issue could not be found.
* Due to the above, the values of 'added_count', 'removed_count', 'updated_count' are being permanently updated to values in the db that are incorrect. Meaning, running `pulp-admin tasks details --task-id` of the automated task will result in the invalid values still being populated in the historic record. This appears to be related to legacy issue #4428
h2. Recreation
The RCA seems to be from updating pulp-smash polling to a much quicker rate:
https://github.com/PulpQE/pulp-smash/blob/e385e6685a82709fd167b5374d19c4dfcfdb78c5/pulp_smash/api.py#L687
To test this, locally have pulp-smash installed and set to a VERY low polling rate. At the time of this writing, anything around `0.3` would fail 50-80% of the time. Adjusting to an even smaller value should increase the probabiltiy of failure on a non-patched system.
Run this test through pytest with the following syntax in an appropriately prepared virtenv:
<pre>
(pulp2) [herring@redherring api_v2]$ pwd
/home/herring/git/Pulp-2-Tests/pulp_2_tests/tests/rpm/api_v2
(pulp2) [herring@redherring api_v2]$ count=1; while [ $count -le 10 ]; do echo -e "Iteration: $count\n"; pytest -svv test_sync_publish.py::SyncRpmRepoTestCase::test_no_change_in_second_sync --disable-warnings; ((count++)); done
</pre>
A loop value of 10 was chosen to ensure math was easy.
It is expected there will be a 100% pass rate. At this time, even with larger values around 2 seconds, there is still ~10% chance of failure.
Also, running the following on a passing Task ID should NOT result in incorrect values for the task:
<pre>
[root@rhel76 ~]# pulp-admin tasks details --task-id d0140034-e1ca-4d1e-a34f-7fdf1d17a697
</pre>
Where the `--task-id` is appropriate for the jobs being ran.
h2. QE Workaround in place
For the time being, the values for pulp-smash when used on Pulp2 polling can be increase to keep these failures from happening.
Once this task issues are resolved, pulp-smash can have the values restored to lower polling values.
h2. References
* Additional Reference needed here from @dkliban about how the values are calculated
https://pulp.plan.io/issues/4428#note-17
* Original investigation information moved down in other Notes to retain investigation history.
.