Pulp

I was able to reproduce the issue. I used the attached test to do so. The output of the test looked like this:

2000: 4000
4000: 6000
6000: 8000
8000: 10000
10000: 12000
12000: 14000
14000: 16000
16000: 18000
18000: 20000
20000: 22000
22000: 24000
24000: 25295
26000: 26597
28000: 27908
30000: 29271
32000: 30531
34000: 31868
36000: 33239
38000: 34526
40000: 35833
42000: 37146
44000: 38486
46000: 39804
48000: 41137
50000: 42435
52000: 43745
54000: 45055
56000: 46362
58000: 47703
60000: 49003
62000: 50342
64000: 51626
66000: 52978
68000: 54312
70000: 54312
Expected: 70000
Actual: 54312

The content started to be non-unique after 24,000 fetched units (the number probably depends upon a running system). And it did not matter whether I was retrieving the content units using page_size=2,000 or page_size=200.

I tried to sort synced content by the field "pulp_created" (as dkliban had mentioned) using the built-in Django ordering options (https://docs.djangoproject.com/en/3.0/ref/models/options/#django.db.models.Options.ordering), but with no success. I added "ordering = ['-pulp_created']" to the meta class of the model FileContent, however, the sync failed with the following error message:

"traceback": "  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py\", line 884, in perform_job\n   
 rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py\", line 664, in perform\n   
 self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py\", line 670, in _execute\n 
   return self.func(*self.args, **self.kwargs)\n  File \"/home/fedora/devel/pulp_file/pulp_file/app/tasks/synchronizing.py\", line 45, 
in synchronize\n    dv.create()\n  File \"/home/fedora/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py\", line 149, 
in create\n    loop.run_until_complete(pipeline)\n  File \"/home/fedora/devel/pulpcore/pulpcore/app/models/repository.py\", 
line 753, in __exit__\n    repository.finalize_new_version(self)\n  File \"/home/fedora/devel/pulp_file/pulp_file/app/models.py\", 
line 68, in finalize_new_version\n    validate_repo_version(new_version)\n  File 
\"/home/fedora/devel/pulpcore/pulpcore/plugin/repo_version_utils.py\", line 138, in validate_repo_version\n    
validate_duplicate_content(version)\n  File \"/home/fedora/devel/pulpcore/pulpcore/plugin/repo_version_utils.py\",
line 96, in validate_duplicate_content\n    ).distinct(*repo_key_fields).count()\n  
File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/django/db/models/query.py\", line 392, in count\n  
  return self.query.get_count(using=self.db)\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/django/db/models/sql/query.py\",
 line 504, in get_count\n    number = obj.get_aggregation(using, ['__count'])['__count']\n 
 File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/django/db/models/sql/query.py\", line 489, 
in get_aggregation\n    result = compiler.execute_sql(SINGLE)\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/django/db/models/sql/compiler.py\", line 1133, in execute_sql\n   
 cursor.execute(sql, params)\n 
 File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/django/db/backends/utils.py\", line 67, in execute\n  
  return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)\n 
 File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers\n  
  return executor(sql, params, many, context)\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/django/db/backends/utils.py\",
 line 84, in _execute\n    return self.cursor.execute(sql, params)\n  
File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/django/db/utils.py\", line 89, in __exit__\n 
   raise dj_exc_value.with_traceback(traceback) from exc_value\n  
File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/django/db/backends/utils.py\", line 84, in _execute\n   
 return self.cursor.execute(sql, params)\n",

    "description": "SELECT DISTINCT ON expressions must match initial ORDER BY expressions\nLINE 1: SELECT COUNT(*) FROM (SELECT DISTINCT ON (\"file_filecontent\"...\n   

I did not experience the above error message during the sync when I added that ordering to the model MasterModel. Still, Pulp did not return unique values. Specifying a default ordering (https://www.django-rest-framework.org/api-guide/filtering/#specifying-a-default-ordering) in FileContentViewSet did not help either way.

The only approach, that was working for me, was to create a custom class that handles the actual ordering, like shown in the commit https://github.com/lubosmj/pulp_file/commit/379ce9298307a4d8886a1cb1f4e2a5bd08906ef7. I just stole the workaround proposed here: https://github.com/encode/django-rest-framework/issues/6886#issuecomment-547120480. What is interesting here is that I did not need to order content by the field "pulp_created" and the test passed as supposed. Yet, It seems like the changes did break some functionality according to Travis (see the build https://travis-ci.org/github/lubosmj/pulp_file/builds/670764582, only one test failed, but I am not experiencing such a failure in my VM).

Also, there is an option to use a different pagination. Currently, we are using LimitOffsetPagination. There exists CursorPagination (https://www.django-rest-framework.org/api-guide/pagination/#cursorpagination) which requires that there is a unique, unchanging ordering of items in the result set.

PR: https://github.com/pulp/pulp_file/pull/372

Lubos did all the work: https://github.com/pulp/pulpcore/pull/634

Applied in changeset pulpcore|5ef9ee839bb23d15f0754e44e53358789ce57eb0.

Applied in changeset pulpcore|8f6fb1bd5aaa7f9708d9576efb4805426756ffce.