Production imports kept aborting with
ActiveRecord::RecordNotUnique on packages_name_key and - on
environments that never got the constraint - silently created
duplicate packages. Root cause: overlapping update_from_deb_repos
runs (scheduled + manual) both do find-by-name then insert; whichever
wins the race aborts the whole run, whichever loses gets a
duplicate if the constraint was missing.
Three things so the mechanism stops being the problem:
- An advisory lock serializes update_from_deb_repos runs. A second
run that finds the lock held simply skips with a warning instead
of racing the first one. The lock is session-scoped, so it is
released automatically when the process exits.
- A per-package rescue for ActiveRecord::RecordNotUnique. If a
create still races (e.g. a package being created by a web upload),
the package is re-fetched and updated instead of aborting the run.
- A real migration for packages_name_key, which previously only
existed where it had been added by hand. From now on db:migrate
creates it everywhere, new databases included.
Also verified with a reproduction: a single sequential run of the
deployed importer against a fresh database (both architectures,
~54k packages) creates zero duplicates.
Tracks every package name seen across all repositories/components/
architectures during update_from_deb_repos and, once the run
completes, destroys database packages that were not among them (e.g.
ones Debian has dropped entirely, or ones from a since-unconfigured
suite/architecture).
Deliberately opt-in (REMOVE_ORPHANED_PACKAGES = false by default),
unlike REMOVE_BLACKLISTED_PACKAGE: a package looking 'orphaned' can
also just mean a mirror had a transient fetch problem, so the removal
refuses to run at all if any component/architecture failed to fetch,
or if no packages were seen this run at all.
Add 'dummy package' and 'dependency package', the other phrases the
Debian Developer's Reference (6.9.7) documents as convention and that
deborphan --guess-dummy looks for. Deliberately skip bare 'dummy' and
'empty package': both have real false positives in production data
(xserver-xorg-video-dummy, a bridge card 'double dummy solver'
library, python3-roscreate's 'empty package template creator').
Verified against production data: 266 combined matches, still zero
with screenshots.
Their short description reliably says so (e.g. 'transitional
package', 'transitional dummy package for foo'). Verified against
production package data: 190 matches, none of them ever had a
screenshot.
Libraries never produce useful screenshots. Excluding by Debian
archive section instead of a lib* name pattern keeps applications
with lib-prefixed names (libreoffice, librecad, ...) importable.
With REMOVE_BLACKLISTED_PACKAGE set, the next import also removes
previously imported libs-section packages and their screenshots.
