Production imports kept aborting with
ActiveRecord::RecordNotUnique on packages_name_key and - on
environments that never got the constraint - silently created
duplicate packages. Root cause: overlapping update_from_deb_repos
runs (scheduled + manual) both do find-by-name then insert; whichever
wins the race aborts the whole run, whichever loses gets a
duplicate if the constraint was missing.
Three things so the mechanism stops being the problem:
- An advisory lock serializes update_from_deb_repos runs. A second
run that finds the lock held simply skips with a warning instead
of racing the first one. The lock is session-scoped, so it is
released automatically when the process exits.
- A per-package rescue for ActiveRecord::RecordNotUnique. If a
create still races (e.g. a package being created by a web upload),
the package is re-fetched and updated instead of aborting the run.
- A real migration for packages_name_key, which previously only
existed where it had been added by hand. From now on db:migrate
creates it everywhere, new databases included.
Also verified with a reproduction: a single sequential run of the
deployed importer against a fresh database (both architectures,
~54k packages) creates zero duplicates.
Tracks every package name seen across all repositories/components/
architectures during update_from_deb_repos and, once the run
completes, destroys database packages that were not among them (e.g.
ones Debian has dropped entirely, or ones from a since-unconfigured
suite/architecture).
Deliberately opt-in (REMOVE_ORPHANED_PACKAGES = false by default),
unlike REMOVE_BLACKLISTED_PACKAGE: a package looking 'orphaned' can
also just mean a mirror had a transient fetch problem, so the removal
refuses to run at all if any component/architecture failed to fetch,
or if no packages were seen this run at all.
Add 'dummy package' and 'dependency package', the other phrases the
Debian Developer's Reference (6.9.7) documents as convention and that
deborphan --guess-dummy looks for. Deliberately skip bare 'dummy' and
'empty package': both have real false positives in production data
(xserver-xorg-video-dummy, a bridge card 'double dummy solver'
library, python3-roscreate's 'empty package template creator').
Verified against production data: 266 combined matches, still zero
with screenshots.
Their short description reliably says so (e.g. 'transitional
package', 'transitional dummy package for foo'). Verified against
production package data: 190 matches, none of them ever had a
screenshot.
Libraries never produce useful screenshots. Excluding by Debian
archive section instead of a lib* name pattern keeps applications
with lib-prefixed names (libreoffice, librecad, ...) importable.
With REMOVE_BLACKLISTED_PACKAGE set, the next import also removes
previously imported libs-section packages and their screenshots.
query_packages now assigns @search_sources - a hash mapping package
ids to the matching strategy that put them into the results: :exact,
:name_prefix, :name_contains, :semantic or :fulltext. The grid and
list views render a small badge with a humanized label per result,
making the search tiers observable. semantic_results() was folded
into search_packages() so each branch tags its own source. Adds the
rails-controller-testing gem for assigns() in tests.
Unique violations on packages_name_key happen on a statement that
cannot logically touch name - log the exact query and a longer
backtrace to identify what really fails.
Drop the threaded worker pool (queue, mutexes, connection pooling).
The task now processes packages one by one via find_each - plenty
fast since the external embedding service dominates latency, and
much easier to reason about.
A single failing package used to kill its whole worker thread silently
because only Vectorizer::Error was rescued per package. Now every
StandardError is caught and logged with class and backtrace, and the
embedding write is narrowed to update_column(:embedding) so the task
only ever issues a minimal UPDATE - it cannot touch any other column
or fire model callbacks. Progress stats no longer serialize database
writes through the mutex.
Package pages get a "Related packages" section: a nearest neighbor
query against the package's own stored embedding (no external service
call), excluding the package itself. Packages without an embedding
render no section.
Package text search now combines several strategies: exact name match,
name prefix promotion, compound word splitting ("sqlite browser" finds
"sqlitebrowser") and semantic nearest neighbor search over description
embeddings computed by an external embedding service (all-MiniLM-L6-v2,
384 dims) stored with the pgvector extension. Classic PostgreSQL
full-text search remains as fallback when the vector service is
unreachable. Gibberish queries without lexical overlap with the package
data return empty results instead of random matches.
Embeddings can be backfilled with bin/rails debshots:compute_vectors.
Also drops the unused lograge gem from the Gemfile.
