Tracks every package name seen across all repositories/components/
architectures during update_from_deb_repos and, once the run
completes, destroys database packages that were not among them (e.g.
ones Debian has dropped entirely, or ones from a since-unconfigured
suite/architecture).
Deliberately opt-in (REMOVE_ORPHANED_PACKAGES = false by default),
unlike REMOVE_BLACKLISTED_PACKAGE: a package looking 'orphaned' can
also just mean a mirror had a transient fetch problem, so the removal
refuses to run at all if any component/architecture failed to fetch,
or if no packages were seen this run at all.
Add 'dummy package' and 'dependency package', the other phrases the
Debian Developer's Reference (6.9.7) documents as convention and that
deborphan --guess-dummy looks for. Deliberately skip bare 'dummy' and
'empty package': both have real false positives in production data
(xserver-xorg-video-dummy, a bridge card 'double dummy solver'
library, python3-roscreate's 'empty package template creator').
Verified against production data: 266 combined matches, still zero
with screenshots.
Their short description reliably says so (e.g. 'transitional
package', 'transitional dummy package for foo'). Verified against
production package data: 190 matches, none of them ever had a
screenshot.
Libraries never produce useful screenshots. Excluding by Debian
archive section instead of a lib* name pattern keeps applications
with lib-prefixed names (libreoffice, librecad, ...) importable.
With REMOVE_BLACKLISTED_PACKAGE set, the next import also removes
previously imported libs-section packages and their screenshots.
Unique violations on packages_name_key happen on a statement that
cannot logically touch name - log the exact query and a longer
backtrace to identify what really fails.
Drop the threaded worker pool (queue, mutexes, connection pooling).
The task now processes packages one by one via find_each - plenty
fast since the external embedding service dominates latency, and
much easier to reason about.
A single failing package used to kill its whole worker thread silently
because only Vectorizer::Error was rescued per package. Now every
StandardError is caught and logged with class and backtrace, and the
embedding write is narrowed to update_column(:embedding) so the task
only ever issues a minimal UPDATE - it cannot touch any other column
or fire model callbacks. Progress stats no longer serialize database
writes through the mutex.
Package text search now combines several strategies: exact name match,
name prefix promotion, compound word splitting ("sqlite browser" finds
"sqlitebrowser") and semantic nearest neighbor search over description
embeddings computed by an external embedding service (all-MiniLM-L6-v2,
384 dims) stored with the pgvector extension. Classic PostgreSQL
full-text search remains as fallback when the vector service is
unreachable. Gibberish queries without lexical overlap with the package
data return empty results instead of random matches.
Embeddings can be backfilled with bin/rails debshots:compute_vectors.
Also drops the unused lograge gem from the Gemfile.
To migrate the application as of April 2020:
• leave models/screenshot.rb with “has_attached_file”
• bundle exec rake db:migrate
• bin/move_paperclip_to_activestorage
• set models/screenshot.rb to “has_one_attached”
