Commit graph

3 commits

Author SHA1 Message Date
2b79470feb Simplify vectorization task to sequential processing
Drop the threaded worker pool (queue, mutexes, connection pooling).
The task now processes packages one by one via find_each - plenty
fast since the external embedding service dominates latency, and
much easier to reason about.
2026-08-23 00:13:50 +02:00
3550a4e2e2 Harden vectorization task against per-package failures
A single failing package used to kill its whole worker thread silently
because only Vectorizer::Error was rescued per package. Now every
StandardError is caught and logged with class and backtrace, and the
embedding write is narrowed to update_column(:embedding) so the task
only ever issues a minimal UPDATE - it cannot touch any other column
or fire model callbacks. Progress stats no longer serialize database
writes through the mutex.
2026-08-23 00:11:52 +02:00
4948fe50e3 Add semantic package search via pgvector embeddings
Package text search now combines several strategies: exact name match,
name prefix promotion, compound word splitting ("sqlite browser" finds
"sqlitebrowser") and semantic nearest neighbor search over description
embeddings computed by an external embedding service (all-MiniLM-L6-v2,
384 dims) stored with the pgvector extension. Classic PostgreSQL
full-text search remains as fallback when the vector service is
unreachable. Gibberish queries without lexical overlap with the package
data return empty results instead of random matches.

Embeddings can be backfilled with bin/rails debshots:compute_vectors.

Also drops the unused lograge gem from the Gemfile.
2026-08-22 22:07:56 +02:00