Add semantic package search via pgvector embeddings

Package text search now combines several strategies: exact name match,
name prefix promotion, compound word splitting ("sqlite browser" finds
"sqlitebrowser") and semantic nearest neighbor search over description
embeddings computed by an external embedding service (all-MiniLM-L6-v2,
384 dims) stored with the pgvector extension. Classic PostgreSQL
full-text search remains as fallback when the vector service is
unreachable. Gibberish queries without lexical overlap with the package
data return empty results instead of random matches.

Embeddings can be backfilled with bin/rails debshots:compute_vectors.

Also drops the unused lograge gem from the Gemfile.
This commit is contained in:
Christoph Haas 2026-08-22 22:07:56 +02:00
parent c7e8109bbd
commit 4948fe50e3
15 changed files with 662 additions and 125 deletions

View file

@ -362,7 +362,7 @@ class PackagesController < ApplicationController
# text search
if params[:search].present?
logger.debug "Searching for #{params[:search]}"
packages = packages.general_search(params[:search])
packages = search_packages(params[:search])
end
case params[:show]
@ -378,6 +378,51 @@ class PackagesController < ApplicationController
packages
end
# Text search for packages: an exact package name match wins without
# touching the vector service. Otherwise packages whose name starts
# with the query (or contains all query tokens, catching compound
# words like "sqlite browser" -> "sqlitebrowser") are promoted above
# the semantic (vector) nearest neighbor results. Classic PostgreSQL
# full-text search kicks in when the vector service cannot be reached.
MAX_PROMOTED_NAME_MATCHES = 8
MAX_SEMANTIC_RESULTS = 100
private_constant :MAX_PROMOTED_NAME_MATCHES, :MAX_SEMANTIC_RESULTS
def search_packages(query)
exact_match = Package.find_by(name: query)
return Package.where(id: exact_match.id) if exact_match
tokens = query.to_s.split(/\s+/)
name_matches = (
Package.name_starts_with(query).limit(MAX_PROMOTED_NAME_MATCHES) +
Package.name_contains_all(tokens).limit(MAX_PROMOTED_NAME_MATCHES)
).uniq(&:id)
results = (name_matches + semantic_results(query)).uniq(&:id)
# Queries that share no vocabulary with our package data can produce
# no meaningful results. Skip the vector service round-trip and show
# an honest empty result instead of random packages.
results.empty? ? Package.none : results
end
# Semantic (vector) nearest neighbor search with a full-text fallback
# for when the external embedding service is unreachable.
def semantic_results(query)
unless Package.lexical_overlap?(query)
logger.debug "Query '#{query}' has no lexical overlap with package data"
return []
end
begin
Package.nearest_to_text(query, limit: MAX_SEMANTIC_RESULTS).to_a
rescue Vectorizer::Error => e
logger.error "Semantic search failed: #{e.message}"
flash.now['error'] = 'Semantic search is unavailable right now. Showing full-text matches.'
Package.general_search(query).limit(100).to_a
end
end
# Store a random identifier and the client's IP address in the session
# for later identification.
# def create_user_token