Christoph Haas
c5cd77e9ee
Production imports kept aborting with ActiveRecord::RecordNotUnique on packages_name_key and - on environments that never got the constraint - silently created duplicate packages. Root cause: overlapping update_from_deb_repos runs (scheduled + manual) both do find-by-name then insert; whichever wins the race aborts the whole run, whichever loses gets a duplicate if the constraint was missing. Three things so the mechanism stops being the problem: - An advisory lock serializes update_from_deb_repos runs. A second run that finds the lock held simply skips with a warning instead of racing the first one. The lock is session-scoped, so it is released automatically when the process exits. - A per-package rescue for ActiveRecord::RecordNotUnique. If a create still races (e.g. a package being created by a web upload), the package is re-fetched and updated instead of aborting the run. - A real migration for packages_name_key, which previously only existed where it had been added by hand. From now on db:migrate creates it everywhere, new databases included. Also verified with a reproduction: a single sequential run of the deployed importer against a fresh database (both architectures, ~54k packages) creates zero duplicates. |
||
|---|---|---|
| .forgejo/workflows | ||
| .github | ||
| .idea | ||
| .kamal | ||
| app | ||
| bin | ||
| config | ||
| db | ||
| doc | ||
| lib | ||
| log | ||
| public | ||
| test | ||
| vendor/javascript | ||
| .dockerignore | ||
| .env.development | ||
| .env.example | ||
| .env.production | ||
| .gitattributes | ||
| .gitignore | ||
| .gitlab-ci.yml | ||
| .node-version | ||
| .rubocop.yml | ||
| .ruby-version | ||
| .tool-versions | ||
| .woodpecker.yml | ||
| AGENTS.md | ||
| app.json | ||
| bun.lock | ||
| config.ru | ||
| dns | ||
| docker-compose.yml | ||
| Dockerfile | ||
| Gemfile | ||
| Gemfile.lock | ||
| Guardfile | ||
| LICENSE | ||
| Makefile | ||
| nginx.conf.sigil | ||
| package.json | ||
| postcss.config.js | ||
| Procfile | ||
| Procfile.dev | ||
| Rakefile | ||
| README.md | ||
| sql | ||
| webpack.config.js | ||
About debshots
Debshots is a web application written in Ruby-on-Rails that powers the screenshots.debian.net web site. Many services like packages.debian.net, Ubuntu Software or Synaptic rely on it.
State of the application
The code you find here resembles the version that is deployed to screenshots.debian.net. It has been refactored in 06/2024 to Rails 7. If you find strange or changed behavior please open an issue at https://salsa.debian.org/debian/debshots/-/issues
Development
Check out the doc/README.Development.md
Package search
The package search combines several strategies, tried in this order:
- Exact package name: Searching for
vimshows exactly this package without calling any external service. - Name prefix matches: Packages whose name starts with the query
rank first, e.g. searching for
sqlitepromotessqlite3,sqlitebrowserand friends above all other results. - Compound word splitting: Packages whose name contains all
query tokens match too, so
sqlite browserfindssqlitebrowser. - Semantic search: The query is embedded via the vector service
(all-MiniLM-L6-v2, 384 dimensions) and the nearest neighbors among
the package description embeddings are returned, using the pgvector
extension with an HNSW cosine index. Embeddings are computed from
the package description and long description and can be (re-)built
with
bin/rails debshots:compute_vectors. - Full-text fallback: If the vector service cannot be reached, classic PostgreSQL full-text search takes over.
Queries that share no vocabulary with the package data (gibberish, keyboard mash) return an honest empty result instead of random semantic matches.
Why is this package in my results? Each result knows which search
strategy produced it. The controller exposes this as
@search_sources (a hash mapping package ids to :exact,
:name_prefix, :name_contains, :semantic or :fulltext). For
debugging you can render a small badge on every result card by adding
show_sources=1 to the URL, e.g.
https://screenshots.debian.net/packages?search=sqlite&show_sources=1.
Package pages also show semantically related packages, computed as a nearest neighbor query against the package's own description embedding - no external service call involved.
Deployment
Read the doc/README.Installation.md