Documentation revamped
This commit is contained in:
parent
844feb7f0e
commit
fbee80f5f1
16 changed files with 377 additions and 1175 deletions
|
|
@ -1,3 +1,5 @@
|
|||
This file collects various notes about how certain aspects are implemented.
|
||||
|
||||
# Full text search
|
||||
|
||||
The pg_search Ruby gem is used for full-text searched in Rails.
|
||||
|
|
@ -21,7 +23,7 @@ SELECT COUNT(*) FROM "packages" WHERE (
|
|||
)
|
||||
)
|
||||
|
||||
# Index:
|
||||
## Create search index to speed up searches significantly
|
||||
|
||||
CREATE INDEX packages_gin ON packages USING GIN( (to_tsvector('simple',
|
||||
coalesce("packages"."name"::text, '')) || to_tsvector('simple',
|
||||
|
|
@ -33,27 +35,21 @@ coalesce("packages"."long_description"::text, ''))) );
|
|||
Used: https://github.com/semaperepelitsa/jquery.fileupload-rails
|
||||
Which is a gem for: https://github.com/blueimp/jQuery-File-Upload
|
||||
|
||||
---
|
||||
|
||||
Paperclip for attachment/screenshot file handlind within ActiveRecord
|
||||
https://github.com/thoughtbot/paperclip
|
||||
|
||||
--
|
||||
Rails 5 is said to contain functionality like that already.
|
||||
Might be worth migrating to that at some time.
|
||||
|
||||
Components:
|
||||
|
||||
- Rails 5 (widespread, up-to-date, solves many common problems)
|
||||
- PostgreSQL (fast, easy, I don't like MySQL, good fulltext search)
|
||||
- Git (most people nowadays use Git)
|
||||
- jQuery (well known and easy to use)
|
||||
- Zurb Foundation (integrates nicer than Twitter Bootstrap)
|
||||
- Guard for automated testing (http://www.rubydoc.info/gems/guard-rails)
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
Importer
|
||||
|
||||
How Debian repositories are organized: https://wiki.debian.org/RepositoryFormat
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
Logging from within the application into the database's "log" table:
|
||||
|
||||
|
|
@ -61,7 +57,7 @@ Log.log "foo"
|
|||
or with section (default='frontend'):
|
||||
Log.log "foo", "importer"
|
||||
|
||||
--
|
||||
---
|
||||
|
||||
To create admin users
|
||||
---------------------
|
||||
|
|
@ -1,5 +0,0 @@
|
|||
To run the Rails server in development mode on port 3001 with HTTPS:
|
||||
|
||||
rails s puma -b 'ssl://0.0.0.0:3001?key=nginx/localhost.key&cert=nginx/localhost.crt&verify_mode=none'
|
||||
|
||||
This is required to be able to use single-sign login with the different services.
|
||||
123
doc/README.Development.md
Normal file
123
doc/README.Development.md
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
# Developing/Contributing
|
||||
|
||||
This document helps you set up the *debshots* Rails application on your
|
||||
own Debian system if you intend to work on the source code.
|
||||
|
||||
## Preparing the development environment
|
||||
|
||||
Most steps required to deploy the application (for production use) are automated
|
||||
using an Ansible playbook that you find at ./ansible/debshots.yml
|
||||
|
||||
The following section documents the steps you need to do to get the application
|
||||
running in development mode. If you are a little experienced in developing Rails
|
||||
applications you may know most of that already.
|
||||
|
||||
Development worked best on a Debian Jessie or Sid system.
|
||||
|
||||
## Install dependencies
|
||||
|
||||
These APT-gettable packages are probably required to run the application:
|
||||
|
||||
apt install python-psycopg2 ruby sudo libssl-dev libreadline-dev \
|
||||
zlib1g-dev libbz2-dev libpq-dev git imagemagick
|
||||
|
||||
## Prepare PostgreSQL database server and user
|
||||
|
||||
PostgreSQL stores data about packages, screenshots and users. Authentication works in production
|
||||
because the applications runs as the system user *debshots* that gets access to PostgreSQL via
|
||||
ident-based authentication. In development mode you will likely work as your own user so you need to
|
||||
create a development database in PostgreSQL and create a database user with a password to access it.
|
||||
Put those credentials into ./config/database.yml
|
||||
|
||||
apt install postgresql
|
||||
su - postgres
|
||||
createuser -P -d debshots_dev
|
||||
createdb -O debshots_dev -E UTF8 -T template0 debshots_dev
|
||||
|
||||
## Install the required Ruby version using rbenv
|
||||
|
||||
Rails uses the *bundler* tool to download and install dependent gems/packages. To avoid installing
|
||||
them system wide you should rather create an *rbenv* sandbox environment where the specific Ruby
|
||||
version and the required gems are installed. That ensures stable operation of the application and
|
||||
does not spoil your otherwise APT-controlled system.
|
||||
|
||||
Documentation on rbenv can be found at https://github.com/rbenv/rbenv
|
||||
|
||||
To install it for your current user:
|
||||
|
||||
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
|
||||
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
|
||||
|
||||
# Load rbenv automatically by appending
|
||||
# the following to ~/.profile:
|
||||
|
||||
Run "rbenv init" and do as said.
|
||||
|
||||
Install the "rbenv install" (https://github.com/rbenv/ruby-build#readme) add-on:
|
||||
|
||||
git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
|
||||
|
||||
## Get the application
|
||||
|
||||
Clone the debshots Git repository from https://salsa.debian.org/debian/debshots
|
||||
|
||||
## Install the required Ruby version and the applications' dependencies
|
||||
|
||||
rbenv install
|
||||
echo "gem: --no-rdoc --no-ri" > ~/.gemrc
|
||||
gem install bundler
|
||||
bundle install
|
||||
|
||||
## Optional: copy screenshots and database from the live website
|
||||
|
||||
scp screenshots.debian.net:debshots-screenshots.tar .
|
||||
tar -C public/ -xvf debshots-screenshots.tar
|
||||
scp screenshots.debian.net:debshots.sql .
|
||||
psql debshots_dev < debshots.sql
|
||||
|
||||
Make sure that the application's database schema is upgraded to reflect the latest state of
|
||||
development.
|
||||
|
||||
bundle exec rails db:migrate
|
||||
|
||||
(The reason to prefix all commands by "bundle exec" to make sure that the Ruby version
|
||||
from *rbenv* is used and that only the gems from the sandbox environment are used.)
|
||||
|
||||
## Start with a blank database
|
||||
|
||||
Create a blank database with the latest database schema:
|
||||
|
||||
bundle exec rails db:schema:load
|
||||
|
||||
If you get database access errors then check the *development* section in your
|
||||
./config/database.yml file.
|
||||
|
||||
## Fill/update the database with information from APT repositories
|
||||
|
||||
Get the latest information on packages from the Debian repositories.
|
||||
|
||||
bundle exec rails debshots:update_from_deb_repos RAILS_ENV=production
|
||||
bundle exec rails debshots:update_longdescription_from_deb_repos RAILS_ENV=production
|
||||
|
||||
## Run the application
|
||||
|
||||
bundle exec rails s
|
||||
|
||||
Point your browser to http://localhost:3000/ and you will see the web site.
|
||||
|
||||
## Optional: Tidy up the screenshot data
|
||||
|
||||
bundle exec rake debshots:remove_broken_screenshots
|
||||
bundle exec rake debshots:remove_duplicate_images
|
||||
|
||||
## Optional: Import new package information
|
||||
|
||||
bundle exec rake debshots:list_deb_repos
|
||||
bundle exec rake debshots:update_from_deb_repos
|
||||
bundle exec debshots:update_longdescription_from_deb_repos
|
||||
|
||||
To run the Rails server in development mode on port 3001 with HTTPS:
|
||||
|
||||
rails s puma -b 'ssl://0.0.0.0:3001?key=nginx/localhost.key&cert=nginx/localhost.crt&verify_mode=none'
|
||||
|
||||
This is required to be able to use single-sign login with the different services.
|
||||
|
|
@ -1,303 +0,0 @@
|
|||
# Installing debshots the easy way using Ansible
|
||||
|
||||
Prerequisites: A Debian Jessie server.
|
||||
|
||||
Install PIP to get a recent version of Ansible:
|
||||
|
||||
apt install python-pip
|
||||
pip install ansible
|
||||
|
||||
Run the Ansible playbook that comes with the debshots repository:
|
||||
|
||||
cd ansible
|
||||
ansible-playbook debshots.yml
|
||||
|
||||
# Installing debshots the manual way
|
||||
|
||||
## Prepare PostgreSQL database server and user (ident-based authentication)
|
||||
|
||||
apt install postgresql
|
||||
su - postgres
|
||||
createuser -P -d debshots
|
||||
createdb -O debshots -E UTF8 -T template0 debshots
|
||||
|
||||
## Create an application user
|
||||
|
||||
adduser debshots
|
||||
|
||||
Allow the user to parse access log files from Nginx:
|
||||
|
||||
adduser debshots adm
|
||||
su - debshots
|
||||
|
||||
## Install the required Ruby version using rbenv
|
||||
|
||||
Documentation: https://github.com/rbenv/rbenv
|
||||
|
||||
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
|
||||
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
|
||||
|
||||
# Load rbenv automatically by appending
|
||||
# the following to ~/.profile:
|
||||
|
||||
eval "$(rbenv init -)"
|
||||
|
||||
Install the "rbenv install" add-on:
|
||||
|
||||
Documentation: https://github.com/rbenv/ruby-build#readme
|
||||
|
||||
git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
|
||||
|
||||
Install the "rbenv vars" add-on:
|
||||
|
||||
cd ~/.rbenv/plugins
|
||||
git clone https://github.com/sstephenson/rbenv-vars.git
|
||||
|
||||
## Get the application
|
||||
|
||||
Clone the debshots Git repository.
|
||||
|
||||
## Install the Ruby dependencies
|
||||
|
||||
rbenv install
|
||||
echo "gem: --no-rdoc --no-ri" > ~/.gemrc
|
||||
gem install bundler
|
||||
bundle install --deployment
|
||||
|
||||
## Optional: copy screenshots and database from the live website
|
||||
|
||||
scp screenshots.debian.net:debshots-screenshots.tar .
|
||||
tar -C public/ -xvf debshots-screenshots.tar
|
||||
scp screenshots.debian.net:debshots.sql .
|
||||
psql debshots < debshots.sql
|
||||
|
||||
## Switch to production environment
|
||||
|
||||
export RAILS_ENV=production
|
||||
|
||||
## Migrate database
|
||||
|
||||
rails db:schema:load RAILS_ENV=production
|
||||
- rails db:migrate RAILS_ENV=production -
|
||||
|
||||
In config/database.yml: production: user/password/host must be commented out
|
||||
|
||||
## Fill/update the database with information from APT repositories
|
||||
|
||||
rails debshots:update_from_deb_repos RAILS_ENV=production
|
||||
rails debshots:update_longdescription_from_deb_repos RAILS_ENV=production
|
||||
|
||||
## Pre-render the static assets
|
||||
|
||||
rails assets:precompile RAILS_ENV=production
|
||||
|
||||
## Test the application:
|
||||
|
||||
rails s -b 127.0.0.1 -e production
|
||||
|
||||
Point your browser to http://...:3000/ and check the web site.
|
||||
|
||||
## Create a user for moderation:
|
||||
|
||||
Start a Rails console:
|
||||
|
||||
bundle exec rails c -e production
|
||||
|
||||
Create a new user record:
|
||||
|
||||
user = User.create(realname: 'Christoph Haas', password: 'foobartest', email: 'email@christoph-haas.de')
|
||||
user.save!
|
||||
|
||||
## Tidy up the screenshot data
|
||||
|
||||
bundle exec rake debshots:remove_broken_screenshots
|
||||
bundle exec rake debshots:remove_duplicate_images
|
||||
|
||||
## Import new package information
|
||||
|
||||
bundle exec rake debshots:list_deb_repos
|
||||
bundle exec rake debshots:update_from_deb_repos
|
||||
bundle exec debshots:update_longdescription_from_deb_repos
|
||||
|
||||
|
||||
# Supported URL paths (aka routes)
|
||||
|
||||
_For up-to-date information please check config/routes._
|
||||
|
||||
/screenshots/:package_inital/:package/:(id)_:size.png
|
||||
* package = name of the package
|
||||
* id = numerical ID of the screenshot
|
||||
* size = "large" or "small"
|
||||
* Returns a PNG image of a screenshot
|
||||
* Should be served as a static asset
|
||||
|
||||
/image/:(id)_:size.png
|
||||
* id = numerical ID of the screenshot
|
||||
* size = "large" or "small"
|
||||
* Returns a PNG image of a screenshot
|
||||
* Usually used for unapproved images that should just be shown to the uploader
|
||||
|
||||
/about
|
||||
* Return an informational HTML page
|
||||
|
||||
/packages
|
||||
* Return a list of packages with pagination
|
||||
|
||||
/packages/with_screenshots
|
||||
* Return a list of packages with pagination
|
||||
* Select only those packages that have screenshots.
|
||||
|
||||
/packages/without_screenshots
|
||||
* Return a list of packages with pagination
|
||||
* Select only those packages that do not have screenshots.
|
||||
|
||||
/packages/games_without_screenshots (deprecated)
|
||||
* Return a list of packages with pagination
|
||||
* Select only those packages that belong to Debian's "games" section
|
||||
|
||||
/json/packages
|
||||
* Return a JSON data structure of all packages
|
||||
|
||||
/json/screenshots
|
||||
* Return a JSON data structure containing packages and screenshots
|
||||
|
||||
/json/packages-without-screenshots
|
||||
* Return a JSON data structure of those packages that do not have any screenshots
|
||||
|
||||
/packages/moderate
|
||||
* Return a HTML page showing packages that need moderation
|
||||
* Must only be available to logged in moderators
|
||||
|
||||
/upload
|
||||
* Return a HTML page containing an upload form for new screenshots
|
||||
|
||||
/guidelines (deprecated)
|
||||
* Return a HTML page showing general guidelines for creating screenshots
|
||||
* Should just redirect to the /upload page
|
||||
|
||||
/upload/:package
|
||||
* ??
|
||||
|
||||
/uploadfile
|
||||
* ??
|
||||
|
||||
/login
|
||||
* Show a login form
|
||||
|
||||
/logout
|
||||
* Logout the user by invalidating the cookie session.
|
||||
|
||||
/package/:package
|
||||
* Show details of a certain package
|
||||
|
||||
/json/package/:package
|
||||
* Return a JSON data structure containing details of a certain package
|
||||
|
||||
/thumbnail/:package
|
||||
/thumbnail-404/:package (deprecated)
|
||||
* Return a small (160x120 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
|
||||
/thumbnail-with-version/:package/:version
|
||||
* Return a small (160x120 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
* Prefer screenshots of a certain version
|
||||
|
||||
/screenshot/:package
|
||||
/screenshot-404/:package (deprecated)
|
||||
* Return a large (320x240 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
|
||||
/screenshot-with-version/:package/:version
|
||||
* Return a large (320x240 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
* Prefer screenshots of a certain version
|
||||
|
||||
/delete_screenshot/:screenshot
|
||||
* Users: request deletion of a screenshot
|
||||
* Moderators: delete a screenshot
|
||||
|
||||
/approve_screenshot/:screenshot
|
||||
* Users: 403
|
||||
* Moderators: approve a screenshot
|
||||
|
||||
/keep_screenshot/:screenshot
|
||||
* Users: 403
|
||||
* Moderators: re-approve a screenshot that was requested for deletion
|
||||
|
||||
## Cron jobs
|
||||
|
||||
Parse Nginx access logs to count the number of visits of each package:
|
||||
|
||||
RAILS_ENV=production bundle exec rake debshots:accesslog2visits[/var/log/nginx/cache-access.log.1]
|
||||
|
||||
As a cron job:
|
||||
|
||||
41 9 * * * cd $HOME/debshots ; RAILS_ENV=production /home/debshots/.rbenv/shims/bundle exec rake debshots:accesslog2visits[/var/log/nginx/cache-access.log.1]
|
||||
|
||||
|
||||
# Authentication
|
||||
|
||||
This web application tries to avoid to create yet another directory
|
||||
of user accounts. So it leverages other common web sites like StackExchange,
|
||||
Debian SSO (using browser/client certificates) or GitHub as single-sign-on
|
||||
services. It is still possible to create local user accounts for administrators
|
||||
but registration is disabled.
|
||||
|
||||
Anonymous uploads are possible but they have to be moderated by an administrator.
|
||||
|
||||
# Authorization
|
||||
|
||||
A user can have one of these access levels.
|
||||
|
||||
## Anonymous
|
||||
|
||||
Without authentication all user actions are moderated. Such users can upload
|
||||
screenshots or report existing/published screenshots as inappropriate. However
|
||||
a user with level 'admin' will have to approve that action.
|
||||
|
||||
## Authenticated by single-sign-login (SSO)
|
||||
|
||||
Users can use different authentication services to login - like Github or
|
||||
StackExchange. Such users internally get an account with a random password
|
||||
created but they will keep logging in via SSO.
|
||||
|
||||
The first upload will have to be moderated. Subsequent uploads will be
|
||||
approved automatically.
|
||||
|
||||
## Authenticated by Debian signle-sign-login (SSO)
|
||||
|
||||
If a user authenticates using a Debian SSO client certificate they will be
|
||||
able to upload screenshots without moderation.
|
||||
(Anohter idea was to restrict them to uploading only screenshots for the
|
||||
packages they maintain. But that may be reliable and is not implemented at
|
||||
this time.)
|
||||
|
||||
## Administrator
|
||||
|
||||
Such a user has an internal user account. The database record has the
|
||||
'admin' field set to 1. Admins can freely upload, moderate and delete
|
||||
screenshots. They can also read the log files.
|
||||
|
||||
|
||||
# Nginx during development
|
||||
|
||||
During development it is advised to run Nginx to proxy ports 3000 (HTTP)
|
||||
and 3001 (HTTPS) to the rails application.
|
||||
|
||||
cd nginx
|
||||
nginx -c `pwd`/nginx.conf
|
||||
|
||||
The SSL certitificate is just a self-signed certificate with the
|
||||
common name "localhost". It was generated using:
|
||||
|
||||
openssl req -x509 -newkey 4096 -nodes -keyout localhost.key -out localhost.crt -days 3650
|
||||
|
||||
Run "rails s" in another window. That will use the built-in Ruby 'Puma'
|
||||
application server that can also be used in production. It requires less
|
||||
configuration than Phusion Passenger but may be slower in regard to
|
||||
performance and multi-threading / multi-core support.
|
||||
|
||||
If you get "400 Bad Request / The SSL certificate error" in the browser then
|
||||
please check the error log (/tmp/error*). Once cause is that the CRL of
|
||||
the Debian SSO has expired and needs to be updated.
|
||||
129
doc/README.Installation.md
Normal file
129
doc/README.Installation.md
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
# Installing debshots
|
||||
|
||||
The easiest way to deploy the application is using Ansible.
|
||||
|
||||
Take a Debian Stretch server (8 GB RAM recommended).
|
||||
|
||||
Install PIP to get a recent version of Ansible:
|
||||
|
||||
apt install python-pip
|
||||
pip install ansible
|
||||
|
||||
Run the Ansible playbook that comes with the debshots repository:
|
||||
|
||||
cd ansible
|
||||
ansible-playbook debshots.yml
|
||||
|
||||
Check out the project repository from https://salsa.debian.org/debian/debshots
|
||||
|
||||
Change to the *ansible* subdirectory and run the main playbook:
|
||||
|
||||
cd ansible
|
||||
ansible-playbook debshots.yml
|
||||
|
||||
This will install PostgreSQL and Nginx, create a database, create a system user
|
||||
and do various configuration.
|
||||
|
||||
Make sure that you set the environment variables as described in README.SSO.md before
|
||||
you start the Rails application. Otherwise most single sign-on providers cannot work.
|
||||
You also need to get the https://salsa.debian.org/debsso-team/debsso/raw/master/update-debsso-ca
|
||||
script and run it in /etc/nginx to update the certficate and CRL used by the Debian
|
||||
single-sign-on provider.
|
||||
|
||||
The production configuration uses Nginx that decrypts HTTPS and either serves static
|
||||
assets (images, Javascript, CSS) itself or requests for dynamic content to the
|
||||
Rails application. You will need a SSL certificate for Nginx. Just get one from
|
||||
LetsEncrypt using the *certbot* tool.
|
||||
|
||||
Finally create an admin user for moderation:
|
||||
|
||||
Start a Rails console:
|
||||
|
||||
rails c -e production
|
||||
|
||||
Create a new user record:
|
||||
|
||||
user = User.create(realname: 'Christoph Haas', password: 'foobartest', email: 'email@christoph-haas.de')
|
||||
user.save!
|
||||
|
||||
## Cron jobs
|
||||
|
||||
Parse Nginx access logs to count the number of visits of each package:
|
||||
|
||||
RAILS_ENV=production bundle exec rake debshots:accesslog2visits[/var/log/nginx/cache-access.log.1]
|
||||
|
||||
As a cron job:
|
||||
|
||||
41 9 * * * cd $HOME/debshots ; RAILS_ENV=production /home/debshots/.rbenv/shims/bundle exec rake debshots:accesslog2visits[/var/log/nginx/cache-access.log.1]
|
||||
|
||||
# Authentication
|
||||
|
||||
This web application tries to avoid to create yet another directory
|
||||
of user accounts. So it leverages other common web sites like StackExchange,
|
||||
Debian SSO (using browser/client certificates) or GitHub as single-sign-on
|
||||
services. It is still possible to create local user accounts for administrators
|
||||
but registration is disabled.
|
||||
|
||||
Anonymous uploads are possible but they have to be moderated by an administrator.
|
||||
|
||||
# Authorization
|
||||
|
||||
A user can have one of these access levels.
|
||||
|
||||
## Anonymous
|
||||
|
||||
Without authentication all user actions are moderated. Such users can upload
|
||||
screenshots or report existing/published screenshots as inappropriate. However
|
||||
a user with level 'admin' will have to approve that action.
|
||||
|
||||
## Authenticated by single-sign-login (SSO)
|
||||
|
||||
Users can use different authentication services to login - like Github or
|
||||
StackExchange. Such users internally get an account with a random password
|
||||
created but they will keep logging in via SSO.
|
||||
|
||||
The first upload will have to be moderated. Subsequent uploads will be
|
||||
approved automatically.
|
||||
|
||||
## Authenticated by Debian signle-sign-login (SSO)
|
||||
|
||||
If a user authenticates using a Debian SSO client certificate they will be
|
||||
able to upload screenshots without moderation.
|
||||
(Anohter idea was to restrict them to uploading only screenshots for the
|
||||
packages they maintain. But that may be reliable and is not implemented at
|
||||
this time.)
|
||||
|
||||
## Administrator
|
||||
|
||||
Such a user has an internal user account. The database record has the
|
||||
'admin' field set to 1. Admins can freely upload, moderate and delete
|
||||
screenshots. They can also read the log files.
|
||||
|
||||
# Nginx during development
|
||||
|
||||
During development it is advised to run Nginx to proxy ports 3000 (HTTP)
|
||||
and 3001 (HTTPS) to the rails application.
|
||||
|
||||
cd nginx
|
||||
nginx -c `pwd`/nginx.conf
|
||||
|
||||
The SSL certitificate is just a self-signed certificate with the
|
||||
common name "localhost". It was generated using:
|
||||
|
||||
openssl req -x509 -newkey 4096 -nodes -keyout localhost.key -out localhost.crt -days 3650
|
||||
|
||||
Run "rails s" in another window. That will use the built-in Ruby 'Puma'
|
||||
application server that can also be used in production. It requires less
|
||||
configuration than Phusion Passenger but may be slower in regard to
|
||||
performance and multi-threading / multi-core support.
|
||||
|
||||
If you get "400 Bad Request / The SSL certificate error" in the browser then
|
||||
please check the error log (/tmp/error*). Once cause is that the CRL of
|
||||
the Debian SSO has expired and needs to be updated.
|
||||
|
||||
# Tidy up the screenshot data:
|
||||
|
||||
bundle exec rake debshots:remove_broken_screenshots
|
||||
|
||||
bundle exec rake debshots:remove_duplicate_images
|
||||
|
||||
102
doc/README.URLs.md
Normal file
102
doc/README.URLs.md
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
# Supported URL paths (aka routes)
|
||||
|
||||
/screenshots/:package_inital/:package/:(id)_:size.png
|
||||
* package = name of the package
|
||||
* id = numerical ID of the screenshot
|
||||
* size = "large" or "small"
|
||||
* Returns a PNG image of a screenshot
|
||||
* Should be served as a static asset
|
||||
|
||||
/image/:(id)_:size.png
|
||||
* id = numerical ID of the screenshot
|
||||
* size = "large" or "small"
|
||||
* Returns a PNG image of a screenshot
|
||||
* Usually used for unapproved images that should just be shown to the uploader
|
||||
|
||||
/about
|
||||
* Return an informational HTML page
|
||||
|
||||
/packages
|
||||
* Return a list of packages with pagination
|
||||
|
||||
/packages/with_screenshots
|
||||
* Return a list of packages with pagination
|
||||
* Select only those packages that have screenshots.
|
||||
|
||||
/packages/without_screenshots
|
||||
* Return a list of packages with pagination
|
||||
* Select only those packages that do not have screenshots.
|
||||
|
||||
/packages/games_without_screenshots (deprecated)
|
||||
* Return a list of packages with pagination
|
||||
* Select only those packages that belong to Debian's "games" section
|
||||
|
||||
/json/packages
|
||||
* Return a JSON data structure of all packages
|
||||
|
||||
/json/screenshots
|
||||
* Return a JSON data structure containing packages and screenshots
|
||||
|
||||
/json/packages-without-screenshots
|
||||
* Return a JSON data structure of those packages that do not have any screenshots
|
||||
|
||||
/packages/moderate
|
||||
* Return a HTML page showing packages that need moderation
|
||||
* Must only be available to logged in moderators
|
||||
|
||||
/upload
|
||||
* Return a HTML page containing an upload form for new screenshots
|
||||
|
||||
/guidelines (deprecated)
|
||||
* Return a HTML page showing general guidelines for creating screenshots
|
||||
* Should just redirect to the /upload page
|
||||
|
||||
/upload/:package
|
||||
* ??
|
||||
|
||||
/uploadfile
|
||||
* ??
|
||||
|
||||
/login
|
||||
* Show a login form
|
||||
|
||||
/logout
|
||||
* Logout the user by invalidating the cookie session.
|
||||
|
||||
/package/:package
|
||||
* Show details of a certain package
|
||||
|
||||
/json/package/:package
|
||||
* Return a JSON data structure containing details of a certain package
|
||||
|
||||
/thumbnail/:package
|
||||
/thumbnail-404/:package (deprecated)
|
||||
* Return a small (160x120 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
|
||||
/thumbnail-with-version/:package/:version
|
||||
* Return a small (160x120 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
* Prefer screenshots of a certain version
|
||||
|
||||
/screenshot/:package
|
||||
/screenshot-404/:package (deprecated)
|
||||
* Return a large (320x240 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
|
||||
/screenshot-with-version/:package/:version
|
||||
* Return a large (320x240 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
* Prefer screenshots of a certain version
|
||||
|
||||
/delete_screenshot/:screenshot
|
||||
* Users: request deletion of a screenshot
|
||||
* Moderators: delete a screenshot
|
||||
|
||||
/approve_screenshot/:screenshot
|
||||
* Users: 403
|
||||
* Moderators: approve a screenshot
|
||||
|
||||
/keep_screenshot/:screenshot
|
||||
* Users: 403
|
||||
* Moderators: re-approve a screenshot that was requested for deletion
|
||||
|
|
@ -57,6 +57,7 @@ Environment variables have to be set before starting the application:
|
|||
export GITHUB_SECRET=…
|
||||
|
||||
Keys can be managed at https://github.com/settings/applications
|
||||
To register a new oauth key go to "Developer Settings" in your Github account.
|
||||
|
||||
Make sure that the allowed callback URL is:
|
||||
https://screenshots.debian.net/users/auth/github/callback
|
||||
|
|
|
|||
357
doc/README.md
357
doc/README.md
|
|
@ -1,352 +1,19 @@
|
|||
== Installing debshots
|
||||
# About debshots
|
||||
|
||||
=== Prepare PostgreSQL database server and user (ident-based authentication)
|
||||
Debshots is a web application written in Ruby-on-Rails that powers the
|
||||
screenshots.debian.net web site. Many services like packages.debian.net,
|
||||
Ubuntu Software or Synaptic rely on it.
|
||||
|
||||
apt install postgresql
|
||||
## State of the application
|
||||
|
||||
su - postgres
|
||||
The application has been revamped a lot and does not resemble the version
|
||||
still used in production. The current state is worth trying but not yet ready
|
||||
for deployment.
|
||||
|
||||
createuser -d debshots
|
||||
## Development
|
||||
|
||||
createdb -O debshots -E UTF8 -T template0 debshots
|
||||
Check out the README.Development.md
|
||||
|
||||
=== Create an application user
|
||||
## Deployment
|
||||
|
||||
adduser debshots
|
||||
|
||||
Allow the user to parse access log files from Nginx:
|
||||
|
||||
adduser debshots adm
|
||||
|
||||
su - debshots
|
||||
|
||||
=== Install the required Ruby version using rbenv
|
||||
|
||||
https://github.com/rbenv/rbenv
|
||||
|
||||
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
|
||||
|
||||
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
|
||||
|
||||
|
||||
# Load rbenv automatically by appending
|
||||
# the following to ~/.bashrc:
|
||||
|
||||
eval "$(rbenv init -)"
|
||||
|
||||
|
||||
https://github.com/rbenv/ruby-build#readme
|
||||
|
||||
git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
|
||||
|
||||
=== Get the application
|
||||
|
||||
Clone the debshots Git repository.
|
||||
|
||||
=== Install the Ruby dependencies
|
||||
|
||||
rbenv install
|
||||
|
||||
echo "gem: --no-rdoc --no-ri" > ~/.gemrc
|
||||
|
||||
gem install bundler
|
||||
|
||||
bundle install --deployment
|
||||
|
||||
=== Copy screenshots and database from the former live website
|
||||
|
||||
scp screenshots.debian.net:debshots-screenshots.tar .
|
||||
|
||||
tar -C public/ -xvf debshots-screenshots.tar
|
||||
|
||||
scp screenshots.debian.net:debshots.sql .
|
||||
|
||||
psql debshots < debshots.sql
|
||||
|
||||
Switch to production environment:
|
||||
|
||||
export RAILS_ENV=production
|
||||
|
||||
Migrate database:
|
||||
|
||||
bundle exec rake db:migrate RAILS_ENV=production
|
||||
|
||||
(config/database.yml: production: user/password/host must be commented out)
|
||||
|
||||
Convert screenshots to new format (paperclip):
|
||||
|
||||
bundle exec rake debshots:screenshots_to_paperclip RAILS_ENV=production
|
||||
|
||||
Remove old screenshots directory structure:
|
||||
|
||||
rm -r public/live
|
||||
|
||||
Render the assets:
|
||||
|
||||
bundle exec rake assets:precompile RAILS_ENV=production
|
||||
|
||||
Test the application:
|
||||
|
||||
bundle exec rails s -b 0.0.0.0 -e production
|
||||
|
||||
http://...:3000/
|
||||
|
||||
Create a user for moderation:
|
||||
|
||||
bundle exec rails c -e production
|
||||
|
||||
user = User.create(realname: 'Christoph Haas', password: 'foobartest', email: 'email@christoph-haas.de', provider: 'local')
|
||||
user.save!
|
||||
|
||||
Tidy up the screenshot data:
|
||||
|
||||
bundle exec rake debshots:remove_broken_screenshots
|
||||
|
||||
bundle exec rake debshots:remove_duplicate_images
|
||||
|
||||
Import new package information:
|
||||
|
||||
bundle exec rake debshots:list_deb_repos
|
||||
|
||||
bundle exec rake debshots:update_from_deb_repos
|
||||
|
||||
bundle exec debshots:update_longdescription_from_deb_repos
|
||||
|
||||
=== Install passenger to run the application behind nginx
|
||||
|
||||
https://www.phusionpassenger.com/library/walkthroughs/deploy/ruby/ownserver/nginx/oss/jessie/install_passenger.html
|
||||
|
||||
Example nginx vhost config:
|
||||
|
||||
##
|
||||
# You should look at the following URL's in order to grasp a solid understanding
|
||||
# of Nginx configuration files in order to fully unleash the power of Nginx.
|
||||
# http://wiki.nginx.org/Pitfalls
|
||||
# http://wiki.nginx.org/QuickStart
|
||||
# http://wiki.nginx.org/Configuration
|
||||
#
|
||||
# Generally, you will want to move this file somewhere, and start with a clean
|
||||
# file but keep this around for reference. Or just disable in sites-enabled.
|
||||
#
|
||||
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
|
||||
##
|
||||
|
||||
#proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=assets:768m use_temp_path=off;
|
||||
|
||||
#add_header X-Cache-Status $upstream_cache_status;
|
||||
#add_header X-Runtime 42;
|
||||
more_clear_headers 'X-Runtime';
|
||||
more_clear_headers 'X-Powered-By';
|
||||
more_clear_headers 'Server';
|
||||
server_tokens off;
|
||||
|
||||
#passenger_show_version_in_header off;
|
||||
|
||||
#log_format proxy '[$time_local] Cache: $upstream_cache_status $upstream_addr $upstream_response_time $status $bytes_sent $proxy_add_x_forwarded_for $request_uri "$http_referer" "$http_user_agent"';
|
||||
#access_log /var/log/nginx/debshots-access.log proxy;
|
||||
|
||||
# Default server configuration
|
||||
#
|
||||
#passenger_log_level 5;
|
||||
|
||||
server {
|
||||
listen 85.25.83.22:80 default_server;
|
||||
#listen 80 default_server;
|
||||
#listen [::]:80 default_server;
|
||||
|
||||
# SSL configuration
|
||||
#
|
||||
# listen 443 ssl default_server;
|
||||
# listen [::]:443 ssl default_server;
|
||||
#
|
||||
# Self signed certs generated by the ssl-cert package
|
||||
# Don't use them in a production server!
|
||||
#
|
||||
# include snippets/snakeoil.conf;
|
||||
|
||||
passenger_enabled on;
|
||||
passenger_ruby /home/debshots/.rbenv/versions/2.3.1/bin/ruby;
|
||||
#passenger_pass_header X-Accel-Redirect;
|
||||
|
||||
#proxy_cache_methods GET HEAD;
|
||||
|
||||
root /home/debshots/debshots/public;
|
||||
|
||||
# Add index.php to the list if you are using PHP
|
||||
#index index.html index.htm index.nginx-debian.html;
|
||||
|
||||
server_name _;
|
||||
#proxy_cache assets;
|
||||
#add_header X-Cache-Status $upstream_cache_status;
|
||||
#proxy_ignore_headers Cache-Control;
|
||||
|
||||
#add_header X-Debian rocks;
|
||||
|
||||
location /assets/ {
|
||||
alias /home/debshots/debshots/public/assets/;
|
||||
expires 1h;
|
||||
#include debshots_params;
|
||||
add_header X-Coffee assets;
|
||||
add_header Cache-Control public;
|
||||
}
|
||||
location /logo/ {
|
||||
alias /home/debshots/debshots/public/logo/;
|
||||
expires 1h;
|
||||
add_header X-Coffee logo;
|
||||
add_header Cache-Control public;
|
||||
}
|
||||
location /screenshots/ {
|
||||
alias /home/debshots/debshots/public/screenshots/;
|
||||
expires max;
|
||||
add_header X-Coffee screenshots;
|
||||
add_header Cache-Control public;
|
||||
}
|
||||
location /images/ {
|
||||
alias /home/debshots/debshots/public/images/;
|
||||
expires 1h;
|
||||
add_header X-Coffee images;
|
||||
add_header Cache-Control public;
|
||||
}
|
||||
|
||||
# Send thumbnails using X-Sendfile / X-Accel-Redirect
|
||||
# The correct thumbnail is computed by the Rails application so it cannot be served directly.
|
||||
location /thumbnail/ {
|
||||
expires 1d;
|
||||
add_header X-Coffee thumbnail;
|
||||
add_header Cache-Control public;
|
||||
|
||||
passenger_set_header X-Sendfile-Type "X-Accel-Redirect";
|
||||
passenger_env_var HTTP_X_ACCEL_MAPPING /home/debshots/debshots/public/=/__send_file_accel/;
|
||||
passenger_pass_header X-Accel-Redirect;
|
||||
}
|
||||
|
||||
# Send public assets using X-Sendfile / X-Accel-Redirect (e.g. public/images/dummy/...)
|
||||
location /public/ {
|
||||
expires 1h;
|
||||
add_header X-Coffee public;
|
||||
add_header Cache-Control public;
|
||||
|
||||
passenger_set_header X-Sendfile-Type "X-Accel-Redirect";
|
||||
passenger_env_var HTTP_X_ACCEL_MAPPING /home/debshots/debshots/public/=/__send_file_accel/;
|
||||
passenger_pass_header X-Accel-Redirect;
|
||||
}
|
||||
|
||||
location /__send_file_accel/ {
|
||||
internal;
|
||||
alias /home/debshots/debshots/public/;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
== Supported URL paths (aka routes)
|
||||
|
||||
/screenshots/:package_inital/:package/:(id)_:size.png
|
||||
* package = name of the package
|
||||
* id = numerical ID of the screenshot
|
||||
* size = "large" or "small"
|
||||
* Returns a PNG image of a screenshot
|
||||
* Should be served as a static asset
|
||||
|
||||
/image/:(id)_:size.png
|
||||
* id = numerical ID of the screenshot
|
||||
* size = "large" or "small"
|
||||
* Returns a PNG image of a screenshot
|
||||
* Usually used for unapproved images that should just be shown to the uploader
|
||||
|
||||
/about
|
||||
* Return an informational HTML page
|
||||
|
||||
/packages
|
||||
* Return a list of packages with pagination
|
||||
|
||||
/packages/with_screenshots
|
||||
* Return a list of packages with pagination
|
||||
* Select only those packages that have screenshots.
|
||||
|
||||
/packages/without_screenshots
|
||||
* Return a list of packages with pagination
|
||||
* Select only those packages that do not have screenshots.
|
||||
|
||||
/packages/games_without_screenshots (deprecated)
|
||||
* Return a list of packages with pagination
|
||||
* Select only those packages that belong to Debian's "games" section
|
||||
|
||||
/json/packages
|
||||
* Return a JSON data structure of all packages
|
||||
|
||||
/json/screenshots
|
||||
* Return a JSON data structure containing packages and screenshots
|
||||
|
||||
/json/packages-without-screenshots
|
||||
* Return a JSON data structure of those packages that do not have any screenshots
|
||||
|
||||
/packages/moderate
|
||||
* Return a HTML page showing packages that need moderation
|
||||
* Must only be available to logged in moderators
|
||||
|
||||
/upload
|
||||
* Return a HTML page containing an upload form for new screenshots
|
||||
|
||||
/guidelines (deprecated)
|
||||
* Return a HTML page showing general guidelines for creating screenshots
|
||||
* Should just redirect to the /upload page
|
||||
|
||||
/upload/:package
|
||||
* ??
|
||||
|
||||
/uploadfile
|
||||
* ??
|
||||
|
||||
/login
|
||||
* Show a login form
|
||||
|
||||
/logout
|
||||
* Logout the user by invalidating the cookie session.
|
||||
|
||||
/package/:package
|
||||
* Show details of a certain package
|
||||
|
||||
/json/package/:package
|
||||
* Return a JSON data structure containing details of a certain package
|
||||
|
||||
/thumbnail/:package
|
||||
/thumbnail-404/:package (deprecated)
|
||||
* Return a small (160x120 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
|
||||
/thumbnail-with-version/:package/:version
|
||||
* Return a small (160x120 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
* Prefer screenshots of a certain version
|
||||
|
||||
/screenshot/:package
|
||||
/screenshot-404/:package (deprecated)
|
||||
* Return a large (320x240 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
|
||||
/screenshot-with-version/:package/:version
|
||||
* Return a large (320x240 pixel) PNG screenshot file of a package
|
||||
* If the package does not have any screenshots then return a 404 code
|
||||
* Prefer screenshots of a certain version
|
||||
|
||||
/delete_screenshot/:screenshot
|
||||
* Users: request deletion of a screenshot
|
||||
* Moderators: delete a screenshot
|
||||
|
||||
/approve_screenshot/:screenshot
|
||||
* Users: 403
|
||||
* Moderators: approve a screenshot
|
||||
|
||||
/keep_screenshot/:screenshot
|
||||
* Users: 403
|
||||
* Moderators: re-approve a screenshot that was requested for deletion
|
||||
|
||||
== Cron jobs
|
||||
|
||||
Parse Nginx access logs to count the number of visits of each package:
|
||||
|
||||
RAILS_ENV=production bundle exec rake debshots:accesslog2visits[/var/log/nginx/cache-access.log.1]
|
||||
|
||||
As a cron job:
|
||||
|
||||
41 9 * * * cd $HOME/debshots ; RAILS_ENV=production /home/debshots/.rbenv/shims/bundle exec rake debshots:accesslog2visits[/var/log/nginx/cache-access.log.1]
|
||||
Read the README.Installation.md
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue