Commit graph

48 commits

Author SHA1 Message Date
7ff36f1c5c More relaxed blacklist for importing packages 2016-10-30 13:03:04 +01:00
046897f1c6 Rake task to clean up broken screenshots works now 2016-08-17 09:27:50 +02:00
b7e82757eb Speed improvement in updating database
Instead of querying the database one package at a time I am now querying
all packages and just update those who have a visits count.
2016-08-11 11:20:37 +02:00
2edae8fbc2 Progress counter in 5% steps added. Comments added. 2016-08-11 09:26:27 +02:00
e895e931fc Regexp now correctly matches empty referers and user agents 2016-08-11 09:25:49 +02:00
04baeb73e7 Added rake task to count package visits from Nginx log files
Example call:
rake debshots:accesslog2visits['cache-access-example.log']
2016-08-10 18:59:59 +02:00
88728c0bba Keep leading spacesd when parsing the long description 2016-06-29 21:50:29 +02:00
670df2f277 Fixed bug that parsed description fields incorrectly
Example:

Description-en: 389 Directory Server suite - server
Based on the Lightweight Directory Access Protocol (LDAP), the 389
Directory Server is designed to manage large directories of users and
resources robustly and scalably.
.
Its key features include:
* four-way multi-master replication;
* great scalability;
* extensive documentation;
* Active Directory user and group synchronization;
* secure authentication and transport;
* support for LDAPv3;
* graphical management console;
* on-line, zero downtime update of schema, configuration, and
    in-tree Access Control Information.

The "Its key features include:" was mistaken as a key.
2016-06-29 10:43:01 +02:00
4d65c071f9 Cruft removed 2016-06-29 10:42:22 +02:00
ff254f8e00 Typo fixed 2016-06-22 18:10:43 +02:00
7914d09676 Dealing with empty/nil version/description fields properly. 2016-06-22 18:09:41 +02:00
b947de24f3 Rake task set_empty_screenshot_version_to_zero added 2016-06-21 17:23:24 +02:00
3eeb9a63dc Using created_at instead of legacy 'uploaddatetime' field 2016-03-04 15:41:50 +01:00
28e4f8e705 Using own custom Foundation 6 paginator 2016-03-03 17:11:54 +01:00
c67f3062dc Package importer fixed 2016-03-02 14:13:44 +01:00
332e8f4b01 version comparison fixed 2016-03-01 12:57:25 +01:00
39149c26ad Adding :uploaddatetime because it is mentioned in the default scope 2016-02-29 15:14:28 +01:00
99059f54df Typo fixed 2016-02-29 15:05:25 +01:00
0d10c8ef0a Cruft removed 2016-02-26 17:33:40 +01:00
26c8605256 Function to compare package versions added 2016-02-26 17:32:04 +01:00
06f09c0486 Count updates/additions correctly. Add fields correctly. Various other fixes. 2015-06-28 15:12:43 +02:00
dade32145e Using .new instead of .create
Otherwise .create() would fail instantly because it does not
populate the fields during the creation and can't create a
valid object.
2015-06-28 15:11:39 +02:00
0ea7f96495 Parse keywords from Debian control file in a non-greedy way
Previously this:
Description: foo: bar
had been parsed into
"Description: foo" -> "bar"

Now it properly parses into
"Description" -> "foo: bar"
2015-06-28 15:09:39 +02:00
8e890fbaa6 Actually updating package data.
Added debug printing which regexp matched.
2015-06-27 00:15:32 +02:00
6ef32e4bea Screenshot/Paperclip migration Rake task checks for validation errors 2015-06-26 21:18:56 +02:00
806a7cbcee Added rake task to remove orphaned screenshot records 2015-06-15 13:05:01 +02:00
019ff7c519 Changing Rake namespace to "debshots"
Let's group all debshots-related Rake tasks under a common namespace
2015-04-27 00:17:21 +02:00
2df1024b9b Added Rake task to remove duplicate screenshots 2015-04-27 00:16:08 +02:00
9a0da2b396 Unused variable removed 2015-04-24 17:00:45 +02:00
da6309400c Refactored check if a package name or section is blacklisted 2015-04-24 17:00:15 +02:00
7c6b5c7205 Removed component argument from DebImporter::Release.new 2015-04-24 16:59:47 +02:00
f511dfbd56 Handle missing files/URLs gracefully.
If no version of a URL is found the importer just skips the URL now instead of throwing an error.
2015-04-24 16:58:08 +02:00
c944e1ba91 Fixed parser to add proper newlines in multi-line values
Example:

Key: Value1
 Value2
 Value3

Before: "Value1Value2\nValue"
After: "Value1\nValue2\nValue"
2015-04-24 13:48:55 +02:00
8031edf2d1 Tidying up 2015-04-24 12:20:19 +02:00
f1059c520c Refactoring the search for compress files (bz2, gz, ...) 2015-04-24 11:18:21 +02:00
63e3c62356 Importer tries to read long descriptions from translations file. Still buggy.
The description text is not properly split after the first line.
2015-04-23 17:19:51 +02:00
c05b52895d Parsing of i18n file added.
Allows us to read long descriptions and add them to the package information
2015-04-23 00:29:30 +02:00
a15b2b6674 DEB importer can now handle bzip2 files 2015-04-23 00:04:06 +02:00
beeb4cd37a Stats counter added. Limiting to wanted archs. 2015-04-22 23:03:49 +02:00
8b8073bdd0 Added "debshots" namespace and list_deb_repos task 2015-04-22 21:45:36 +02:00
1034afa92d Added "/" to URL paths when reading HTTP repositories 2015-04-22 21:45:06 +02:00
13218841f4 Optimisation: only query database if the package is not obviously blacklisted
Performance gain: 20x
2015-04-20 20:33:40 +02:00
e4e8c0c0db Changed "puts" to proper logging 2015-01-04 14:20:54 +01:00
38df37057a Added migration script to add screenshots to Paperclip 2015-01-04 14:20:31 +01:00
c4390af52a Script tidied up 2014-12-07 22:49:37 +01:00
ef30db8b55 Added Rake task to import packages from Debian repositories 2014-12-05 01:23:45 +01:00
5df82dc0e5 Added library to parse Debian control files 2014-12-05 01:23:13 +01:00
3af24f56dc Blank rails 4.0 project started 2013-07-27 00:19:29 +02:00