89 Commits

Author SHA1 Message Date
Trenton H
c8a62715ec Feature: Allow setting backend configuration settings via the UI (#5126)
* Saving some start on this

* At least partially working for the tesseract parser

* Problems with migration testing need to figure out

* Work around that error

* Fixes max m_pixels

* Moving the settings to main paperless application

* Starting some consumer options

* More fixes and work

* Fixes these last tests

* Fix max_length on OcrSettings.mode field

* Fix all fields on Common & Ocr settings serializers

* Umbrellla config view

* Revert "Umbrellla config view"

This reverts commit fbaf9f4be30f89afeb509099180158a3406416a5.

* Updates to use a single configuration object for all settings

* Squashed commit of the following:

commit 8a0a49dd5766094f60462fbfbe62e9921fbd2373
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 23:02:47 2023 -0800

    Fix formatting

commit 66b2d90c507b8afd9507813ff555e46198ea33b9
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 22:36:35 2023 -0800

    Refactor frontend data models

commit 5723bd8dd823ee855625e250df39393e26709d48
Author: Adam Bogdał <adam@bogdal.pl>
Date:   Wed Dec 20 01:17:43 2023 +0100

    Fix: speed up admin panel for installs with a large number of documents (#5052)

commit 9b08ce176199bf9011a6634bb88f616846150d2b
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 15:18:51 2023 -0800

    Update PULL_REQUEST_TEMPLATE.md

commit a6248bec2d793b7690feed95fcaf5eb34a75bfb6
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 15:02:05 2023 -0800

    Chore: Update Angular to v17 (#4980)

commit b1f6f52486d5ba5c04af99b41315eb6428fd1fa8
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 13:53:56 2023 -0800

    Fix: Dont allow null custom_fields property via API (#5063)

commit 638d9970fd468d8c02c91d19bd28f8b0796bdcb1
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 13:43:50 2023 -0800

    Enhancement: symmetric document links (#4907)

commit 5e8de4c1da6eb4eb8f738b20962595c7536b30ec
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 12:45:04 2023 -0800

    Enhancement: shared icon & shared by me filter (#4859)

commit 088bad90306025d3f6b139cbd0ad264a1cbecfe5
Author: Trenton H <797416+stumpylog@users.noreply.github.com>
Date:   Tue Dec 19 12:04:03 2023 -0800

    Bulk updates all the backend libraries (#5061)

* Saving some work on frontend config

* Very basic but dynamically-generated config form

* Saving work on slightly less ugly frontend config

* JSON validation for user_args field

* Fully dynamic config form

* Adds in some additional validators for a nicer error message

* Cleaning up the testing and coverage more

* Reverts unintentional change

* Adds documentation about the settings and the precedence

* Couple more commenting and style fixes

---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2023-12-29 15:42:56 -08:00
frozenbrain
27edc4e1c5 Fix: Delay consumption after MODIFY inotify events (#4626) 2023-11-19 14:18:43 -08:00
Trenton H
7769b8968c Fixes the returning of a set, change to a list instead (#4442) 2023-10-25 16:18:13 -07:00
Trenton H
7291f11be7 Ensures all old connections are closed in certain longer running locations (#4265) 2023-09-25 12:52:22 -07:00
Trenton H
b162d7afc8 Handle when INotify fails to import but the polling is set to 0 still (#4230) 2023-09-20 14:04:42 -07:00
Trenton Holmes
34b80a4d8e Removes support for Python 3.8 and lower from the code base 2023-09-10 11:42:59 -07:00
Trenton H
30655f1b73 Fixes ruff not running isort against the codebase 2023-04-26 09:35:27 -07:00
Trenton H
d2c02b9102 Configures ruff as the one stop linter and resolves warnings it raised 2023-04-01 17:03:52 -07:00
Trenton H
36a6df0bae Creates a data model for the document consumption, allowing stronger typing of arguments and setting of some information about the file only once 2023-04-01 11:05:34 -07:00
Trenton H
8d0e65d369 Limit the number of threads waiting for files to be ready during polling 2023-03-21 07:46:57 -07:00
Trenton H
a9c77216b2 Updates ignore path filtering so files in a folder in an ignored folder will be ignored correctly 2023-02-16 09:05:11 -08:00
Trenton H
873522fd82 Updates the exporter to use pathlib and add a few more tests for coverage 2023-01-14 06:33:12 -08:00
Michael Shamoon
1268daf6fe Celery JSON serializable object for override_tag_ids 2022-11-07 07:41:23 -08:00
Trenton Holmes
09287701ae Transitions the backend to celery and celery beat 2022-09-26 11:25:34 -07:00
Trenton Holmes
f06cb40141 Corrects the timeout's interaction with files waiting to be consumed 2022-08-14 17:47:59 -07:00
Trenton Holmes
9e091e333d Attempts to make production consumer more event driven while still allowing unit testing 2022-08-14 17:47:59 -07:00
Trenton Holmes
5d2062a076 Includes the actual OSError string in the log, instead of assuming it's a busy file 2022-07-11 11:57:02 -07:00
Trenton Holmes
836373bc3b Adds configuration variable to the inotify debounce timing 2022-05-15 11:48:12 -07:00
Trenton Holmes
74815a489f Minor improvements for quality of life 2022-05-09 12:05:29 -07:00
Trenton Holmes
dcc2e018ff Adds additional checking for both inotify and polling around document still being busy before consuming it 2022-04-10 12:21:34 -07:00
Trenton Holmes
6635fa5f0d Runs the pre-commit hooks over all the Python files 2022-03-11 11:34:28 -08:00
kpj
c56cb25b5f Format Python code with black 2022-02-27 15:26:41 +01:00
Daniel Albers
bc685e8edb Make ignores configurable
Adds config file setting PAPERLESS_CONSUMER_IGNORE_PATTERNS.
2021-08-18 22:23:18 +02:00
jonaswinkler
3e42ceef38 ignore macOS specific files 2021-05-19 19:56:01 +02:00
jonaswinkler
b957531100 run the polling file change checks on individual threads to speed up queueing of new files 2021-02-21 12:43:55 +01:00
jonaswinkler
692557a364 increased default delay when waiting for file changes with polling 2021-02-21 12:14:54 +01:00
jonaswinkler
555e37958f better exception logging 2021-02-11 22:16:41 +01:00
jonaswinkler
e5a7dc0cc7 rework most of the logging 2021-02-05 01:10:29 +01:00
jonaswinkler
eeff7b3bdb code style 2021-02-02 23:58:25 +01:00
jonaswinkler
ba48e0ca1a revert a change 2021-01-21 22:29:47 +01:00
jonaswinkler
6e29f64a8e revert changes for #351 2021-01-20 11:56:09 +01:00
jonaswinkler
280ba2fcc2 fixes #351 2021-01-19 14:43:55 +01:00
jonaswinkler
9ded48acab fixes #300 2021-01-09 01:54:51 +01:00
jonaswinkler
33ca08f794 tags from folders: case insensitive 2020-12-09 00:07:22 +01:00
jonaswinkler
ba7bf9b2d2 removed slugs entirely, since their only purpose was purely cosmetic anyway. 2020-12-09 00:04:37 +01:00
jonaswinkler
e4eeb29f54 checking file types against parsers in the consumer. 2020-12-01 15:26:05 +01:00
jayme-github
a90b7a647e Create tags from sub directories
The names of sub directories in the consumer directory will be added as
tags for the document to be consumed.
To enable this, set:
PAPERLESS_CONSUMER_RECURSIVE=1
PAPERLESS_CONSUMER_SUBDIRS_AS_TAGS=1

Fixes #50
2020-11-30 14:22:35 +01:00
jonaswinkler
c6627eac1f fix warnings about unclosed files. 2020-11-27 13:19:58 +01:00
jonaswinkler
29867ba6bd inotify: cleanup descriptor when done 2020-11-27 13:12:34 +01:00
jonaswinkler
72b4f817df moved consumption dir check into the correct spot 2020-11-27 13:12:13 +01:00
jonaswinkler
221c1e76e9 couple changes to the consumer. 2020-11-26 18:55:05 +01:00
jonaswinkler
dac7971cd6 Apparently there was a very good reason to use inotify. fixes #46 complete with test cases for inotify and polling. 2020-11-26 17:57:03 +01:00
Jonas Winkler
9c23207b84 workaround for a bug in django-q: task results with too long names would not show up in the result lists. 2020-11-22 13:53:19 +01:00
Jonas Winkler
afc3753e58 code cleanup 2020-11-21 14:03:45 +01:00
Jonas Winkler
5eb5aa6fb6 removed unused code. 2020-11-18 00:54:51 +01:00
Jonas Winkler
24bb8c71c9 Merge branch 'dev' into mail_rework 2020-11-17 00:23:10 +01:00
Jonas Winkler
e30f0b274b added more testing 2020-11-16 23:16:37 +01:00
Jonas Winkler
2119eb4c15 added option for polling 2020-11-16 18:52:13 +01:00
Jonas Winkler
bd04c966c5 first version of the new consumer. 2020-11-16 18:26:54 +01:00
Jonas Winkler
d99b4623f8 first implementation of the mail rework 2020-11-15 23:56:22 +01:00