From e530750fc6e405bf3a37981d9da8dbb0d33c840a Mon Sep 17 00:00:00 2001 From: tooomm Date: Thu, 19 May 2022 22:05:43 +0200 Subject: [PATCH 01/32] update heading levels for v1.7.0 --- docs/changelog.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/changelog.md b/docs/changelog.md index 9ba2f9aa0..9d35ee050 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -70,12 +70,12 @@ ## paperless-ngx 1.7.0 -Breaking Changes +### Breaking Changes - `PAPERLESS_URL` is now required when using a reverse proxy. See [\#674](https://github.com/paperless-ngx/paperless-ngx/pull/674). -Features +### Features - Allow setting more than one tag in mail rules [\@jonasc](https://github.com/jonasc) (\#270) @@ -107,7 +107,7 @@ Features - Parse dates when entered without separators [\@GruberViktor](https://github.com/gruberviktor) (\#250). -Bug Fixes +### Bug Fixes - add \"localhost\" to ALLOWED_HOSTS [\@gador](https://github.com/gador) (\#700). @@ -155,7 +155,7 @@ Bug Fixes - Fix: Include excluded items in dropdown count [\@shamoon](https://github.com/shamoon) (\#263). -Translation +### Translation - [\@miku323](https://github.com/miku323) contributed to Slovenian translation. @@ -168,7 +168,7 @@ Translation - [\@Prominence](https://github.com/Prominence) contributed to Belarusian translation. -Documentation +### Documentation - Fix: scanners table [\@qcasey](https://github.com/qcasey) (\#690). - Add [PAPERLESS\_URL]{.title-ref} env variable & CSRF var @@ -180,7 +180,7 @@ Documentation - Fix minor sphinx errors [\@shamoon](https://github.com/shamoon) (\#322). -Maintenance +### Maintenance - Add `PAPERLESS_URL` env variable & CSRF var [\@shamoon](https://github.com/shamoon) (\#674). From 163231d3076562da4079a13842b5e13cd7470611 Mon Sep 17 00:00:00 2001 From: tooomm Date: Thu, 19 May 2022 23:12:40 +0200 Subject: [PATCH 02/32] Link issues, capitalization and minor fixes --- docs/changelog.md | 174 +++++++++++++++++++++++----------------------- 1 file changed, 87 insertions(+), 87 deletions(-) diff --git a/docs/changelog.md b/docs/changelog.md index 9d35ee050..319592b5d 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -17,7 +17,7 @@ ### Bug Fixes - Feature / fix saved view \& sort field query params [\@shamoon](https://github.com/shamoon) ([\#881](https://github.com/paperless-ngx/paperless-ngx/pull/881)) -- mobile friendlier manage pages [\@shamoon](https://github.com/shamoon) ([\#873](https://github.com/paperless-ngx/paperless-ngx/pull/873)) +- Mobile friendlier manage pages [\@shamoon](https://github.com/shamoon) ([\#873](https://github.com/paperless-ngx/paperless-ngx/pull/873)) - Add timeout to healthcheck [\@shamoon](https://github.com/shamoon) ([\#880](https://github.com/paperless-ngx/paperless-ngx/pull/880)) - Always accept yyyy-mm-dd date inputs [\@shamoon](https://github.com/shamoon) ([\#864](https://github.com/paperless-ngx/paperless-ngx/pull/864)) - Fix local Docker image building [\@stumpylog](https://github.com/stumpylog) ([\#849](https://github.com/paperless-ngx/paperless-ngx/pull/849)) @@ -78,129 +78,129 @@ ### Features - Allow setting more than one tag in mail rules - [\@jonasc](https://github.com/jonasc) (\#270) -- global drag\'n\'drop [\@shamoon](https://github.com/shamoon) - (\#283). + [\@jonasc](https://github.com/jonasc) ([\#270](https://github.com/paperless-ngx/paperless-ngx/pull/270)) +- Global drag\'n\'drop [\@shamoon](https://github.com/shamoon) + ([\#283](https://github.com/paperless-ngx/paperless-ngx/pull/283)) - Fix: download buttons should disable while waiting - [\@shamoon](https://github.com/shamoon) (\#630). -- Update checker [\@shamoon](https://github.com/shamoon) (\#591). + [\@shamoon](https://github.com/shamoon) ([\#630](https://github.com/paperless-ngx/paperless-ngx/pull/630)) +- Update checker [\@shamoon](https://github.com/shamoon) ([\#591](https://github.com/paperless-ngx/paperless-ngx/pull/591)) - Show prompt on password-protected pdfs - [\@shamoon](https://github.com/shamoon) (\#564). + [\@shamoon](https://github.com/shamoon) ([\#564](https://github.com/paperless-ngx/paperless-ngx/pull/564)) - Filtering query params aka browser navigation for filtering - [\@shamoon](https://github.com/shamoon) (\#540). + [\@shamoon](https://github.com/shamoon) ([\#540](https://github.com/paperless-ngx/paperless-ngx/pull/540)) - Clickable tags in dashboard widgets - [\@shamoon](https://github.com/shamoon) (\#515). + [\@shamoon](https://github.com/shamoon) ([\#515](https://github.com/paperless-ngx/paperless-ngx/pull/515)) - Add bottom pagination [\@shamoon](https://github.com/shamoon) - (\#372). + ([\#372](https://github.com/paperless-ngx/paperless-ngx/pull/372)) - Feature barcode splitter [\@gador](https://github.com/gador) - (\#532). -- App loading screen [\@shamoon](https://github.com/shamoon) (\#298). + ([\#532](https://github.com/paperless-ngx/paperless-ngx/pull/532)) +- App loading screen [\@shamoon](https://github.com/shamoon) ([\#298](https://github.com/paperless-ngx/paperless-ngx/pull/298)) - Use progress bar for delayed buttons - [\@shamoon](https://github.com/shamoon) (\#415). + [\@shamoon](https://github.com/shamoon) ([\#415](https://github.com/paperless-ngx/paperless-ngx/pull/415)) - Add minimum length for documents text filter - [\@shamoon](https://github.com/shamoon) (\#401). + [\@shamoon](https://github.com/shamoon) ([\#401](https://github.com/paperless-ngx/paperless-ngx/pull/401)) - Added nav buttons in the document detail view - [\@GruberViktor](https://github.com/gruberviktor) (\#273). + [\@GruberViktor](https://github.com/gruberviktor) ([\#273](https://github.com/paperless-ngx/paperless-ngx/pull/273)) - Improve date keyboard input [\@shamoon](https://github.com/shamoon) - (\#253). -- Color theming [\@shamoon](https://github.com/shamoon) (\#243). + ([\#253](https://github.com/paperless-ngx/paperless-ngx/pull/253)) +- Color theming [\@shamoon](https://github.com/shamoon) ([\#243](https://github.com/paperless-ngx/paperless-ngx/pull/243)) - Parse dates when entered without separators - [\@GruberViktor](https://github.com/gruberviktor) (\#250). + [\@GruberViktor](https://github.com/gruberviktor) ([\#250](https://github.com/paperless-ngx/paperless-ngx/pull/250)) ### Bug Fixes -- add \"localhost\" to ALLOWED_HOSTS - [\@gador](https://github.com/gador) (\#700). -- Fix: scanners table [\@qcasey](https://github.com/qcasey) (\#690). +- Add \"localhost\" to ALLOWED_HOSTS + [\@gador](https://github.com/gador) ([\#700](https://github.com/paperless-ngx/paperless-ngx/pull/700)) +- Fix: scanners table [\@qcasey](https://github.com/qcasey) ([\#690](https://github.com/paperless-ngx/paperless-ngx/pull/690)) - Adds wait for file before consuming - [\@stumpylog](https://github.com/stumpylog) (\#483). + [\@stumpylog](https://github.com/stumpylog) ([\#483](https://github.com/paperless-ngx/paperless-ngx/pull/483)) - Fix: frontend document editing erases time data - [\@shamoon](https://github.com/shamoon) (\#654). + [\@shamoon](https://github.com/shamoon) ([\#654](https://github.com/paperless-ngx/paperless-ngx/pull/654)) - Increase length of SavedViewFilterRule - [\@stumpylog](https://github.com/stumpylog) (\#612). + [\@stumpylog](https://github.com/stumpylog) ([\#612](https://github.com/paperless-ngx/paperless-ngx/pull/612)) - Fixes attachment filename matching during mail fetching - [\@stumpylog](https://github.com/stumpylog) (\#680). + [\@stumpylog](https://github.com/stumpylog) ([\#680](https://github.com/paperless-ngx/paperless-ngx/pull/680)) - Add `PAPERLESS_URL` env variable & CSRF var - [\@shamoon](https://github.com/shamoon) (\#674). + [\@shamoon](https://github.com/shamoon) ([\#674](https://github.com/paperless-ngx/paperless-ngx/discussions/674)) - Fix: download buttons should disable while waiting - [\@shamoon](https://github.com/shamoon) (\#630). + [\@shamoon](https://github.com/shamoon) ([\#630](https://github.com/paperless-ngx/paperless-ngx/pull/630)) - Fixes downloaded filename, add more consumer ignore settings - [\@stumpylog](https://github.com/stumpylog) (\#599). + [\@stumpylog](https://github.com/stumpylog) ([\#599](https://github.com/paperless-ngx/paperless-ngx/pull/599)) - FIX BUG: case-sensitive matching was not possible - [\@danielBreitlauch](https://github.com/danielbreitlauch) (\#594). -- uses shutil.move instead of rename - [\@gador](https://github.com/gador) (\#617). + [\@danielBreitlauch](https://github.com/danielbreitlauch) ([\#594](https://github.com/paperless-ngx/paperless-ngx/pull/594)) +- Uses shutil.move instead of rename + [\@gador](https://github.com/gador) ([\#617](https://github.com/paperless-ngx/paperless-ngx/pull/617)) - Fix npm deps 01.02.22 2 [\@shamoon](https://github.com/shamoon) - (\#610). + ([\#610](https://github.com/paperless-ngx/paperless-ngx/discussions/610)) - Fix npm dependencies 01.02.22 - [\@shamoon](https://github.com/shamoon) (\#600). -- fix issue 416: implement PAPERLESS_OCR_MAX_IMAGE_PIXELS - [\@hacker-h](https://github.com/hacker-h) (\#441). -- fix: exclude cypress from build in Dockerfile - [\@FrankStrieter](https://github.com/FrankStrieter) (\#526). + [\@shamoon](https://github.com/shamoon) ([\#600](https://github.com/paperless-ngx/paperless-ngx/pull/600)) +- Fix issue 416: implement `PAPERLESS_OCR_MAX_IMAGE_PIXELS` + [\@hacker-h](https://github.com/hacker-h) ([\#441](https://github.com/paperless-ngx/paperless-ngx/pull/441)) +- Fix: exclude cypress from build in Dockerfile + [\@FrankStrieter](https://github.com/FrankStrieter) ([\#526](https://github.com/paperless-ngx/paperless-ngx/pull/526)) - Corrections to pass pre-commit hooks - [\@schnuffle](https://github.com/schnuffle) (\#454). + [\@schnuffle](https://github.com/schnuffle) ([\#454](https://github.com/paperless-ngx/paperless-ngx/pull/454)) - Fix 311 unable to click checkboxes in document list - [\@shamoon](https://github.com/shamoon) (\#313). + [\@shamoon](https://github.com/shamoon) ([\#313](https://github.com/paperless-ngx/paperless-ngx/pull/313)) - Fix imap tools bug [\@stumpylog](https://github.com/stumpylog) - (\#393). + ([\#393](https://github.com/paperless-ngx/paperless-ngx/pull/393)) - Fix filterable dropdown buttons arent translated - [\@shamoon](https://github.com/shamoon) (\#366). + [\@shamoon](https://github.com/shamoon) ([\#366](https://github.com/paperless-ngx/paperless-ngx/pull/366)) - Fix 224: \"Auto-detected date is day before receipt date\" - [\@a17t](https://github.com/a17t) (\#246). + [\@a17t](https://github.com/a17t) ([\#246](https://github.com/paperless-ngx/paperless-ngx/pull/246)) - Fix minor sphinx errors [\@shamoon](https://github.com/shamoon) - (\#322). + ([\#322](https://github.com/paperless-ngx/paperless-ngx/pull/322)) - Fix page links hidden [\@shamoon](https://github.com/shamoon) - (\#314). + ([\#314](https://github.com/paperless-ngx/paperless-ngx/pull/314)) - Fix: Include excluded items in dropdown count - [\@shamoon](https://github.com/shamoon) (\#263). + [\@shamoon](https://github.com/shamoon) ([\#263](https://github.com/paperless-ngx/paperless-ngx/pull/263)) ### Translation - [\@miku323](https://github.com/miku323) contributed to Slovenian - translation. + translation - [\@FaintGhost](https://github.com/FaintGhost) contributed to Chinese - Simplified translation. + Simplified translation - [\@DarkoBG79](https://github.com/DarkoBG79) contributed to Serbian - translation. + translation - [Kemal Secer](https://crowdin.com/profile/kemal.secer) contributed - to Turkish translation. + to Turkish translation - [\@Prominence](https://github.com/Prominence) contributed to - Belarusian translation. + Belarusian translation ### Documentation -- Fix: scanners table [\@qcasey](https://github.com/qcasey) (\#690). -- Add [PAPERLESS\_URL]{.title-ref} env variable & CSRF var - [\@shamoon](https://github.com/shamoon) (\#674). +- Fix: scanners table [\@qcasey](https://github.com/qcasey) ([\#690](https://github.com/paperless-ngx/paperless-ngx/pull/690)) +- Add `PAPERLESS_URL` env variable & CSRF var + [\@shamoon](https://github.com/shamoon) ([\#674](https://github.com/paperless-ngx/paperless-ngx/pull/674)) - Fixes downloaded filename, add more consumer ignore settings - [\@stumpylog](https://github.com/stumpylog) (\#599). -- fix issue 416: implement `PAPERLESS_OCR_MAX_IMAGE_PIXELS` - [\@hacker-h](https://github.com/hacker-h) (\#441). + [\@stumpylog](https://github.com/stumpylog) ([\#599](https://github.com/paperless-ngx/paperless-ngx/pull/599)) +- Fix issue 416: implement `PAPERLESS_OCR_MAX_IMAGE_PIXELS` + [\@hacker-h](https://github.com/hacker-h) ([\#441](https://github.com/paperless-ngx/paperless-ngx/pull/441)) - Fix minor sphinx errors [\@shamoon](https://github.com/shamoon) - (\#322). + ([\#322](https://github.com/paperless-ngx/paperless-ngx/pull/322)) ### Maintenance - Add `PAPERLESS_URL` env variable & CSRF var - [\@shamoon](https://github.com/shamoon) (\#674). + [\@shamoon](https://github.com/shamoon) ([\#674](https://github.com/paperless-ngx/paperless-ngx/pull/674)) - Chore: Implement release-drafter action for Changelogs - [\@qcasey](https://github.com/qcasey) (\#669). -- Chore: Add CODEOWNERS [\@qcasey](https://github.com/qcasey) (\#667). + [\@qcasey](https://github.com/qcasey) ([\#669](https://github.com/paperless-ngx/paperless-ngx/pull/669)) +- Chore: Add CODEOWNERS [\@qcasey](https://github.com/qcasey) ([\#667](https://github.com/paperless-ngx/paperless-ngx/pull/667)) - Support docker-compose v2 in install - [\@stumpylog](https://github.com/stumpylog) (\#611). + [\@stumpylog](https://github.com/stumpylog) ([\#611](https://github.com/paperless-ngx/paperless-ngx/pull/611)) - Add Belarusian localization [\@shamoon](https://github.com/shamoon) - (\#588). + ([\#588](https://github.com/paperless-ngx/paperless-ngx/pull/588)) - Add Turkish localization [\@shamoon](https://github.com/shamoon) - (\#536). + ([\#536](https://github.com/paperless-ngx/paperless-ngx/pull/536)) - Add Serbian localization [\@shamoon](https://github.com/shamoon) - (\#504). + ([\#504](https://github.com/paperless-ngx/paperless-ngx/pull/504)) - Create PULL_REQUEST_TEMPLATE.md - [\@shamoon](https://github.com/shamoon) (\#304). + [\@shamoon](https://github.com/shamoon) ([\#304](https://github.com/paperless-ngx/paperless-ngx/pull/304)) - Add Chinese localization [\@shamoon](https://github.com/shamoon) - (\#247). + ([\#247](https://github.com/paperless-ngx/paperless-ngx/pull/247)) - Add Slovenian language for frontend - [\@shamoon](https://github.com/shamoon) (\#315). + [\@shamoon](https://github.com/shamoon) ([\#315](https://github.com/paperless-ngx/paperless-ngx/pull/315)) ## paperless-ngx 1.6.0 @@ -216,46 +216,46 @@ include: - Updated Python and Angular dependencies. - Dropped support for Python 3.7. - Dropped support for Ansible playbooks (thanks - [\@slankes](https://github.com/slankes) \#109). If someone would - like to continue supporting them, please see the [ansible + [\@slankes](https://github.com/slankes) [\#109](https://github.com/paperless-ngx/paperless-ngx/pull/109)). If someone would + like to continue supporting them, please see our [ansible repo](https://github.com/paperless-ngx/paperless-ngx-ansible). - Python code is now required to use Black formatting (thanks - [\@kpj](https://github.com/kpj) \#168). + [\@kpj](https://github.com/kpj) [\#168](https://github.com/paperless-ngx/paperless-ngx/pull/168)). - [\@tribut](https://github.com/tribut) added support for a custom SSO - logout redirect (jonaswinkler\#1258). See + logout redirect ([jonaswinkler\#1258](https://github.com/jonaswinkler/paperless-ng/pull/1258)). See `PAPERLESS_LOGOUT_REDIRECT_URL`. - [\@shamoon](https://github.com/shamoon) added a loading indicator - when document list is reloading (jonaswinkler\#1297). + when document list is reloading ([jonaswinkler\#1297](https://github.com/jonaswinkler/paperless-ng/pull/1297)). - [\@shamoon](https://github.com/shamoon) improved the PDF viewer on - mobile (\#2). + mobile ([\#2](https://github.com/paperless-ngx/paperless-ngx/pull/2)). - [\@shamoon](https://github.com/shamoon) added \'any\' / \'all\' and - \'not\' filtering with tags (\#10). + \'not\' filtering with tags ([\#10](https://github.com/paperless-ngx/paperless-ngx/pull/10)). - [\@shamoon](https://github.com/shamoon) added warnings for unsaved - changes, with smart edit buttons (\#13). + changes, with smart edit buttons ([\#13](https://github.com/paperless-ngx/paperless-ngx/pull/13)). - [\@benjaminfrank](https://github.com/benjaminfrank) enabled a - non-root access to port 80 via systemd (\#18). + non-root access to port 80 via systemd ([\#18](https://github.com/paperless-ngx/paperless-ngx/pull/18)). - [\@tribut](https://github.com/tribut) added simple \"delete to - trash\" functionality (\#24). See `PAPERLESS_TRASH_DIR`. + trash\" functionality ([\#24](https://github.com/paperless-ngx/paperless-ngx/pull/24)). See `PAPERLESS_TRASH_DIR`. - [\@amenk](https://github.com/amenk) fixed the search box overlay - menu on mobile (\#32). + menu on mobile ([\#32](https://github.com/paperless-ngx/paperless-ngx/pull/32)). - [\@dblitt](https://github.com/dblitt) updated the login form to not - auto-capitalize usernames (\#36). + auto-capitalize usernames ([\#36](https://github.com/paperless-ngx/paperless-ngx/pull/36)). - [\@evilsidekick293](https://github.com/evilsidekick293) made the - worker timeout configurable (\#37). See `PAPERLESS_WORKER_TIMEOUT`. + worker timeout configurable ([\#37](https://github.com/paperless-ngx/paperless-ngx/pull/37)). See `PAPERLESS_WORKER_TIMEOUT`. - [\@Nicarim](https://github.com/Nicarim) fixed downloads of UTF-8 - formatted documents in Firefox (\#56). + formatted documents in Firefox ([\#56](https://github.com/paperless-ngx/paperless-ngx/pull/56)). - [\@mweimerskirch](https://github.com/mweimerskirch) sorted the - language dropdown by locale (\#78). + language dropdown by locale ([\#78](https://github.com/paperless-ngx/paperless-ngx/issues/78)). - [\@mweimerskirch](https://github.com/mweimerskirch) enabled the - Czech (\#83) and Danish (\#84) translations. + Czech ([\#83](https://github.com/paperless-ngx/paperless-ngx/pull/83)) and Danish ([\#84](https://github.com/paperless-ngx/paperless-ngx/pull/84)) translations. - [\@cschmatzler](https://github.com/cschmatzler) enabled specifying - the webserver port (\#124). See `PAPERLESS_PORT`. + the webserver port ([\#124](https://github.com/paperless-ngx/paperless-ngx/pull/124)). See `PAPERLESS_PORT`. - [\@muellermartin](https://github.com/muellermartin) fixed an error - when uploading transparent PNGs (\#133). + when uploading transparent PNGs ([\#133](https://github.com/paperless-ngx/paperless-ngx/pull/133)). - [\@shamoon](https://github.com/shamoon) created a slick new logo - (\#165). + ([\#165](https://github.com/paperless-ngx/paperless-ngx/pull/165)). - [\@tim-vogel](https://github.com/tim-vogel) fixed exports missing - groups (\#193). + groups ([\#193](https://github.com/paperless-ngx/paperless-ngx/pull/193)). Known issues: From e568b3000e9304c1aa1febfd6ab6749fc59e09a3 Mon Sep 17 00:00:00 2001 From: shamoon <4887959+shamoon@users.noreply.github.com> Date: Tue, 7 Jun 2022 15:28:49 -0700 Subject: [PATCH 03/32] Add lsio to issue form --- .github/ISSUE_TEMPLATE/bug-report.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml index 3568c9621..556cef93d 100644 --- a/.github/ISSUE_TEMPLATE/bug-report.yml +++ b/.github/ISSUE_TEMPLATE/bug-report.yml @@ -63,9 +63,11 @@ body: attributes: label: Installation method options: - - Docker + - Docker - official image + - Docker - linuxserver.io image - Bare metal - Other (please describe above) + description: Note there are significant differences from the official image and linuxserver.io, please check if your issue is specific to the third-party image. validations: required: true - type: input From 26b12512b1fd25dba7e1180bcf1dbf70b66b8dba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Moritz=20St=C3=BCckler?= Date: Mon, 20 Jun 2022 12:06:54 +0200 Subject: [PATCH 04/32] fix: update scanner capability The Brother ADS-A1700W does indeed support SFTP. I've just bought it, and set it up like this. --- docs/scanners.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/scanners.rst b/docs/scanners.rst index e8685fadc..0e89fbfd1 100644 --- a/docs/scanners.rst +++ b/docs/scanners.rst @@ -18,7 +18,7 @@ Physical scanners +---------+-------------------+-----+------+-----+----------+------+----------+----------------+ | | | FTP | SFTP | NFS | SMB | SMTP | API [1]_ | | +=========+===================+=====+======+=====+==========+======+==========+================+ -| Brother | `ADS-1700W`_ | yes | | | yes | yes | |`holzhannes`_ | +| Brother | `ADS-1700W`_ | yes | yes | | yes | yes | |`holzhannes`_ | +---------+-------------------+-----+------+-----+----------+------+----------+----------------+ | Brother | `ADS-1600W`_ | yes | | | yes | yes | |`holzhannes`_ | +---------+-------------------+-----+------+-----+----------+------+----------+----------------+ From d8cda7fc1b878c43ae10733f6b807c13d50239e9 Mon Sep 17 00:00:00 2001 From: shamoon <4887959+shamoon@users.noreply.github.com> Date: Sat, 2 Jul 2022 17:51:39 -0700 Subject: [PATCH 05/32] Use any-of-labels for stalebot [ci skip] --- .github/stale.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/stale.yml b/.github/stale.yml index ef287a3fd..91ab0e8ae 100644 --- a/.github/stale.yml +++ b/.github/stale.yml @@ -2,8 +2,7 @@ daysUntilStale: 30 # Number of days of inactivity before a stale issue is closed daysUntilClose: 7 -onlyLabels: - - unconfirmed +any-of-labels: unconfirmed,cant-reproduce,stale # Label to use when marking an issue as stale staleLabel: stale # Comment to post when marking an issue as stale. Set to `false` to disable From 8efb97ef4ebfad8690c32ac9e4ae0b328b1c13e1 Mon Sep 17 00:00:00 2001 From: shamoon <4887959+shamoon@users.noreply.github.com> Date: Sat, 2 Jul 2022 19:06:32 -0700 Subject: [PATCH 06/32] Update stale.yml [ci skip] --- .github/stale.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/stale.yml b/.github/stale.yml index 91ab0e8ae..c15e06f90 100644 --- a/.github/stale.yml +++ b/.github/stale.yml @@ -2,7 +2,7 @@ daysUntilStale: 30 # Number of days of inactivity before a stale issue is closed daysUntilClose: 7 -any-of-labels: unconfirmed,cant-reproduce,stale +any-of-issue-labels: unconfirmed,cant-reproduce,stale # Label to use when marking an issue as stale staleLabel: stale # Comment to post when marking an issue as stale. Set to `false` to disable From 278453451ec49366f993a7b9cce22a3dcaab5f1d Mon Sep 17 00:00:00 2001 From: tooomm Date: Fri, 15 Jul 2022 21:18:38 +0200 Subject: [PATCH 07/32] only run on certain labels --- .github/stale.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/stale.yml b/.github/stale.yml index c15e06f90..6b0169bd9 100644 --- a/.github/stale.yml +++ b/.github/stale.yml @@ -1,14 +1,23 @@ # Number of days of inactivity before an issue becomes stale daysUntilStale: 30 + # Number of days of inactivity before a stale issue is closed daysUntilClose: 7 -any-of-issue-labels: unconfirmed,cant-reproduce,stale + +# Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled) +onlyLabels: [unconfirmed,cant-reproduce] + # Label to use when marking an issue as stale staleLabel: stale + # Comment to post when marking an issue as stale. Set to `false` to disable markComment: > This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. + # Comment to post when closing a stale issue. Set to `false` to disable closeComment: false + +# See https://github.com/marketplace/stale for more info on the app +# and https://github.com/probot/stale for the configuration docs From 2ab2d9127df146910130591b541258c3bb6cd4c4 Mon Sep 17 00:00:00 2001 From: shamoon <4887959+shamoon@users.noreply.github.com> Date: Fri, 15 Jul 2022 20:19:28 -0700 Subject: [PATCH 08/32] Use cant-reproduce for stale --- .github/stale.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/stale.yml b/.github/stale.yml index 6b0169bd9..9b473eef5 100644 --- a/.github/stale.yml +++ b/.github/stale.yml @@ -5,7 +5,7 @@ daysUntilStale: 30 daysUntilClose: 7 # Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled) -onlyLabels: [unconfirmed,cant-reproduce] +onlyLabels: [cant-reproduce] # Label to use when marking an issue as stale staleLabel: stale From 4c697ab50e3a4ecc92291659c9ca93921421d61d Mon Sep 17 00:00:00 2001 From: Quinn Casey Date: Sun, 17 Jul 2022 15:23:28 -0700 Subject: [PATCH 09/32] Bump version to beta --- src-ui/src/environments/environment.prod.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src-ui/src/environments/environment.prod.ts b/src-ui/src/environments/environment.prod.ts index 4d64db657..d076f8a78 100644 --- a/src-ui/src/environments/environment.prod.ts +++ b/src-ui/src/environments/environment.prod.ts @@ -5,7 +5,7 @@ export const environment = { apiBaseUrl: document.baseURI + 'api/', apiVersion: '2', appTitle: 'Paperless-ngx', - version: '1.7.1-dev', + version: '1.8.0-rc1', webSocketHost: window.location.host, webSocketProtocol: window.location.protocol == 'https:' ? 'wss:' : 'ws:', webSocketBaseUrl: base_url.pathname + 'ws/', From b1b6d50af602f2d52a2557fb921f36367e9be38c Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Mon, 18 Jul 2022 09:46:31 -0700 Subject: [PATCH 10/32] Adds a couple packages to the Docker image for ocrmypdf and pikepdf --- Dockerfile | 4 ++++ docs/setup.rst | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 981a194c9..187c75cb1 100644 --- a/Dockerfile +++ b/Dockerfile @@ -90,6 +90,10 @@ ARG RUNTIME_PACKAGES="\ tesseract-ocr-fra \ tesseract-ocr-ita \ tesseract-ocr-spa \ + # Suggested for OCRmyPDF + pngquant \ + # Suggested for pikepdf + jbig2dec \ tzdata \ unpaper \ # Mime type detection diff --git a/docs/setup.rst b/docs/setup.rst index 60fbc212e..3aea55467 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -339,7 +339,7 @@ writing. Windows is not and will never be supported. * ``qpdf`` * ``liblept5`` * ``libxml2`` - * ``pngquant`` + * ``pngquant`` (suggested for certain PDF image optimizations) * ``zlib1g`` * ``tesseract-ocr`` >= 4.0.0 for OCR * ``tesseract-ocr`` language packs (``tesseract-ocr-eng``, ``tesseract-ocr-deu``, etc) From 9c1ae96d336b499355cb5053516a36daa60983a0 Mon Sep 17 00:00:00 2001 From: Quinn Casey Date: Mon, 18 Jul 2022 09:48:03 -0700 Subject: [PATCH 11/32] Create PR for changelog instead of direct commit --- .github/workflows/ci.yml | 37 +++++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3c17f72a3..d6db6e6d1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -305,6 +305,10 @@ jobs: publish-release: runs-on: ubuntu-20.04 + outputs: + prerelease: ${{ steps.get_version.outputs.prerelease }} + changelog: ${{ steps.create-release.outputs.body }} + version: ${{ steps.get_version.outputs.version }} needs: - build-release if: github.ref_type == 'tag' && (startsWith(github.ref_name, 'v') || contains(github.ref_name, '-beta.rc')) @@ -348,6 +352,13 @@ jobs: asset_path: ./paperless-ngx.tar.xz asset_name: paperless-ngx-${{ steps.get_version.outputs.version }}.tar.xz asset_content_type: application/x-xz + + append-changelog: + runs-on: ubuntu-20.04 + needs: + - publish-release + if: needs.publish-release.outputs.prerelease == 'false' + steps: - name: Checkout uses: actions/checkout@v3 @@ -358,11 +369,33 @@ jobs: id: append-Changelog working-directory: docs run: | - echo -e "# Changelog\n\n${{ steps.create-release.outputs.body }}\n" > changelog-new.md + git branch ${{ needs.publish-release.outputs.version }}-changelog + git checkout ${{ needs.publish-release.outputs.version }}-changelog + echo -e "# Changelog\n\n${{ needs.publish-release.outputs.changelog }}\n" > changelog-new.md CURRENT_CHANGELOG=`tail --lines +2 changelog.md` echo -e "$CURRENT_CHANGELOG" >> changelog-new.md mv changelog-new.md changelog.md git config --global user.name "github-actions" git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com" git commit -am "Changelog ${{ steps.get_version.outputs.version }} - GHA" - git push origin HEAD:main + git push origin ${{ needs.publish-release.outputs.version }}-changelog + - + name: Create Pull Request + uses: actions/github-script@v6 + with: + script: | + const { repo, owner } = context.repo; + const result = await github.rest.pulls.create({ + title: '[Documentation] Add ${{ needs.publish-release.outputs.version }} changelog', + owner, + repo, + head: '${{ needs.publish-release.outputs.version }}-changelog', + base: 'main', + body: 'This PR is auto-generated by CI.' + }); + github.rest.issues.addLabels({ + owner, + repo, + issue_number: result.data.number, + labels: ['documentation'] + }); From daad634894831b410b9348587ffdde389bf72ae2 Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Fri, 15 Jul 2022 13:45:23 -0700 Subject: [PATCH 12/32] Adds a CI job for hadolint over all the Dockerfiles, fixes the minor thing it complained about --- .github/workflows/reusable-ci-backend.yml | 23 +++++++++++++++++++ Dockerfile | 27 +++++++++++------------ 2 files changed, 36 insertions(+), 14 deletions(-) diff --git a/.github/workflows/reusable-ci-backend.yml b/.github/workflows/reusable-ci-backend.yml index b6459c110..2b54c6a8f 100644 --- a/.github/workflows/reusable-ci-backend.yml +++ b/.github/workflows/reusable-ci-backend.yml @@ -106,3 +106,26 @@ jobs: run: | cd src/ pipenv run coveralls --service=github + + dockerfile-lint: + name: "Lint ${{ matrix.dockerfile }}" + runs-on: ubuntu-20.04 + strategy: + matrix: + dockerfile: + - Dockerfile + - docker-builders/Dockerfile.qpdf + - docker-builders/Dockerfile.jbig2enc + - docker-builders/Dockerfile.psycopg2 + - docker-builders/Dockerfile.pikepdf + fail-fast: false + steps: + - + name: Checkout + uses: actions/checkout@v3 + - + uses: hadolint/hadolint-action@v2.0.0 + with: + failure-threshold: warning + dockerfile: ${{ matrix.dockerfile }} + ignore: DL3008,DL3013,DL3003 diff --git a/Dockerfile b/Dockerfile index 981a194c9..c57f0b806 100644 --- a/Dockerfile +++ b/Dockerfile @@ -117,19 +117,18 @@ COPY gunicorn.conf.py . # setup docker-specific things # Use mounts to avoid copying installer files into the image # These change sometimes, but rarely -ARG DOCKER_SRC=/usr/src/paperless/src/docker/ -WORKDIR ${DOCKER_SRC} +WORKDIR /usr/src/paperless/src/docker/ COPY [ \ - "docker/imagemagick-policy.xml", \ - "docker/supervisord.conf", \ - "docker/docker-entrypoint.sh", \ - "docker/docker-prepare.sh", \ - "docker/paperless_cmd.sh", \ - "docker/wait-for-redis.py", \ - "docker/management_script.sh", \ - "docker/install_management_commands.sh", \ - "${DOCKER_SRC}" \ + "docker/imagemagick-policy.xml", \ + "docker/supervisord.conf", \ + "docker/docker-entrypoint.sh", \ + "docker/docker-prepare.sh", \ + "docker/paperless_cmd.sh", \ + "docker/wait-for-redis.py", \ + "docker/management_script.sh", \ + "docker/install_management_commands.sh", \ + "/usr/src/paperless/src/docker/" \ ] RUN set -eux \ @@ -162,15 +161,15 @@ RUN --mount=type=bind,from=qpdf-builder,target=/qpdf \ && apt-get install --yes --no-install-recommends /qpdf/usr/src/qpdf/libqpdf28_*.deb \ && apt-get install --yes --no-install-recommends /qpdf/usr/src/qpdf/qpdf_*.deb \ && echo "Installing pikepdf and dependencies" \ + && python3 -m pip install --no-cache-dir /pikepdf/usr/src/wheels/pyparsing*.whl \ && python3 -m pip install --no-cache-dir /pikepdf/usr/src/wheels/packaging*.whl \ && python3 -m pip install --no-cache-dir /pikepdf/usr/src/wheels/lxml*.whl \ && python3 -m pip install --no-cache-dir /pikepdf/usr/src/wheels/Pillow*.whl \ - && python3 -m pip install --no-cache-dir /pikepdf/usr/src/wheels/pyparsing*.whl \ && python3 -m pip install --no-cache-dir /pikepdf/usr/src/wheels/pikepdf*.whl \ - && python -m pip list \ + && python3 -m pip list \ && echo "Installing psycopg2" \ && python3 -m pip install --no-cache-dir /psycopg2/usr/src/wheels/psycopg2*.whl \ - && python -m pip list + && python3 -m pip list # Python dependencies # Change pretty frequently From 329a317fdf04ce905b9e3bfcbefb7e3a21f04659 Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Tue, 19 Jul 2022 13:54:33 -0700 Subject: [PATCH 13/32] Configure Hadolint in a single location for both hooks and CI --- .github/workflows/reusable-ci-backend.yml | 4 +--- .hadolint.yml | 8 ++++++++ .pre-commit-config.yaml | 7 ------- 3 files changed, 9 insertions(+), 10 deletions(-) create mode 100644 .hadolint.yml diff --git a/.github/workflows/reusable-ci-backend.yml b/.github/workflows/reusable-ci-backend.yml index 2b54c6a8f..663a954b5 100644 --- a/.github/workflows/reusable-ci-backend.yml +++ b/.github/workflows/reusable-ci-backend.yml @@ -124,8 +124,6 @@ jobs: name: Checkout uses: actions/checkout@v3 - - uses: hadolint/hadolint-action@v2.0.0 + uses: hadolint/hadolint-action@v2.0 with: - failure-threshold: warning dockerfile: ${{ matrix.dockerfile }} - ignore: DL3008,DL3013,DL3003 diff --git a/.hadolint.yml b/.hadolint.yml new file mode 100644 index 000000000..e195127fe --- /dev/null +++ b/.hadolint.yml @@ -0,0 +1,8 @@ +failure-threshold: warning +ignored: + # https://github.com/hadolint/hadolint/wiki/DL3008 + - DL3008 + # https://github.com/hadolint/hadolint/wiki/DL3013 + - DL3013 + # https://github.com/hadolint/hadolint/wiki/DL3003 + - DL3003 diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 46ad91ee8..70496a4d0 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -74,13 +74,6 @@ repos: rev: v2.10.0 hooks: - id: hadolint - args: - - --ignore - - DL3008 # https://github.com/hadolint/hadolint/wiki/DL3008 (should probably do this at some point) - - --ignore - - DL3013 # https://github.com/hadolint/hadolint/wiki/DL3013 (should probably do this too at some point) - - --ignore - - DL3003 # https://github.com/hadolint/hadolint/wiki/DL3003 (seems excessive to use WORKDIR so much) # Shell script hooks - repo: https://github.com/lovesegfault/beautysh rev: v6.2.1 From 27721aef71529e133487294e79585bc2c8f6f451 Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Tue, 19 Jul 2022 14:01:47 -0700 Subject: [PATCH 14/32] Fixes and updates the Hadolint action version --- .github/workflows/reusable-ci-backend.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/reusable-ci-backend.yml b/.github/workflows/reusable-ci-backend.yml index 663a954b5..333a94fc1 100644 --- a/.github/workflows/reusable-ci-backend.yml +++ b/.github/workflows/reusable-ci-backend.yml @@ -124,6 +124,6 @@ jobs: name: Checkout uses: actions/checkout@v3 - - uses: hadolint/hadolint-action@v2.0 + uses: hadolint/hadolint-action@v2.1.0 with: dockerfile: ${{ matrix.dockerfile }} From 16b0f7f9ee96a5fdf3c1c989dba0db9279bc907c Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Tue, 19 Jul 2022 14:18:47 -0700 Subject: [PATCH 15/32] Removes a Dockerfile I can't find referenced anywhere --- docs/Dockerfile | 17 ----------------- 1 file changed, 17 deletions(-) delete mode 100644 docs/Dockerfile diff --git a/docs/Dockerfile b/docs/Dockerfile deleted file mode 100644 index bb4b35e2d..000000000 --- a/docs/Dockerfile +++ /dev/null @@ -1,17 +0,0 @@ -FROM python:3.5.1 - -# Install Sphinx and Pygments -RUN pip install --no-cache-dir Sphinx Pygments \ - # Setup directories, copy data - && mkdir /build - -COPY . /build -WORKDIR /build/docs - -# Build documentation -RUN make html - -# Start webserver -WORKDIR /build/docs/_build/html -EXPOSE 8000/tcp -CMD ["python3", "-m", "http.server"] From 4e78ca5d82cb9b047639d92e0692436434d3a556 Mon Sep 17 00:00:00 2001 From: Michael Shamoon <4887959+shamoon@users.noreply.github.com> Date: Wed, 20 Jul 2022 11:15:35 -0700 Subject: [PATCH 16/32] remove merge error ng-select css --- src-ui/src/styles.scss | 2 -- 1 file changed, 2 deletions(-) diff --git a/src-ui/src/styles.scss b/src-ui/src/styles.scss index a6199b24d..18413276a 100644 --- a/src-ui/src/styles.scss +++ b/src-ui/src/styles.scss @@ -215,8 +215,6 @@ a, a:hover, .btn-link, .btn-link:hover { top: 7px; } - .paperless-input-select .ng-select .ng-select-container - .ng-dropdown-panel .ng-dropdown-panel-items .ng-option.ng-option-marked { background-color: var(--pngx-bg-darker) !important; color: var(--pngx-body-color-accent) !important; From 7a99dcf69309a464648db39e59498a97715238c4 Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Thu, 21 Jul 2022 08:02:11 -0700 Subject: [PATCH 17/32] Adds configuration for database timeout, documentation and troubleshotting suggestion --- docs/configuration.rst | 9 ++++++++- docs/troubleshooting.rst | 16 ++++++++++++++++ src/documents/signals/handlers.py | 28 ++++++++++++++++++++-------- src/paperless/settings.py | 7 +++++++ 4 files changed, 51 insertions(+), 9 deletions(-) diff --git a/docs/configuration.rst b/docs/configuration.rst index c4203472c..fd68f61cb 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -31,7 +31,7 @@ PAPERLESS_REDIS= PAPERLESS_DBHOST= By default, sqlite is used as the database backend. This can be changed here. - Set PAPERLESS_DBHOST and PostgreSQL will be used instead of mysql. + Set PAPERLESS_DBHOST and PostgreSQL will be used instead of sqlite. PAPERLESS_DBPORT= Adjust port if necessary. @@ -60,6 +60,13 @@ PAPERLESS_DBSSLMODE= Default is ``prefer``. +PAPERLESS_DB_TIMEOUT= + Amount of time for a database connection to wait for the database to unlock. + Mostly applicable for an sqlite based installation, consider changing to postgresql + if you need to increase this. + + Defaults to unset, keeping the Django defaults. + Paths and folders ################# diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst index 3db9a069e..1605fed11 100644 --- a/docs/troubleshooting.rst +++ b/docs/troubleshooting.rst @@ -301,3 +301,19 @@ try adjusting the :ref:`polling configuration `. The user will need to manually move the file out of the consume folder and back in, for the initial failing file to be consumed. + +Log reports "Creating PaperlessTask failed". +######################################################### + +You might find messages like these in your log files: + +.. code:: + + [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/src/../consume/SCN_0001.pdf: OS reports file as busy still + +You are likely using an sqlite based installation, with an increased number of workers and are running into sqlite's concurrency limitations. +Uploading or consuming multiple files at once results in many workers attempting to access the database simultaneously. + +Consider changing to the PostgreSQL database if you will be processing many documents at once often. Otherwise, +try tweaking the ``PAPERLESS_DB_TIMEOUT`` setting to allow more time for the database to unlock. This may have +minor performance implications. diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py index f7a04ad51..e2f4fb4f7 100644 --- a/src/documents/signals/handlers.py +++ b/src/documents/signals/handlers.py @@ -11,6 +11,7 @@ from django.contrib.contenttypes.models import ContentType from django.db import DatabaseError from django.db import models from django.db.models import Q +from django.db.utils import OperationalError from django.dispatch import receiver from django.utils import termcolors from django.utils import timezone @@ -506,21 +507,28 @@ def add_to_index(sender, document, **kwargs): @receiver(django_q.signals.pre_enqueue) def init_paperless_task(sender, task, **kwargs): if task["func"] == "documents.tasks.consume_file": - paperless_task, created = PaperlessTask.objects.get_or_create( - task_id=task["id"], - ) - paperless_task.name = task["name"] - paperless_task.created = task["started"] - paperless_task.save() + try: + paperless_task, created = PaperlessTask.objects.get_or_create( + task_id=task["id"], + ) + paperless_task.name = task["name"] + paperless_task.created = task["started"] + paperless_task.save() + except OperationalError as e: + logger.error(f"Creating PaperlessTask failed: {e}") @receiver(django_q.signals.pre_execute) def paperless_task_started(sender, task, **kwargs): try: if task["func"] == "documents.tasks.consume_file": - paperless_task = PaperlessTask.objects.get(task_id=task["id"]) + paperless_task, created = PaperlessTask.objects.get_or_create( + task_id=task["id"], + ) paperless_task.started = timezone.now() paperless_task.save() + except OperationalError as e: + logger.error(f"Creating PaperlessTask failed: {e}") except PaperlessTask.DoesNotExist: pass @@ -529,8 +537,12 @@ def paperless_task_started(sender, task, **kwargs): def update_paperless_task(sender, instance, **kwargs): try: if instance.func == "documents.tasks.consume_file": - paperless_task = PaperlessTask.objects.get(task_id=instance.id) + paperless_task, created = PaperlessTask.objects.get_or_create( + task_id=instance.id, + ) paperless_task.attempted_task = instance paperless_task.save() + except OperationalError as e: + logger.error(f"Creating PaperlessTask failed: {e}") except PaperlessTask.DoesNotExist: pass diff --git a/src/paperless/settings.py b/src/paperless/settings.py index bfb9507ba..e7fd0f3f0 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -327,6 +327,13 @@ if os.getenv("PAPERLESS_DBHOST"): if os.getenv("PAPERLESS_DBPORT"): DATABASES["default"]["PORT"] = os.getenv("PAPERLESS_DBPORT") +if os.getenv("PAPERLESS_DB_TIMEOUT") is not None: + _new_opts = {"timeout": float(os.getenv("PAPERLESS_DB_TIMEOUT"))} + if "OPTIONS" in DATABASES["default"]: + DATABASES["default"]["OPTIONS"].update(_new_opts) + else: + DATABASES["default"]["OPTIONS"] = _new_opts + DEFAULT_AUTO_FIELD = "django.db.models.AutoField" ############################################################################### From 0e2e5f3413ba265ac209ec9e755702671e47f30a Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Tue, 19 Jul 2022 13:57:00 -0700 Subject: [PATCH 18/32] Creates utiliy to ensure all paths in settings are normalized and absolute --- src/paperless/settings.py | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/src/paperless/settings.py b/src/paperless/settings.py index bfb9507ba..5c698532c 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -4,6 +4,7 @@ import math import multiprocessing import os import re +import tempfile from typing import Final from typing import Optional from typing import Set @@ -56,6 +57,13 @@ def __get_float(key: str, default: float) -> float: return float(os.getenv(key, default)) +def __get_path(key: str, default: str) -> str: + """ + Return a normalized, absolute path based on the environment variable or a default + """ + return os.path.abspath(os.path.normpath(os.environ.get(key, default))) + + # NEVER RUN WITH DEBUG IN PRODUCTION. DEBUG = __get_boolean("PAPERLESS_DEBUG", "NO") @@ -66,14 +74,14 @@ DEBUG = __get_boolean("PAPERLESS_DEBUG", "NO") BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -STATIC_ROOT = os.getenv("PAPERLESS_STATICDIR", os.path.join(BASE_DIR, "..", "static")) +STATIC_ROOT = __get_path("PAPERLESS_STATICDIR", os.path.join(BASE_DIR, "..", "static")) -MEDIA_ROOT = os.getenv("PAPERLESS_MEDIA_ROOT", os.path.join(BASE_DIR, "..", "media")) +MEDIA_ROOT = __get_path("PAPERLESS_MEDIA_ROOT", os.path.join(BASE_DIR, "..", "media")) ORIGINALS_DIR = os.path.join(MEDIA_ROOT, "documents", "originals") ARCHIVE_DIR = os.path.join(MEDIA_ROOT, "documents", "archive") THUMBNAIL_DIR = os.path.join(MEDIA_ROOT, "documents", "thumbnails") -DATA_DIR = os.getenv("PAPERLESS_DATA_DIR", os.path.join(BASE_DIR, "..", "data")) +DATA_DIR = __get_path("PAPERLESS_DATA_DIR", os.path.join(BASE_DIR, "..", "data")) TRASH_DIR = os.getenv("PAPERLESS_TRASH_DIR") @@ -83,15 +91,18 @@ MEDIA_LOCK = os.path.join(MEDIA_ROOT, "media.lock") INDEX_DIR = os.path.join(DATA_DIR, "index") MODEL_FILE = os.path.join(DATA_DIR, "classification_model.pickle") -LOGGING_DIR = os.getenv("PAPERLESS_LOGGING_DIR", os.path.join(DATA_DIR, "log")) +LOGGING_DIR = __get_path("PAPERLESS_LOGGING_DIR", os.path.join(DATA_DIR, "log")) -CONSUMPTION_DIR = os.getenv( +CONSUMPTION_DIR = __get_path( "PAPERLESS_CONSUMPTION_DIR", os.path.join(BASE_DIR, "..", "consume"), ) # This will be created if it doesn't exist -SCRATCH_DIR = os.getenv("PAPERLESS_SCRATCH_DIR", "/tmp/paperless") +SCRATCH_DIR = __get_path( + "PAPERLESS_SCRATCH_DIR", + os.path.join(tempfile.gettempdir(), "paperless"), +) ############################################################################### # Application Definition # From 91ba0bd0af089e59157305ea23331c8b86bd8644 Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Fri, 22 Jul 2022 08:53:02 -0700 Subject: [PATCH 19/32] Locks dependencies to the final versions for the beta --- Pipfile.lock | 234 ++++++++++++++++++++++++----------------------- requirements.txt | 12 +-- 2 files changed, 127 insertions(+), 119 deletions(-) diff --git a/Pipfile.lock b/Pipfile.lock index ba1ee3149..e121d9ed2 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -198,11 +198,11 @@ }, "channels-redis": { "hashes": [ - "sha256:5dffd4cc16174125bd4043fc8fe7462ca7403cf801d59a9fa7410ed101fa6a57", - "sha256:6e4565b7c11c6bcde5d48556cb83bd043779697ff03811867d2f895aa6170d56" + "sha256:78e4a2f2b2a744fe5a87848ec36b5ee49f522c6808cefe6c583663d0d531faa8", + "sha256:ba7e2ad170f273c372812dd32aaac102d68d4e508172abb1cfda3160b7333890" ], "index": "pypi", - "version": "==3.4.0" + "version": "==3.4.1" }, "charset-normalizer": { "hashes": [ @@ -724,11 +724,11 @@ }, "ocrmypdf": { "hashes": [ - "sha256:4c3d9ead76d2cbf248fa764bf6950acacc5586a153895e136332d3df0af4f4f5", - "sha256:ed2ad72ef796770c38edf5eb43392c6d166eb8959ad14a19ea0350c510b9c121" + "sha256:c98061536decf1686e057cfe555b435605d68ab0838de3fccef0e7f3cc591eab", + "sha256:d123e8e1c530fa2a7c0a880e048334867ab691907927d63594b0adf94dbcc5e8" ], "index": "pypi", - "version": "==13.6.0" + "version": "==13.6.1" }, "packaging": { "hashes": [ @@ -764,41 +764,41 @@ }, "pikepdf": { "hashes": [ - "sha256:021021b30c0d8bc0637c73463f85141154f8434870d60b18a9b8c9f3763e7bf4", - "sha256:0438140aee2e46c11379fe14a5a2b982b311ad2683a6749091ce60a28ee9fbf2", - "sha256:0d6ff379b9b9f5efb25e1a35e4d18c6e2f2fc6b9515f29620fafdb1a81a7b926", - "sha256:2e2b5fa3889e05ca78c2e7c8737a31f22e5689a38bab70a78d41325c9532cdfb", - "sha256:35d4913fb161fc9d8e32f970a80c8f5a8fda013393072a5b83d05738e06e7f2a", - "sha256:451435fcbbde528ea5650e61af00840c2983b218fcb58ac2d734d9c96836e53d", - "sha256:4ad6926ccdb65830a7477db96dd3fe500d6be4f5f1195caf38ed151c77f873ed", - "sha256:4f62bab6f017974a5cb640493881e1f1af9f653dc3c60c1568332e188d62bce9", - "sha256:692bcd0e1802a4d478734fd5d3a557a9050b906862e70fab2abe942bd8bd0be3", - "sha256:69e081faabcc60a652b95bdf6330198346bf2c79dd60c6935fb1b4aa7fdb7472", - "sha256:714c47c0d8d4ca07f301974b6497acacc9bb9d4387d2d5eb0cc52c0d366f80ce", - "sha256:7bd8e8bd7532c5652927061ae58668297d564d77a8844d54fd83f0a03a27c355", - "sha256:7ded2776ba03340ffd8543cf9a02322ad29f18ad32dff95d3262965d54dc4bd3", - "sha256:87b409871ab24818e49b2061cc7fc64d27f88173263945b318172b4b13b3326f", - "sha256:8a7174e748c1e442aee47a2efdc772ef13947c5da94ecae99e5cf08dde1c6a52", - "sha256:908e5cd64eaa45a3527cb13497baa09fb68aa91faf746e53269d1f3286dd6959", - "sha256:90b6a7fe632462feba79f2981b88b225a7e286f7535c87fac1d9f3c1dc165bba", - "sha256:9b7d0a38c71114c58b3c4ab4f37aa809466e9ccc16762a1cd1badbf6df194545", - "sha256:a184b83c47a22092ecbe4b849440a0a50f5bffb392f89fe5a863e115b0f6cb46", - "sha256:a60e4a6f41c0f97de81a0d9a84e1968e8ef5feb3bfa599ad77975bd9ca8266bc", - "sha256:a97b930121c2a19ad4ce8cc733fb4b45f79152becf21debf949bec51fcc95c29", - "sha256:b06d847c93857127ccfc38a578b64dc460cee39981e50ba856ffa122525b838b", - "sha256:b0f75244c48e81655a1b91fce960bfd42f1c978af273ce0d5f6ee20093741249", - "sha256:b54729e983d8c0289eb94ceb48cf170555ab0928b364e9d3b7a1204fde979647", - "sha256:b637d18b9d6a59199ad1b061e821e9eb6e6a81e3483a8233a03fa66bb40d2957", - "sha256:b8c9e5cc1c73fa4d0dae9f4b184adbc72949973a011f2746c851affec3c0ede3", - "sha256:bc7374c33c7977bdfba05546fdb07b2e5b2c53ac7a54cc1da001431dc6375e26", - "sha256:cfb76f360f914954340fa8c05c8f849f54c55a76bf94a114351ffa9a3665ad4d", - "sha256:d3b0228da670b194cdc470ddc84239f499a11ebade12c9d721195171b78c1866", - "sha256:e3c7c2e5f4cacf76a6340d43c68829a8f4a841728472d03cb996023b350e8548", - "sha256:e6f5267179115af4de26d8d83e6b5f721b3a632cd68f931acd83c78f236c7f3c", - "sha256:eb51f78841e61d2921379d5c275315c7a7fcda35aecd6ddf9e3556bad8a2e142" + "sha256:125d46d635634355ce21d6a6de87a5fff2f12a1086b5d0581639fb981a4ecec2", + "sha256:13d50d69e0d6f7d9ff0d08d38ec714acc907e8f2f443a97f57f5c6bbf7c5d440", + "sha256:173c5089cb3fc4c0944d45e33cb6c676dc817a076df19a30d66d30e9ef0dee55", + "sha256:2720fdc92685003155a7f5c1dc166c4d26621a03eb47987d0ce7f84078cac286", + "sha256:3b3762b31926b145a9014cf59a749c101b6d77e0e87f36c46043a59f4d31ed60", + "sha256:43bf8912f7a3b7d32a66c9a1c33f757d66746a47826a7e087fa81fe6ee4c3fcf", + "sha256:48274fe2afbe66990a86a41b986a759738ded7cbeca3d8b03cc3203a5aefbc2e", + "sha256:59f84c618e6db4655ea33f023621be1b5537d8ebe82d6de7be4bd2b54a53022c", + "sha256:6216355a15d7e93ea8ed1f4ab91ab4b26e5d11b7df4229d0a6501476aa3d80e4", + "sha256:6bef943ce87837d206b3cbb6eabe63018a26eddad9dac34c2d16c67a490629c9", + "sha256:73136cbc8183d43cefb53fc84ecd8cda4d01a568fdf8d7214d8048e98e9ac577", + "sha256:76748ea30d24f3a8c4348c2d4ea649024e6aa51a760b2939d33710f3438df50c", + "sha256:7b550d1ba6d53436b1c42ca82c7fbf66ebd7c82a96f1567c398510fd7ea8562e", + "sha256:811c2b7b85e801cc30754340ae39dae75d0284e0db9ab5113376768f4d981077", + "sha256:8c55a1b1cfb5c68104f26893482bff4bac9d42fca27d212d8a5b78e7851ff98d", + "sha256:8d2420e4211c87c2da5b30610a3de7a26153c7aa4d86150808db7308f6dd751e", + "sha256:8d86189a8679699eddbaacc15d95f55cb27b2447abd9ac15f798567a0bf3694f", + "sha256:993eb93d1beea5b6b2edc3e730f1b528438340eadf3248b9e1c898f4dac57fbf", + "sha256:ac6a85d15b6b3a008fbbc61e7d30c3a92ffd0ef768af1212b362b666d3456036", + "sha256:ade58804ae816d4aac479468fcb45f354a87f1ce69807c4404239e974b7e3d9f", + "sha256:bd41e636920f3afecdc8be3e2b4cebaeb9dcdd6266822c903b54a7c2053cfe52", + "sha256:d0a37b4093e88753e8c689092c9444b120ebb2d1c1228e283724f01da4681d16", + "sha256:d29a9b4b6972026c1146a7fb2d582f9e4ef6d474e0762885cd0c6991fd0cb2d9", + "sha256:d364e7d723d261123b7e05590e8e5dceb5d01e32c70c0f03ddf2bfa82a36904e", + "sha256:d397a061959a5cd7d9869a381dda005eccfef59c33dd3eba6a8e19ce036168be", + "sha256:d6d23916d9a4645ffbb27acae011ee052e479ce7d0dce6c4b9e16cb7a2423eb9", + "sha256:ddc45fb735918616ff96f24639628c00db9067117f250a04e4c430ba81ea8721", + "sha256:e7b2c99181dcb00c0faad1e5a16294e6c75c6ec497262f1df1ed882b6224b40e", + "sha256:eaa64e6583c6be54cc8b33cd793cf070e1e52168f7680e184f34a2febd6b723c", + "sha256:ec03ca399d4855c224e3a4b573b54049730ddd6937ee08cd0562aeba3043e151", + "sha256:ed3183288cdf84d88b76eae3afbbf008800c5a97102fa4fea65a1c740db61a60", + "sha256:f5018081e849cdd3472a4901e00dc2fdacb38985ea9beeeb20e8a6126c76e61c" ], "index": "pypi", - "version": "==5.3.1" + "version": "==5.4.0" }, "pillow": { "hashes": [ @@ -1255,11 +1255,11 @@ }, "setuptools": { "hashes": [ - "sha256:16923d366ced322712c71ccb97164d07472abeecd13f3a6c283f6d5d26722793", - "sha256:db3b8e2f922b2a910a29804776c643ea609badb6a32c4bcc226fd4fd902cce65" + "sha256:0d33c374d41c7863419fc8f6c10bfe25b7b498aa34164d135c622e52580c6b16", + "sha256:c04b44a57a6265fe34a4a444e965884716d34bae963119a76353434d6f18e450" ], "markers": "python_version >= '3.7'", - "version": "==63.1.0" + "version": "==63.2.0" }, "six": { "hashes": [ @@ -1424,20 +1424,25 @@ }, "watchfiles": { "hashes": [ - "sha256:56abed43e645d1f2d6def83e35999cc5758b051aff54ca1065cbfcaea15b3389", - "sha256:65ca99a94fcab29d00aa406526eb29cf198c0661854d59a315596064fed02141", - "sha256:67d4c66e46a564059df4aeedab78f09cba0b697bf36cc77566b0a7015dfb7f5d", - "sha256:6e0e8829d32b05151e6009570449f44f891e05f518e495d25f960e0d0b2d0064", - "sha256:715733c2ac9da67b2790788657ff6f8b3797eb31565bfc592289b523ae907ca2", - "sha256:7b81c6e404b2aa62482a719eb778e4a16d01728302dce1f1512c1e5354a73fda", - "sha256:82238d08d8a49f1a1ba254278cd4329a154f6100b028393059722ebeddd2ff3d", - "sha256:955e8f840e1996a8a41be57de4c03af7b1515a685b7fb6abe222f859e413a907", - "sha256:cab62510f990d195986302aa6a48ed636d685b099927049120d520c96069fa49", - "sha256:d1f9de6b776b3aff17898a4cf5ac5a2d0a16212ea7aad2bbe0ef6aa3e79a96af", - "sha256:d4f45acd1143db6d3ee77a4ff12d3239bc8083108133e6174e9dcce59c1f9902", - "sha256:f7f71012e096e11256fae3b37617a9777980f281e18deb2e789e85cd5b113935" + "sha256:059bd9596429f8c13604b2eb30888a5661b3c79099edc506f11b63be7afe3ca4", + "sha256:09490d258be8fdd7f5141a39b468dede0b4aa4a52f2b2dbfb0f3835ae7c23eca", + "sha256:1bb5f0117c8b93f8e1b22ac0be60cfeb00332959a72e6bbe2073fea27ed086e5", + "sha256:3d3f0397c9128971398a5cbb0fb45852ab2fa4472ac9724c031071e1e39970c0", + "sha256:43d1d517faffa8955c2da0e6f64268e38442d43b50ca73cb686df25f891e49a1", + "sha256:4f712dbe9d8c0365bf46ffe0dd9c6a62cc0acf05ba951f1a53de2b4d5bb63299", + "sha256:59498853d3214d1e4d9b1cb3a06b0011a11f24d31708b1734d9cd7f5a30fe1af", + "sha256:5e3d4c92091d16bca1d61920575dab5d6dcbceda76dccd5fb91da0b7390b4ee9", + "sha256:5fa786d102e7eabef22b2147af531aa70194aabcb35335be81c07c26382b0050", + "sha256:750e40db5efcf3f5f11602dbc6fdf8e96a0eefdbccd271093efe9fa2e9d02ed2", + "sha256:7c80e3907d21ca3f1689f42632d239fdc40ffc1d5f32f564997480f85e94c474", + "sha256:8d635dcba3aab2909bf568765547696d7465d30e2e9c6f5ab99da877b58d29bb", + "sha256:a5f64674559fac56a6bf2f5e086cb3758740140c80711fe3e016f5443b84ef15", + "sha256:bcd085980389bc64fe509188a9caffa4fe13b2616e2e3e674cde58f916b2a8ee", + "sha256:c9e3756cd2ba17e5042e8c9399a08e4bdbe1a366156a164e8373bda30ca096d0", + "sha256:cbdb7814ca43f85ab8569206ab2c3bcd51dd5d1ba582914246784414e6ada62e", + "sha256:d5fb4f3b5c884d4f22f643b0697edbb04942bcad961a8f9a9bfadb73e7a1e229" ], - "version": "==0.15.0" + "version": "==0.16.0" }, "wcwidth": { "hashes": [ @@ -1588,12 +1593,12 @@ }, "zipp": { "hashes": [ - "sha256:56bf8aadb83c24db6c4b577e13de374ccfb67da2078beba1d037c17980bf43ad", - "sha256:c4f6e5bbf48e74f7a38e7cc5b0480ff42b0ae5178957d564d18932525d5cf099" + "sha256:05b45f1ee8f807d0cc928485ca40a07cb491cf092ff587c0df9cb1fd154848d2", + "sha256:47c40d7fe183a6f21403a199b3e4192cca5774656965b0a4988ad2f8feb5f009" ], "index": "pypi", "markers": "python_version < '3.9'", - "version": "==3.8.0" + "version": "==3.8.1" }, "zope.interface": { "hashes": [ @@ -1747,51 +1752,54 @@ "version": "==0.4.5" }, "coverage": { + "extras": [ + "toml" + ], "hashes": [ - "sha256:01c5615d13f3dd3aa8543afc069e5319cfa0c7d712f6e04b920431e5c564a749", - "sha256:106c16dfe494de3193ec55cac9640dd039b66e196e4641fa8ac396181578b982", - "sha256:129cd05ba6f0d08a766d942a9ed4b29283aff7b2cccf5b7ce279d50796860bb3", - "sha256:145f296d00441ca703a659e8f3eb48ae39fb083baba2d7ce4482fb2723e050d9", - "sha256:1480ff858b4113db2718848d7b2d1b75bc79895a9c22e76a221b9d8d62496428", - "sha256:269eaa2c20a13a5bf17558d4dc91a8d078c4fa1872f25303dddcbba3a813085e", - "sha256:26dff09fb0d82693ba9e6231248641d60ba606150d02ed45110f9ec26404ed1c", - "sha256:2bd9a6fc18aab8d2e18f89b7ff91c0f34ff4d5e0ba0b33e989b3cd4194c81fd9", - "sha256:309ce4a522ed5fca432af4ebe0f32b21d6d7ccbb0f5fcc99290e71feba67c264", - "sha256:3384f2a3652cef289e38100f2d037956194a837221edd520a7ee5b42d00cc605", - "sha256:342d4aefd1c3e7f620a13f4fe563154d808b69cccef415415aece4c786665397", - "sha256:39ee53946bf009788108b4dd2894bf1349b4e0ca18c2016ffa7d26ce46b8f10d", - "sha256:4321f075095a096e70aff1d002030ee612b65a205a0a0f5b815280d5dc58100c", - "sha256:4803e7ccf93230accb928f3a68f00ffa80a88213af98ed338a57ad021ef06815", - "sha256:4ce1b258493cbf8aec43e9b50d89982346b98e9ffdfaae8ae5793bc112fb0068", - "sha256:664a47ce62fe4bef9e2d2c430306e1428ecea207ffd68649e3b942fa8ea83b0b", - "sha256:75ab269400706fab15981fd4bd5080c56bd5cc07c3bccb86aab5e1d5a88dc8f4", - "sha256:83c4e737f60c6936460c5be330d296dd5b48b3963f48634c53b3f7deb0f34ec4", - "sha256:84631e81dd053e8a0d4967cedab6db94345f1c36107c71698f746cb2636c63e3", - "sha256:84e65ef149028516c6d64461b95a8dbcfce95cfd5b9eb634320596173332ea84", - "sha256:865d69ae811a392f4d06bde506d531f6a28a00af36f5c8649684a9e5e4a85c83", - "sha256:87f4f3df85aa39da00fd3ec4b5abeb7407e82b68c7c5ad181308b0e2526da5d4", - "sha256:8c08da0bd238f2970230c2a0d28ff0e99961598cb2e810245d7fc5afcf1254e8", - "sha256:961e2fb0680b4f5ad63234e0bf55dfb90d302740ae9c7ed0120677a94a1590cb", - "sha256:9b3e07152b4563722be523e8cd0b209e0d1a373022cfbde395ebb6575bf6790d", - "sha256:a7f3049243783df2e6cc6deafc49ea123522b59f464831476d3d1448e30d72df", - "sha256:bf5601c33213d3cb19d17a796f8a14a9eaa5e87629a53979a5981e3e3ae166f6", - "sha256:cec3a0f75c8f1031825e19cd86ee787e87cf03e4fd2865c79c057092e69e3a3b", - "sha256:d42c549a8f41dc103a8004b9f0c433e2086add8a719da00e246e17cbe4056f72", - "sha256:d67d44996140af8b84284e5e7d398e589574b376fb4de8ccd28d82ad8e3bea13", - "sha256:d9c80df769f5ec05ad21ea34be7458d1dc51ff1fb4b2219e77fe24edf462d6df", - "sha256:e57816f8ffe46b1df8f12e1b348f06d164fd5219beba7d9433ba79608ef011cc", - "sha256:ee2ddcac99b2d2aec413e36d7a429ae9ebcadf912946b13ffa88e7d4c9b712d6", - "sha256:f02cbbf8119db68455b9d763f2f8737bb7db7e43720afa07d8eb1604e5c5ae28", - "sha256:f1d5aa2703e1dab4ae6cf416eb0095304f49d004c39e9db1d86f57924f43006b", - "sha256:f5b66caa62922531059bc5ac04f836860412f7f88d38a476eda0a6f11d4724f4", - "sha256:f69718750eaae75efe506406c490d6fc5a6161d047206cc63ce25527e8a3adad", - "sha256:fb73e0011b8793c053bfa85e53129ba5f0250fdc0392c1591fd35d915ec75c46", - "sha256:fd180ed867e289964404051a958f7cccabdeed423f91a899829264bb7974d3d3", - "sha256:fdb6f7bd51c2d1714cea40718f6149ad9be6a2ee7d93b19e9f00934c0f2a74d9", - "sha256:ffa9297c3a453fba4717d06df579af42ab9a28022444cae7fa605af4df612d54" + "sha256:0895ea6e6f7f9939166cc835df8fa4599e2d9b759b02d1521b574e13b859ac32", + "sha256:0f211df2cba951ffcae210ee00e54921ab42e2b64e0bf2c0befc977377fb09b7", + "sha256:147605e1702d996279bb3cc3b164f408698850011210d133a2cb96a73a2f7996", + "sha256:24b04d305ea172ccb21bee5bacd559383cba2c6fcdef85b7701cf2de4188aa55", + "sha256:25b7ec944f114f70803d6529394b64f8749e93cbfac0fe6c5ea1b7e6c14e8a46", + "sha256:2b20286c2b726f94e766e86a3fddb7b7e37af5d0c635bdfa7e4399bc523563de", + "sha256:2dff52b3e7f76ada36f82124703f4953186d9029d00d6287f17c68a75e2e6039", + "sha256:2f8553878a24b00d5ab04b7a92a2af50409247ca5c4b7a2bf4eabe94ed20d3ee", + "sha256:3def6791adf580d66f025223078dc84c64696a26f174131059ce8e91452584e1", + "sha256:422fa44070b42fef9fb8dabd5af03861708cdd6deb69463adc2130b7bf81332f", + "sha256:4f89d8e03c8a3757aae65570d14033e8edf192ee9298303db15955cadcff0c63", + "sha256:5336e0352c0b12c7e72727d50ff02557005f79a0b8dcad9219c7c4940a930083", + "sha256:54d8d0e073a7f238f0666d3c7c0d37469b2aa43311e4024c925ee14f5d5a1cbe", + "sha256:5ef42e1db047ca42827a85e34abe973971c635f83aed49611b7f3ab49d0130f0", + "sha256:5f65e5d3ff2d895dab76b1faca4586b970a99b5d4b24e9aafffc0ce94a6022d6", + "sha256:6c3ccfe89c36f3e5b9837b9ee507472310164f352c9fe332120b764c9d60adbe", + "sha256:6d0b48aff8e9720bdec315d67723f0babd936a7211dc5df453ddf76f89c59933", + "sha256:6fe75dcfcb889b6800f072f2af5a331342d63d0c1b3d2bf0f7b4f6c353e8c9c0", + "sha256:79419370d6a637cb18553ecb25228893966bd7935a9120fa454e7076f13b627c", + "sha256:7bb00521ab4f99fdce2d5c05a91bddc0280f0afaee0e0a00425e28e209d4af07", + "sha256:80db4a47a199c4563d4a25919ff29c97c87569130375beca3483b41ad5f698e8", + "sha256:866ebf42b4c5dbafd64455b0a1cd5aa7b4837a894809413b930026c91e18090b", + "sha256:8af6c26ba8df6338e57bedbf916d76bdae6308e57fc8f14397f03b5da8622b4e", + "sha256:a13772c19619118903d65a91f1d5fea84be494d12fd406d06c849b00d31bf120", + "sha256:a697977157adc052284a7160569b36a8bbec09db3c3220642e6323b47cec090f", + "sha256:a9032f9b7d38bdf882ac9f66ebde3afb8145f0d4c24b2e600bc4c6304aafb87e", + "sha256:b5e28db9199dd3833cc8a07fa6cf429a01227b5d429facb56eccd765050c26cd", + "sha256:c77943ef768276b61c96a3eb854eba55633c7a3fddf0a79f82805f232326d33f", + "sha256:d230d333b0be8042ac34808ad722eabba30036232e7a6fb3e317c49f61c93386", + "sha256:d4548be38a1c810d79e097a38107b6bf2ff42151900e47d49635be69943763d8", + "sha256:d4e7ced84a11c10160c0697a6cc0b214a5d7ab21dfec1cd46e89fbf77cc66fae", + "sha256:d56f105592188ce7a797b2bd94b4a8cb2e36d5d9b0d8a1d2060ff2a71e6b9bbc", + "sha256:d714af0bdba67739598849c9f18efdcc5a0412f4993914a0ec5ce0f1e864d783", + "sha256:d774d9e97007b018a651eadc1b3970ed20237395527e22cbeb743d8e73e0563d", + "sha256:e0524adb49c716ca763dbc1d27bedce36b14f33e6b8af6dba56886476b42957c", + "sha256:e2618cb2cf5a7cc8d698306e42ebcacd02fb7ef8cfc18485c59394152c70be97", + "sha256:e36750fbbc422c1c46c9d13b937ab437138b998fe74a635ec88989afb57a3978", + "sha256:edfdabe7aa4f97ed2b9dd5dde52d2bb29cb466993bb9d612ddd10d0085a683cf", + "sha256:f22325010d8824594820d6ce84fa830838f581a7fd86a9235f0d2ed6deb61e29", + "sha256:f23876b018dfa5d3e98e96f5644b109090f16a4acb22064e0f06933663005d39", + "sha256:f7bd0ffbcd03dc39490a1f40b2669cc414fae0c4e16b77bb26806a4d0b7d1452" ], "markers": "python_version >= '3.7'", - "version": "==6.4.1" + "version": "==6.4.2" }, "coveralls": { "hashes": [ @@ -1803,10 +1811,10 @@ }, "distlib": { "hashes": [ - "sha256:6564fe0a8f51e734df6333d08b8b94d4ea8ee6b99b5ed50613f731fd4089f34b", - "sha256:e4b58818180336dc9c529bfb9a0b58728ffc09ad92027a3f30b7cd91e3458579" + "sha256:a7f75737c70be3b25e2bee06288cec4e4c221de18455b2dd037fe2a795cab2fe", + "sha256:b710088c59f06338ca514800ad795a132da19fda270e3ce4affc74abf955a26c" ], - "version": "==0.3.4" + "version": "==0.3.5" }, "docopt": { "hashes": [ @@ -1856,11 +1864,11 @@ }, "identify": { "hashes": [ - "sha256:0dca2ea3e4381c435ef9c33ba100a78a9b40c0bab11189c7cf121f75815efeaa", - "sha256:3d11b16f3fe19f52039fb7e39c9c884b21cb1b586988114fbe42671f03de3e82" + "sha256:a3d4c096b384d50d5e6dc5bc8b9bc44f1f61cefebd750a7b3e9f939b53fb214d", + "sha256:feaa9db2dc0ce333b453ce171c0cf1247bbfde2c55fc6bb785022d411a1b78b5" ], "markers": "python_version >= '3.7'", - "version": "==2.5.1" + "version": "==2.5.2" }, "idna": { "hashes": [ @@ -2189,11 +2197,11 @@ }, "setuptools": { "hashes": [ - "sha256:16923d366ced322712c71ccb97164d07472abeecd13f3a6c283f6d5d26722793", - "sha256:db3b8e2f922b2a910a29804776c643ea609badb6a32c4bcc226fd4fd902cce65" + "sha256:0d33c374d41c7863419fc8f6c10bfe25b7b498aa34164d135c622e52580c6b16", + "sha256:c04b44a57a6265fe34a4a444e965884716d34bae963119a76353434d6f18e450" ], "markers": "python_version >= '3.7'", - "version": "==63.1.0" + "version": "==63.2.0" }, "six": { "hashes": [ @@ -2301,7 +2309,7 @@ "sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc", "sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f" ], - "markers": "python_full_version < '3.11.0a7'", + "markers": "python_version >= '3.7'", "version": "==2.0.1" }, "tornado": { @@ -2355,12 +2363,12 @@ }, "zipp": { "hashes": [ - "sha256:56bf8aadb83c24db6c4b577e13de374ccfb67da2078beba1d037c17980bf43ad", - "sha256:c4f6e5bbf48e74f7a38e7cc5b0480ff42b0ae5178957d564d18932525d5cf099" + "sha256:05b45f1ee8f807d0cc928485ca40a07cb491cf092ff587c0df9cb1fd154848d2", + "sha256:47c40d7fe183a6f21403a199b3e4192cca5774656965b0a4988ad2f8feb5f009" ], "index": "pypi", "markers": "python_version < '3.9'", - "version": "==3.8.0" + "version": "==3.8.1" } } } diff --git a/requirements.txt b/requirements.txt index dbab771b5..4532c8210 100644 --- a/requirements.txt +++ b/requirements.txt @@ -13,7 +13,7 @@ blessed==1.19.1; python_version >= '2.7' certifi==2022.6.15; python_version >= '3.6' cffi==1.15.1 channels==3.0.5 -channels-redis==3.4.0 +channels-redis==3.4.1 charset-normalizer==2.1.0; python_version >= '3.6' click==8.1.3; python_version >= '3.7' coloredlogs==15.0.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' @@ -51,12 +51,12 @@ langdetect==1.0.9 lxml==4.9.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' msgpack==1.0.4 numpy==1.23.1; python_version >= '3.8' -ocrmypdf==13.6.0 +ocrmypdf==13.6.1 packaging==21.3; python_version >= '3.6' pathvalidate==2.5.0 pdf2image==1.16.0 pdfminer.six==20220524 -pikepdf==5.3.1 +pikepdf==5.4.0 pillow==9.2.0 pluggy==1.0.0; python_version >= '3.6' portalocker==2.5.1; python_version >= '3' @@ -82,7 +82,7 @@ requests==2.28.1; python_version >= '3.7' and python_version < '4' scikit-learn==1.1.1 scipy==1.8.1; python_version < '3.11' and python_version >= '3.8' service-identity==21.1.0 -setuptools==63.1.0; python_version >= '3.7' +setuptools==63.2.0; python_version >= '3.7' six==1.16.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' sniffio==1.2.0; python_version >= '3.5' sqlparse==0.4.2; python_version >= '3.5' @@ -98,11 +98,11 @@ urllib3==1.26.10; python_version >= '2.7' and python_version not in '3.0, 3.1, 3 uvicorn[standard]==0.18.2 uvloop==0.16.0 watchdog==2.1.9 -watchfiles==0.15.0 +watchfiles==0.16.0 wcwidth==0.2.5 websockets==10.3 whitenoise==6.2.0 whoosh==2.7.4 wrapt==1.14.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' -zipp==3.8.0; python_version < '3.9' +zipp==3.8.1; python_version < '3.9' zope.interface==5.4.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' From ef790ca6f4336095610a3fca2a4ad6507c26455e Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Fri, 22 Jul 2022 11:08:52 -0700 Subject: [PATCH 20/32] Fixes the copy and paste of the log line --- docs/troubleshooting.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst index 1605fed11..6d94d7100 100644 --- a/docs/troubleshooting.rst +++ b/docs/troubleshooting.rst @@ -309,7 +309,7 @@ You might find messages like these in your log files: .. code:: - [WARNING] [paperless.management.consumer] Not consuming file /usr/src/paperless/src/../consume/SCN_0001.pdf: OS reports file as busy still + [ERROR] [paperless.management.consumer] Creating PaperlessTask failed: db locked You are likely using an sqlite based installation, with an increased number of workers and are running into sqlite's concurrency limitations. Uploading or consuming multiple files at once results in many workers attempting to access the database simultaneously. From ecc4553e673440d18f68d88c8579ef4f53f4dc80 Mon Sep 17 00:00:00 2001 From: Michael Shamoon <4887959+shamoon@users.noreply.github.com> Date: Fri, 22 Jul 2022 15:10:33 -0700 Subject: [PATCH 21/32] fix occasional code block color legibility --- docs/_static/css/custom.css | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/_static/css/custom.css b/docs/_static/css/custom.css index 87694a598..c0b3ed83a 100644 --- a/docs/_static/css/custom.css +++ b/docs/_static/css/custom.css @@ -439,7 +439,8 @@ a.image-reference img { } .rst-content code.literal, -.rst-content tt.literal { +.rst-content tt.literal, +html.writer-html5 .rst-content dl.footnote code { border-color: var(--color-border); background-color: var(--color-border); color: var(--color-text-code-inline) From b4d20d9b9a4f1ff3cb90945dbbcf321e6f84c6ea Mon Sep 17 00:00:00 2001 From: Trenton Holmes Date: Sun, 24 Jul 2022 10:22:53 -0700 Subject: [PATCH 22/32] Fixes document import copying PNG files to .webp extensions without actual conversion --- .../management/commands/document_importer.py | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/src/documents/management/commands/document_importer.py b/src/documents/management/commands/document_importer.py index 1a1316059..db85460da 100644 --- a/src/documents/management/commands/document_importer.py +++ b/src/documents/management/commands/document_importer.py @@ -3,6 +3,7 @@ import logging import os import shutil from contextlib import contextmanager +from pathlib import Path import tqdm from django.conf import settings @@ -14,6 +15,7 @@ from django.core.serializers.base import DeserializationError from django.db.models.signals import m2m_changed from django.db.models.signals import post_save from documents.models import Document +from documents.parsers import run_convert from documents.settings import EXPORTER_ARCHIVE_NAME from documents.settings import EXPORTER_FILE_NAME from documents.settings import EXPORTER_THUMBNAIL_NAME @@ -192,7 +194,7 @@ class Command(BaseCommand): document_path = os.path.join(self.source, doc_file) thumb_file = record[EXPORTER_THUMBNAIL_NAME] - thumbnail_path = os.path.join(self.source, thumb_file) + thumbnail_path = Path(os.path.join(self.source, thumb_file)).resolve() if EXPORTER_ARCHIVE_NAME in record: archive_file = record[EXPORTER_ARCHIVE_NAME] @@ -209,7 +211,20 @@ class Command(BaseCommand): create_source_path_directory(document.source_path) shutil.copy2(document_path, document.source_path) - shutil.copy2(thumbnail_path, document.thumbnail_path) + + if thumbnail_path.suffix in {".png", ".PNG"}: + run_convert( + density=300, + scale="500x5000>", + alpha="remove", + strip=True, + trim=False, + auto_orient=True, + input_file=f"{thumbnail_path}[0]", + output_file=str(document.thumbnail_path), + ) + else: + shutil.copy2(thumbnail_path, document.thumbnail_path) if archive_path: create_source_path_directory(document.archive_path) # TODO: this assumes that the export is valid and From 45a6b5a43676d8e62b09c37594e01ad98c432fba Mon Sep 17 00:00:00 2001 From: Michael Shamoon <4887959+shamoon@users.noreply.github.com> Date: Sun, 24 Jul 2022 20:15:26 -0700 Subject: [PATCH 23/32] Add redo OCR button to document edit --- src-ui/messages.xlf | 161 +++++++++++------- .../document-detail.component.html | 6 + .../document-detail.component.ts | 36 ++++ 3 files changed, 142 insertions(+), 61 deletions(-) diff --git a/src-ui/messages.xlf b/src-ui/messages.xlf index a8c43a998..331d9bf5d 100644 --- a/src-ui/messages.xlf +++ b/src-ui/messages.xlf @@ -812,7 +812,7 @@ src/app/components/document-detail/document-detail.component.html - 168 + 174 src/app/components/document-list/save-view-config-dialog/save-view-config-dialog.component.html @@ -1138,7 +1138,7 @@ src/app/components/document-detail/document-detail.component.html - 69 + 75 src/app/components/document-list/document-list.component.html @@ -1391,11 +1391,22 @@ 25 + + Redo OCR + + src/app/components/document-detail/document-detail.component.html + 34 + + + src/app/components/document-list/bulk-editor/bulk-editor.component.html + 90 + + More like this src/app/components/document-detail/document-detail.component.html - 34 + 40 src/app/components/document-list/document-card-large/document-card-large.component.html @@ -1406,49 +1417,49 @@ Close src/app/components/document-detail/document-detail.component.html - 37 + 43 Previous src/app/components/document-detail/document-detail.component.html - 44 + 50 Next src/app/components/document-detail/document-detail.component.html - 49 + 55 Details src/app/components/document-detail/document-detail.component.html - 66 + 72 Archive serial number src/app/components/document-detail/document-detail.component.html - 70 + 76 Date created src/app/components/document-detail/document-detail.component.html - 71 + 77 Correspondent src/app/components/document-detail/document-detail.component.html - 72 + 78 src/app/components/document-list/bulk-editor/bulk-editor.component.html @@ -1471,7 +1482,7 @@ Document type src/app/components/document-detail/document-detail.component.html - 74 + 80 src/app/components/document-list/bulk-editor/bulk-editor.component.html @@ -1494,7 +1505,7 @@ Storage path src/app/components/document-detail/document-detail.component.html - 76 + 82 src/app/components/document-list/bulk-editor/bulk-editor.component.html @@ -1513,21 +1524,21 @@ Default src/app/components/document-detail/document-detail.component.html - 77 + 83 Content src/app/components/document-detail/document-detail.component.html - 84 + 90 Metadata src/app/components/document-detail/document-detail.component.html - 93 + 99 src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts @@ -1538,95 +1549,95 @@ Date modified src/app/components/document-detail/document-detail.component.html - 99 + 105 Date added src/app/components/document-detail/document-detail.component.html - 103 + 109 Media filename src/app/components/document-detail/document-detail.component.html - 107 + 113 Original MD5 checksum src/app/components/document-detail/document-detail.component.html - 111 + 117 Original file size src/app/components/document-detail/document-detail.component.html - 115 + 121 Original mime type src/app/components/document-detail/document-detail.component.html - 119 + 125 Archive MD5 checksum src/app/components/document-detail/document-detail.component.html - 123 + 129 Archive file size src/app/components/document-detail/document-detail.component.html - 127 + 133 Original document metadata src/app/components/document-detail/document-detail.component.html - 133 + 139 Archived document metadata src/app/components/document-detail/document-detail.component.html - 134 + 140 Enter Password src/app/components/document-detail/document-detail.component.html - 156 + 162 src/app/components/document-detail/document-detail.component.html - 186 + 192 Discard src/app/components/document-detail/document-detail.component.html - 166 + 172 Save & next src/app/components/document-detail/document-detail.component.html - 167 + 173 @@ -1668,6 +1679,66 @@ 459 + + Redo OCR confirm + + src/app/components/document-detail/document-detail.component.ts + 479 + + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 387 + + + + This operation will permanently redo OCR for this document. + + src/app/components/document-detail/document-detail.component.ts + 480 + + + + This operation cannot be undone. + + src/app/components/document-detail/document-detail.component.ts + 481 + + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 364 + + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 389 + + + + Proceed + + src/app/components/document-detail/document-detail.component.ts + 483 + + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 391 + + + + Redo OCR operation will begin in the background. + + src/app/components/document-detail/document-detail.component.ts + 491 + + + + Error executing operation: + + src/app/components/document-detail/document-detail.component.ts + 502,504 + + Select: @@ -1771,13 +1842,6 @@ 84,88 - - Redo OCR - - src/app/components/document-list/bulk-editor/bulk-editor.component.html - 90 - - Error executing bulk operation: 363 - - This operation cannot be undone. - - src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 364 - - - src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 389 - - Delete document(s) @@ -1959,13 +2012,6 @@ 366 - - Redo OCR confirm - - src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 387 - - This operation will permanently redo OCR for selected document(s). @@ -1973,13 +2019,6 @@ 388 - - Proceed - - src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 391 - - Filter by correspondent diff --git a/src-ui/src/app/components/document-detail/document-detail.component.html b/src-ui/src/app/components/document-detail/document-detail.component.html index 7948d82e3..2c4a5c85f 100644 --- a/src-ui/src/app/components/document-detail/document-detail.component.html +++ b/src-ui/src/app/components/document-detail/document-detail.component.html @@ -28,6 +28,12 @@ + +
@@ -16,7 +16,7 @@ -  Download + Download
@@ -31,13 +31,13 @@