From 48b042bfb0af66ef495c9af095d5cdb8697e04db Mon Sep 17 00:00:00 2001 From: Jonas Winkler Date: Mon, 1 Feb 2021 20:14:29 +0100 Subject: [PATCH 001/225] troubleshooting for #216 --- docs/troubleshooting.rst | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst index f55d57af5..ae7fe14c7 100644 --- a/docs/troubleshooting.rst +++ b/docs/troubleshooting.rst @@ -60,7 +60,8 @@ required so that the user running paperless inside docker has write permissions to these folders. This happens when pointing these directories to NFS shares, for example. -Ensure that `chown` is possible on these directories. +Ensure that ``chown`` is possible on these directories. + Classifier error: No training data available ############################################ @@ -73,6 +74,7 @@ This may have two reasons: with Inbox tags. Verify that there are documents in your archive without inbox tags. The algorithm will only learn from documents not in your inbox. + UserWarning in sklearn on every single document ############################################### @@ -91,6 +93,7 @@ in most cases. This warning will disappear automatically when paperless updates If you want to get rid of the warning or actually experience issues with automatic matching, delete the file ``classification_model.pickle`` in the data directory and let paperless recreate it. + Permission denied errors in the consumption directory ##################################################### @@ -106,6 +109,38 @@ different from ``1000``. See :ref:`setup-docker_hub`. Also ensure that you are able to read and write to the consumption directory on the host. + +OSError: [Errno 19] No such device when consuming files +####################################################### + +If you experience errors such as: + +.. code:: shell-session + + File "/usr/local/lib/python3.7/site-packages/whoosh/codec/base.py", line 570, in open_compound_file + return CompoundStorage(dbfile, use_mmap=storage.supports_mmap) + File "/usr/local/lib/python3.7/site-packages/whoosh/filedb/compound.py", line 75, in __init__ + self._source = mmap.mmap(fileno, 0, access=mmap.ACCESS_READ) + OSError: [Errno 19] No such device + + During handling of the above exception, another exception occurred: + + Traceback (most recent call last): + File "/usr/local/lib/python3.7/site-packages/django_q/cluster.py", line 436, in worker + res = f(*task["args"], **task["kwargs"]) + File "/usr/src/paperless/src/documents/tasks.py", line 73, in consume_file + override_tag_ids=override_tag_ids) + File "/usr/src/paperless/src/documents/consumer.py", line 271, in try_consume_file + raise ConsumerError(e) + +Paperless uses a search index to provide better and faster full text searching. This search index is stored inside +the ``data`` folder. The search index uses memory-mapped files (mmap). The above error indicates that paperless +was unable to create and open these files. + +This happens when you're trying to store the data directory on certain file systems (mostly network shares) +that don't support memory-mapped files. + + Web-UI stuck at "Loading..." ############################ @@ -148,4 +183,4 @@ This might have multiple reasons. SENDFILE=0 - to your `docker-compose.env` file. \ No newline at end of file + to your `docker-compose.env` file. From 0a28ccf62fac76d1fef0e18434d81ce1c3597ca2 Mon Sep 17 00:00:00 2001 From: Jonas Winkler Date: Mon, 1 Feb 2021 21:43:50 +0100 Subject: [PATCH 002/225] adjustments to setup --- docs/setup.rst | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/setup.rst b/docs/setup.rst index afb3784d6..a86c4ce27 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -151,13 +151,16 @@ Install Paperless from Docker Hub .. code:: - /home/jonaswinkler/paperless-inbox:/usr/src/paperless/consume - + Don't change the part after the colon or paperless wont find your documents. 5. Modify ``docker-compose.env``, following the comments in the file. The most important change is to set ``USERMAP_UID`` and ``USERMAP_GID`` - to the uid and gid of your user on the host system. This ensures that + to the uid and gid of your user on the host system. Use ``id -u`` and + ``id -g`` to get these. + + This ensures that both the docker container and you on the host machine have write access to the consumption directory. If your UID and GID on the host system is 1000 (the default for the first normal user on most systems), it will @@ -175,15 +178,11 @@ Install Paperless from Docker Hub on such a file system, paperless will not pick up new files with the default configuration. You will need to use ``PAPERLESS_CONSUMER_POLLING``, which will disable inotify. See :ref:`here `. - -6. Now head over to: https://hub.docker.com/r/jonaswinkler/paperless-ng and choose your preferred - image and copy the link. To download this image do a `docker pull` followed by the link. Do this within the directory with the .yml files. - Depending on your network connection and CPU this will take a while. You have time to get a beverage. -7. Run ``docker-compose up -d``. This will create and start the necessary - containers, but your are not done yet! +6. Run ``docker-compose up -d``. This will pull the image, create and start the necessary + containers. -8. To be able to login, you will need a super user. To create it, execute the +7. To be able to login, you will need a super user. To create it, execute the following command: .. code-block:: shell-session @@ -193,8 +192,8 @@ Install Paperless from Docker Hub This will prompt you to set a username, an optional e-mail address and finally a password (at least 8 characters). -9. The default ``docker-compose.yml`` exports the webserver on your local port - 8000. If you haven't adapted this, you should now be able to visit your +8. The default ``docker-compose.yml`` exports the webserver on your local port + 8000. If you did not change this, you should now be able to visit your Paperless instance at ``http://127.0.0.1:8000`` or your servers IP-Address:8000. Use the login credentials you have created with the previous step. From 2efb99192a67fb28269ec513ef1f2f3acddb511c Mon Sep 17 00:00:00 2001 From: Jonas Winkler Date: Mon, 1 Feb 2021 21:45:45 +0100 Subject: [PATCH 003/225] add pull to docker setup --- docs/setup.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/setup.rst b/docs/setup.rst index a86c4ce27..5c3c9c361 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -179,8 +179,8 @@ Install Paperless from Docker Hub with the default configuration. You will need to use ``PAPERLESS_CONSUMER_POLLING``, which will disable inotify. See :ref:`here `. -6. Run ``docker-compose up -d``. This will pull the image, create and start the necessary - containers. +6. Run ``docker-compose pull``, followed by ``docker-compose up -d``. + This will pull the image, create and start the necessary containers. 7. To be able to login, you will need a super user. To create it, execute the following command: From c78d54e71e96386a3b6a28545925a60521ae27e3 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 2 Feb 2021 19:54:56 +0100 Subject: [PATCH 004/225] update the easy install script and documentation --- docs/setup.rst | 19 ++++++++ install-paperless-ng.sh | 96 +++++++++++++++++++++++++++++++++++------ 2 files changed, 103 insertions(+), 12 deletions(-) diff --git a/docs/setup.rst b/docs/setup.rst index eda03550e..5e0e3e4a6 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -81,6 +81,7 @@ Installation You can go multiple routes to setup and run Paperless: +* :ref:`Use the easy install docker script ` * :ref:`Pull the image from Docker Hub ` * :ref:`Build the Docker image yourself ` * :ref:`Install Paperless directly on your system manually (bare metal) ` @@ -101,6 +102,24 @@ it includes the same sensible defaults, and it simultaneously provides the flexi .. _CLI Basics: https://sehn.tech/post/devops-with-docker/ .. _idempotent: https://docs.ansible.com/ansible/latest/reference_appendices/glossary.html#Idempotency +.. _setup-docker_script: + +Install Paperless from Docker Hub using the installation script +=============================================================== + +Paperless provides an interactive installation script. This script will ask you +for a couple configuration options, download and create the necessary configuration files, pull the docker image, start paperless and create your user account. This script essentially +performs all the steps described in :ref:`setup-docker_hub` automatically. + +1. Make sure that docker and docker-compose are installed. +2. Download and run the installation script: + + .. code:: shell-session + + $ wget https://raw.githubusercontent.com/jonaswinkler/paperless-ng/master/install-paperless-ng.sh + $ chmod +x install-paperless-ng.sh + $ ./install-paperless-ng.sh + .. _setup-docker_hub: Install Paperless from Docker Hub diff --git a/install-paperless-ng.sh b/install-paperless-ng.sh index ad992fe8c..8fb44e222 100755 --- a/install-paperless-ng.sh +++ b/install-paperless-ng.sh @@ -15,7 +15,7 @@ ask() { fi array=$3 if [[ -z $3 || " ${array[@]} " =~ " ${result} " ]]; then - ask_result=$result + ask_result=$result return else echo "Invalid option: $result" @@ -23,6 +23,47 @@ ask() { done } +ask_docker_folder() { + while true ; do + + read -p "$1 [$2]: " result + + if [[ -z $result ]]; then + ask_result=$2 + return + fi + + if [[ $result == /* || $result == ./* ]]; then + ask_result=$result + return + else + echo "Invalid folder: $result" + fi + + + done +} + +if [[ $(id -u) == "0" ]] ; then + echo "Do not run this script as root." + exit 1 +fi + +if [[ -z $(which wget) ]] ; then + echo "wget executable not found. Is wget installed?" + exit 1 +fi + +if [[ -z $(which docker) ]] ; then + echo "docker executable not found. Is docker installed?" + exit 1 +fi + +if [[ -z $(which docker-compose) ]] ; then + echo "docker-compose executable not found. Is docker-compose installed?" + exit 1 +fi + echo "" echo "############################################" echo "### Paperless-ng docker installation ###" @@ -48,14 +89,13 @@ echo "The consume folder is where paperles will search for new documents." echo "Point this to a folder where your scanner is able to put your scanned" echo "documents." echo "" -echo "HINT: If paperless is unable to pick up any files from this directory after" -echo "installation, you might need to configure PAPERLESS_CONSUMER_POLLING." -echo "See the documentation for details." -echo "" -echo "CAUTION: You must specify an absolute path starting with /" +echo "CAUTION: You must specify an absolute path starting with / or a relative " +echo "path starting with ./ here. Examples:" +echo " /mnt/consume" +echo " ./consume" echo "" -ask "Consume folder" "$TARGET_FOLDER/consume" +ask_docker_folder "Consume folder" "$TARGET_FOLDER/consume" CONSUME_FOLDER=$ask_result echo "" @@ -64,9 +104,10 @@ echo "Leave empty and docker will manage this folder for you." echo "Docker usually stores managed folders in /var/lib/docker/volumes." echo "" echo "CAUTION: If specified, you must specify an absolute path starting with /" +echo "or a relative path starting with ./ here." echo "" -ask "Media folder" "" +ask_docker_folder "Media folder" "" MEDIA_FOLDER=$ask_result echo "" @@ -74,8 +115,11 @@ echo "The data folder is where paperless stores other data, such as your" echo "SQLite database (if used), the search index and other data." echo "As with the media folder, leave empty to have this managed by docker." echo "" +echo "CAUTION: If specified, you must specify an absolute path starting with /" +echo "or a relative path starting with ./ here." +echo "" -ask "Data folder" "" +ask_docker_folder "Data folder" "" DATA_FOLDER=$ask_result echo "" @@ -166,8 +210,36 @@ done ask "Email" "$USERNAME@localhost" EMAIL=$ask_result -echo "Done collecting data. Press any key to install." -read +echo "" +echo "Summary" +echo "=======" +echo "" + +echo "Target folder: $TARGET_FOLDER" +echo "Consume folder: $CONSUME_FOLDER" +if [[ -z $MEDIA_FOLDER ]] ; then + echo "Media folder: Managed by docker" +else + echo "Media folder: $MEDIA_FOLDER" +fi +if [[ -z $DATA_FOLDER ]] ; then + echo "Data folder: Managed by docker" +else + echo "Data folder: $DATA_FOLDER" +fi +echo "" +echo "Port: $PORT" +echo "Database: $DATABASE_BACKEND" +echo "Tika enabled: $TIKA_ENABLED" +echo "OCR language: $OCR_LANGUAGE" +echo "User id: $USERMAP_UID" +echo "Group id: $USERMAP_GID" +echo "" +echo "Paperless username: $USERNAME" +echo "Paperless email: $EMAIL" + +echo "" +read -p "Press any key to install." echo "" echo "Installing paperless..." @@ -209,7 +281,7 @@ sed -i "s/- 8000:8000/- $PORT:8000/g" docker-compose.yml sed -i "s#- \./consume:/usr/src/paperless/consume#- $CONSUME_FOLDER:/usr/src/paperless/consume#g" docker-compose.yml if [[ -n $MEDIA_FOLDER ]] ; then - sed -i "s#- data:/usr/src/paperless/media#- $MEDIA_FOLDER:/usr/src/paperless/media#g" docker-compose.yml + sed -i "s#- media:/usr/src/paperless/media#- $MEDIA_FOLDER:/usr/src/paperless/media#g" docker-compose.yml fi if [[ -n $DATA_FOLDER ]] ; then From 82730ab491cb8d7ea093c5aa3564829da5d86e32 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 2 Feb 2021 20:31:25 +0100 Subject: [PATCH 005/225] update dependencies --- Pipfile | 4 +- Pipfile.lock | 121 +++++++++++++++++++++++++++-------------------- requirements.txt | 10 ++-- 3 files changed, 78 insertions(+), 57 deletions(-) diff --git a/Pipfile b/Pipfile index cbe5c5103..e5e13ccac 100644 --- a/Pipfile +++ b/Pipfile @@ -25,7 +25,7 @@ pdftotext = "*" pathvalidate = "*" # pinned to 8.1.0, since aarch64 wheels might not be available beyond that https://github.com/python-pillow/Pillow/issues/5202 pillow = "==8.1.0" -pikepdf = "~=2.2.5" +pikepdf = "~=2.5.0" python-gnupg = "*" python-dotenv = "*" python-dateutil = "*" @@ -41,7 +41,7 @@ whitenoise = "~=5.2.0" watchdog = "*" whoosh="~=2.7.4" inotifyrecursive = "~=0.3.4" -ocrmypdf = "~=11.4.5" +ocrmypdf = "~=11.6" tqdm = "*" tika = "*" # TODO: This will sadly also install daphne+dependencies, diff --git a/Pipfile.lock b/Pipfile.lock index ff9e96717..6d200ce4b 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "d80d2539a4528a8fd9e848875c2e2d5bcdb3e98154f45b612706094b84ecaaea" + "sha256": "3e5b28740397136ffd0a397889e232192414eca2e2e6e9284f452d60cbbceda8" }, "pipfile-spec": 6, "requires": {}, @@ -107,6 +107,7 @@ "sha256:6bc25fc545a6b3d57b5f8618e59fc13d3a3a68431e8ca5fd4c13241cd70d0009", "sha256:798caa2a2384b1cbe8a2a139d80734c9db54f9cc155c99d7cc92441a23871c03", "sha256:7c6b1dece89874d9541fc974917b631406233ea0440d0bdfbb8e03bf39a49b3b", + "sha256:7ef7d4ced6b325e92eb4d3502946c78c5367bc416398d387b39591532536734e", "sha256:840793c68105fe031f34d6a086eaea153a0cd5c491cde82a74b420edd0a2b909", "sha256:8d6603078baf4e11edc4168a514c5ce5b3ba6e3e9c374298cb88437957960a53", "sha256:9cc46bc107224ff5b6d04369e7c595acb700c3613ad7bcf2e2012f62ece80c35", @@ -218,11 +219,11 @@ }, "django": { "hashes": [ - "sha256:2d78425ba74c7a1a74b196058b261b9733a8570782f4e2828974777ccca7edf7", - "sha256:efa2ab96b33b20c2182db93147a0c3cd7769d418926f9e9f140a60dca7c64ca9" + "sha256:169e2e7b4839a7910b393eec127fd7cbae62e80fa55f89c6510426abf673fe5f", + "sha256:c6c0462b8b361f8691171af1fb87eceb4442da28477e12200c40420176206ba7" ], "index": "pypi", - "version": "==3.1.5" + "version": "==3.1.6" }, "django-cors-headers": { "hashes": [ @@ -587,11 +588,11 @@ }, "ocrmypdf": { "hashes": [ - "sha256:416a9c4321bfc844f250694b8c68ebb538f60609bbc8686bd9f84a13c5127d68", - "sha256:f45fc7e844e6026d6080a623a2936be120fc077d99aaa599df022acf35fb31e6" + "sha256:a54634d017a2f44aa2115b0b6ae5aa41a7cec018f5c53d16ad3abec1e70b3db7", + "sha256:d0e2da48d4abd90f48f0937b2cd4ba57503b56c603f5e3aa91e20e3b21a036cd" ], "index": "pypi", - "version": "==11.4.5" + "version": "==11.6.0" }, "pathvalidate": { "hashes": [ @@ -618,30 +619,32 @@ }, "pikepdf": { "hashes": [ - "sha256:0e67e5beeeed5422b3b8e862e4777fed5a4cd3c72e711e2a449a65d9ee641448", - "sha256:138155ae1f71634cd6eca79f5517f77b2067ef0bd5b627ea9414e308fe868dc5", - "sha256:15cf648dd760a47c55a4106b601b92bb653ae98155b10f04310553629c6695dd", - "sha256:1d6a011ae4c501c78509caf19cbe152c2e3cb5c267f7b47bc3db8cd3436585a7", - "sha256:211f529313953e44ae42eb896c2b688668385e6e8f9d04d21484bddb3c42b34c", - "sha256:22049ad288d603a7fc68e90a0722770d307886788373ddfe71fbf614ced0f5b2", - "sha256:24f7c371f6ecbee8f0ae30030992fc75cd32cd575dcfca8d466a03a8290377ca", - "sha256:26cdf561632866d584fedb6b1c1fce78cefa49b5cae54c65aa6a6ca5fe6de4ac", - "sha256:2c37afcd21a2eb1da1773687e853327fa8ec7d2c5cd90cdcd70180f55f0221e1", - "sha256:65b8ec6403814f51e1b9c7e18a8ff26087fcc7a199b1405583e5ff9eb931db56", - "sha256:66a03103aadb2e2738271cb18c89837ac3980fa0b4687195c4c150228b7e79de", - "sha256:6e8f0124354c53a66f83ec5a18111b760aeff1a64db3a86e7ee5fed8e8624707", - "sha256:70f2836cd468aa25bc8b09a2b9561364bd75d3e6ddb0e50a25d248d7da6cff25", - "sha256:82cebf68952cfb65c86d880eb782a0c558b37531cdae59f2e11fcd0f2bb4669c", - "sha256:84ad3e8fd5f3251fb5b534614da64b04a264ce9348f0fe35b781c0fb378b0f82", - "sha256:af13fbc022efa85d1ae161129d4cde66493479db52b9adb74d525b890a078208", - "sha256:c1d40fb8f8192c75f54f0e74a569ccf45e4e13bed8da78a78a5b488be29979bf", - "sha256:d147ec1ab58512871fdf40a161809f698eaa75720b4a230198e7e028582b20a1", - "sha256:dedad1f68d6b0b54000f7f99386351f1c6e19c8cf70a9700d8dd06b9809c54fb", - "sha256:e72c3f5b624b9c7341fd6a7e657926d4cf12a7ea453681ffd7332cabc3530c62", - "sha256:eb75f22e261b3bc69b6fc9a17b1d6966c95e79d3e792b7737a018a2bf6a2b07f" + "sha256:02815df9499d3a6dfac2e07e4d2fdbe25fcbefae208970e76bff90af4a402d49", + "sha256:06b0c3004cc9e9068ebc62bb59c3c3a54e7af13867f4a326690d79c69a1cf288", + "sha256:1a471c6ca288fbcd0e1b0e128ef12bb14c5e7db745786308ba292fc7cff30bb5", + "sha256:489ed0fd1281beb0343a34fe8b9d94407c440ed0419ab2e6f5ea297a41824a31", + "sha256:5106b27f7085ed449e057b9988f07c80a87292d2bf46c585a8635ac7a3ccf0d5", + "sha256:51acffba6f3d21674eea7a0432ce1adaf0743641d57844a5e3dc92b4a7e81c85", + "sha256:53d694d70dd072a47bd2dd71329dcef0f809dcd8084d1d11c31baf3b64cd345c", + "sha256:6a640fef52dc785abd354d6800a52ecc02656c98dbfc2ecde559323b001bd43f", + "sha256:7006ef95f847412605dea6e772019f637949eaeaf65363d5d6afd9aa96bf5623", + "sha256:81e13b62877dbc089095e7efa03c27834bdf6b49d404d064cb227b0e179ce049", + "sha256:8fc3e97b24dafbd4b1ac057c8f144c18467ce55d342a1a8c42688890ead58ef3", + "sha256:9158dc4d3ef4e2301fb1879d5825530fdb32143ced770d60fa8e5badeee70a35", + "sha256:961337a10b42bd656b59116ce1c574eafd515b45a513221df6ae1f11734bfb6c", + "sha256:994ccac972357a7b9b147217e1beea2f7688697944b862dbb2a3e64aa9a5ff14", + "sha256:a8e9abf7db0351357b55c3f935979e7dc14f3f259a25d15bcc86abce730955a7", + "sha256:b836eda7f70b9dd75ccdeaf4e78b38393118a66821a69a10054b1430f945d1fd", + "sha256:b859a225f6bd953472c50f4df612b4575df646e560189e3720310ad65b6805a0", + "sha256:cae106bd461cfad73c554c09f6db1d5f2c6a28f5b8cb0602b63046840f488226", + "sha256:cdc75b8fa5a650f4fc91214a315358fde7470e09b95a00981b73a7c4ac5ddb97", + "sha256:e2c28da4f37ad9a3efbedbbfc6f1084941bde43903d30dfdbb338d5ba94c9f50", + "sha256:e596bd8fdbf40bfb8dc8068cdbc7e5b72052188d1ad8ca84da9d6b77658a8b31", + "sha256:ee957b9c60b6def20cbdf656d35859ce211eec02dafa3abb9d5ca937d32a3c3b", + "sha256:f9428d4b1f70af4f4560be4dccbbc5ab5308c00c5b62ed2f1c44ce9e2591b3d2" ], "index": "pypi", - "version": "==2.2.5" + "version": "==2.5.2" }, "pillow": { "hashes": [ @@ -819,12 +822,12 @@ }, "python-levenshtein": { "hashes": [ - "sha256:108edd3c271f1afda8b21a8d9da81886414dfb6940a085fa7903c592e0f8f54b", - "sha256:1ff19b712c5974080b003fd26ef365cd93dfc1a5e690be621f79f3e63e00a7cc", - "sha256:554e273a88060d177e7b3c1e6ea9158dde11563bfae8f7f661f73f47e5ff0911" + "sha256:212db61934fcb819f5cb2fcb5ad5c0e2e43f3161f524eef4a135f7285dfb308a", + "sha256:d92fe5c3b10c8ad8f2d880499f5e96ed24dbc7cd0414a665c2e2505feaf4ec58", + "sha256:dc2395fbd148a1ab31090dd113c366695934b9e85fe5a4b2a032745efd0346f6" ], "index": "pypi", - "version": "==0.12.1" + "version": "==0.12.2" }, "python-magic": { "hashes": [ @@ -836,10 +839,10 @@ }, "pytz": { "hashes": [ - "sha256:16962c5fb8db4a8f63a26646d8886e9d769b6c511543557bc84e9569fb9a9cb4", - "sha256:180befebb1927b16f6b57101720075a984c019ac16b1b7575673bea42c6c3da5" + "sha256:83a4a90894bf38e243cf052c8b58f381bfe9a7a483f6a9cab140bc7f702ac4da", + "sha256:eb10ce3e7736052ed3623d49975ce333bcd712c7bb19a58b9e2089d4057d0798" ], - "version": "==2020.5" + "version": "==2021.1" }, "pyyaml": { "hashes": [ @@ -1558,12 +1561,11 @@ }, "jinja2": { "hashes": [ - "sha256:3f172970d5670703bd3812e8ca6459a9a7e069fa8e51b40195f83c81db191ec4", - "sha256:89aab215427ef59c34ad58735269eb58b1a5808103067f7bb9d5836c651b3bb0", - "sha256:f0a4641d3cf955324a89c04f3d94663aa4d638abe8f733ecd3582848e1c37035" + "sha256:03e47ad063331dd6a3f04a43eddca8a966a26ba0c5b7207a9a9e4e08f1b29419", + "sha256:a6d58433de0ae800347cab1fa3043cebbabe8baa9d29e668f1c768cb87a333c6" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'", - "version": "==2.11.2" + "version": "==2.11.3" }, "markupsafe": { "hashes": [ @@ -1573,8 +1575,12 @@ "sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5", "sha256:13d3144e1e340870b25e7b10b98d779608c02016d5184cfb9927a9f10c689f42", "sha256:19536834abffb3fa155017053c607cb835b2ecc6a3a2554a88043d991dffb736", + "sha256:195d7d2c4fbb0ee8139a6cf67194f3973a6b3042d742ebe0a9ed36d8b6f0c07f", + "sha256:22c178a091fc6630d0d045bdb5992d2dfe14e3259760e713c490da5323866c39", "sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff", "sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b", + "sha256:2beec1e0de6924ea551859edb9e7679da6e4870d32cb766240ce17e0a0ba2014", + "sha256:3b8a6499709d29c2e2399569d96719a1b21dcd94410a586a18526b143ec8470f", "sha256:3d61f15e39611aacd91b7e71d903787da86d9e80896e683c0103fced9add7834", "sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1", "sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e", @@ -1584,36 +1590,51 @@ "sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1", "sha256:6788b695d50a51edb699cb55e35487e430fa21f1ed838122d722e0ff0ac5ba15", "sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1", + "sha256:6f1e273a344928347c1290119b493a1f0303c52f5a5eae5f16d74f48c15d4a85", + "sha256:6fffc775d90dcc9aed1b89219549b329a9250d918fd0b8fa8d93d154918422e1", "sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e", "sha256:7952deddf24b85c88dab48f6ec366ac6e39d2761b5280f2f9594911e03fcd064", "sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b", "sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905", + "sha256:7fed13866cf14bba33e7176717346713881f56d9d2bcebab207f7a036f41b850", + "sha256:84dee80c15f1b560d55bcfe6d47b27d070b4681c699c572af2e3c7cc90a3b8e0", "sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735", "sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d", + "sha256:98bae9582248d6cf62321dcb52aaf5d9adf0bad3b40582925ef7c7f0ed85fceb", "sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e", "sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d", "sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c", + "sha256:a6a744282b7718a2a62d2ed9d993cad6f5f585605ad352c11de459f4108df0a1", + "sha256:acf08ac40292838b3cbbb06cfe9b2cb9ec78fce8baca31ddb87aaac2e2dc3bc2", "sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21", "sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2", "sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5", + "sha256:b1dba4527182c95a0db8b6060cc98ac49b9e2f5e64320e2b56e47cb2831978c7", "sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b", + "sha256:b7d644ddb4dbd407d31ffb699f1d140bc35478da613b441c582aeb7c43838dd8", "sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6", + "sha256:bf5aa3cbcfdf57fa2ee9cd1822c862ef23037f5c832ad09cfea57fa846dec193", "sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f", + "sha256:caabedc8323f1e93231b52fc32bdcde6db817623d33e100708d9a68e1f53b26b", "sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f", "sha256:cdb132fc825c38e1aeec2c8aa9338310d29d337bebbd7baa06889d09a60a1fa2", + "sha256:d53bc011414228441014aa71dbec320c66468c1030aae3a6e29778a3382d96e5", + "sha256:d73a845f227b0bfe8a7455ee623525ee656a9e2e749e4742706d80a6065d5e2c", + "sha256:d9be0ba6c527163cbed5e0857c451fcd092ce83947944d6c14bc95441203f032", "sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7", - "sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be" + "sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be", + "sha256:feb7b34d6325451ef96bc0e36e1a6c0c1c64bc1fbec4b854f4529e51887b1621" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", "version": "==1.1.1" }, "packaging": { "hashes": [ - "sha256:24e0da08660a87484d1602c30bb4902d74816b6985b93de36926f5bc95741858", - "sha256:78598185a7008a470d64526a8059de9aaa449238f280fc9eb6b13ba6c4109093" + "sha256:5b327ac1320dc863dca72f4514ecc086f31186744b84a230374cc1fd776feae5", + "sha256:67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", - "version": "==20.8" + "version": "==20.9" }, "pluggy": { "hashes": [ @@ -1722,10 +1743,10 @@ }, "pytz": { "hashes": [ - "sha256:16962c5fb8db4a8f63a26646d8886e9d769b6c511543557bc84e9569fb9a9cb4", - "sha256:180befebb1927b16f6b57101720075a984c019ac16b1b7575673bea42c6c3da5" + "sha256:83a4a90894bf38e243cf052c8b58f381bfe9a7a483f6a9cab140bc7f702ac4da", + "sha256:eb10ce3e7736052ed3623d49975ce333bcd712c7bb19a58b9e2089d4057d0798" ], - "version": "==2020.5" + "version": "==2021.1" }, "requests": { "hashes": [ @@ -1863,11 +1884,11 @@ }, "virtualenv": { "hashes": [ - "sha256:219ee956e38b08e32d5639289aaa5bd190cfbe7dafcb8fa65407fca08e808f9c", - "sha256:227a8fed626f2f20a6cdb0870054989f82dd27b2560a911935ba905a2a5e0034" + "sha256:147b43894e51dd6bba882cf9c282447f780e2251cd35172403745fc381a0a80d", + "sha256:2be72df684b74df0ea47679a7df93fd0e04e72520022c57b479d8f881485dbe3" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", - "version": "==20.4.0" + "version": "==20.4.2" }, "zipp": { "hashes": [ diff --git a/requirements.txt b/requirements.txt index c3f32002b..536e84e3b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -31,7 +31,7 @@ django-extensions==3.1.0 django-filter==2.4.0 django-picklefield==3.0.1; python_version >= '3' django-q==1.3.4 -django==3.1.5 +django==3.1.6 djangorestframework==3.12.2 filelock==3.0.12 fuzzywuzzy==0.18.0 @@ -53,11 +53,11 @@ langdetect==1.0.8 lxml==4.6.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' msgpack==1.0.2 numpy==1.19.5; python_version >= '3.6' -ocrmypdf==11.4.5 +ocrmypdf==11.6.0 pathvalidate==2.3.2 pdfminer.six==20201018; python_version >= '3.4' pdftotext==2.1.5 -pikepdf==2.2.5 +pikepdf==2.5.2 pillow==8.1.0 pluggy==0.13.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' psycopg2-binary==2.8.6 @@ -69,9 +69,9 @@ pyopenssl==20.0.1; python_version >= '2.7' and python_version not in '3.0, 3.1, python-dateutil==2.8.1 python-dotenv==0.15.0 python-gnupg==0.4.6 -python-levenshtein==0.12.1 +python-levenshtein==0.12.2 python-magic==0.4.18 -pytz==2020.5 +pytz==2021.1 pyyaml==5.4.1 redis==3.5.3 regex==2020.11.13 From d198a89f22c5a72f93a8a7fe07331f33d903eda0 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 2 Feb 2021 20:43:07 +0100 Subject: [PATCH 006/225] update dependencies --- Pipfile | 3 +-- Pipfile.lock | 6 ++++-- requirements.txt | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/Pipfile b/Pipfile index e5e13ccac..293d71603 100644 --- a/Pipfile +++ b/Pipfile @@ -17,7 +17,7 @@ django-filter = "~=2.4.0" django-q = "~=1.3.4" djangorestframework = "~=3.12.2" filelock = "*" -fuzzywuzzy = "*" +fuzzywuzzy = {extras = ["speedup"], version = "*"} gunicorn = "*" imap-tools = "*" langdetect = "*" @@ -29,7 +29,6 @@ pikepdf = "~=2.5.0" python-gnupg = "*" python-dotenv = "*" python-dateutil = "*" -python-Levenshtein = "*" python-magic = "*" psycopg2-binary = "*" redis = "*" diff --git a/Pipfile.lock b/Pipfile.lock index 6d200ce4b..78cfecf95 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "3e5b28740397136ffd0a397889e232192414eca2e2e6e9284f452d60cbbceda8" + "sha256": "658e2907c680baea81e77da9081ee063ff843b185d02c95d6483b747f0acd84d" }, "pipfile-spec": 6, "requires": {}, @@ -282,6 +282,9 @@ "version": "==3.0.12" }, "fuzzywuzzy": { + "extras": [ + "speedup" + ], "hashes": [ "sha256:45016e92264780e58972dca1b3d939ac864b78437422beecebb3095f8efd00e8", "sha256:928244b28db720d1e0ee7587acf660ea49d7e4c632569cad4f1cd7e68a5f0993" @@ -826,7 +829,6 @@ "sha256:d92fe5c3b10c8ad8f2d880499f5e96ed24dbc7cd0414a665c2e2505feaf4ec58", "sha256:dc2395fbd148a1ab31090dd113c366695934b9e85fe5a4b2a032745efd0346f6" ], - "index": "pypi", "version": "==0.12.2" }, "python-magic": { diff --git a/requirements.txt b/requirements.txt index 536e84e3b..e1b76a4a0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -34,7 +34,7 @@ django-q==1.3.4 django==3.1.6 djangorestframework==3.12.2 filelock==3.0.12 -fuzzywuzzy==0.18.0 +fuzzywuzzy[speedup]==0.18.0 gunicorn==20.0.4 h11==0.12.0; python_version >= '3.6' hiredis==1.1.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' From 85f46424a812707f63360432107b69846c2384d3 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 2 Feb 2021 23:58:11 +0100 Subject: [PATCH 007/225] fixes layout issues #482 --- .../src/app/components/app-frame/app-frame.component.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src-ui/src/app/components/app-frame/app-frame.component.html b/src-ui/src/app/components/app-frame/app-frame.component.html index 70329d537..34a978fe5 100644 --- a/src-ui/src/app/components/app-frame/app-frame.component.html +++ b/src-ui/src/app/components/app-frame/app-frame.component.html @@ -163,13 +163,13 @@
  • -
    - +
    +  GitHub - + @@ -177,7 +177,7 @@
    • -
  • +
  • {{versionString}}
    From bdc247ce49a73a810e813e2db011e2e86016cc73 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 2 Feb 2021 23:58:25 +0100 Subject: [PATCH 008/225] code style --- src/documents/management/commands/document_consumer.py | 2 +- src/documents/parsers.py | 5 ++--- src/paperless_tesseract/parsers.py | 4 ++-- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/src/documents/management/commands/document_consumer.py b/src/documents/management/commands/document_consumer.py index 9ba1b1a87..232a5b23c 100644 --- a/src/documents/management/commands/document_consumer.py +++ b/src/documents/management/commands/document_consumer.py @@ -21,7 +21,7 @@ logger = logging.getLogger(__name__) def _tags_from_path(filepath): - """Walk up the directory tree from filepath to CONSUMPTION_DIr + """Walk up the directory tree from filepath to CONSUMPTION_DIR and get or create Tag IDs for every directory. """ tag_ids = set() diff --git a/src/documents/parsers.py b/src/documents/parsers.py index 3f0879b3c..a2eb14cef 100644 --- a/src/documents/parsers.py +++ b/src/documents/parsers.py @@ -273,10 +273,9 @@ class DocumentParser(LoggingMixin): self.date = None self.progress_callback = progress_callback - def progress(self, current, max): - print(self.progress_callback) + def progress(self, current_progress, max_progress): if self.progress_callback: - self.progress_callback(current, max) + self.progress_callback(current_progress, max_progress) def extract_metadata(self, document_path, mime_type): return [] diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py index 52b50e983..4e5ceac1b 100644 --- a/src/paperless_tesseract/parsers.py +++ b/src/paperless_tesseract/parsers.py @@ -190,11 +190,11 @@ class RasterisedDocumentParser(DocumentParser): # Also, no archived file. if not self.text: # However, if we don't have anything, fail: - raise ParseError(e.__class__.__name__ + ": " + str(e)) + raise ParseError(f"{e.__class__.__name__}: {str(e)}") except Exception as e: # Anything else is probably serious. - raise ParseError(e.__class__.__name__ + ": " + str(e)) + raise ParseError(f"{e.__class__.__name__}: {str(e)}") if not self.text: # This may happen for files that don't have any text. From 32c9369ba18e1f109c3b6707a49dd459767925e7 Mon Sep 17 00:00:00 2001 From: Fabian Koller Date: Wed, 3 Feb 2021 07:53:49 +0100 Subject: [PATCH 009/225] Apply consistent name structure to vars --- ansible/README.md | 2 +- ansible/defaults/main.yml | 16 ++++++++-------- ansible/tasks/main.yml | 16 ++++++++-------- docs/setup.rst | 20 ++++++++++---------- 4 files changed, 27 insertions(+), 27 deletions(-) diff --git a/ansible/README.md b/ansible/README.md index 6cd9e9f9b..0940cf072 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -107,7 +107,7 @@ Example Playbook paperlessng_db_type: postgresql paperlessng_db_pass: PLEASEPROVIDEASTRONGPASSWORDHERE - paperless_secret_key: AGAINPLEASECHANGETHISNOW + paperlessng_secret_key: AGAINPLEASECHANGETHISNOW paperlessng_ocr_languages: - eng diff --git a/ansible/defaults/main.yml b/ansible/defaults/main.yml index aaeffa507..f1cbb8f88 100644 --- a/ansible/defaults/main.yml +++ b/ansible/defaults/main.yml @@ -23,14 +23,14 @@ paperlessng_filename_format: paperlessng_virtualenv: "{{ paperlessng_directory }}/.venv" # Hosting & Security -paperless_secret_key: PLEASECHANGETHISFORTHELOVEOFGOD -paperless_allowed_hosts: "*" -paperless_cors_allowed_hosts: http://localhost:8000 -paperless_force_script_name: -paperless_static_url: /static/ -paperless_auto_login_username: -paperless_cookie_prefix: "" -paperless_enable_http_remote_user: False +paperlessng_secret_key: PLEASECHANGETHISFORTHELOVEOFGOD +paperlessng_allowed_hosts: "*" +paperlessng_cors_allowed_hosts: http://localhost:8000 +paperlessng_force_script_name: +paperlessng_static_url: /static/ +paperlessng_auto_login_username: +paperlessng_cookie_prefix: "" +paperlessng_enable_http_remote_user: False # OCR settings paperlessng_ocr_languages: diff --git a/ansible/tasks/main.yml b/ansible/tasks/main.yml index dba409549..ed05cd05e 100644 --- a/ansible/tasks/main.yml +++ b/ansible/tasks/main.yml @@ -286,21 +286,21 @@ line: "PAPERLESS_FILENAME_FORMAT={{ paperlessng_filename_format }}" # Hosting & Security - regexp: PAPERLESS_SECRET_KEY - line: "PAPERLESS_SECRET_KEY={{ paperless_secret_key }}" + line: "PAPERLESS_SECRET_KEY={{ paperlessng_secret_key }}" - regexp: PAPERLESS_ALLOWED_HOSTS - line: "PAPERLESS_ALLOWED_HOSTS={{ paperless_allowed_hosts }}" + line: "PAPERLESS_ALLOWED_HOSTS={{ paperlessng_allowed_hosts }}" - regexp: PAPERLESS_CORS_ALLOWED_HOSTS - line: "PAPERLESS_CORS_ALLOWED_HOSTS={{ paperless_cors_allowed_hosts }}" + line: "PAPERLESS_CORS_ALLOWED_HOSTS={{ paperlessng_cors_allowed_hosts }}" - regexp: PAPERLESS_FORCE_SCRIPT_NAME - line: "PAPERLESS_FORCE_SCRIPT_NAME={{ paperless_force_script_name }}" + line: "PAPERLESS_FORCE_SCRIPT_NAME={{ paperlessng_force_script_name }}" - regexp: PAPERLESS_STATIC_URL - line: "PAPERLESS_STATIC_URL={{ paperless_static_url }}" + line: "PAPERLESS_STATIC_URL={{ paperlessng_static_url }}" - regexp: PAPERLESS_AUTO_LOGIN_USERNAME - line: "PAPERLESS_AUTO_LOGIN_USERNAME={{ paperless_auto_login_username }}" + line: "PAPERLESS_AUTO_LOGIN_USERNAME={{ paperlessng_auto_login_username }}" - regexp: PAPERLESS_COOKIE_PREFIX - line: "PAPERLESS_COOKIE_PREFIX={{ paperless_cookie_prefix }}" + line: "PAPERLESS_COOKIE_PREFIX={{ paperlessng_cookie_prefix }}" - regexp: PAPERLESS_ENABLE_HTTP_REMOTE_USER - line: "PAPERLESS_ENABLE_HTTP_REMOTE_USER={{ paperless_enable_http_remote_user }}" + line: "PAPERLESS_ENABLE_HTTP_REMOTE_USER={{ paperlessng_enable_http_remote_user }}" # OCR settings - regexp: PAPERLESS_OCR_LANGUAGE line: "PAPERLESS_OCR_LANGUAGE={{ paperlessng_ocr_languages | join('+') }}" diff --git a/docs/setup.rst b/docs/setup.rst index 5c3c9c361..e040db790 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -88,7 +88,7 @@ You can go multiple routes to setup and run Paperless: The Docker routes are quick & easy. These are the recommended routes. This configures all the stuff from the above automatically so that it just works and uses sensible defaults for all configuration options. -Here you find a cheat-sheet for docker beginners: `CLI Basics `_ +Here you find a cheat-sheet for docker beginners: `CLI Basics `_ The bare metal route is complicated to setup but makes it easier should you want to contribute some code back. You need to configure and @@ -126,7 +126,7 @@ Install Paperless from Docker Hub If you want to use the included ``docker-compose.*.yml`` file, you need to have at least Docker version **17.09.0** and docker-compose - version **1.17.0**. + version **1.17.0**. To check do: `docker-compose -v` or `docker -v` See the `Docker installation guide`_ on how to install the current @@ -151,7 +151,7 @@ Install Paperless from Docker Hub .. code:: - /home/jonaswinkler/paperless-inbox:/usr/src/paperless/consume - + Don't change the part after the colon or paperless wont find your documents. @@ -159,7 +159,7 @@ Install Paperless from Docker Hub most important change is to set ``USERMAP_UID`` and ``USERMAP_GID`` to the uid and gid of your user on the host system. Use ``id -u`` and ``id -g`` to get these. - + This ensures that both the docker container and you on the host machine have write access to the consumption directory. If your UID and GID on the host system is @@ -194,7 +194,7 @@ Install Paperless from Docker Hub 8. The default ``docker-compose.yml`` exports the webserver on your local port 8000. If you did not change this, you should now be able to visit your - Paperless instance at ``http://127.0.0.1:8000`` or your servers IP-Address:8000. + Paperless instance at ``http://127.0.0.1:8000`` or your servers IP-Address:8000. Use the login credentials you have created with the previous step. .. _Docker: https://www.docker.com/ @@ -223,7 +223,7 @@ Build the docker image yourself webserver: image: jonaswinkler/paperless-ng:latest - + and replace it with a line that instructs docker-compose to build the image from the current working directory instead: .. code:: yaml @@ -426,7 +426,7 @@ Install Paperless using ansible This role currently only supports Debian 10 Buster and Ubuntu 20.04 Focal or later as target hosts. -1. Install ansible 2.7+ on the management node. +1. Install ansible 2.7+ on the management node. This may be the target host paperless-ng is being installed on or any remote host which can access the target host. For further details, check the ansible `inventory `_ documentation. @@ -461,7 +461,7 @@ Install Paperless using ansible .. code:: sh cd paperless-ng - git checkout ng-0.9.14 + git checkout ng-1.0.0 3. Create an ansible ``playbook.yml`` in the paperless-ng root directory: @@ -511,7 +511,7 @@ Install Paperless using ansible .. code:: yaml - paperless_secret_key: PleaseGenerateAStrongKeyForThis + paperlessng_secret_key: PleaseGenerateAStrongKeyForThis paperlessng_superuser_name: YourUserName paperlessng_superuser_email: name@domain.tld @@ -584,7 +584,7 @@ Migration to paperless-ng is then performed in a few simple steps: .. code:: shell-session $ docker volume ls | grep _data - + and adjust the project name in the ``.env`` file so that it matches the name of the volumes before the ``_data`` part. From d17de457916b46491ea74ca467146a757d599165 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 3 Feb 2021 14:51:04 +0100 Subject: [PATCH 010/225] fix typo --- src/paperless_tesseract/parsers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py index 4e5ceac1b..5a08c601b 100644 --- a/src/paperless_tesseract/parsers.py +++ b/src/paperless_tesseract/parsers.py @@ -200,7 +200,7 @@ class RasterisedDocumentParser(DocumentParser): # This may happen for files that don't have any text. self.log( 'warning', - f"Document {document_path} does not have any text." + f"Document {document_path} does not have any text. " f"This is probably an error or you tried to add an image " f"without text, or something is wrong with this document.") self.text = "" From a186d6fae2f55d521c58348de102c56766fac632 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 3 Feb 2021 15:22:20 +0100 Subject: [PATCH 011/225] bugfix --- .../components/document-detail/document-detail.component.ts | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src-ui/src/app/components/document-detail/document-detail.component.ts b/src-ui/src/app/components/document-detail/document-detail.component.ts index a7cce715e..071c8e249 100644 --- a/src-ui/src/app/components/document-detail/document-detail.component.ts +++ b/src-ui/src/app/components/document-detail/document-detail.component.ts @@ -115,9 +115,13 @@ export class DocumentDetailComponent implements OnInit { this.document = doc this.documentsService.getMetadata(doc.id).subscribe(result => { this.metadata = result + }, error => { + this.metadata = null }) this.documentsService.getSuggestions(doc.id).subscribe(result => { this.suggestions = result + }, error => { + this.suggestions = null }) this.title = this.documentTitlePipe.transform(doc.title) this.documentForm.patchValue(doc) From 2129cda38d45819a367b658123c41e1d6b6bd8ca Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 3 Feb 2021 16:01:32 +0100 Subject: [PATCH 012/225] display http errors --- src-ui/messages.xlf | 43 +++++++++++-------- .../upload-file-widget.component.ts | 2 +- 2 files changed, 26 insertions(+), 19 deletions(-) diff --git a/src-ui/messages.xlf b/src-ui/messages.xlf index 86198ca7f..a80bd082f 100644 --- a/src-ui/messages.xlf +++ b/src-ui/messages.xlf @@ -188,35 +188,35 @@ Confirm delete src/app/components/document-detail/document-detail.component.ts - 199 + 203 Do you really want to delete document ""? src/app/components/document-detail/document-detail.component.ts - 200 + 204 The files for this document will be deleted permanently. This operation cannot be undone. src/app/components/document-detail/document-detail.component.ts - 201 + 205 Delete document src/app/components/document-detail/document-detail.component.ts - 203 + 207 Error deleting document: src/app/components/document-detail/document-detail.component.ts - 210 + 214 @@ -1470,6 +1470,13 @@ 126 + + HTTP error: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 136 + + Upload new documents @@ -1637,21 +1644,21 @@ Document already exists. src/app/services/consumer-status.service.ts - 14 + 15 File not found. src/app/services/consumer-status.service.ts - 15 + 16 Pre-consume script does not exist. src/app/services/consumer-status.service.ts - 16 + 17 Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1659,7 +1666,7 @@ Error while executing pre-consume script. src/app/services/consumer-status.service.ts - 17 + 18 Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1667,7 +1674,7 @@ Post-consume script does not exist. src/app/services/consumer-status.service.ts - 18 + 19 Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1675,7 +1682,7 @@ Error while executing post-consume script. src/app/services/consumer-status.service.ts - 19 + 20 Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1683,49 +1690,49 @@ Received new file. src/app/services/consumer-status.service.ts - 20 + 21 File type not supported. src/app/services/consumer-status.service.ts - 21 + 22 Processing document... src/app/services/consumer-status.service.ts - 22 + 23 Generating thumbnail... src/app/services/consumer-status.service.ts - 23 + 24 Retrieving date from document... src/app/services/consumer-status.service.ts - 24 + 25 Saving document... src/app/services/consumer-status.service.ts - 25 + 26 Finished. src/app/services/consumer-status.service.ts - 26 + 27 diff --git a/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts b/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts index f604a3fc1..2412290c0 100644 --- a/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts +++ b/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts @@ -133,7 +133,7 @@ export class UploadFileWidgetComponent implements OnInit { break; } default: { - this.consumerStatusService.fail(status, `${error.status} ${error.statusText}`) + this.consumerStatusService.fail(status, $localize`HTTP error: ${error.status} ${error.statusText}`) break; } } From ee487529fe23a8382f893c63c88a2dcd2b0dd880 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 3 Feb 2021 15:11:49 +0000 Subject: [PATCH 013/225] Translate /src-ui/messages.xlf in de translation completed for the source file '/src-ui/messages.xlf' on the 'de' language. --- src-ui/src/locale/messages.de.xlf | 62 +++++++++++++++++-------------- 1 file changed, 35 insertions(+), 27 deletions(-) diff --git a/src-ui/src/locale/messages.de.xlf b/src-ui/src/locale/messages.de.xlf index 44fa03f5c..cc5e1d68a 100644 --- a/src-ui/src/locale/messages.de.xlf +++ b/src-ui/src/locale/messages.de.xlf @@ -214,7 +214,7 @@ Löschen bestätigen src/app/components/document-detail/document-detail.component.ts - 199 + 203 @@ -222,7 +222,7 @@ Möchten Sie das Dokument "" wirklich löschen? src/app/components/document-detail/document-detail.component.ts - 200 + 204 @@ -230,7 +230,7 @@ Die Dateien dieses Dokuments werden permanent gelöscht. Diese Aktion kann nicht rückgängig gemacht werden. src/app/components/document-detail/document-detail.component.ts - 201 + 205 @@ -238,7 +238,7 @@ Dokument löschen src/app/components/document-detail/document-detail.component.ts - 203 + 207 @@ -246,7 +246,7 @@ Fehler beim Löschen des Dokuments: src/app/components/document-detail/document-detail.component.ts - 210 + 214 @@ -427,7 +427,7 @@ Original document metadata - Original-Metadaten + Metadaten Original src/app/components/document-detail/document-detail.component.html 121 @@ -435,7 +435,7 @@ Archived document metadata - Metadaten des archivierten Dokuments + Metadaten Archiv src/app/components/document-detail/document-detail.component.html 122 @@ -451,7 +451,7 @@ Hello , welcome to Paperless-ng! - Hallo , Willkommen zu Paperless-ng + Hallo , willkommen zu Paperless-ng! src/app/components/dashboard/dashboard.component.ts 33 @@ -1678,6 +1678,14 @@ 126 + + HTTP error: + HTTP-Fehler: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 136 + + Upload new documents Neue Dokumente hochladen @@ -1746,7 +1754,7 @@ You can start uploading documents by dropping them in the file upload box to the right or by dropping them in the configured consumption folder and they'll start showing up in the documents list. After you've added some metadata to your documents, use the filtering mechanisms of paperless to create custom views (such as 'Recently added', 'Tagged TODO') and they will appear on the dashboard instead of this message. - Sie können Ihre Dokumente hochladen, indem Sie sie in die Hochladebox auf der rechten Seite ziehen oder in den konfigurierten Ablageordner kopieren/verschieben. Danach werden sie inder Dokumentliste erscheinen. Nachdem Sie den Dokumenten einige Metadaten zugewiesen haben, können Sie die Filtermechanismen zur Erstellung individueller Ansichten benutzen (zum Beispiel "Kürzlich hinzugefügt" oder "Mit TODO markiert") und die Ansichten werden auf der Startseite erscheinen und diese Nachricht ersetzen. + Sie können Ihre Dokumente hochladen, indem Sie sie in die Hochladebox auf der rechten Seite ziehen oder in den konfigurierten Ablageordner kopieren/verschieben. Danach werden sie in der Dokumentenliste erscheinen. Nachdem Sie den Dokumenten einige Metadaten zugewiesen haben, können Sie die Filtermechanismen zur Erstellung individueller Ansichten benutzen (zum Beispiel "Kürzlich hinzugefügt" oder "Mit TODO markiert") und die Ansichten werden auf der Startseite anstelle dieser Nachricht erscheinen. src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html 6,7 @@ -1762,7 +1770,7 @@ Once you've got a couple documents in paperless and added metadata to them, paperless can assign that metadata to new documents automatically. - Nachdem Sie einige Dokumente hochgeladen und mit Metadatebn haben, kann Paperless diese Metadaten automatisch passenden neuen Dokumenten zuweisen. + Nachdem Sie einige Dokumente hochgeladen und diesen Korrespondenten, Tags und Dokumenttypen zugewiesen haben, kann Paperless-ng diese Metadaten automatisch passenden neuen Dokumenten zuweisen. src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html 10 @@ -1770,7 +1778,7 @@ You can configure paperless to read your mails and add documents from attached files. - Sie können in Paperless Ihre E-Mail-Konten einrichten und Paperless wird automatisch Dokumente aus Dateianhängen erzeugen. + Sie können in Paperless Ihre E-Mail-Konten einrichten und Paperless wird automatisch Dokumente aus Dateianhängen hinzufügen. src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html 11 @@ -1869,7 +1877,7 @@ Dokument existiert bereits. src/app/services/consumer-status.service.ts - 14 + 15 @@ -1877,7 +1885,7 @@ Datei nicht gefunden. src/app/services/consumer-status.service.ts - 15 + 16 @@ -1885,7 +1893,7 @@ Pre-Consume-Skript existiert nicht. src/app/services/consumer-status.service.ts - 16 + 17 Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1894,7 +1902,7 @@ Fehler beim Ausführen des Pre-Consume-Skripts. src/app/services/consumer-status.service.ts - 17 + 18 Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1903,7 +1911,7 @@ Post-Consume-Skript existiert nicht. src/app/services/consumer-status.service.ts - 18 + 19 Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1912,7 +1920,7 @@ Fehler beim Ausführen des Post-Consume-Skripts. src/app/services/consumer-status.service.ts - 19 + 20 Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1921,7 +1929,7 @@ Neue Datei erhalten. src/app/services/consumer-status.service.ts - 20 + 21 @@ -1929,7 +1937,7 @@ Dateityp wird nicht unterstützt. src/app/services/consumer-status.service.ts - 21 + 22 @@ -1937,7 +1945,7 @@ Verarbeite Dokument... src/app/services/consumer-status.service.ts - 22 + 23 @@ -1945,7 +1953,7 @@ Erzeuge Miniaturbild... src/app/services/consumer-status.service.ts - 23 + 24 @@ -1953,7 +1961,7 @@ Ermittle Datum des Dokuments... src/app/services/consumer-status.service.ts - 24 + 25 @@ -1961,7 +1969,7 @@ Speichere Dokument... src/app/services/consumer-status.service.ts - 25 + 26 @@ -1969,7 +1977,7 @@ Abgeschlossen. src/app/services/consumer-status.service.ts - 26 + 27 @@ -2222,7 +2230,7 @@ Any: Document contains any of these words (space separated) - Irgendein Wort: Dokument enthält eins der folgender Wörter + Irgendein Wort: Dokument enthält eins der folgenden Wörter src/app/data/matching-model.ts 12 @@ -2270,7 +2278,7 @@ Regular expression: Document matches this regular expression - Regulärer Ausdruck: Dokument passt zum folgenden Audruck + Regulärer Ausdruck: Dokument passt zum folgenden Ausdruck src/app/data/matching-model.ts 15 @@ -2286,7 +2294,7 @@ Fuzzy: Document contains a word similar to this word - Ungenau: Dokument enthält ein zum folgendem Wort ähnliches Wort + Ungenau: Dokument enthält ein zum folgenden Wort ähnliches Wort src/app/data/matching-model.ts 16 From 611b4fe31bc38e448618896723f7028f6f29699c Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 3 Feb 2021 15:25:14 +0000 Subject: [PATCH 014/225] Translate /src-ui/messages.xlf in nl_NL translation completed for the source file '/src-ui/messages.xlf' on the 'nl_NL' language. --- src-ui/src/locale/messages.nl_NL.xlf | 224 +++++++++++++++------------ 1 file changed, 123 insertions(+), 101 deletions(-) diff --git a/src-ui/src/locale/messages.nl_NL.xlf b/src-ui/src/locale/messages.nl_NL.xlf index 0e164d688..becc0e797 100644 --- a/src-ui/src/locale/messages.nl_NL.xlf +++ b/src-ui/src/locale/messages.nl_NL.xlf @@ -214,7 +214,7 @@ Bevestig het verwijderen src/app/components/document-detail/document-detail.component.ts - 199 + 203 @@ -222,7 +222,7 @@ Wilt u het document echt verwijderen ""? src/app/components/document-detail/document-detail.component.ts - 200 + 204 @@ -230,7 +230,7 @@ De bestanden voor dit document worden definitief verwijderd. Deze bewerking kan niet ongedaan worden gemaakt. src/app/components/document-detail/document-detail.component.ts - 201 + 205 @@ -238,7 +238,7 @@ Verwijder document src/app/components/document-detail/document-detail.component.ts - 203 + 207 @@ -246,7 +246,7 @@ Fout bij het verwijderen van het document: src/app/components/document-detail/document-detail.component.ts - 210 + 214 @@ -649,12 +649,20 @@ 10 + + Notifications + Meldingen + + src/app/components/manage/settings/settings.component.html + 115 + + Saved views Opgeslagen views src/app/components/manage/settings/settings.component.html - 128 + 133 @@ -777,68 +785,12 @@ 98 - - Notifications - Meldingen - - src/app/components/manage/settings/settings.component.html - 102 - - - - Consumer status - Status van de verwerker - - src/app/components/manage/settings/settings.component.html - 106 - - - - Show notifications when new documents are detected - Toon meldingen wanneer nieuwe documenten worden gedetecteerd - - src/app/components/manage/settings/settings.component.html - 109 - - - - Show notifications when document consumption completes successfully - Toon meldingen wanneer documenten met succes zijn verwerkt. - - src/app/components/manage/settings/settings.component.html - 110 - - - - Show notifications when document consumption fails - Toon meldingen wanneer het verwerken van documenten faalt. - - src/app/components/manage/settings/settings.component.html - 111 - - - - Suppress notifications on dashboard - Onderdruk meldingen op het dashboard - - src/app/components/manage/settings/settings.component.html - 112 - - - - This will suppress all consumer related status messages on the dashboard. - Dit verbergt alle statusberichten op het dashboard die met de verwerker te maken hebben. - - src/app/components/manage/settings/settings.component.html - 112 - - Bulk editing Massabewerking src/app/components/manage/settings/settings.component.html - 116 + 102 @@ -846,7 +798,7 @@ Toon dialoogvenster ter bevestiging src/app/components/manage/settings/settings.component.html - 120 + 106 @@ -854,7 +806,7 @@ Er zal altijd om bevestiging gevraagd worden voor het verwijderen van documenten. src/app/components/manage/settings/settings.component.html - 120 + 106 @@ -862,7 +814,55 @@ Toepassen bij het sluiten src/app/components/manage/settings/settings.component.html - 121 + 107 + + + + Document processing + Verwerking van documenten + + src/app/components/manage/settings/settings.component.html + 118 + + + + Show notifications when new documents are detected + Toon meldingen wanneer nieuwe documenten worden gedetecteerd + + src/app/components/manage/settings/settings.component.html + 122 + + + + Show notifications when document processing completes successfully + Toon een melding wanneer documenten met succes zijn verwerkt + + src/app/components/manage/settings/settings.component.html + 123 + + + + Show notifications when document processing fails + Toon meldingen wanneer het verwerken van documenten faalt + + src/app/components/manage/settings/settings.component.html + 124 + + + + Suppress notifications on dashboard + Onderdruk meldingen op het dashboard + + src/app/components/manage/settings/settings.component.html + 125 + + + + This will suppress all messages about document processing status on the dashboard. + Dit verbergt alle statusberichten op het dashboard die met het verwerken van documenten te maken hebben. + + src/app/components/manage/settings/settings.component.html + 125 @@ -870,7 +870,7 @@ Komt voor bij src/app/components/manage/settings/settings.component.html - 140 + 145 @@ -878,7 +878,7 @@ Toon op het dashboard src/app/components/manage/settings/settings.component.html - 143 + 148 @@ -886,7 +886,7 @@ Toon in de zijbalk src/app/components/manage/settings/settings.component.html - 147 + 152 @@ -894,7 +894,7 @@ Geen opgeslagen views gedefinieerd. src/app/components/manage/settings/settings.component.html - 157 + 162 @@ -1119,7 +1119,7 @@ Beheren src/app/components/app-frame/app-frame.component.html - 112 + 107 @@ -1127,15 +1127,15 @@ Beheer src/app/components/app-frame/app-frame.component.html - 154 + 149 - - Misc - Andere + + Info + Informatie src/app/components/app-frame/app-frame.component.html - 160 + 155 @@ -1143,7 +1143,7 @@ Handleiding src/app/components/app-frame/app-frame.component.html - 167 + 162 @@ -1151,7 +1151,15 @@ GitHub src/app/components/app-frame/app-frame.component.html - 174 + 170 + + + + Suggest an idea + Ideeënbus + + src/app/components/app-frame/app-frame.component.html + 176 @@ -1167,7 +1175,7 @@ Open documenten src/app/components/app-frame/app-frame.component.html - 92 + 87 @@ -1175,7 +1183,7 @@ Alles sluiten src/app/components/app-frame/app-frame.component.html - 106 + 101 @@ -1662,14 +1670,22 @@ 123 - - Waiting for consumer... - Wachten op het verwerken... + + Upload complete, waiting... + Upload voltooid, klaar voor meer... src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts 126 + + HTTP error: + HTTP fout: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 136 + + Upload new documents Nieuwe documenten opladen @@ -1701,21 +1717,23 @@ src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html 4 + This button dismisses all status messages about processed documents on the dashboard (failed and successful) - - more hidden - niet getoond + + {VAR_PLURAL, plural, =1 {One more document} other { more documents}} + {VAR_PLURAL, plural, =1 {Nog één document} other {Nog documenten}} src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html - 24 + 25 + This is shown as a summary line when there are more than 5 document in the processing pipeline. Open document Open document src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html - 41 + 45 @@ -1859,7 +1877,7 @@ Document bestaat al. src/app/services/consumer-status.service.ts - 14 + 15 @@ -1867,7 +1885,7 @@ Bestand niet gevonden. src/app/services/consumer-status.service.ts - 15 + 16 @@ -1875,39 +1893,43 @@ Pre-verwerkingsscript bestaat niet. src/app/services/consumer-status.service.ts - 16 + 17 + Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation Error while executing pre-consume script. Fout tijdens het uitvoeren van het pre-verwerkingsscript src/app/services/consumer-status.service.ts - 17 + 18 + Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation Post-consume script does not exist. Post-verwerkingsscript bestaat niet. src/app/services/consumer-status.service.ts - 18 + 19 + Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation Error while executing post-consume script. Fout tijdens het uitvoeren van het post-verwerkingsscript src/app/services/consumer-status.service.ts - 19 + 20 + Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation Received new file. Nieuw bestand ontvangen. src/app/services/consumer-status.service.ts - 20 + 21 @@ -1915,7 +1937,7 @@ Bestandstype niet ondersteund. src/app/services/consumer-status.service.ts - 21 + 22 @@ -1923,7 +1945,7 @@ Document wordt verwerkt... src/app/services/consumer-status.service.ts - 22 + 23 @@ -1931,7 +1953,7 @@ Voorbeeldweergave wordt gemaakt... src/app/services/consumer-status.service.ts - 23 + 24 @@ -1939,7 +1961,7 @@ Datum wordt gezocht in document... src/app/services/consumer-status.service.ts - 24 + 25 @@ -1947,7 +1969,7 @@ Document wordt opgeslagen... src/app/services/consumer-status.service.ts - 25 + 26 @@ -1955,7 +1977,7 @@ Klaar. src/app/services/consumer-status.service.ts - 26 + 27 From 28c79f1abb337ceb840178861ece2842129f4b59 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 3 Feb 2021 17:07:37 +0100 Subject: [PATCH 015/225] documentation --- docs/changelog.rst | 2 ++ docs/faq.rst | 36 ++++++++++++++++++++++++++---------- docs/index.rst | 2 +- docs/setup.rst | 45 ++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 73 insertions(+), 12 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 1da8621bb..2b9821408 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -37,6 +37,8 @@ paperless-ng 1.1.0 Paperless will continue to work with WSGI, but you will not get any status notifications. + Apache ``mod_wsgi`` users, see :ref:`this note `. + * Paperless now offers suggestions for tags, correspondents and types on the document detail page. * Added an interactive easy install script that automatically downloads, configures and starts paperless with docker. diff --git a/docs/faq.rst b/docs/faq.rst index e41f9aace..0f4527d62 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -62,9 +62,9 @@ file extensions do not matter. **A:** The short answer is yes. I've tested it on a Raspberry Pi 3 B. The long answer is that certain parts of -Paperless will run very slow, such as the tesseract OCR. On Raspberry Pi, +Paperless will run very slow, such as the OCR. On Raspberry Pi, try to OCR documents before feeding them into paperless so that paperless can -reuse the text. The web interface should be a lot snappier, since it runs +reuse the text. The web interface is a lot snappier, since it runs in your browser and paperless has to do much less work to serve the data. .. note:: @@ -76,7 +76,14 @@ in your browser and paperless has to do much less work to serve the data. **Q:** *How do I install paperless-ng on Raspberry Pi?* **A:** Docker images are available for arm and arm64 hardware, so just follow -the docker-compose instructions, or go the bare metal route. +the docker-compose instructions. Apart from more required disk space compared to +a bare metal installation, docker comes with close to zero overhead, even on +Raspberry Pi. + +If you decide to got with the bare metal route, be aware that some of the +python requirements do not have precompiled packages for ARM / ARM64. Installation +of these will require additional development libraries and compilation will take +a long time. **Q:** *How do I run this on unRaid?* @@ -95,12 +102,21 @@ occasionally build the image on and see if it works. **Q:** *How do I proxy this with NGINX?* -.. code:: +**A:** See :ref:`here `. - location / { - proxy_pass http://localhost:8000/ - } +.. _faq-mod_wsgi: -And that's about it. Paperless serves everything, including static files by itself -when running the docker image. If you want to do anything fancy, you have to -install paperless bare metal. +**Q:** *How do I get WebSocket support with Apache mod_wsgi*? + +**A:** ``mod_wsgi`` by itself does not support ASGI. Paperless will continue +to work with WSGI, but certain features such as status notifications about +document consumption won't be available. + +If you want to continue using ``mod_wsgi``, you will have to run an ASGI-enabled +web server as well that processes WebSocket connections, and configure Apache to +redirect WebSocket connections to this server. Multiple options for ASGI servers +exist: + +* ``gunicorn`` with ``uvicorn`` as the worker implementation (the default of paperless) +* ``daphne`` as a standalone server, which is the reference implementation for ASGI. +* ``uvicorn`` as a standalone server diff --git a/docs/index.rst b/docs/index.rst index a083fb3d1..4035d822e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -70,8 +70,8 @@ Contents configuration api faq - extending troubleshooting + extending contributing scanners screenshots diff --git a/docs/setup.rst b/docs/setup.rst index 6d601378d..139db549a 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -588,7 +588,7 @@ Migration to paperless-ng is then performed in a few simple steps: paperless. 3. Download the latest release of paperless-ng. You can either go with the - docker-compose files from `here `_ + docker-compose files from `here `__ or clone the repository to build the image yourself (see :ref:`above `). You can either replace your current paperless folder or put paperless-ng in a different location. @@ -787,3 +787,46 @@ For details, refer to :ref:`configuration`. well as on any other device. .. _redis: https://redis.io/ + + +.. _setup-nginx: + +Using nginx as a reverse proxy +############################## + +If you want to expose paperless to the internet, you should hide it behind a +reverse proxy with SSL enabled. + +In addition to the usual configuration for SSL, +the following configuration is required for paperless to operate: + +.. code:: nginx + + http { + + # Adjust as required. This is the maximum size for file uploads. + # The default value 1M might be a little too small. + client_max_body_size 10M; + + server { + + location / { + + # Adjust host and port as required. + proxy_pass http://localhost:8000/; + + # These configuration options are required for WebSockets to work. + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + + proxy_redirect off; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Host $server_name; + } + } + } + +Also read `this `__, towards the end of the section. From 49d2084d8a41b7c76a9651cf5f1cc5daf4020442 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 3 Feb 2021 17:10:45 +0000 Subject: [PATCH 016/225] Translate /src-ui/messages.xlf in fr translation completed for the source file '/src-ui/messages.xlf' on the 'fr' language. --- src-ui/src/locale/messages.fr.xlf | 202 ++++++++++++++++-------------- 1 file changed, 109 insertions(+), 93 deletions(-) diff --git a/src-ui/src/locale/messages.fr.xlf b/src-ui/src/locale/messages.fr.xlf index b3c84f14c..0b7e86165 100644 --- a/src-ui/src/locale/messages.fr.xlf +++ b/src-ui/src/locale/messages.fr.xlf @@ -214,7 +214,7 @@ Confirmer la suppression src/app/components/document-detail/document-detail.component.ts - 199 + 203 @@ -222,7 +222,7 @@ Voulez-vous vraiment supprimer le document "" ? src/app/components/document-detail/document-detail.component.ts - 200 + 204 @@ -230,7 +230,7 @@ Les fichiers liés à ce document seront supprimés définitivement. Cette action est irréversible. src/app/components/document-detail/document-detail.component.ts - 201 + 205 @@ -238,7 +238,7 @@ Supprimer le document src/app/components/document-detail/document-detail.component.ts - 203 + 207 @@ -246,7 +246,7 @@ Une erreur s'est produite lors de la suppression du document : src/app/components/document-detail/document-detail.component.ts - 210 + 214 @@ -649,12 +649,20 @@ 10 + + Notifications + Notifications + + src/app/components/manage/settings/settings.component.html + 115 + + Saved views Vues enregistrées src/app/components/manage/settings/settings.component.html - 128 + 133 @@ -777,68 +785,12 @@ 98 - - Notifications - Notifications - - src/app/components/manage/settings/settings.component.html - 102 - - - - Document processing - Traitement de document - - src/app/components/manage/settings/settings.component.html - 106 - - - - Show notifications when new documents are detected - Afficher des notifications lorsque de nouveaux documents sont détectés - - src/app/components/manage/settings/settings.component.html - 109 - - - - Show notifications when document processing completes successfully - Afficher des notifications lorsque le traitement des documents se termine avec succès - - src/app/components/manage/settings/settings.component.html - 110 - - - - Show notifications when document processing fails - Afficher des notifications en cas d'échec du traitement des documents - - src/app/components/manage/settings/settings.component.html - 111 - - - - Suppress notifications on dashboard - Supprimer les notifications du tableau de bord - - src/app/components/manage/settings/settings.component.html - 112 - - - - This will suppress all messages about document processing status on the dashboard. - Cela supprimera tous les messages liés au traitement de documents sur le tableau de bord. - - src/app/components/manage/settings/settings.component.html - 112 - - Bulk editing Edition en masse src/app/components/manage/settings/settings.component.html - 116 + 102 @@ -846,7 +798,7 @@ Afficher les messages de confirmation src/app/components/manage/settings/settings.component.html - 120 + 106 @@ -854,7 +806,7 @@ La suppression de documents requiert toujours une confirmation. src/app/components/manage/settings/settings.component.html - 120 + 106 @@ -862,7 +814,55 @@ Appliquer lors de la fermeture src/app/components/manage/settings/settings.component.html - 121 + 107 + + + + Document processing + Traitement de document + + src/app/components/manage/settings/settings.component.html + 118 + + + + Show notifications when new documents are detected + Afficher des notifications lorsque de nouveaux documents sont détectés + + src/app/components/manage/settings/settings.component.html + 122 + + + + Show notifications when document processing completes successfully + Afficher des notifications lorsque le traitement des documents se termine avec succès + + src/app/components/manage/settings/settings.component.html + 123 + + + + Show notifications when document processing fails + Afficher des notifications en cas d'échec du traitement des documents + + src/app/components/manage/settings/settings.component.html + 124 + + + + Suppress notifications on dashboard + Supprimer les notifications du tableau de bord + + src/app/components/manage/settings/settings.component.html + 125 + + + + This will suppress all messages about document processing status on the dashboard. + Cela supprimera tous les messages liés au traitement de documents sur le tableau de bord. + + src/app/components/manage/settings/settings.component.html + 125 @@ -870,7 +870,7 @@ Apparaît sur src/app/components/manage/settings/settings.component.html - 140 + 145 @@ -878,7 +878,7 @@ Montrer sur le tableau de bord src/app/components/manage/settings/settings.component.html - 143 + 148 @@ -886,7 +886,7 @@ Montrer dans la barre latérale src/app/components/manage/settings/settings.component.html - 147 + 152 @@ -894,7 +894,7 @@ Aucune vue sauvegardée n'est définie. src/app/components/manage/settings/settings.component.html - 157 + 162 @@ -1119,7 +1119,7 @@ Gestion src/app/components/app-frame/app-frame.component.html - 112 + 107 @@ -1127,15 +1127,15 @@ Administration src/app/components/app-frame/app-frame.component.html - 154 + 149 - - Misc - Divers + + Info + Info src/app/components/app-frame/app-frame.component.html - 160 + 155 @@ -1143,7 +1143,7 @@ Documentation src/app/components/app-frame/app-frame.component.html - 167 + 162 @@ -1151,7 +1151,15 @@ GitHub src/app/components/app-frame/app-frame.component.html - 174 + 170 + + + + Suggest an idea + Suggérer une idée + + src/app/components/app-frame/app-frame.component.html + 176 @@ -1167,7 +1175,7 @@ Documents ouverts src/app/components/app-frame/app-frame.component.html - 92 + 87 @@ -1175,7 +1183,7 @@ Fermer tout src/app/components/app-frame/app-frame.component.html - 106 + 101 @@ -1670,6 +1678,14 @@ 126 + + HTTP error: + Erreur HTTP : + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 136 + + Upload new documents Charger de nouveaux documents @@ -1861,7 +1877,7 @@ Le document existe déjà. src/app/services/consumer-status.service.ts - 14 + 15 @@ -1869,7 +1885,7 @@ Fichier non trouvé. src/app/services/consumer-status.service.ts - 15 + 16 @@ -1877,7 +1893,7 @@ Le script de pré-traitement n'existe pas. src/app/services/consumer-status.service.ts - 16 + 17 Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1886,7 +1902,7 @@ Erreur lors de l'exécution du script de pré-traitement. src/app/services/consumer-status.service.ts - 17 + 18 Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1895,7 +1911,7 @@ Le script de post-traitement n'existe pas. src/app/services/consumer-status.service.ts - 18 + 19 Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1904,7 +1920,7 @@ Erreur lors de l'exécution du script de post-traitement. src/app/services/consumer-status.service.ts - 19 + 20 Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation @@ -1913,7 +1929,7 @@ Réception d'un nouveau fichier. src/app/services/consumer-status.service.ts - 20 + 21 @@ -1921,7 +1937,7 @@ Type de fichier non pris en charge. src/app/services/consumer-status.service.ts - 21 + 22 @@ -1929,7 +1945,7 @@ Traitement du document... src/app/services/consumer-status.service.ts - 22 + 23 @@ -1937,7 +1953,7 @@ Génération de vignette... src/app/services/consumer-status.service.ts - 23 + 24 @@ -1945,7 +1961,7 @@ Extraction de la date du document... src/app/services/consumer-status.service.ts - 24 + 25 @@ -1953,7 +1969,7 @@ Enregistrement du document... src/app/services/consumer-status.service.ts - 25 + 26 @@ -1961,7 +1977,7 @@ Terminé. src/app/services/consumer-status.service.ts - 26 + 27 From ba3c8308b369a329f1e92e4419f6d8ad697c667f Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 3 Feb 2021 20:44:10 +0100 Subject: [PATCH 017/225] websocket authentication --- src/paperless/consumers.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/src/paperless/consumers.py b/src/paperless/consumers.py index 21a0e3ede..b3d31b402 100644 --- a/src/paperless/consumers.py +++ b/src/paperless/consumers.py @@ -1,14 +1,19 @@ import json from asgiref.sync import async_to_sync +from channels.exceptions import DenyConnection, AcceptConnection from channels.generic.websocket import WebsocketConsumer class StatusConsumer(WebsocketConsumer): + def connect(self): - self.accept() - async_to_sync(self.channel_layer.group_add)( - 'status_updates', self.channel_name) + if not self.scope['user'].is_authenticated: + raise DenyConnection() + else: + async_to_sync(self.channel_layer.group_add)( + 'status_updates', self.channel_name) + raise AcceptConnection() def disconnect(self, close_code): async_to_sync(self.channel_layer.group_discard)( From c51fa6017235fe14b757a6d536f74064cff6c578 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 3 Feb 2021 21:53:39 +0100 Subject: [PATCH 018/225] close connection when not authenticated anymore. --- src/paperless/consumers.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/src/paperless/consumers.py b/src/paperless/consumers.py index b3d31b402..45f6ad9c5 100644 --- a/src/paperless/consumers.py +++ b/src/paperless/consumers.py @@ -7,8 +7,11 @@ from channels.generic.websocket import WebsocketConsumer class StatusConsumer(WebsocketConsumer): + def _authenticated(self): + return 'user' in self.scope and self.scope['user'].is_authenticated + def connect(self): - if not self.scope['user'].is_authenticated: + if not self._authenticated(): raise DenyConnection() else: async_to_sync(self.channel_layer.group_add)( @@ -20,4 +23,7 @@ class StatusConsumer(WebsocketConsumer): 'status_updates', self.channel_name) def status_update(self, event): - self.send(json.dumps(event['data'])) + if not self._authenticated(): + self.close() + else: + self.send(json.dumps(event['data'])) From b9e1ed7d7e2a60752434a6493be429040d502c75 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 3 Feb 2021 23:18:28 +0100 Subject: [PATCH 019/225] added test for web socket authentication --- src/paperless/tests/test_websockets.py | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 src/paperless/tests/test_websockets.py diff --git a/src/paperless/tests/test_websockets.py b/src/paperless/tests/test_websockets.py new file mode 100644 index 000000000..e5745ce77 --- /dev/null +++ b/src/paperless/tests/test_websockets.py @@ -0,0 +1,25 @@ +from unittest import mock + +from channels.testing import WebsocketCommunicator +from django.test import TestCase + +from paperless.asgi import application + + +class TestWebSockets(TestCase): + + async def test_no_auth(self): + communicator = WebsocketCommunicator(application, "/ws/status/") + connected, subprotocol = await communicator.connect() + self.assertFalse(connected) + await communicator.disconnect() + + @mock.patch("paperless.consumers.StatusConsumer._authenticated") + async def test_auth(self, _authenticated): + _authenticated.return_value = True + + communicator = WebsocketCommunicator(application, "/ws/status/") + connected, subprotocol = await communicator.connect() + self.assertTrue(connected) + + await communicator.disconnect() From 883a6b26a4e71e94e65749389a6d82812f5dfced Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 4 Feb 2021 00:17:23 +0100 Subject: [PATCH 020/225] update gunicorn.conf --- gunicorn.conf.py | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/gunicorn.conf.py b/gunicorn.conf.py index 36bde9cf7..a67b20d0c 100644 --- a/gunicorn.conf.py +++ b/gunicorn.conf.py @@ -1,21 +1,7 @@ bind = '0.0.0.0:8000' -backlog = 2048 -workers = 3 +workers = 2 worker_class = 'uvicorn.workers.UvicornWorker' -worker_connections = 1000 timeout = 20 -keepalive = 2 -spew = False -daemon = False -pidfile = None -umask = 0 -user = None -group = None -tmp_upload_dir = None -loglevel = 'info' -errorlog = '-' -accesslog = '-' -proc_name = None def pre_fork(server, worker): pass From 44ec3a3d9c5fe0b5916839495fcd3461d4af106d Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 4 Feb 2021 13:17:24 +0100 Subject: [PATCH 021/225] lazy loading for parsers --- src/documents/tests/test_parsers.py | 4 ++-- src/paperless_tesseract/signals.py | 8 ++++++-- src/paperless_text/signals.py | 8 ++++++-- src/paperless_tika/__init__.py | 0 src/paperless_tika/signals.py | 8 ++++++-- 5 files changed, 20 insertions(+), 8 deletions(-) create mode 100644 src/paperless_tika/__init__.py diff --git a/src/documents/tests/test_parsers.py b/src/documents/tests/test_parsers.py index 392c0504f..8da6470a4 100644 --- a/src/documents/tests/test_parsers.py +++ b/src/documents/tests/test_parsers.py @@ -114,8 +114,8 @@ class TestParserAvailability(TestCase): self.assertEqual(get_default_file_extension('application/zip'), ".zip") self.assertEqual(get_default_file_extension('aasdasd/dgfgf'), "") - self.assertEqual(get_parser_class_for_mime_type('application/pdf'), RasterisedDocumentParser) - self.assertEqual(get_parser_class_for_mime_type('text/plain'), TextDocumentParser) + self.assertIsInstance(get_parser_class_for_mime_type('application/pdf')(logging_group=None), RasterisedDocumentParser) + self.assertIsInstance(get_parser_class_for_mime_type('text/plain')(logging_group=None), TextDocumentParser) self.assertEqual(get_parser_class_for_mime_type('text/sdgsdf'), None) self.assertTrue(is_file_ext_supported('.pdf')) diff --git a/src/paperless_tesseract/signals.py b/src/paperless_tesseract/signals.py index 1e1cd1e1a..fedd08a92 100644 --- a/src/paperless_tesseract/signals.py +++ b/src/paperless_tesseract/signals.py @@ -1,9 +1,13 @@ -from .parsers import RasterisedDocumentParser + +def get_parser(*args, **kwargs): + from .parsers import RasterisedDocumentParser + + return RasterisedDocumentParser(*args, **kwargs) def tesseract_consumer_declaration(sender, **kwargs): return { - "parser": RasterisedDocumentParser, + "parser": get_parser, "weight": 0, "mime_types": { "application/pdf": ".pdf", diff --git a/src/paperless_text/signals.py b/src/paperless_text/signals.py index 1e0493f4f..833d0be28 100644 --- a/src/paperless_text/signals.py +++ b/src/paperless_text/signals.py @@ -1,9 +1,13 @@ -from .parsers import TextDocumentParser + +def get_parser(*args, **kwargs): + from .parsers import TextDocumentParser + + return TextDocumentParser(*args, **kwargs) def text_consumer_declaration(sender, **kwargs): return { - "parser": TextDocumentParser, + "parser": get_parser, "weight": 10, "mime_types": { "text/plain": ".txt", diff --git a/src/paperless_tika/__init__.py b/src/paperless_tika/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/src/paperless_tika/signals.py b/src/paperless_tika/signals.py index 137416abb..dbda377ca 100644 --- a/src/paperless_tika/signals.py +++ b/src/paperless_tika/signals.py @@ -1,9 +1,13 @@ -from .parsers import TikaDocumentParser + +def get_parser(*args, **kwargs): + from .parsers import TikaDocumentParser + + return TikaDocumentParser(*args, **kwargs) def tika_consumer_declaration(sender, **kwargs): return { - "parser": TikaDocumentParser, + "parser": get_parser, "weight": 10, "mime_types": { "application/msword": ".doc", From 8711a206a0eb7dafa40ba172c51f0d6ce8821845 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 4 Feb 2021 13:26:58 +0100 Subject: [PATCH 022/225] fix websocket test cases --- src/paperless/tests/test_websockets.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/paperless/tests/test_websockets.py b/src/paperless/tests/test_websockets.py index e5745ce77..4f98fe2b5 100644 --- a/src/paperless/tests/test_websockets.py +++ b/src/paperless/tests/test_websockets.py @@ -8,14 +8,16 @@ from paperless.asgi import application class TestWebSockets(TestCase): - async def test_no_auth(self): + @mock.patch("paperless.consumers.async_to_sync") + async def test_no_auth(self, async_to_sync): communicator = WebsocketCommunicator(application, "/ws/status/") connected, subprotocol = await communicator.connect() self.assertFalse(connected) await communicator.disconnect() + @mock.patch("paperless.consumers.async_to_sync") @mock.patch("paperless.consumers.StatusConsumer._authenticated") - async def test_auth(self, _authenticated): + async def test_auth(self, _authenticated, async_to_sync): _authenticated.return_value = True communicator = WebsocketCommunicator(application, "/ws/status/") From 758631c53f29ca5d8637fc44a6fb643237c790ce Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Thu, 4 Feb 2021 13:31:20 +0100 Subject: [PATCH 023/225] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 0fc3185a1..80b9c85fe 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ [![ci](https://github.com/jonaswinkler/paperless-ng/workflows/ci/badge.svg)](https://github.com/jonaswinkler/paperless-ng/actions) +![Ansible Role](https://github.com/jonaswinkler/paperless-ng/workflows/Ansible%20Role/badge.svg) [![Documentation Status](https://readthedocs.org/projects/paperless-ng/badge/?version=latest)](https://paperless-ng.readthedocs.io/en/latest/?badge=latest) [![Gitter](https://badges.gitter.im/paperless-ng/community.svg)](https://gitter.im/paperless-ng/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Docker Hub Pulls](https://img.shields.io/docker/pulls/jonaswinkler/paperless-ng.svg)](https://hub.docker.com/r/jonaswinkler/paperless-ng) From 866c8fc848d1b5918bdbd76b87caff71b9e9cf50 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 4 Feb 2021 14:17:18 +0100 Subject: [PATCH 024/225] websocket testing --- src/paperless/tests/test_websockets.py | 43 +++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/src/paperless/tests/test_websockets.py b/src/paperless/tests/test_websockets.py index 4f98fe2b5..e1b948edf 100644 --- a/src/paperless/tests/test_websockets.py +++ b/src/paperless/tests/test_websockets.py @@ -1,23 +1,31 @@ from unittest import mock +from channels.layers import get_channel_layer from channels.testing import WebsocketCommunicator -from django.test import TestCase +from django.test import TestCase, override_settings from paperless.asgi import application +TEST_CHANNEL_LAYERS = { + 'default': { + 'BACKEND': 'channels.layers.InMemoryChannelLayer', + }, +} + + class TestWebSockets(TestCase): - @mock.patch("paperless.consumers.async_to_sync") - async def test_no_auth(self, async_to_sync): + @override_settings(CHANNEL_LAYERS=TEST_CHANNEL_LAYERS) + async def test_no_auth(self): communicator = WebsocketCommunicator(application, "/ws/status/") connected, subprotocol = await communicator.connect() self.assertFalse(connected) await communicator.disconnect() - @mock.patch("paperless.consumers.async_to_sync") + @override_settings(CHANNEL_LAYERS=TEST_CHANNEL_LAYERS) @mock.patch("paperless.consumers.StatusConsumer._authenticated") - async def test_auth(self, _authenticated, async_to_sync): + async def test_auth(self, _authenticated): _authenticated.return_value = True communicator = WebsocketCommunicator(application, "/ws/status/") @@ -25,3 +33,28 @@ class TestWebSockets(TestCase): self.assertTrue(connected) await communicator.disconnect() + + @override_settings(CHANNEL_LAYERS=TEST_CHANNEL_LAYERS) + @mock.patch("paperless.consumers.StatusConsumer._authenticated") + async def test_receive(self, _authenticated): + _authenticated.return_value = True + + communicator = WebsocketCommunicator(application, "/ws/status/") + connected, subprotocol = await communicator.connect() + self.assertTrue(connected) + + message = { + "task_id": "test" + } + + channel_layer = get_channel_layer() + await channel_layer.group_send("status_updates", { + "type": "status_update", + "data": message + }) + + response = await communicator.receive_json_from() + + self.assertEqual(response, message) + + await communicator.disconnect() From d8e0ef257e73f28bc8eb7ae71a6d56f1c518ea7d Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 4 Feb 2021 15:15:11 +0100 Subject: [PATCH 025/225] don't load sklearn libraries unless needed --- src/documents/classifier.py | 10 ++++++---- src/documents/tests/data/model.pickle | Bin 0 -> 156345 bytes src/documents/tests/test_classifier.py | 9 +++++++++ src/paperless/settings.py | 13 ++++++++----- 4 files changed, 23 insertions(+), 9 deletions(-) create mode 100644 src/documents/tests/data/model.pickle diff --git a/src/documents/classifier.py b/src/documents/classifier.py index b427264c8..3051b82a5 100755 --- a/src/documents/classifier.py +++ b/src/documents/classifier.py @@ -5,10 +5,6 @@ import pickle import re from django.conf import settings -from sklearn.feature_extraction.text import CountVectorizer -from sklearn.neural_network import MLPClassifier -from sklearn.preprocessing import MultiLabelBinarizer, LabelBinarizer -from sklearn.utils.multiclass import type_of_target from documents.models import Document, MatchingModel @@ -109,6 +105,10 @@ class DocumentClassifier(object): pickle.dump(self.document_type_classifier, f) def train(self): + from sklearn.feature_extraction.text import CountVectorizer + from sklearn.neural_network import MLPClassifier + from sklearn.preprocessing import MultiLabelBinarizer, LabelBinarizer + data = list() labels_tags = list() labels_correspondent = list() @@ -265,6 +265,8 @@ class DocumentClassifier(object): return None def predict_tags(self, content): + from sklearn.utils.multiclass import type_of_target + if self.tags_classifier: X = self.data_vectorizer.transform([preprocess_content(content)]) y = self.tags_classifier.predict(X) diff --git a/src/documents/tests/data/model.pickle b/src/documents/tests/data/model.pickle new file mode 100644 index 0000000000000000000000000000000000000000..db303ec807671383449aa8ac83a427d814a5ca2f GIT binary patch literal 156345 zcmcG#c|29^_cxBrWJ-hvQcrAWq%nUgUyAq|Ff3~|Uj&+|O9j^mgK(IBFd zNQrPas(#0P-=FXI`@Ej#_j+ExKYr&A``XuD*R}UvYrWssT5Ded>D3vf0_kJ7(ERgL z;q_X|QK6mcH!gIskP!>ss5xa^*+5(uDWUsEgs{WS-oe`3)k(_6+T7j4)!NM3+uhaN z(%sJ4Ny?qtCK5PRoIRY}^{g%3on7sGtzC&R#J#tO{zP#CgPoI$2UUW}(%H$~+KJj? zwsx{~wz6}wB@!4s+-)R}5((^9*3>TQysoa!u0#T(o4c!>B~^6edc(!c+|rUd(v7G= zV0Uq~c5!vKw03i&3ThBo+@0;Mo&LU~27%e!$=tzb{q6*MFK1UPB7w!h*~{A1(%j9O z7)sdqx2&0qxx2fys}pr{aq!q{Y+>wW>?L7rK_sxcxjVaMftCRIHY)-bW=8k5r=1#WOM0Hwqx?t+8p3au$79I}f zu0HFtTG%=TGpaa;*?&Ks znVl6ej4@D8$mdT0f$e|v6C;TPw*NV9-2n9NSMA)0>YN0+^#ePB*~;0{!*Sgb>Wl<> z8&_vXqB;wK&QhAFPETN37ZzR@?xBu-mRJW5db1=8C zb|4ZqUGQ*lw>!UnqP%XF|J8AsEFH|<+^pB_y3x(W+||w6%-O?z-IwnX*v%}>Ew56I zWVP<5RL}cY){IEl{I@-&ES+7gr5xAiH+OY4r`ng@%$4fnPE?2Xpbnv$`9FqdaJU1+qr`im|6JWbpglp<(9k&dhx+^bk2;Mz z(c+#87yCbNPb2y7kx1?Szboy*@V`ov{EyO{tUX-K9n74p-Km(cmooeNZ5+&ftX<7q zsGzcTad%zU{l*LDFRHAo%Ff2_e*&L%ogh5T*GYnkM^|eHk9A->J5Y^AJvX;9cckjT z+`;9lITd4qa&JCtRgomHT9~_AUNv*0nqpnZ!`+#BgTJ|lYB(ye+^OBnf2+}aol1EA zmrG`LPIm6p$u3%4c>K9wjlkrxj()TCu2=uQ9P>J2?Wp)Cs&^CU9h}|VsPj2E+nW8; zO35&!XNwBe***_Cu=wA!#I1onf=$*Pyz>) zYaM+42E5BUP2MAH@-%m_v-(??W;XvaFLj!Rg!Nw{l~t_W&CR6$$Hf2i9HRCtetk5? z7a~DmvUYK^bD-KnO;Ob+;>|&VJb}yUZ&jE%Ih$Et{fD!tIR2Z{Je-IZ;^he(PX9%3 zROOlx)w!vT^f!9U{wA`&5kvUDz>MSn4Ph)!)>Os+jYTSHu(&w8{pXDKe}^2CkJ&n5 zQW3`cuOm`nD{1FMMW8u#(OF+B{sHOV>&-f4{{IL{ng7qA{1=Nf|56jp{~eT94_=ig zusZ!qI&M_u(-AhVuPy)Wd<5Qq$mX9;DKi&4OX@;Rg@xI_a)0}%+4`cq&NxgLbfk|R zJA8x~MCF(DrD3Cmox7Q>wUad!SX7a#=Lrn|c=h_l4ffVP#H)v|%B{1~)uaD%(gl}w zQc@t$dC2@X6(^@nXUK1oa@`=!+*3? zlg;?~nsx^|@zeS3;wS^^^|_)hmALSO;&aV9E}ApS?7w7Nf}EDt-FH{FL^xNc}LT?pYrPB+aBuS^$o9+Y(LT$9AnULw&>69 z-*~7c=a(MWBIgxnO_O=43yrf$5hGq&YA2cK3{Iu@8t2hboOM1p_XzedMoI*`Y!&w? zyM=2mp+=9bm)@Li)e(3(;*#{JYF7QwH5&#GVRPRS=PV;JNhS8O-O)k^FZ@htIrX@K zblFw&-u-W77xFkr10}0bzK7@D=V})EAwx*FM*i-Ls5QNj(;EwFhI=8z%-b1YaxLv!$_FGdz%s^72WOxii*Q zR7)rQAzDf~Lo|l}b*P#fbH-;DmKO_x0dn7LglFAM?)d26W?!1y|NdEqan`CP%VJ-( z7vV4MLsx$DS22q|>C8^uHD0tobXm$%4?p=aPtkd2#ak?7M7iCJNi_rv7bKh+pedY+bqovx(gWD9l9M{=)u@Y zk-xe1L%r>hBt?UK4F%p=w!0P;4pW(1{3xeo&Oc9R&OX20AlIPwNbbjSTa&G`se!FD zwJ&lB&r+nl@={bDaSdIU`fGx6h~b@L=%4esS}(OTXR>7|9%_bz!W=$J_rL9KA;&5| zd*fJp2^I7cr)}By*Dxz^F5Uu%*@fjV_OoBlw6}k_K4WA?^HL^GYtSp6_@4jn@hmmn zY9noBhulw5q`yFHf9)*+&0-VseT|-zswe%VhUMpt6SO&EyuZ!K#Z`uiJfHdXs=@N$ z(7aMa1+Ae_tH|y#2|1U&yiU5jI=)U*=@E9v z#(v&UobUKuVU|6mOj>$gyDQl1bh}w|$woi+bI#D<_}JJfNd8R4!>vXpuS!ouGxC+x zUAizI{DAF%`^#2)+Nf%$<8S?Xd2X=_e_aYWT*m-oKdF68j=4|#e!bIlEXdL<@{ zZ+7()(?@A0mzq~aE}qaDWN+}mQ87Qc$bhtP8^t3O{! zeoK^yjfDB{dBW${dj><@{_H%@-q1$ds{8xlkGkTgvi`Z%3^yWFUv8%Fj$B$hetx`X z|77X6ja-e&wDmcpXvHVnbWV8b?GZUEq0FG1@QF9d-$sY-Oye$<$SENjExK=`t0^C{ zd~P~tXEynL6nyvc*^SC(LH|kh`grM$1+HVl#y$)gE862FnfK#)8wC`U@Z0T<>TBet z&zYx_FS{<>YGl=z+88+-T;i6sa7XvqC)pRyS5-PrOq!T)-*f#`*M!jKm)cLt9M#{% zjYm0rJxmTA{I>V4R@FAwyEG@y@YwWV5a>-RK=P*C!)lub)-fS46XEd=9o-!MS)04vdN9GzysmG&xO%?zo!eqAU9a4>`C)_2??0c6FFerR z(QLR7eM8mXLYSB}otd)P*E{o!iDjXBRMZ`r)f?>^<4qkE+%S0B*V3Ty2=~(QzQQFs z^;oa2Q}U-O%0kMuckrjaq+Kh%u$eqI_$n}AxbRQOwiMqz9NAJgx|nxp9^aF>s&l4z z%XS`#A%~x@k2di51s(7%T-}qan)pZ6Tc^LF-t*qK9sHeEtOJfsdhzcDxfp6QN_3~T zzpsdReNnZoZiy07TIuroSGx?WO0lf;bD^cAKGN7E_czhciu@CoKY3>fOG~@two5Mu z3PP$C_vNDE$Ke&1kZDg{fm@)g$S>7~O!J>3l%2mj?`^94JTz9@&wqP_DVxbzeRlqo zdKbFL9OJ(lLMBXb^2$`adb8V6fJ91sES*l_68Ms*IWOAW|%GoIiW>s_|cO7V?ik;kMouje@C6ZMLQ3@2XDw_;MxQkxxP%v=d!qS9$&QDYJThF<)rz9TOT%5eKup>D$eazq(xk=$x5ZaAI2AU z=z(DQC}nz{n{YW`k0mhhUb zo!$p4!&n_{PPCx$SJjanJsQ}wIc+X@OD`y`RS9=eI$-Ea#t8x2cHFR|s7-5c0X9kr z8{Ek%!QzkZWS4;~NYR`<#FWwtaVx?J+WX0%YwH$S!xW9wSlMm6*R-iOBI1__O^_%!&2#`;|G4c-+3FDbx?*1cG2}Rb|W%^PR z!P1*>iMO@@3q$v{w9w?ivwq4`>v{?ft8eeQa=0Gr)Z=^mR4>4Gd;Qb(9L``ELMOgU zivkp9rBTZ{WpFtA(_w>P8Kg&^&dnYU#t)78^?Qw5KuLJVAKJWXY_==6{-sC3`naeS zy22(XU_H4+S5}7?=8bjQqbgu&x&63lV=Z!JbA40hC4uMOYF?A+cHm(;W<@tmf#n5* zJJF*3U}^oUL8hS}-}mZPrK+n!aYz~eyhRFb?>GG^b*&3$n|oB37%IUdTww90R~E!* zsGgrm?MHXk)NVTdPLz17WczWY6(a7~e)?!~5v2;oe^hyrz~7US`rD9e+4u)%+c$Lm5j4EjC(-t;mCotfUM6q2(c{=@4% zDZ-s_p@+9^!$cb)H$(p} z%gZO74#6kWpZ9Y&bYO_$jozxSJ+Rc`?IVdhJiN3S z^}pU=SfHH2SG1hzy0g0>)$^K0woe-x$PuT`eP{)l%)#{QQw3m{GVqGJK0r?dze-08 z3Ger5CTuisMQn2%Xk?6r38g1|QKzcF{5*@rXP!)mF8KaV;CBhSNIA(C-6Ud=#Gchu zi8|z}4kr;eZ~3>S9vpNB}BR(UlD;HsYaVi)(*m4v;N3*Mg!bs zJ=f9U-3twQJcC309q?t1?U7?+2To)&dvL_5;G@xVjAZF%P_-jZM17Wnhk1!v2O3&n z;(4rDLsB0WOe#Lnu5QENA9|9^j+GGaQ5OQm;v|vv^Ix=pX3hD}iA02@w zalReSH|*rd-CB(rtPAQ7}4guV{cnN3Y)U4@{oMmMz0{V0{dPkm3hw6g5LS! zJ05;y6o&nbKZ^;x8ZN0UneTNQqR4SOu7j(gd%*~^J+~wfV zqjJ`=-+54JyqiUCq8l4d9AMMGU5m!I9X01R7UHF+;#P`F%^1skjq}!O4n#YwRVkVV z;mRh{_KCgjB+y0{V**2_~89lrO=qqYVT{4hYJ16y(7VGP`6fi;HkA2?BKcP zv6YpC#Ti#ar28{q4`m?Dm%AAQJT|?!J}LzzL&iNHs@iZ&kd2*%TnoCA>V-)Ma^Xk{ zWtCTix<0jA4BLk_MoY2!6d3_0X7%2Q{MwLd-f*uSA4D^G6$Q^xf|+adTE5oLNd9j16{zl&+=NExB^Y z%-t%YU_K04EiZl;T~Pu02fER}<2yjxHf&UEQ#(W+*>IKnYB~fGE_^E5-VX~I(NoSQ zwQ$(LsIMxY1Z$O^$yH8yAW>9(M(_d!tL|O*e!L?dKGwav&&yu|an7Y*5=07NQ;OOA zC^s4J6f4|3v)YXu{`Z8M7K-3*pzqqZhz@WZ-seeKEWp9%5%)q3j$=a0a`%)^3mlZ! z44bJZ!)H#Giwe!qeut@ZqP-1-?!Nm>cYT0*ylb&}sTnxmMcqf%U-m;)z?Q zbUA1?F)&}8)_Oh|IRn?y@VfMi+nQpIj(~VwUXulC31XTuWX(`mht8ySWcJ%h!Sq zXHr@OS0gIK`3%l)$-%rAKlZ!zlELLdmT?e&5Bzd2B&m*2e4%biI6T51F%zY~L%hv=RVLXx*-8j6fv=4t6CM&nTE5v$fjg2ojL(#9P zV45&M#^*Z=M{8vJ;Mph1xedo^kX_y~H*%pBeh;KRl=#(;eK+9li^djAc)ZV3lIV#{ zg(9C4N22hp2+ABVse-23A2v%*+OXR~Y^N%LjNK9p`ii&n;GxmhcjUTue19r9r;wlm z0l9p(A-4*kiB`*A-ntccwvG#5I@*ANKA%&fbgn>}G|5c!Kra}a-tu1et3D3)MLUv2 z>QVcs9unT3mg0S1hd#a*nR(I%KKh{UeN< z8WRT~nD?j8r9B%_+Jk1bk**0j%s6&-9m_=nKG)t(L3-%D&%CzFx*xvN&DL*uLI#nR zyU)(awPUH+T*%p_0bF)IS3Z!g0+ng~!`q*ggGu$r-j4DDxSbowEW-&O(0m=-ERU0Za>~y&K792)~Gx3qaWhCJWEgKcR-3Cdl(I&3R~WDn$Tsm zVzJ89i669nxHvv>(rT3q115{hudX1-Nw`IdQE0y@1-nXAK6riV0**VzTF;`X`KlV% z)&pu4FzlWqG|^TEbN;QG;@>F96U{|nwKQx}wA2{aM4U305nrBcScCAsC$9^CLnHzF%^P|2meVAcn&3#BYFSE$}y%L+V z&8v3k)IxYxaYE67e)yCD8)jphz%Jred5do%yq(eFhP4)TfKRe0DV`{|H^RyY`KU+rdF_3YdbI``U zAqLIHKf7Jqeh$wpf0Gp|sfJ78?_Ss>P*D85wDp6_$zWuX86Vrw17{;>rgQu#aBR0g z`;qVvI4hd_xviaoyuTKX7q}#YvV@j4_qi69z`66d)Eh28i6OfOqBlPDv9^f*&VQ`*chMjPP>? zU(ZOzH7%O{AsPyPrN)If^|G-1<&UqDt>LI(qBQo)rwO(k&Pg7TqhPG%sX<%qdR*PT zX?iFr4E?0R^~MD;^i=gWpKR=aeR%XKZDKa8scG##YDR=AGeHSy|86*Qsr=KndnDX= zWiLAFxMQ39;^_+MeE6|wIDM$Q5I<35Y_^jUA!VCvrL;E*VnV)b|6$q)RwFHiu}s5| z$WD3DOT`y&Fr&X(kP_Te8J`PNuSV2q*f0M*6#e%PFo-iEd@lU6cE#oadZ=|Y-s2<# z(a3>iiaQ!+1XuLa%X6TXT(dklNWxvpub}UX9t3jbCMq(w!GMij$h(zhXj2fU8)Auq z!B3ZsRI3L-THyQq4O0@hiGE+4 z1#H;4BT2`!3r4hr*-~G(<5E+Pd*^Z*GJf64ui4N5b-U)uRjIiCB+7sIg8KlxTRnek zL~%DRr3i|cx>dmOxv=Grt#OdJAnnVeqJhMM(p+~=G8!jYD&-R%BHQ^5A`DD@NH9p& z_%q)Hefn1-AJitn#9vx_%BtHjX1ma)SuZj4iuX>Mxz-6fq38M3%kt46P};dBvKP`K z#pDw1Bm#k9Vv?dl1{+1zi$seatm4f6ZZDsK;>!==V=Wb@LrIsj*IMx736uWvn0z#n zQ`1})2*$z4(QElh9axx68uHIGhb|pHj>9dH@X2gH0^HR-6*AStVo8{>4Y8Htopn;*r-8S`C8P>>7VnKZIjOEH5g5 z*~2BRK6Ac+T<8#UC+bM{0)N0{yUeC`G<>EYF`0H3R9c3J*i@p4;!+N4p{TlJUe# zv8Ky{U7+KlbRagf3Uj|u`Wgnzfz9ho%9?LG#;$yvRkm+N9iMo*x$HW)=KM-=A&&$+ z)m+v1Ov_WaPF4=NuE`}$@2bV$b9YLg-LC}J-4Cz#Tp+_9u^kF7zuNH6YvpF9 z##&Gmm%4TPTOU-u{Ug*mNCRf-RZm>2$T-k7DWz)K3NPJ$mCzLTg5e2HiwM?X(7Rt! z+;gr44L0*n5iU@9?`zKu??bKl^IE)1eP$cJUf_8;CRhcfRcc%f=PNM#9Nqp-Wq@wU zBzB*=Y}9KLlQ_SX6<^Tuy#CSNg0Vk?&U*WH!b0dXx{O=^F5RK}o=g&avw1yN+folp z3(Aad)>@$O$uR3`O)$#s{$$uOLxJ8$8e8JuwIBz}^IDJa^PqEiIeyW$9QSnmVR|Lo z1UmcA1sY!&gvV!vZrV|5P|x6gx7pl``*f3X53u*4(vo+v zSz8j&Y7ZPAoNUBWSzSkep1XLsL+i$;kV@d1I*?uAn|J`#i;`C|$P?okaL2CZ`-X&GwD|V>+GckW3S2H* zb6{^mc^B;s0v1h3xHS6b_d+`y?$wQX=1qoU{FVvyR9?RI+af@>ya;YoM#1y#L_Gdz ztNiFYGG6%Ukp24&HJ^$h46*bF;?FA!w|e*2z@jD-zn6I={yyotTW^0Tjx^>%v2!ul zDx5Utr+tya5Zk5?~u> zYx{Y&LbZ) z^uynmD|DcLj@+5@eJRL&`LvJ}b02Os_|x%Fpc!2CF9&U=<_S_;4!igr&V?IufB27W zDZ)_m+2v;^Gf}FAFcw3tn=8LIzWT_Ugq)17r|w-QAdzAGN?1=7l59y_J8yA9$lIBlE~&4yoHt5sxbezZM4 zplN8Q0aW-_>7CwIhYv0#S(nl@!_Mp(xN3P1ULGyYaI?7x4VN$KM*HOh*WIm`N#96F zo2);5Or{JK|Li%GMNYzc`9spg-FcWL>SiQYKtZhz52ak^GVl-@80t@{gQ~W>zg#8t zaKz!APT3i1ol5z#r2e-sthqTWXG}_mU;&v1(XLE2esyT8N7@9gwEfh2NVD!u$%5c! zq|F;ZS%G1ABN%C*Qd|YcWZxPX03GCnKG)+uuZJZh2{5$k0s8)-S+$_87-;m>z=qBd zdOiHcQWfulfZ3N(;$j`%|4<-G=re))4-6O%I&{GSx;yb-Z`Wh+X%+Q=nn8@TZY_&9 z9YRXF^ZCKlR%*V?!Z++h{rV=fX#3G+_biI;>4MW>FjIS@bIw8Hy@RJdb$gu_DpWQKZbB~pVJXP%{>!s5RC+OuK&6Hvu+@`6Oi755q;t zBmI<8GD-(!sEZA?L*Z0x+Man4+!xloZ|qVvSP%Q%lMZeMvJ-jN?=6jJH2q9|s^cc) zU2kS6yHJ5*(?;=|PPW3x!L@x}EXBapA7y5GBnxKF?bf|GUy4_(t7l`?8sLG%9dpuf z5wysh)Yo{^iM?yChH(NU+`3R2XBFCj>^py0NZo0Jp_7VlMrFumt zQ>*dr_It<9u(iS@W-nD*RzrBQ7EJ&(59S)H6g&Ji7t;VQR@MF>eMdl1LgdV zldTzzxVrdY%kt=bsQ(=B%kNJCwpi6W`@J55YaC&J1_=?+t`QT%rceU(N!kY_8)Hy% zXudmTt_Z%?nl21&ZGj5u6}vqXm7rka;&hF37=Dk7dMWN~hF_%Q#9e$lRSls@F{^x!lRC9cHEwAm)&qb>?%Q`9*qO{;vzz=}BGzgDYutPKwz^SCYD zOM*>=BztYKV*D(0GV<^!1?irt^Z(l41=DvOp3oJSgNuCF<;1;%K(K+#Y3eYn-SyZ2-gBd0@0o{HoudqvOW@N2L-LDeOHwCL<$IHH3a(ff7 z!pvqvXJRzOuU+tRszB6d|IyzvQ-}KuS$(4%+M#Y~fvL#44t+yUCw&`x03y)ZT^bjP zu=Ju#Pp%QipF5vmElG#rf{M%M-J@aiE5};NQW8#xFL3x058;-QKEw1^@=z_Mc{W10 z4TY+tWgP31k?&%mW6B46l)Yp>SC^FlsR8mYZ>nd* zoFCy8iQURFGwEQngRLt2cNX;J4sx{A*WoII>F3T(_K?eJ?yTS!gAz%nPiTFj);%TO zb+}<0wpsQhFJ+d)Rmt7#eAN08-Fqv!Kv@!+8vE;I7!Sh@w^P@ZPf*~_=18mTlWp)+ z@Hu@@TM^2AEJ&19y9E!+&rvQ1SHPlz8$o|WAIe|vZENB}ctq!UOVV^ac<-Ovzpaau-|X$ucz!6;Evqn%HWO;43HNKOZm}?=^>#((*EVR$=*BF zSCRxNzpvgW{^-UNWzRln%}~eHeN4amvkjhwML&=%sl#2FfnvT)To{u}-t-{39qB8U zcAXg}LI$5Or^x1dxG*w&DmC*qWS43N8g6N(=6Pn}gZtW{Z(~$rrhF%)t9R}%_(=gL zJ;p0E)c8@}B)av@at*Tb1I_I;ecYEe5)eV?gM7VxVGg{Nu(ejI&5If@9$C?rnEiJ4be&rc9GA(sb!=}N#(idIl8R`Ak*DcvPjjwgx(3O9 zo~;`>m-jxocxV_tUC%7wGOb4vKd<PWkgh=N8=zy9_ z+oPP?Nl0TFc<4b-8}e^+-4#TQQ=FtWdk-}sWPe$-7Qb@`8=|IyE4*t_z4$a`G@fHh5wwD_RBHodZrCf0Y6rM-D!#m#l~2LHms7Osx=}pssygaS-yG1B&bN z8ZqaDmg8e;UfCYE9GOgCi`?U5G^0cLI45%aWqQzEc-@dlBmArlVQfmaZ37W@y*X!a zH?$E|)t>BJY3YE}3H`vv=atz1>c+|J#e9%Wd(-Qc6b~OLHQ)1+3Xr4Mq$kC=1KFQ< zPTXjvKvl^`AF28p-0}Jl7lTGS)Q0Sg=%oaYS=gLenkJ1ng$qKZwD?c=WcY) z623YNY%)_0`2k(G$~$CRUb?d@@!ltY}Yct6M;Y8l&3Y)79o zUsax;O_)yWwQ}no1@~7jMoXq7gO`Lr-xOU04xakmFB40G_M>+y6W-TDRj_i`hJ$3J z1P~QwPCvxkQA5!mZg*h9LRsd(4jqvCt+v&qzYN&B)f2n!AU;lbdwIuTJ7%)kxOn+c zzr*d3s7r2+0<+@a;C``c{OeE%X?xWmaN+#Rj?p|&u~Q6@u*`zDo6fvc5KRShR%)4k z zw_Tv71#ghf_V*pQiFL}5?m`bR(urhPoGF8guOu29seZnt=&*hUzYA6bh5Kk5w&67;0e*cd zZ*Nv(;oA@%jE8@jpLt?c3-?By!h3n@P|cM0A(u$G!}g^ZU|Th$dqEAXpRxMe+og5N4$CzS&U6-TY> z*&T&IQ@DfjlQ|E6l=PifS1!k?Hzan0E7IQh!i_CC7!1@ChD)`sPCLA)m) zRe?qKv#SnXQy5wAECX}7KA9iFk8cId)_3Ki45dPllmQP5aj3&SG?6t84CH3y9gXJaeMv1MH=oUW|ebc%4V`c2Hj@ zatkclG9;0)zJ}9JD>@IF^#7QK=k!6GR@D8)nGPf!x|L~HT@4d~!HV^1qImHrIJZVM zVWZ{Vu#Z9pxGN~vawH}l>5sDbM|uw7c2m3AT8=j0l-<7iJd1>q9U?cryy=DOVWSk< zZ4Y6ZDJ<_9PZ`Xo(`LWilmHS&=MO3jmjQo*Js;O%Ib7`Azjw(>7C)O5_47)SkikJi za9)(UA1Y~){@kuesCl8zTQS~@bDkXC-)Y=XeWrvbHl_~wY|P_-9q+T{Sq$4hPKFNmA&v;kUYwrRafMUO zSl7Kn@7&BKOuthxz(vJHGuN@o7_pmZu6?`kDNP5s36DPNoNh(Jm0XV4=OolvXn1VP zm5pZ~2bpJd)#Hs4dKOZ56144=U#zld#r-RZZkXB+v7=3Pm`%~$F^~bKr3r&$|{7iRDWX}+0uyuTt3Tj6SZns}gFSg;yURy&gDKhFN z9pEoo8AO~uoNyzbglF8}c-I9s!(8(3+E3JdF|G`HtRI?uL7|H=(n?q#Rr)Vye$lH! z8pr;PUR=m{hPA{M?A%MOQ=~h3!c)d)Te)>)(d6Z^sOXl5NBTo#GF5sPS7ijuS5l)q{|-;Eotx3X*lL|9ah?fi$d23WZ64W(L~N3FdzL$n-<=V@5E)SqQV-%edDO7*_(Pu{U!A| zKcgR0_=nRxYclb~&gD&nly=w>QAG{Ns!^b3(?MsVq)h_R!m0Xnr*96PBu)V zG27VRwP9)GBVKF0Rv=_==sN~wI4ZZ3F4e6Vxn2vD%U}e^7VJng+nj;d7E_No9%@2M z3(q9afffL{~ZynDq=6|g31Vg8tw@0s*!6e!Dj@PYp5bdzHpFEI) zl}3i`pY}_E*R#Op_}*3sGu{&8R2q*SD}N+S{D**Y;@p!Hel=LmFkrLAoQ#HfSv6NG z+;B7hW8wFtHYDInxeIS~A+>Unw>hL3ABO9jNXjF4sjBXg{XjvTD1EMLIY}t0Fxaa* zT?>4@s~;kE4#TZA`G{kIh?Pn|xf-965e3>mH0ETW(%BfD`ornS`KtV4W+-((UqEU_ zib4%M6^?Cmkmy6)6!)b(e({Z3_u-Q%AX#A2S)uAuez*qebvQZ#I7Z z+%m-fC=nLJn5Ij})fx!7`aS?EywI#%bE0V9Tn+ ztKZY??+@HX$GObwvJMS6VK}3^<#iEiOa^{DALob0o15=UqdQkQONR5au#eR|4Xck@P20|s@m$v~rZQ^1G`_D; zMq5r79y6;PId_Z%FIoa-SAUZ6KHV`t(cnIevF-Wzh&dj6&f@8ov`pN4`+4|W4hghp zKD3iXTR>!w?$>X59ng_4V=c1*@wbO?_=v_~oYHs~wlA2vfA;X7K?%kR$S`*EXx~k( zpB;Pg-p>C#wvBLZEElZCbCHr$zBv>KdN43?!6pX4`|7cQ@p6+HbZr{V@(q;;!>p=^U(7G*w=V5 zQ)w1q`-qr(rKL5znu@!}#Lxh|62B>_HmQ(JcP}JyOFNQky6xoz7h3g~uj!uNSd9md zh!^aCSC9GPN7+9v?nFEHoAo^jM0o4!Yc-xzg|{nXU+OC9A?$w!v~uXXp9!@dx7TZqwWS?i-<^7p_PGp-O?On)-pE4JSsfL;y#{{@2C?`~Nxfq{WiqOBa zpkRYlat~W1bssic+P;pgW~fsVj*u`+!-%fI`q%VC(A=Lmd|v7?IQNMZJS8STT-Cc` zifB7(tXe%95~Y5xKkya`8pzN-u-27F`w(XYEIOx4E?~?4gs*ob4G^<@__zwqHcVD47s}hA3tIQvxXP6ZU~i7frj2&g{VE=w?n;}oz*#zkY{^rD zer*DG%pVWJm6lwIa^^a4kT7wZ;f@14A5KO-av5BGJgY0mc?jmc{8x^zQtN#NXCBub z?7{N?hpzVy=Q@1f#!(?jNh%S_2vLY+y>4X7$Sxy$WF#vovPJgZ*^#~XecOBQy&?)} zh@$j+eV*qzzP~=d-|_r=9(Rtzk?Vb)*Lhvn`DQ2lK4l$^9#GxbJrUf9 zMId&?CrA84I$|M?v|1o81+PngrtGD)1Knv~jktWAj|au&&GM>{y7BmALn8?o6K%d6 z@~j7ZE9n*Ah^Pfy#&QX4_6dNp?d3u@)4PzJQTRq`o`b%>zL<(Kl?G}Q1b z@<9lGhw)VErYyyG2$DwE^;f1f(OT1j_B2%pk~T24mmcXy%nZ#Z$CSE|eKXzV?X6aH zx`ITh65k6=c=Pl6Aa5ITy{@;}K`smUxiq}4SOuW3KCehVmUe+Hf(^|GaxD~}$xh#I zN(&lCPx1U(tpe{o%bP%L6JQRA#4h4~#hA>qhM&1zh%>T5!`H71F%MG^*xU6Z?{jg7 z^nQhC6rTzMP75si99t3xdQ->W*#Tu1fNO67c|FJaXwuN3#j| z7Mj`dP@x{gc*{6+N@;>2mJ^N(p4rH3Pd__sBMJB_PX~U!)C^7#ayoL!HKJ(tdS>&m zI&l0K376HEEOa!cO5cz*45e(H)s`(2 zZH666nC!HmO%_2SeV##JmEt{Cwv>+dP(! z4Vv?>RG{m_Nk6h54*~Vut%CRY1E_YPzT)7#0UD5}q0cITh`*&V+;T1v{l3iD=p=I) zTnJ*qg7_E8Hk!ew~KL!5?U_tUs$B=m#gFQXf1L zO+`d(x|I4%sVGADD^*`gH{f>Jk!$tEEx(U^ACFz{K*8#9=EVc;DCWABUx`K=_-MIj zsoaG?i3G|?zk;s@e2d_U&Taukuix{9=yZeo4=#6+P_>~DonqF|&^nMc6`B{S(2qP^ z!;e%;)nn40O9Z0yb?DjxeCUg{B6i2$Ol(bvqw(zlm~z}Xp}rn9m`AVy-J)>AhD@r|EbF}vo6Bzqi|?mp-Me(QUT zGrGOtNYR9$v40NY_1xqU2xFkw!Y$bGuy}jOCzQR(DxtjV7R-%XITa zR~^bx^XjXW%SMMNq~cSPDw1IS9oRI~hQ6<6aQss90@2>$6@hzQ=kkrye}wS_ zGcq5mN34vMs<%ALf&Ki2_UqJ*K-e$u_M3+tK-cv`!l6tfGN}mq+0s@Fj($H_cG2lU zTTO}rv=$w>bay`{;%|>y?r_dN!CSno!1JnYr#l2HIHD)xE$s63q7Ui zuLC1}*Ml!zDMrr1rwF0(AQ(&1PFVYuhMEaaaEmm*1Csk8qpK@q*?_7^76Ww7iVL^m)7_Xzy+tC)gr$V_co+cMp(OY&j0-cH8454{etlezi06)m2#{+?(00+1_QA!xo>3}$5&SCe8I!Rx48-D3xy z;1UC|e4Ju6YUS^Krd?Kw__ipOLk_q-aJmk7)e! zqKd&-r-o8T=v$<9@o5B|UkQp-%9$9}ZbLsmu~d1y?E%fFKgDiFt-!0%_otOFWuiaF zZn&gg9t84NOE}lXhXB)e62>U$2GC(B7a&Lf8d(d(7nDM?y*fYg?^JLQ4nGJbHO#YB~knLN5lwsM|qw{$F*! zxhgbxMyb#2(lA(AG7TTu8V1TmJqf%d!$6GleOKnGO4M?^utNmk)W}J@TDLO*?zNo? zdXWav+eg2D+{I7fq@Eol{7%S4GlrkbpH}1}mxnKj(v>Sfz~$&UCbD9r9@iOkC8z*3 z-g8jW$WBH#1dA@QB|}s+wpr0(J_t1S#0St@d?o8X^<_nlL2yQs_|N%zTsnVASz&6A zLve}i4>i~E6*Vfx+Gx2B6l$xN5bmr3SY7tDH``vKc{?v(^O-u3eouELap8W=l|2^WXF2b@|{w6l03Vh~c?!|0^!)|907n2Q2-Eau|;g z5G4J$IHoe%hAzL5E2Yd$2aZ<~o@UA9qHkO!2i{Y~NcGIAFYd|%DETsbOYVLXIPy_L znn$YyjlL%p{FYINCU=-GGZu6MDShJFj2`?dH$1@Ac?X}aSl_+38r_PxyvCZXT?SFx z*YJSHIAyFS)u!*Hb)sv0Drs38-oRYQS&mz{9X&Pt?i6k_1p03=R-SOJMBKiahoTAT zXl5~^p8HWecsp|Sn{8MLVkFW$QP$Q48gqyj)$WI)n+}GphI;v^<;%GWCi_}2C`CGZ z&(s%*g@*G|(l(*HZz#;E^&1gyjoLShlik4lf$QDK>H%5AGZ^Rv?ef^?8p}3dNfDx+Mbw8lH#$1l zi+jLW=HSU^Z%a{viOtAm)k3hPnnTb0x&r)?x@cmTR*yc=9F?9r(ui7_Z**BI#UqcP zCc`ktZuD2`wZ0}nA-M8>3yG~%B4E#3^4K>Lyt^TOmGF2iO2aDi8q$iv{h>v5f|OF^ zPISNRqE!{j;ily1w5vwN`uhIudY#~|dj5Hj?Go^AN-u7JtqdI3@hZ|YOhX^rU6gu% zwSu|H=?qrg&xo~$nTr){tK~q z>$)^W{P-7Qd@`x2)VA|4G+o9?#6j?15T{+mK&JTje_yFLSnGxaeESy~KFE$UpZxkS zM6wpqe<5@CvE7oK z&3~cEbH5~iW^eopN&72fpzzbb&{5w+XN&re|3VqthO&l(>;FO#o=fl1#@fG-=%Y(= ztW^I6Y2-h>ULpSB-&Zr4!fdt9tN%hqQBm1dB`g0zzSml^o_v4*FC=(eiL_Jrzo3A2 z3ICgc%m2QjQS*D;JHPZVq^B4gnkDpK5a4{H5TCsG@2g#7(Kp`2{{_8zl3n_7WB%V) zMTdeu+hcS8LYn?=HL~5a|3X2GI$O^>Xa0p;{r&XRhNl08c!TBy@|ULmg*c?&sW$FS z{tLaZD3@5H~a>;#j7Kp ze&oL(*3br?S8^l&zA`0PUi}a={4Zn?Ph{A+G4wBV^3~#{CYk?&ZY=$ox%Y1H-&b08 z85KTM{{TLR- zNW_Q6{|S%9vpw!othcg5rOIBQ41P_lMwnPcx#A zB{%-hZ$ke+v5giC{}Zcd!TA5iDH7tp0-k#Ee}3-&H#qPA#b#$Qm?l1@M`_4*a;_#;@+G_6&c8;%NJeXAw) zQw6IbYrc+^VF;thqace^fDat5WMyJ)aQT<_rp;j`EWYNc8?r0}WYlldpYF2(FWsMV ze5PE&K61WNYOl`A$L)t?m9Zt}Red_B%l?!9^h5 zS-*MqX*xVdmrZr?u`#ffiy3M!uz^?jBc3Q#+M_osjSijPe1SYjh<@0=2nU>bP8Cjd zK$`=9!jthmkTib&U{NL<>GVWs)R6=uMgI4ATu>k)V0kO@q{SPlhdgQ}j-JH6Y<$r4 zDRD-{ksgXacQrAyqrp8Cxz$iExbu1ei#Yf+x5-vpQ;d<@k8*)u>>!>+@_JOmD{vt) zzFq655cti<9p0+k32Q=;qww)|OvxfX%k}*de7VtA;~hN)z1}q&^m^GtOy%QyLfb@4 zrceH^9U~7|91*xF`{E|t>2)Pwrw9Z^7UsE5hetua>OETCx(Jj~P`GlKH4Ik+x(XK? z#X-mLm)~0_08mULNn~UjgQbDDYR&U)&WUIk*mZ%T06{%C-w#p}BWUtgQi>W{9wm-76 z0GmR+$J<2Y!Nmuodk2r*Q5JbRG{#x|>9iXx8VWPFcE7nI9!c6k*9pdq=APv5ot(5Dnl#)};J zaJf{jN1?<9zl#6ppAS7?rlxeI?B!)hIxpO>aoz=ahc0zdkclC}?s+Mq=^=Qp)HTz< z+y<<1Ci7mCoq~b4>*8WGY~aR>##Mqn1L&JoW@`Lo9xA3kI8*R09LW(f{gKE~1sZHu zE7GgNkoroD4C~%3tR}lh##=ZIVnp{_rG`+h{%jeA6w&VKziCY|J;eb4ojWgT9~q((aV}D z$M5r&SV!Lw>l)KEw#xE!-Dg@1yv}_6eq(4J7B0WeZN762{Cu~mFoxHM%ZbJW6=oqo zCpmSDT3{9iva{HAbwfDKpynF!vK_1WW<)IbdjjTvI!|`vtPOVV$saOpu38vz&ER%b zGB@x{%yF@$7{%JUx|scl1A*3#_XYA_$H6Z%QjEGb0$n-gQ5!-28aAok9{J5C0oc1& z^B8}K0(tQ{%{GvSk=$YUcsIia$u^mO^1|zb^Fa93E&6CkxmKKLz4;gOnP_6F9A3h_ zT-hs}qdX9q`;%QaK4)-Sdh#=;WC_fSzMl1)xNf_@O zJZ++6F&OR!5;_*Y#?dH(B`0AyJ9%A@wcdKFqa*+wk4(>CQ^&83LJxXahvQ++`URV@ z1RG#~tDg?@>VO(nkz|{X?_vq7iyIi73u-@8-_t28hV1!G&el1O!Y{vyZQ`Ee^-=RO zX3FOg#?O7m^$g1(tgWU$>iFxYsIq2GxzEW(XqL6lYFrqO)3scF5K zi%D+Rw0=88dv&WkY^M@(bH};JBS3vCD>y#Ee)kQ&HU0ya% zAO3ox*eHhA(KHKr9FNH;R!!g@bWb@RU9t%AzU z%98WrL(tEc?@n~I4NBWjuolAWLsX%5r|MS+cJj=+s!A*o(Emjs#8h|>W^Miw5lHqx zF-mus&s=c^27}%C!BPv*B+z+ZwK5KB_Qq;>|(#E0x{ zXF0he&fAM@bdLR)4kO$6hGQ~nQxJ&>c`^)>1l1Xu@cQ^FFpJff-4`4>2UxVcq7-IsJm0UsjOHLT*$@6h41qZ9P7>&O=6HJm>f z=qWzt`YVDqOR~T5w{=0jQ(})megI;4VM#-$;fIDZ$$aB!<}lgCocHM< z2BjZyN{769pN%8ftKkpA?$SmQArKKKwu0e&*8YTrxKl?3Q;mGetl~rkqV+l5iOx-7 z0$axHEI5Azw{cqhN*;#8Z)oh+9F|~;q5&26nR6hqwVyF?g&MkCGg7beGX@KMSy!HI z%wvX&)YzADdz6ryY9;^G7cA}z-*XLHz`83=3etgmOq#NX%UVemi*`L;6d)ar+E3;a zI1vY<#fuhP1vq~goR6Mh!|P-5^*vgLu`x`Uo1-f>#~Fo+Lh&xSFnA^ZJKXb`h4U$D z0X^2@poNZ>)q1KI3SBNwkEh@O9#LP=HS`LoJ|`K~NfH9rLJlggF15qWXPu5g9W9s| z_X)%)6~ z+9PPE7ZR(NpYtSS#O7m)OVkrEfSDL6yeU>I}$`<&J{$T{=y{><-bEU@U|F^cI4NXTB=+Hr;+ z_}`nY!zf0u$LL$auwfXGZHWczF9|?^NbbrxYb)d>8AF&jGYw_Uq>obUL}3(K>kag0 zNWmnN_Sw&(O>pdv^l8nz(I|#8=MgK(GPG^z(o*~82qJcSSE-yPU;r1Fq+oh9(k7}J zvE!bH2dtv)b}oaku;jaEKU9eY@r#>Dl2uDpTHo<<~ zityLD>nlcU6Oid}I7nmk5-7Y|700(t1-nmtIaIRuMlmq)-6{5yU@DaD3r&+bw&{V2XFPIf7Zd0Q5VV_6yhB5p{;|WAoy`L948X%fK^QhAeKIwMAm(vh~Q2VNPE zV0>)o3145{k51nRN9)ZWLblngLHH-$`}*-q@Kwrfwf3i~D3MC*)%hL|)NqQo)NPdq zd}pxvs5sk7B-mGD4Qcd-yJpI^c zSm>lnyAVL1r0t~#Z9oo9|I4Ch-mttwXrxPw3VbP+`K&@<2v4ez(neg=M#FwJ@D%L; zOp>Qh`*C9olPCPF*ysgbYGm01~s%aLdL)h(x>@|#Uw?M{1Cnwfre zbjug0vea-=pK-zlc#_P>^M)}pA=4Yjx|bl0uzl|_Twkp^jgEfA`;W%ncE2Q?Km0tl zKw+~t+DqBFHn6z_nF>z}I=pj6V-q8s;6pR~_MtRy)~poDebzf$bxRzWMtwY+lQ)UY zT-{6K+~fcvj#a1WYhQuq%)dm0sf2-JgnoAY?-^(&>GgiMU>5tSRE*{*N4G@q3m6ttLEG98<X>9DyBM1GW1`TxTZ&C}(ixYZO+wif^q=hB%wm5FK74WE>%nYHtT$eYFoISZ_EqbT z^H4JKTff5_b#zztD)$Jf8jvNw+AhI_UrFZZ-92%031*mZU23Olgy^&wof`XbU|fCe zS@9xp`I83Y+AmS8uZx3bnVd-Qw5MXv?l z@=Bn6#U$#`0ZX5I%h%}=fn&*;lH@Pyuy4=yTrb*$BNhoRt^PVmpcr)GuxY^vy}3U2 z?ZtKiip&>}cGa1N>ymx(x5T26%i;px`s+N{0#EnRrYyt1YNyYKD!l@|!jltaETtH6 zjlr&YwhmBN4yV3690_jwG(Bkbbw*F-JumcD*F(*VZ!?#USHWRn*N?h(i}34go>6<> zBFrr5AYqaTL}s#V-is-Rn5W5E1LG8K(69G6lP9qaGv5E|r0&xKh1xHvG|%s9>N>xIb3zW0tiKMJZ=W=W=v0#K>n!JK=%J2HT4 z3Ib9JScb35%VAqW@Kt*2y4huMmUAX!g4=qy(bMcO z3((otYqa$A6jT{k{;R{`1xlFYDvxr>gDd?r!zWIM1LY%Xu=KVKOcRmlAe2u){_Jbv zPmnWem}~TF`Rs^h%tOuUCae%8<@;+n`YfP7LSOPWS0KV;p?t_iRN%?K3J>d~(~wD8 zjFO)xn2TF1d@Z zjf9seyh?FDjo>v&YOW2&MA~6;q~I)2aoZ7zq@2W3QL<|IcP+$Z&ETmWF#;RiFAQlW zhGIttqq8S3p9j(-X4~ssldz~x!E)Qg7`zJzrnntak3F@4dITc<#m79QP2Olt z-p@NiaT4;z7jE57aYjTJ+=gO1OR;Z8oXl?fBbdKq>A~Cc;^6kkua_fveOSKbdLn}? zCwR4{kbMch0(SPqGUevN;2e?C{MPvq%;(vCjhC$mGc77fj5KY-`^(R3*31i-7FG+9 zeXBH%;nQ=fzPAI4F7 zFJF@_Vq*d=&d=l}z<25Ey*2X~o@!xP?E0w!TKtNpKha`?JnkM7y5A9tG(&=Z%(TPIqMT^o0_?MfsaWs(XV=5kIq;QhH)|C?7c9!nVe zv%ryKc>i%b*Qnt$&L08|=o9}UD{!>)XA*tyhF*ev2jlZv*lslO3>nTJ*>Mq?a&pe7 zHY7S_#}?n>Rf$kP`m-DUWE83(h;;*_^>X$SY>I%HIro_ZwJz|;xM^vC_a95M82F-s zr!iG<5A8(ZKJKwNW6n4meAQ{Jdx~WdEBHRR>HDY|`#M7IH+97Y$qCBdJWC;l!oTT! z86^d;diMY?(3=1h*CDA^_$8voH0$PVQ6cBQ38U; z$W*L>Z=z7UYYQbb7Aw^$nOTuTTPMov4(q8-5^|>dB3e?lFw- z*F~ozm%n4@Qlje6n_>v9e2feGt{~Bep)UgNhX8uXD08gI7q}g5^Z#=q91+OmUSkyP zgc2FFSO2o(*G@z}1ce{wFqMecMl~rdSfKxTCPM*#psF$KX!Ar5o4I^9hzL*3>>zyH zwmjel?i}xP_$cCn#9k6wO6~y33^!#ax$eV|-Mw{T~ zVb)mqHP(B{2qZ(&z&8&B=UyZ4X#SFV+F*1=M?~uh{Rd3***CtE4dg&~T=zFsdklIL z_QqJ$D*-JVWmb`YPDV~ae0?`)$2g9wCdJ`#nrnmSg z8848?^t{Sn>K69!aDDR~hXwpXa(sTu+8OCw;(e9PuME3C%3LCTn1GoUmK}zF3P&ok zPA&eN)?lmmwgJOH1ZH`h(mK&a713(D6_AX0py!#7`TXX%0YBYp38ASW5GdC7v@&}E zS`$x;TrmHE-Tq-!LaY>nTv#V&UWazV51-uPoB=pzMe@v3cfyyfr)CSEs-qjn zx%V$!Rs$V-`BkS_qR_hjiLI<7BUr`)H3gON60|dRd!+R875|X;Hld0 zx~n$e!=D_A+f3wOZAa5I3D;KzRFfN^qYC48mQ3b#vqyK&$xt$!i~)yiOmn8PqtLd- zpnMolVGPoAo)BJ{!QS;)Z9TLNN11&lT&*XiKzCzxPTG_aiX7ruNcfn5q-CSo7H=&= z!Zl;PP|;}g(>9g@WE*1Dn8!15gJFz5PwkY|y;p!Q+4{I2>jWfyUy(WPuLH7#G9sT2 zM1ni1Ws9Y+oDun3HVPw&Zp`a^cB096Gxj5|NUA-z7qS?gLT{gqW7P7s27|c1q9=J8 zqk3`&Yt;8zyvfUrr?p76uosQPO_c|`dqgL&JfR*QdAm^gH?^ADYL=qT$#bb05;|m`@@@=aXj-KB@*yRzZ zey2!Gh!yY8%QgY{EC-_bzg!xj2nU6EM<2-02VtFz#FhBrIeK>7Iy4$Oqmc8w&SKvk zk%R#|N8Y>@Vx<%|5Y}b^(jr73j93HF`kf%F?xKEdTXC){0-t|qzH9W_#`TpcA@yzk z^(Yk0bBjK-ath)Xfr1pDUGUUf5$l0HH&jPyyIY0ptIq{}VyBdS5dl}fcJp03l)+9^ zK4G$qMKwzY|A~u0=iAphOPsx7Wh5`~d6t9GJ#|d1!&i_qqq|8^tRUKFBCys{sfAfx zT90`;aR2J3WHNut4E%h`P+nJK3{FbQE*JPRf+yFkPkC&NLaE;4Ukr`HK;5-m**u?J z?E8xsS;jbjU@KwO;suQut$82YJ^ONuDLtr}o8br$Q~s55-|jL-UCx~nrxcBLEAQj? zn{zPW()Hgr4jjRS^n19!!&*n)LZvbBbsw3FxIgvj#y# z?DraM$-w)74}Fiv{E+%*VkLo>Em-(*)eMlSl@lMX=}=WOW22 zWa!4Qu`R>6`!DB&jx53y^XPW6qHt8QrE0ZPVh^8|4(MCnUWR|18jr~h(t)yM2aU?|)wq9^!bV`fNFR*2KA!vVbafI-8E$+O>`M&1^;U^>hrJ;~mK*vpH|e4bal zz)wUWcMhn`@96OWZ4V})YvG1KJu6n7{NW3rd$^=@fjkr2 zOg~3#s~CfN^~ttkq~>7F+%(tB&OH3NKkSloFA5#qj=N_?bO{jKzh#PM&V|9GckeSg z*?@zyUp~w~Zo(`mzK2*}(82ci>6{!)hA@Vomd8j8A?9Bsa_3lHINFuD93on&3g}-) zWxlKkL)V@*&)!X%fHzOy^V|+tgopb1bFLjDuz%fUJdi~Qe5)u)okBK%RAR5}>WKyT z>2io6OB*$KasQ`th}Q%RLCH~SN3@Y|<7t`<)*;Bbnd~8%+krW|MiD>AJ`XHJ>5UTw zJE7%m!cWnb#vqK5LfLhF33J`xseaUk`&YSwo-+7;$&0&)cfnu@o;kOF%anKo{yu&5 zugla7BvVO|*pv%Lk-ih!HOGUIympwhlUpFlAbt>$2EEZU5&D85G>diIYScf-b4Ksl z_xaO`M&L<*osT-R(~yFmW2#D69PIz-4^1gug!^}gZJc)40oBUZmsxy1=wi?z;NdC^ zTI_OlcMq&Fg|yO~-GwY{TdX{oDs>8b_WG(egUcd}X|CV+H70>|8uYy<6Viee1Cw2k zsCfYEG@Y-h&Il&&ruX&)J|FB4PHea>MhuP%QW%Wk{dtx4)`JzFMeN;k^YGAAaln>7 zE!19&0X4c1Mt1T!tntK}Cw;69;$Tw9`j3BAt0T^KJ1G?N+>tjt@cMy;5c;uMzmA9A zKax|+@%<9}L#LcuOwNEq8&%&d$NRb#FY>+mCAj!z)Jv-y10Q+setKsR2!7fa%B89} zfb$+avq_?wKud0HH3^@y=|3_#V^mNLZwSau_XI3M6NcF%IVl-IJls)4JL9*mq9tf?dBa1%DH-MEPp(FY3`14%u~TOLLC94{uZ(Ea3JnES zv~lIIg8KJg;(z1wLB{&n?%lNyqFuh9!q3X2Ay=7Ol2FVU$&#Py702fvp&w{_4T%(C z`kRUAO9^g(Ge?HV>a`*mb+oFzfzQuRP@W16$O}Nrc|PP8_x;dV(z;nN{S-FCrJuW* zXalrM_@D7m)Iw^dzvdKgt>HfL?46GsF34TyO^+h67&^^#@5Ak%y_n5acFG`JUlCuL zHk2hwf;v(Ov!uGj>$1O;pI9*8r?qJY`ZC*yR=nEL&!_G_oxA@lrH=kbd+fV1E_Yo8T408R#v zPJQ=89MzP}-?{6tSf-B2ZclsEnHSs!d`zkXIEM_nM+YOp-Yebr8^O*<*iucr zB)1czU5VQzw{C-{>n><0sI*`zH+|@qSR3H*Q0ap&(t&8=N!xwZFT2=YUfy$Rygq*G zj^~6dIY1K^{os!I}!4jLs(RUwLG$dbxQa z(chfBNd#nI#lX_M#SGu0uyEU4!Tqb7$A$`}Gx{)4Z6cXucN84I)Vyrx=7jwn#{CEP zUpV;zT{;pQwO8Iqm%>f>oUJdMAA9^^l3*Naa+n@i*y+QkTd5`&u6lt}1Zo$@1LZ)7 zndA5x?q5yQxSdR*oP)ou)Z*-A640XK9WC8TXQc3n7Jk`vL?>JP546Xv(CLNe&&LN@ z!Gk}OGN$Z-=+x^m)yI3rgwF&<+{b9kFS>x57wekqk$@!fz1?5t)=C# z`Q!qa2Zp|E|8~Pq+N7VE2^#<|)&A>5>b^+k{Z?U+v>jTu!YCFf(_vzE1guGnK+kIz z=dTJd!&blQTS`oeFukFAvh+a$D$m(`X~!yr?mXS#O_ZI*h#T5hX`k5uTWnmLX=IMpMirDk7c1An=O3l%yD<>t z8Qlynnua=uTmsS+_UIvB!H)Q*F9@Po@eOvM!(8;2XuMS0;2AoO;+4m9u=hbDk0?I> z(3&Oi{f+ZS{!V>5v3nr;Xw*uuQ16Wh-gDXIBrU?+_^eX{_8||Ff7yw_*YB&MciD5WGrnpb)(IYH@zE{H zI-EZU%4dZsHx{wXuq4PQ-;J5CJ$*sxg8>P%=G}OFzvR&)^@SJs{*{V+bB>pwCiucB zBx;M#KcrqW7Om?p!5fzQBJ&ANaIHLeiTIukR7!yi+!XVWjH;76^R_%_RL{uU#^)d0 zO(XATaG#XsNxf|wKL1dCcgS^9xDq3dF#XW@DhS=PnWg(UX@%JMSknfsUIm$Ji?-q8 zZfMLTA@dX29DF>;@w^uIub%$mRl0}sM@T^Bx6?R(JdQ;)9%tnt^QHueU6dQp7y9;e z_^~3$Xgo{EPN)kCL~fnct_VQ1mya5a;{KI!89&GIwia00{4Ur#$OaflM^J~;0L(eW zy+Czu7+y$joD0MEOBj{UrR5ZgA&Do)%q>67z*jvdYw7U$IcL}XV>yNGP+6s4&Ft$4 zmRw%OlhDQh@@Ykq_gs3h!toae@A3KhdV~2T^6k$Usj^_0lc^Otv#&Ij5ao87&B1_Yesi3M}J|VQY%kRd|rTkXKK!T&Upi!1;Sb8 zO2SdSeW$JAhzg9TT6Wyc>xNgT#y?pRFoN>Y=raD3%TQ_U%7-BrZ)CURP^puB0@O%B zDq-BW_yzb($LYiHd$D`2)mn>iSNNGQzh@JCGvCCLCDjPGP0z7qEWAb%7Z?UO7=jU@ z=MRr2=P}rzu-I_w&2g~A_+UD}D+X~5bgfW&CLjXUv7Z%N_|>;eB85_sH^}Gn>8?1h z3GD7Z_XPS~kZfxp%C!1Dw0013IFcq(_SF;ME07$$Ow$J`n?i%e*xOwIXEKl+!-Au z<*~Y@{SKOaR+?Rj9fzrApFUb{gd?K>R=qd;*5GW@xljr&My&Z0YtxfvRn!?ol`7Tm zfvn~NYzyx40w;g21nNLT@P1P;m(B78SlYXpL@vIAF&s#qZNlg0B7NAOjE+Efj_g)j z+}lDZvUjWOy(+GULiptSp3?xD!EWw6(?yK8B}9q&LJ|_(A&^})Uxb*5V*kXkX81SS z%F=|20bG=Qk$KB#7Arf^q#eWE$S_j-+HSjMJjNI*S@ zp_|Ez6<)`|`=v)a{WC&9FFnD`G%g&y89AV(3)jKS)A-V!;{H`n$?frJsR2w!F7liL zLoD)_O6?XX34&eR<3Z&-!;nO0zcrl41~lf&NJvJW2J7Oz4FaL$-ZY(rmH(Y2FNpb%wg;Z$^F-U>5 z3Et+2c_Z{U^~|WnMgoeMyl;K~%Mi4S+SA^~^;Pk6a_3OCDJ=UV{y|kf7AmUo{Tpia}YYfLj=7JZYllt(RYEeGyDdP}v z4C^rDShA11#Xb&?$&adv;{Go8Gi4bO=R@qzvap+k4ma4zA?rTamcxL(!jLeouT~YD znJzPtf?3i}Zapvp#on)RyCTR4;`Sd|m*0{ASxeI-5fQpASCKJ7KX^ zJcC_a3v52PbquiikLE_;^TCrgMV<7?f#~X4ht*A7Uom(FzwUSS0|h&8K3MyDBME0h zj_C7vUw0{soh@_)`%7cmE;kipbQ6e_es-&9xTA^ zAG|e9xPKLA`qe<7$QcPl4^Dj~a6%kombn=VR_Km)QJJk6Dh25V<88w>J{kKW@}H9jB2`h4YnW+M2CgF&oh!v&y_Kj*|gx*_Xn_wFP2 z3;=fMXgrV42lbz4tY6l!LrM|1SGmN#VU<7F_RVm8)r$6SKPa|=ckh#R5$auslJ7!R zTeK69s!K<@vcDi&Fi6lB5-`Hn^J4DToKM12VmOrKw>q#ROj#sK=}WNa)J5f~*9<_T zrm6kg;V`uCy20;itB#`dBBlP2tAYNnAF1^5`j|Mkn3k{Chuz}fxy^OwGWPV^tMx*; zqu?eDg_U})AXo{Ts;0*KkGOM!0@R!MUJlC>Oi0xUI8h%&pYa;R0t$*IY#3uvN6dw9 z8#1}@3n|(8G`1uxhAV%kiQ5KGvrW3Yf9(wTK+nr%y5os_&J2{rhBw3DjrUGjPWCAE zf$w1_-hX7tcqS4u&OuVCXA)+_Kd|Y<0@Z0IM(liE{WEs!a1_`n@${aM6u8*fpz;d$ zciTA!+xoW>knM3tLAneZjQI1Vt4BIQG!@2d6g5cO~6hd)krzm_402U{^bdH*PtpMW=88|1e2S z2{w-6+qe9LbFL&1W`o2$!;bk7p2@Xe>i^IUd0BT$m`G?ai*du&6_oEgg#EZ~rgP5~@eE?5v7r}h@ew98=f)zwz^FiGyKG?} z?2TQCyh5!amSR-K6NCK67dEy(jr@)HW8Mc(%u#=SJ>yX5eT6fj@`jhQ&#ohZYX={` zJ@SO$k=5O3)h2*7_lB|`p#FU0_=7R6R}=W<1xd~rq_33Mq!LJ76A86TRNZe-zB>|3 zL-XQ%AKuzMc&&Nf1qUCq;4q8w-KSHB9VGYc2xOgc7XVW; zoITmiX8b0GFdUp>9-mo{AD#^)bCI9K{}$(YD~a%-_986&%ZV%hhjp(1kKIX53t%VR z=%>HT9i@JMSJ;Ekm^g6A-T6T1n6kf6vib>o!IDZ!^dc6oFyJ2>yNV=Mc&?FAP7OYP z?L0<$qOu?+ih9+4R)hr&AcG>3Bo7<1myo*zYwa`&j=7q|sJxRXRV7pD6pjBs(+(wOQYqZBMEZ-qGwvULLzk5K{TqUD5o8 zkP-aCrt{8QY&!I<)b@%p#LZW%G9C`#r>^O|72-A^e8^1r&Omby3w>lD*@)}K`}SV$ za_Jhtmn?gKox530pkTlaGPYL`#+u~3gOjK54l9a3>jXPtO`d(G&w3|5T6!z@N6t_D z^8CdFmKqnryBe|hgnlQB+zDYKL{bPe3)G@Fpeh_^tf;O5z+}hZ~u6tbOsY_hkjge)vLuf ztzFvuP4XU6YSxdZ*-qmpou+KoEIT1jZ!6zTzVCQXYx!5+n~2&nS)o~Z6+vSD(W{@m4*)gmBT}=dhRMy=IC;hIpD;o*U%Jd_5|&+aIB_v! z5-;c&+o7G%2ZIj#z1}6iB$T|5me`SffqmvS+is*Cfl8m0aFr)Y@t2J&AGcxut17Sg z6U3Z~d9B2IjFV>*+$oyu>i*dh4Cja#rDnVdjsqvOlD-6B7Hl*bmz8Vb__SJ93UeI{ znsrGHVrs_o==0mAULJ$2=QAxdLI?4~T55M0Y#Z?o%pBuoie@l_%XB=TWeP7qLopQ) zn?@ip<~DLK&cY(|l4r;!dA=d9Xg04>67z5%HdDSq?!>ItACD z_<_*8RyWO-I)OiJEW5L`(gOu2oy~X(*$7?H6PdSZxClCFV>JEjafBZvA0kH|CSyYr zE3!wfGz9J1w%gTxB-m;`nd)7G4}{i5&(l)}353^ba$ruNm{70QD7tYjosg8T6|PS5 z5i(Rvzv6VR!G{V9=`6k;!3XjrrFXFZ!aw=2VSOrc5dOIAexAFf4K0o8b)nM8Cycha zVSg&0gkz@R^0OTl71#(CBmjC37)OgaKXc0u1nVjTNj|1;>=1Ra0_tQ9uvwXSe~ZipZ=H&!+J|LMGE@x-N&ZMSIax_ z>Xi#$LcK@vR~s~!?x>AH<%0d8OhPpbAn`dIjkh4UGn9^qmk8)sTaAz}QVdQX>B5kawZ63jZFEfV-EV-%)`@k8Qj{HYTN zze(@dK2#SGJ}j?58OgVVa?b6L$#b6w1gYb4ZzPkkpximhjv!-#?_4=8!*zWEM;+@U zS;h>)l#Ko9_U3$oY-KZRy~%6BO^=$G9ufA^P8s>yHhn@{vaS@BY85Su^Q`95pxg-~ID~ zRe_caZM4V5Oku;{JD7j1`F<4AX#YL`-GX@01K|L^|4>oiN~niG#T*7fHzYr`En>^^VJjM`SoA5vHS59R!@Mq*N{O^540~Bt}BMs2*ZP z8tuQ=J>0-cy~N`G=l;KsQ~ap8Vu5BH|GVeA;&S$tzb?kqk?XI5GGd9e6C^zm&i}8^ z`|tJtUE2ol&TXL?c@3>kH|Wd{pj&rEo8oJU|9Rg;r|4rVISz!VmMyX2o8*|0ya90< z>QMeoW@P$2;=(p+a&6>D(Gg&+@2T$k_#h(DS{_%Sz*+`6TZ%1%2`wqT_w3%H4*q{X zRJMykf94YX&-<^J+}lk>8qKJsb~EFo58>V?8WyJYeptxXV$*L#f!Q=j5}6{6mc`kW z(vulN7@3`17DqFV@uA+llP?0W^K29>H$qTm_`e_AI}CHpa(@5wbABOr>V6N(h?(E> z2UBb-Vs(R)hHSLa|MT@XmPy!IkVaEUjQ`1mG}?EG^`|?39nlFHd*lXa#&I95p|3U< zz^*ur{AouX=D)8)@S=Fdi!$Q+GtyasFB1tS&^L+;X|(^ICk7dhyirEHdA8*5JJi8z z@`U~oMIF3m%VF)Lraxhvoh92$&5n?{7@YU@{46YBS@JuMJj{Qe|KAm#-4|@5`--6N zxa3tmB>R5=(0_Z8#PE}N*$Y$=&dC~8{kn-J{{MfQ0g*WW|5x4rf3J?0>(Gzg(hLA; zCVr*{QVnoGP$&GYbu{>=uiUfcr3W11J_l+g`GbJpw_XVckb{C{zLe>obXZGrISPT`$Fv*u}eSydQk`u;nJkNPlTPb1O$<1LWk3LL4>fTqs(p-yz%UX=(CbqD8Dcg z&+Hrz)QSdvgjfB5-D81Mm#Za!+VS_cdF{Ti``q5^Z^wfGU+N#_9APsc;9^sresv7? zv${uncV7i7vB`aBuA|sBH0N=X3JnlhAARUvQw2}Omh_#^IfW^i9$RuAj{`cdvU4_P ztbxOfu#kM212D~~{>78Sgn8XOP*1)=j?EZ%L=PF1$2WPw`>IpjU#9N^O+)^`$oSt-P`Kg6LlHGF^6-1I>05p1dY<8HW;QGK z`Tgbdjqwy1bK2;8!6qA!|0^n#Cc+AI*kRWr?j?eGp45a7_k97;3DMW3vI5{eFn6za zxB||bv~?^GFko@M&i!gb>$uZwcg2k9B9mu7jnE@rry{{rxjU)4mY^7cuF9lk0|} z+CHkad5Pc-9>zLztit=RP9_;fplMpk=_B)x65w2b@Hc~I1JG|mN-6o#7&^Duh!6LY zVz{`wpVB3A@& zWst;znO2@>Hf4v9CGnmErCc90YvUCUm3sgP=F)SGW-Qosl|OHIETX`1&i>09PmbU$ zJ(ps?Q7mL$UDWm(9D*XT^1cth)Ib%>bg>k1Js=pn?JB?^jWyr6lA5hN1ZOxsNwzPX z#mHoim7Ka|4(yx8u8dZaVX|~jC?QK7l>FtHtfRdR$Fm(uW;OPp9dkjGPx5bgdfGDR zDu*IaDv^}H2joLb?x`7ddSTGert*n(J`wbh_uesX8;4BC@9wOg`v%SA&N!*cyaW#Q z|8D+8__!LkrUT4DhSf-Txpp1}7_T z#nk=~08BZ5wwBe4;CC^)3wSSMuwz_hxZ6|(jk>oZO*2kmMdzn3hu#YYnpBgnhI_rZ zg@HMzY1&TQR)tZnRyGrMa?9yQD#FLnT<5XV2p^mJ;WOJG_aHGP>=1pig5z7n9*{li zh1R^C^idDV0d09;o@HMMc;P7>t{3$i>Jm=i|XL z?)K9N9|;R_%W~}Az^;4Mgy#Vhmg~3uB;{Q>JbNFoUVh1pIW%dXOd6K}c_C2&lG*&& za5i0{kGM}K=@!B^!7{qdl~4TezadD(+&M^rEwXG zCxXawK;Y$Eg`etHyl=c7gmU!_zE|Ycfn{Z-!l%3KIQ6gvWwUN$c;MgucBz9DQ=^zm zAMWpip9`yA*Fs`UTgbYjaCi;MmNv2$OhkgdXPIl4lRQCpwBbxwf))t8ZaS7#5Czn& z7%h$R522$t1JP&cKA7&d^QvgiA5<;oj7pfYV0J}@)MY}kKVEeSq`!JARI9lLW(6P zc_Ruu)^&N-UrvUt1vN~)Vy%O8ez=Kc-EByxC1|^OcMo22@h7E^{SCdf^PNQ$WkH_2 z{Rzp(`A}*nBI(*yYj9PBjV7J-G`O=I`^3Kot&Xkgak|3y4Mv$Rfw4kwK;{|#T}Gh+ zYCSNq^QSrn@KT;$^n1^N@k7H@;~@(iv8C$ZIxZH38EZU}o<#Wgliq#$+A@5_PuB1R z@kehSfwJwBHqdm>)AFOJz==Gey9DoS;yzAtJ$9dJg8U;cmhE}9@QnMBX5)%4p#F3p zpY!bp{2csvUvY>JECjecK4|lWUGl=}ny3-e@u#j6qJIiLbiOtXL>BPGHJei=ir&Dq zSA&D{b`gB>;EWIW5Db#jzNgc*RKdRto9rZ6r?AuXvuD|L4{^sNPjlv1$ARxQ4BmW3 zMqsTz)+`zE#}Mnk2J_qGSOSC%GaLWl4}*#BtA%?|-y0h4GHZd^{=uD1y#hEj+ z{Hw4?f?|3bS;&S=Yu6OL7%=`gInCTd2K+RZQqk)(hTIXhXE(b@v5Q8o_wr}@pzxE{ z$ln+-cE{FJas}bTWhz0N+-(;(qF#C-zbO&2^8PjtRKJ2x7RMX62p|3t)zx2Qm zq}KR2^|7u7&WuUjm`G&>Vo`jhaR?t|Z%9;$wT9r+W<_&M;w-kIG(~CFzJ}wRI9b|? z_=9ahqcBdO4!&`Y4WfCt4d=YYpXsab!Pr}=&c$)R;g?_2zMUdbfTcp3Ugc#zq!ro} zqzFj>?c$NDMBNWSz3yS{YX3Ma{Ip)#i10C!@NGxaF9y_Av_8v_X@GKW<Uyg0f;EwrZnGYfU2;<3onsIv>Dk&SBU~*){mKObASG~0c zv!YQZ6P;Iq-L)Wtg4_4O8$~Wz$@wPuMn)i%?p-Z3ub}gi@S4IMD#_91B7LRXu&s0Q zFbHG@RlOb@^o8nMzns3wc>oRf#+P#13E(x)-EUB4436(A{}y&^##J1>W{c1+f{lak zgnW;pfg4}yf+9LqDCJM9*vsrw7}dCJ?AQDlV06$l{6Tr0Pmk;{ytapB5=x zi3E}fD#9-jK2nEOxL=%f1RQ&W&lV{NAHfD}Pjgr?b*0|Reh44p%~zg2I`tIXNiLbc zU0?=;H^b+Aq7}j0?ds19V#MIHVt1&_;$t9Y$|Dm!>IP=dY!;Lv{=gDXzuNT<0%eZ2 zrF%$U72JF^`ZBv5dY_S(Ge-Pz$?3>)cI7JY_&6P2kMPkU&z2uMN`#T5e)5Wx3HSt8=^@1)4OmRYboIeR29d_NtHc-? zeIKe~pn->c`pMz$tRM+;1vtV1`xFko(A@tN`)AG2|2eVVm zHANDmKvgAs*;R8E?BkB-L$X2*kUX%XPlfcA@-lCmGtyVTTD|Zpy+cq!d0VQzu?Fgv zX*6b>jsmy7rjHvVeN`*$Z$V}-1Ow*R@9`o2XgD|suix|nkCS;{X?`Tb7R-hHoH^@Y z3_;jGUT+&FNgv9wA$?__LJ@T*;WxDFOm|s1%MZR?di|GRoe%p=%-wAM*5Up>lgcUl za~Z@pOkZN_9EXDEIT!rczClhhm1VW+bb)}!Ddx~`_IeZ zmPKt#b8P+zCx55i&rbwaXc7_51H_ zZ*Hv3hG9zhS~R%dG&Yk|=K`pTbl;YLnxJ1_=Yv?uv_w%inZl+hl!cMKJ ziNhVppy|v}^JX7@?$8~reSZRwuGbp9dFl+JqBJE^zSsaMU$39ywBwM|V)f34uNx3g znmggZY6l-Awd@i`GoZKPPq`pNR*ZV+6&>JW$0Q}JXcdyqVROY&;VPoXFxroziT-_o z(ACVomHyu~pgUWfi*1uY54Y1E(dyw~)=iH4djK64F5EiCOce#b-*|5LZ;}@))1K~H z9Ona<2bBVUmc_uo-zu#sCL7`XbgHd*q!Z6N&%Dbh4+kEqTFRLVPeJF0>txi$#qg@P z&qVI11IYgU`Vyb3Fxblc8+^564rY_*{TeQ+gP-jl+26}g06AS)`o)}l$TfjQjm0d$ zq>?^CC(e4975DyIhb0?UbnNF>!=-I_J8OCEet#?+NMpVo_52i8daL$F3c_uw`Qif= zr6iEedD6LUg$sLmay}dOO<7bT6_aNQ(tfdTq-Rnk-Fue7F)c1Rkl==g?TSrO3pW%lY0zOcv1=*S~UPE#W#m9=>Jca z5fky)?I3{N;fX2N)`#Q1TGUH#S+TO7QzZNe6qsqiw4a7_1gNsfE;|++3B;zGU-a|^ zgEK{6c^xlBgUi8|#o^Qs!S}5`)8Tn1kZm4M+=%qmvB8RQ?gCmS?b~#a|48V#@y}z72wRSzy;3P308;X~YfOG-Lz`a8 z8y9c!VX2G9zPjJq!?j$rzr`xu4V%ADG+(}R6SvIK!!nNi$NCe_?#9GHDAoI~E3Ii0 zS3SK&qjI1Q3>~801R(y9c>4WnQ77ULO*T8d;XcTpEcUqx=_{Y@`2?P^HK?#}Hb-id z09w6lI_o@rLFiEtqr+1bu*5}FywD*G)|b24T#-K?;+p&uajg#u=#UEc&1i$8Sb34j zM=aQ>!rRGVn~~sdUE-C0NMD`MeO*N584H`rUaN2S4Z*uFt|U>{*TBb=d0(>dksyxx zYd#C&k9Cp)59wP&Q0Hr~gErDv_(!GD(SPP}TU9JnrHDV)JA%#=bJaoh8(gQz5r6br zO4NP4w+B;;g+!ENe#1hVwFI<~7I^a&G}OE0!~FY999>VMKu-npriMBXkccrEU+)@+ z0-KkW!_R($11eO)XG=7I?RO4Vd-}83%(b9pChJI$w(ZgPTG0V0@eH;&nRmiD-v&;L zgJ=*+L{n)k-vMdFZzisv-hdqVhu!L@PGJRd-&Tsg#v}iE<$mFc5twNbaXM2x533Jx z5?v_2x~-}#n$SdnWqjFJ-s9}WE#XZmLK<1Indcjv&*b*l}Z!eG978X*Mt~ zN@9$a7oCL9SOc;$Nuv2PBalnS#$nD$7g&}2(i39j!3x};t8L|{LA}`^9}dwIm_5m* zYg45n*z8(vWQa^CNLSw3Ql_KB`o0jFFNYxCODnpp%a{yI6nw^Ymri41OR{9dK^lNU zgET}zwg(nPyww_Q7svWXjCU9RP+@+9@8?di>4P(>YSPYYcY(+)r$^fKaUk+W^%wrJ zIB>FhFL9iI2JX3*M!2!`!2q^{NiL+Xo~mWu{?uX#Qa;b*EZ!W2+WbUhzRBfKGlj;o z!-E;4TXOd1BuR&dHiChV9Agn)G@;;G7sRG;%tE+oKESL=O?~_C$>7@7h8~hx74ZJOhen z+)+QviSRDE-Jiw23gs_a`u!E<#lqFkebR^gSaMS(czOEY3l(V+ zSorJ4)0NF2(4)9{;(HJsra#Y|828~2Lb@l6cIE8YA+P&db`%{*t^1uR-N}prbAeG8 z(k6J8>ccaM&Le1C&~)XiUo5QNFLQZ-{J9`Q{cYJ>Jut)Arsmz+6t01N;fpb&G{*3p zUh<^QE>!2QSZsgJk3H3-^fWh)0Wn59cB~YXm|RAsd+T-|98~(v=@?FmDU1l&t{IVG zrhOfv@e-Txj|?*h_e3nLln@bH{>*^&{Maojx)cIZX$@+0%mcw|f9Z*4g>x90c%FDp z(k8rJsn)Ul*A;|Z4DQvx)&RdBzaXeuzm2=Kh5+US zm7UYbf80n7W%0Pa425P)p3tHE>Qff+=qR}#`14M}cNYDfMefJXk<%!GjtQ%1y`3g_ z1U(hakpIy5vmh66f*QDL66Jr!|A1w6nbX@h1HsrWZ)N{lUpS;-v=jTp3tZ#>QTW8@ z0wDWFxaECi43fvYn4NhU1UzK*OLOIl;7qWD=ZOhD(CfWs5ZPY^!;=0`k|BQ{(iqe{ z_w5DHnwEYWjJ~e~>#vkYzwCfU!&-LpTP7^p;~`haEpm)g?Teh#)<1Z2Om5Jpcn=cM zmYCx61HtoxKZ<=1dm(GiqPg+~O`un^eq-st3*?-XK9h+2ozYaQ<=loNFy>d4TK}a7 z8^qwdU&x<}Ir`5iCQ@L-<5!;8_lrJ*NSgs!%7&3nQ z^VA6CyEZx6j+AKEGArlqDnFF(roDfzK|8eu;||k4oz3x^joe zK@v#0zy3wmI3HsA=cYUDV!^oakavcex-RtqXKR|+_5}G4;xWaRmxr97$FvOcBmeOs&;8n)7tvrT z+Ftx=Iz863()@96H57<_UQ*8YbnfX{0=*s?s>;D7cp#y@rX@*@>1K;p2<7<#9!05x1)dRTM3PBW zva1TZ!$geUHTAh0W&Dbb2na0l>(3WP$vL_W(~P{wxXR7&P_Hgirxp z^IxQ1?UMwZ*;Wf>y-z`q-zu*{aTJg)a7=4tcLW2v7e26^n}J$a1w&sm_ra6xg)Wa} z6G4$o%Xc14N1&W`^T%aZ3QWw1J&Hnr7u1=`cxOt6fY=TxcQV8uZ@t<*2j`5zvw4k6 zA47+r6QN&vKeH0@o_uhD<=+mhT&{O)&fbUK9C>_hdv1W$wy2cbR2qn>9y6V9mIKxC zi;ouR$Dxg6S4`=r4ahmS(D0qp4rbk6@YO~9fmiczE7NDirnCO6G4ZiujWtFh0mDnID6MjF9F`e zQ;C1F^WpQ`JhY8z3y@*!sjdW7J!~oXj$3-nh6S-Vo3XEL!%v5j9dw943e~xdH=Ryl zm&jd+IIh`(BmcNMGGKs#iF# zei`7X%6wd`3vs`s2RiZ|cfwhTjFb28+knT$HE)kDbil=*O0FV28&EcV|F#IrDeQpC zVu@eP6V&K9P1cDffcRUxtc`{9Q2FB}SFhP7DB0G;UfD{439uS`H`>ty7po>j@|##O znfA{2Wv(V*WWD6WNu;lU^us%H$d8Jv+;z~q^c?)!j`BNko*dKcuI#eD)earp7o-|m zd*S@os*O9{_dvU*MR6t4SH!kr8=MQ0m;#(PJZ|j`R^Q(%Oh)>O_fiu4!ngN2NicIB8=wq_6U1e+)1diC|J8dNvC2v49}jJF-JfhgE!2`xtfe2=_vn zk2^Hc9lWDy$e3J4yP2K1@bz!AKMFXYdOis^|R5P#gi_#L}L z=?~6S1toP|8-?yfMwh=M{$TsWNBrZ$- zvSSQG4UA^8lAszy9JF^aV>zF4?Ab}0VEjSpc4XHPtYI74x*iz|En|9xq>;W_@88r+ zfjv;<4@C_@y7dxg9@Mik^_kt6Ym}c#& z!WGhfc#}h)#S`f(jp1XL-kYFYiz%l;M{*N>a5egMWhfR}>{bzD)eP7*xmPm7T_WH! zzm=Cm+YMmd@;f60>8q}m>|DRso6zR*cxdy64%q#xD8tVqgoU|c{aUhuI3D@{j-(-S3m!5|x-)qoaxWIvKDx7Co?ZxT8oO*>b{GSR31T{f zgF9foma%%WU>**ueBqby_<{Q^=FYUiM1_$w^cm={M1h6Wx1X>yddv-f?%}nYvVbY#2ed`4M{|5@&~aL>dR(C$uG>6h-@Y{rm0sDYGF>tQ zuk^c8&m;b5{~O-Gdx09uEgfl>C)NgeVShjM@Q=WkCibc{yV0O+IJ*IR{rnHBr9qy7i8N&pgyp%*L!NU^Xwh6V~odyv2Hq2m>!05sTr zq@=m2Fs7@;pDx{D1*J;K9r>nx@H-c&nK9yzP8xPxQCkK4Lm4}&t{Vl6`+DZW0%8Gf zr@BOGJsR9ztzgJ*P5|cv*o`l8&%j$@Co(+Q`e5w?(NNRfM3BXIbcana7RWe09SuhO z@m65Ujag9uSUNQ~5OXI1ex3yjN`#LF@47dXwN7A8NHRmydk8*>!EuCzSHhaI3s&B{ zJ22Ij^r=yi0<#y5y za8I_Mk<>E;i#o$$Y-4m7>3s#d6qH}RPa}Az*)d|RO98i40|LM+3z9q{#W*nk@@vIP zxfdv@o|ZNI-2`o_@?)0@YvJ?q3ReLh)K9z`A9+;w1CHr#A6e@K0F&PBF4itz=-(51 zv)|Sgym@=SC@k0-ye>D$dM!5wL+<~|%6V-A8*mj zWy_N*sQ=J#>o%)7mjEuYmO7-~^#mvT{kZ>bhJ%+oE^X)Yn6S|J)FB4sKdgF(>9*JY zp3xc@ZqhC@Llu8|CP%9_8pW_>sDQRL5kIbz$f_mR%mcx%% z-eC!ic@gK7x%X=oF|CL__ zpI{FBg#4S_@9VonsQ=I&ILYtAEDMOf(3Lr%`I7!MSMhw*f9(9F*U-Kn1jK&bHp1^G zfETCt9}3PIL$bv0#}rZj(S*k&b*KAa`J7_2)HP!4)v=v(Q8R1seW330iG}-sJdyXb z=u{FQr`RBwPGSHl?%rlEavXu9^rrYo;UNs+%F?P+?t>*VL%JeksLx6u9@$6!!@j+o z)kM}EoSEoLGJkvoQ`j}02qFI=7)tx#*w7HPTXsHbM*ZMr{=0k5vKMh?1+0E}3wk>aP8}-ixcb{kp5Z?M;+2vPmM1BWgiFx_9ZoLWYu0kud25%8SzJCYpL|!92fJ{pW+S@iC+^5C`StTx zR^ThALIylXj@>iS&mBhkiemk^O;BSmqzXNDH1$Xiod1Itk45@wYnfG#6y;Y>r;TYZ zdtU`)e@mI0_(ouvreosjr)wbSu3Nq}0}s~4dtY|>RT^xnvUx&4`s#M}V;N~gz$?G~kX!{4sNT=ERr-+) zINHm&StEVb*`ns%^X?G-3tp7h`^1iM7@0L3JB#Lzw6%Sg+L$quOp8#`twzWlw7FZ- zeFXECV+V76Vxi8N2f;%KA0-=;l27G(;Q0shi(|LMLAK8PgHYwI{cKuywW>dwv z;KGk}{p2ZXj|&F+c)Bi!7jQEVOpsO`rV2fPI@UekJ&^5%<_hvC8O4EFe)k*?lN~4!f+LzH=>k6W-C8S|DzY z0DmGa4HD%Wpx$A|rR#YyAg{xan*L)vcvE~;^9_X;cwm>QA5EM9Zc8bhU!9=CSiK5J z)#MO=oUCIoLH);7*DL4S#(v>uqX&OczX}CuL6kc`r(=O(s@bKAi6-ded$%MOZ+x>M7r_AgaCZTKz zt~WRwH}#GJ@d+9~U!wlQpW||z0^$$Vwe{HC^?#7(biz2}yFDltt@vugJ`P9?J8nkpB=7O7WwMHUdg|7K1|%nXpyEhG%0a->tjb%5fgyqwl@_Y7UzF^0HZ@xP|cX zYkmLU4)V2)MbZ_y5A{Lnn&_> zfay=s3s$_VaH^nI_aEYqlR^3R;@t5-OVBTx=z%@3`H)=~(q#-&=i7^W(R|6&=kU>K zG=D{>B6w>X_2*5>u0GFE{~=>dezuG|0dP$S69-*-1Wx|D<>O6!5{wqq4B@$=K|)@g z#$}XW4XQ;||3m#pjXZ&>ZO;{O`NG#hN5B_fY?#ukZ1=M-}D2jVhdth(A>Qsb{ZJ58}l2 zynp0>BEu%Jd3+du9W0#q=y(OqCmN9tv57z2gMLw699Ag5dTbk*#hq&g&eE3j{8Z0} z2CZt`Zj&3hh({}S86ut_C@o#{6~S@4bv8_b~phuXV2Ui zh957q4&EHN4&;BnJvEN}$B@Fi5BQ7J*hbr%BK3EApyu82@~_CBi;)o3GM|CqHvAIO z+|dt1?qNdYA4#z8El-{-N@&7=6x_~dpLL>_2=Pw2HhQBDxl-Bjql8%2HS=3m*z){D7s~(<34RHYc z2vdm^@ua{+&mFz$x8el=O?=bV*aMdu`uGsCbT~s!E0$TA2&x`u=`*AJO4};GgyT&m zWQsdsCAqN!&x|IunP=?7QE8`*?U7jEV4ku5kopmz@4al^A#Dsenbb?iPL0E%c7+%B zkpH-NPxh7PFIyPum|!*AmjTyCZON*PS+QduGIBrguw(Pom-OUOes!*+?Mm1AV_0UO zMB%-TKuC2?v*oXq9T0Z$&)gO#fsaETR0KS61P+OR0~is1dcl^>j{&MwnnZrt z#qb`h(g#wMgRQR~+)yY({ki-lN*n)J+~)YR&gbFzkR(K! zhazbKHWe}l_3_n1<<|l$CJRQlMz8>k=5^}PG_GwMHRxmsiB+tF!FIyG6z3IA~Qaf1SQlwb8SU)<4eb_T(w zaQ31S+GRE6ndN6oNzl#w(WHyh9W+@B+h_`tV-MA1#y8d4VMVZlZ$@V?{Bd$tEw9iI zP~8;R=Hl3bWJM`K3-glLr~TZCMaKZJ?C-}niRMe5KE87?>dhlS+0?jIf%2<1&2uN+ zL((9=rS|?=q_1p-h$X{|MKG#IRKsEpF9Cbzfa4RSuQ)Pt&Mce`2Eq$t^RXjgAev?z zJAv}6$N;ArE3OE@%|x!Yd!YwjR;tZ;j`~5q0AiSd^c8UzbzZz?9C$LfN6H`j5vS}V zr<8N_64cuTuIS48fxDNo8z=;4p#I9=*Y>C%Org_yRfhQE>j%5(bVgn`-wHFJaG2em%CjSQGVAL3D>_JD4KPxi zVX%XHW&+D`-!kAviKurD>IV&+++LKktwKl2$+1f)zv_$H`XY_;EB?P6=KnO;;5Vwa zI>qoXKt(hk>mxu4Y5WUqS8fG>GgB!S-B7;U*zXuPjru{ud~R(Q*3w_fjop2|T*p>g)8 zCvx_fK>CVEQEfo+0m`q;lCl|*zDf_K&R~u)0vx-V`{PJo?LSqb-C6I03W{VWJ5YZ0 zrg;D5BI*bA!xD_!E^NY^u=V`Su~;ZR{&YMV=_}nkmQf<_sKMtu1T%e)6`VF@d2ztC zb69J)z=LL#UrjbNlW&h5;Tor?ZofwOxTsWco{!Q7T$efSy^Bu(=jhK9zd`w~E*0&k zhu1to^7L0u8}SY}ByFmo&9wpT7wDATk^gYzuj2og;s}C+atpa`HsR85{ZP_G`L0#X zw@JmxCg>)IUtdP~E~h$KTPRT!h@nsxOl@Gr+Mf4n?GCwsqa|^JMT8I0b6EjAJ~u%m zeS+}n-WAY2@a&cp!iVFv!N>K8KRO<=rB5{XLcL9y#G#L-Ad<57?Kbisn?JgQ*pUAa z|D)$dcMKF}I0jV=>QR0b{vKMhBLC5ro`$beEXLgeowt+}Mq#SVN86O#a#&1|ci2My+#%tj z_N9X~Xp}4DH--GU`0rUpz09rsF{z>skm4!iRtSuwE7NAOCADmBs&S3Dy5& zb-V-H$jFyBVz?OnS3k6(yg)Sb;RTBtU95BIgKo8<0w&l#(e6EX2~5|@QSICa18rs( zly!*ofx&cezOO(Z&SBG%*5lV46uTSi!4%vMA72l#%6}XRUOBvq=)nmC)u{6OFAFg2 z)}z!YVTQ|C@Izr{)pz`ukeCod_9icOb=PaudjAH-`Gl0?F3`rJ&UCcKsiuIJo$C3v zfv>^HW4PRq!N-cp2ToaC8%xqi?|)Q=NEp}Zt`EQ5Wb)XRMnD2M&~xb%VM_FYUF=e*dNfnhWF zt@DdXw}Ji-Nk{*J6kyXoFnB>8V7FI}B1&SfVUx9sDt3pFVDo)2(*qPGQs1PDx2j11 zL>eAJ%MW~kg+S}T05uLXHLZ8|zPADmWBHi>d4vIZ?xY>V7B%1$N+LWeqJ$Y+oj70; z6~huqV{eWbnqYx0*}wW^E@6WY0x55tW5IGi+IP}(qmAv?m9Cmsn}B`hA^M1oN1!P7 zLo2=GQ=mjdE&AwYG?0(@eD5|x5WqJ)$UWz73DVB}xjx#$54aL8Nkw}mgRAn@+Wa_1 zj4MrW_3{V<_B^8ejLWPW9Q}Qbn5N1Qn%D^ZAnI+0g3(bb)jK_~k`jMvl_eF&nRP3^`s<^1J5i9Z>+qxccN`R@`@iUV?|3f1|9{-f2-&11DKjangd;n$D$3rXgp7;4$$ zJZL5}IfG=s(%tj!VRM&Vc_P_y+2cGG+Dqq1)v1CcPFRlmGKnHPs(qz`F{hB3$ZNC9 zu?2Fz-{eD%EQiWD=f1AP=Pb*OFc~%4j=Yj9s?j_D-9rwQq-pr>EQq1rR9&lKF>SOg z`HN~!gBv}4V{`k3y$xEgQE%LToeQZZ1q#^SmPIvYY}o#ryK*wA`ah^g4j|?}s;LZZ z9;A@Yw|@VGGPcC7o9%(;6ImyzgY8;hD`Xe7q$zxTl4U(=qKb9QB4w8zN~K#47|B-h zkS+xs;6^`=E0{Ihr$+hBr`_vLGa_LVm-PoY4pf`6Yx1zm7SSK({-FHkD!RxXCGqH9 zsEnb&CHn?bTiHfzBcj`{KFjHxNRe*||1HPLGAa|*#Dq?NywjG2XGG=98mU@AJn`Ps;^aNU0t}n{tYLYLl0lcjQdGj?YO@k|X+?B2kemeR7Z91U*%4 zt(Kz*EbqEcog~ZhoXUdb>tk8caEXK?;*4l9sY=M^CJ(x?A^tXH_#E0VpfC^_b_w~K zOD;Fhs-uT|98tXW28c?cqqz2TsBD?{!Rl{Rv9cGlOz0D%Zlj2O#qYmm>7)E?d~3iP zY@-yRq_M_*hmq|$uXom5;>hh(P0MWeLB#TO@<5Ze0!q%P>7bXBMP;JQ>9_u{Aj6Q$ zCxl;~MB({&%~!TUWc4>{lSFClWR;tm+Z{_J(Nc2Mp@JK9=&{aa&7&y?WCM1snU3C9 zlr7YeUJw-uk{u3T;JTE0R`%7YRw)rW9W-6*p_wYYEVoP9QDQi899cf1W|pVmLI00W z%>TJMeofRN$OOY^|E?1L_qUvlx>*F%Vg5ajKUg+scWF?LcBYRYba+F~Z0FruTMVQ9 zd;j0xAlc4AMl43WECWx4Rp8;f!L_kO?yk{5j#Mq|^M75atw;Qzf|!_2XJ1G5qT zdv2ul^)fRnH)`;?;4>JSE_b?XGmRU=XqEo0yw&Bx|1I~ptxNdM0mlEH|6AVfh*Zh$ zmm~kZUf@_>KX(?JkglvdiDEY5)L`K`aZHE#m;V3t>mOKc@ap8OT%huMIwnkqX>@%j zK!stnf9VgHRq4iHKKSdmW=xr}HM)e4xm}T1jAQ&;civN&jri5gbkjP_2FAY5k{pc1 zIEplH4(rD^$a!4}c3*z8C+Br9gyy9n^?&P&r~63$9m8n<)^D=Kc!dnZXe93_9XGFj zl~Xx*^Ro-Rv)qnb?2pcoOu0@b6S#kj6Ui_LolYUr?W5l$lPZx&Kt{i+s zN29;?-}>HMp;2$TPxjyQ3Na;?hnSAP>VKB{B<6$v%(>-z9P`2d`+ODeIFo}IM(aNl<$9)UKXbZ09H>ICM%Yj9D-!W{B)uOti7K_rzf9n-6TwqDTFj|ydu$-#l zmK;0z-m@W0hxxbOEib&#VN8dif0g;7i*!uR^R!dJQ4FIsRMh?=Vm4yk)q5=Am=C^3 z+Dq#l=7VQE(bT`%^1t~0Z#`xA%;WVH`BD3$6rW2;|1XU8@9OyK&%`>O`>yECp&IYT zyBGc$ZZ__gp_C3zuC8V=w`EC5NbHpDRNO5^{3v3_JOA^P(9^DW9BeFH97P;$++8dj z%p7gpJe*wah?w1VcW|?Ju<){RF>|(Yv9fV?b8&Kv38mM(qNC_w;p%E{Yj5Kc6CESq zALARtA4+Rs@u~LlcTGfg`-7L^f7wkD@X6y2ou^3ZTQDfrYBPn#BrHz|~+1dZC z;{RMZhh6#hbKo8OGoer_8)sL0hrh+-l)b_}iH1stGCKal3NuG1Gb=j_$6Gcr8nmHQ ze>r69?iiz)ARWr!Xl83;;pXmQgE9W^%@_@mP#R|^*MHAQL+M?eEnHk} z%$(faoZbJjkIKu;&CA)w3=^V%9_@t*#{baRJ7Oeee_3qxR}QhZ#)Psty5Dv760ve} zu@U)4a4lS1EU+?kW-d1W$d8M=l^Z6kp_Km+VsF^d`adMmSYrZW6iF>(p2b{^;nyJ5 z_&cD)Ln*EQKL2kKGW+u}&x8JRmpqhICPqt3OZ^Xa{PRbH1k;?JD>DAQ@b|z**+0kM zuN}s{|KXJxCMK@`42^xFBIm?KL}T17?L_UQLunoVr_Wd~G%!A3LjS*sTPW*)`t_fS zB4*C^R@i9827=lD(ETHoW>#)ae@9{{mFD%c7cPiN#ROno`8z1-E$!XRZrM26U_%Kb zvAYsV`5!s{dz1W*jaSV7q4Q}v|LwnQD4DzX|LV2frTu~pBrSh_>9(Ib;nPM@ zHA4SXH$Ap`WG6>|*@AJ)KZ`H_JV7G|9R4-scE`JbUzk_M)5I$9alp6i#`(wfLCV##?vyNl3-TCr1?@&hU`N z$;>8*&@>8)tWmIqT=D2+PN2R+x-otDrs+ob5p7;(=<=FYmsxkZ)DTtKZBd?q)bnL|a`Cy58gt zJ^G(*zGbfN|Lt!4;g7cqyX1kSRF}n%eT@&rdVUiO&yKvll&E98LI{wkv6Kz-KdOK5 zbW3&&pOp&rjHYsD?jSlz;Mdkku+NIg2zBaX0tA~VxnEWUH^b8(kq+zV0u z&n?%TwdWKTC%6oT5B5{F+WkDnEJZ%!L?3dx15^UDZ19dtUHR?2d82K=Zjs#zKK-$} zH`a_h??x;e;re+d!}RalG*Z?LgvgVJm6pdcbNIUyu|06@)VspTV=QCnCUfrGwte#a zfl;06weAZ^Y-hPIkZ|vIgVp$^Ox#Nbanl>$15VV(ct55w3JZJH^)b391LEW40nh(?J$uXB+J(u^acTVA5?qfO=Sx2kdmLt~K%8RAa(l2=H z=@XxKgC~TvTz^p|wVvSiuyHo-i@NSOUYqdSZZyM_-Rg=jRaXgx6hlZpKiyU5X-)?Y z0cX?SM?AY0qidfjUh@jL_-w9P??w29y@CUz59Y&O-thEE+cMZ)3VD7^O2&DA57S(u z)7R`9DbuM}X~rX)8{c}CuxPK!s`9Vk$gc#eZ+bJxZhf4}QT(x|P>Dkm#*N>|`CeA*5s(4|i0TM(PKcBD1Ln_&n@SnI#`$Cf zr|Zj3JzPEe!i55z5A(a1`@^ntkkPQs>L-=9>jx)CoRbHt7c|b@wxx9Ew(u@>$~NLV zEl*c|GU|}1=1yv>(r_crz(xCc$X2;#J_GJ$>Gue4vCoTn*NU4r_L940A9P9z+`xA9 zExk+Pl5;

    &#qCTg!0UAv5j%X1S(}t!4hYxB3#N)*&Lf$1<)?G+Lu+;$HBRcBPAm|ywb&=ylXG6BRjhsC8CgdBAR15NY#}(bB%`P{a3Dsl3TXi3$7+lylyz>%ng8afratkyGbfFAyjVDBUDfL}llZEPSWL^lub&Nk z_9pqlr5*;)WEIRWE;dg&_pbGtCLd#FtIyc`)9w+U7ejk)Ny^q{!Y1voNA|iS8HZf3 zeb;+Z>hK84l5abPItkrIMh)FubRreC*d}f4L#^C=M^d;YKD3`76KU$F@Y~`Y(>`Kl zChB(3$o|~pz#{j3odoGehnE{}NhQnOEYOl=UD*HBveMyW7N-wEMf}R!#Fm^ZM{nlc ztdq+7QgO@V@Isnj8%f>!ywEqPXFc*$6$coH?uq=EAe^I|mkauHB~N$qTGm{SIKf@j zaFCn9Yc*u+WNSyP!kbTb>#lzj8LeyZW|%lGDt%t$81 z<8%i-5@HrPo?gsW)vGbOrr?mb5`p`H_=M~H4{Dc~bcAU2TvEQ|Bl1dm$vE*EL$v4C zXUVv#pyO}n_NE%GM2D8-!zxJ)549aX`A$gES%B41kM+8@PTdJL-W@h}43Ji< zj6ZB-GF2uMwU52D{;K9uVC4QYZffmQF+F!mUM{`V%+}z$nZE1j7Z;!H4QrAAXK_O3Nxn{{e!)nX-<2o|Bg9qfM8V6&pXKt8S$Y1=_$0-q!`gyj* zxz(pKDIWFQ&h9Uem*$K(ZYyN*dnt5n`qW^M>mQygbdBw#ZF;}sx9hRZxqb6$C?A9= zPqI;Thp+BlyfWS+{Gn`%p0P=Rv>_K4CHMNsbr}!+Q^zk0DNrgTuCPY<+FmD9YvNT5 z|9FT*muzeFd+Ktw*CVH#tY)7tT=SD}9#pk(`F_x7NH|Mh=<<%+*o!iAR!9PC z(?MBz*m_K?x!uvcmZg$n;PTbKiB{_)efUCPscZJvCwgyIB;Gq=o72dAFtIp#>Yh^9 z#38oHYp=`iYJ7?tk8t=Y))6$gB`~X7eZ=J{$t5*r+uq~Bg^a1moX0r}HW_IyQXMPk z`!Rc2C{B=nlHa83%(25YGV|80WaBmRLk}5tKQP^QQlRdYUEf@|%)Ybr`n=+c>rTs` zP5n-t7+;;IF*W(FRlqhz+y9A6*qr~VR(yy5>qhB|%+5jbM5ku`vFgR{?@yzS)!zFM z7JKiq)^n{CH7T|>Ia(|MoY`?^&!ADZ7i-s;@HT^qh8FdG<0UmqZS$_*^2mB6k1V~q z$+oz5DWT{Lp4ejeHR^$~uO)X(_DohPo403{nu%4BMnuFD@$V1tTI0>cN^DKJinrCx zA}OZTi+x3_WE!y^T}skQmF3SWt{vk@n5)|LQw%vG54xz2 zXGO@uJ^^Pui@u-AQ%?G$?0LPvvElyntz#UW*0eA0HtQ$M z4>D5LWtQrFJi1sJHm#%FUcX9sR#xRay@wa4RV`}WXkav$#92d1qPYjPYD1}mP~ z+-J`^=i$$;`*EQ(YbJx@lHnJXSAptOeW9zL#PL1l#w?4S5dzQe&gU%kYeVnq`Z*qt zP~}iLX)G)$X>@_J_yrETXC0vv2Ux``r#_v$dk}|98a|srU_7{)r&ZF{F8orw@Wj4% zzB`Ri{^SphRhPcD6zn_j=1e$W(l|#B+jrw1ahB2pNcOQlr&i&|tZVyMFV0u1czH7> zJrPpSUSRBaU}=duI5q6WYC#N{fSPJ*9 zM)TixpkJ7^IketoEJrDR-llOfM?ISnO9i})-VX((^c^wm>~;9@4@@P;uy~9nDVcjH zjh%(7*)0oqOi{uxI^}=WRR5zy#ApbIl3^%Z<7Oz$|0qy@5iEJ=0Sv;M z{V!b*!{&b#E_+uudn*jfTRGX-Vss`k{Qn2=)L_!E4*Ums+Wv2N`~QwSxh1eB^*n0O zXd@t2L)zk&r?DuNVkc%&B@qo8HZFu2<0;eycC zRTfer();8TE$ZC{GsdsC*ouqL*7!r6^%OkhH{YdoJv)G&&n~@P>n8xi`WLYqlGng6 zz>REMZ9hDFTo*-U(+X#9K6b1dX+WeuE??cRXMl>>)l#!>yP$~c{y@E*3G|G+RdT+q z@Kc?F`QBPCu<=}b*;!tNqJ|ep&P;S8lf5p5u-r&w7olq~!Bva&EcMRQg>=B-&v*MQ z`?8T^Ly=NHr3Lc5RJZBN-2<1$KN-m-H$v^H=>gi$cn~upB=%UwfM2L9J#(co;Frbt zMs>U4m4(srDm0?|%Wa)=v{mT$qgahyV;ss8#RtA{>VmJ$z8`+SZU*jS7hhPDSs<26 zk`g)eMQGIW*a54hY>2fN9h$mVh>CA^sr(Gj0LME&HA(h$z^Z>jJ{FD%;S}~)PuW$Y zrhc1CZK)-&{7{ghc&!7{LN?_$4)md?b#>ofyzhiJIyaECawJ+0no;c9%0-Jbg2}YT zQXP?|hTFI_= z4sa9DXQ3BsbX!@7Qm?$!)}|9&&S>?>t2RQK!6l9=QD2C9bBN}>Y8RMAgg@WB*#KM- z#y+)x2QJm#lvQps^dy1kwVs^=iL_*TdCPbZdP!>OMx6`S=zc_n$T&fD7jad7n8|((Ce5@So0H5JDO17vBXyG(t__{6t7jKY}JuAya zMT{=9WJaB6U-P*!x!GD&1I>Fl0|L}8%6~tZHH1n$d&YMR3Xs*+mLUt1e)RaH#Q-s} z4R~9R8>Ix=Az|+Bk*nV_p@B~7<)Iw{ivO({Kv`&oIs}~LB=w8YXD=RyYkN3Y>A0z` zL{$bAbFcDlIN%VuAl2SVZU=CE@(E>g4@a`1t1-8B8==iaH}33=gex0&-7ivg|JEL=jFY9i_CL;4vfpjqP_Gn25RU*;y0N zzB8Md#G65oT`~4`KsN^&+~>6%J>G%k^XfV+ynzSXI-V#lNJEYHc2sDx0uaTQiN%Ne z+hA;->TzU1HVkj1cV4os2Ap?d_wKn~2&g`KJ4(JDt!Pa<^h=gOvikh!(hd$icFBqn zMeT@Z{ivvDRtKzYEp930B|*fa4a;YkJb2O>EPA`N!yzThDI41slybcsU8ku*d(-6+ zJB1D)=Xq@XLVO3Xq(|e9ylw*}-QcVh-4?K_9CyD?dJk6KV1cad9pHrj_Uc?e4s&zL z%(_M5V5DcSeCQPbv3NdRJc)p+xoeK-^|YeLAFk+VnAV`NRb~dU;40XP({50X?FM>3 z;SxipA*3UHE#=xe0Swcxx}F)W1Wn<6yZlLoV4mns_|A<(9h!fl9q8MT|C6PT#nmBn zz)x3KNWK&0L}eC)4rjttzjwZ9Od}K-+|q~_$HU82x5zoU2y{>-hKKcQ2V#1?Dt~0I z9gMjG+-LIAVTJdk@hRyl2n{!(7AFO`Po5ob%|=9rj@ytL-EKt`N~#=B%kyFEw(ybU z>s7$t5WLLrpagy5%n;njX@QRI`K}H|BHXQ?K4O3*5WHW%hGA@ep+;||TKoT*3lVjuUP=_3M(&h7l1!A|r-{+chJTrvC+ zF%A(LsYWY5bhpce%g}j6+wY&h<3MsKdTjsNAga<$AZ(?Tqu2I=dVb|~aPb2^Y1!j2 zn@e-T%SDl90|4^FD0q~9f9+MGM#fqUqtp?EFY&mcn8Y}ATu3Qo^hw51{62k9GO zyParH&gN^TaSMnsWLPyTcA?8-myUn=jDzQwoxZ$0-G!K*s_*Se6Vb%!7iSaNGr?uA zv@$HA1orUg;zzqq5Em?Xt2va0jv?=-!p5~o-&#hrTvrTd5iwWU1--&6qvnIP=Zav z2f?LEXk2kgS6{0~p6L|A1H55q-jSPLq>hLlTTD$IC#`^2FJ|u~Wj2A7$Mcy+SySYb z{%+D_^dYjo&(G==g+r{3R&09xxrjd6eA!U774Y&@M#ITTh`-*Rx2w7rz4GOKe^n|K z-J0k(5(I+Y!b))C_aC@WRL3BucgwG+h z421YBXWKP)<$f&0{gI|fhleL82DD8EQE)-5_@oIAMWq_4t*s29{h@~q2Or>oYGbwL zKvfn}WYw5bVQGN)_0Jynw#q?!itAYR(N^RWdO<&tp#}uif~7Ae`U7udU@TQ%6$nzj zS81@WK)(Y|ChZHUMn5aAMJ%jWAW?-7f_!`%k{exgm#wTv*IO=&efib|+94Y6A{MjY z;mq*qC+Cw&Nf{#9*gL&^r!wJ3f}0e{4k^e&=V zqXp6p)ep#S;1E&op<;Q71(dvc`}~bw7dqkeovD7N6BRdXFHqln2KFL>BaiurKwUxp zi2HOZiWs@v=Nle`Tz0P{9#W}=v?>bowx4FGA#3@vaCZ(85BRddb|xR*^BN^c&$lA= zyN~brkvs)&R*|fqbNy&W>(YLTY9g$4oN!T&E=G|{kFli!jj->$XcdEWI}G*h@)ujR z!Qge5*$FoS$O|UCILloFRwXhSH$+>|&IO+Gjc<=Z^5v{+KBX6o=iOXLiLV0-Us227 zk8sFN?xWgm>jvcNS91SNO)5Zr)FMO zERln_jYJbR<~u-!R*@%E*cU0ITGi3`0>~a3H{a$hghv^XTFs9e(2wem%JqaoWF&U5 zR<`W~&3KAjkXe9~Ll}HqBx?UXDUkXJRqIAW4JK;X3 zI<^gADg06sipX3j0GFzwOkH;z3=()u9vK&5d0N@FxMm4ZY@yOE`m7t29w|SG$ZkX< zdi1jEZwr8E*8L-)Hy@1FN}rLoV(%$4^jC5O4*KjGPd%vhfR+m+(^mul^kTX``F-j_ z>yNw2k9pLA!GVlyvwMSx%*{{6PO22NM~*47IA^1s(>{6W{*@@6M_tn`pbUxdeo>r$ znvB?YcKkWaVquf(_I46utudHin6x)^%% z_rV;LCJ>VTrMm)JE~{lX zp5CYGl~HJiW;(ea(gW4t-n{i{P^$~*53EO83-rLHfXbie7yud@-|;I^wIhlnGik90 zdy!S&C8>K)+JXIbScZfZy>_@AygB*o zOEt>P3Ey50>qS{xJ*JvucsNKM^;3tn5(QHakQ9&&qL+$e^I4W%U{8ZCm-e+olNSHB z#QS_itFbI^I+Q~BEO!w7ae-ZMRR7HfMhg@TwC8Okyu+w z>%Orv`1WeS;aX)QfONxww~2UU#1ghISGWSc7;(G1(3Zfb`>$t5^{RkGTgmh1K?1rf zLBW*Pgi5*4z2m}(ws zMo+dF;-=hcvB+;yql35I5n*aU>Bn{|dPR%Be5|Dt*_n?&DP#?R#YADsLnD|`&9<4v zG`9s}qLa5&z1pDWD=Xie(+ik871wz`Ar~$m_9Q*8UXFhAJeS*Fj_Elv=jU91HNYUP zJIyU=B4iQ|$S+t{qL5J@?}{`c5|L&6{H&oD*=qR3OYtYb#(ek&?fDECJwP7M)uah* ztb{j&xH1&aeng9rF9S{7nc!M(P6Cd8(j%Km_2_PqA})iw3vRqANVHGGp{6*Gw6e4zlT%bz??_jIS;1!7wpk~1H#j^aGgLvM4KrmU ze6hf;H+xU>ClSc9&nA*-lmZ|5^8tpD0!Xk5AMWlYz`Zk!Ok)%sXsWOy^*l)m)JX3T z?j{}pp?UK3I3XPD6V$mweXA3No2znf9F2w*Ws6@`>Z;JpuA%#^EFK2F<~BL_cf$=r zz`j=?stI*(ytky!F~G$ zQydZYQxq9DJ*z_!}O7Kp|sj(f^BBT@~| zT88RGlw4~Ozue}J#aF*=hqx#^*^qq5-fdHTE zOy7SDiv__mW07|kf>2(y^lM?Bcr-zA+YRfkq2K_4LzM)_@d)DlxD*$uug_U z^-n2p&S*D4Bm;$VcLEU&2As@ZX^w^2hL~pYgXQ3V!Oi8>+eR=`S=!y3k4O7d#gE&Q zcfw@-fji@8D!}U6^#zjEO7t-yL|@r44~+RN?bdp$kbA0-I_;lAP}*u6D^2l&FBvip zlV)iUZ+=>>=SU??IqIA`&W}eloEoK9<-5=~8kouttb;d%KMP!U8c^!Pg_?^-c~EU1 z!`(9*jZD?cl%4wsAbDu5$)}bGY`PRJAu=^6td#ABfKvgS_p&Vh*f@lSSJk(wT=8fh zU*od@?GCUXm*LA?sepJo$|nsrHE`4ATIiS@9)0QgK%%Hojwm=BCz9^9AwemzcwOyi zbmK;?jo2)P8(;GZG~KO5+k(G8l8vMxq4_4!k2*N0dpSqvtmBLN#^&nIgb|SDyTX{h zi7d1#xHdR&w-v)L+H_}bWkYeBqV1It9MC5mE~&|S4s$#A{s{S(A|=T%d)xcX;Cy0b zBV)c24D9x4^ZB)*9KA%^`MDTaSxhr{5r^q1#-;Do5FRm>zB-~Doq?+F_*v=hYe65_ zs=ql4+(nCgC2P%R3em%8sgd6+7%t1Na^+l2hE$nOswv8RG&XX_|L{RP1P(W}_TMu> z0__B^gO%;*OXkVU(!ee#zyI^dXX8%TzLIPu)|ibx?XOS|QtShdeN_IH_;%2yQ8@ng zqBn4)FV67JSE0*SH7;E}-T<`@2UBW-@{rx{!Pk$hnh-5{Pkq^>Jo58&+xMur8M&$L zM^({TU~#nH+c&idaL!M*jdt-M>BdC6=2Q*jxbu=}e~Q5gZHitEu11ZaviCo!5uiHw z#IrydRruOd+VaP$5~=b<+w#)`>Kyx(CO*}RzNqc(mdw;aqV~y>3)dTg@1u<8LH&G0 zL3g;;mb4xo3k_W6{L}&UM^em{RtG_zmbiaPqY(IRPF@yrje|q4)r1qN0sPKHoDY1^ zj`%C}E;Cc^L(g?{V=4{X!9;saFV+>aLomPoyg#M~aEDK(T_44x$w)KFhl(_ENYa-eYoSa8v@esCjX5rBsW_LC6`F^dc9JT)5 zwXl%wL6@iBP8q&$0IAz~=~~?#NdFK^SPM%cEIgxFAD+oqqed2?%?{`g`w@Px5bRmD}IFGuVA zWDgOM#yithNvRs7{-b*J#I+|-G)ci;jyH#-_II0)eeuZJ)9Xr9aR-zOKJXO0UWc@Y zY0IfzV)U;^@R`NEXhi+}7CVM7%#OtArzI1h#qPr4n#2ZF@~x~*pE!Vwc0MwiN(KRm zu#*0M_BxpL@;g#LL`i&)l0C)6m`vyjo zA^J^y$8VQoQMq>6q*yc#{d#N2=@(Fq%IDfPq>FHHZ>5Lh0#7Bhb1OaLe_RKT!UsaM zR;xj0;GxXvSG~x2&-nLZ%n-PVlzRd8<$xaZ#k5xFu5r9?g z=w7{S5X|PJ=fMb|%G#Z+AfGC*pg1Nu_LGSEOOzxTn2sE<2LgU6uwg|R1z*}GwJ#S(w1tjgoe%0@ zvG;XB4etP&vM%uwznYIsls0n8$ID^KW%Swv_C7yH4Q8x0*Q1ffDYaLs<*50CO0wXc zHuUl6lc2nlF~C`O*>Q5Q6(l_MuF6UnA;||P7l}o=h||7gC@{Sho-zM<(rb_9o8DGD zL6@3}s`>BS&OX|O)Xd0CE?3rp<+*VOmrtFbHnXDY_(T_ON1Z$}N6be)3fLy}nBG%l zFk(;}cmr(Cam_2;c>wa~(=Phn%0&HDWiR&P1|akiJL&P7dN47Ha!U;%AX2xZUkkGb z&}0#hICF;u*d8oe)OG3r>svomu3+|(I3M$w&74XAnO_@Ur3=syc|nDDYA5u3IK*7mt#`jX?I1d%?mBW=uLzCHHepn~FypzA z@Y*dFm1!OHS21seYx-h7AwdL4C>t8v6vF{m7<<@Y=rK?xBrFnP>h%kvY)c4Y_LME;W5$7# z4mG`dEe;A2W>;LXc|bZ)Q_?xG2L3$Qr5%0J0LHIRPPfI2BNj<5_Ct(#_#AqJoS!od z(9QXMDwsYNe6B0@2&T8ha4_U8R@Xon;p;IxdkuJ;*I6YL2P4kG@R;%5dNfS87I{@4 zpmnZeL5CwBhWWq6`<*C8(mee#$Aa>JmuH~stk(n7TRT4cP1zXr7T5T@Z{cB|r1^{M zyIr7?E+$d*up4#Xf0b}}f(T|WIGlEpaOeT=`H#D5`B+|au`8K?Lq9ZCDTP@Fq5Rya zu&{bR)Q9_ytzgr~x|a4p;BQeBn88P0i_I_R)aF?Au0MsP<*ew-wCRYEnIW!(xelHz zzrCaoT8Hd|3-5pM>_R)X3^QcPcpyuzl4~VP1(rBF(zH3fh(WM~!CL_xFW;xg; z(#L)aO+tgRKh{Kpnh@*k(&(^J5=uOGZIg2lv!lH?Rgo<_jlPo0?~G3R!R#8>Z;KyI zFz+ZS@)Wc0e3h>X+o#z8q&T3qNx)U3)6=e!B1P>|zd84Eg{4St*_-S`C6JE_2Getd5LBf5-lCqPP;BOPTO`vH<-9CZ>!t*$k zUhI{Bl?@LmrzM)l#{$scx4=tLr^QeJ&27OZ%w9G3J!(tnOfS6m6bn~8U8ru)^y^)J%?NpCaYLQI6-(0D&m78}YjMjgoInruARW)4`_MF~Zn zJBWmTRh|?w!ElQQA*SbO2lUgt`7Nl}jUIjzkNaI!ix>~$d#6W-z|}#mn_MXnnT;o( zwtw0KX(^;_w5sJ`(@Edy(1%AgMg1#Yn4h#FWkzG=TqHO#em_rjG!wesgq$bWCjhgP zgKLf$9+(pO1~!5TXs<$RewDWbj%w$piRl2OGWp%Ke^!lzKjpc+@FBo{EtZfk;-%nS zZm%c3(gBk;b~YC=|C0N8X;ycv{Ev|p$8e)Mv^A-is7cce17qwuwTw6v=pt&gXVHqt zv^h*g`J>QqjVtFbff|UYd|jzODZ>%fv0BLKab~mAeh$(vT>_UH^FdYh^&zTKEDoUQ&P)Xd1yY)(58vHs1+B-j z3+)$cfP5qShhjnLnC`dpsigu(4 zv-6=oIwwF+TZJQ~&SayzosTE#F{e}qjZ&r^c?I~+vYVW9XhXE}oUbMCbf6y-0&_(^ z_NeBa>v4zJIOH^7^h}H7DALcJ&Hm6-h01xQ^+vj*U|p!m**mEY8h14wEaVd)ujEn5 z{7Nl+K4!@@q+E%z8}gKXlg2~7Rhl0?R}Z@9M7q^y(}2F<-1=^+b%Bh;%eh?9R+O8? z&LS>c2{yGE?WIypNP_MDc2h+TBzylXi8;^-UymBIHdpsSXR7yyWtkXo|DqFDQB#FT zeG-#I0&(cTPSWY5b{z8FmSkBf#-UdiPU$FP`OF{XJ-aGyX@&-A^E50La*!?6wO5=%;kv}V=--u|>eb+J5AR?g{<-@hp*n6o| zXo<`zL7YAeZKnA+VDya6)OH?%19F!xpOxST2`Nn`!iyRd&LNPOtYHS{W}DiAU30)o z61)PtJK)nK-yz5CPDryVbE@bIhW9E0Z29XQun{NS8SoXLv;Y3r^rtvPnlfzcs#1;k z{8hfpS+@gqQdE0$2oCZ^E>v8W%Lo0a2Zv``iEzJ&RzZ5U7t~z@?_XaFK@ZRVq2m#* z0o_ZNjZ$=*p*c3@seAS%^v+bTDwCuQTc+tu7?kZmJBcy{b+x@9X`-GxeFu+BU7k}g zcC^E`zgxA}urd0P&V1vB;Y;v+Z7TQfRtnIwj?ew3%}1xs9j8BQoPyTZm;V%taYIN7 zB5;55fLZzA+_UGi!NOE5jq@b|t^Ij)%JVfq)8$bGgSQ>f%D8|6)qJ( z|MfdHhrXpEjm=Eoh3x|1@vDo9?@C2np9iw4=m>};qX1un%_~%m3%V6DwMce=Y$l4C zfOhYb34LEkN2Y9~Rqa(p;Oyp8O2aaQh<4iJ9It!eN^ODC@FxN|5F;LB-*}F$^o<3t zHKia?qHNT}5)Q?F4U>1>z>*mIm6bA$R6`b1?rNf_Il4z{8-D0EX7^}Qx^Zs}%f%Wa z@rOLQ5)`antoshMf#dS&QX2LF)b>!R^xi`Pa`%qq*0xH6u??GC|LG=#ijkiBjTWdo zwB8Z>q!JqDBtrJ=9srMT8I z@k7fpp1Jpaw1KeE!zm}ygJ{OKUafMs9aLp^HSMJJpm{x>y5vDWwp=ppvB4w`4O~9^ z**Y{A7!KvRwKO%s(8CH>_BcQ@R-~2O9i4C^`uzzz@eEk;*r^WL*9QEfyYKd`6u|tx zm@i)^oiRUo4d>m&4w#xr^DeXQNBSA(tq?UH?2jkO4?itLq4qB&CL4NTL$CS#Yzc-t zj~MBHne0UsPne(5Rf!|+Y%yMAtlqcBMn0O05FvJ8_U)O|(a2nz@6;KEW+3xBLH_AP z9%8TD2^tN@AqNHN_wSdCQSNfCJh)dOsJ&6=pXrWz;}uulYP2E6{sUYq+KsT;?kW9) zUk4-^*nbX;cA=ri``)w$K0$#;@yAeq6CCXQLzx#-32#`eCnlwyp{uw5=nH}{!*kKc}pBBx`^4;PD|)NyfGz(iHQvi_l@vFpzb9JcS zR8n^b8#H8YO?;JA)j)ZH)t1Mi3}ve2f9b^VH6ff^Lg5h?Xg6F4;u+0BuihK|avEqy z9JT#&(k=whmwbAmW?uzpGQ5p2lx#b*QWjF6n{dN{Se~z1)Y~xB_*l7BkFyp@vpjNp z*Z}bLP0^??Lmf<%4-=2<;1S1+*`G5D1lZ@NcbLf@!~Jn7q=a45edxRX`51)kKaxinxnO80@-FZ|U?w360X(iKsTZi{Sggrudy_IBZs3DO!k zv&}#KLcI>{bg>`5_M#1ijnwp6@?!embOy5SX@is6cK2vII?*5h`RMcX9dNa!`v zMlie*`S$w3JoyT{3`d@w`R~tCeI($YJdsd-TYL+I&?#}C%e@k90BR5RhB8H zU$jT}{`9W`MW5rJ>rDx0bAnbytE>|-FCGui<5V`AuHL!7R2r_s_^CC@~F7 zU?{5uN_mC>!%EEmQhu(_xV{~7+Z7If79hZ$mBz-|eVCoD6DR#_jEI_7#Ork1+5x9f zqUgc*7-~=S1(I1;qGp4hs<}heD9!SVatG!&y*R>p?L}-R_(^NZ#(4{1Q`mDlp;IMD zIgb_zq+yUlmko0e)PmM z8U3EMX#1Gjf-K!DE%;3Ekd*QBxI|<<>K(H3K0n!lGCNHSESzE?H6DMxvGUzxmq+3deeodU{-0imev?xEh*tecc?6 zn|;})E9*fq6~SRoG5h@uP2V>+Rx5#jFFHY08~d&+=AYi$xDCyjKc^jR>(FPS;i;%t zestn3-rFv;2Bl=4DLoUW2pOigx)`SFP$ur;*+lmoB+->D1(+Q(_W&_-E4cznii~}R zTf(94sqeF{#y)8At50$6NXBxIP4!CZQ^?(ZvEQY%8@2sDT)a;$03u%OIJP$9 zQRq>%a~4O4u&8iB^;SSQ5}UX>n{}ZP8n5&%TdTCAOScjO&*WyK!%Oo4?@QYesiV+o zs#{TL+_(37m&5=fQ@?eZm9hj52^x!Le~*V!aOhHP&VY3%%MoE&`brhV_*E*UTEyp z30y8-?Dw0y;QfLSj)$TTOqHjJm((&)QIP`Q&C(P|vb?dC--mtAeA?*)#0OEmoLZpC zqYAV#e`(K@p$>UJETyVSL?Fc2_=91*7QMn>)#hTzMt_{n7Ku>-5Da^xOX2{e3h(Ht zaulM0<{ll6;7-i$cjnmXJ6m#mMc>ul?h(N6g)7B%-x#<(m1}~UD`36m?VpFQ?_%FU z=0Q<%9D1kyi}{jW33?O4FjkB`r&4M@W(%{&-B?Yyt$eWI8|Nk&5Z-eJFaKcD2a96At_Lz0@tQMWmTNK_AOnVbjip$EBkInfwYj za<~}<&mC_21?|-!Qhd+)XOdR5^PqZ9q`C>XLf>=y+!;cvKZ}I=$oi1F{;C=kS2zlt zYq{NN*9>n3-QJ4vwxIQI2htL1JJHEMYEKOgc7kN!YQD?CF3=Q!*B_R0K<&iarx7Qy zc&CZG4w_92uy=eP=aOe55Y z>URrz@an?Z>XBw}xMp$dXIM2#9%CQ1cf~`U9r03GLkjji_R}9zErf%|8_XjV8d0?G zwX)3rkFKwds`A^~m6TFYL{w5tKtM3SdKVy|bO=aDcXy|hN(s`^ozjAUtVMTscPN62 z1xR?`-#Ons*E8<;{++xq#%Axi=X_#4>r@l^ty{A78OOr`xdUgqgbkT1fwaD(d?b}65-2cy)sX^Esfo@q%{4n5;M}=nOBw;L3D9>~m$Q{P_1NENKtD_B| z^w#wq0q#O%apL(L38fBrh1IhNFuVtbcJaG^y3&xy&xgjmj*UQ)-?})nJr(4>kbhcr zu@TYUOgN%0C;h;?Z55h#=Cwtvu!#9!AO zZ%AoEwW7Cj2j?5mqWmvE_NP8b^AO#}Q_W5gJ@fdM&Z{<%E_P2z27e!o42L3?OB+$; z^WJmwM$zDbT)S?KP8vwmFNuDlm*w$(TqparaFczo z6YZZncc$b}4^W{&@>kZXLH3lhxKc|L;0xy}awcfPV@$pte#@T+3iZzR?=iN6w-xJG zZ9XWVR{^8jZV!S`wDoSQoU1=FsW<$ZqSFq7b{wwAa`d5=2_4@{68h-#Gfl(q>n-Tp zW%k#9qT7INoM+YwRXtjj2vA#v9iXK-Y22|_2NbQuePN49MRt+L29Js+qM#i~OBMlb zpen9L$FJRlR_h-1Jr;>TCxeC`i2F65^R`2L-GuMaN8P(~%Of$s({^N7Hn|HZc+|_N z@W-LXFqx`jhg(qf=ddNw^(Ih?Rl3*8H>3At>92lmyajjV+RtY)BSgXL8BqGV8|Vxk zk?R;~0U|>NArs#-kpV%zeb?JCkTEm=_8i-LTO0L` zv)v%eJ(d%cu?(gavk?G<(OngN?>zT5tC8Sb;QWIZR*iO(^m>$M_1;QZ|B z>o|&cVESm1iUxHR;ua5DSZ>0f`)GpTOk(i-Ge zx*=Lb+YI*X)z>eU)}g1rV@FomYS0{uP0%Yt+;^DfKJ_31=kM#Jg?`C2AEpA7W06%19`9heL=+L1#A`^x^dmnV+!;)GKV+&I0*4rn%kPEnmgaUUD>XBcd z?1f9>5WO01x294~1Jq{2p%)(QCl12jgx_Luu5Z!zn;6-!$a>Vd{>{1QK|io2zk2TqRXym8 z72UdBfUmCH?1?p3hN2@8D!nN0GZxtn z>b{D;?awYklgZP>S-7R;D{oh|tX?j7bSHLXz^)!m|Bl^QYU~6CKSg05Ziyu-tm8D) zZUQCWUo^e@Qx6V~bUII;4h8fVIvyWq>p{jgYcn?A-+(`KBp>g7DnSh@_frqX@{q3Z zGU=1mKJcXA-go-yHYB$q8}O*E1>A?v2n-0{A?E&q0h;!AD2}rDVEbAzICq0PfY?ME zcl{{_mw!YZKWIHE46yocP<-Q(JyssU_rKKt#--hjw>9+Q1Y_W@!* zy{uQh=Q|b1W6l}$*?fN!?)c?psyxjM#AS%i|bE~vEM#H~OEiL3qK z9bi7QAtj|||9Bk$By%d3Bv8$AH zFJqDC?ipr6!j^wW=mb-mSmL2A{{-OtZzmV=$e91zQbnHndr`GzK#a3eiEkG;CUN8id3VS7`wQ$Z@|#U_+|`w%eJYZm-T7y-f^ zoJT%>*Wgv5PCS=f=1Oy@4?5u3eQlZ0iO{CJz^nEa zH0CO!!dmebjd&tOHi>eSbNl?#_%Za7lN7FGfh-hC`Wj_qJ2Z)ar`KZUm}RAl?v zybU<8R9yB{*(hkl*utw8O&4j+bm8&I+-3KrB(0BQAuThjKp>cB(0DJb<8g?$>< zG^c7nDsOL}o8K!$(VKZi&t?mO5wP0eE@?vgcy#Pqd=A)olV-ekp%Q4;Nzb2idxyxf zE{962SEJoRXY$OSEoiaeaPXjcIEp#9YhI^>yOTOouU&I~4VHeA*X1^+BEEaavMN1s zHG4r|+w4aw*wwVWlAzoO?D|qY32~OBewslzLA?eo^@NqLysZLhqbcOx5Ta*fL4%## zwIGi~%XpUlEx2GzJT1N)hBBPnv^upQvW_jGq0B?%UoM)Lkd>n=?~{8ExKjXO}4& zr{KbnEuu9)ltJIp00h_m=!N6z*b{TZlVXb%AbnabW1F`V@b30k9>kOaR&l?kpxeG+ z)wcb<*>(XqB+%8CY!ME!{mqwO_qF~nQ;FbT4RoI@=Pph75={{L=Tx#mFy+6JtMnXf z9q}BXe~t|E;)zmvI_5h6gxwfh-X{~|kw_$OTnCW>c;1cii+@V};|xa$NCvNeZ&~|R|1J$l&~@@}t^8d&H_PC}dGWtX z!kX-C8q0q__;-oaeeC`F>;F~iqGLTI7`pKHQ<}~sc6e06 z-=*>1eqyBhUnOPxn>qJe=Kg+aucWVtmjAy>_vI$dBvfYqerhN|$1Xy4=I>Gk!FD2t z!t~#zwS6Ip42`M3OV5eK?lD?~5<;QpUTv2=^P>G`MboKeE##$q3*v+MfA$2d?~yBF0GoY zkW(M#sNP2ElayWPzDj;2~a)2LFVWY3W%x;5loOlz1>3o>ca)lr%g! z?g*Y~6-=RJWo>U_@$U$*|1~rY4>7~D#r`?$|NkDxGuMs?;rVZ}|M~s@M1qmxsb4x4 zcK>Q_Q)7E$cViy|tHbTg*;{2ai!u+2=HDi|l1fChQ{=c9z0{kW5 z=>-4tcmKaoaD-x1EJ(FAeZeR{sv^Co7QfNlA|`z#Tb6jy$+_sTuy1up8a^+=s=gxROoOhNG+E z1%ugKtxK@9@ zFr31cjP!LY>s|uSlIwQ-HDQ2edP;?+SrMClowz3++J>pym5(aw(}BafXYaniT^PS; zOtvcW>fm?cs&Rc=Z#0hSu$@+!hNoq-m)1q>!3IO6^;!R5)IMl?o$>{2~rOKe{LpKBxJ&u-7 z+oD($ZU!2nK)cFQo`=I(uyjpja-lj6OX6=_-8J$-tk+kAV?2EkomHfZVzLyt^i>nu zJQ{&NrCXhX<})GB>{XIa{0e~e>E(~TVZ(63YPwvOAOXGcwKlx1tqX&sPLCx57o->_ zeNFe2G9c<4Dk?Q-g4?P>X^O`J03Z(6F!9yF?3E?wC65h4PD)+J_b0|+kl@}mJN*f0 zdZO`6I`15ORDXrlqo5A>BiXC`!RrQapmyP##3~aZK6iW zb{?*U-T(X~$p!_>XcsVFY{lTo`~eTCa8%}7| z1Kl|vKo)n%%WN?ga|k5b80(Kl`IZOum`)gIp~;{yc8LNIFsZtDq1j!3wXO1Ktl&z|QY z3-Cstr@3vqgPoC5k;b;)zydTccj;7e7=jX($!s@HMFKs`U<-v|b7Y#6Xx87|h! zl)9^<0YX`UxciJ1=$LTe3G#GD-R0XSOymnNHQNE5Bm5Ik)a@a|OkFrSnr$xUR98>fromEeqe! zKIo}(?E4E51)!IwyLUjbfMF#P*ZcS5QT(lxIfA$rc#KyD-4k*_A42-Yg+-J>f~B!T z!SQjZ?0M?mVW z-%F))pMV8rkKF!T9mWa?H#7X09MF;9FU_;?{#e*=Li+3rR3h=^BP+`y z99N?`n^NxzB6Yr~Wr!MsSHlZmeU?M986pE+Y%Kv*8ecm-Rvv{pK2YLI!RJR;>RV~Q zL}x^~9esdNM#7y$Q%>&D9;l{VrIm#D$JesG@vdoeBsr~NtZ;e&6R{X*q)yWSGwErVF)@Mx7djI6o(>BPY(N%cz~)9ioAKeKVEV8yK8LNqHiJ`>cw?Ypox+W&OIK6 z)M)};mX_U+RKReW5udB3AAKC6pZcPM=T+Agm!-hzx5JC_B>nJ$Z*I};Gdp* zHx`(Vm!+B@>cwX0Uw^B|`{MzPOJ&pHVQlLzP12sQ3u?Hl)=tK&47iI^9$wIRk7+YX z^bAr2fFy;n!#G`a8oNWSa4t!NJeoZRlkn)9SZN=FGo?)FD(cgi z@%eMeY9az8o0 zgx+W*aeLOI33r>2J#Jr`c8vm0?(KGo>5ss$_ar10J=2iS$IkDI=LB5MlE_;Nz+F3r z12SHlqDVB2mf&UC81`oP(#P3CXFySY=Csp#IwZ~2CBMM-0v(o5l8#EA!iIalr4lx~ z0!`N+N$UK@AiUHocGh$p+d8x?#r-h>-DF67jUCGzUSV!x;>v9g6~%@jO?fH zY|LX;YZ0Pl*8T8uXaDRAx=3)ZZ||x`gE>0iV>&;#tO!Z`W?M}RH2^8^FaP!pD{$t= z^*$>&t5B5;1Vd@m=O+=3zk^!jH`pn>7Y{wjjvEZXnZaBa}!t2Z&O>a@Q;}Zj!N1!U2H_b6YEw<3Af^nA0>8V z%X8S`fv&kcM-1ZWh}Q_A)C8DVL1|Lg5X`miiH}`ZL?R78#4^v@B8zO|v9f4yAm29Z z6tt3t+2`CURgM?Ot|V0d@rsB9duIe5hcfd4nXFQIUwnl-LGvW7jC2tSD+t5d~WJVy8td6%7OJS&EMUO*5APo!5e~`{X zk^~~6tqx=p@%^!niT%qbd*DjTO7!lEBe>Mca?7qX1vxDdUu>}Nf|2#rz)EWouEw}z z*VsG2(3(=!*rEV%lKC9b)kQZ%Si>b%WrDxH7kgd1<*9Ib@%*Ktufve|W4y<8ZAoO- z*Q!f!Ed_}AUCJ?A;RD&;L}AWD?x5sB?xtf;Gwjr;ra48IhE}Mw_#CE#(6imKp6+|L zz~-74d6-uhJVvy=crQE?B7}FHb)yYdBoKUb=a7see)d`4WMCKFpX_nfxBJ&b8au((eTrT7Z>`c zuF*Sa~$d8#n`9d5b5`BO&o{5Pb?8!Y4S#QsTYXuaPdQ3t3`~*K_TRYu{U~B>7l{Zg!ke1#A_qM_F(MZ=X(gB zADy8jS8q5Ag3}cnt%+p*h@4&FS;C+JFnBppM!qP9v4qJwes!|KYH#dk$$!T8t0!KT z<)j{9isz@~hL{_EdTyQP-ncEoEaopW;P|8eBSR8m{Y;et3XzAXGi?U2l-SgXo{$N+UVB|{|A7K1|M)qjfp!You5-#Y{fS4^ zWy12t8>1MNV5ynFT^IDTq|xH(ZDo+2$1u0RG6mO&rgj48@crtO?OYS}G`6plBTPtP z4L=V}8<^dghaE&AX7s_8keJuyyDnCQRSSOKtyp@8E%=}L@%B(DX3GBf%o}`u_`Ntu zva=Th&I{!J=~{n^hGKth8oU^Wu7`KupGvnu)w$}GxuunmZ&aM*8*&*3$^gT=ZDjc+_fV4K5UsM`xC=pG%9u)fuHokfX}m;8$LKb z_j^B5HhwJwmVSSjewGx$>1ero4<&lx6@^UlFM$yteRF>Og#CHIy>NoPn05%>Xgo5y z{!9vq%mJ{`vVPnHtPRA&=;yh}cO^*`Ww0`qT{mR(SAI zHpcsO6gu?E@HT7Z9QN$8ufvl-Z*+Ug5LHu3fxvLytuCbj=xF6@aVB^UOT6XO@-x90 zwNm{dp!QHleSv>?ixS^r_JWCY+AoEGiqIYF{%j%C;i&61R<8hBOTMRdDNI3mQ8tFh zUo$brSB4A3NrvE>UBBw#I&W}jHBl+3IT#hb7_iK}h+lsC{>qnXUUhW%OybLir6TA( zkU;(?D-5w2kS(XaRR`V??Z!QuMqn)T2jk`F39KW_G+1o60xs*fS7o#b0UCmk7>XHv zv|z|~#QjYjMm6LrRL*PxWU7iPx~#mAPa(%`5q!VGuTNHIi0pzv(~^PG+wY;}{%`AE z{GQ+P2)ol|I6i;kQe*Lz#T3?xKJuDlegw2?M3{mPFTpclD#}w(9pwYX zpa7M%6>hb;>W@@zowMu_a0 zi+n)fs0o3qxfEErKu>!`SQ&I9d4pr7g(-SOEY_ef zFb99^i1gpo2|-bOl0vQ5yRpYuN38d#5#n`WG~nD3Me-q)vB{isn5@+=89E$)j0%Yj zXPv6WepmnawTjP=yXT&tAeg-if7hDl7`3?q+T*m8mvMgg40XH(=>u!XUu&M2gY&y1 zyuq_NzIAZb>uSm;KX0^tJmSHlIA=to8k}cw_Z`;iI9U~Oe;#to9T}O$=STWWMlU5C zf23p{Ff@JXhj%|&nY<6t056CI&UWJXoRHir?~uDYiZ8FzxOyiK-q)M_E$O5M`H;Sq z0X{!=CwC<7oS1`d$x67p98X1QrU>!H`KvGw+P3dFK4*-i2&mOugz?`x#V2uoxA@sq z=*+A-ekeJb;3JMdqNukvt5XW07)ftsOywZt=H^L~z99tsC==a3!{S-9H zCX7@{ufz802%ng@zXahD;YRy7J`epK`;ao~BFs@{WAL7|!F+5^Fy64C19yC8W2U5E zAm6^}@X6{vNa|53zlQU>G-s**Xo_`V@`_z?$ME?Jcb zl|4S$qL~oG<7eul01v@E)4E$>(Di%J$;%6)n9*^DlXCc6O=BECo#N$-X4spH=aZzs zg=`AuZbAgbrdhXXKP_S*5XuDE^M+_W6-*Wc=ezWVb! zAG6~j)7&ummZ}8&+*IjGVJn3_+qQnuK`{i=$=*J(BXdCGpSof9vcyRf>t_Q!+L;fNusg@i}B0)Fy(eJG7#0(#yNpSb03 z1HF%kEZxTWtLWqTVkEQU(BYl9`Yw(?+Ty!DJjUmT)R+K^FOEM%Uvx|c-5i3<2|qsm zdAbPml;Wm5qr9;*P9ub|FCsu6TjQ5W0}0T2?xCW=cPC8Hu*EJ()E!|Zg=06h>am`C z!;E=nLjVc4{fV;_!|=2ig^j_cCu*Y(bUMQAg-A6f`O=qTU{O~cL> zME;})FrMgx!j|U_6q)e%k&Gklqxd59pmD)E@kQ}6U|%v)Bx^QGd3TG zbFoIj&$o~EDNlkJ^l^yW$T{3V@T>i(LVY3&EQXhe|Aj?eb&LUDg3O9 zwjO{-GHY)7bol_!p!D`*(q@?INC{~TwHpei@;2+c?G28mh-0zSWf&6g^LBk`54aEN z3Zrr&kgn3AX5MNT`uO|yT9;8Vrj>EW=@4lWFk;t^i%9cE@uxpdK3lX0S$(Gkr`R1q zmgSLY>9Q1*{G0yt(G!_iq$=(3(XW=!@3o>M$GLf^y8XW8dTsz9vO$yBlsGS{ zQu?0H8a@YCZxL%8pR0wK@4{ue$X={|cfPs(ursI&J@II&r4*9zH>oi#=EK1YQnpb! zL)dk%LdR`_D(o1~O*3jzCqVh^T};*SNFWhm-udy@Jba{a;Z!M=4w5x5jM#Bg!E)_cM%c=*x1R`FuXBeX-zfYIb^3~A}>+Q`s`8G^9 ztXI^QUkLCM8^~E+hv*UrevWv6wON)Sj?X#@_|Iv@wFCo=sX z!;p@Q(8mlMpOf9_NeuehiJe{g!>piT4T6R%+n9dHA>+?UV>)iCXza#n^`xK#@K{u; zZ^!qmK;}2U(lZy}^Ui?FoizIJ0n3J&pS2I-Pa)d;@=6tu57x3#@i`;$TlWOdKTE*y z%M?WBvnF7+rPsWYtpc!5&my{OB#A7`s!#k*vH>GO{m+dP`GCc$#NqUQDL@|CE>W_#=mW@U->*9EO1##Qd=O^v3=49=Tp(|Ue10O7Y1l1Fk+jC&krv4 z8d>XW12FvF$41Y|VK~oNI{%b*4pLLTF3d2DKx=2>xGO}hq0lZ?ewNe%kY5NKXu$a^ zV#$5qV{hl7jnb2>L0sQ>XOeUEL;DPtbzOu#_fJ2}&e!66pl1iCdSA;i^<-g2grYx} zdVPRjJWCCY+ycB77dmx;)D4mP|NMRuc!P18jG@fpNysW5GG8rW4=(N+NWIL6KnE%O zmjC$P=*qK&=2a%KNiFdU4{`pAi))EVCEgo_o!DPUUa$ugrhH-#xEz7vvQj=B&R@Y0 zzYfFMc&uSW#y!^j9hP}`^arK+W$ejRl=?Q#Ukym*n$AzVA(`mHgynNXFpu0m<5S5z z)OwUG7!flFPu;lea7RuOsSNgYIN|)2SJz`*dD?4$Y^OZ$+^{?N9GCCg>neoBecT0UDh%aV3l9Lz43wLp-Jxu$ABU zG{N^iti*LGH7|DxlP+af7^ZLrNqOab7gnb*CFh@ngWWG-;79J7i@njJ z#BEAuXBsE)?!g0cb)3H{VrD-Xj8(zTwRFKRM}0s_{Hf%hX>+hid8pw0qCeUnyC<^E zIfZ=;J^f6CAsjR$+{ABwd4(>T3;7?$`KvoeMb&#fN8q(tF0ur*5M*Lh@BQKC0;W0J zKb3>?SN&4(M|71aio0>Py>@mAl3L(brKLH82Ohh-x$Alu!EdAQO-wHkYr{+ai_dki z%XXc&_iJ4NpgVD@7T3@HG1hwTs6{xUDwec}_46YZ zW`Ul5WW#F$yZ)9ZH?fsoK8f!DGxxbwf4!&2Mj=b2S4r)`I#B5t$p>QoM;nkyB zjF|Mo_pH}W*C z7DgJh=Yx;E_)@-?9OCsT;YOjeoBk zZRZGVA{oRy6b7a66^r{1uajnNcjRe@M%{$e);> zhhg$+oXlxO*m8o%6J=aKrxmX@VsZCH6#|}j-G-&`Q-QC?NW+ z&r<=ceYEB~WFLWrw(SI7!_z?V+sY=1Mkb)awp5oquM6^7n6&<>stgJ$FZ&)QN07Yv zM&G3q0U&H*Ca35*U%GD5j+0mmP#gNmtspPr(C(;Xg_R1%oC_PO;2++WggZ0$%- z>M-0`Y@F?u2}g!k*^7VsKfAE26P+q0TER<^*e7^!w-F= zYgxEHI8!y2{07%Q(&CzElX3ovy!Ea_rOzn5#80?U;yj7fSv-GQX|xDk*GxhppGSZ% zHj18yec6CQGtI-OlvXIBCtM?3B89%{e;srnnS*yS{R36`wZYH4?3jG5Iq1S+_#_tJ zuN+Evrhi?uLpD#?B8=_x;f!N{5a*dFRCw6cx^MV7y!3POQVHH4>op?*jti1th5ZG& zu=iW63X{qf+#H2>_)L%YCmEvyFs#1ds*dg|#x@-&nZui*Z+iyag~02tdY28`gwP~g zKPT6a0^sLTDSxIm0^eKFh~Ei$iV2u7iM>g|_bVl?Ie8p^+)g4S1fBSP^}TCZgjE^v zeB`7*enK6+*d*JC!SM=5qug{!ZW!VzUAypZMjiN;%Urqk%?Olpo=XXgmB127j%zB` z%))hkzLlznLST$Soh^Gw9}z9=i>|uWVxl>9X(~AW$SZ4jvTE*y^xQgt5AH9q_|C<4 zi8usiEe|=|xZQ-!yZm%%t)K^jN-U>#aQ;eAEx{$1R2#yA{GG8mEfPei%^;E{KDjnk!+251ygt)SnL-FV916r86AP zUly@QYr^1%xIUOe^_r+#O%<#)F|v=|az?F%)=H$*!|=P#Spw4x3&_LO$W|n#08||C zb4d=8=zO1al0d!WVk;Y$BQl+D77x7OkgqR&mNVbn1ZEJd4ZGt z6R_9TDtWBw0hs80n)q}l2HY7G3unUpS22Ro*O@cAF}>B%zKwF+U($NzC+#0cz!yUq zvwg}CIGCC1X9qSyTk8!`{Ubi$`HIcFp6ER0{*=2p^^6MQc6`h>5S)sMogEf9kMCFU zx3vApM)gp4b2Uk%hbhY0t*|O+8HK+nA4JjN_+ySSc-*n18S^<;n)CCk5&9|cxuSkW z6b;RL|52-iP*p)b&%V?d{G#sImy4@|wrYNTHn{)F^#1pl>RUZ@k40$>sSk4N{JzuQ z;EjOr;omtrlkmKCsJs%cf3PL~Fx|#em-L0<>N$LVbmax>FBRltJgn-iOh>z5aIf6s zC~YA?Zu|Y=2i$+EBedQXqwBXvkV;6f&(zjJ{&%gLEMeE1v)b@Jq9OrTg^^yzkuZ zqK5ku2i~4>KRC$?Q!lDX<>CC*L++ucYYfUDgKN$20!J>+)B4!((gpwqUftRF2mM$e z=J}fB{S0KUBx6GU^>AQ$)}9LL!HARHj^yGUn8#y#mF9L!c(?L;41?}CJQg@37DMlV z1j;Q)-{bg$OfcfmC0sv`n7DuA9K>}_ncx;j++Si~Vi0{n881 zEvG0TuRmkvsXYVTEk6DF!{~*{ycohPb@4-1_u5S5aepEUOB7@MLkY0jf0{x~GaWl# zKAOM!&>b}kutpE{4Z&514|xO}A>jA}3~b5w z!4C1V97Y^}taKl6^duxe?#b#c#hpRyM;4#M!WnlE6Xme);xK@PO-vSaE%;+P)59^^ zIDh35_>jPnK@&*woLu$5dCZuhgE4~{MRZLeUeKP`7X9>cUThEd1{x*mYBYtl*mu#* z$q(=9V7;w?6JcZ|Xp0&a&u#Pu21C|-&N%*f=Nzk(%H*O61WZ%&f=)0=9X zzhWbmAzn7jhpkzXnEE9P;7K;YGK%w8Ln7=~>o?~xj`{>?&tnVFAz3uj{zWNP)^YRb zqlY7qo2b~OXJHzyt~ObFVE#HaRt?7=V~;FLMm~fgex8ip&qft+`ppZo%Y;d|O*Y1JI>{SJ zu~dCiU9kthzq)h3IPC}onI0Clm!u$tE5Fk@_~&7ZtMyDq#{^`hJ=lb@_p!c>t=gV~ z0KjT(l+@bqhKl!M99%h1U=HENbCfub-5d9e&G`HxWO#U&RkBJFu_*G{VK{#kNJ({~ zH;@ky6d4u1?{UXda|Few&P>AsT1&g(6KQBBm8OkoE(p=5PiCZHxc;$vL<93y!{V;a z?LK}sh&eZLY)Lla{FNoS+&$t+NTvK!^oq+QTzVJxs-&S6qbJ;HTKeq-Vjdmbzi4HI zL2>EMk*#@1qV&?uKeYmW;O8_l%jtnE1ovO19CpG}4s+NoaQ@1huZ7>ws{@k`f3m5J z&yNoT30aTdEWkr$4Tr<#{gJH6=XVJN=~&uV-A;ZSe;j(m-}M96&sEOmi{TOnO4$4T zldWwQ+uYOhH&G5j9GvSX5^%j@(YEq1r++juUavI%vwaRoH$-a~F)zZ&@eQXjWht~X zy&ctp`w{EYsn0U9YXcHbx}xWw#-W7um%_;bIYh-!cK9fh9r_wnmvmxa0K4>2G1#0T z3JI%(+Jv}|LhbItUx37=G}_Z3Oea zF|ZO4V+i;^^JIzDdxK?_6vZf9A57q`NYzrueRF~hd@BcPX!tUh!miO6B(~bms>Aic zeX`X{Y6a?mWUUnjA6;@tl*Rp~7(?CsZ6ToL=i{_Gr;mz%ng{;5 zHVJDeuqaw43-HZD&2`qn3puLxHVfeTha)Me>{s#%*hd|H{Q8skaLBOmBwH#S*o|zS z-^J(0@cHW;wfu9K`O}iQ^`8nri2E&}pq5g#)VQnrHm}ISdJjAL;hT_p8zXJ*S4249qaJUW7@(8vMXy!ftWPBh~@U$9{pT zsO}x(6USMBiazWNY*n4HQ7utwFXGWl8iU?@dc%ybXYx3YISsFJMmA zpO@{E0yXAFYoV0NU}g6h^)lWcG}4945;u*I9dQqX2+r>YCr_<-am-`yYvYW_O4_g} zG3?BPuiE3WspTFH*3FE|qVz{KF-MR&f4GB!%*_(OE-4-h59wYdr=NJ@c0H4uubJ^HI9_ zn;zfm%6`ze-c&)SL(jjCahiv_P5N)1T9#bk1N2-4N$66su`1;i!nO2W85QK0#{ z32Dj#R4jELv*vvSHYA%_#yV%PPU@_a9J^|W__0)4+B0WFQLXgj#xX%4^SQEG_l*OR zwPR-giu1cc$2&qS@cvjj=4{=4VFZ%#s#g%oT7%=~W+vKy%b|;s=j9Cz@%}Ilzb_^# z0R&i^joEPgfoc9OR0tS>xjYR0B^;BGjYECo9gaUZ!RxY^097C{+rm7B_s3bbO}6lP zCoH-1%|?i2GDf#*Ez0Pt0FFLR);tm>iN@8ma^-RTT$6@xS#gvX48(IZD14Cuzh)9r zeStENU^0{S!uc!K81uSM*Nl)@%j_dwe16C~aL>KQU-zhA918W*YC0)o(VN7K3_d;kv96y^SgIseBM$ecKs}a=)xujPq9@u z2gzBGEJ_p!A{kkrBn6QSqGS{#h#(@_)9(G~`Fdz!tfI^+4ybo`K`LOh5M!g5!>d(%g2ycs>6Y}0 z@UgYT@^4W{i<;&&jX5`0ynPylDRoEz7VJDe^IDJ|v-p~l`)2YvR>i)fa;>-qZ@KRv zEyjfsh4>DhF<2;v=PXT=swOJ24l;!}!LBgunC@CeE86&Xo{*A$T&4t59Q5V+sZowM zdX&d`m+(`g;V8Mgr0EP4P)a!bnzRt3FROD);(LTe-C8;Qt+NR~<$r$N$l3^w%;U49 z5Ji{M-t?kh)d0hcB10B>;_=E+?VQ?UY_NAi zl_{+%#rV@4K^ZoCkML-#K?7THOT40MOuYohB%}yZiF=|s(HKedjsMb%Wc(T8#6+CN zmlvgF+tkMS(~Vu{C?Y$0Uch&U6p1FHi|~m|l&!S&#aKW2)V;$kS(s$|k1c|iN8#He z9Z@`UrC6ey|1B~NQ@m)hB*$FLL+q)4o8^bCca0kt-UpXRKg2&=;L9%6X=prmp;N5G zrvP8NJp17(dsk!o+vsefxdNE}HdctRIPbvLv|EGL_=#v`B zBXc~_n4Gg?qR!uTd6M{(^tzX^Jaaef>+8$-b=rG+8Fd=})gDO9^KC_%V#$qf&wgt3 z!}xXL4(fJ9;@Pj9RU#8W*mG8^yNBt2p4(;}GFqG;i+5Vi2@dJh!KgG9Rm(Qb{_DAg z=Y6>Ed0ocdg=t0JTsHpuT}NDn>vrjFFh90$>=h`6OZKo5K@^H{{L`O5=fjSQspn=G zv1rLxyg6!Nf7@TI^&4&?y!N;L=RT$n&yVTg=Vw{iMPl9YG?XS{lERYMhtJ>M!WtL+ zV?zD3dBlNn)_Xs`_c`|ObCyUY^XPBbV-`9laR$HC@YknOI6HlA;_qoIo}xXVfO)ru zRVFHi{;mJ#yX;2D2tOi@co_e|bkyl4M&(7o6^Xh3SG}%t8F{DgXkgc?r%GF`6!DE> zg{cp+95AnQD_>n~MKIB3oJ~if5~iyj(n2V49ivNf8@i2r@GKuZ4aOwR@ZvX=I2SKC z{_W?V_iwO1-L#(({;&7_DX*fJiBa^F#Ryg?sihD{jFGa7cdhvUtB*E3Gk&5LZbKi5*E9HqBLT=9dthfi#LG{U=x ziI36wTK+9(`J6MkhB&Z4^{0)r$txEBHy!k6&wvrmnghh`asP;!NhFc$zpIJ=zgPWk z8)cT#rg*{Rno@B(33+h*v_A9F+BnpHt~Xvo76dGhQzkqcS%&0%GF60BUZC}QuFuTT zez^LL$(y9U9!C6hYI#~80IswUsb%>wV{hnfH9qV`0*xm*qLD>t71kh*sHATaQsLy8 z?OV8jmR(~O|Ke>Nl~O+4#vBE9>ug}e9!oNGzUp?+VwxUQiq}7RFR~5obJkC%rS(9E zaPPA!kGi2o#HTsd2fDyuFIYah#~#p6oWA@}p8$FD&&-)PUIEgFsEBph)?j@Po5(AX z1-SU(nDgDCKKSO2V7F1pNuYY7XPFmIiydyVD#=pXf&DHd``0`OF>8^_losu>fX^;C zQFxCQjHgaA#1aj`J4rqZCe(yjV*Ht#Btv$fSV*6)fhQPzXMDrxyR-{GPqi|Pn~`8Y z=^4((WgY}~chi1G1sH+-(h3v#=iacqW~fH#{vz}<3*pG8&;l1Pjkc+nEx~ZH7gx`^ zg#hJ+L!X&#MqyR5z8TiN23wden|{vQf$`i`=1R}eYC^vGHx!ZOkk%#ixI5t$a5ZQ| zST0`|To9NHAF89rD$HKlXr7*hW&*!f616&Dc8J{zDHIh$8SzP!YL^k)cq@=hU?~PZ zZqKBBixUE;v5`yTw??45)_KE1-EkN?*6Ynb5dfdrigy+1dVwpqhj5{HMFE&q8Aogp zWRy^|Vz2W6I+H2K7HOHVcjMFts!OQ>|B=_4qS@2Xwd6^-P+%8KziOoII(8dm=lzV+ zPFaB|UFR$ct*Jq3XM|K+sWSjX)0Y@^d_lk+Rx{;*0B}jWrS14yIM_cdM8im<1(XVj zlqH@|;|2?^?7DG}!JHe##_T1GSoSCR?3}hvc|noT z346C@)j8Iq{ggWs?RSd1cUo=U)Dc|g@a8MR=gdM0^~RQh&x}AW7j81 zY-p0a;8UXm1vxY!Kx|JzL~3UocK^De&Se`4cGx-M(XK=`YD>4VD@nDt>iqYHmB95AfION8ry$ItF@Y6@*bIzy@Xqscw6s2tI2 z$=%Si`<={d){EejBGGD`TmWE__7x>JCcq+0hJG{;I)E)!+;!C>YmndNMEG6a1<1E_ z@fPuiKDhk+g;{n?2(bR;&DYjUi}`AuiXT?offV>JAv$h^SVit8$Nk&{@Z6%Z>UB;i z5G5h>t|A|ThHNjq=#hMA5T3|1(BcI_g5P%yOv1tW(~5b$%e&Cv3Cco;b| z86+PiAqkn8NIpuW-B&!3e5|N<+#;=*hF`l=tDTX2s2GRGm`K_IwWzy&+$2W8eW_-; zt%e?3h#MjK| zM%@MGU&2+s6?lRr(R|LM^+ceXF7#}%W(-(cj^1|_G6RIJAksHG0N4#BH+)<{SpfwE zaO8qxFe#(zEf10pPah-IGd-QqqfTfL=i&$)*LaV;GaG5X|97DyT-EcmL|{(_8piL z>-0$gK(}UGb9)@RjCVKc9n=Oqqp|l*XP2R>?#`!@9aQ;uN8WJRD_d>ax)Tn-q?>4Bw8Ib;VzyJ5%WFC4v- z%D_P0$l^(tEGStJofU@*sky2#r;=vThp zmoq90)PD*u+B<_S*uJVnV)1)3CRX^_@O~7TKb>-QXGUX7I z<|dFLrX<8JA9}OZF>)8Y`j9#A!to2I>&2oS{B0MCYE*bNS&?A7L>2xI)pbErAQu&( zpFha@(bhU#;|*t(E{Jo*E<#y}mpiO`>flj4*>^_cCHVAEQ0cU`G8ii5mpWxU3T^MO zCCId|!QOW)VVp%fa8V}Biyz5{V9g7E^_X(FnLZWbOe})S>$HrxuX_#LaVdy!tER_z z&c~g;hUBC4(nRMJl8=p!ugU}dWpKIy_o#fE5hFa*e=_>A2e>7C)BTl=CCH*SB|dd$ z1ZH^l$lo>@hf2j7?eR+i@FZM(!1U7|(5UjpxVTyYGQnpPQSS$!!3&n3p`We63&QfQ zJR~2w7oFX@9=Lco+QniR7+)pc|<0xj?R< zNd>y7hxf!yZi8>X<}8PT^g)6R^ZVBglkmpGGWBMV1W>uJbKztp1-9=M-#%hQjOG83 zktMoIjWsfPy~z#q0+Jc4y_6iGpnF5N=H%l5P}a9u_~IowmOsa}5ULrMWhn9R2XS$4xHhqIyWk87V9; z?+?hCn$|~sn6V@_R*Hb5PT>C7Y@|?4AH1PS_2~tYkL^c!F7`69;OsX4SpG8yARD1w zf8h%S)+K-SGY>^F)OWV{8h3>fxYmkFyhQq=f41z*gY+IqXMXAPL|8YJReBe;z7-Dw zuY@o^3Jw9%6hA!-kp4)S$lsddGX`(3MA#D{`>Oe>jhzO_cbZw<`xtEvFc>6_ghm^rx7?tQ)Jzz_`o)o>$* zoDeG|Q&Dl(cLRZj*L0K7?;SSN2Eh6*eDKV*PR^19t?uAx3F33?sl0)JuB$XKAB7zEZoEx+y$0p5 zZTbeJKWrWndf%&?hNn{ITW%xysJuu+6W|^XTw0@y`Zi60y-)Ii3S?jHx=2bVbIw97 zW+~GyIlnd3)p51w*i0b(lNaWCR-e$lxFq?$exTQ*0LuH@VSG{@5JovmO5o%S?23~)9AJUXZG9S%P_SwtEIRyX z7#`v>&Pypv0O4Wf0~vvf(3#1O|??C@MI3~D~Xr$NJPfxYJr3YC$L3-_{qQG`5*{vZmQcQWd z8@E9h4ZX`ADBa|62Qp3f{i=$Cp%Hz3m>kvt7bgPsx#hay#Ix4-q_si-nu?S+KN!NL z`83ly+3mrzFLLUSKAeJ^4pq{46(TJ7{(a+Py|XYy+2JrvdM|t*@FrjEkS6e?Ie56u znh`~}tLcb5z6!3$3Ez;)`~@4&-g#W9aR*fSCa$gNg#tQG#o^{uM<4-?&22o{h1Gls zUws@Rz-5c`vp))Up{-&pW4q)5tkQ7vMs{}sSSmk3wc~ORWYHb-dM#`KGF|BByXOr- zdu;2|7PEMezr5T0z>@&Hu6?2{KK%*W92q5(5%mKKR7G;+S_5!r%aU21X$5AHEa!jW zP5`*NBq|DlUbw*fZGrgF7L+spWg$?o3N>D9>wRMr0{yl^=kNK^V(yn4j!1D#Kyj7j z<#?x7C{x2`lh0EDNm+7%I@e+BJ*S!{%Z52%>I>uA?2ZS|ygBX|Q856iY6ewTsWG?{ zmX_1N)(maTh^Ys%BEV0sv|Vd0L9kfsekL!}6AYA7YV`z$z=<9uPZxa#EPOw4S&|_g zN;K1MtqpyITsN7;-0AeeQ<+V_xJhQrKIPSSUHU~B)p{a%-eVQEtzPy!PGSM{@_a+~ z*}MU*9}jQ;=>&K-&N8)o&Jx(TW-4aXn1D3uDvAURdyselm)21>2p8blAJu|d_1w z+CDgRO4I_-=$O-6=#4?n9O@Hv-y%Ti_cNGNA+k4Sep0x`6JQF~MOp+pWEg2FLqO%K zGtklgqBQUS1=`n^ZIiG(>^%!Ql&P%V-mLbxkTWql|X!3EHCV2X3L!5Sj0!zJUqPxqH z4EMXb-hDCD1Qkj3?ivW6SCzh|(8=zBE_^w`jRoCsF>zv!r5@p^;}#NBN(sR9b=QEa^{tJq4;`B=3;@FMb1BKqOudOKgVca6@ zQ=hv@v=k3M&0Qtmyk)hK*WF^Ee-`omeGG-Rc42Y!hXJ(_`b zcXCTsEiH{LhgHIB0TbQipb}(noV~t@W1;j5eN#)1HBD~u5ucfbxE$uW&}$v=e5Kjf z3cWI@yej=#4dIXDqqw7FEJh$MjeE%}${k#IK>eBj#t3}fUVF0Ma2!7NbzCb!`20zs zl50qy6X+{H;w4rc0N$xMPbd-nKPC;HbD>sxJ)_CU%~Eb${On$f#eva4mBh zc8BY+b47JQrGnIB?>d6O#q%E>nlo3R)bcv#4qXTk7Up)fUE08HD9tUrRl+SO9Y=xT6{fr&BmB{^;|q5XKBufwx6GOs27FJYAMj3S0IA2K zdYalcV5C(x;TqB(c&@ST<4V%d@wm(&C~yU3!%#S$I32n}HHHVKdSJ}mp8e@eF08`c zd?!SYHcH66tme}m3*FgtYLC#>!81P|C?4oE0634*`uFKfn9}K`;ll}$fOxE?hOD6v zUeC=Jp?1%N#m|57P^&qCPQH=6I~0*%f4NHjn{GQKsAkU4uV?ZV)XB+g2CJ5I?~1nQSiN>g`)3O4a6-pG4o_j!Jpxo zl{H-(P`*YgS$#SWcC(*WaScg>KKqB7KJq~z7dFn1S0Tn~)1LJEBL3=Bc01p=QXQN! zz2mUh9001W`g5&Y%Yne0LvJe({$Q;wD8kz{KynL_Z9+ONuvOgkeY|iN7Wwo|jp5he zLLtAZ1G2Aj?!Hid%54GCunXqKWa=PcQ04UQ(TP8|=+g6b5te!2*s^L{Ha zFr8(_xCNu`Y=13*eB1WV4mhvE!%l``h0hE@Y9o#cXR83P2OL)<$r2!u#MyUbHKE{V zPxBmW2^+vu#XttDP|(V0b7%QYD(qAkzcXCOh*iIkv6^V^g#C^MS38>HfB}aCqmXz$ zJi%q*^3azEdtPHOQ2n6_e$V%st6GZ%UBpGY-Dxv$o{@u}YD+4(b;lOYGRfYDqaVT5fs+)dN9dK`UN57s~hHX4K$FH6u z2XAJw6$TN1b@szGM^41=KIgkQxAXlw?v>fM9x=pUEu=K6e!^>lSG_b^x1aUF(L)}7 zL_L#G!OSF>PhA0={#8f4eLWg1TgiRpSVZ#NK2LBA;g1@>(L2>ij-dXH<|7iZZMgJ0 zHk0mA5BxCt=8AAaH%yCB;L;pF4@C5q3U=`afPNd}Fhu%8|45nJa&kDRJVUBshxCV7 zyi#~S;;-sYcU^hd-3Lvn`6(;k#)GwE&2)~4-yNFJeYA)0N9CX=i3-9WciwWer9H<1 zlaE2RJ*zH2#A+di8R3t60;{7Q6oeQ)wxFQ1=pB&$m{=)m!#s_xN<-%WWVx7nG1>YpzrYH?LqS;$QLb6 z;aP4DxSp#fxmb?Ec-yZZ7~ZVG+(3>AGNeB=EE-58k^UGLDq`Y5_~S(w?ZeO6u3&zN zmscm-5fD_ZWPd>Td}AWrqJVW4_UwD{qcsJv`f_~kW1liuou2!gYKIZa6<9pw=*9^+ zmekhpvp#?)!uCwyO~mi6{P?)3KMs#5kXbV${Xr~Y`<3mZ1|VRukFeT|17nBv3d4H` zpwXjqBF7tWK(^*RWeURQg$ioEG574jRA-&~#K0qoQh+7fL;xLY`XLhWyjIDQu1 z%DeYn!CcF3$HdVwSg2mr_qdo5W5+LkXh-~2*rW7*pRZ1U|D=6q|KWUiNBT?Xe4GU+ zOYP?#(AkHAG}ji!J!L^r)tgHbCj-F930;n?k90tkwIDR9ZW3Oe`<9>m-XBQ(xRAks z^oMS+li!*tF_xNp>6{zFA16<+q*H!HIBO@j@BnoJn7HNT}`2{WZy9W_8%qf8rj2eV7qvm+4ee^Utg+T@?V6xx48(aGTp)QXx;deo_8=f zeM;e_Tsv%?d)C~rc@hY=UMaZ0IRGmqHM*Mc2ZdHIwl?#Ep;QWeoCMeqf&?*>PlC9Je^59hU)qzbLMoe--MBmLe2H48%&UfGc1)mk?U#Am00k}eX%uEj4 z12;`Kk~kTz1Fh4Q6@;a`@E(m0mG1`=kW`bTy_LHQU*##>KOu7foA=OOEYFDpVICU! zp(2t%0Y9GhJTM5%c+pU?t^@pS{M`O?*s;Zp22Iq{{&~wM2b$9eZlc^ zEqQp|7=R8{tZV#7R-nNJ_Y=Y-SHRe@;-^hKz0h9!Imdj#7F_3PnDZ}Lg|{N>B$=f# z5MOYP?TSAwCdoYf{4?(ad^dixklvydUNnM_15Z@Ib-^C#PbUszUk`FJ?$wHbYc2HN zl#mYGDvopZS&jnd4y1L_i;Y3q^z~!3VF2nIasAEi_YG#4u}$x1H~Ee% zLViaTA3lUXDma{2OqX1MBacb8ak(pa{G(dq+}Q+J!_0I0)UinL`l3Bm#GpDDY>=k3 z@3sNivSbwf#e>iscR}Yzk`(x9dNW(%@dYrY-&>?L=mr)YPmkaO6+tu$u?n#t5%$36 zVq1Xu4m`vC>P5fYMjdHFqPly9?X2&wNrW@Y!C@)F~!F zyLQwHJxL_Mik}Hi(cB=zW_I#7#u!h6AI8r!V-fyX{~mDSX+aFwJ3jp-ii!m5PzJpG zGg7cCBkz-8N)UMc-Xg1uC>_SXOV)KcZVv>*6lx=pePw*Q+bC#22Yju0qQvDe(&E1L-m!_zIY&N1Rk=UJLP_R8#+sWuq!o-f$yyk9bt3} zg74>S?doV9f!B8yn?YPC$SpL@VQ;O0fhB5`gV|FM&htLQz1@I6Ipb8|R30>uy0I$a zo(AngcUGE@YJm6cng;`wi80ThwQ_c~9jO0fm~&IH4zl)fQ;<=jhZPLda(w@m6gnV=keFzuFLiQcF)8Nn|sb`(OArv^%x!H*9s{*EDl!=-ZuXocKJm zKD+e@EK6i`OTL^0aPW!b?Qmu z8&cut1)|U=J{@39sjkm8ZyF3WewRd;-bQlS|> zn$1g%F?tON-;3`ZTh{`j{$cU}*;fX<7oK)}jR5xvf9dBU`zl4;Us!?70Y)t;5}e8m z1U$DamW9)Yq2$3mQ96|vppzqhMKEj;Y6Ke64Z#S2{kpY_?|la?sOU7(RobD5-OYo{ zhOQv_=l3V?e-1$Uq<~A|enFsYC68%0)gOo~xb$*6zlWS=9Trj>jv(t&P6=t&IWSG5 zy10h;E4eL+(-uNRX1C>8lc zYXBx$xR=5s(3L=)|U(j^@rS=@H6A*jo_WAowPvmDD%&1BV2XZUV zet1;uLbCMykrUo@;AzFM(ct4=ZK0$p7&UN@T7UaMAQNOG@0JSrqco@eD ztg?MP5GWc1@IqR7G`zjgL@lP=J$nld^Kw?)Dq4m3*l7|8d?4Vu6Gm5r_^ShFOOOL~ z0yfZVZGAIqg~!(=Sqslpz)Q~i29J*)#ym=wrhihN0U3c>x75-UK&98yDiyAKII)QJ zM=H`|aIP_RRgSqCUhoXRp4Z_D*eu?^^LZ5mG-%g8(^t5F-B#c9C!rzmi&#mYq5%Wu z8WH)<*&oa=rkF8C1v+KxV~em?E^pV*a}_?lkWh60 z!fg;x+=&udqczp6uQ)3A<6*m$|NZr3Ss*$l?S6OI9&llGV?{y1;IzlU@z~Nqn8_8< z8vZ^Q^z3U~Va7N?tM~!o!#pQJnB&AnkF49km`#`WA>ywBe&BW_5P$Vt*np8`MiGpD z()YWcGXt;aOpT~qiv*dM_n+33k(*qJyG!a0*@~~8u;-t7N}TleoZGM!NB4IrXA}Aos(3q$fIaNHWgX2 zGqdB6^``T~w^P%&tWo!Xwee-xwUau2#FiS=$5%K}q4<(M8sB_dee@t@I(4hx3YW(5KL1&xf&1(k|-la%Tatu z(H2JDBAN{EDT-BS7{`JANcQcMg4<9wFD=(PqX!0TY39mhcSE}%?gI0ZzMxzs_n-yw zGVXy?#{HKFf7Iv7-ggNL1EyU=UnG(Ld|K*N&MoBsNZLDJUNznaRU~hY(dBca&};(p z`9|bF_b*i)Q{I8r0D`XfT{;Kq* z7_-aaSj1z#mU_(}1WZ??rzMg9JTu1i+I@u28}IlB`p|`gu#>-LUBhF*Zjx?mVwE=> z>pM>{gZzUTPguf~+6+LH=7l#gCQA^GtYp7Z2?v%;R}#6*MxiQU;3=)HH5mTUGGGMh zk0YBycHGGSF(~iyNj<3?HVRP-eEi`Eh8WI&=z8D`G(F20?2-SxG2*ha`N>)M!)I(V z0@m}i9{bfm2qI*Vi}@qs>FG0LR;su_si($=Pl&%-yJBK2b2t#F zygo;xnlcSp-k!5jz261BSu?r&rZhmvDs6c|`Utb) zLz_BbCUxwjoG|9@nDqH z2jpJx*^>AY2qxcd^^%Rp17g+5W~x@?KX;=s+K-WfLA!5O@2VOCPs-;tk_73nMKq{+ z$TA!xW=0gtAp45jBzo|kUj%3gyx?~M*;i96G(x1O>Yz~l7j7Ki1t5N0>X%Ih6L$5T znH$0QF+i*QY-Xgc4<@-;eR$`c3G>58oDa)g1tw@`$t_Y}uzW~{svX%^qjslUUkQf8 zh3>M-!1uI3hP&5GQEwYkpJAuJdou=FtZFVvx(30hJtndXwq8I?tcsWoFoXJtQSIl5 zzp8a*^3X*1Bk0P>oW`CF_+rsrmvANz@^0^c?(|KAm#3>+hnvE|Cjm3Q3^ih`i;qI_ z6T;_@yZCHI5q~A7j?=3zeTnO+YzC}rVc^QTva=4duiVSYe-R@7s$MoB2=A>9ECb|r zJ|p}=alrPF5aO@Y?sq5JoRJHa3_@0;KoWM35Ie5Ija4+M|T;o%|g!tWl%Wo>reL~;~1@+2Q#9tk{@4({Ak`Av)a1-eIAp2@) z(Pb^T3x+|DNQlf3-CK=Nm9 zKAU5DAlWKSz10!%S3zj!?|WYGn^n{kxnvD+q;k3W<@z`zrF3jySkMHgzpc$*Me$eG ztMg&u`Bs2O5qrEq-wzqTv>#8dsYkmhc!j=vFAjv{XEvo!dZ z$bUYQ#Msk<{2zjEnHzheyWs%fygxDT0jOhQi^!KYak)QPt2vSWNTc_EkQb~5rW^PB z=2_OD|3#+LRiXjGc<$CtxPA1Z8l zW9>n>c?-KdFJnS0=p+-;&CC_XWkm@U?@J`%0J4zkNgCGIs%$n?erD~k*`NCWF3Jv^{!CX_g7C-WXizfG<1^s+J=v57Csz>IvK<7S-owHh zwiFM;oj`|=hmqLk5isoSZ4{39tC;5{+g%*dF!`Zj;KzrqU}!CJv=reFf$$@5ebhRj z1GN_Y1&MCRLuJ8#tWX`~_{@ZGoYn$73wN$0+wZ|m!}Ss4oGGZC< z+E#8iAQqfwA9$5^RSE>({UM`p)eD$~Z)wzD3I#SQ43_?Cw}8OUrlVl$6P&$yXG-OU z3|J^AO~|`C0Q+c<7K^g1Kq5K)B}NA&@Zf91>My=t_`rQlmK@;^9XbN1=(1I~Qng)M zoG1&_Wt0b~d}uL;Z~LU>Todq$Yv`jC>sFYQOtE*4qyl<}9ob_>_`@uqP|r^w8Z3I* zD@7kfc0)tlhVa1-oa9Tl1_9g{v}Zi$o_weoKGq?0_c%ueCO^b{Jd1ej1LkXQC5_{O zWc+y^f7=i^I@pnOSdRfa^me5-9Pu{!o=-$cnGh`%bG ze`r2twgYPyaAJg8Vs`rb&E4y~ijO~2~Ig9`(eIbq1YQV(JxRsZ4-E>;#eQ*^~b=CaR@ z+$?oaW_~we_`EadqT{d%%4EV$z6onD-?|HK=1!kTDeQx?4>AO|Ei<7-~uUE$XQyGT(0&4hs|3hkG0f5E}tB)^sFgJ_dew%9{C4Jgc+4|0;B`Nkbm&d{P%2y<~2xtXgK~cich4k%kj^llm(SW zAJaVp)WD;~qAxL(-jKuSRl_#Y9~lu0iA%&0z$>X^ky)@1Zt#-4xfx0VNUz6am6)c% z>eAD^f=f+MmZzHL71AHfwk@+}O^Ck&cgGw#HlgB8Z^jmc&)bIXQ)@ejfEEv()1Hf6 zF#8vjP}q0~Ne*IHF7T$ojN=^9&-e}Dqt@dY7ID*%>ii@8LyidWG$uV#;>!S#LX%g+#h)xFyi=j_`B6Utt%@*?^0QK(!HL-ti!Xk-f^vab|w zIJhYtj0P_r(Q#pAM&Puor*Ix$0>t>)opDn?>QLNPno@0OrajhMJTht|kp?4SD#q5XOKi%?tY%kblqsZJ+)KQnuSO@i-4?U5&?FUE% zmc;@Q{-`W%D~WON1)T=jKiCld5E4;NRd&dPEUvBEc)u9nJs;5c#>xROUJxUO`t2~i z@;tM=b~waj{eljb%7b^U=Y2U0wqfk?%}7GE7`W$oPfrTTM>hpW$&i>8&~@RNF+3Iz za*pv&RUr8YnH+u`itxwjQhsk46DIo_B}E6Ys>45|g8b(-Tkps|Ao*y|>vV1~ zi2^6*`g(bacA@?D({E`AfBbA^k2`_vN{E&&ohpQRbFnohQTKqZv54b>SsFa!@|<+@a}(6VhfTyGe6GsP znxRuS1XtS#CejhVyFcI7`~b;E%_-N!sclbCbt|gxHIfh7Ue%CaI}agE*kvy_)-)*E z&}tWMXb3f=_l6@7{+O8A$ZG8N0()096r%zHLFvgNNjK9G*s@^2a~jD9&(jneRU{wZ zC4|a9yk-EL$r0boR?mXDZ0CMj6rcEYtE%9ddk7>+Qu544_#-1_?eW*CG}spwPAKG= z1)E*0$y1`c;C8$)s{!J#g2gJkG?09F&#Qf>K>A}!%`{rhGY-rO1o2dk`GdA=A1-(h zCcs1W2Ybv;#Dkrr+RHT=Rk-{0^C9DLySS@!FH0m|q{4Jcm(MsPAGNbRL+K-(@F;~^ z@63@%VB`Yl=|uA3Lt)RLRK#D|P(6=nME-MPx(^1fT|R)`soCoV;;*{jFs6_+-UV&_ z7vjyT(%_2!krT08%D_uJm|7Ld$2GO;GGBy0UI>*|d@+6v2OhGlY|kQZ14l-bQLt#R0T4dO$ zNAUfCJG@|a%6sU@c@&zS(pY$#0b*;e#@0e=XR(Kl4@&rvDmc=0Cef`t2rvx1QB#V$ zirr+sl1OzBVo?N_m;!cBuh9eZ-@V?EoN1!zXx+18*Wph(vn#cSWN22I3=E z)T}vpe2DKB&WZ^Wb|lSWE9AfuO{>?Xo>F3z%{5%fABnJDH5$?Q+Hf%QvLkn2I}99? zHH~+`v0*RC`75*7f$alrTHl+8-JI-oskXSv2IjwNox>J0~FZ1R=65ET0f z4bNVdq@=aN#nhY4tZJRW#_j1u4DS(O6*nt1HMc3THdTk*3(mHnsdQ{>H9`>Eu{=ad zhEzSVP{ze`7a1`sVGS~BH*-*{Qs_B(gbn-QFF92yBZS>1^fk5vtXL6CT&c@UEI3kl z?8EH`j97fKLI}wfWe}P%*uLY=iTy7k^gpY9eV-3CpX`x>e^xujo2xrS|4RFNeYeIw zO&sZ&Hn?49*y(LC^cPCl4$(p&g}2`Na?ima9y3 zldgFP;$K`_qC*@Qp1(!weDUhvb|y>*&nF-bEaoC(_v-dTn3Hpf2;GLPUc`Zs(T254_BQ@kJuSMxWzC@hh`Ymhxf5|< zu@CF58Ig^)-H@NYu$=~<%iewK5SRt^HGLb*BT$S=kL=vnX=aQ{w_1P$`51j_#t(_R z|IhdSc|W=QdE5BI3IFxJKjjy{-@91bmI?(2llrHhFk-UON5?^D=imA~U%IR4IPzh{ z?N1hb2*V5uZ(bQiHrk)|`+ASb_%mkyUdIeA;H<0v=l3TU5oCCK^gB$j)m3zmYliy! zOx(7J1N+nd#h!u!4*B5!Tt6)T!1N7qVE=ak`m^eHU5Uqx=_BJmVrJrE7h;owKS-YZ2k;Dcl%W-1lK?7@h zbAv=#?C81aMOJBkluX&c*~nba3FS`yUCG7S4!ywI#K6(k!q!aB(ZCtqK4fHP>*Q>J zvM)!{|G$d#ENm^D(ZkdXmHSo;cOyyd?cC7E{=PN&pUxe!F>u$ja7Le?*bzzmdmzyB ztnJMH$={5gXKrC^Y+|eTZ!dp;g;S!Uad2WJsgt>jsj0O|;=M>pM*~}9I~zUp3FvE7 zA`dv*S)(@uc}~fe9QKMNMg23hb23Q`izIh5aQvs9qcgfXWMlU`akPsKs=U%3$|tg|5i+lq*nUfK{a*aH2G! zVPa^aMk0?Qp&}8|L}HN#jeo!YcM~EDiNvsb|CSzzB*Z1EsHiCXLdQRU6bX<;7$!~o z`^N7BWd#4{_${Pp^!_tedT5$B{UaJAD9eP%DWGR>VT8sWOv>KQrP{g%W?Qe{oy3l~JsCkCUg{1!?oLknj; zGZR}ABuc1?xl-i8f9LV*i+4s>Q20SYsgKyW%Gz} zJS(9>W~1c{`;}Y6i?8pKrR7?9inN4q?NNnT2yJj4=Xx?Db;@nTU?Q)PyUHdoqU=&2 zyR^>?7vX)}Y>EI`Ew8{TF=K?1TN{1KAuGb~Q>-_1zsInta?{IHqz#xh?N}(^kbLz2 z5ccNLRKDT+XjG^WAtgi;MSUW3ac53M148DhQK*Da=1iIAd7kHa*xY-v%|l6%gcK(f4}ox>zuXDS?m0__g?#t{l5Eo@8`O%`+Dvx^peEkJl|>2PRd*6#e)TFkB{0A zRW9u}%h#8vd?Q_|8!Y3Xwf5ZVZ0mW-d84fMwamM3V{G(;^qXC<%kY!kV%E}*br;ij z8;|bomf4w{c5iN?r|I#ju63%3#L%#E(nZs+)Bx#PYsGLMLBqYr2|2MMHX1C`7u0D@ zdp{QzzQ5zl{UAKA%eao2!Ihj5DX{xJqs67UlH1OoN;*{IpPdx_M9_EA`=~Pame2Ub z-d^U`8^8ANlHEM*$QF8>j2eO2c9grHI|@jL@?W<7zRq|(v-<%TlRjz9@pwMuXM^M^GFEE zZxUMTM4PdnXX5+VrOdCc1RQLLedN3CT0~&Y)h(xZshRbwdHom3k9rRdr8%lTetF|| z-rh?_qb>Z~4U~43U6#y$+Wk>wG|!jSO!2Nw9YMer{6{{QmCaLZrw)Ev3wbN1d;d3c za_d1MS34)uo@o8MV|9t^H(qAC?Y7bJV(uv2BFi3HAj*2#X==|M-lIjG{ab1`*K zRj;@QoPPST#^70m{J+AT3?4J#eOKK)(^rkxK8FVHmp$V|pBw$L$#Eg)YU)(lQ>fB0-XO3JtdK};X38Av4O%_UmUymx|kxw z0-g9pT`GKV%_->G3%jNF8g2S}$A_F!`fFyjC2!epbrG_9ROXm-P2~7l){4W?`%Yf? zo7SrKqKRmHPcJxhwcKD=a5ku)C%NUu`WgTCCGDf@kZC@7Ol*;WL89VH@M#)6F zx@SoFjgm_08700%pHRs<7{faiq)W<%Zx;9-%dFZ9&E7ZjbHD1#y7WnW?sb-F z&Whf)#hzN%upd_>E!TN#SPl-;a?=lum52u|Yq}rjlv+A-Dwe>s@X6To-?&JG?BkH> z?82G3xt2+%?r+`ZDf@TvG-UqUAh{;w$8MMWENgG~)-IjjHD~!QBd=1(&5kz=ED@1g zOBer|Tukh|cCE3KkM%@l-76{1-Gi+{A_A#G((g!euTC`gZt-3{^h!^_#^U6Iz1MC^ z`UV!eFwv+APx!wyUYAW#GAh(nUB_ZBzZ5{IJ()60QYls{I^;&L@7nrAls{Q3~47$-gvyCHrHp6xBu3WI%}BeJOPH za4R`Z<@I}q`pc+zHF^3vt9TuY;?6}MxHG%3{PXtguX7a27w3=HEH;fx#a|k5O^lu6 z^*^1XX;6FZiprh*uaU$bAR=DxvsbUwj2x=lrL3;(dE&XkXVauB>@jYupJd{zgM>#u z{+n#FIXU?GY!=qv{PDi=#8gT1Bx;rtZ!#!m7O!Q$p$^KyUfp0!Zwk1kC3wOPQnIxT9BpA z=F-4UrOBp&SpBT;W>#lUzwg;29hvs&L#b1%XH{|n zbluA7Ej+8RCsNp6%xe8}*te-813~vU4(PBpkr>(x))W3Tl)jSo%CFt(5w1SYv!yd) zY3;PmSeN*_@>MpDW)+6UJYuxcuz>y1)t{ax)W>>vw>yza~AvzUVoK-<9I`ro^@bmC3wVHt>f)Jp7ATg z6%N|( z9}^uHHS73qKY#6+8QWIIvD&kPkJ;DWaXxfZVd+*}{yD3)`|s+ooa!@u$1k7Ey^kCm zTbkKsZnmOZ$n$D@?|VLROHqH_1hUU?lfvm;PC;ke9a{`v)y#FS_($)rbAA_24`$H~ z)=fPp%R>(h9o+{O!n@ppUh+Kqc7>B-H;`mv)zC9mTKl3=b&ZW~m z#Y>FZajqR|3TjmqPb;tN=S?4HSS!82LmnNN^iFzSyiq2Q_UH(E?g@_$mi>CCk7Tdt zpDW!XxJzvC&fh7yrd^%^|G5>f9LZNt-cWbb?`>*)7`(clmu9=Y&!NRIab|#HYkgLk z!Gz#kRru6JbyC9;^=Wyv)6_qT)OOWUX^D}2ODR3X(RZAy2fr!tzBT^pmLnt~;gnC2 zSoY?FbX!j2lF}E?sxBk*k%prC-qOPV)SI%c{ti(OtkZ(I8omvV*7x%I4l(C4J8I8< zR@3f4Cn?68+eOnxDeN#)r4Gher~jU?VDRdkSP)a!4P-$udU ztD3T5>tj7TU;h^&l03$n%d=woBi>q}9~6BJ_vjW)WMA2_bb6*n!~GFQvY(iW-Yf^% z!`d3id$eyjzgjU|SQ=KVEx1##?UQ+ck{Rba;k=rKvuC6}j8F1fa25T#93y(`4%_Sp zyM4+=Fnr{{?{Gy!qOD@GO*X5*d2@TmD zA{oV3`QmZ0Jw@i{5{JT@LoW;!-LJhF3`*@0Frjtd`SE{X>Sj6?kG;Ua=n}T=hShzG z>sBswqD05&TmP?Dn*Rf7H__Q7^Z$^DSnUI0jC2&PtroWJe+bn7MzEX1cG4ld#sAF( z>Dc^#6Rw;0AKbK|V|g1#JA3-~%wk3VU%*qFQ`($j5ctCmR@6{l5b>il6$Rc7Tck@JjS~})g(z>uPp7e#_x_3NxQY1peshg!MeH%e0C*Bt?Cs+`w5;;4ByGndU4yE2v-`;1!wFJ zX>Y`RuUU8K{#Obt%BGIRw1+sz+5BIbb{k6dXeGX?DF79Ihk41lR^T`%oaDPZ36D<2 zGq}12&~r6XyKaiLLyz0p;Uc*vc-Loq)ZzPqc4PVKN}TNlGUgRUHj?=8klN8iV2>lW8Xw zX0!S_l;xwvKg%9_$vEuC+@5e_GTyRcnjAFB5&th{3&?LTf={kT_Nm^7{SFW|q?$XGZ* zVR~Yn+Yae+?IyR55b&FXIkOhI1oyeeINrD#i7&j&9G4_>G4T18u}iYmxc^%7zE=V< z_|wZXTH+}Yc%&G6&scO~?EJl3VXxaz*;!&5cIx4%pqJPQl>*p#Or`w%of_OI5~x+k|C0JUT;q#Pw{Lw-$hv0psCr-#;(&es4k7? zS@_ijo}%_@3mXKywIHu2UD}Q-%v#G~eW&pHcgCzWKN?VfQS_Xvh!7>1r*Ca?9n+(C zRko{j(q;Vkd&`ccK(O1Jwb{24m1~Dq_p>2*Y?>?@o~r}he==j|6$a5gO3cAyM-fU_ zUJ}(wB}1^XyTXTj3eG*0n$Nyw3ehFOOW0?vlK@DbVU4?Gq*O7kIPSadPtcv@P7 z;tE~(mxl&Re%t5H6YEj(FOf-mlplR>-R36#XAKtTi4Q&Zc0$#wsqz=IR4n?_`XM;6 z8LHf0soi?r0EFJ32_Z#QIJqv#TCCrI|0y>7-rhq+QJ&~0r4?;xvCIAan0*ww&ASOa zXz75hTiDRZ_$oAYu}Jj>y@S=t>lZZ=|2mhU6-1Pyh29O@_$8-JQ1AN|+M>b*ZKr(ebYosv?)vNWJ>YKaJbHBzPkLgw@jCYcX zd{U0%t>XVu{Au{jUoJ3*kPDONuPMmwb%rY%KD8&)o8aQPxr{!BYS>#kl)?13hraIA zbleX%gP*86;WH}%%3D4L9LS>J#H`JqrQOvSwpi%Y<5LVQm388OS*bweH9$x!-BBf!HME@!!hYjdmHHat z?1sAee=iBZbYt;?a9%%%dNCNS97KqeJD+r0y91_$sxvS7cEV_Os^7n&cBnFWs-qy> zg?zg7=5T_!Eaj+e7}F-AoHF!ywFJq776VD!=7Uor~lMpQ(XC(-%dU7(`d$U zRBQkQ6D((fk5FLGR<6q?Jh^z{>+NCpm-%qa@?@QOTNXN}l*oAO=!C@`Y%$rzWH^7z zb9KI%9{X&2(fx{T6>^8|^tU)jqUT{otV^oXkY&7Ae_@e~{{(Lt>>i^)y?CJIaB(%7 zIn4R_4V1vbr{_9V0!4UoS?u(qiX>2=9QViw^97!@P@&Vr9M~Bnkk03rjSTi>V#{{< zQ17~DTcchlEWh2d_1|wpNSFxvlo~*TyFa!TIx4rIe7sWPXiO4Vz0?~p*<%jw_2>DM zyDN~YIupLrq8?qnV*OTaI`GC5<+N>b&2Y5i-VbS!Zv6PFtZgN*7Cs!4Vk;ecihO^* z|KqMGfa-34squ4E7&7O|IPFct?btIFwBUqNe94>sv{is)cj?H)3kqB*6^voMMSv3B zQ;An8CBV6KrSpqXC0vlu=?Z9Vg<_JZ7fS^N@9~YClUKbhPX7_B+LQyX`B`=ZrQ3n^+6zP``A> zU8n*U{@rd3>m*d1jrN{9*MUp>^XsLw;y^m`yWPdwHr&F%PAJtQVMv!OHvi3qT88@# zwvFv*vk)P^{Rj0f>G+U|;mGxHaKmO}w_G=RKJhbby!76m?B#5*Y`a<#Z(yt}D`C#QKSkTH@fl=x_@r&}M zaQ|&@MaNn-uFpK@yY12d4LU)y_n-kzHvYF+)U5+VNNVn885Ho^^Yy3Tb`m};U*j>Y zY{$)7DLR%b&A_ulX32_a!@AXL8D|uIaP_;(`F&ng=yum17L}%Aw1@^v!|fh8Ihfq+ zlx>ck8@$c|nrRTvvtvjpyBsder5X4>Vv&AL+=CjuIT?D;)RZZN!#4rQKq4MEK~T)y7`cj>|dy z)9XxtS9O^U<{8-fAL_NlL`a7j(jEvO%><771tB_A*oG!?lk% zkEv}wOoFvVfuom}^WgqXb9GQEzbSGRRy(64^MS=)bOOtCRG!$%q)x0Ih9$4CGO75E) z!Dj2$)6a{WK|Ocxq}){+G?o-ip0h2(0RgpnT5>g{?qs^ZoKlSI$7Q=GcXR_GMQkf~ z=~=kTGi$Q7jff0P71sUpeK296)^4X;h-qzKn>eCO!B;DFi|(^dU|y^LeDbm%REZS+ zK9P3`F7)|+dm%!F@z|fQ+~k~LV>0T^6KqHR46fD0@g#^exm0{PoBvg-HHM@=A#bsx z!nSudq@xB1$3IZ6_ghm(kO)$iC%rfCvC9U<#u}eFCr*vcu2<`hz75HTz9;8)PZXI4r3OPDBRRV z_BpxHjyk(Tw;na6-_Ff74o|Zv*)mxL0Sdz+`CK~mNFDj z)yD=PDS#XmexHI_L+`Vi4asmY`72E3!Pl|RzbwvA#c1$yDN`LUM1GF}>rd#f|!vS@=)s8?#&rTbeH?w_M z?BYuwirNVjW81syzf-W_>iXLT+aTC>P~DBCkA`P>`-IL*5@EP-yUB$p3T%2gZPM=B z2yckC+kR!EvCvHjLWD{fvc?E5`m3tw0FC2CT7K({~bboM9#=9|CQ zpU|oT-nx@>=bS1*L)p~skbgb$x9jZp|K0@No21*XS1F;MW>z?dc?a0eQCL4dugAEA zyA8}5M6@x<)RcWggs}_D@2~z>2RYYVE*34eK)A3vueW18&N@}6+Us>7?=>E7v-N&- zdVVE}XR#KDhd-&`qw}=q-K<6iyq)*~Vt z5r@>v>IVE1*85~RrUo?m&Tq14rsA9DmBC|UJy@gps^ahA7QC#Q`qUsu4SYnJ9Fi-E z@axGA+hIZ*JjhRxZ_Cby{Or@aE}8d%SnjBpr$Y@2alT0{yi^6Y6Yb(na}8L{_)&+- zR)x1I2?G_gt+>CUg*UFC1dUk6Ei#(hz;uE;CqlXe!e_5>S_;s0pXhu!uUaC$H=1Yv zC{hXs0tojc_SGYsv3rVnKn;q;j#`Y<`Rpr$JLe`3G(f=TPi4jH{!rCNWn-cZ!22mn z-ZVV|=m{9qAG*^C>ROv6X9nwGC+Y2?OJpagsC%y8&!iwmk8 z>!q`#VNnCA&cpEvl_Ypah6t;34G;@ir(bhZQJ^@ff7hl4u;a6MTT16EN!1Q7%k+F; z>G1uf+n55Xz^^J2+_}@wsQ8W2O@p`T}>Wy*D(wKKa4%jUe z4wYiw14EJG%xF;dOGqC{tHW>B%dF%6M7YGoKEB+V35(O3m-REKP_}S6pnJ6rlE2Eo z`NQLahda2XkFAkG;Q`Ng_pSkC@C;2U^lrhZ&*Mewhsa*HbZu- zY1(;#KFF|&i%*cRM5XqjsFCbkbTcs#Tv^D4G1p1!yL}|MOgx{SXoa9#-g4(Zr*0?> zNlVFgAmg&x+>aX-by)H6Zg=5T67KGCAuyVhVwh7m|2KhTe6snqGF71+@?tHgrQckF zqejd&l0if~et~YWJ3$8Dz`=~t<5WzOh_+VjO@;h@U%7373@TLvOU&6dF!v&}BhaY{ z6D0dq)m7*^=1J4NQgzk%+xbNJ`>+n!V^H$vI=LM_3L6DmyEH-c@RZy1$3}Q{?AWOU zXByU|+0^W!*Y%`rD}pub-Dp_R_P+mYJ>*-6>yo@l@Mv%4-$%Etk%8sX;lAK{{OxT< z3cXQ+@;5#5ER(yTfKz3(f?5O4#plL*==Ju4z0j@VgcRJ;{K2|b-3VhV7;>3rtAWWb za0jPaAq0jUWoNU^ge&z0c8M}Xq@ItEW?{PzF`HhFDeUV5aqHJ(zy3T%yL~`&S?q|Gf>@1<5`8_VgptD^G(c~*{)fE`g-};H+P8G54up)&_Wn{TMN_*C+ZzJ| z5V~E4dp5fZwkb=TP<>9rX$b?P+gfzIq+Na2zMc$x?k8r9i|+wl=@u!TiH4mu)x)D4 z?bx@babfau2XJq{AJg|W823s&_3~(m!6LOM8HaQl=y*Yp_xtrsR5zOnY+k6vn^{7l ze^~P2%sJxd}oGm}I?7h1fR@?UuZ{>;Q3e;?i$E{y>){$9*2u6Gi8yTl_=#FAg<)t4qp$*hUtH8 zfV4DoCVT043?hoBZ2cyL5{)N*JO0-RX%VH%f(10}N&M);<&Xpwo_PfQ_%=+@jtP|b zUWlr_(n%t8y=9-({dq6DSX^&)<~yI&4-t~l^RnjjY5nRD^)ogfGm50&WzqMK6|&L4 z*RJ&aC%bpN`DrT(^-EI{8A;e$!obV@t{SCWcI39)PJ%$goZ{;oouIGyGU)@46`-bZ z0Wp?}W#@$_>_>CJ`+410{wONsCdZQ`6ZK*KZu{T$S_)kBX{oYFA|gScNb7kwLe`VU zwUb?K@SoaUUi-Ri2z~n1lE0aV4Z+{eboZ0+knGFfm$@j|<`fY#6V-tSI#d0!f5pIb z>FLQuO>a1Oejy{awjQcKd~os5ZoqquuiPebRX~7kB8%yRHLh4)R2z$@*XtYSm3gE( z@MyB$equ!#7Drw#5;jYOz+l6zJ(oMs=8vpcRTF(5tKKHnqS1+i{kwlMU!fw8{JY#$ z`gIi-HsxhEs6-$cK%?v^+*=V z-273-Oxa~zUJ324ZY)o&sNl0*;DR$31>ADn*A^ee zXtqg$oxmZI!o=`m%LOso9O)EyfG2^{*&FkpN^YqpGd>r zO;l{Yot44YtcSY_<$W!P1b9DeY;$GO8l&qZzi*zVK+Vgrg3$(gzH&4}$%f?sL}+g* z`c3C?-ml`NH~nkIjYYzD0#hj-eM=ud!`zCaDmBSQ$LM@-j@Rf|7zsVhVp64jn{Ypp z%Qfw*phIZ8*x0)P6!MfDU@RpDacxo@cj^yw@T1Z;i|{wMlunf=+3YjJ!^+6t6Nk`EcC@BAm~*+Rwf46a~!Ab7Y7Vp#9|=ds34l!KIFsnFW}kvi-i^h0}X*XzCXD}ehD!T*X&BR=%}@h>K)615b2 zMAlqr=yI1UKdX#}j`wxL%a4{|r|8r0)D03+^##9velq~`yDdeR=euy%T!Xf6fdfPb z&$0*R7vmmDfiJK42jIEHOlV(vAbhu9II6V06Rm!0apVx_eVnHinieSq$lbT|_T5D# zm{X!f8JujvGXe(&HdWL^YM|ZzF)I>$WU#xy@vaXHLxkN=LmfVI5Ur`SD8sYt#7#rS zmG~m#M#KJg0@#aJ+K-G>qS2A5n}vP7P}UMY@OX^`U0hYWmTl^g6r+C7JPa{^$n2}* z_GXNj`!|GRPR$r03>_9HyrCgW$GT zJ|RZ!^!u7GLDo%x+#;tvf3(}6Zs1M$&W#)lc*t{(BLOkc&1#C4goKZ2dLBe&$ z(3sd|srkATSu-EqH71i`RI1s%>sJBr82X+v38Z4O*siLhe9<7RozgIQw;9eZE6@Hl zO~qh;Cd-ZfLiox*rFP=lLrk`R@}wlB0R`-mHEvVO;jn>3)!rL5V8Q3r-&@pzf@fF* z21RJ7y6LrZpH(%UJuH>`FOv$Yf)8rHvZO)fLoo(^0vSx412aadi(vohlChR&^yhy} z{I2wrFP0gc<>VgA!Tx2-lPc6)m>sV4Z)B%owe4}1HbNh0-6^m8_>qd~dDFhVXK7G( z-|@80LLcz!I0T6m=b)MYj!E|mB(ORiZ|E6T25+qfcj@E@&48Ut^shtNLTQRxnWxl%(2{$Kn z5gYd8V?>PRAHDc8jK1R1cJ`+gcwc{C-e%B%1+%}Fyt`tNxc4|E=1dz%J$o!<{k;)# zWhuI=4HobsNYhm(o(Qtkfon-8dSQu+&DsA!HNNB3G7?j%M;`A}k`ERW@zRNB@3fse zz^%7paeGn&xOwfXEp#owD~$UX3s1KKSJYZR6ZOl{7OZS%Zi>p+f-<8evnXdftjBC5+i;MORhK5`Yghxqr(R`> zl{BF;W5bI3C<*DK6y!`py5XsM?>vdM4K6k$dIaD zj=PdDaJ$T>ti#n{c4pI;Glpb5dxo9C0Q<{9EEYAGqbSd>I7x)!HQmHloPD5dvUgMXhc@Vp+NWYYRtsT#w}1S; z)Buwu4B7bW z#`)&k+%&N8EUuXtL*I-`4oHxFQPHsMf|d%=@fQv?f_e9}YcjpwuKG?LSd?kOM%@>?#~&Bt zV56mTutO0(d-LA?PNXxM48{1_i5-XG+Qvr?yPA;x7dresM@EWImH6eER&Y2a_sHu` z5wZlUZe7o((Zg2rLVoe~;K-hqpA0>{5PgLywKR~7q11?{S@hr0p|)pbBDE483%_}4 z-BGHDL-L|(M5rIGo&H!z!seN$J zesqx0i5&fN1Lj8A@UIMLi16?V)16djo1#{Jtd>Qb4L=?oOl{yEs+Rs z*K~!u=IOdwR@L4)TPj?>b&7>!q8Y=tXzL4?p@c_9~ zZT_%S#^mCo?t>dWM~nXGwWGyjNL1EoMWxs5FGS;SLZRm9)(&MNJSB{ldKkHYRl-%9Z!~WE4*g6hSt@g zoV=XJ=)#Qg;>2vg7KB)IeqzlnHy?S@9h$UR>mYqFv-yFK z8CWJ}^E*Y-@SqO!ZubOw{m@r)<*BW~V5>ro#Q1u+-VthcTHFg7r0 z6`f?rWe^yXyF~@Ia>f`t2O3KEC6==Hkm1Upork8@6|UhB@ouW_pwt6&X}&g7U7`fA|e{dG&3 z@@Dw;@D|gcP&Hf?YqnMNEP@A9GatmbqCr$)w4`<+1x}U@PL|y-z=`V&Yg$oc{NP5q zw?Cc=)_$t&_Iv9v`O{aDh8+=K_;S=-AU1(!5(7^s2MN+o`DAe__T%}XY1WnHE;!VE zN|W=yHlW<1%DL^K!n4`1VL9u5aBMMS|3=sCluEni{u8J3YNHdncb(`yt@nA+lH-kV z>rIlX_DTc%;Py1i+8G1%LRX~6)QqhpiC-7(vwYImFZZr9bK)*^iOXDUhjxZBGo4_41oscGba@YV&J1EfR=Ev((E(QX%iY z$rA^!q8r;KJR^0j9hY`I&7MYex=)8-^ezgPI^+r)!q8I z`;6X-kS7IvO}mA8{2Gxv;d^qq$^$fCofdw?-31HhyqP*@3_;`RjxN~-fCiVazB%Dk z2x#@)cIc-&YIgdKG`ggMWCg)PwV@L!d`_SD*OG8u!tM3tjUX^-wD=ulSbzt@*%#$R zLSRt4XGr!i5jL$Z-Y+{s!Cn$C-}mGO=#MPA7jY;PsUzdPER95T9G|c^6N`mQ%y&Nr zU(Q19>)Rc;RGQ(P-n-@jmtv48w~3Jv>B9HzU&^w*`ysJhpLdf+4z8U!e*8X5Gi2%h z@RO(4ONnufZYhN#s2q`Iy)qYzhT(l)yI83(qLfjtvq8Z?A%m~43Y*|@c4dCzE*ef9 zEt;ulr~^y3-E8ypzQEtO55aRw6?kEGc30@pPTY1kqi5Bi9~n<*f1Si*A0Fz2T5^fukCGm+XDOak8EppVGcO5uavEXp@}9tQ)n1582zc?x zuLG}Aw9cI-D8j$86#dfACt>oc%afF9YdkuhHWy;u30m{l*cadSqN29jbF&l@ewgZ* zZ2Zv(Kfyfw+Vp*JvAqAY_PHUO>HpWYiQP};6fNgfon6g7}W_hL^ z$J^xxjSK1hz2rFomC`VD{rgQ@*P{`xzFCxgaD{?{g7Z@w1680paKR>os}7APPQH@^ z5*9Xdz4@=X6Ykgg?elV?`>50&|9ql92sx=^3vL}HQ0FWVQ^Mbds(#Uy?7o5EGPX~z z?N1|;)7veShHSw)K4qq!ry8uj3wCU!>vV7CL(cVA=Hbm1s)bg6F+Alq)s7WS#IqbM z#=An(fL7zp{;Z@N&M(xGC?>^t>Y?y^`DBC|fx4ZigyK+5Bafb0-v|S#tBlJk6p*d^ zFOs&02Hdl8Kjkz-;E5yOyyyY}FK=Bo4yA|I>qMq4Q8`J#75g*L-O3U&2z>4ZyWhqkb(RzZoIVb$6G4iG)(?#r^A z0neGu1f=M`uF&(63kiq2fXC<95b=Bl@b58>604)@?DLKn?1kxiTiorB^z;C@Rdi0! zy|oXO_GYvHkgP{#Y2n8P#$8zCT#H9)a z*wh0WNSR;!_;gn-zI{{H?|+XB#V5x9d>o{}2)AzL^K+&+dpL4jhQ8h$PdeVoNUFzw zlnXHh0$p&B@5@2eU5WU;=g6DUjV8SEa_WhJU^V(`vVL3n*^CFY&MG)aQjj{=Bx~21 z3}V+;CQD;IVXx7ztqq^su;ASMp_#x|jKAdeRVla<**;nJXm^#Nyl=2N)1Mj~b7$|B zB;=yJ_m7!>&$>Wumh%l)HWhbut1%dmNf?GknJd0jB)PtNb;PF&j{g{s*Vm@+(}nHp z-zw5TA)d+7F|G`TD%_u|A0xolZ(=oN-E{pR`^bafwM?8OFJ9kxSc(jfTduEN&jF@4 z{!4juADeYi-S{Bg|MaBR>Y@6tYAl-yCOu*80?`&vqtcr-p!Olwyzn9ge=+iXT)bL> z``ZF{`(3L?kAJ#Xyz`!dliTd|!OJBu%9wIS@k}NDdnp-kwWI^WEuIN0lyzciU<6I9 zy9)B+&L$WC%0;#T`EmiB9Mn&?UndCULDJA0-^)i@fMe=Gj{6D$qpRNRHryEpf{*#d zc~+V+Kp^OIXDbztoj<~HuDb!u4H`Xn{Uc+X|HdosPy%YYo2Fcmq2W_6vnTiIzDd8D zv8Iz-YH(CqUvM_50eilrir!13A3Kex51@s0L4=1c?TS@Cq`M#13t{Mn%8RGU66kf& zayT!%wY>mBJKj1JeGG=O_M1OTDO6yT{x`Ka*Z_k}H91S{By2cv@UX_aE^JABWV-7> z38s2i|G3rM0$zJ(`0{plAseTocF%ANy2)LXy+zmShtEe9-27dSkA}YO-oZgdZ9W;s zX~He~K6cngWj6`#YVjmjTGKFCaYR0QqYtiohJ04Hqx*?P&++}(UW0Ewvc!MhmJ5-c zDiZ^-y>N9T=IL5xJ<{k!=v7Y#%vXH;HRM`~@?xt5<;@LH_e!tn-qS8a+LUoAj9bF-DToC=59=4Q1{7lIONyww)bE{r&XY>pX; zXba=>s?8L-k4-^(eR~HQrz9kib?N6%E?ykI9oUA4CWR(5Gn_!N`SWffp#UV$SP-_V zcER&x;wkR-SUB!c>L%`%Mej$%`+a)TjGS0p?lVBYfAiLx?CHKgZ>|x|0uwr3C=Pz> z{HzToR(RB3GI!GZv`5qT{px{&Iw`Vho{4=%CI+cp4 zZ6-I@GYat#NA6`B2Mr<)p8xcgtqCAscR_uO2s-r4Nr6Lkc**=^8E-KaEpKm{>T79% zH-RgPNnbMXr+kJaIXw*$FU1lwZ2aL*KI?~pfGi};$9W0V#N)D}M`TDfZ)Pv&;>P{6L9cl`Kp;BoEB9~#G>VW-)5JrOp6(hfL*Ey~lvsG=TM^iqu5SH% zq7YucxVHDYX%=X`RJwa1uLWMaM&Q6(0^D(pv&+aK!#VApU(^Ds>B+;PHzbYP(BN~p z8a0qa_XDwfITX-?=Q|@S`RV6wI^y)BQcQ?2@K8p~3Yu^$d00AJHw(=lZeLAR>p)#s znT&R7JNRt0avokH!%4fNO*@X2Lf-*T4te2r{B!x+z8lU}=sD1JWsi9-UR+lg52gE9 zjSCbmx}G6JWXFwmDG5n%nV9fZ3?_l?Yt0x2|0>A*=^tkx-HEH!6YuYccOomhme;sE zUEe;c%dyw44l+Woc|C5T^Ns)iNmUs~cm#8q-EjVh$a=yj5)>Bt=$+b12K@?i&AZ3@ z>DE7)+Sh(WWZ@Bf)qmzWG`l~4DO*XWK9?)sE!-^w^{}qdy&5FESog0r;}YGv^!(Ve zrDz8*1P?^JGBskAO0a-;QWp~PUAZh0{+fi*)G;j20fWJr>EM_(JGaJUsZ+-k60}AcAxIU0F|(@XpaQ=etIQ6{8}H* zTtD@5y|ED@cjgK=ziq~aP1|BB)7mjhlZ9V0s~wu-gHk05dLWw-?Mt=~Vv-C-co|c% zXGo7ZHLV#Ygp|5!yu0yOYQooW`caXTf16C3a=X!ErE6qqqznTN-%p%;-U`e1Vb;68 z^dd*C=FuX$b#I!n^O|#NBcz|K5WGE6kGHa7{%pO|0QLmW)9;PDp`p5a%1^Wt{a0C3 z_6gsC6GxpcY~`dNTa;PG&Ei4)vq_i<*n6=*i=)Br*kfe9Yaq#0UWY3(zrWu~N<)JL zg;nMS8WbiN%{GkEA>?t3!-jQ@DE;!wJ`ME_)cP$PdMls@IK^9jPpsmLFJw{tm zePV8^yhlRTgfG9g@BM%ns~Rety@dwLs$D?#b@c zg%}%M0LYL>w`zx;{9lZ{by!tzw>C-%ih`61l8PdwNQi{Y2Sh1pMUay2l9F!e?h;7_ z6qJ$>pNB3H>5fH-C@2!5qH^ZD_x`=x{e5Sj>+sLI)?lvb64qdh`@YAx9|uz$vb}#1 z_-Gv=t+qolBiA8lxc+?uzNkPa_6qHX{BY=Y&owsAtP4C>PIJ+I%krNVI?;!%9dOf@ ze;MHXZ<~v_WsH?W#8nPbBBDsf1kz6S4s@!4RM~vzDL9?46d&eZ14<6azn}lzhWnGc zo@hvWiq;q6#^kGCAya$Ov!5Rzpc2ICyXRSrLfwdoG;6!SkKqiXX{t)3LnoH{@Ma&P zR%3`VJ@*0xWEEJ>yV!v=DlzJ@n|0{2>`?!e*lO_R&)V1BFoae<@u>KwbfS*hI`6pF zN>uV#Kxs_=F*>A9INRTdvwp=p+ou~5NPlx7Oa~uZj_*kL;P$BlZEYNtkJx&JVlMx< znom>(`16bg*>ATaySBqL4mg^j2yS0t83{nVzAp!)ck0pCsN7?VL$yeYEB=1e$8f-N zoq4_UKr^~$OZGvEz8BOjbe8@qet~=l;>Ubf^ufXL`|s6GB%uQ*sGWiov{Cl*fJqE( zD>xh|;>i^L5U>uD*>DV20}0#E`B?ILz)SnAbNWy_pdVGkcMEEe;!!SP89SU+E*V7TEjsj<|syh)?BVhTi_y|KnJE#=PYzPr8#H-O) z^vDQ!oiZU!1PRgyYlEk~!fk>AxCL-t-!o z)~7mPTv__SY^55hURL_C{0<^pg-@>+9~A@A(7Cc1yn9V8F*^amashLXcl5^XK5&#^ z&J&(kj|2pBn0ky`k-itFYA*XjkbuQjvzK;*%FA=as^3-vancLsAm0lM5B!Lk?I z(X+JfYbCT@XiP}&&Yo{0kSbK(dlu4(;%A)ZPxm#0dp-MFwFJ}Q*}p@L*E5Dc=+FEeT0GEH?*DD-?~v&|&7G$5$-hGbQ>1T5eJB17 zo#K|Vj@BOkJ7j5DvCl0%_IIebp1pfdc=Yd(Y;`SPnb64JA@&!t=jtSf{|>R_u$Cju zp}#{fGDJyJy$1gd&3ukY#hwlP9V%hepfgX@BauX^?wQ~{?zyPRW5d| zw?X{dze628d?R`|+40wJC?|PV))ZoL7Sp`7yd`+ z*@V1qie<~+S5-ZCr13I^6=zK0e{LSZ{;;Uua{6Z+n3k zYEnZBPKn?%MY0@i<2=lu5rr3Bp?}{@NoI=3PN0UTHRj%~^h{s@SBt;C(NYD6lq>@ccvH|shFptF{Vbd!qdYxckbuSI75$RkZ<5dM6AsQTb?1mZtc9W(eU8c6f6FYrD$gslm#4~{h^ zq3ou1^OH8=$m!0YgPOgl*1YB|W^nuR0f?jN;2tF|v51qUoGr+u4mB34MPp z%0>`S|0#73$V))SKHVDa@Qy^}pKH&Q_4okE+Z-GxI2+(6M>g4wB3Tf?x{x>dCJrd) zZUyg_H0X&%Q8I-2>nT8k)}%oGuCz1R>T>wgRHuZ@sB6i zm9@AOZ=nacpPCLEe+vRT)6c0zDJNi3Py(lf`w}*}?K@F45Q#Ruhj>ZwHLIMouC(ug z?ua<4mgrutFZeC`K8YyJ9gKGM_({=+qs))~SL=AkvG~zVGVO;ks6A)mYoLb-STi=9 z_TU+Tj~@4N+iOZd#klM8Sv(=A`XVj=4~+@T=ra}N?MX)zrNtw5-Ovbm$Q|Yx^Ui@) zr$Zg)F3&)%!@50>=^c>w$PS2Js)oF^8X-qtPeL(xY1vQ*VG>v-1ybJ0OYpY zq;pLKS(mPbEO3uQ&r1VgRl+Gijw|LEE1n+@&A0tj{RdQWoqLJjuPcE-Mo{m7hLiV+yt}jvC2uG|o8gKGF?ZJG5g`fY~1vsSlds2?k5G$E1&__h^sGghLZd-dATl3Gb8?+S! zGBVdbB|S_)<~LgdBJuq2vHrm3*z5xy+~Esz^_s-`t{UmEy_E%{>};Cno8!Pi`51d? zz8ECs`g^b1SpgiaXZ9oD`N8_KQjihPkNgK0?GT+M~wBv%`RQY(Y#hZ6er6^}9EAZyvg{E~a>sdI3$r#5>`- zdLa9KF?r(mAh7%8&<9DOL1^}Qwe#b_F^Cx*nhoiQL@6Xb7hH$^(L4S43iT*=G#(qE z`#awkXr$gB_?_$y>@904sBpEDZQM!sW>hfhs*_wb4@mX>`$#bx& z`&aqf83fCtA6eN5grG~zSwj55E%3?GI_0@RM@09IE-}m52(>rowJJr=V@0uwl>#Pq zm}8M_!ond3WM(pMSzTZPB;E(5BWymajp@z$&-tam~cj3qV*jdm%Sr( z{EZ?wS*?Ao8)RaIvSkMhyhgFI!$n_*h(2M4bKg>PU8V{F$LoijSCtuJ5Q(u%QO^VYc#nTwrMESxM|_5pNC+}`1b z`mx?`lZ4YGs^CP-T`6_`6m*Ixm?VH<92?tq`+c-H0Xbxj=m(tahm8;0;*3ZWk$Y9Y z#kD^H;0a5h>;9ZHcHp2~vCLuw5@gX0%597W7gx!Cj4x8cvxN7_3V43-4yq-rS%o9l z=Pna5N%nx&Vvk14djuP-()ij_HI9Yz73dN6lBq3txTN(jmGAjJ|3W6}H8Ji#b8k8fRmWt_5L84Tu709{`fnX^=5;-v@;&$q;3zzJX(O3s;Z4Q*XN-0 zWHWo&Q$r+qg#9snkR70#q+^rx>c(6J2U0s#I^nIXNdd9E25iM~mG5;@33hs(>du*F z3!pj2pf$4C4EMRNnlU{J1Nnp5j#9@GLDPGk@`$r`nDYpo`Y~2c#g=%s0~>g#s$lyl2Hh_FepUz1k8r7vr;6Ni;Sb}disD7R zSjyX?PnY;Y5IL_>@0Q^>CQ6>`k}&Rw4-GBPf53-u&fm$ruIMm<`KWwU6pACj%~yRV zR~Q|Tiqo)9nQb8^9?V)rG`;}k4Ojk@=yk()HLLTIME%hI=;yCv_~y!=i*Us%MiD$Y z@S7L~FTfDrchq8L$=FYc41u~B1g$wp%pV7};E&gaFKT*$RyBU6f8~dBci)?Pojz%d(#7c`!K;$TSPQ8eD6<&JzGOEGYip!GH8?E7-B@uK)T{;S=WV?U&IF*N zr7?x&w`QQ9TA*h)g)@@!xh5?pumE2@T02u2x`4rBqPG&>`v3{X?lS=x}!f!00_MqEL zf%X2ub}UPHCvb_W7wQx!Xe*M$qk+~RO&f##(8^Em@udbquw~Q1Kbn((pZ2-`Pyx>m zYL9~-!+Y`mYK^HmeR&#^P-~2^n#+PazP-_!J#oNn+$8$Q`51Kbd}AD5$tWU@`6M}>AG{XN7`J=bP^4|15<267`c}f$>kb>B{es5z?8+#x;na*z z947#d`qh;`js9qx`g^M{^C)}=-%o|M#)FnqGoRW6N1>G?J2}ydVR&Aq=)S?IA!=Ol zEdJ2c>U1Z`4LPQ>cC#y1ov%eynxiB@A-{*e4gV)&^% z4fEiKYsT~=a12nrE$fOz$AM_Z_fdZ&6v_Tt)x#Y*?=pTkQS1vM&-apT<+y|GZ-HkKiZNEN8BFD1 z&AS!rV?8z?z2dvVpDPn{+8!;NQfA~ZY?8? z-fS-rx_@?9qj~`Q9%bHW^fVdWJwH*`ZJZ7aiSeVTxHSReocU&CVi4-}fi5@HCNP$F zBVXGt&p{T}!MX5hKab-@Vbb|`bKyfANk9P`x>so6Xc3ZBqM$x8SRW3giuWPWvl zXs4#_uoLqftTas)kHTGJV}52tqP!%86`UGiQg8r=tU0FG&#R!1AD2C{?VZ5*@y+Q{ z`3&&nvPsVouP6|gLt-XdGXPV)B+j8qbD*!sQ^UVx4C>1L7Tg#TkmrF0?c&ENNVR|E ztw>BTke51s$(*DI6Dn6Xv#v@;T#AVi%*x}~!A-6c1kYydI;PWea!?6~Q#hm)=1oAJ zM%sEMk1EWSv+BT;$Meub09q+TEn%e>>;)So#^62Q&mLh)1Q_a4|H^Q(8!O>w$noWg zLBvnGs#R1|k*D^(-J|-|nAW?^r|Sm;P?*e-^RwYq@Klp&PL<^Xq!d4w!chTi2Fqqflu&S56@d0b_a6vy`mx7)xU8n4c zhSuc+x9eXR0nuq6jjLPg;J9V`)aC7Q*zCEpYdu^Ezcif9$@6dq!RDb;bT*?f+NOH; z%42iz{wvxI?+^x@ZoOZ#6ax^0lB)T;I&19x!c1%VhXp7oudXxAm5AzpoD*xfOMq=e zXYMJ4tYUaUN%Gk1!K|S|(DUnFV839gxEaUisqNwhVJ%~@;uoozJYGLO^=;^JGDL!` z6#I*_&4a8!;A33rMPEyc>SQSZ0_L0>xbjwyNj>E2*&+$$zX9V0yXN! zr0o8ye*8G22YBi*v#I;BPa2XC$5vU?Q5=8dQhB>Uyne_K8`szAj=;=5l49*GcV$J~vWfU=h=r%zW3f@s6Lvl4iJrA7KV`lXX+Mv)1?o1gnFC-O&s>Rz>| z;W+Kbz+3q^ynak8sA{C^_hbIAEQ8L#HyBUJ(?GRVL$pv>wO+V3P`8OgPzhbt_=c`2&VZ(ftI?y?O_*26F$J(pI+jQ3aV zJr9{%IS9~7`^zr^trrNkX=-)1(E}wsJre#nK2P|S^2aV}5~ERJ&ivjBVMqrrHr|c% z7w)HIYiInCU?I2petRT^AS8 z{Gw)rtj?+qxTN2Q?X@mZ@g@y^Jt~!|18lZ{RxFhpZO1J?9 zMPBho%^|qXsZC!#?E`LHQWiG>ozP2yf0!dc6?_h*8XWO;M#(bo93F*5qL!4j-AWvv zKl-K8PNuzt9lhc3_TyX-^62^K&buD~;E6@UxL^XD&Xtsw;r2wE4lC7SEz#f+|L+mw z@?>=7_p4V?an6A5gvhkae{~~w z<4g?y5^R@jykE~H2uR5y=G~EE~Y1M7Xo|}PB@4onb zF2WJ@KZ|Q_E0~3ID!XyJokLhtdVi(;B@JZPbTu`@#}2%lI!$r=xdqhuey~h4qW$&n z^>*2T8$D3@>L;^j?bVoDHuH8ywFQ`-@P7?yrZJ7j(^L;HTOfV2=1)^=PN<%s>aKrz z2-E%jrb$Ue2i;xNSiIt+2S)bPxI)&0aMLu)oiAC@pr$Xx>sHAu)^*r&dH$mnC^0U* z8Tti(zZ7fNR&WIiXIj%{UZ!W+TUE8 zz8-)+e@dqM@p}Xw8{Llm1rQpF|wU)p+@-CrMx^+ zD*&5U{T+L23Fm>@#SDHw?7-MdM{eD+QU&B5#vu{{DM;f$V)?jB9`=-#W8)Ie??!^o zR7?B_0HbPoGH&8T)cN3>XxUBxP_VC+Oc7dwMWaE;eKrDBo+t{LN6~=qH1$?~%pkV4 zmfmw1CZSonm!#q5xH)T}^rg%Ud!Te?Bt!A&A|(2#>U-L85tCM1Rp7wuM|Xh8#?tjX ztXd4dbMq4fYh31vC-M^z$CwbMI*vb9&jf1FxA}l0+DCXPDr?|7<8ASi4`hKe?#esZ z5eFQOFP9wWk3q+N-=^GEPypM?F-&H7{m97*tdMd~MtuwNOllVs5Q*55QLrNm%DykX zm?q$X&eBtjjVGQ@nm0pI`Y?`^q2bQR#HNrtio85T!|-!1*hy zkDMpOv!>uo;jF?nUMq-V?mvICXox7)?RkPc>;T(4o<$~-8CYFmFT$z+2Fjk|8@}a7 zfGO`WrdJz7(B`+QMM1X(U|arel`%%ZxK%`A>T!H-eDYmF#?eGj?dZaq=VFVsO0=G0 zJLm=cNledjUC{%jH8BNjIDb`SP*GzhjPoTnCUib`7U03}jx#6TM4~m#E@`y|f0P^F zkn}aw9d)*}(5m5PMdC$i_h~ZSfycK8ZK*guFCRV|Z@B{Cn`n0T!kieyH-D5z9LMLB zs|l0dHVv5Dnb5%q=1It}prdGk5S@VgX@8n-$WB19mvb`76Vot%LP%1r0mqkA7j7F_ zH(=(kt$%q9D1u(+M)DqWN7!J^e({0aJT#e3Zd0Lb#muR1iG_?tpo2XX*Y@>NFvr>R zgweKCBwZyC7%n&snA@8?;b}bR^5+cv3p}g^f`n zX5f5!Uiz~-JX-k?Zr2P!Jz|RH#;AB8&nbY}$3>#K#cbc@YY3(-mt5!h?-H!s+Ka+kDKPN!-9%`ZIvF4!k3dJt* zyfN@@jQ4#(pMU`0(K_Oy6vSVWEz_D142n)n+TA3_aj-}8;XxdK@D?3T{y01iXWkqp z8d)Gj>DBo}`Y_`Vwr#ugO`SD+VR+*`<(aq#~8=Dz@X2W03xi_gUK`0jTli@LtCB zI3()rR`=WO$MoW9+%MquBT0xflWHX#Q~1&AccWC^0Wrh8fucL1Z%V zL?#FwyqhX(cV-;2*v~|tMKJ&zRZX22gPgr2HF`@kSV`u?!(** z?0Sye=%f42;7w^BzYu<$CUXE!&1RlC009e@@`}P>klZNcEy2-!II%f>FC4BFB zaG!vMJDxq3b1o6xF;c`Bf6c-p`Q^I$Gh>iBDt$@_=Y2fvPP<9UdjY1;qpeW?9wORh zHHmSVhJr)BV36Q~ctk3X$X>NVKZP8!st>t=(jNWmbTArF&J&M=3nA!H{$d5kD=n}U zS?U_GWdJDV^9fJssxWKXpI1L#(L|HFDHK#AT)_Dq`JeRm0qmG^uirf0Us0Oz9VNRn z4H}Mf^ZfD|222Z4>#(XifJ{s_!yrXpZSu(QQd3y3@s579*-Phn@3q{vv z@?N`{xg%yUANu9s5^R_va`bxC51$|ENKGP*M;zZpG$iyTFh3kEoH!wfn>TK%Hs10= zjB&g3ex8x&bE}vwb%`zt45;L+&L4y^^e#>ViLxMR?mK6E`aj+>bag$Tl7)sVArJ!9H_o??wwYZ4Ju!MSp1Al5(wPU zR;*gIMt=+}MU(OV>gwR0H)osyc*H{!A;gdb1yv@urtKZkA(M4yxG;|$R1Dh*Dy_q& zqL(HzB{WdfSrf~!2X>&b?@~qahe@c)6sYTS`z18uQ?k9fF$Z%yVSksTH0Dt4BzfwU z1&9=tn5Lm!fJG|5&mlewG%mVdUi8BWSv^Z4oMLIjo<2&*%#qSTB3n|cTe$vVX2h_* z{51&pNzrInw6~WcNY6Z%*^Q$RI3_+Hpa+v)qW3=FCe`CT=57|+R zXtl(+gHQ7oA#s%!;EB3x62Gh_*kbX&bUn!y489ky_hAl0_a{cr`j1S(%fbUTG4rn2 zu7X1X)e%!rm$W_>pVbScf611=J!*)mBe$G8n$^(V>#Q#$n@h0zR{6VJl~d5POIP{1-RG-f)s2}|9QeXeYz`=m%IUR|g~7hvwaYW|0mv#Mrskdb6vpZ7VSN=T0^__W zNjaWGbZAg{;G1&%ouItX#XB5k$3lrLUI18SXSKbm1+vO?}^BLcEn!WZDLxKxAG z+x<}8G9I+QWIx!&pN_vhv5S&@}p#3~k0{frt z%9I{%h4%DxuPzsOfj_-^k59NRz}$?NlBY_NQGG&Zs5M?cOphhGix_KyfX$eQ)|en< z8Zq;BbFl@A3eeZ{Y0O|dwku3UgdWT>+334N!*ke)8D(!x zha*7=cYB)ykrnMX{5aS?C`KluEx}le-FZC6KKw8VZQSTe_@Lte&S}b?IGv_~48Fuu z4&HMDa>65?39=c0JY-@j$2AH}TeU1sXN+MZ5xlqd;-%&4tMHWe^U4Q!{m}8HQz#IafGV7iLlV_yU~1z%L{R7w${?y9t4?e(?Wvn3U%QSrLIQX63OBH+) zkg;6C^^f?jb$fVm7V^&4-MWwCk7_L!v8=KQj3;?n!vN>6Y_5_9590GBU(To~v`@@n z>@ioHzNuWjy*mr(mA7nWUnik}XBFF`xIUP-7JiM@#U2nfYbl3| z%whFGld%^p#^7jdQY9CzpWoZ{KAO`sj%_`>*;=h52(H)zreOSViw)MtD{j7#Xt^YY zBfQlIU@nW-dg4_2J+=nUs;v65e`|29b+tvLCyB%>#p7=VEJd_?{VWsEY?17@ogdX?P2VFnAFOxKa^x`JqABy zcsISZ>4F<9M_sVkP6(_HII_3o!qk1nZ>odw*aruxp?Q5pAiu=y*0V4RcS)3Y7h@-| zX^M@r+ajHqVIesuJI;$*v&vZM%X$IjYG3Rg&R>Nk08Hun93)qEWj~GgS6?5gVV~r! z&|Qx9Tj|7Z;QG1SYd3NJDyN+)M@%#XZJabyfBpbr#OH7>om8WNP5hLAW zQt6cuC^Y)^T+^y7=+b8@Bk8k8=ht3VF(*Ah_Ax{&4kMh1|HlzBwlE*$6~?+pD(nJo z6z1R2uCPUCa#9Kg9MM(cLcS2jtgp;MY z1|Wmx{Cs6_K6c|En~|lLBO*Fs00(6m zfl=57yZ6Y$on2=JV{=>6hm^Tkf^zuf>2Lx(ena1Wu-pPvwtq99;Fy757|NbSh+Ck= zPld}-8%~IAqq37;=`O~4o7le%=dXrqmf6+s>Vd)ToDOkZKX1IoU1XgR4Mg`Y5XN!c z*X6Tx%&lE3pdaE`g?<=-bJ}%gT`!H%g~5b=tq48jBsnbSgwJ2ej(Q{3I-D0!)svOi z*91EYGsObAwt(;JqUIB}FeId8q!_GDfN6HDAI}HWU>&?t?7wJC!8Jl#T

    A{K-_j zUN32gAcaWlFAp_Dwfadj($@oCv@E=vG_?RruRZI}=(_=QSr^^DnGe8??>b`AxV}VP zuC^l*FATV>6WK;(1CZWJlTD%z2%a-{FQ&lxE0?`aqLbW-$VZ;5e>`sy`c(*QB%6%D zx7GK24mVH1n>)XzJn;F%N!hn2dt4F$?<&b!@)>-tWu;`08?PUHv9-*yxc;FYA${)4 zjw^Vu&-sA=+B}R8k9+g+W*K(1xXU2CM++Fe++RF#HUJd}LCTY~eb^)0>-u5D&d5E6 zoW^T87n^4GJ~!Sz0yDJyGC0P4fRBu&fgWu;d`{M6!{VR{0^TNnA>jH4yy}r`vx;BSnL(x2d!Bmud}6*p#=T3{;po=px9gVC_JGRx&KgT*Cnjeh?tDhA8q;*i zzf_3p=Vj87{E<8DkYM1oX|{V2yji4g{fg@!y|hB<)c%o3UfiBj5c+^Z_1&Q!sZk8x zIpvc~kOgLCi|RS}{ME>T>cUaHewCzzKZ5gx@KctRL4u z^nbFnZ?e-vMOWukn# z856&9TjqRMJcxI^aGb8P3oCq+uk0T527Arw%=@9y5S^)fMrj>p2hbRQgE!d%yf_+h zntXf)?)_fzV^CSd)CO*crMPEcGLQFjp1-sJ;=6X*k9x;o9&cfBQ(73v+PiZ8U-Kn3 z)!zipI%Q#{d6$^FslC9d?*r!d?e##69%TXdP7ttlx~rKSG>j3bO+NGs&%)^NAeWHd zNF=yS5lxQkAJKY-!?k$*XeiE=`G)J~)9RygWw?GGPJZ!DFRq{OMQKw%x%&t^jd)JF zKZ!vUPY*O+@x9HVV5n>3(tKxc(-{OQdpgposCgFq%KeOy9Qj`LU9_P{5=v`;jt0>jo!6}K_kViSS1vKJ1wP{oN=aJ`2KSb@kmt8y_%5(fczM|dBps9dnRtO4 z9J@)`;XFaW#AGk4%HaHp)I{E^1AL>{`6Jpq0{0i7o_Ov!M_fFx5S>u0i;P5~dsS^l zZ+qb^i9pc%Q%kUy^+fa{dm!-XdwzS&Zv@^seL=xYunh)MtK55+<^^&n*{)9DbD73g zKjyS>{1Kg4gAE&{1Dc{4g0_Yxfc`1+jIlu|<))0b^TS1$tH!G@R4$3#t>@9L+Kb1c z>j;GbReIRg^|&y7wIxiLaC*O#3dbKOVo}^%d_I&V$Gfc|5G72lj_Hl%VUJWBro!tg zVOKv!w4aU7d5y`ImZ&%Y#b*IrP8}-fwu|YH_byIgt_)nuxRwC~7-Lr3ar{AB<-%&e zuz3OajBP40D4FmNt86MU~;6eC=^ zu-S#bTi<|=`ZRgHf99ek8*$f^j#fv zuNs>)caFog1&-Lj&0cK9_if0faA%mRV3$}+<;6s&usNpsFxkln#I79s z{gaXx2&%oz6Qy|%S$MyguAx}MlqW8E?Be?1LCUl`9ktUV37dXXTFCL(~SOMaN(Pm!mS|BS527^64jd{a7HUzc$ocgmq~r z&t5t#2o8RL(WmZup_JwuDw%$fXluYX&TmNm{$WGO`yq>O0Wamo6L4|vINN8q^={TyokFyuX>vHbLA2!&UBX58l*72Wov7L{oS`Ycc6barbvnab>Y`deH~m zk&KNkRB6KKx<$cIm@1Go*|cDHcSd)d7%o73{)#z7b37ZLPdpL%_}oBKJ(QSG5+5MO zl3QtKAt(CLdn5Iuj}P*EpJo3Y=XdKxC|x{o{rvpX;145I9Eh5Na=87e4SG5C@~4Z2 z1khw~@2#7*M#4UR+vkj((C&ypc21rFa6ck!`6GG|vJ+Pe*Q7ckEmfsgkB7!E;IPLy z2@yPAU%5V}tbyV`e0B|uv;+19jMj5!n_-9fVY-Jrbr4n-OdH^?eNH!$RgdtFK_;QB zxkL39;AwhoR;v6Wwl3XVC@gA$jI&0qGmLak^Gn1kEQVofY)`*a}w?aN4T9sq?d`UhT z7UF~J=hL2x2RGC$v21yvMX_3QkS?doD%>Ir1lI1 z1BO$!9}jSGgKsX3sk-vZ@R;oOuZ{V1;2fTNg?x7lDh0H*YPBq5<`O()BatpZKi}j- zOn5Jr#Smm@wk`y|MFeO!9Kii?&$}yj;(lMxTZxBs?xcYeV@+?<-_>CP7kHMxeystw zykm*a+74CB0P3=p$#4Y-?2${E_Xj zXHztWO)5`)F__K+ub=B*W^LC3Z$fh^X|>-&pcYdvd>WWlX;=0+?Nzc z@Dgw{opO~ceFj3*Cl~MGl4%&9*|ebMG{z|F8diXx`(`uJxcrJQ0G!b-@rj8`0sIy3p;j&)Q5I@ z40Djr52-Tr1Dvk5PVi?d%*f8`^>y$=ThAKa{#iMM99)>p(_VNWqK%A_)%bU?hN9qM zGRs+Th0%n-$^ICLN;xvANizO#DfHhf_w#mI%xdW)n2#ORZCkuYYhkoDpI7k$T>hMl zfj4oD@xQ)$$to0xR$Y*njxl?iiWSn4XEGQb@&4EXo!A|9X>7# z0`DB9rN85a3i_4&JAEQiUjWnNgJZh?zAl&6bUm{~7N1FVB&wRVM=`gGO*4}ppz$~F z>kVnoBJcZje=g%3*!%5dHoMC%K;r8a+x0wK)R#S5Qt^}x6-^D++Z5ZN+)Xa#c4rB2 zr(3Fp8DUl1h0ga7_{c7Ijwbb5euVE@OLmz#$7Lro3TUejAC5@84a zk8l59PdmE`kB>K6{QJ80%vNp=2@ABWFEv)W?ey>a`)zMl)e7pMBi>p1#Rhudz%9eExhsX>w5=-ZkFO2<1S~QQ+f?YXDPyDpgODgAbuFKOW(^xv&=;(eWF(65Jd!sxj zND59}v~^kC#X?_E8p^)1lq#Jnow*7!6iM8?-!axk0r3T`kC^hJzI46l09Mv^ zCdcKbtkfs9gZ;MLU06#kc~+y~7S`}#nskY61B+fAG3Mup!KcP$3dL|%+lA;`;LB%3 z;NapPD&teT82whJt?|aL)R)4uL?YGASXM-V-$lkWjN3YbVQv32Hf2S>om{$y{Zi`j z*e#vFQZr=b$sfl_p_#BG-p(EDA({4{EFT~xdi%4aeBkL0(Qm|X5e zZQ?R!sc^P2w&uPRWz8;Espt-tm~UF&{xuUbQJCLL4|SIE{=Kxh{Q%7l;HnU*I!0O*?w=pu!J-sB_T?xq#`P)aPFb8KtMoB z1f&ruY3Y=1kWe~R8WcnX=~ARqL>lSt=lZ{E-M{zC^L&~$7Z<|}YxX+!5qqCc54)-g z1Cg>=|LNOoVaQp)NfRgf2X5MVd|te&j-*Fz#0Qoypt92}`ocPtFcC3Xa+rEiEU2FK zu2alBq`$U5IiIixCj-wW2o*aZ->b57@&kA{BK$jRTK`uck);MP6s@3B{`MBlS0gn1 zgEC@xNfgb!ieR#kodL7)h?k!mhGA4LR@|}O1%2k?5*|sbL5dkUvl>ZY%Kqq`%H-7|1zFBgdEXTvOm6kIs@t#3f48IuA&Tj+#(3J;n zYcl8;4;4-ntB>5gx^KArF*qf6oM`_CJ5_P_)q?-vEF^q=)5tHq0bkqgZ;o9yMf$nb zOyNgI;N3#gCdxJqHPc?_R2Dx2VfS#UP}pmf^g_6KJGDWqft7tu(r_IFr#7_;cHPnb z8}9uYtUgk1eqGV#+=1S+CL3~U(U>P%UB1L6dl_6pm`Ijbt$&3 z?s%hUWqRlHS2PhB!Mh*dW1Yd@=u&jW;Rc*O6=N!0+9`I!bl#iHpAqHk6Oil@hoR`S z=!cOj{%A4cc4TnoI0zlM7&&!XqdU#UBz*#t@cC=dqDdfZZxq=9Rj;Hk`2AxpDGB?Ct_g>=~#p)y4EoW|o zV;Vl?Zb)q&dY6}_BZ@LzH#uRj4sNdb%H*=IkelatV$ zDr5D*lsz{wtTYcwUC-<#__}~Z;}+?a%N}UMv()tDm1fvlGa%yTUx$b)xoGusD=;6} zUoQ$FP!j_|0jv{XHn{k5DCY!@@JQH<2Ez~C-k{D;l)FeqSlO(+vKG`+5+mzGGn!J-!-43F^QbV ze!-$jG=QzSDBnb&pP^|thGP1MjoG` zw2>jk{Fk|&Gw7S~I*=mWht?I*X zVE9q3D2V4@v;rrrr|Q^fTVa`tL)%)`3r&$d5lH3;L1*OCO20_Yfb6|DB(Lj+VeG1g zAK@uaRIrucFw;~B43i(~b#h;$1fjlWk+?u)%(E-$)42>K^eInIE{CD>cSuDHGe_Wj z?T#5m1s=}nV)3DWY6$vw#c{@HbsI!>w(mr^2cigq4_Z@xyYM4dZ_)v)j~xS2vnOk| zs0KH4RcvS$Oy%TBMI<-iU{6Nh(8(DE6q(?gA0NRhE4hzm+|zL4!K1E$jnB}3MPr*- zSQXh&Q+iq@>Y>JMRfQbmbx7rH&hJq(LHloa2Y;i1_^MiHo~nXY{y3P_2xT8WTmoik_QsOjePD5p zj3)g!2*Zhr4K`<9BIDZ?vW)ecK;dBd=|TNV#D0k`6TjXX?VZ+nWVh%HRs5XU8CZS1 z`?`0HQriP1kZA=bzIl!G-s!C_ZJj`!PUiH{p25h%S}SBDeH_SKCX((Sxg$yH(&sui zCSh{(5l*CZ7tR*U#D%?3NADN{Z^VjpVc%;8Cqb;g(qi#0bpAa7Woi5xniK9Qe1$A* zM&Ab2*XbTQpjGHjx*1(8)CZjpE>tw8egS5qSH#DU`eC!+!Vu594siY%b*vsb1))^# zhk|55h(>CB^||OK5IsGiZMB~O*8+_Yin2pNdiZ{_1VJc9J^e~88ikDavx>MtV`TJ` z@ByA=F1*)VI5Z>jgdpnR!BJ-lq-#78D@Nmi_$v)^J{W(7>L9|oTg>WcKJlkb>PLKZ zj#%E8^i()1UdnsAV8D;ok4zTydxyo+rpFIyF#MSREveQo&4##}?_SRN^cvA*`?Ye? zOQSbt^kf{zlQ6sQpb~3-2K|d&$#>nt!(|#Nc}kMHBkJ&vCLPoyxLC_+g^K-6VCo8b z!&UEtGWYkiMA_D0TUF|rd%^$+xf!&!L@*(1_D!)C#t8`P^h^3jdLGqA{Ls0nod;oa z8I5;kDj_9G;kb!>RV;Cta&GpoAF}M9OrD5-g;WEwGuq?q(2^QwVPiKwu2-Ha<3qqS zNS+fpx5^ucG@n{OJ_b@$!oJ}bPO=6x2i2GSUBVEH;u~K9oEeIBAlqFjse%I1`DpG^ zGxUtON}c7*A-q{#q1|63#Ob-@ux2EAfMeYQ5&ENU$QY(!6C89xV8-NDE9i|%et*Gn z1$!a^+ruW`_o^s;+->s~W-(loo>S8@oQ0lu&FvGm*2vEDiV$ahA27$UpX{LWMqE-h zK_q3?XfyZNQ4@a;;+4sPWi$=k>ttoc6(T^!hW7T}?sw2>*w6EFBf<$s-j{ghunE5o zLR`L;o1r3t(<*N)_>LKHjfpMciL>`wGL`_$BWEb~OAC#xS#SyNngA5`@62H1Fz>1uKEw3ydE@j!P zs9E_o3Luo+&ok)*F8!;lH)?MpMcMW_JIO%gdaje(lyD97UbO6+9(p2tsi+S^m0MuQ zlrBTlKMMTg-a0Jnl~777n*ZbJ7+hHqv0=J8Dz=vu2%N~cdM~H!usAI2b%oZlkSkvNDwa* zU<-mU|~dw%jlb0Mh2+#~lqe2q30ok;En zg&^+Z!bR$z%P?1J%&HKF@yE`A&J?V_Ld?ESBPDpaK*@<_)$6iIwk_-l zCk;k1VZm!E7=Gx_$+4{!j6v+nPibX&SYK(o$Qd;{3p67X!!{Ux;L5q$M(8Eb^GoM1 zryU-_Mc11?uUMxcpkVsi?aj|Xvi|s7cApI@jJ)ku%;k+f_|*~;V))@W9DK9OjyB}g`BVjJ1~&bY;m z@sBs%cRhKf;~?vU+35QE5;P^0j2mX|13Npf#CX9V7}wUC8w5N-_-~}xFJkynm21-@ z!exvKHTr($-Oxh|C(`yWP`kjGzs9XvjGynQo<7-T;Ejl}5&K$JKlI+lujb)n7i4N| zrz%TFiw4iWRuxYj2j$dq#}tlN$e{2&h2fP+C~Z(CHpluaV@{O_IddPh^)1XL3Bya@ z>w{W47=HA>?J2j!_&LqJUTaUOSLl$ctc958E*j7^`g9*wfr&2@PvmAF(4~V#OYs-@ zC#?8L4(qR8JE>}mMs+}kqrUYp#y`F@WND;CTcKi7+lM!BoACTmQy2MG0`MMrMQhND zBG)-(UIP_ps?F66>^`D%;{r6*VtzL^LZ#Y2T`k|)1b zg`nb~_J$Y!*c4Ox3jg>egia4$u=o~ij+~#oHawqVjr@%{=a1t}k?H9jlh={{D4YJx zHCl|HpD`~wNEf$9%Z%hgPvgnabWyO4nJFa{%vHS+$MB<6m&uzab{?+vehF#kD1tGe zl6q-bQxp(*&+N-$Ihe0KP75zi2b!o~djH5;K~9)X&Y5`)<~SpI9vyKm))dglMzzKmet0etc%OW1iAr4+mtHn61DS(H_dgL0 zbj;xRfg*1N<|dz0@$rsA(Z|*F7hcm)RiC`_598-N>ir^t?O~{v_BnI>g$4N1rKs9j zGzM9tOYiNqe37BpbeVC}G~_-ni#3(n0L2ToqZ?-rK_#pG_|bL=99(Z#n?ITY@5-1P zw`Q7v(dXCkSvgO1dA!u-qM{{A7ksb!2h*=UyrVglAs>bgEu1A8$gnLHhBjOalPhpc zzIr}VdmbjKxAP6He85%U`>w;L2@z=Xi}dhwp}L~}L8!|zc$*nxN16zO zZ4+U2WAGSI(MEXueo+R+4h?sOCx+mYm62*z#12yRTD4_ds)2(YBKiA-GO*fqM3^a@ z4&@i}om4UkVZixNU=-u|1bp@3vLhY{-E3MOQ+N-$wD~`}eZ3H6NTqI;mK7qh{Cd(W z!VttcuARAp^;cb`ztYOXe~SHvGIA43`|?SV62b2f&G1f8Cf`F!bxJE~o>S}##BhsTl1 zm&jlGp@Za78kUGcIB4#z)K#npPua6aUsI~#MyE5?&q-SpuDkyVLk0M9*(X3=XbZe9 za-KL>6N2PgCKv3pDB<5v(LB`!TXbwVLi_>a9~vhk%T_V|(HU{q#%=AZUdywFN-5)C>$J_;{3;48sMpuQW0E zW{5cKMa*zRAv|Qazy9Wj3{oYWSD{=YMBhDzFJ8d-x%$3}majB94Ca)dCsO>v9XRIB1iSftvP&3#IGC5- zUwuCan}bg4isnX0D6C6P5!0_i_SRg4EZk7c{F)Mh=r6JGi(!>DL@r>r=|l7OU<3Y~ zivQeP;(;!PV#Z&qE}A#&-bq&QMLcBj-hbX$p}Tw{E>oC()fA$AnvmZQ`SN-EZo&BZ z3!mnU#G+m3B5c1d?mZ#qt)mcMhkagd!f(&HFn-PzxNpab=~uqd0Yi7Ij8KZOc}CsY zK;#um;CUIh3LK+Ptae5Fz_px^%j)YFXnmpNaG=r;Le5u}r!js`dGl*MYt$4x{U9QJ zTQx$g`xirOp~xnjXf&Uk!1(#@s%bnFZdDfDPu&dCVlAHq|6tuc{DVxhf;@e(rrs8qDaqkLKk zy~zBeOL8;`_nyqnJN43`JJipU4>s{|L;ngZ_;TD)&NJEiE*cV?p%ss|@a85E=p}WX zu~S6#o%4&r7=HXV7*vVK9)JRptRBjHKFF;4o&6N`1Q3o})vszJq?J6YEMc1m5=;-# zb=gW7A@D1+K5vb9We7^$#c9xxLixYqf>p7lAAbiPh+Cl-*M8cW{=&y~&vSaO22I1z zCq0~Snk2fmW~wZ8D+Im2`Dvyd;~)4^1)DOqVMvi;m&dZk58d30u)!;=0;79$T6o%C zC{;86qCEW}To;=Aad(ao=Sjuqbu-ZesLLH&RWbhYy8p7^JAPs$SR*j`efTmMYs*!qWUaj+RVBkK25tVL@q$mX;=P##F# zPItD2t!!%E`=)Lvn<}WK?c{ZIcRQ$bQ1%;;{l@jxUI;)X(`*ikfrYSdJf5lZpdKtP zhF7q>t%kR~7kgAHFdjA`D=~+B5q&JDd=mc_T>jp>TQ4hs^WbUl*Nf$Glmu@rMF_-W)>C}I3u zYe=PEHgN#Xgf{VtV*F#jWR8;8D-6=26?J}0iJ;0g#w(1cQb5^GS|=4MAvwH-=<{DQ zRQ5E{G8*fzTyqaOSNGk~Yf6=({EQJ8RBRr%;KBSLy48m)A=3~TX|CL|v<(3rw|wr) z1)+0aEf1Y}7eMS>peSe77_j0UDpKF^LdA5?E~4sb_%<^Cyh>~XTrDNIO&AVAhb&0? z>s$%2mVe@t*qDNYi|rpqvHt3+`sU{T-%upFmfjHj(-^(gx|En_x(+YCQHT6qNEf>* z<`phQ>IbbcQTo6BtU!e|3-5;VJcO}z_gJX;Krr*-*0!<*!v9%AMSQ6lzBh35QPJiC z-d(No7q2jVHk9laMY%9gs@jzPa2J{+(SJs0m*;=@m%_vM3406|BFK)a`FRYZwBP_@e5q12pK0*lp!a{`X+$HbXo; z5`ff`jo?hC1hU5ymWz%y1aohbx8_G1@N=U4CCBkA)EH{~`@I~X>|m##r^$TKoPLj^ zn|7p_J@gD+!t^Vhikr3C>aS4&^F-#$h6I?NvE}D9Cu;*O3oRZ0AT|8jxmBY_?1w9I4)*|>Qiik4{;g5K10#DaTEl#_2eAFPtxFN z#2rF;w?JS)o{b!@JK&uArD47BBQ>Je-z(bm(6*29G z>L2fX_TD*x=%=Eo>tmuIj4HfFQa~SR9xYt!F3ke#m|(WbBYRlQP~+$*e&V!b10!rcyw&{1pWSga>FDP>#sCy@`x1bq4?TypHNmcH0?a4Y&vp9Wh2*= z{$c&q!td2Ciz`@t3t!GS@1cx@SU&CsyzLNE9;T|7+Mk4%%mQ;YEmo)~=48xgOuy1{ zyUQN(=qb8Ac2sAN;fGylc*nP`P0$+GrtZAtkIeoYH4>j&gMp3Y2A!kp2lTOj7d zK-4nGd~Sr!7&Uymrm5A3kDHbCYLR?B4SBk63t}?7Q9!No%+gIwRiU)*@mVGGw<*Hy&*v)G`*UML&L9xerFz!BP#=QnBci>@Swh^6ft&A|vpnE+ z$+VyNRyRD8J2yC&tArL$CmZmPd7=Xz@hA1tx+t9SOXQrU5Q>kKu_H^dKp*}1HCXj$ zK}G5D_dA;mNdFR%)n;WMHZ8iAS~2?!B^8M1^^W4u73u?HP7FWT3Q2lACemQQi#xFM zMFiMhT0QN5@Eu+==yOCm5#h?8uogGiZbGjloufd3Ju>jZM@95;U=^!e@gpV)SOVl4 z+1_t}UpZfQ>|=b~LR8nmK#nOUzG(+NIcOy z2k&C~erNbX_Ka0RUKh>%WO(S=vnVF{f~Q4{&lEFJ1qM^*oHx6L_i9CE(pXk zat}djlIK(h@z>z{-GvWCV`|9bi4?E$=PihRbyd{5Zxqt&(}(y+Dq*14x8aioro$&6 zq_0rN1%<~aj=>9&KQ8jr5oS1{c0JD|vX)dUc*(Oj?@)63) ziYdy(@MD*u>4cPvDSCF$t;9&@4spZ2DmSccVPp86NT|LRdZV^_SBpym4V?B~v$*>W zsB6}1hWka3ZVYpJtZyO6^Sot>RjLPf<;}pT>}v4hUyEZo9gbL!CPXG&JkXVb!Th}I zTM&43nNvQS5B(Exy)QdR2{buc&%Nb>QTnAR%b>`4&_ClNs*2@H_UtXc#oZ z6BsYV;E>3Q&UkNaA@n=M=-xkTj_6XE*`)Bz(QtAD-Tsed@RjN9D}CpIJO~ye%W_A+ zUZq0!3+8{^u&yGaCGbK!2}hRg82>;gY~AWYosoIGMAj`#zq%Oe*dUDMOFl@nTz;PF zi9~)0x>Eg}g=WUC*Yucv<+K%JS1|2`8gGX0ED#)n=)IWFM3_G)`^^7_@!V%1e`tQ? zfZrRr?4BvyQMO0KGUL*!m_H~MVv)?>=7{jF4f{NvH5sd%duZ+%Hp29!7VhFRoX5xXt4vMnzikMP!YGPX{&m?5Gl=I~B8q zZEpg$p%u>9&%0n`M0@Ha#Ra5%a*&Q2^Uvv6K0WQh_{S{K4C%GW3Fy-xy;1u65~?t& zp=g(PMrW!A%1XpmftcVxk?nRLl*}*CIp%(Wu_Mi=BM3?X zj{2GX&0xfQR##;o^FM<1pBq)JCqO^h1N?1LS@bWWJ*Zmp1fs#GQwx+Bg{A$SHTxhp zBxmQFcU>?S@|B9?u9ABKnMGh|XEg_6C+acG5wt<9jXkPQu`N6^QU>|u>h>r$JZ3G* zNehjBd{dv>p@9gVx3f%Hv!NcMylLmH(tbV{44Me`A+EB$To zyKGGX3CPM@fMg07k)Ysc=sYMec-FFT7D3KHc)G8+H)79;A`hA@huqk6EjQEBLB1#W zcEIUYP~b6d2*&saYutu$W$pl+qhS8rB|ZbuiIM{&AH(2$V+!7zPeJI7rt$+O<`j6` z2U4c_|>SL=|zC@keGjt%t2PPY)QtQ9kqKs34 zg*pPwK==Nu*}ttE5XGa2B5_-WA?sugg`69ZmR95X#BU5Ly*t^0bj#r0`KE8ZvqM0B z;lgv>kR4=ji|WM1y&7mRI4Jk5m4=DnJ*ilQba-&Pt+%UaaFKkRk(xKD~DI`CX? zr^a-;gM5KbHCA+Dg-G7R#21xn?-2B+j05pacL$kX4@9sMaYke`0m8pN@H=>12)Tvs zVeV*2?1khNrwUBJN~0N|@5TBnq1vcmTFgH`bbXg8bhrw`R|B)C@?WFy%f=7q{J===f$;qpu4nt+%NsQG9CxP&AaaL7pYqUZq*1u{9hGr5q* z_|s_e`wXQ0Ione>=7SciXre76#n7{y@NTl?D5$iyuo|6yh2{l`Bl_C0ZK$Wzl%D>y zhu`PAMk09dk^9@gDGIM3G^Af-EW>OL=hga(eaB4EucPAf>oJ93wEbRfS-u_)6gB;s z600FH>vqOs_C4eoY)?Ch^;b3cqjZ)6TaZ}pIee}>1O?$!s=HQ^!>IDk!_g`?G&HEI z9fsu-TW4cR46*#x&O5C*mnA<`LdqLkfZ@k2uOw=5%>STH-l_?~w$qN!g+_|u{88QD z-_imHLG(}5FWp22JHy10UkqP~F_aEJlA! zSCbE4&TCkPq`HbcOBq#U5C}@77=9GqVrGpg#lx*KuUCIs@)kSs@yjDItUlBPF0L!l z1fbI7VRsfxzjEKq3E+8;Z4;C=bG|5e8AYU&c~BkXg*c~Sr26CKjGfP5>NIytV^RafXLGxc-nxxG@v(Q_Fj)tdk-8%^poaRw zpDo4_`+=*2yU8(@FTv$S@G)s(`rc=vx^Dtqa5;an&_eMgI{kf8xP-qMI+DUpbz%9d z{@3SY@vwY}1I@!GY;O+WvpRlb!u<1i{NBzbEMHRPPrF}=<*(EVStYMYZ6nglH>yR; zYQdgF)Z*_&%s(eN-cQ2(b4&Ldlh5<_;b+T!##F^1$oTTDhR`aaR|h(GYuYwJJl>$B zf=C3-do#UJ{aQ2Q#A`?!T^9SRRw{ z+M7q-nEz3|^1Xr_o5pN7-oO9gl{H$;l-vp=@*-A0}ilLO!_~+XWXvul?v>fx#Pw&nQFNvVb(m9uJgDS6hx)q7$f@b5@ZF&+?t>nU`qZ&`NgvVP%K+;F7u%AID(&Roh~zw2-+L5=wTuw9U8%9j+%DIbsLk`5Mn< zDPAQs&XLRK!-VB2bZE;>uG=6-+PCMu9rB=9N^tqDR3!wjO5kM)`k{T=83PUmKO}t7 zi#J#BC32;|@UiN7AiCv!=BjfqKJK~F$)f9i(|{~L)=e7fqpYhd4m$#2h$1ewbBJgS zUJ$D%w^G}p5QetuQ_YXiQ)|Dhiv?Bila177g*^eXrIh9JB19#jsr>0%e%8Q4@AsgX7LpNt=yJ7T7Tt9>zLVTifm z{#K>-EKt(jQErdIzGo?4$4?aWflHD14WBG_)a7(tjaEVg0YQ~$;_fa?FAq%&({!_)rT}hBH@N6E3zL+=D4??}f*lyX2ul=8Y-Z{Pdi|r8_da7~L z`Pwb?lkXzy^mk`y6DHsppbSLZgy#A?X1*vh{zoRg`*rlG>h%F$sts~ukbP{?;De4g zs?^i5eBKdGCB$XvInsBX_-W(60RJ>K9%;zn;f&}v{a(9@R%|GT;ZBkb{NqdoCc#QI+_a(t5SpNaXE{NLkW+;N9a(+mQ^5L>ufcMuXh;E4G`4;WR#_0eBi zdPT0s-=HUQ-$#3DR_x7N&x3^ULfBp#*etkT52;+0gngOSAb4iz?#Q7HQYtldA*+9d zk`Hu?G%^1?uCf?^CfE!G5nE*>_E12>d=#lI{uIz?Cekxv`NRW5#+_xsA{c$Zlc_x( zipoMfUq_vDvNigYDWLdlp*025uYN71Dz%1LB8#?rHre&dFnl5Hh=xrB4g6B2 z62bDjPsNw-9dVC>OU)f++puYv*SNatj^*2x3*0kT-kTwZYpHuNTnlhxxVN1Y%U_wg zI2j-H;v+ou@9Ws%@Sxo!F}986+eObm#81QOqk09snOGpQ6e`JTc1z5exe0|U>x@)eY7=Extt54+{T}1aqdrD&BHyPg9||2{-j`eMB5`937$i-0Gcpz|!| z4^BzhSsOZ(K~&am;pM+W;Gmwo74>unWxug6MH@8`@~l5fEmsCQvs5drdDCGe?8G## ztPn)guixx@5CWIp_UFo9cchcAv54-Qpv^D(!%{(pz-v>! zOm<=uvJ#413BUOuhcVmoZ++S*oyFLhZPgO(&RuBub0~<8f;{Y^uzcdF@3LulfAtZA z{QV04ngr01`Rm7RUI;NM|Jn}!7^3HWac^Vey`d**K+y>6uU=j}nrXoLEAF8oVXuSd(i*OPY9j)9ZvL2-5mZsKJgRRXpL_E ztXpICD+HZ{fk^&)^?-5}f9>T~LzzWN$3=PvWN=yKiUQ`JXPyyvq`kHUl`p3)+A+O8 zv8(K~tvCgY^*H>BAf5sv`)sT3dO%9KNwdeiN)r zhaU>vy*8MCZV+>l#tN&CM*2(VdNBOh_xyq*!TiBb-_B<7d<_GFoNBU|Vja}F8^hI0 zn*vpi+7-QbD&f4d<4cuLL-eO(=;LtvGK9*xpS_ZDAH`k&{ex zseCMdW$pX%-~*OVG=4E8Ecr(gIr_gOGsfyeQ#zl35%UKbqP(~?zW5-&mp1aHjng2? zNEzQKz5&+@Jf_sK{FQuZtC90T3H-CGQ0Us5g4DYrmE!YFkX>N*wp>WoyRtfZY>?`*II*(qj4DI6A80@f8r9ZMDZK&O`R|;a>A_A27I=`EyM@ z80{94Q@LXK#DMu@Hy2E&lokq67I$3+J-q!zj)WUvS!Mpt&ut8txGfahFnw3iVseQG z^UvR!+*q)#*+n5%AAE=QYv2Lhg|vB2%s&qjWvdWOhun6~Kbr3gAuz$AY{4Z2<~myp zed#RFS7uk$ny2sKR$XZPB?bqi(%-WlVR;Wlyf6Rd^Ue^2%RV-;VEm&(`^Oh)QU&yq z?ozwd841MH(_mSCD;%AF|Lk<~tQ`s_?y9T9`YW#_{K$;rV3cd~*4w^60SJXhf);fO z!B(#=NE*Y1(d{$Jt@hq9vK4G4iuG3nCVGO=Sbg+W3anqi>Z7$)fSdN4BbHMq8mkD5=@z)vja*_K5hJh(fTROizHym(BJE73_1KjQnc=@jOF{JtwV zOqC1WPlfF58)jg}T2HQlF$4{78nxlqzeadUo=ouxQ81`;?Q)!;4w6eJd9OB<1tUZP zH;uOKfihj^YI2k^5-q=0T=F>tMdGS%&0+p|MaxEckyw}5VzJ1Se|#YvjkXDpVfia{ zp5)|$l4@*g$@paZ^)MvUI&PVQ)kps9KjPWT82_LT8Sy9cK>zO(R|fa-{Qh@Nu>I5q zcb||Kw)jsdm^l6bxQ7JV+KmpOncBjEgy=Hv*O7Xu+f#QGvh;0|!1Owferfiv0UrsD zYstRr%4Y_g=7r}&c~_}%^8WE26xA&i{-8t{cdAN{EHotkwkhNQR8q&#o#1M}@{ z;mybNI05EqQV+&u@H-X9kuJ!!6`3NB{Fozj_6kA5kHdws3Vr^I{0C(I!X2 zb!=3GcHS?iJZl|_DV^^V`Rk&j$NVngdGW~oLX`tv(P`XyZ0RZQ&L!Lvfm53;xBdcG zCadpIycG&adS{eWTnajz%z^zf+}&9U3LInZ0?MYu$0fEhJPS@C z##P-XyiX)1hhrLv3X6`k2CJJ30@q?`aS^GrtsnLn3&y;L#gxds)2 zS}nSx2r`qaThD(%mzc6qi$OPB7QUMG(PawG-PIg^cbNjW_dzO>^cfS5DA6_W+t5W^ z`Qs|Qrp!;^=xO^WfOZ0lMyy*{PTa)t)k+5UFgT$(`Wb8;Bo@VFQ)jUn--lW)zxX*C zH8gf>(B1wa4Q@2eNXbn37VgK{L5g5bW|Xj4abNeM0}4Cq^|>sv8xGf6?=f3ELhnvm zzM8>10GEWQ_CuUCW|c??@p|OIQ=?`$CU z|4~B!|Eb^ljPdH%nG{g+B*9Gu>(N&AWEPDu3_4R&vRHDp2>zRXl-w9}om(vj^5J$K z`RsJ)=I(yBOW6wbv*bEYSk~Zkh4!v;$^dY>W_bu>J({Qj?DfA6``_<3-mMm2W=Mg? z`H%Yx7>D`qb%|EV%I;wr*niI%zvSF2#%2(^4n%f2unbq^%P`S+%tQH%H@Cfqas2=8 z3-TE9K7DBcQihplMzY8LcYVa)=iF6H0~?7<4Sj-Pkof7^L@LZ%_LYph^m1!ym`-RV=C2jx%C z+m3vE{&yeS(%wBCYzFZS(^>ie(p<1#S~{KSz6?6A{&M}*7Y3~rHLFvY23B)XyE@CL z3~oo>a9)`jg0JNpUg>Xlkom?%iIMFZaD8fux|L^+$*Q2z~vA!tgG5+`SttEzY zBO?i5{rE=Xx_06JuHUOU&2|**4L7&BQZuj~jUih6rr7udd|~!;Z#=Q~zx&>-Co2*t zWx&&)@A1C327&ewoqUW&8kSkjFK)y%u(^Ck*OG`1Xv}}2r09TUodZaQJjr|DUt1Sr z5K}Hx-m5GAKMm}^_w(Q5?gcrLBhRS+U2kg5JML)6f?Y%_xBSQcf7k!J|Aji$nlFL* z|E>$F-6AH#JjNG)W_L0$4Jtns j4u0ZYb7@6?4V*3853DM{mcIkN&isF literal 0 HcmV?d00001 diff --git a/src/documents/tests/test_classifier.py b/src/documents/tests/test_classifier.py index 43c38b691..14673ae65 100644 --- a/src/documents/tests/test_classifier.py +++ b/src/documents/tests/test_classifier.py @@ -130,6 +130,15 @@ class TestClassifier(DirectoriesMixin, TestCase): new_classifier.reload() self.assertFalse(new_classifier.train()) + @override_settings(MODEL_FILE=os.path.join(os.path.dirname(__file__), "data", "model.pickle")) + def test_load_and_classify(self): + self.generate_test_data() + + new_classifier = DocumentClassifier() + new_classifier.reload() + + self.assertCountEqual(new_classifier.predict_tags(self.doc2.content), [45, 12]) + def test_one_correspondent_predict(self): c1 = Correspondent.objects.create(name="c1", matching_algorithm=Correspondent.MATCH_AUTO) doc1 = Document.objects.create(title="doc1", content="this is a document from c1", correspondent=c1, checksum="A") diff --git a/src/paperless/settings.py b/src/paperless/settings.py index b6d01ba53..56e91695e 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -4,7 +4,6 @@ import multiprocessing import os import re -import dateparser from dotenv import load_dotenv from django.utils.translation import gettext_lazy as _ @@ -491,7 +490,11 @@ if PAPERLESS_TIKA_ENABLED: # List dates that should be ignored when trying to parse date from document text IGNORE_DATES = set() -for s in os.getenv("PAPERLESS_IGNORE_DATES", "").split(","): - d = dateparser.parse(s) - if d: - IGNORE_DATES.add(d.date()) + +if os.getenv("PAPERLESS_IGNORE_DATES", ""): + import dateparser + + for s in os.getenv("PAPERLESS_IGNORE_DATES", "").split(","): + d = dateparser.parse(s) + if d: + IGNORE_DATES.add(d.date()) From 5718f50c4f16982232128ea6da83067e48b6bc5a Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 4 Feb 2021 15:46:51 +0100 Subject: [PATCH 026/225] increase worker timeout for Raspberry Pi --- gunicorn.conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gunicorn.conf.py b/gunicorn.conf.py index a67b20d0c..f80be548d 100644 --- a/gunicorn.conf.py +++ b/gunicorn.conf.py @@ -1,7 +1,7 @@ bind = '0.0.0.0:8000' workers = 2 worker_class = 'uvicorn.workers.UvicornWorker' -timeout = 20 +timeout = 60 def pre_fork(server, worker): pass From 222fb483789ef5ec78064427f74c8afef07c0294 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 4 Feb 2021 16:55:48 +0100 Subject: [PATCH 027/225] dismiss completed dismisses successful items only --- .../upload-file-widget/upload-file-widget.component.html | 2 +- .../upload-file-widget/upload-file-widget.component.ts | 4 ++-- src-ui/src/app/services/consumer-status.service.ts | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html b/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html index eb1e1da73..c4f7fa21c 100644 --- a/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html +++ b/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html @@ -1,6 +1,6 @@

    - + Dismiss completed  diff --git a/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts b/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts index 2412290c0..e74e2bc41 100644 --- a/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts +++ b/src-ui/src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts @@ -92,8 +92,8 @@ export class UploadFileWidgetComponent implements OnInit { this.consumerStatusService.dismiss(status) } - dismissAll() { - this.consumerStatusService.dismissAll() + dismissCompleted() { + this.consumerStatusService.dismissCompleted() } ngOnInit(): void { diff --git a/src-ui/src/app/services/consumer-status.service.ts b/src-ui/src/app/services/consumer-status.service.ts index 38f29a528..23df0871f 100644 --- a/src-ui/src/app/services/consumer-status.service.ts +++ b/src-ui/src/app/services/consumer-status.service.ts @@ -176,8 +176,8 @@ export class ConsumerStatusService { } } - dismissAll() { - this.consumerStatus = this.consumerStatus.filter(status => status.phase < FileStatusPhase.SUCCESS) + dismissCompleted() { + this.consumerStatus = this.consumerStatus.filter(status => status.phase != FileStatusPhase.SUCCESS) } onDocumentConsumptionFinished() { From 138f38d1f379a405b63b36cab9f8d8b7d8458746 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 4 Feb 2021 23:32:11 +0100 Subject: [PATCH 028/225] fix lost messages when queuing many files --- src/paperless/settings.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/paperless/settings.py b/src/paperless/settings.py index 56e91695e..5a3c206ed 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -160,6 +160,8 @@ CHANNEL_LAYERS = { "BACKEND": "channels_redis.core.RedisChannelLayer", "CONFIG": { "hosts": [os.getenv("PAPERLESS_REDIS", "redis://localhost:6379")], + "capacity": 2000, # default 100 + "expiry": 15, # default 60 }, }, } From 0d3ab3aaf7479d2b644c1746497f669d9f43d461 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 4 Feb 2021 23:40:53 +0100 Subject: [PATCH 029/225] remove lots of unused code --- .../management/commands/document_archiver.py | 7 +------ .../commands/document_create_classifier.py | 3 +-- .../management/commands/document_exporter.py | 3 +-- .../management/commands/document_importer.py | 3 +-- src/documents/management/commands/document_index.py | 9 +-------- src/documents/management/commands/document_logs.py | 12 ------------ .../management/commands/document_renamer.py | 9 +-------- .../management/commands/document_retagger.py | 9 +-------- .../management/commands/document_thumbnails.py | 10 +--------- src/documents/mixins.py | 9 --------- 10 files changed, 8 insertions(+), 66 deletions(-) delete mode 100644 src/documents/management/commands/document_logs.py delete mode 100755 src/documents/mixins.py diff --git a/src/documents/management/commands/document_archiver.py b/src/documents/management/commands/document_archiver.py index 01bd819c5..9e0d8fabf 100644 --- a/src/documents/management/commands/document_archiver.py +++ b/src/documents/management/commands/document_archiver.py @@ -17,7 +17,6 @@ from whoosh.writing import AsyncWriter from documents.models import Document from ... import index from ...file_handling import create_source_path_directory -from ...mixins import Renderable from ...parsers import get_parser_class_for_mime_type @@ -62,7 +61,7 @@ def handle_document(document_id): parser.cleanup() -class Command(Renderable, BaseCommand): +class Command(BaseCommand): help = """ Using the current classification model, assigns correspondents, tags @@ -71,10 +70,6 @@ class Command(Renderable, BaseCommand): modified) after their initial import. """.replace(" ", "") - def __init__(self, *args, **kwargs): - self.verbosity = 0 - BaseCommand.__init__(self, *args, **kwargs) - def add_arguments(self, parser): parser.add_argument( "-f", "--overwrite", diff --git a/src/documents/management/commands/document_create_classifier.py b/src/documents/management/commands/document_create_classifier.py index fbfb7f7e6..a4ede88b5 100755 --- a/src/documents/management/commands/document_create_classifier.py +++ b/src/documents/management/commands/document_create_classifier.py @@ -1,10 +1,9 @@ from django.core.management.base import BaseCommand -from ...mixins import Renderable from ...tasks import train_classifier -class Command(Renderable, BaseCommand): +class Command(BaseCommand): help = """ Trains the classifier on your data and saves the resulting models to a diff --git a/src/documents/management/commands/document_exporter.py b/src/documents/management/commands/document_exporter.py index 37fcf2024..1505b0856 100644 --- a/src/documents/management/commands/document_exporter.py +++ b/src/documents/management/commands/document_exporter.py @@ -16,10 +16,9 @@ from documents.settings import EXPORTER_FILE_NAME, EXPORTER_THUMBNAIL_NAME, \ EXPORTER_ARCHIVE_NAME from paperless.db import GnuPG from ...file_handling import generate_filename, delete_empty_directories -from ...mixins import Renderable -class Command(Renderable, BaseCommand): +class Command(BaseCommand): help = """ Decrypt and rename all files in our collection into a given target diff --git a/src/documents/management/commands/document_importer.py b/src/documents/management/commands/document_importer.py index a2e19e3cc..7e5d32085 100644 --- a/src/documents/management/commands/document_importer.py +++ b/src/documents/management/commands/document_importer.py @@ -15,7 +15,6 @@ from documents.models import Document from documents.settings import EXPORTER_FILE_NAME, EXPORTER_THUMBNAIL_NAME, \ EXPORTER_ARCHIVE_NAME from ...file_handling import create_source_path_directory -from ...mixins import Renderable from ...signals.handlers import update_filename_and_move_files @@ -28,7 +27,7 @@ def disable_signal(sig, receiver, sender): sig.connect(receiver=receiver, sender=sender) -class Command(Renderable, BaseCommand): +class Command(BaseCommand): help = """ Using a manifest.json file, load the data from there, and import the diff --git a/src/documents/management/commands/document_index.py b/src/documents/management/commands/document_index.py index 08e20e1d2..5faa70b9f 100644 --- a/src/documents/management/commands/document_index.py +++ b/src/documents/management/commands/document_index.py @@ -1,24 +1,17 @@ from django.core.management import BaseCommand from django.db import transaction -from documents.mixins import Renderable from documents.tasks import index_reindex, index_optimize -class Command(Renderable, BaseCommand): +class Command(BaseCommand): help = "Manages the document index." - def __init__(self, *args, **kwargs): - self.verbosity = 0 - BaseCommand.__init__(self, *args, **kwargs) - def add_arguments(self, parser): parser.add_argument("command", choices=['reindex', 'optimize']) def handle(self, *args, **options): - - self.verbosity = options["verbosity"] with transaction.atomic(): if options['command'] == 'reindex': index_reindex() diff --git a/src/documents/management/commands/document_logs.py b/src/documents/management/commands/document_logs.py deleted file mode 100644 index 06efc3850..000000000 --- a/src/documents/management/commands/document_logs.py +++ /dev/null @@ -1,12 +0,0 @@ -from django.core.management.base import BaseCommand - -from documents.models import Log - - -class Command(BaseCommand): - - help = "A quick & dirty way to see what's in the logs" - - def handle(self, *args, **options): - for log in Log.objects.order_by("pk"): - print(log) diff --git a/src/documents/management/commands/document_renamer.py b/src/documents/management/commands/document_renamer.py index 745d2d03d..f311715c6 100644 --- a/src/documents/management/commands/document_renamer.py +++ b/src/documents/management/commands/document_renamer.py @@ -5,23 +5,16 @@ from django.core.management.base import BaseCommand from django.db.models.signals import post_save from documents.models import Document -from ...mixins import Renderable -class Command(Renderable, BaseCommand): +class Command(BaseCommand): help = """ This will rename all documents to match the latest filename format. """.replace(" ", "") - def __init__(self, *args, **kwargs): - self.verbosity = 0 - BaseCommand.__init__(self, *args, **kwargs) - def handle(self, *args, **options): - self.verbosity = options["verbosity"] - logging.getLogger().handlers[0].level = logging.ERROR for document in tqdm.tqdm(Document.objects.all()): diff --git a/src/documents/management/commands/document_retagger.py b/src/documents/management/commands/document_retagger.py index b2f5d8918..a37ab9ef2 100755 --- a/src/documents/management/commands/document_retagger.py +++ b/src/documents/management/commands/document_retagger.py @@ -4,11 +4,10 @@ from django.core.management.base import BaseCommand from documents.classifier import load_classifier from documents.models import Document -from ...mixins import Renderable from ...signals.handlers import set_correspondent, set_document_type, set_tags -class Command(Renderable, BaseCommand): +class Command(BaseCommand): help = """ Using the current classification model, assigns correspondents, tags @@ -17,10 +16,6 @@ class Command(Renderable, BaseCommand): modified) after their initial import. """.replace(" ", "") - def __init__(self, *args, **kwargs): - self.verbosity = 0 - BaseCommand.__init__(self, *args, **kwargs) - def add_arguments(self, parser): parser.add_argument( "-c", "--correspondent", @@ -61,8 +56,6 @@ class Command(Renderable, BaseCommand): def handle(self, *args, **options): - self.verbosity = options["verbosity"] - if options["inbox_only"]: queryset = Document.objects.filter(tags__is_inbox_tag=True) else: diff --git a/src/documents/management/commands/document_thumbnails.py b/src/documents/management/commands/document_thumbnails.py index 6f6451599..cf2cbeb77 100644 --- a/src/documents/management/commands/document_thumbnails.py +++ b/src/documents/management/commands/document_thumbnails.py @@ -7,7 +7,6 @@ from django import db from django.core.management.base import BaseCommand from documents.models import Document -from ...mixins import Renderable from ...parsers import get_parser_class_for_mime_type @@ -30,16 +29,12 @@ def _process_document(doc_in): parser.cleanup() -class Command(Renderable, BaseCommand): +class Command(BaseCommand): help = """ This will regenerate the thumbnails for all documents. """.replace(" ", "") - def __init__(self, *args, **kwargs): - self.verbosity = 0 - BaseCommand.__init__(self, *args, **kwargs) - def add_arguments(self, parser): parser.add_argument( "-d", "--document", @@ -51,9 +46,6 @@ class Command(Renderable, BaseCommand): ) def handle(self, *args, **options): - - self.verbosity = options["verbosity"] - logging.getLogger().handlers[0].level = logging.ERROR if options['document']: diff --git a/src/documents/mixins.py b/src/documents/mixins.py deleted file mode 100755 index 949be02e8..000000000 --- a/src/documents/mixins.py +++ /dev/null @@ -1,9 +0,0 @@ -class Renderable: - """ - A handy mixin to make it easier/cleaner to print output based on a - verbosity value. - """ - - def _render(self, text, verbosity): - if self.verbosity >= verbosity: - print(text) From 431d4fd8e460bf9e93b912fb5a55b76aaa6933df Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Fri, 5 Feb 2021 01:10:29 +0100 Subject: [PATCH 030/225] rework most of the logging --- src/documents/classifier.py | 20 +++++----- src/documents/consumer.py | 2 + src/documents/file_handling.py | 5 ++- src/documents/index.py | 2 +- src/documents/loggers.py | 26 ++++--------- .../management/commands/document_archiver.py | 2 +- .../management/commands/document_consumer.py | 13 ++----- .../management/commands/document_retagger.py | 5 ++- src/documents/matching.py | 2 +- src/documents/parsers.py | 6 ++- src/documents/signals/handlers.py | 39 +++++++++---------- src/documents/tasks.py | 9 +++-- src/documents/views.py | 5 ++- src/paperless_mail/mail.py | 2 + src/paperless_mail/tasks.py | 7 +++- src/paperless_tesseract/parsers.py | 2 + src/paperless_text/parsers.py | 2 + src/paperless_tika/parsers.py | 2 + 18 files changed, 80 insertions(+), 71 deletions(-) diff --git a/src/documents/classifier.py b/src/documents/classifier.py index 3051b82a5..2acebe04a 100755 --- a/src/documents/classifier.py +++ b/src/documents/classifier.py @@ -13,7 +13,7 @@ class IncompatibleClassifierVersionError(Exception): pass -logger = logging.getLogger(__name__) +logger = logging.getLogger("paperless.classifier") def preprocess_content(content): @@ -115,7 +115,7 @@ class DocumentClassifier(object): labels_document_type = list() # Step 1: Extract and preprocess training data from the database. - logging.getLogger(__name__).debug("Gathering data from database...") + logger.debug("Gathering data from database...") m = hashlib.sha1() for doc in Document.objects.order_by('pk').exclude(tags__is_inbox_tag=True): # NOQA: E501 preprocessed_content = preprocess_content(doc.content) @@ -162,7 +162,7 @@ class DocumentClassifier(object): num_correspondents = len(set(labels_correspondent) | {-1}) - 1 num_document_types = len(set(labels_document_type) | {-1}) - 1 - logging.getLogger(__name__).debug( + logger.debug( "{} documents, {} tag(s), {} correspondent(s), " "{} document type(s).".format( len(data), @@ -173,7 +173,7 @@ class DocumentClassifier(object): ) # Step 2: vectorize data - logging.getLogger(__name__).debug("Vectorizing data...") + logger.debug("Vectorizing data...") self.data_vectorizer = CountVectorizer( analyzer="word", ngram_range=(1, 2), @@ -183,7 +183,7 @@ class DocumentClassifier(object): # Step 3: train the classifiers if num_tags > 0: - logging.getLogger(__name__).debug("Training tags classifier...") + logger.debug("Training tags classifier...") if num_tags == 1: # Special case where only one tag has auto: @@ -202,12 +202,12 @@ class DocumentClassifier(object): self.tags_classifier.fit(data_vectorized, labels_tags_vectorized) else: self.tags_classifier = None - logging.getLogger(__name__).debug( + logger.debug( "There are no tags. Not training tags classifier." ) if num_correspondents > 0: - logging.getLogger(__name__).debug( + logger.debug( "Training correspondent classifier..." ) self.correspondent_classifier = MLPClassifier(tol=0.01) @@ -217,13 +217,13 @@ class DocumentClassifier(object): ) else: self.correspondent_classifier = None - logging.getLogger(__name__).debug( + logger.debug( "There are no correspondents. Not training correspondent " "classifier." ) if num_document_types > 0: - logging.getLogger(__name__).debug( + logger.debug( "Training document type classifier..." ) self.document_type_classifier = MLPClassifier(tol=0.01) @@ -233,7 +233,7 @@ class DocumentClassifier(object): ) else: self.document_type_classifier = None - logging.getLogger(__name__).debug( + logger.debug( "There are no document types. Not training document type " "classifier." ) diff --git a/src/documents/consumer.py b/src/documents/consumer.py index f8f7576ef..23988c7c1 100755 --- a/src/documents/consumer.py +++ b/src/documents/consumer.py @@ -47,6 +47,8 @@ MESSAGE_FINISHED = "finished" class Consumer(LoggingMixin): + logging_name = "paperless.consumer" + def _send_progress(self, current_progress, max_progress, status, message=None, document_id=None): payload = { diff --git a/src/documents/file_handling.py b/src/documents/file_handling.py index dd798e8ab..6ee97f5c3 100644 --- a/src/documents/file_handling.py +++ b/src/documents/file_handling.py @@ -8,6 +8,9 @@ from django.conf import settings from django.template.defaultfilters import slugify +logger = logging.getLogger("paperless.filehandling") + + class defaultdictNoStr(defaultdict): def __str__(self): @@ -140,7 +143,7 @@ def generate_filename(doc, counter=0, append_gpg=True): path = path.strip(os.sep) except (ValueError, KeyError, IndexError): - logging.getLogger(__name__).warning( + logger.warning( f"Invalid PAPERLESS_FILENAME_FORMAT: " f"{settings.PAPERLESS_FILENAME_FORMAT}, falling back to default") diff --git a/src/documents/index.py b/src/documents/index.py index 51197e252..bc17daeb1 100644 --- a/src/documents/index.py +++ b/src/documents/index.py @@ -12,7 +12,7 @@ from whoosh.qparser.dateparse import DateParserPlugin from whoosh.writing import AsyncWriter -logger = logging.getLogger(__name__) +logger = logging.getLogger("paperless.index") class JsonFormatter(Formatter): diff --git a/src/documents/loggers.py b/src/documents/loggers.py index 863bc0c34..4b0f03552 100644 --- a/src/documents/loggers.py +++ b/src/documents/loggers.py @@ -4,33 +4,21 @@ import uuid from django.conf import settings -class PaperlessHandler(logging.Handler): - def emit(self, record): - if settings.DISABLE_DBHANDLER: - return - - # We have to do the import here or Django will barf when it tries to - # load this because the apps aren't loaded at that point - from .models import Log - - kwargs = {"message": record.msg, "level": record.levelno} - - if hasattr(record, "group"): - kwargs["group"] = record.group - - Log.objects.create(**kwargs) - - class LoggingMixin: logging_group = None + logging_name = None + def renew_logging_group(self): self.logging_group = uuid.uuid4() def log(self, level, message, **kwargs): - target = ".".join([self.__class__.__module__, self.__class__.__name__]) - logger = logging.getLogger(target) + if self.logging_name: + logger = logging.getLogger(self.logging_name) + else: + name = ".".join([self.__class__.__module__, self.__class__.__name__]) + logger = logging.getLogger(name) getattr(logger, level)(message, extra={ "group": self.logging_group diff --git a/src/documents/management/commands/document_archiver.py b/src/documents/management/commands/document_archiver.py index 9e0d8fabf..d2ff9c8c2 100644 --- a/src/documents/management/commands/document_archiver.py +++ b/src/documents/management/commands/document_archiver.py @@ -20,7 +20,7 @@ from ...file_handling import create_source_path_directory from ...parsers import get_parser_class_for_mime_type -logger = logging.getLogger(__name__) +logger = logging.getLogger("paperless.management.archiver") def handle_document(document_id): diff --git a/src/documents/management/commands/document_consumer.py b/src/documents/management/commands/document_consumer.py index 232a5b23c..c01743628 100644 --- a/src/documents/management/commands/document_consumer.py +++ b/src/documents/management/commands/document_consumer.py @@ -17,7 +17,7 @@ try: except ImportError: INotify = flags = None -logger = logging.getLogger(__name__) +logger = logging.getLogger("paperless.management.consumer") def _tags_from_path(filepath): @@ -108,12 +108,7 @@ class Command(BaseCommand): # This is here primarily for the tests and is irrelevant in production. stop_flag = False - def __init__(self, *args, **kwargs): - - self.logger = logging.getLogger(__name__) - - BaseCommand.__init__(self, *args, **kwargs) - self.observer = None + observer = None def add_arguments(self, parser): parser.add_argument( @@ -161,7 +156,7 @@ class Command(BaseCommand): logger.debug("Consumer exiting.") def handle_polling(self, directory, recursive): - logging.getLogger(__name__).info( + logger.info( f"Polling directory for changes: {directory}") self.observer = PollingObserver(timeout=settings.CONSUMER_POLLING) self.observer.schedule(Handler(), directory, recursive=recursive) @@ -176,7 +171,7 @@ class Command(BaseCommand): self.observer.join() def handle_inotify(self, directory, recursive): - logging.getLogger(__name__).info( + logger.info( f"Using inotify to watch directory for changes: {directory}") inotify = INotify() diff --git a/src/documents/management/commands/document_retagger.py b/src/documents/management/commands/document_retagger.py index a37ab9ef2..854117fdd 100755 --- a/src/documents/management/commands/document_retagger.py +++ b/src/documents/management/commands/document_retagger.py @@ -7,6 +7,9 @@ from documents.models import Document from ...signals.handlers import set_correspondent, set_document_type, set_tags +logger = logging.getLogger("paperless.management.retagger") + + class Command(BaseCommand): help = """ @@ -65,7 +68,7 @@ class Command(BaseCommand): classifier = load_classifier() for document in documents: - logging.getLogger(__name__).info( + logger.info( f"Processing document {document.title}") if options['correspondent']: diff --git a/src/documents/matching.py b/src/documents/matching.py index fa9ce6af3..21102bc98 100644 --- a/src/documents/matching.py +++ b/src/documents/matching.py @@ -6,7 +6,7 @@ from fuzzywuzzy import fuzz from documents.models import MatchingModel, Correspondent, DocumentType, Tag -logger = logging.getLogger(__name__) +logger = logging.getLogger("paperless.matching") def log_reason(matching_model, document, reason): diff --git a/src/documents/parsers.py b/src/documents/parsers.py index a2eb14cef..98af4f080 100644 --- a/src/documents/parsers.py +++ b/src/documents/parsers.py @@ -36,7 +36,7 @@ DATE_REGEX = re.compile( ) -logger = logging.getLogger(__name__) +logger = logging.getLogger("paperless.parsing") def is_mime_type_supported(mime_type): @@ -261,6 +261,8 @@ class DocumentParser(LoggingMixin): `paperless_tesseract.parsers` for inspiration. """ + logging_name = "paperless.parsing" + def __init__(self, logging_group, progress_callback=None): super().__init__() self.logging_group = logging_group @@ -316,5 +318,5 @@ class DocumentParser(LoggingMixin): return self.date def cleanup(self): - self.log("debug", "Deleting directory {}".format(self.tempdir)) + self.log("debug", f"Deleting directory {self.tempdir}") shutil.rmtree(self.tempdir) diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py index 76c1e76ed..848bcb900 100755 --- a/src/documents/signals/handlers.py +++ b/src/documents/signals/handlers.py @@ -19,8 +19,7 @@ from ..file_handling import delete_empty_directories, \ from ..models import Document, Tag -def logger(message, group): - logging.getLogger(__name__).debug(message, extra={"group": group}) +logger = logging.getLogger("paperless.handlers") def add_inbox_tags(sender, document=None, logging_group=None, **kwargs): @@ -48,23 +47,23 @@ def set_correspondent(sender, selected = None if potential_count > 1: if use_first: - logger( + logger.info( f"Detected {potential_count} potential correspondents, " f"so we've opted for {selected}", - logging_group + extra={'group': logging_group} ) else: - logger( + logger.info( f"Detected {potential_count} potential correspondents, " f"not assigning any correspondent", - logging_group + extra={'group': logging_group} ) return if selected or replace: - logger( + logger.info( f"Assigning correspondent {selected} to {document}", - logging_group + extra={'group': logging_group} ) document.correspondent = selected @@ -92,23 +91,23 @@ def set_document_type(sender, if potential_count > 1: if use_first: - logger( + logger.info( f"Detected {potential_count} potential document types, " f"so we've opted for {selected}", - logging_group + extra={'group': logging_group} ) else: - logger( + logger.info( f"Detected {potential_count} potential document types, " f"not assigning any document type", - logging_group + extra={'group': logging_group} ) return if selected or replace: - logger( + logger.info( f"Assigning document type {selected} to {document}", - logging_group + extra={'group': logging_group} ) document.document_type = selected @@ -138,9 +137,9 @@ def set_tags(sender, return message = 'Tagging "{}" with "{}"' - logger( + logger.info( message.format(document, ", ".join([t.name for t in relevant_tags])), - logging_group + extra={'group': logging_group} ) document.tags.add(*relevant_tags) @@ -155,10 +154,10 @@ def cleanup_document_deletion(sender, instance, using, **kwargs): if os.path.isfile(f): try: os.unlink(f) - logging.getLogger(__name__).debug( + logger.debug( f"Deleted file {f}.") except OSError as e: - logging.getLogger(__name__).warning( + logger.warning( f"While deleting document {str(instance)}, the file " f"{f} could not be deleted: {e}" ) @@ -177,13 +176,13 @@ def cleanup_document_deletion(sender, instance, using, **kwargs): def validate_move(instance, old_path, new_path): if not os.path.isfile(old_path): # Can't do anything if the old file does not exist anymore. - logging.getLogger(__name__).fatal( + logger.fatal( f"Document {str(instance)}: File {old_path} has gone.") return False if os.path.isfile(new_path): # Can't do anything if the new file already exists. Skip updating file. - logging.getLogger(__name__).warning( + logger.warning( f"Document {str(instance)}: Cannot rename file " f"since target path {new_path} already exists.") return False diff --git a/src/documents/tasks.py b/src/documents/tasks.py index e0d726d3e..8c7d91585 100644 --- a/src/documents/tasks.py +++ b/src/documents/tasks.py @@ -12,6 +12,9 @@ from documents.models import Document, Tag, DocumentType, Correspondent from documents.sanity_checker import SanityFailedError +logger = logging.getLogger("paperless.tasks") + + def index_optimize(): ix = index.open_index() writer = AsyncWriter(ix) @@ -45,18 +48,18 @@ def train_classifier(): try: if classifier.train(): - logging.getLogger(__name__).info( + logger.info( "Saving updated classifier model to {}...".format( settings.MODEL_FILE) ) classifier.save_classifier() else: - logging.getLogger(__name__).debug( + logger.debug( "Training data unchanged." ) except Exception as e: - logging.getLogger(__name__).warning( + logger.warning( "Classifier error: " + str(e) ) diff --git a/src/documents/views.py b/src/documents/views.py index c2b31c996..e413d4b35 100755 --- a/src/documents/views.py +++ b/src/documents/views.py @@ -58,6 +58,9 @@ from .serialisers import ( ) +logger = logging.getLogger("paperless.api") + + class IndexView(TemplateView): template_name = "index.html" @@ -488,7 +491,7 @@ class SearchView(APIView): try: doc = Document.objects.get(id=r['id']) except Document.DoesNotExist: - logging.getLogger(__name__).warning( + logger.warning( f"Search index returned a non-existing document: " f"id: {r['id']}, title: {r['title']}. " f"Search index needs reindex." diff --git a/src/paperless_mail/mail.py b/src/paperless_mail/mail.py index e1916c1dd..4a62ffd95 100644 --- a/src/paperless_mail/mail.py +++ b/src/paperless_mail/mail.py @@ -102,6 +102,8 @@ def get_mailbox(server, port, security): class MailAccountHandler(LoggingMixin): + logging_name = "paperless.mail" + def _correspondent_from_name(self, name): try: return Correspondent.objects.get_or_create(name=name)[0] diff --git a/src/paperless_mail/tasks.py b/src/paperless_mail/tasks.py index 68fb859a4..086edef7b 100644 --- a/src/paperless_mail/tasks.py +++ b/src/paperless_mail/tasks.py @@ -4,6 +4,9 @@ from paperless_mail.mail import MailAccountHandler, MailError from paperless_mail.models import MailAccount +logger = logging.getLogger("paperless.mail.tasks") + + def process_mail_accounts(): total_new_documents = 0 for account in MailAccount.objects.all(): @@ -11,7 +14,7 @@ def process_mail_accounts(): total_new_documents += MailAccountHandler().handle_mail_account( account) except MailError as e: - logging.getLogger(__name__).error( + logger.error( f"Error while processing mail account {account}: {e}", exc_info=True ) @@ -27,4 +30,4 @@ def process_mail_account(name): account = MailAccount.objects.get(name=name) MailAccountHandler().handle_mail_account(account) except MailAccount.DoesNotExist: - logging.getLogger(__name__).error(f"Unknown mail acccount: {name}") + logger.error(f"Unknown mail acccount: {name}") diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py index 5a08c601b..0a976b569 100644 --- a/src/paperless_tesseract/parsers.py +++ b/src/paperless_tesseract/parsers.py @@ -19,6 +19,8 @@ class RasterisedDocumentParser(DocumentParser): image, whether it's a PDF, or other graphical format (JPEG, TIFF, etc.) """ + logging_name = "paperless.parsing.tesseract" + def extract_metadata(self, document_path, mime_type): namespace_pattern = re.compile(r"\{(.*)\}(.*)") diff --git a/src/paperless_text/parsers.py b/src/paperless_text/parsers.py index c1afe07fc..c307bf10b 100644 --- a/src/paperless_text/parsers.py +++ b/src/paperless_text/parsers.py @@ -11,6 +11,8 @@ class TextDocumentParser(DocumentParser): This parser directly parses a text document (.txt, .md, or .csv) """ + logging_name = "paperless.parsing.text" + def get_thumbnail(self, document_path, mime_type): def read_text(): diff --git a/src/paperless_tika/parsers.py b/src/paperless_tika/parsers.py index cde6543be..b888af820 100644 --- a/src/paperless_tika/parsers.py +++ b/src/paperless_tika/parsers.py @@ -14,6 +14,8 @@ class TikaDocumentParser(DocumentParser): This parser sends documents to a local tika server """ + logging_name = "paperless.parsing.tika" + def get_thumbnail(self, document_path, mime_type): if not self.archive_path: self.archive_path = self.convert_to_pdf(document_path) From 96d7114fa7d281a9f0735af3cc7086379dbfb1d1 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Fri, 5 Feb 2021 01:11:03 +0100 Subject: [PATCH 031/225] lazy load fuzzy only when required --- src/documents/matching.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/documents/matching.py b/src/documents/matching.py index 21102bc98..edbbdc1b8 100644 --- a/src/documents/matching.py +++ b/src/documents/matching.py @@ -1,7 +1,6 @@ import logging import re -from fuzzywuzzy import fuzz from documents.models import MatchingModel, Correspondent, DocumentType, Tag @@ -123,6 +122,8 @@ def matches(matching_model, document): return bool(match) elif matching_model.matching_algorithm == MatchingModel.MATCH_FUZZY: + from fuzzywuzzy import fuzz + match = re.sub(r'[^\w\s]', '', matching_model.match) text = re.sub(r'[^\w\s]', '', document_content) if matching_model.is_insensitive: From 00903bd90cf4b052579f1186518a71c3594631b5 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 00:13:49 +0100 Subject: [PATCH 032/225] update statistics when new documents are added --- .../saved-view-widget.component.ts | 2 +- .../statistics-widget.component.ts | 27 ++++++++++++++----- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.ts b/src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.ts index 04e7e56bd..36feeea1e 100644 --- a/src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.ts +++ b/src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.ts @@ -12,7 +12,7 @@ import { DocumentService } from 'src/app/services/rest/document.service'; templateUrl: './saved-view-widget.component.html', styleUrls: ['./saved-view-widget.component.scss'] }) -export class SavedViewWidgetComponent implements OnInit { +export class SavedViewWidgetComponent implements OnInit, OnDestroy { constructor( private documentService: DocumentService, diff --git a/src-ui/src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.ts b/src-ui/src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.ts index c3f45d63c..5207d56ab 100644 --- a/src-ui/src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.ts +++ b/src-ui/src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.ts @@ -1,6 +1,7 @@ import { HttpClient } from '@angular/common/http'; -import { Component, OnInit } from '@angular/core'; -import { Observable } from 'rxjs'; +import { Component, OnDestroy, OnInit } from '@angular/core'; +import { Observable, Subscription } from 'rxjs'; +import { ConsumerStatusService } from 'src/app/services/consumer-status.service'; import { environment } from 'src/environments/environment'; export interface Statistics { @@ -14,20 +15,34 @@ export interface Statistics { templateUrl: './statistics-widget.component.html', styleUrls: ['./statistics-widget.component.scss'] }) -export class StatisticsWidgetComponent implements OnInit { +export class StatisticsWidgetComponent implements OnInit, OnDestroy { - constructor(private http: HttpClient) { } + constructor(private http: HttpClient, + private consumerStatusService: ConsumerStatusService) { } statistics: Statistics = {} - getStatistics(): Observable { + subscription: Subscription + + private getStatistics(): Observable { return this.http.get(`${environment.apiBaseUrl}statistics/`) } - ngOnInit(): void { + reload() { this.getStatistics().subscribe(statistics => { this.statistics = statistics }) } + ngOnInit(): void { + this.reload() + this.subscription = this.consumerStatusService.onDocumentConsumptionFinished().subscribe(status => { + this.reload() + }) + } + + ngOnDestroy(): void { + this.subscription.unsubscribe() + } + } From 2f68f236bb8d0cb637c8ad2736d94a27dcea6026 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Sat, 6 Feb 2021 00:09:49 +0000 Subject: [PATCH 033/225] Apply translations in en_GB translation completed for the source file '/src/locale/en-us/LC_MESSAGES/django.po' on the 'en_GB' language. --- src/locale/en_GB/LC_MESSAGES/django.po | 650 +++++++++++++++++++++++++ 1 file changed, 650 insertions(+) create mode 100644 src/locale/en_GB/LC_MESSAGES/django.po diff --git a/src/locale/en_GB/LC_MESSAGES/django.po b/src/locale/en_GB/LC_MESSAGES/django.po new file mode 100644 index 000000000..5f6901f7f --- /dev/null +++ b/src/locale/en_GB/LC_MESSAGES/django.po @@ -0,0 +1,650 @@ +# SOME DESCRIPTIVE TITLE. +# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER +# This file is distributed under the same license as the PACKAGE package. +# FIRST AUTHOR , YEAR. +# +# Translators: +# Ali Bates , 2021 +# +#, fuzzy +msgid "" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"Report-Msgid-Bugs-To: \n" +"POT-Creation-Date: 2021-01-28 22:02+0100\n" +"PO-Revision-Date: 2020-12-30 19:27+0000\n" +"Last-Translator: Ali Bates , 2021\n" +"Language-Team: English (United Kingdom) (https://www.transifex.com/paperless/teams/115905/en_GB/)\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Language: en_GB\n" +"Plural-Forms: nplurals=2; plural=(n != 1);\n" + +#: documents/apps.py:10 +msgid "Documents" +msgstr "Documents" + +#: documents/models.py:33 +msgid "Any word" +msgstr "Any word" + +#: documents/models.py:34 +msgid "All words" +msgstr "All words" + +#: documents/models.py:35 +msgid "Exact match" +msgstr "Exact match" + +#: documents/models.py:36 +msgid "Regular expression" +msgstr "Regular expression" + +#: documents/models.py:37 +msgid "Fuzzy word" +msgstr "Fuzzy word" + +#: documents/models.py:38 +msgid "Automatic" +msgstr "Automatic" + +#: documents/models.py:42 documents/models.py:352 paperless_mail/models.py:25 +#: paperless_mail/models.py:109 +msgid "name" +msgstr "name" + +#: documents/models.py:46 +msgid "match" +msgstr "match" + +#: documents/models.py:50 +msgid "matching algorithm" +msgstr "matching algorithm" + +#: documents/models.py:56 +msgid "is insensitive" +msgstr "is insensitive" + +#: documents/models.py:75 documents/models.py:135 +msgid "correspondent" +msgstr "correspondent" + +#: documents/models.py:76 +msgid "correspondents" +msgstr "correspondents" + +#: documents/models.py:98 +msgid "color" +msgstr "color" + +#: documents/models.py:102 +msgid "is inbox tag" +msgstr "is inbox tag" + +#: documents/models.py:104 +msgid "" +"Marks this tag as an inbox tag: All newly consumed documents will be tagged " +"with inbox tags." +msgstr "" +"Marks this tag as an inbox tag: All newly consumed documents will be tagged " +"with inbox tags." + +#: documents/models.py:109 +msgid "tag" +msgstr "tag" + +#: documents/models.py:110 documents/models.py:166 +msgid "tags" +msgstr "tags" + +#: documents/models.py:116 documents/models.py:148 +msgid "document type" +msgstr "document type" + +#: documents/models.py:117 +msgid "document types" +msgstr "document types" + +#: documents/models.py:125 +msgid "Unencrypted" +msgstr "Unencrypted" + +#: documents/models.py:126 +msgid "Encrypted with GNU Privacy Guard" +msgstr "Encrypted with GNU Privacy Guard" + +#: documents/models.py:139 +msgid "title" +msgstr "title" + +#: documents/models.py:152 +msgid "content" +msgstr "content" + +#: documents/models.py:154 +msgid "" +"The raw, text-only data of the document. This field is primarily used for " +"searching." +msgstr "" +"The raw, text-only data of the document. This field is primarily used for " +"searching." + +#: documents/models.py:159 +msgid "mime type" +msgstr "mime type" + +#: documents/models.py:170 +msgid "checksum" +msgstr "checksum" + +#: documents/models.py:174 +msgid "The checksum of the original document." +msgstr "The checksum of the original document." + +#: documents/models.py:178 +msgid "archive checksum" +msgstr "archive checksum" + +#: documents/models.py:183 +msgid "The checksum of the archived document." +msgstr "The checksum of the archived document." + +#: documents/models.py:187 documents/models.py:330 +msgid "created" +msgstr "created" + +#: documents/models.py:191 +msgid "modified" +msgstr "modified" + +#: documents/models.py:195 +msgid "storage type" +msgstr "storage type" + +#: documents/models.py:203 +msgid "added" +msgstr "added" + +#: documents/models.py:207 +msgid "filename" +msgstr "filename" + +#: documents/models.py:212 +msgid "Current filename in storage" +msgstr "Current filename in storage" + +#: documents/models.py:216 +msgid "archive serial number" +msgstr "archive serial number" + +#: documents/models.py:221 +msgid "The position of this document in your physical document archive." +msgstr "The position of this document in your physical document archive." + +#: documents/models.py:227 +msgid "document" +msgstr "document" + +#: documents/models.py:228 +msgid "documents" +msgstr "documents" + +#: documents/models.py:313 +msgid "debug" +msgstr "debug" + +#: documents/models.py:314 +msgid "information" +msgstr "information" + +#: documents/models.py:315 +msgid "warning" +msgstr "warning" + +#: documents/models.py:316 +msgid "error" +msgstr "error" + +#: documents/models.py:317 +msgid "critical" +msgstr "critical" + +#: documents/models.py:321 +msgid "group" +msgstr "group" + +#: documents/models.py:324 +msgid "message" +msgstr "message" + +#: documents/models.py:327 +msgid "level" +msgstr "level" + +#: documents/models.py:334 +msgid "log" +msgstr "log" + +#: documents/models.py:335 +msgid "logs" +msgstr "logs" + +#: documents/models.py:346 documents/models.py:396 +msgid "saved view" +msgstr "saved view" + +#: documents/models.py:347 +msgid "saved views" +msgstr "saved views" + +#: documents/models.py:350 +msgid "user" +msgstr "user" + +#: documents/models.py:356 +msgid "show on dashboard" +msgstr "show on dashboard" + +#: documents/models.py:359 +msgid "show in sidebar" +msgstr "show in sidebar" + +#: documents/models.py:363 +msgid "sort field" +msgstr "sort field" + +#: documents/models.py:366 +msgid "sort reverse" +msgstr "sort reverse" + +#: documents/models.py:372 +msgid "title contains" +msgstr "title contains" + +#: documents/models.py:373 +msgid "content contains" +msgstr "content contains" + +#: documents/models.py:374 +msgid "ASN is" +msgstr "ASN is" + +#: documents/models.py:375 +msgid "correspondent is" +msgstr "correspondent is" + +#: documents/models.py:376 +msgid "document type is" +msgstr "document type is" + +#: documents/models.py:377 +msgid "is in inbox" +msgstr "is in inbox" + +#: documents/models.py:378 +msgid "has tag" +msgstr "has tag" + +#: documents/models.py:379 +msgid "has any tag" +msgstr "has any tag" + +#: documents/models.py:380 +msgid "created before" +msgstr "created before" + +#: documents/models.py:381 +msgid "created after" +msgstr "created after" + +#: documents/models.py:382 +msgid "created year is" +msgstr "created year is" + +#: documents/models.py:383 +msgid "created month is" +msgstr "created month is" + +#: documents/models.py:384 +msgid "created day is" +msgstr "created day is" + +#: documents/models.py:385 +msgid "added before" +msgstr "added before" + +#: documents/models.py:386 +msgid "added after" +msgstr "added after" + +#: documents/models.py:387 +msgid "modified before" +msgstr "modified before" + +#: documents/models.py:388 +msgid "modified after" +msgstr "modified after" + +#: documents/models.py:389 +msgid "does not have tag" +msgstr "does not have tag" + +#: documents/models.py:400 +msgid "rule type" +msgstr "rule type" + +#: documents/models.py:404 +msgid "value" +msgstr "value" + +#: documents/models.py:410 +msgid "filter rule" +msgstr "filter rule" + +#: documents/models.py:411 +msgid "filter rules" +msgstr "filter rules" + +#: documents/serialisers.py:383 +#, python-format +msgid "File type %(type)s not supported" +msgstr "File type %(type)s not supported" + +#: documents/templates/index.html:20 +msgid "Paperless-ng is loading..." +msgstr "Paperless-ng is loading..." + +#: documents/templates/registration/logged_out.html:13 +msgid "Paperless-ng signed out" +msgstr "Paperless-ng signed out" + +#: documents/templates/registration/logged_out.html:41 +msgid "You have been successfully logged out. Bye!" +msgstr "You have been successfully logged out. Bye!" + +#: documents/templates/registration/logged_out.html:42 +msgid "Sign in again" +msgstr "Sign in again" + +#: documents/templates/registration/login.html:13 +msgid "Paperless-ng sign in" +msgstr "Paperless-ng sign in" + +#: documents/templates/registration/login.html:42 +msgid "Please sign in." +msgstr "Please sign in." + +#: documents/templates/registration/login.html:45 +msgid "Your username and password didn't match. Please try again." +msgstr "Your username and password didn't match. Please try again." + +#: documents/templates/registration/login.html:48 +msgid "Username" +msgstr "Username" + +#: documents/templates/registration/login.html:49 +msgid "Password" +msgstr "Password" + +#: documents/templates/registration/login.html:54 +msgid "Sign in" +msgstr "Sign in" + +#: paperless/settings.py:286 +msgid "English" +msgstr "English" + +#: paperless/settings.py:287 +msgid "German" +msgstr "German" + +#: paperless/settings.py:288 +msgid "Dutch" +msgstr "Dutch" + +#: paperless/settings.py:289 +msgid "French" +msgstr "French" + +#: paperless/urls.py:114 +msgid "Paperless-ng administration" +msgstr "Paperless-ng administration" + +#: paperless_mail/admin.py:25 +msgid "Filter" +msgstr "Filter" + +#: paperless_mail/admin.py:27 +msgid "" +"Paperless will only process mails that match ALL of the filters given below." +msgstr "" +"Paperless will only process mails that match ALL of the filters given below." + +#: paperless_mail/admin.py:37 +msgid "Actions" +msgstr "Actions" + +#: paperless_mail/admin.py:39 +msgid "" +"The action applied to the mail. This action is only performed when documents" +" were consumed from the mail. Mails without attachments will remain entirely" +" untouched." +msgstr "" +"The action applied to the mail. This action is only performed when documents" +" were consumed from the mail. Mails without attachments will remain entirely" +" untouched." + +#: paperless_mail/admin.py:46 +msgid "Metadata" +msgstr "Metadata" + +#: paperless_mail/admin.py:48 +msgid "" +"Assign metadata to documents consumed from this rule automatically. If you " +"do not assign tags, types or correspondents here, paperless will still " +"process all matching rules that you have defined." +msgstr "" +"Assign metadata to documents consumed from this rule automatically. If you " +"do not assign tags, types or correspondents here, paperless will still " +"process all matching rules that you have defined." + +#: paperless_mail/apps.py:9 +msgid "Paperless mail" +msgstr "Paperless mail" + +#: paperless_mail/models.py:11 +msgid "mail account" +msgstr "mail account" + +#: paperless_mail/models.py:12 +msgid "mail accounts" +msgstr "mail accounts" + +#: paperless_mail/models.py:19 +msgid "No encryption" +msgstr "No encryption" + +#: paperless_mail/models.py:20 +msgid "Use SSL" +msgstr "Use SSL" + +#: paperless_mail/models.py:21 +msgid "Use STARTTLS" +msgstr "Use STARTTLS" + +#: paperless_mail/models.py:29 +msgid "IMAP server" +msgstr "IMAP server" + +#: paperless_mail/models.py:33 +msgid "IMAP port" +msgstr "IMAP port" + +#: paperless_mail/models.py:36 +msgid "" +"This is usually 143 for unencrypted and STARTTLS connections, and 993 for " +"SSL connections." +msgstr "" +"This is usually 143 for unencrypted and STARTTLS connections, and 993 for " +"SSL connections." + +#: paperless_mail/models.py:40 +msgid "IMAP security" +msgstr "IMAP security" + +#: paperless_mail/models.py:46 +msgid "username" +msgstr "username" + +#: paperless_mail/models.py:50 +msgid "password" +msgstr "password" + +#: paperless_mail/models.py:60 +msgid "mail rule" +msgstr "mail rule" + +#: paperless_mail/models.py:61 +msgid "mail rules" +msgstr "mail rules" + +#: paperless_mail/models.py:67 +msgid "Only process attachments." +msgstr "Only process attachments." + +#: paperless_mail/models.py:68 +msgid "Process all files, including 'inline' attachments." +msgstr "Process all files, including 'inline' attachments." + +#: paperless_mail/models.py:78 +msgid "Mark as read, don't process read mails" +msgstr "Mark as read, don't process read mails" + +#: paperless_mail/models.py:79 +msgid "Flag the mail, don't process flagged mails" +msgstr "Flag the mail, don't process flagged mails" + +#: paperless_mail/models.py:80 +msgid "Move to specified folder" +msgstr "Move to specified folder" + +#: paperless_mail/models.py:81 +msgid "Delete" +msgstr "Delete" + +#: paperless_mail/models.py:88 +msgid "Use subject as title" +msgstr "Use subject as title" + +#: paperless_mail/models.py:89 +msgid "Use attachment filename as title" +msgstr "Use attachment filename as title" + +#: paperless_mail/models.py:99 +msgid "Do not assign a correspondent" +msgstr "Do not assign a correspondent" + +#: paperless_mail/models.py:101 +msgid "Use mail address" +msgstr "Use mail address" + +#: paperless_mail/models.py:103 +msgid "Use name (or mail address if not available)" +msgstr "Use name (or mail address if not available)" + +#: paperless_mail/models.py:105 +msgid "Use correspondent selected below" +msgstr "Use correspondent selected below" + +#: paperless_mail/models.py:113 +msgid "order" +msgstr "order" + +#: paperless_mail/models.py:120 +msgid "account" +msgstr "account" + +#: paperless_mail/models.py:124 +msgid "folder" +msgstr "folder" + +#: paperless_mail/models.py:128 +msgid "filter from" +msgstr "filter from" + +#: paperless_mail/models.py:131 +msgid "filter subject" +msgstr "filter subject" + +#: paperless_mail/models.py:134 +msgid "filter body" +msgstr "filter body" + +#: paperless_mail/models.py:138 +msgid "filter attachment filename" +msgstr "filter attachment filename" + +#: paperless_mail/models.py:140 +msgid "" +"Only consume documents which entirely match this filename if specified. " +"Wildcards such as *.pdf or *invoice* are allowed. Case insensitive." +msgstr "" +"Only consume documents which entirely match this filename if specified. " +"Wildcards such as *.pdf or *invoice* are allowed. Case insensitive." + +#: paperless_mail/models.py:146 +msgid "maximum age" +msgstr "maximum age" + +#: paperless_mail/models.py:148 +msgid "Specified in days." +msgstr "Specified in days." + +#: paperless_mail/models.py:151 +msgid "attachment type" +msgstr "attachment type" + +#: paperless_mail/models.py:154 +msgid "" +"Inline attachments include embedded images, so it's best to combine this " +"option with a filename filter." +msgstr "" +"Inline attachments include embedded images, so it's best to combine this " +"option with a filename filter." + +#: paperless_mail/models.py:159 +msgid "action" +msgstr "action" + +#: paperless_mail/models.py:165 +msgid "action parameter" +msgstr "action parameter" + +#: paperless_mail/models.py:167 +msgid "" +"Additional parameter for the action selected above, i.e., the target folder " +"of the move to folder action." +msgstr "" +"Additional parameter for the action selected above, i.e., the target folder " +"of the move to folder action." + +#: paperless_mail/models.py:173 +msgid "assign title from" +msgstr "assign title from" + +#: paperless_mail/models.py:183 +msgid "assign this tag" +msgstr "assign this tag" + +#: paperless_mail/models.py:191 +msgid "assign this document type" +msgstr "assign this document type" + +#: paperless_mail/models.py:195 +msgid "assign correspondent from" +msgstr "assign correspondent from" + +#: paperless_mail/models.py:205 +msgid "assign this correspondent" +msgstr "assign this correspondent" From 968614a289ae89468cf496502aaef05ae0e78bb4 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Sat, 6 Feb 2021 00:12:00 +0000 Subject: [PATCH 034/225] Translate /src-ui/messages.xlf in en_GB translation completed for the source file '/src-ui/messages.xlf' on the 'en_GB' language. --- src-ui/src/locale/messages.en_GB.xlf | 2313 ++++++++++++++++++++++++++ 1 file changed, 2313 insertions(+) create mode 100644 src-ui/src/locale/messages.en_GB.xlf diff --git a/src-ui/src/locale/messages.en_GB.xlf b/src-ui/src/locale/messages.en_GB.xlf new file mode 100644 index 000000000..4df278c69 --- /dev/null +++ b/src-ui/src/locale/messages.en_GB.xlf @@ -0,0 +1,2313 @@ + + + + + Document added + Document added + + src/app/app.component.ts + 51 + + + + Document was added to paperless. + Document was added to paperless. + + src/app/app.component.ts + 51 + + + + Open document + Open document + + src/app/app.component.ts + 51 + + + + Could not add : + Could not add : + + src/app/app.component.ts + 59 + + + + New document detected + New document detected + + src/app/app.component.ts + 65 + + + + Document is being processed by paperless. + Document is being processed by paperless. + + src/app/app.component.ts + 65 + + + + Documents + Documents + + src/app/components/document-list/document-list.component.ts + 49 + + + + View "" saved successfully. + View &quot;&quot; saved successfully. + + src/app/components/document-list/document-list.component.ts + 109 + + + + View "" created successfully. + View &quot;&quot; created successfully. + + src/app/components/document-list/document-list.component.ts + 130 + + + + Select + Select + + src/app/components/document-list/document-list.component.html + 7 + + + + Select none + Select none + + src/app/components/document-list/document-list.component.html + 10 + + + + Select page + Select page + + src/app/components/document-list/document-list.component.html + 11 + + + + Select all + Select all + + src/app/components/document-list/document-list.component.html + 12 + + + + Sort + Sort + + src/app/components/document-list/document-list.component.html + 39 + + + + Views + Views + + src/app/components/document-list/document-list.component.html + 64 + + + + Save as... + Save as... + + src/app/components/document-list/document-list.component.html + 72 + + + + Save "" + Save &quot;&quot; + + src/app/components/document-list/document-list.component.html + 71 + + + + {VAR_PLURAL, plural, =1 {Selected of one document} other {Selected of documents}} + {VAR_PLURAL, plural, =1 {Selected of one document} other {Selected of documents}} + + src/app/components/document-list/document-list.component.html + 85 + + + + {VAR_PLURAL, plural, =1 {One document} other { documents}} + {VAR_PLURAL, plural, =1 {One document} other { documents}} + + src/app/components/document-list/document-list.component.html + 86 + + + + (filtered) + (filtered) + + src/app/components/document-list/document-list.component.html + 86 + + + + ASN + ASN + + src/app/components/document-list/document-list.component.html + 105 + + + + Correspondent + Correspondent + + src/app/components/document-list/document-list.component.html + 111 + + + + Title + Title + + src/app/components/document-list/document-list.component.html + 117 + + + + Document type + Document type + + src/app/components/document-list/document-list.component.html + 123 + + + + Created + Created + + src/app/components/document-list/document-list.component.html + 129 + + + + Added + Added + + src/app/components/document-list/document-list.component.html + 135 + + + + Confirm delete + Confirm delete + + src/app/components/document-detail/document-detail.component.ts + 203 + + + + Do you really want to delete document ""? + Do you really want to delete document &quot;&quot;? + + src/app/components/document-detail/document-detail.component.ts + 204 + + + + The files for this document will be deleted permanently. This operation cannot be undone. + The files for this document will be deleted permanently. This operation cannot be undone. + + src/app/components/document-detail/document-detail.component.ts + 205 + + + + Delete document + Delete document + + src/app/components/document-detail/document-detail.component.ts + 207 + + + + Error deleting document: + Error deleting document: + + src/app/components/document-detail/document-detail.component.ts + 214 + + + + Delete + Delete + + src/app/components/document-detail/document-detail.component.html + 15 + + + + Download + Download + + src/app/components/document-detail/document-detail.component.html + 23 + + + + More like this + More like this + + src/app/components/document-detail/document-detail.component.html + 38 + + + + Close + Close + + src/app/components/document-detail/document-detail.component.html + 44 + + + + Details + Details + + src/app/components/document-detail/document-detail.component.html + 56 + + + + Content + Content + + src/app/components/document-detail/document-detail.component.html + 72 + + + + Metadata + Metadata + + src/app/components/document-detail/document-detail.component.html + 81 + + + + Discard + Discard + + src/app/components/document-detail/document-detail.component.html + 130 + + + + Save + Save + + src/app/components/document-detail/document-detail.component.html + 132 + + + + Page + Page + + src/app/components/document-detail/document-detail.component.html + 4 + + + + of + of + + src/app/components/document-detail/document-detail.component.html + 8 + + + + Download original + Download original + + src/app/components/document-detail/document-detail.component.html + 29 + + + + Archive serial number + Archive serial number + + src/app/components/document-detail/document-detail.component.html + 60 + + + + Date created + Date created + + src/app/components/document-detail/document-detail.component.html + 61 + + + + Date modified + Date modified + + src/app/components/document-detail/document-detail.component.html + 87 + + + + Date added + Date added + + src/app/components/document-detail/document-detail.component.html + 91 + + + + Media filename + Media filename + + src/app/components/document-detail/document-detail.component.html + 95 + + + + Original MD5 checksum + Original MD5 checksum + + src/app/components/document-detail/document-detail.component.html + 99 + + + + Original file size + Original file size + + src/app/components/document-detail/document-detail.component.html + 103 + + + + Original mime type + Original mime type + + src/app/components/document-detail/document-detail.component.html + 107 + + + + Archive MD5 checksum + Archive MD5 checksum + + src/app/components/document-detail/document-detail.component.html + 111 + + + + Archive file size + Archive file size + + src/app/components/document-detail/document-detail.component.html + 115 + + + + Original document metadata + Original document metadata + + src/app/components/document-detail/document-detail.component.html + 121 + + + + Archived document metadata + Archived document metadata + + src/app/components/document-detail/document-detail.component.html + 122 + + + + Save & next + Save & next + + src/app/components/document-detail/document-detail.component.html + 131 + + + + Hello , welcome to Paperless-ng! + Hello , welcome to Paperless-ng! + + src/app/components/dashboard/dashboard.component.ts + 33 + + + + Welcome to Paperless-ng! + Welcome to Paperless-ng! + + src/app/components/dashboard/dashboard.component.ts + 35 + + + + Dashboard + Dashboard + + src/app/components/dashboard/dashboard.component.html + 1 + + + + Do you really want to delete the tag ""? + Do you really want to delete the tag &quot;&quot;? + + src/app/components/manage/tag-list/tag-list.component.ts + 30 + + + + Tags + Tags + + src/app/components/manage/tag-list/tag-list.component.html + 1 + + + + Create + Create + + src/app/components/manage/tag-list/tag-list.component.html + 2 + + + + Filter by: + Filter by: + + src/app/components/manage/tag-list/tag-list.component.html + 8 + + + + Name + Name + + src/app/components/manage/tag-list/tag-list.component.html + 9 + + + + Color + Color + + src/app/components/manage/tag-list/tag-list.component.html + 20 + + + + Matching + Matching + + src/app/components/manage/tag-list/tag-list.component.html + 21 + + + + Document count + Document count + + src/app/components/manage/tag-list/tag-list.component.html + 22 + + + + Actions + Actions + + src/app/components/manage/tag-list/tag-list.component.html + 23 + + + + Documents + Documents + + src/app/components/manage/tag-list/tag-list.component.html + 38 + + + + Edit + Edit + + src/app/components/manage/tag-list/tag-list.component.html + 43 + + + + Do you really want to delete the document type ""? + Do you really want to delete the document type &quot;&quot;? + + src/app/components/manage/document-type-list/document-type-list.component.ts + 26 + + + + Document types + Document types + + src/app/components/manage/document-type-list/document-type-list.component.html + 1 + + + + Logs + Logs + + src/app/components/manage/logs/logs.component.html + 1 + + + + Filter + Filter + + src/app/components/manage/logs/logs.component.html + 7 + + + + Saved view "" deleted. + Saved view &quot;&quot; deleted. + + src/app/components/manage/settings/settings.component.ts + 67 + + + + Settings saved successfully. + Settings saved successfully. + + src/app/components/manage/settings/settings.component.ts + 87 + + + + Use system language + Use system language + + src/app/components/manage/settings/settings.component.ts + 91 + + + + Use date format of display language + Use date format of display language + + src/app/components/manage/settings/settings.component.ts + 95 + + + + Error while storing settings on server: + Error while storing settings on server: + + src/app/components/manage/settings/settings.component.ts + 111 + + + + Settings + Settings + + src/app/components/manage/settings/settings.component.html + 1 + + + + General settings + General settings + + src/app/components/manage/settings/settings.component.html + 10 + + + + Notifications + Notifications + + src/app/components/manage/settings/settings.component.html + 115 + + + + Saved views + Saved views + + src/app/components/manage/settings/settings.component.html + 133 + + + + Appearance + Appearance + + src/app/components/manage/settings/settings.component.html + 13 + + + + Display language + Display language + + src/app/components/manage/settings/settings.component.html + 17 + + + + You need to reload the page after applying a new language. + You need to reload the page after applying a new language. + + src/app/components/manage/settings/settings.component.html + 25 + + + + Date display + Date display + + src/app/components/manage/settings/settings.component.html + 32 + + + + Date format + Date format + + src/app/components/manage/settings/settings.component.html + 45 + + + + Short: + Short: + + src/app/components/manage/settings/settings.component.html + 51 + + + + Medium: + Medium: + + src/app/components/manage/settings/settings.component.html + 55 + + + + Long: + Long: + + src/app/components/manage/settings/settings.component.html + 59 + + + + Items per page + Items per page + + src/app/components/manage/settings/settings.component.html + 67 + + + + Document editor + Document editor + + src/app/components/manage/settings/settings.component.html + 83 + + + + Use PDF viewer provided by the browser + Use PDF viewer provided by the browser + + src/app/components/manage/settings/settings.component.html + 87 + + + + This is usually faster for displaying large PDF documents, but it might not work on some browsers. + This is usually faster for displaying large PDF documents, but it might not work on some browsers. + + src/app/components/manage/settings/settings.component.html + 87 + + + + Dark mode + Dark mode + + src/app/components/manage/settings/settings.component.html + 94 + + + + Use system settings + Use system settings + + src/app/components/manage/settings/settings.component.html + 97 + + + + Enable dark mode + Enable dark mode + + src/app/components/manage/settings/settings.component.html + 98 + + + + Bulk editing + Bulk editing + + src/app/components/manage/settings/settings.component.html + 102 + + + + Show confirmation dialogs + Show confirmation dialogs + + src/app/components/manage/settings/settings.component.html + 106 + + + + Deleting documents will always ask for confirmation. + Deleting documents will always ask for confirmation. + + src/app/components/manage/settings/settings.component.html + 106 + + + + Apply on close + Apply on close + + src/app/components/manage/settings/settings.component.html + 107 + + + + Document processing + Document processing + + src/app/components/manage/settings/settings.component.html + 118 + + + + Show notifications when new documents are detected + Show notifications when new documents are detected + + src/app/components/manage/settings/settings.component.html + 122 + + + + Show notifications when document processing completes successfully + Show notifications when document processing completes successfully + + src/app/components/manage/settings/settings.component.html + 123 + + + + Show notifications when document processing fails + Show notifications when document processing fails + + src/app/components/manage/settings/settings.component.html + 124 + + + + Suppress notifications on dashboard + Suppress notifications on dashboard + + src/app/components/manage/settings/settings.component.html + 125 + + + + This will suppress all messages about document processing status on the dashboard. + This will suppress all messages about document processing status on the dashboard. + + src/app/components/manage/settings/settings.component.html + 125 + + + + Appears on + Appears on + + src/app/components/manage/settings/settings.component.html + 145 + + + + Show on dashboard + Show on dashboard + + src/app/components/manage/settings/settings.component.html + 148 + + + + Show in sidebar + Show in sidebar + + src/app/components/manage/settings/settings.component.html + 152 + + + + No saved views defined. + No saved views defined. + + src/app/components/manage/settings/settings.component.html + 162 + + + + 404 Not Found + 404 Not Found + + src/app/components/not-found/not-found.component.html + 7 + + + + Do you really want to delete the correspondent ""? + Do you really want to delete the correspondent &quot;&quot;? + + src/app/components/manage/correspondent-list/correspondent-list.component.ts + 26 + + + + Correspondents + Correspondents + + src/app/components/manage/correspondent-list/correspondent-list.component.html + 1 + + + + Last correspondence + Last correspondence + + src/app/components/manage/correspondent-list/correspondent-list.component.html + 22 + + + + Confirmation + Confirmation + + src/app/components/common/confirm-dialog/confirm-dialog.component.ts + 17 + + + + Confirm + Confirm + + src/app/components/common/confirm-dialog/confirm-dialog.component.ts + 29 + + + + Cancel + Cancel + + src/app/components/common/confirm-dialog/confirm-dialog.component.html + 12 + + + + Create new correspondent + Create new correspondent + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.ts + 21 + + + + Edit correspondent + Edit correspondent + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.ts + 25 + + + + Matching algorithm + Matching algorithm + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html + 10 + + + + Matching pattern + Matching pattern + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html + 11 + + + + Case insensitive + Case insensitive + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html + 12 + + + + Create new tag + Create new tag + + src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.ts + 21 + + + + Edit tag + Edit tag + + src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.ts + 25 + + + + Inbox tag + Inbox tag + + src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.html + 21 + + + + Inbox tags are automatically assigned to all consumed documents. + Inbox tags are automatically assigned to all consumed documents. + + src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.html + 21 + + + + Create new document type + Create new document type + + src/app/components/manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component.ts + 21 + + + + Edit document type + Edit document type + + src/app/components/manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component.ts + 25 + + + + Search results + Search results + + src/app/components/search/search.component.html + 1 + + + + Invalid search query: + Invalid search query: + + src/app/components/search/search.component.html + 4 + + + + Showing documents similar to + Showing documents similar to + + src/app/components/search/search.component.html + 7 + + + + Search query: + Search query: + + src/app/components/search/search.component.html + 11 + + + + Did you mean ""? + Did you mean &quot;&quot;? + + src/app/components/search/search.component.html + 13 + + + + {VAR_PLURAL, plural, =0 {No results} =1 {One result} other { results}} + {VAR_PLURAL, plural, =0 {No results} =1 {One result} other { results}} + + src/app/components/search/search.component.html + 18 + + + + Paperless-ng + Paperless-ng + + src/app/components/app-frame/app-frame.component.html + 11 + + app title + + + Search documents + Search documents + + src/app/components/app-frame/app-frame.component.html + 15 + + + + Logout + Logout + + src/app/components/app-frame/app-frame.component.html + 45 + + + + Manage + Manage + + src/app/components/app-frame/app-frame.component.html + 107 + + + + Admin + Admin + + src/app/components/app-frame/app-frame.component.html + 149 + + + + Info + Info + + src/app/components/app-frame/app-frame.component.html + 155 + + + + Documentation + Documentation + + src/app/components/app-frame/app-frame.component.html + 162 + + + + GitHub + GitHub + + src/app/components/app-frame/app-frame.component.html + 170 + + + + Suggest an idea + Suggest an idea + + src/app/components/app-frame/app-frame.component.html + 176 + + + + Logged in as + Logged in as + + src/app/components/app-frame/app-frame.component.html + 34 + + + + Open documents + Open documents + + src/app/components/app-frame/app-frame.component.html + 87 + + + + Close all + Close all + + src/app/components/app-frame/app-frame.component.html + 101 + + + + Correspondent: + Correspondent: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 29 + + + + Without correspondent + Without correspondent + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 31 + + + + Type: + Type: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 36 + + + + Without document type + Without document type + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 38 + + + + Tag: + Tag: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 42 + + + + Without any tag + Without any tag + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 46 + + + + Filter tags + Filter tags + + src/app/components/document-list/filter-editor/filter-editor.component.html + 12 + + + + Filter correspondents + Filter correspondents + + src/app/components/document-list/filter-editor/filter-editor.component.html + 20 + + + + Filter document types + Filter document types + + src/app/components/document-list/filter-editor/filter-editor.component.html + 27 + + + + Reset filters + Reset filters + + src/app/components/document-list/filter-editor/filter-editor.component.html + 50 + + + + Not assigned + Not assigned + + src/app/components/common/filterable-dropdown/filterable-dropdown.component.ts + 166 + + Filter drop down element to filter for documents with no correspondent/type/tag assigned + + + Apply + Apply + + src/app/components/common/filterable-dropdown/filterable-dropdown.component.html + 26 + + + + Last 7 days + Last 7 days + + src/app/components/common/date-dropdown/date-dropdown.component.ts + 24 + + + + Last month + Last month + + src/app/components/common/date-dropdown/date-dropdown.component.ts + 25 + + + + Last 3 months + Last 3 months + + src/app/components/common/date-dropdown/date-dropdown.component.ts + 26 + + + + Last year + Last year + + src/app/components/common/date-dropdown/date-dropdown.component.ts + 27 + + + + After + After + + src/app/components/common/date-dropdown/date-dropdown.component.html + 13 + + + + Before + Before + + src/app/components/common/date-dropdown/date-dropdown.component.html + 29 + + + + Clear + Clear + + src/app/components/common/date-dropdown/date-dropdown.component.html + 18 + + + + View + View + + src/app/components/document-list/document-card-large/document-card-large.component.html + 50 + + + + Created: + Created: + + src/app/components/document-list/document-card-large/document-card-large.component.html + 67 + + + + Filter by correspondent + Filter by correspondent + + src/app/components/document-list/document-card-large/document-card-large.component.html + 20 + + + + Filter by tag + Filter by tag + + src/app/components/document-list/document-card-large/document-card-large.component.html + 24 + + + + Score: + Score: + + src/app/components/document-list/document-card-large/document-card-large.component.html + 62 + + + + View in browser + View in browser + + src/app/components/document-list/document-card-small/document-card-small.component.html + 40 + + + + Error executing bulk operation: + Error executing bulk operation: + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 73 + + + + "" + &quot;&quot; + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 112 + + + + "" and "" + &quot;&quot; and &quot;&quot; + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 114 + + This is for messages like 'modify "tag1" and "tag2"' + + + , + , + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 116 + + this is used to separate enumerations and should probably be a comma and a whitespace in most languages + + + and "" + and &quot;&quot; + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 117 + + this is for messages like 'modify "tag1", "tag2" and "tag3"' + + + Confirm tags assignment + Confirm tags assignment + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 126 + + + + This operation will add the tag "" to selected document(s). + This operation will add the tag &quot;&quot; to selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 129 + + + + This operation will add the tags to selected document(s). + This operation will add the tags to selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 131 + + + + This operation will remove the tag "" from selected document(s). + This operation will remove the tag &quot;&quot; from selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 134 + + + + This operation will remove the tags from selected document(s). + This operation will remove the tags from selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 136 + + + + This operation will add the tags and remove the tags on selected document(s). + This operation will add the tags and remove the tags on selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 138 + + + + Confirm correspondent assignment + Confirm correspondent assignment + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 158 + + + + This operation will assign the correspondent "" to selected document(s). + This operation will assign the correspondent &quot;&quot; to selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 160 + + + + This operation will remove the correspondent from selected document(s). + This operation will remove the correspondent from selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 162 + + + + Confirm document type assignment + Confirm document type assignment + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 181 + + + + This operation will assign the document type "" to selected document(s). + This operation will assign the document type &quot;&quot; to selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 183 + + + + This operation will remove the document type from selected document(s). + This operation will remove the document type from selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 185 + + + + Delete confirm + Delete confirm + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 200 + + + + This operation will permanently delete selected document(s). + This operation will permanently delete selected document(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 201 + + + + This operation cannot be undone. + This operation cannot be undone. + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 202 + + + + Delete document(s) + Delete document(s) + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 204 + + + + Select: + Select: + + src/app/components/document-list/bulk-editor/bulk-editor.component.html + 10 + + + + All + All + + src/app/components/document-list/bulk-editor/bulk-editor.component.html + 20 + + + + Edit: + Edit: + + src/app/components/document-list/bulk-editor/bulk-editor.component.html + 27 + + + + Suggestions: + Suggestions: + + src/app/components/common/input/select/select.component.html + 26 + + + + Save current view + Save current view + + src/app/components/document-list/save-view-config-dialog/save-view-config-dialog.component.html + 3 + + + + Show all + Show all + + src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.html + 3 + + + + Statistics + Statistics + + src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html + 1 + + + + Documents in inbox: + Documents in inbox: + + src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html + 3 + + + + Total documents: + Total documents: + + src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html + 4 + + + + Processing: + Processing: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 32 + + + + Failed: + Failed: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 35 + + + + Added: + Added: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 38 + + + + Connecting... + Connecting... + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 118 + + + + Uploading... + Uploading... + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 123 + + + + Upload complete, waiting... + Upload complete, waiting... + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 126 + + + + HTTP error: + HTTP error: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 136 + + + + Upload new documents + Upload new documents + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 1 + + + + Drop documents here or + Drop documents here or + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 13 + + + + Browse files + Browse files + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 13 + + + + Dismiss completed + Dismiss completed + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 4 + + This button dismisses all status messages about processed documents on the dashboard (failed and successful) + + + {VAR_PLURAL, plural, =1 {One more document} other { more documents}} + {VAR_PLURAL, plural, =1 {One more document} other { more documents}} + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 25 + + This is shown as a summary line when there are more than 5 document in the processing pipeline. + + + Open document + Open document + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 45 + + + + First steps + First steps + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 1 + + + + Paperless is running! :) + Paperless is running! :) + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 5 + + + + You can start uploading documents by dropping them in the file upload box to the right or by dropping them in the configured consumption folder and they'll start showing up in the documents list. After you've added some metadata to your documents, use the filtering mechanisms of paperless to create custom views (such as 'Recently added', 'Tagged TODO') and they will appear on the dashboard instead of this message. + You can start uploading documents by dropping them in the file upload box to the right or by dropping them in the configured consumption folder and they'll start showing up in the documents list. After you've added some metadata to your documents, use the filtering mechanisms of paperless to create custom views (such as 'Recently added', 'Tagged TODO') and they will appear on the dashboard instead of this message. + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 6,7 + + + + Paperless offers some more features that try to make your life easier: + Paperless offers some more features that try to make your life easier: + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 8 + + + + Once you've got a couple documents in paperless and added metadata to them, paperless can assign that metadata to new documents automatically. + Once you've got a couple documents in paperless and added metadata to them, paperless can assign that metadata to new documents automatically. + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 10 + + + + You can configure paperless to read your mails and add documents from attached files. + You can configure paperless to read your mails and add documents from attached files. + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 11 + + + + Consult the documentation on how to use these features. The section on basic usage also has some information on how to use paperless in general. + Consult the documentation on how to use these features. The section on basic usage also has some information on how to use paperless in general. + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 13 + + + + Metadata + Metadata + + src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts + 18 + + + + Select + Select + + src/app/components/common/select-dialog/select-dialog.component.ts + 18 + + + + Please select an object + Please select an object + + src/app/components/common/select-dialog/select-dialog.component.ts + 21 + + + + Yes + Yes + + src/app/pipes/yes-no.pipe.ts + 9 + + + + No + No + + src/app/pipes/yes-no.pipe.ts + 9 + + + + (no title) + (no title) + + src/app/pipes/document-title.pipe.ts + 12 + + + + English (US) + English (US) + + src/app/services/settings.service.ts + 82 + + + + German + German + + src/app/services/settings.service.ts + 83 + + + + Dutch + Dutch + + src/app/services/settings.service.ts + 84 + + + + French + French + + src/app/services/settings.service.ts + 85 + + + + Document already exists. + Document already exists. + + src/app/services/consumer-status.service.ts + 15 + + + + File not found. + File not found. + + src/app/services/consumer-status.service.ts + 16 + + + + Pre-consume script does not exist. + Pre-consume script does not exist. + + src/app/services/consumer-status.service.ts + 17 + + Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation + + + Error while executing pre-consume script. + Error while executing pre-consume script. + + src/app/services/consumer-status.service.ts + 18 + + Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation + + + Post-consume script does not exist. + Post-consume script does not exist. + + src/app/services/consumer-status.service.ts + 19 + + Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation + + + Error while executing post-consume script. + Error while executing post-consume script. + + src/app/services/consumer-status.service.ts + 20 + + Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation + + + Received new file. + Received new file. + + src/app/services/consumer-status.service.ts + 21 + + + + File type not supported. + File type not supported. + + src/app/services/consumer-status.service.ts + 22 + + + + Processing document... + Processing document... + + src/app/services/consumer-status.service.ts + 23 + + + + Generating thumbnail... + Generating thumbnail... + + src/app/services/consumer-status.service.ts + 24 + + + + Retrieving date from document... + Retrieving date from document... + + src/app/services/consumer-status.service.ts + 25 + + + + Saving document... + Saving document... + + src/app/services/consumer-status.service.ts + 26 + + + + Finished. + Finished. + + src/app/services/consumer-status.service.ts + 27 + + + + Error + Error + + src/app/services/toast.service.ts + 35 + + + + Information + Information + + src/app/services/toast.service.ts + 39 + + + + ASN + ASN + + src/app/services/rest/document.service.ts + 17 + + + + Correspondent + Correspondent + + src/app/services/rest/document.service.ts + 18 + + + + Title + Title + + src/app/services/rest/document.service.ts + 19 + + + + Document type + Document type + + src/app/services/rest/document.service.ts + 20 + + + + Created + Created + + src/app/services/rest/document.service.ts + 21 + + + + Added + Added + + src/app/services/rest/document.service.ts + 22 + + + + Modified + Modified + + src/app/services/rest/document.service.ts + 23 + + + + Light blue + Light blue + + src/app/data/paperless-tag.ts + 6 + + + + Blue + Blue + + src/app/data/paperless-tag.ts + 7 + + + + Light green + Light green + + src/app/data/paperless-tag.ts + 8 + + + + Green + Green + + src/app/data/paperless-tag.ts + 9 + + + + Light red + Light red + + src/app/data/paperless-tag.ts + 10 + + + + Red + Red + + src/app/data/paperless-tag.ts + 11 + + + + Light orange + Light orange + + src/app/data/paperless-tag.ts + 12 + + + + Orange + Orange + + src/app/data/paperless-tag.ts + 13 + + + + Light violet + Light violet + + src/app/data/paperless-tag.ts + 14 + + + + Violet + Violet + + src/app/data/paperless-tag.ts + 15 + + + + Brown + Brown + + src/app/data/paperless-tag.ts + 16 + + + + Black + Black + + src/app/data/paperless-tag.ts + 17 + + + + Light grey + Light grey + + src/app/data/paperless-tag.ts + 18 + + + + Create new item + Create new item + + src/app/components/common/edit-dialog/edit-dialog.component.ts + 50 + + + + Edit item + Edit item + + src/app/components/common/edit-dialog/edit-dialog.component.ts + 54 + + + + Could not save element: + Could not save element: + + src/app/components/common/edit-dialog/edit-dialog.component.ts + 58 + + + + Automatic + Automatic + + src/app/components/manage/generic-list/generic-list.component.ts + 39 + + + + Do you really want to delete this element? + Do you really want to delete this element? + + src/app/components/manage/generic-list/generic-list.component.ts + 97 + + + + Associated documents will not be deleted. + Associated documents will not be deleted. + + src/app/components/manage/generic-list/generic-list.component.ts + 104 + + + + Delete + Delete + + src/app/components/manage/generic-list/generic-list.component.ts + 106 + + + + Error while deleting element: + Error while deleting element: + + src/app/components/manage/generic-list/generic-list.component.ts + 114 + + + + Any word + Any word + + src/app/data/matching-model.ts + 12 + + + + Any: Document contains any of these words (space separated) + Any: Document contains any of these words (space separated) + + src/app/data/matching-model.ts + 12 + + + + All words + All words + + src/app/data/matching-model.ts + 13 + + + + All: Document contains all of these words (space separated) + All: Document contains all of these words (space separated) + + src/app/data/matching-model.ts + 13 + + + + Exact match + Exact match + + src/app/data/matching-model.ts + 14 + + + + Exact: Document contains this string + Exact: Document contains this string + + src/app/data/matching-model.ts + 14 + + + + Regular expression + Regular expression + + src/app/data/matching-model.ts + 15 + + + + Regular expression: Document matches this regular expression + Regular expression: Document matches this regular expression + + src/app/data/matching-model.ts + 15 + + + + Fuzzy word + Fuzzy word + + src/app/data/matching-model.ts + 16 + + + + Fuzzy: Document contains a word similar to this word + Fuzzy: Document contains a word similar to this word + + src/app/data/matching-model.ts + 16 + + + + Auto: Learn matching automatically + Auto: Learn matching automatically + + src/app/data/matching-model.ts + 17 + + + + + \ No newline at end of file From 1d50d017acf871b0a0a7b7b6006578aa4a33f8eb Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 14:54:00 +0100 Subject: [PATCH 035/225] update bare metal instructions --- docs/setup.rst | 63 +++++++++++++++++++++++++++----------------------- 1 file changed, 34 insertions(+), 29 deletions(-) diff --git a/docs/setup.rst b/docs/setup.rst index 139db549a..4025b6721 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -221,7 +221,7 @@ Install Paperless from Docker Hub .. _setup-docker_build: -Build the docker image yourself +Build the Docker image yourself =============================== 1. Clone the entire repository of paperless: @@ -252,14 +252,14 @@ Build the docker image yourself 4. Run the ``compile-frontend.sh`` script. This requires ``node`` and ``npm >= v15``. -5. Follow steps 2 to 7 of :ref:`setup-docker_hub`. When asked to run - ``docker-compose up -d`` to start the containers, do +5. Follow steps 3 to 8 of :ref:`setup-docker_hub`. When asked to run + ``docker-compose pull`` to pull the image, do .. code:: shell-session $ docker-compose build - before that to build the image. + instead to build the image. .. _setup-bare_metal: @@ -309,7 +309,7 @@ writing. Windows is not and will never be supported. 2. Install ``redis`` >= 5.0 and configure it to start automatically. 3. Optional. Install ``postgresql`` and configure a database, user and password for paperless. If you do not wish - to use PostgreSQL, SQLite is avialable as well. + to use PostgreSQL, SQLite is available as well. 4. Get the release archive from ``_. If you clone the git repo as it is, you also have to compile the front end by yourself. @@ -333,8 +333,14 @@ writing. Windows is not and will never be supported. * Set ``PAPERLESS_OCR_LANGUAGE`` to the language most of your documents are written in. * Set ``PAPERLESS_TIME_ZONE`` to your local time zone. -6. Setup permissions. Create a system users under which you wish to run paperless. Ensure that these directories exist - and that the user has write permissions to the following directories +6. Create a system user under which you wish to run paperless. + + .. code:: shell-session + + adduser paperless --system --home /opt/paperless --group + +7. Ensure that these directories exist + and that the paperless user has write permissions to the following directories: * ``/opt/paperless/media`` * ``/opt/paperless/data`` @@ -342,30 +348,32 @@ writing. Windows is not and will never be supported. Adjust as necessary if you configured different folders. -7. Install python requirements from the ``requirements.txt`` file. +8. Install python requirements from the ``requirements.txt`` file. It is up to you if you wish to use a virtual environment or not. .. code:: shell-session - pip3 install -r requirements.txt + sudo -Hu paperless pip3 install -r requirements.txt + + This will install all python dependencies in the home directory of + the new paperless user. - -8. Go to ``/opt/paperless/src``, and execute the following commands: +9. Go to ``/opt/paperless/src``, and execute the following commands: .. code:: bash # This creates the database schema. - python3 manage.py migrate + sudo -Hu paperless python3 manage.py migrate # This creates your first paperless user - python3 manage.py createsuperuser + sudo -Hu paperless python3 manage.py createsuperuser -9. Optional: Test that paperless is working by executing +10. Optional: Test that paperless is working by executing .. code:: bash # This collects static files from paperless and django. - python3 manage.py runserver + sudo -Hu paperless python3 manage.py runserver and pointing your browser to http://localhost:8000/. @@ -380,7 +388,7 @@ writing. Windows is not and will never be supported. This will not start the consumer. Paperless does this in a separate process. -10. Setup systemd services to run paperless automatically. You may +11. Setup systemd services to run paperless automatically. You may use the service definition files included in the ``scripts`` folder as a starting point. @@ -396,14 +404,7 @@ writing. Windows is not and will never be supported. These services rely on redis and optionally the database server, but don't need to be started in any particular order. The example files depend on redis being started. If you use a database server, you should - add additinal dependencies. - - .. hint:: - - You may optionally set up your preferred web server to serve - paperless as a wsgi application directly instead of running the - ``webserver`` service. The module containing the wsgi application - is named ``paperless.wsgi``. + add additional dependencies. .. caution:: @@ -412,10 +413,13 @@ writing. Windows is not and will never be supported. however, the documentation of GUnicorn states that you should use a proxy server in front of gunicorn instead. -11. Optional: Install a samba server and make the consumption folder + For instructions on how to use nginx for that, + :ref:`see the instructions below `. + +12. Optional: Install a samba server and make the consumption folder available as a network share. -12. Configure ImageMagick to allow processing of PDF documents. Most distributions have +13. Configure ImageMagick to allow processing of PDF documents. Most distributions have this disabled by default, since PDF documents can contain malware. If you don't do this, paperless will fall back to ghostscript for certain steps such as thumbnail generation. @@ -432,7 +436,7 @@ writing. Windows is not and will never be supported. -13. Optional: Install the `jbig2enc `_ +14. Optional: Install the `jbig2enc `_ encoder. This will reduce the size of generated PDF documents. You'll most likely need to compile this by yourself, because this software has been patented until around 2017 and binary packages are not available for most distributions. @@ -441,9 +445,10 @@ writing. Windows is not and will never be supported. Install Paperless using ansible =============================== - .. note:: - This role currently only supports Debian 10 Buster and Ubuntu 20.04 Focal or later as target hosts. +.. note:: + + This role currently only supports Debian 10 Buster and Ubuntu 20.04 Focal or later as target hosts. 1. Install ansible 2.7+ on the management node. This may be the target host paperless-ng is being installed on or any remote host which can access the target host. From e35723e7af318f3fa1bc82c5c77fb05e00679eb2 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 15:30:47 +0100 Subject: [PATCH 036/225] fix some logging messages --- src/documents/consumer.py | 4 ++-- src/documents/matching.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/documents/consumer.py b/src/documents/consumer.py index 23988c7c1..3baedb5c3 100755 --- a/src/documents/consumer.py +++ b/src/documents/consumer.py @@ -200,8 +200,6 @@ class Consumer(LoggingMixin): MESSAGE_UNSUPPORTED_TYPE, f"Unsupported mime type {mime_type}" ) - else: - self.log("debug", f"Parser: {parser_class.__name__}") # Notify all listeners that we're going to do some work. @@ -222,6 +220,8 @@ class Consumer(LoggingMixin): document_parser = parser_class(self.logging_group, progress_callback) + self.log("debug", f"Parser: {type(document_parser).__name__}") + # However, this already created working directories which we have to # clean up. diff --git a/src/documents/matching.py b/src/documents/matching.py index edbbdc1b8..3b70930b6 100644 --- a/src/documents/matching.py +++ b/src/documents/matching.py @@ -11,7 +11,7 @@ logger = logging.getLogger("paperless.matching") def log_reason(matching_model, document, reason): class_name = type(matching_model).__name__ logger.debug( - f"Assigning {class_name} {matching_model.name} to document " + f"{class_name} {matching_model.name} matched on document " f"{document} because {reason}") From 019dfeb3eec3962ff219d17bae4e0c999e44283a Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 16:59:04 +0100 Subject: [PATCH 037/225] separate mail logging --- src/paperless_mail/mail.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/paperless_mail/mail.py b/src/paperless_mail/mail.py index 4a62ffd95..28a8a0784 100644 --- a/src/paperless_mail/mail.py +++ b/src/paperless_mail/mail.py @@ -102,7 +102,7 @@ def get_mailbox(server, port, security): class MailAccountHandler(LoggingMixin): - logging_name = "paperless.mail" + logging_name = "paperless_mail" def _correspondent_from_name(self, name): try: From 5696c02ac319657c262a00250aff46aef341a3c2 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 17:00:06 +0100 Subject: [PATCH 038/225] remove log admin --- src/documents/admin.py | 17 ----------------- 1 file changed, 17 deletions(-) diff --git a/src/documents/admin.py b/src/documents/admin.py index 78437f91c..266422d51 100755 --- a/src/documents/admin.py +++ b/src/documents/admin.py @@ -116,22 +116,6 @@ class DocumentAdmin(admin.ModelAdmin): return format_html("<{} {}/>", kind, attributes) -class LogAdmin(admin.ModelAdmin): - - def has_add_permission(self, request): - return False - - def has_change_permission(self, request, obj=None): - return False - - list_display = ("created", "message", "level",) - list_filter = ("level", "created",) - - ordering = ('-created',) - - list_display_links = ("created", "message") - - class RuleInline(admin.TabularInline): model = SavedViewFilterRule @@ -149,5 +133,4 @@ admin.site.register(Correspondent, CorrespondentAdmin) admin.site.register(Tag, TagAdmin) admin.site.register(DocumentType, DocumentTypeAdmin) admin.site.register(Document, DocumentAdmin) -admin.site.register(Log, LogAdmin) admin.site.register(SavedView, SavedViewAdmin) From fe435f66538f82e73d3cb37c5d18b8ae2b0a70ff Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 17:02:00 +0100 Subject: [PATCH 039/225] update API access for logs --- src/documents/serialisers.py | 13 ------ src/documents/tests/test_api.py | 24 +++++++++++ src/documents/tests/test_logger.py | 66 ------------------------------ src/documents/tests/utils.py | 4 ++ src/documents/views.py | 36 ++++++++++------ src/paperless/urls.py | 2 +- 6 files changed, 52 insertions(+), 93 deletions(-) delete mode 100644 src/documents/tests/test_logger.py diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py index aa4ef4bf8..fb0962902 100644 --- a/src/documents/serialisers.py +++ b/src/documents/serialisers.py @@ -153,19 +153,6 @@ class DocumentSerializer(DynamicFieldsModelSerializer): ) -class LogSerializer(serializers.ModelSerializer): - - class Meta: - model = Log - fields = ( - "id", - "created", - "message", - "group", - "level" - ) - - class SavedViewFilterRuleSerializer(serializers.ModelSerializer): class Meta: diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py index 9e4b77189..54d2b292d 100644 --- a/src/documents/tests/test_api.py +++ b/src/documents/tests/test_api.py @@ -4,7 +4,9 @@ import shutil import tempfile from unittest import mock +from django.conf import settings from django.contrib.auth.models import User +from django.test import override_settings from rest_framework.test import APITestCase from whoosh.writing import AsyncWriter @@ -717,6 +719,28 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): v1 = SavedView.objects.get(id=v1.id) self.assertEqual(v1.filter_rules.count(), 0) + def test_get_logs(self): + response = self.client.get("/api/logs/") + self.assertEqual(response.status_code, 200) + self.assertCountEqual(response.data, ["mail", "paperless"]) + + def test_get_invalid_log(self): + response = self.client.get("/api/logs/bogus_log/") + self.assertEqual(response.status_code, 404) + + @override_settings(LOGGING_DIR="bogus_dir") + def test_get_nonexistent_log(self): + response = self.client.get("/api/logs/paperless/") + self.assertEqual(response.status_code, 404) + + def test_get_log(self): + log_data = "test\ntest2\n" + with open(os.path.join(settings.LOGGING_DIR, "paperless.log"), "w") as f: + f.write(log_data) + response = self.client.get("/api/logs/paperless/") + self.assertEqual(response.status_code, 200) + self.assertListEqual(response.data, ["test", "test2"]) + class TestBulkEdit(DirectoriesMixin, APITestCase): diff --git a/src/documents/tests/test_logger.py b/src/documents/tests/test_logger.py deleted file mode 100644 index bbc9c2b5d..000000000 --- a/src/documents/tests/test_logger.py +++ /dev/null @@ -1,66 +0,0 @@ -import logging -import uuid -from unittest import mock - -from django.test import TestCase, override_settings - -from ..models import Log - - -class TestPaperlessLog(TestCase): - - def __init__(self, *args, **kwargs): - TestCase.__init__(self, *args, **kwargs) - self.logger = logging.getLogger( - "documents.management.commands.document_consumer") - - @override_settings(DISABLE_DBHANDLER=False) - def test_that_it_saves_at_all(self): - - kw = {"group": uuid.uuid4()} - - self.assertEqual(Log.objects.all().count(), 0) - - with mock.patch("logging.StreamHandler.emit") as __: - - # Debug messages are ignored by default - self.logger.debug("This is a debugging message", extra=kw) - self.assertEqual(Log.objects.all().count(), 1) - - self.logger.info("This is an informational message", extra=kw) - self.assertEqual(Log.objects.all().count(), 2) - - self.logger.warning("This is an warning message", extra=kw) - self.assertEqual(Log.objects.all().count(), 3) - - self.logger.error("This is an error message", extra=kw) - self.assertEqual(Log.objects.all().count(), 4) - - self.logger.critical("This is a critical message", extra=kw) - self.assertEqual(Log.objects.all().count(), 5) - - @override_settings(DISABLE_DBHANDLER=False) - def test_groups(self): - - kw1 = {"group": uuid.uuid4()} - kw2 = {"group": uuid.uuid4()} - - self.assertEqual(Log.objects.all().count(), 0) - - with mock.patch("logging.StreamHandler.emit") as __: - - self.logger.info("This is an informational message", extra=kw2) - self.assertEqual(Log.objects.all().count(), 1) - self.assertEqual(Log.objects.filter(group=kw2["group"]).count(), 1) - - self.logger.warning("This is an warning message", extra=kw1) - self.assertEqual(Log.objects.all().count(), 2) - self.assertEqual(Log.objects.filter(group=kw1["group"]).count(), 1) - - self.logger.error("This is an error message", extra=kw2) - self.assertEqual(Log.objects.all().count(), 3) - self.assertEqual(Log.objects.filter(group=kw2["group"]).count(), 2) - - self.logger.critical("This is a critical message", extra=kw1) - self.assertEqual(Log.objects.all().count(), 4) - self.assertEqual(Log.objects.filter(group=kw1["group"]).count(), 2) diff --git a/src/documents/tests/utils.py b/src/documents/tests/utils.py index dfefc4061..ebf79f6c5 100644 --- a/src/documents/tests/utils.py +++ b/src/documents/tests/utils.py @@ -19,12 +19,15 @@ def setup_directories(): dirs.originals_dir = os.path.join(dirs.media_dir, "documents", "originals") dirs.thumbnail_dir = os.path.join(dirs.media_dir, "documents", "thumbnails") dirs.archive_dir = os.path.join(dirs.media_dir, "documents", "archive") + dirs.logging_dir = os.path.join(dirs.data_dir, "log") os.makedirs(dirs.index_dir, exist_ok=True) os.makedirs(dirs.originals_dir, exist_ok=True) os.makedirs(dirs.thumbnail_dir, exist_ok=True) os.makedirs(dirs.archive_dir, exist_ok=True) + os.makedirs(dirs.logging_dir, exist_ok=True) + dirs.settings_override = override_settings( DATA_DIR=dirs.data_dir, SCRATCH_DIR=dirs.scratch_dir, @@ -33,6 +36,7 @@ def setup_directories(): THUMBNAIL_DIR=dirs.thumbnail_dir, ARCHIVE_DIR=dirs.archive_dir, CONSUMPTION_DIR=dirs.consumption_dir, + LOGGING_DIR=dirs.logging_dir, INDEX_DIR=dirs.index_dir, MODEL_FILE=os.path.join(dirs.data_dir, "classification_model.pickle"), MEDIA_LOCK=os.path.join(dirs.media_dir, "media.lock") diff --git a/src/documents/views.py b/src/documents/views.py index e413d4b35..2443ae342 100755 --- a/src/documents/views.py +++ b/src/documents/views.py @@ -29,7 +29,7 @@ from rest_framework.views import APIView from rest_framework.viewsets import ( GenericViewSet, ModelViewSet, - ReadOnlyModelViewSet + ViewSet ) import documents.index as index @@ -40,16 +40,14 @@ from .filters import ( CorrespondentFilterSet, DocumentFilterSet, TagFilterSet, - DocumentTypeFilterSet, - LogFilterSet + DocumentTypeFilterSet ) from .matching import match_correspondents, match_tags, match_document_types -from .models import Correspondent, Document, Log, Tag, DocumentType, SavedView +from .models import Correspondent, Document, Tag, DocumentType, SavedView from .parsers import get_parser_class_for_mime_type from .serialisers import ( CorrespondentSerializer, DocumentSerializer, - LogSerializer, TagSerializer, DocumentTypeSerializer, PostDocumentSerializer, @@ -307,16 +305,28 @@ class DocumentViewSet(RetrieveModelMixin, raise Http404() -class LogViewSet(ReadOnlyModelViewSet): - model = Log +class LogViewSet(ViewSet): - queryset = Log.objects.all() - serializer_class = LogSerializer - pagination_class = StandardPagination permission_classes = (IsAuthenticated,) - filter_backends = (DjangoFilterBackend, OrderingFilter) - filterset_class = LogFilterSet - ordering_fields = ("created",) + + log_files = ["paperless", "mail"] + + def retrieve(self, request, pk=None, *args, **kwargs): + if not pk in self.log_files: + raise Http404() + + filename = os.path.join(settings.LOGGING_DIR, f"{pk}.log") + + if not os.path.isfile(filename): + raise Http404() + + with open(filename, "r") as f: + lines = [l.rstrip() for l in f.readlines()] + + return Response(lines) + + def list(self, request, *args, **kwargs): + return Response(self.log_files) class SavedViewViewSet(ModelViewSet): diff --git a/src/paperless/urls.py b/src/paperless/urls.py index 5b58de454..40f4bd754 100755 --- a/src/paperless/urls.py +++ b/src/paperless/urls.py @@ -31,7 +31,7 @@ api_router = DefaultRouter() api_router.register(r"correspondents", CorrespondentViewSet) api_router.register(r"document_types", DocumentTypeViewSet) api_router.register(r"documents", DocumentViewSet) -api_router.register(r"logs", LogViewSet) +api_router.register(r"logs", LogViewSet, basename="logs") api_router.register(r"tags", TagViewSet) api_router.register(r"saved_views", SavedViewViewSet) From 9fa3f91e8a870c165094034c4d84337f365dd544 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 17:05:07 +0100 Subject: [PATCH 040/225] update logging config --- Pipfile | 1 + Pipfile.lock | 158 +++++++++++++------------------------- src/paperless/settings.py | 51 +++++++----- 3 files changed, 86 insertions(+), 124 deletions(-) diff --git a/Pipfile b/Pipfile index 293d71603..43b9afd3f 100644 --- a/Pipfile +++ b/Pipfile @@ -48,6 +48,7 @@ tika = "*" channels = "~=3.0" channels-redis = "*" uvicorn = {extras = ["standard"], version = "*"} +concurrent-log-handler = "*" [dev-packages] coveralls = "*" diff --git a/Pipfile.lock b/Pipfile.lock index 78cfecf95..4da3d625d 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "658e2907c680baea81e77da9081ee063ff843b185d02c95d6483b747f0acd84d" + "sha256": "70117fd0d9bc10ab46b05ec2c58b59ca0e52b32a3c3e4e130c83bcfb70b51d5f" }, "pipfile-spec": 6, "requires": {}, @@ -60,11 +60,11 @@ }, "autobahn": { "hashes": [ - "sha256:410a93e0e29882c8b5d5ab05d220b07609b886ef5f23c0b8d39153254ffd6895", - "sha256:52ee4236ff9a1fcbbd9500439dcf3284284b37f8a6b31ecc8a36e00cf9f95049" + "sha256:93df8fc9d1821c9dabff9fed52181a9ad6eea5e9989d53102c391607d7c1666e", + "sha256:cceed2121b7a93024daa93c91fae33007f8346f0e522796421f36a6183abea99" ], "markers": "python_version >= '3.6'", - "version": "==20.12.3" + "version": "==21.1.1" }, "automat": { "hashes": [ @@ -173,6 +173,14 @@ "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'", "version": "==15.0" }, + "concurrent-log-handler": { + "hashes": [ + "sha256:00d5ca24d463a7013c3479b026f34b76da4b50df8d76194132b8d8403c014379", + "sha256:b12f79abed3f94121c25ce9c24cdb57d889282ec6ff61f5535ab2068dc37d409" + ], + "index": "pypi", + "version": "==0.9.19" + }, "constantly": { "hashes": [ "sha256:586372eb92059873e29eba4f9dec8381541b4d3834660707faf8ba59146dfc35", @@ -420,14 +428,6 @@ ], "version": "==0.4.0" }, - "importlib-metadata": { - "hashes": [ - "sha256:ace61d5fc652dc280e7b6b4ff732a9c2d40db2c0f92bc6cb74e07b73d53a1771", - "sha256:fa5daa4477a7414ae34e95942e4dd07f62adf589143c875c133c1e53c4eff38d" - ], - "markers": "python_version < '3.8'", - "version": "==3.4.0" - }, "incremental": { "hashes": [ "sha256:717e12246dddf231a349175f48d74d93e2897244939173b01974ab6661406b9f", @@ -550,44 +550,34 @@ }, "numpy": { "hashes": [ - "sha256:012426a41bc9ab63bb158635aecccc7610e3eff5d31d1eb43bc099debc979d94", - "sha256:06fab248a088e439402141ea04f0fffb203723148f6ee791e9c75b3e9e82f080", - "sha256:0eef32ca3132a48e43f6a0f5a82cb508f22ce5a3d6f67a8329c81c8e226d3f6e", - "sha256:1ded4fce9cfaaf24e7a0ab51b7a87be9038ea1ace7f34b841fe3b6894c721d1c", - "sha256:2e55195bc1c6b705bfd8ad6f288b38b11b1af32f3c8289d6c50d47f950c12e76", - "sha256:2ea52bd92ab9f768cc64a4c3ef8f4b2580a17af0a5436f6126b08efbd1838371", - "sha256:36674959eed6957e61f11c912f71e78857a8d0604171dfd9ce9ad5cbf41c511c", - "sha256:384ec0463d1c2671170901994aeb6dce126de0a95ccc3976c43b0038a37329c2", - "sha256:39b70c19ec771805081578cc936bbe95336798b7edf4732ed102e7a43ec5c07a", - "sha256:400580cbd3cff6ffa6293df2278c75aef2d58d8d93d3c5614cd67981dae68ceb", - "sha256:43d4c81d5ffdff6bae58d66a3cd7f54a7acd9a0e7b18d97abb255defc09e3140", - "sha256:50a4a0ad0111cc1b71fa32dedd05fa239f7fb5a43a40663269bb5dc7877cfd28", - "sha256:603aa0706be710eea8884af807b1b3bc9fb2e49b9f4da439e76000f3b3c6ff0f", - "sha256:6149a185cece5ee78d1d196938b2a8f9d09f5a5ebfbba66969302a778d5ddd1d", - "sha256:6373751c4b6fd325606d29dd98dc2bf7092485ad20aafbfc6a177acd3b89059e", - "sha256:759e4095edc3c1b3ac031f34d9459fa781777a93ccc633a472a5468587a190ff", - "sha256:7fb43004bce0ca31d8f13a6eb5e943fa73371381e53f7074ed21a4cb786c32f8", - "sha256:811daee36a58dc79cf3d8bdd4a490e4277d0e4b7d103a001a4e73ddb48e7e6aa", - "sha256:8b5e972b43c8fc27d56550b4120fe6257fdc15f9301914380b27f74856299fea", - "sha256:99abf4f353c3d1a0c7a5f27699482c987cf663b1eac20db59b8c7b061eabd7fc", - "sha256:a0d53e51a6cb6f0d9082decb7a4cb6dfb33055308c4c44f53103c073f649af73", - "sha256:a12ff4c8ddfee61f90a1633a4c4afd3f7bcb32b11c52026c92a12e1325922d0d", - "sha256:a4646724fba402aa7504cd48b4b50e783296b5e10a524c7a6da62e4a8ac9698d", - "sha256:a76f502430dd98d7546e1ea2250a7360c065a5fdea52b2dffe8ae7180909b6f4", - "sha256:a9d17f2be3b427fbb2bce61e596cf555d6f8a56c222bd2ca148baeeb5e5c783c", - "sha256:ab83f24d5c52d60dbc8cd0528759532736b56db58adaa7b5f1f76ad551416a1e", - "sha256:aeb9ed923be74e659984e321f609b9ba54a48354bfd168d21a2b072ed1e833ea", - "sha256:c843b3f50d1ab7361ca4f0b3639bf691569493a56808a0b0c54a051d260b7dbd", - "sha256:cae865b1cae1ec2663d8ea56ef6ff185bad091a5e33ebbadd98de2cfa3fa668f", - "sha256:cc6bd4fd593cb261332568485e20a0712883cf631f6f5e8e86a52caa8b2b50ff", - "sha256:cf2402002d3d9f91c8b01e66fbb436a4ed01c6498fffed0e4c7566da1d40ee1e", - "sha256:d051ec1c64b85ecc69531e1137bb9751c6830772ee5c1c426dbcfe98ef5788d7", - "sha256:d6631f2e867676b13026e2846180e2c13c1e11289d67da08d71cacb2cd93d4aa", - "sha256:dbd18bcf4889b720ba13a27ec2f2aac1981bd41203b3a3b27ba7a33f88ae4827", - "sha256:df609c82f18c5b9f6cb97271f03315ff0dbe481a2a02e56aeb1b1a985ce38e60" + "sha256:0d28a54afcf46f1f9ebd163e49ad6b49087f22986fefd01a23ca0c1cdda25ca6", + "sha256:1264c66129f5ef63187649dd43f1ca59532e8c098723643336a85131c0dcce3f", + "sha256:1abc02e30e3efd81a4571e00f8e62bf42e343c76698e0a3e11d9c2b3ee0d77a7", + "sha256:2445a96fbae23a4109c61be0f0af0f3bc273905dc5687a710850c1dfde0fc994", + "sha256:2bf0e68c92ef077fe766e53f8937d8ac341bdbca68ec128ae049b7d5c34e3206", + "sha256:33edfc0eb229f86f539493917b34035054313a11afbed48404aaf9f86bf4b0f6", + "sha256:3d8233c03f116d068d5365fed4477f2947c7229582dad81e5953088989294cec", + "sha256:4d592264d2a4f368afbb4288b5ceb646d4cbaf559c0249c096fbb0a149806b90", + "sha256:5ae765dd29c71a555f8102281f6fb15a3f4dbd35f6e7daf36af9df6d9dd716a5", + "sha256:894aaee60043a98b03f0ad992c810f62e3a15f98a701e1c0f58a4f4a0df13429", + "sha256:89bd70c9ad540febe6c28451ba225eb4e49d27f64728357f512c808002325dfa", + "sha256:93c2abea7bb69f47029b84ceac30ab46dfcfdb99b671ad850a333ff794a765e4", + "sha256:abdfa075e293d73638ece434708aa60b510dc6e70d805f57f481a0f550b25a9e", + "sha256:afeee581b50df20ef07b736e62ca612858f1fcdba96651d26ab44e3d567a4e6e", + "sha256:b51b9ef0624f4b01b846c981034c10d2e30db33f9f8be71e992f3900741f6f77", + "sha256:b66a6c15d793eda7cdad986e737775aa31b9306d588c14dd0277d2dda5546150", + "sha256:bd08254aac78b363b14acac456038ce6438bad848e02b275837930115e41ea21", + "sha256:cb257bb0c0a3176c32782a63cfab2eace7eabfa2a3b2dfd85a13700617ccaf28", + "sha256:cf5d9dcbdbe523fa665c5309cce5f144648d94a7fddbf5a40f8e0d5c9f5b596d", + "sha256:d1bc331e1706fd1809a1bc8a31205329e5b30cf5ba50461c624da267e99f6ae6", + "sha256:db5e69d08756a2fa75a42b4e433880b6187768fe1bc73d21819def893e5128c6", + "sha256:e3db646af9f6a145f0c57202f4b55d4a33f975e395e78fb7b394644c17c1a3a6", + "sha256:e9c5fd330d2fedf06051bafb996252de9b032fcb2ec03eefc9a543e56efa66d4", + "sha256:eee454d3aa3955d0c0069a0f265fea47f1e1384c35a110a95efed358eb6e1562", + "sha256:f1e9424e9aa3834ea27cc12f9c6ea8ace5da18ee60a720bb3a85b2f733f41782" ], - "markers": "python_version >= '3.6'", - "version": "==1.19.5" + "markers": "python_version >= '3.7'", + "version": "==1.20.0" }, "ocrmypdf": { "hashes": [ @@ -696,6 +686,14 @@ "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", "version": "==0.13.1" }, + "portalocker": { + "hashes": [ + "sha256:afa66c85041d7bac1532acb74ff26cc2406472dcf7145c90c31ff8ca4f1cc146", + "sha256:e5f6ffb2f360e9aef615a7c284143d2a93bb640c62e8e45a703e6083fc5aa114" + ], + "markers": "python_version >= '3'", + "version": "==2.2.1" + }, "psycopg2-binary": { "hashes": [ "sha256:0deac2af1a587ae12836aa07970f5cb91964f05a7c6cdb69d8425ff4c15d4e2c", @@ -1144,15 +1142,6 @@ "markers": "python_version >= '3.6'", "version": "==20.12.1" }, - "typing-extensions": { - "hashes": [ - "sha256:7cb407020f00f7bfc3cb3e7881628838e69d8f3fcab2f64742a5e76b2f841918", - "sha256:99d4073b617d30288f569d3f13d2bd7548c3a7e4c8de87db09a9d29bb3a4a60c", - "sha256:dafc7639cde7f1b6e1acc0f457842a83e722ccca8eef5270af2d74792619a89f" - ], - "markers": "python_version < '3.8'", - "version": "==3.7.4.3" - }, "tzlocal": { "hashes": [ "sha256:643c97c5294aedc737780a49d9df30889321cbe1204eac2c2ec6134035a92e44", @@ -1278,14 +1267,6 @@ "index": "pypi", "version": "==2.7.4" }, - "zipp": { - "hashes": [ - "sha256:102c24ef8f171fd729d46599845e95c7ab894a4cf45f5de11a44cc7444fb1108", - "sha256:ed5eee1974372595f9e416cc7bbeeb12335201d8081ca8a0743c954d4446e5cb" - ], - "markers": "python_version >= '3.6'", - "version": "==3.4.0" - }, "zope.interface": { "hashes": [ "sha256:05a97ba92c1c7c26f25c9f671aa1ef85ffead6cdad13770e5b689cf983adc7e1", @@ -1506,11 +1487,11 @@ }, "faker": { "hashes": [ - "sha256:0783729c61501d52efea2967aff6e6fcb8370f0f6b5a558f2a81233642ae529a", - "sha256:6b2995ffff6c2b02bc5daad96f8c24c021e5bd491d9d53d31bcbd66f348181d4" + "sha256:190f0d3ce037866b5d230f0b9fd0f513f07c25dc326dcad6ee019849c68d441c", + "sha256:db7adc3b4755005fc960cf96fb4ed46b54b6eb21413741ab3f31a9595f379905" ], "markers": "python_version >= '3.6'", - "version": "==5.8.0" + "version": "==6.0.0" }, "filelock": { "hashes": [ @@ -1537,22 +1518,6 @@ "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", "version": "==1.2.0" }, - "importlib-metadata": { - "hashes": [ - "sha256:ace61d5fc652dc280e7b6b4ff732a9c2d40db2c0f92bc6cb74e07b73d53a1771", - "sha256:fa5daa4477a7414ae34e95942e4dd07f62adf589143c875c133c1e53c4eff38d" - ], - "markers": "python_version < '3.8'", - "version": "==3.4.0" - }, - "importlib-resources": { - "hashes": [ - "sha256:885b8eae589179f661c909d699a546cf10d83692553e34dca1bf5eb06f7f6217", - "sha256:bfdad047bce441405a49cf8eb48ddce5e56c696e185f59147a8b79e75e9e6380" - ], - "markers": "python_version < '3.7'", - "version": "==5.1.0" - }, "iniconfig": { "hashes": [ "sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3", @@ -1861,20 +1826,11 @@ }, "tox": { "hashes": [ - "sha256:76df3db6eee929bb62bdbacca5bb6bc840669d98e86a015b7a57b7df0a6eaf8b", - "sha256:854e6e4a71c614b488f81cb88df3b92edcb1a9ec43d4102e6289e9669bbf7f18" + "sha256:65d0e90ceb816638a50d64f4b47b11da767b284c0addda2294cb3cd69bd72425", + "sha256:cf7fef81a3a2434df4d7af2a6d1bf606d2970220addfbe7dea2615bd4bb2c252" ], "index": "pypi", - "version": "==3.21.3" - }, - "typing-extensions": { - "hashes": [ - "sha256:7cb407020f00f7bfc3cb3e7881628838e69d8f3fcab2f64742a5e76b2f841918", - "sha256:99d4073b617d30288f569d3f13d2bd7548c3a7e4c8de87db09a9d29bb3a4a60c", - "sha256:dafc7639cde7f1b6e1acc0f457842a83e722ccca8eef5270af2d74792619a89f" - ], - "markers": "python_version < '3.8'", - "version": "==3.7.4.3" + "version": "==3.21.4" }, "urllib3": { "hashes": [ @@ -1891,14 +1847,6 @@ ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", "version": "==20.4.2" - }, - "zipp": { - "hashes": [ - "sha256:102c24ef8f171fd729d46599845e95c7ab894a4cf45f5de11a44cc7444fb1108", - "sha256:ed5eee1974372595f9e416cc7bbeeb12335201d8081ca8a0743c954d4446e5cb" - ], - "markers": "python_version >= '3.6'", - "version": "==3.4.0" } } } diff --git a/src/paperless/settings.py b/src/paperless/settings.py index 5a3c206ed..d518249c6 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -4,6 +4,7 @@ import multiprocessing import os import re +from concurrent_log_handler.queue import setup_logging_queues from dotenv import load_dotenv from django.utils.translation import gettext_lazy as _ @@ -62,6 +63,8 @@ MEDIA_LOCK = os.path.join(MEDIA_ROOT, "media.lock") INDEX_DIR = os.path.join(DATA_DIR, "index") MODEL_FILE = os.path.join(DATA_DIR, "classification_model.pickle") +LOGGING_DIR = os.getenv('PAPERLESS_LOGGING_DIR', os.path.join(DATA_DIR, "log")) + CONSUMPTION_DIR = os.getenv("PAPERLESS_CONSUMPTION_DIR", os.path.join(BASE_DIR, "..", "consume")) # This will be created if it doesn't exist @@ -306,14 +309,19 @@ USE_TZ = True # Logging # ############################################################################### -DISABLE_DBHANDLER = __get_boolean("PAPERLESS_DISABLE_DBHANDLER") +setup_logging_queues() + +os.makedirs(LOGGING_DIR, exist_ok=True) + +LOGROTATE_MAX_SIZE = os.getenv("PAPERLESS_LOGROTATE_MAX_SIZE", 1000000) +LOGROTATE_MAX_BACKUPS = os.getenv("PAPERLESS_LOGROTATE_MAX_BACKUPS", 20) LOGGING = { "version": 1, "disable_existing_loggers": False, 'formatters': { 'verbose': { - 'format': '{levelname} {asctime} {module} {message}', + 'format': '[{asctime}] [{levelname}] [{name}] {message}', 'style': '{', }, 'simple': { @@ -322,34 +330,39 @@ LOGGING = { }, }, "handlers": { - "db": { - "level": "DEBUG", - "class": "documents.loggers.PaperlessHandler", - }, "console": { "level": "DEBUG" if DEBUG else "INFO", "class": "logging.StreamHandler", "formatter": "verbose", + }, + "file_paperless": { + "class": "concurrent_log_handler.ConcurrentRotatingFileHandler", + "formatter": "verbose", + "filename": os.path.join(LOGGING_DIR, "paperless.log"), + "maxBytes": LOGROTATE_MAX_SIZE, + "backupCount": LOGROTATE_MAX_BACKUPS + }, + "file_mail": { + "class": "concurrent_log_handler.ConcurrentRotatingFileHandler", + "formatter": "verbose", + "filename": os.path.join(LOGGING_DIR, "mail.log"), + "maxBytes": LOGROTATE_MAX_SIZE, + "backupCount": LOGROTATE_MAX_BACKUPS } }, "root": { - "handlers": ["console"], - "level": "DEBUG", + "handlers": ["console"] }, "loggers": { - "documents": { - "handlers": ["db"], - "propagate": True, + "paperless": { + "handlers": ["file_paperless"], + "level": "DEBUG" }, "paperless_mail": { - "handlers": ["db"], - "propagate": True, - }, - "paperless_tesseract": { - "handlers": ["db"], - "propagate": True, - }, - }, + "handlers": ["file_mail"], + "level": "DEBUG" + } + } } ############################################################################### From 9993381cda1fad164625259ede9dc57cb6620ad1 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 17:07:25 +0100 Subject: [PATCH 041/225] rework of the log viewer --- .../manage/logs/logs.component.html | 33 ++++------- .../manage/logs/logs.component.scss | 8 +++ .../components/manage/logs/logs.component.ts | 57 ++++++++++--------- src-ui/src/app/data/paperless-log.ts | 27 --------- src-ui/src/app/services/rest/log.service.ts | 17 ++++-- 5 files changed, 62 insertions(+), 80 deletions(-) delete mode 100644 src-ui/src/app/data/paperless-log.ts diff --git a/src-ui/src/app/components/manage/logs/logs.component.html b/src-ui/src/app/components/manage/logs/logs.component.html index be14faf1d..8a2a28ef8 100644 --- a/src-ui/src/app/components/manage/logs/logs.component.html +++ b/src-ui/src/app/components/manage/logs/logs.component.html @@ -1,27 +1,18 @@ -
    - -
    - -
    -
    -     -
    + + + +
    + +

    - {{log.created | customDate:'short'}} - {{getLevelText(log.level)}} - {{log.message}} -

    + class="m-0 p-0 log-entry-{{getLogLevel(log)}}" + *ngFor="let log of logs" style="white-space: pre;">{{log}}

    diff --git a/src-ui/src/app/components/manage/logs/logs.component.scss b/src-ui/src/app/components/manage/logs/logs.component.scss index dee9b10dc..a99506c12 100644 --- a/src-ui/src/app/components/manage/logs/logs.component.scss +++ b/src-ui/src/app/components/manage/logs/logs.component.scss @@ -13,4 +13,12 @@ .log-entry-50 { color: lightcoral !important; font-weight: bold; +} + +.log-container { + + overflow: scroll; + + height: calc(100vh - 190px); + top: 70px; } \ No newline at end of file diff --git a/src-ui/src/app/components/manage/logs/logs.component.ts b/src-ui/src/app/components/manage/logs/logs.component.ts index b131796ee..0754e4253 100644 --- a/src-ui/src/app/components/manage/logs/logs.component.ts +++ b/src-ui/src/app/components/manage/logs/logs.component.ts @@ -1,5 +1,4 @@ -import { Component, OnInit } from '@angular/core'; -import { LOG_LEVELS, LOG_LEVEL_INFO, PaperlessLog } from 'src/app/data/paperless-log'; +import { Component, ElementRef, OnInit, ViewChild } from '@angular/core'; import { LogService } from 'src/app/services/rest/log.service'; @Component({ @@ -11,38 +10,42 @@ export class LogsComponent implements OnInit { constructor(private logService: LogService) { } - logs: PaperlessLog[] = [] - level: number = LOG_LEVEL_INFO + @ViewChild('logContainer') private logContainer: ElementRef + + logs: string[] = [] + + logFiles: string[] = [] + + activeLog: string ngOnInit(): void { - this.reload() - } - - reload() { - this.logService.list(1, 50, 'created', true, {'level__gte': this.level}).subscribe(result => this.logs = result.results) - } - - getLevelText(level: number) { - return LOG_LEVELS.find(l => l.id == level)?.name - } - - onScroll() { - let lastCreated = null - if (this.logs.length > 0) { - lastCreated = new Date(this.logs[this.logs.length-1].created).toISOString() - } - this.logService.list(1, 25, 'created', true, {'created__lt': lastCreated, 'level__gte': this.level}).subscribe(result => { - this.logs.push(...result.results) + this.logService.list().subscribe(result => { + this.logFiles = result + if (this.logFiles.length > 0) { + this.activeLog = this.logFiles[0] + this.reloadLogs() + } }) } - getLevels() { - return LOG_LEVELS + reloadLogs() { + this.logService.get(this.activeLog).subscribe(result => { + this.logs = result + }) } - setLevel(id) { - this.level = id - this.reload() + getLogLevel(log: string) { + if (log.indexOf("[DEBUG]") != -1) { + return 10 + } else if (log.indexOf("[WARNING]") != -1) { + return 30 + } else if (log.indexOf("[ERROR]") != -1) { + return 40 + } else if (log.indexOf("[CRITICAL]") != -1) { + return 50 + } else { + return 20 + } } } diff --git a/src-ui/src/app/data/paperless-log.ts b/src-ui/src/app/data/paperless-log.ts deleted file mode 100644 index 61a6fce99..000000000 --- a/src-ui/src/app/data/paperless-log.ts +++ /dev/null @@ -1,27 +0,0 @@ -export const LOG_LEVEL_DEBUG = 10 -export const LOG_LEVEL_INFO = 20 -export const LOG_LEVEL_WARNING = 30 -export const LOG_LEVEL_ERROR = 40 -export const LOG_LEVEL_CRITICAL = 50 - -export const LOG_LEVELS = [ - {id: LOG_LEVEL_DEBUG, name: "DEBUG"}, - {id: LOG_LEVEL_INFO, name: "INFO"}, - {id: LOG_LEVEL_WARNING, name: "WARNING"}, - {id: LOG_LEVEL_ERROR, name: "ERROR"}, - {id: LOG_LEVEL_CRITICAL, name: "CRITICAL"} -] - -export interface PaperlessLog { - - id?: number - - group?: string - - message?: string - - created?: Date - - level?: number - -} diff --git a/src-ui/src/app/services/rest/log.service.ts b/src-ui/src/app/services/rest/log.service.ts index 797d9b6b9..3abf2fb6f 100644 --- a/src-ui/src/app/services/rest/log.service.ts +++ b/src-ui/src/app/services/rest/log.service.ts @@ -1,14 +1,21 @@ import { HttpClient } from '@angular/common/http'; import { Injectable } from '@angular/core'; -import { PaperlessLog } from 'src/app/data/paperless-log'; -import { AbstractPaperlessService } from './abstract-paperless-service'; +import { Observable } from 'rxjs'; +import { environment } from 'src/environments/environment'; @Injectable({ providedIn: 'root' }) -export class LogService extends AbstractPaperlessService { +export class LogService { - constructor(http: HttpClient) { - super(http, 'logs') + constructor(private http: HttpClient) { + } + + list(): Observable { + return this.http.get(`${environment.apiBaseUrl}logs/`) + } + + get(id: string): Observable { + return this.http.get(`${environment.apiBaseUrl}logs/${id}/`) } } From 80b3fa75a540ce6507334fe2224babe1b6b99400 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 17:21:32 +0100 Subject: [PATCH 042/225] pycodestyle --- src/documents/loggers.py | 5 ++++- src/documents/views.py | 4 ++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/src/documents/loggers.py b/src/documents/loggers.py index 4b0f03552..90f6770e0 100644 --- a/src/documents/loggers.py +++ b/src/documents/loggers.py @@ -17,7 +17,10 @@ class LoggingMixin: if self.logging_name: logger = logging.getLogger(self.logging_name) else: - name = ".".join([self.__class__.__module__, self.__class__.__name__]) + name = ".".join([ + self.__class__.__module__, + self.__class__.__name__ + ]) logger = logging.getLogger(name) getattr(logger, level)(message, extra={ diff --git a/src/documents/views.py b/src/documents/views.py index 2443ae342..5ab4ca9df 100755 --- a/src/documents/views.py +++ b/src/documents/views.py @@ -312,7 +312,7 @@ class LogViewSet(ViewSet): log_files = ["paperless", "mail"] def retrieve(self, request, pk=None, *args, **kwargs): - if not pk in self.log_files: + if pk not in self.log_files: raise Http404() filename = os.path.join(settings.LOGGING_DIR, f"{pk}.log") @@ -321,7 +321,7 @@ class LogViewSet(ViewSet): raise Http404() with open(filename, "r") as f: - lines = [l.rstrip() for l in f.readlines()] + lines = [line.rstrip() for line in f.readlines()] return Response(lines) From 5ef6f2697ea949a571672557b081658b483a54d6 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 17:21:40 +0100 Subject: [PATCH 043/225] default settings --- src/paperless/settings.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/paperless/settings.py b/src/paperless/settings.py index d518249c6..718738f71 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -313,7 +313,7 @@ setup_logging_queues() os.makedirs(LOGGING_DIR, exist_ok=True) -LOGROTATE_MAX_SIZE = os.getenv("PAPERLESS_LOGROTATE_MAX_SIZE", 1000000) +LOGROTATE_MAX_SIZE = os.getenv("PAPERLESS_LOGROTATE_MAX_SIZE", 1024*1024) LOGROTATE_MAX_BACKUPS = os.getenv("PAPERLESS_LOGROTATE_MAX_BACKUPS", 20) LOGGING = { From e8a1fde1c7ab0d086fc15210e98d02948990377b Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 17:22:05 +0100 Subject: [PATCH 044/225] documentation --- docs/changelog.rst | 7 +++++++ docs/configuration.rst | 27 +++++++++++++++++++++++---- 2 files changed, 30 insertions(+), 4 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 2b9821408..537e898c7 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -58,6 +58,13 @@ paperless-ng 1.1.0 * Better info section in the side bar. + * Paperless no longer logs to the database. Instead, logs are written to rotating log files. This solves many "database is locked" + issues on Raspberry Pi, especially when SQLite is used. + + * By default, log files are written to ``PAPERLESS_DATA_DIR/log/``. Logging settings can be adjusted with + ``PAPERLESS_LOGGING_DIR``, ``PAPERLESS_LOGROTATE_MAX_SIZE`` and + ``PAPERLESS_LOGROTATE_MAX_BACKUPS``. + paperless-ng 1.0.0 ################## diff --git a/docs/configuration.rst b/docs/configuration.rst index 36b124350..a307cc25a 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -72,13 +72,13 @@ PAPERLESS_CONSUMPTION_DIR= container. Change the local consumption directory in the docker-compose.yml file instead. - Defaults to "../consume", relative to the "src" directory. + Defaults to "../consume/", relative to the "src" directory. PAPERLESS_DATA_DIR= This is where paperless stores all its data (search index, SQLite database, classification model, etc). - Defaults to "../data", relative to the "src" directory. + Defaults to "../data/", relative to the "src" directory. PAPERLESS_MEDIA_ROOT= This is where your documents and thumbnails are stored. @@ -86,7 +86,7 @@ PAPERLESS_MEDIA_ROOT= You can set this and PAPERLESS_DATA_DIR to the same folder to have paperless store all its data within the same volume. - Defaults to "../media", relative to the "src" directory. + Defaults to "../media/", relative to the "src" directory. PAPERLESS_STATICDIR= Override the default STATIC_ROOT here. This is where all static files @@ -94,7 +94,7 @@ PAPERLESS_STATICDIR= Unless you're doing something fancy, there is no need to override this. - Defaults to "../static", relative to the "src" directory. + Defaults to "../static/", relative to the "src" directory. PAPERLESS_FILENAME_FORMAT= Changes the filenames paperless uses to store documents in the media directory. @@ -102,6 +102,25 @@ PAPERLESS_FILENAME_FORMAT= Default is none, which disables this feature. +PAPERLESS_LOGGING_DIR= + This is where paperless will store log files. + + Defaults to "``PAPERLESS_DATA_DIR``/log/". + + +Logging +####### + +PAPERLESS_LOGROTATE_MAX_SIZE= + Maximum file size for log files before they are rotated, in bytes. + + Defaults to 1 MiB. + +PAPERLESS_LOGROTATE_MAX_BACKUPS= + Number of rotated log files to keep. + + Defaults to 20. + Hosting & Security ################## From d09ea8408c74f7d35a0fb71c56a5ac1b4eea6339 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 17:36:31 +0100 Subject: [PATCH 045/225] keep numpy below 1.20, since that breaks python 3.6 --- Pipfile | 2 ++ Pipfile.lock | 66 ++++++++++++++++++++++++++++++---------------------- 2 files changed, 40 insertions(+), 28 deletions(-) diff --git a/Pipfile b/Pipfile index 43b9afd3f..504b267c4 100644 --- a/Pipfile +++ b/Pipfile @@ -21,6 +21,8 @@ fuzzywuzzy = {extras = ["speedup"], version = "*"} gunicorn = "*" imap-tools = "*" langdetect = "*" +# numpy 1.20.0 drops python 3.6 support +numpy = "~=1.19.5" pdftotext = "*" pathvalidate = "*" # pinned to 8.1.0, since aarch64 wheels might not be available beyond that https://github.com/python-pillow/Pillow/issues/5202 diff --git a/Pipfile.lock b/Pipfile.lock index 4da3d625d..fafec4310 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "70117fd0d9bc10ab46b05ec2c58b59ca0e52b32a3c3e4e130c83bcfb70b51d5f" + "sha256": "aec5aa96fc4cad6d06f00d0ada0f8bd193f42874f476d39e76fc69d05c78f9f6" }, "pipfile-spec": 6, "requires": {}, @@ -550,34 +550,44 @@ }, "numpy": { "hashes": [ - "sha256:0d28a54afcf46f1f9ebd163e49ad6b49087f22986fefd01a23ca0c1cdda25ca6", - "sha256:1264c66129f5ef63187649dd43f1ca59532e8c098723643336a85131c0dcce3f", - "sha256:1abc02e30e3efd81a4571e00f8e62bf42e343c76698e0a3e11d9c2b3ee0d77a7", - "sha256:2445a96fbae23a4109c61be0f0af0f3bc273905dc5687a710850c1dfde0fc994", - "sha256:2bf0e68c92ef077fe766e53f8937d8ac341bdbca68ec128ae049b7d5c34e3206", - "sha256:33edfc0eb229f86f539493917b34035054313a11afbed48404aaf9f86bf4b0f6", - "sha256:3d8233c03f116d068d5365fed4477f2947c7229582dad81e5953088989294cec", - "sha256:4d592264d2a4f368afbb4288b5ceb646d4cbaf559c0249c096fbb0a149806b90", - "sha256:5ae765dd29c71a555f8102281f6fb15a3f4dbd35f6e7daf36af9df6d9dd716a5", - "sha256:894aaee60043a98b03f0ad992c810f62e3a15f98a701e1c0f58a4f4a0df13429", - "sha256:89bd70c9ad540febe6c28451ba225eb4e49d27f64728357f512c808002325dfa", - "sha256:93c2abea7bb69f47029b84ceac30ab46dfcfdb99b671ad850a333ff794a765e4", - "sha256:abdfa075e293d73638ece434708aa60b510dc6e70d805f57f481a0f550b25a9e", - "sha256:afeee581b50df20ef07b736e62ca612858f1fcdba96651d26ab44e3d567a4e6e", - "sha256:b51b9ef0624f4b01b846c981034c10d2e30db33f9f8be71e992f3900741f6f77", - "sha256:b66a6c15d793eda7cdad986e737775aa31b9306d588c14dd0277d2dda5546150", - "sha256:bd08254aac78b363b14acac456038ce6438bad848e02b275837930115e41ea21", - "sha256:cb257bb0c0a3176c32782a63cfab2eace7eabfa2a3b2dfd85a13700617ccaf28", - "sha256:cf5d9dcbdbe523fa665c5309cce5f144648d94a7fddbf5a40f8e0d5c9f5b596d", - "sha256:d1bc331e1706fd1809a1bc8a31205329e5b30cf5ba50461c624da267e99f6ae6", - "sha256:db5e69d08756a2fa75a42b4e433880b6187768fe1bc73d21819def893e5128c6", - "sha256:e3db646af9f6a145f0c57202f4b55d4a33f975e395e78fb7b394644c17c1a3a6", - "sha256:e9c5fd330d2fedf06051bafb996252de9b032fcb2ec03eefc9a543e56efa66d4", - "sha256:eee454d3aa3955d0c0069a0f265fea47f1e1384c35a110a95efed358eb6e1562", - "sha256:f1e9424e9aa3834ea27cc12f9c6ea8ace5da18ee60a720bb3a85b2f733f41782" + "sha256:012426a41bc9ab63bb158635aecccc7610e3eff5d31d1eb43bc099debc979d94", + "sha256:06fab248a088e439402141ea04f0fffb203723148f6ee791e9c75b3e9e82f080", + "sha256:0eef32ca3132a48e43f6a0f5a82cb508f22ce5a3d6f67a8329c81c8e226d3f6e", + "sha256:1ded4fce9cfaaf24e7a0ab51b7a87be9038ea1ace7f34b841fe3b6894c721d1c", + "sha256:2e55195bc1c6b705bfd8ad6f288b38b11b1af32f3c8289d6c50d47f950c12e76", + "sha256:2ea52bd92ab9f768cc64a4c3ef8f4b2580a17af0a5436f6126b08efbd1838371", + "sha256:36674959eed6957e61f11c912f71e78857a8d0604171dfd9ce9ad5cbf41c511c", + "sha256:384ec0463d1c2671170901994aeb6dce126de0a95ccc3976c43b0038a37329c2", + "sha256:39b70c19ec771805081578cc936bbe95336798b7edf4732ed102e7a43ec5c07a", + "sha256:400580cbd3cff6ffa6293df2278c75aef2d58d8d93d3c5614cd67981dae68ceb", + "sha256:43d4c81d5ffdff6bae58d66a3cd7f54a7acd9a0e7b18d97abb255defc09e3140", + "sha256:50a4a0ad0111cc1b71fa32dedd05fa239f7fb5a43a40663269bb5dc7877cfd28", + "sha256:603aa0706be710eea8884af807b1b3bc9fb2e49b9f4da439e76000f3b3c6ff0f", + "sha256:6149a185cece5ee78d1d196938b2a8f9d09f5a5ebfbba66969302a778d5ddd1d", + "sha256:6373751c4b6fd325606d29dd98dc2bf7092485ad20aafbfc6a177acd3b89059e", + "sha256:759e4095edc3c1b3ac031f34d9459fa781777a93ccc633a472a5468587a190ff", + "sha256:7fb43004bce0ca31d8f13a6eb5e943fa73371381e53f7074ed21a4cb786c32f8", + "sha256:811daee36a58dc79cf3d8bdd4a490e4277d0e4b7d103a001a4e73ddb48e7e6aa", + "sha256:8b5e972b43c8fc27d56550b4120fe6257fdc15f9301914380b27f74856299fea", + "sha256:99abf4f353c3d1a0c7a5f27699482c987cf663b1eac20db59b8c7b061eabd7fc", + "sha256:a0d53e51a6cb6f0d9082decb7a4cb6dfb33055308c4c44f53103c073f649af73", + "sha256:a12ff4c8ddfee61f90a1633a4c4afd3f7bcb32b11c52026c92a12e1325922d0d", + "sha256:a4646724fba402aa7504cd48b4b50e783296b5e10a524c7a6da62e4a8ac9698d", + "sha256:a76f502430dd98d7546e1ea2250a7360c065a5fdea52b2dffe8ae7180909b6f4", + "sha256:a9d17f2be3b427fbb2bce61e596cf555d6f8a56c222bd2ca148baeeb5e5c783c", + "sha256:ab83f24d5c52d60dbc8cd0528759532736b56db58adaa7b5f1f76ad551416a1e", + "sha256:aeb9ed923be74e659984e321f609b9ba54a48354bfd168d21a2b072ed1e833ea", + "sha256:c843b3f50d1ab7361ca4f0b3639bf691569493a56808a0b0c54a051d260b7dbd", + "sha256:cae865b1cae1ec2663d8ea56ef6ff185bad091a5e33ebbadd98de2cfa3fa668f", + "sha256:cc6bd4fd593cb261332568485e20a0712883cf631f6f5e8e86a52caa8b2b50ff", + "sha256:cf2402002d3d9f91c8b01e66fbb436a4ed01c6498fffed0e4c7566da1d40ee1e", + "sha256:d051ec1c64b85ecc69531e1137bb9751c6830772ee5c1c426dbcfe98ef5788d7", + "sha256:d6631f2e867676b13026e2846180e2c13c1e11289d67da08d71cacb2cd93d4aa", + "sha256:dbd18bcf4889b720ba13a27ec2f2aac1981bd41203b3a3b27ba7a33f88ae4827", + "sha256:df609c82f18c5b9f6cb97271f03315ff0dbe481a2a02e56aeb1b1a985ce38e60" ], - "markers": "python_version >= '3.7'", - "version": "==1.20.0" + "index": "pypi", + "version": "==1.19.5" }, "ocrmypdf": { "hashes": [ From d2c34fb9c7e7426479e439a7be392115136d00f0 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 17:37:47 +0100 Subject: [PATCH 046/225] update requirements.txt --- requirements.txt | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/requirements.txt b/requirements.txt index e1b76a4a0..7986f92fe 100644 --- a/requirements.txt +++ b/requirements.txt @@ -12,7 +12,7 @@ arrow==0.17.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, asgiref==3.3.1; python_version >= '3.5' async-timeout==3.0.1; python_full_version >= '3.5.3' attrs==20.3.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' -autobahn==20.12.3; python_version >= '3.6' +autobahn==21.1.1; python_version >= '3.6' automat==20.2.0 blessed==1.17.12 certifi==2020.12.5 @@ -22,6 +22,7 @@ channels==3.0.3 chardet==4.0.0; python_version >= '3.1' click==7.1.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' coloredlogs==15.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' +concurrent-log-handler==0.9.19 constantly==15.1.0 cryptography==3.3.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5' daphne==3.0.1; python_version >= '3.6' @@ -44,7 +45,6 @@ hyperlink==21.0.0 idna==2.10; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' imap-tools==0.37.0 img2pdf==0.4.0 -importlib-metadata==3.4.0; python_version < '3.8' incremental==17.5.0 inotify-simple==1.3.5; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' inotifyrecursive==0.3.5 @@ -52,7 +52,7 @@ joblib==1.0.0; python_version >= '3.6' langdetect==1.0.8 lxml==4.6.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' msgpack==1.0.2 -numpy==1.19.5; python_version >= '3.6' +numpy==1.19.5 ocrmypdf==11.6.0 pathvalidate==2.3.2 pdfminer.six==20201018; python_version >= '3.4' @@ -60,6 +60,7 @@ pdftotext==2.1.5 pikepdf==2.5.2 pillow==8.1.0 pluggy==0.13.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' +portalocker==2.2.1; python_version >= '3' psycopg2-binary==2.8.6 pyasn1-modules==0.2.8 pyasn1==0.4.8 @@ -88,7 +89,6 @@ tika==1.24 tqdm==4.56.0 twisted[tls]==20.3.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' txaio==20.12.1; python_version >= '3.6' -typing-extensions==3.7.4.3; python_version < '3.8' tzlocal==2.1 urllib3==1.26.3; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4' uvicorn[standard]==0.13.3 @@ -99,5 +99,4 @@ wcwidth==0.2.5 websockets==8.1 whitenoise==5.2.0 whoosh==2.7.4 -zipp==3.4.0; python_version >= '3.6' zope.interface==5.2.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' From 0e764416843c3788862b2b9f267bf8d370f42438 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 18:51:31 +0100 Subject: [PATCH 047/225] fix log viewer for empty logs --- src-ui/src/app/components/manage/logs/logs.component.html | 2 +- src-ui/src/app/components/manage/logs/logs.component.ts | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src-ui/src/app/components/manage/logs/logs.component.html b/src-ui/src/app/components/manage/logs/logs.component.html index 8a2a28ef8..56ffd25bb 100644 --- a/src-ui/src/app/components/manage/logs/logs.component.html +++ b/src-ui/src/app/components/manage/logs/logs.component.html @@ -11,7 +11,7 @@
    -
    +

    {{log}}

    diff --git a/src-ui/src/app/components/manage/logs/logs.component.ts b/src-ui/src/app/components/manage/logs/logs.component.ts index 0754e4253..d77fcde4a 100644 --- a/src-ui/src/app/components/manage/logs/logs.component.ts +++ b/src-ui/src/app/components/manage/logs/logs.component.ts @@ -10,8 +10,6 @@ export class LogsComponent implements OnInit { constructor(private logService: LogService) { } - @ViewChild('logContainer') private logContainer: ElementRef - logs: string[] = [] logFiles: string[] = [] @@ -31,6 +29,8 @@ export class LogsComponent implements OnInit { reloadLogs() { this.logService.get(this.activeLog).subscribe(result => { this.logs = result + }, error => { + this.logs = [] }) } From fd8c40fee7f8ad9a8fb717661b93ea763bdcad19 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 18:51:53 +0100 Subject: [PATCH 048/225] increase worker timeout --- gunicorn.conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gunicorn.conf.py b/gunicorn.conf.py index f80be548d..6fcbe6b0e 100644 --- a/gunicorn.conf.py +++ b/gunicorn.conf.py @@ -1,7 +1,7 @@ bind = '0.0.0.0:8000' workers = 2 worker_class = 'uvicorn.workers.UvicornWorker' -timeout = 60 +timeout = 120 def pre_fork(server, worker): pass From ed39e9e15f0ce8edefe5e23794b966c15e1c5caa Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 20:48:23 +0100 Subject: [PATCH 049/225] update copyright #506 --- docs/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index b2442ddc9..71abe2079 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -28,7 +28,7 @@ master_doc = 'index' # General information about the project. project = u'Paperless-ng' -copyright = u'2015, Daniel Quinn' +copyright = u'2021, Daniel Quinn, Jonas Winkler' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the From 0024c2aae49b10f3d121db031d621ce8bf2c8ee2 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 20:51:04 +0100 Subject: [PATCH 050/225] caching --- Pipfile | 1 + Pipfile.lock | 11 ++++++++++- src/paperless/settings.py | 12 +++++++++++- 3 files changed, 22 insertions(+), 2 deletions(-) diff --git a/Pipfile b/Pipfile index 504b267c4..7fcf3de26 100644 --- a/Pipfile +++ b/Pipfile @@ -51,6 +51,7 @@ channels = "~=3.0" channels-redis = "*" uvicorn = {extras = ["standard"], version = "*"} concurrent-log-handler = "*" +django-redis = "*" [dev-packages] coveralls = "*" diff --git a/Pipfile.lock b/Pipfile.lock index fafec4310..c3d57fc7d 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "aec5aa96fc4cad6d06f00d0ada0f8bd193f42874f476d39e76fc69d05c78f9f6" + "sha256": "0c2003b9d3d95d1af594f749a2740b55079551ea0ae512177ee9524bb327281e" }, "pipfile-spec": 6, "requires": {}, @@ -273,6 +273,15 @@ "index": "pypi", "version": "==1.3.4" }, + "django-redis": { + "hashes": [ + "sha256:1133b26b75baa3664164c3f44b9d5d133d1b8de45d94d79f38d1adc5b1d502e5", + "sha256:306589c7021e6468b2656edc89f62b8ba67e8d5a1c8877e2688042263daa7a63", + "sha256:f2b25b62cc95b63b7059aaf8e81710e7eea94678e545d31c46e47a6f4af99e56" + ], + "index": "pypi", + "version": "==4.12.1" + }, "djangorestframework": { "hashes": [ "sha256:0209bafcb7b5010fdfec784034f059d512256424de2a0f084cb82b096d6dd6a7", diff --git a/src/paperless/settings.py b/src/paperless/settings.py index 718738f71..60e7b45bd 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -169,6 +169,16 @@ CHANNEL_LAYERS = { }, } +CACHES = { + "default": { + "BACKEND": "django_redis.cache.RedisCache", + "LOCATION": os.getenv("PAPERLESS_REDIS", "redis://localhost:6379"), + "OPTIONS": { + "CLIENT_CLASS": "django_redis.client.DefaultClient", + } + } +} + ############################################################################### # Security # ############################################################################### @@ -398,7 +408,7 @@ Q_CLUSTER = { 'name': 'paperless', 'catch_up': False, 'workers': TASK_WORKERS, - 'redis': os.getenv("PAPERLESS_REDIS", "redis://localhost:6379") + 'django_redis': 'default' } From ffe96c8fffa8ca22c5911ad82aea48aa77d13cc1 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 20:54:58 +0100 Subject: [PATCH 051/225] classifier caching --- src/documents/classifier.py | 76 ++++++++++++-------------- src/documents/tasks.py | 2 +- src/documents/tests/test_classifier.py | 63 ++++++++++----------- src/documents/tests/test_tasks.py | 3 +- 4 files changed, 68 insertions(+), 76 deletions(-) diff --git a/src/documents/classifier.py b/src/documents/classifier.py index 2acebe04a..5151d453c 100755 --- a/src/documents/classifier.py +++ b/src/documents/classifier.py @@ -5,6 +5,7 @@ import pickle import re from django.conf import settings +from django.core.cache import cache from documents.models import Document, MatchingModel @@ -30,22 +31,28 @@ def load_classifier(): ) return None - try: + version = os.stat(settings.MODEL_FILE).st_mtime + + classifier = cache.get("paperless-classifier", version=version) + + if not classifier: classifier = DocumentClassifier() - classifier.reload() - except (EOFError, IncompatibleClassifierVersionError) as e: - # there's something wrong with the model file. - logger.error( - f"Unrecoverable error while loading document " - f"classification model: {str(e)}, deleting model file." - ) - os.unlink(settings.MODEL_FILE) - classifier = None - except OSError as e: - logger.error( - f"Error while loading document classification model: {str(e)}" - ) - classifier = None + try: + classifier.load() + cache.set("paperless-classifier", classifier, version=version) + except (EOFError, IncompatibleClassifierVersionError) as e: + # there's something wrong with the model file. + logger.error( + f"Unrecoverable error while loading document " + f"classification model: {str(e)}, deleting model file." + ) + os.unlink(settings.MODEL_FILE) + classifier = None + except OSError as e: + logger.error( + f"Error while loading document classification model: {str(e)}" + ) + classifier = None return classifier @@ -55,10 +62,6 @@ class DocumentClassifier(object): FORMAT_VERSION = 6 def __init__(self): - # mtime of the model file on disk. used to prevent reloading when - # nothing has changed. - self.classifier_version = 0 - # hash of the training data. used to prevent re-training when the # training data has not changed. self.data_hash = None @@ -69,30 +72,23 @@ class DocumentClassifier(object): self.correspondent_classifier = None self.document_type_classifier = None - def reload(self): - if os.path.getmtime(settings.MODEL_FILE) > self.classifier_version: - with open(settings.MODEL_FILE, "rb") as f: - schema_version = pickle.load(f) + def load(self): + with open(settings.MODEL_FILE, "rb") as f: + schema_version = pickle.load(f) - if schema_version != self.FORMAT_VERSION: - raise IncompatibleClassifierVersionError( - "Cannor load classifier, incompatible versions.") - else: - if self.classifier_version > 0: - # Don't be confused by this check. It's simply here - # so that we wont log anything on initial reload. - logger.info("Classifier updated on disk, " - "reloading classifier models") - self.data_hash = pickle.load(f) - self.data_vectorizer = pickle.load(f) - self.tags_binarizer = pickle.load(f) + if schema_version != self.FORMAT_VERSION: + raise IncompatibleClassifierVersionError( + "Cannor load classifier, incompatible versions.") + else: + self.data_hash = pickle.load(f) + self.data_vectorizer = pickle.load(f) + self.tags_binarizer = pickle.load(f) - self.tags_classifier = pickle.load(f) - self.correspondent_classifier = pickle.load(f) - self.document_type_classifier = pickle.load(f) - self.classifier_version = os.path.getmtime(settings.MODEL_FILE) + self.tags_classifier = pickle.load(f) + self.correspondent_classifier = pickle.load(f) + self.document_type_classifier = pickle.load(f) - def save_classifier(self): + def save(self): with open(settings.MODEL_FILE, "wb") as f: pickle.dump(self.FORMAT_VERSION, f) pickle.dump(self.data_hash, f) diff --git a/src/documents/tasks.py b/src/documents/tasks.py index 8c7d91585..f74a3d420 100644 --- a/src/documents/tasks.py +++ b/src/documents/tasks.py @@ -52,7 +52,7 @@ def train_classifier(): "Saving updated classifier model to {}...".format( settings.MODEL_FILE) ) - classifier.save_classifier() + classifier.save() else: logger.debug( "Training data unchanged." diff --git a/src/documents/tests/test_classifier.py b/src/documents/tests/test_classifier.py index 14673ae65..1efe564d1 100644 --- a/src/documents/tests/test_classifier.py +++ b/src/documents/tests/test_classifier.py @@ -1,7 +1,6 @@ import os import tempfile from pathlib import Path -from time import sleep from unittest import mock from django.conf import settings @@ -85,37 +84,19 @@ class TestClassifier(DirectoriesMixin, TestCase): self.assertTrue(self.classifier.train()) self.assertFalse(self.classifier.train()) - self.classifier.save_classifier() + self.classifier.save() classifier2 = DocumentClassifier() current_ver = DocumentClassifier.FORMAT_VERSION with mock.patch("documents.classifier.DocumentClassifier.FORMAT_VERSION", current_ver+1): # assure that we won't load old classifiers. - self.assertRaises(IncompatibleClassifierVersionError, classifier2.reload) + self.assertRaises(IncompatibleClassifierVersionError, classifier2.load) - self.classifier.save_classifier() + self.classifier.save() # assure that we can load the classifier after saving it. - classifier2.reload() - - def testReload(self): - - self.generate_test_data() - self.assertTrue(self.classifier.train()) - self.classifier.save_classifier() - - classifier2 = DocumentClassifier() - classifier2.reload() - v1 = classifier2.classifier_version - - # change the classifier after some time. - sleep(1) - self.classifier.save_classifier() - - classifier2.reload() - v2 = classifier2.classifier_version - self.assertNotEqual(v1, v2) + classifier2.load() @override_settings(DATA_DIR=tempfile.mkdtemp()) def testSaveClassifier(self): @@ -124,10 +105,10 @@ class TestClassifier(DirectoriesMixin, TestCase): self.classifier.train() - self.classifier.save_classifier() + self.classifier.save() new_classifier = DocumentClassifier() - new_classifier.reload() + new_classifier.load() self.assertFalse(new_classifier.train()) @override_settings(MODEL_FILE=os.path.join(os.path.dirname(__file__), "data", "model.pickle")) @@ -135,7 +116,7 @@ class TestClassifier(DirectoriesMixin, TestCase): self.generate_test_data() new_classifier = DocumentClassifier() - new_classifier.reload() + new_classifier.load() self.assertCountEqual(new_classifier.predict_tags(self.doc2.content), [45, 12]) @@ -252,25 +233,39 @@ class TestClassifier(DirectoriesMixin, TestCase): self.assertFalse(os.path.exists(settings.MODEL_FILE)) self.assertIsNone(load_classifier()) - @mock.patch("documents.classifier.DocumentClassifier.reload") - def test_load_classifier(self, reload): + @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}) + @mock.patch("documents.classifier.DocumentClassifier.load") + def test_load_classifier(self, load): Path(settings.MODEL_FILE).touch() self.assertIsNotNone(load_classifier()) + load.assert_called_once() - @mock.patch("documents.classifier.DocumentClassifier.reload") - def test_load_classifier_incompatible_version(self, reload): + @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}) + @override_settings(MODEL_FILE=os.path.join(os.path.dirname(__file__), "data", "model.pickle")) + def test_load_classifier_cached(self): + classifier = load_classifier() + self.assertIsNotNone(classifier) + + with mock.patch("documents.classifier.DocumentClassifier.load") as load: + classifier2 = load_classifier() + load.assert_not_called() + + @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}) + @mock.patch("documents.classifier.DocumentClassifier.load") + def test_load_classifier_incompatible_version(self, load): Path(settings.MODEL_FILE).touch() self.assertTrue(os.path.exists(settings.MODEL_FILE)) - reload.side_effect = IncompatibleClassifierVersionError() + load.side_effect = IncompatibleClassifierVersionError() self.assertIsNone(load_classifier()) self.assertFalse(os.path.exists(settings.MODEL_FILE)) - @mock.patch("documents.classifier.DocumentClassifier.reload") - def test_load_classifier_os_error(self, reload): + @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}) + @mock.patch("documents.classifier.DocumentClassifier.load") + def test_load_classifier_os_error(self, load): Path(settings.MODEL_FILE).touch() self.assertTrue(os.path.exists(settings.MODEL_FILE)) - reload.side_effect = OSError() + load.side_effect = OSError() self.assertIsNone(load_classifier()) self.assertTrue(os.path.exists(settings.MODEL_FILE)) diff --git a/src/documents/tests/test_tasks.py b/src/documents/tests/test_tasks.py index d008f995a..ed280441f 100644 --- a/src/documents/tests/test_tasks.py +++ b/src/documents/tests/test_tasks.py @@ -2,7 +2,7 @@ import os from unittest import mock from django.conf import settings -from django.test import TestCase +from django.test import TestCase, override_settings from django.utils import timezone from documents import tasks @@ -52,6 +52,7 @@ class TestTasks(DirectoriesMixin, TestCase): load_classifier.assert_called_once() self.assertFalse(os.path.isfile(settings.MODEL_FILE)) + @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}) def test_train_classifier(self): c = Correspondent.objects.create(matching_algorithm=Tag.MATCH_AUTO, name="test") doc = Document.objects.create(correspondent=c, content="test", title="test") From 7702f5012b5eb2ed311d8abcb5d38b5ed8be1c85 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 21:03:32 +0100 Subject: [PATCH 052/225] classifier cache timeout --- src/documents/classifier.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/documents/classifier.py b/src/documents/classifier.py index 5151d453c..81f860314 100755 --- a/src/documents/classifier.py +++ b/src/documents/classifier.py @@ -39,7 +39,8 @@ def load_classifier(): classifier = DocumentClassifier() try: classifier.load() - cache.set("paperless-classifier", classifier, version=version) + cache.set("paperless-classifier", classifier, + version=version, timeout=86400) except (EOFError, IncompatibleClassifierVersionError) as e: # there's something wrong with the model file. logger.error( From 656b4708ac448de629a439e26199960234c16f4a Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 6 Feb 2021 21:16:06 +0100 Subject: [PATCH 053/225] update requirements.txt --- requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/requirements.txt b/requirements.txt index 7986f92fe..5379725b5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -32,6 +32,7 @@ django-extensions==3.1.0 django-filter==2.4.0 django-picklefield==3.0.1; python_version >= '3' django-q==1.3.4 +django-redis==4.12.1 django==3.1.6 djangorestframework==3.12.2 filelock==3.0.12 From 40f3e8e45af8c6548c94a14ce5d8df599542cc70 Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Sat, 6 Feb 2021 23:26:44 +0100 Subject: [PATCH 054/225] Update README.md --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index 80b9c85fe..7ef647561 100644 --- a/README.md +++ b/README.md @@ -11,10 +11,6 @@ Paperless-ng is a fork of the original project, adding a new interface and many other changes under the hood. For a detailed list of changes, have a look at the [change log](https://paperless-ng.readthedocs.io/en/latest/changelog.html) in the documentation. -# Survey - -If you already used Paperless-ng for a bit, would like to give some anonymous feedback, and help me decide on what to focus on next: I've created a survey, [see here](https://github.com/jonaswinkler/paperless-ng/issues/402). Thank you! - # How it Works Paperless does not control your scanner, it only helps you deal with what your scanner produces. From bcee17c3ac342341dafd76733a1d8009bf7645e6 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 7 Feb 2021 00:14:24 +0100 Subject: [PATCH 055/225] github actions: build release tags --- .github/workflows/ci.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5eaca123a..08630e176 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,6 +2,7 @@ name: ci on: push: + tags: ng-* branches-ignore: - 'translations**' pull_request: From f555bb95aef3dd42c66091797643851c0d02a6ce Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 7 Feb 2021 00:49:53 +0100 Subject: [PATCH 056/225] possible fix for the ansible roles --- ansible/molecule/default/verify.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ansible/molecule/default/verify.yml b/ansible/molecule/default/verify.yml index 185840783..7646b223e 100644 --- a/ansible/molecule/default/verify.yml +++ b/ansible/molecule/default/verify.yml @@ -38,7 +38,7 @@ - name: verify uploaded document has been accepted uri: - url: "http://{{ paperlessng_listen_address }}:{{ paperlessng_listen_port }}/api/logs/" + url: "http://{{ paperlessng_listen_address }}:{{ paperlessng_listen_port }}/api/logs/paperless/" headers: Authorization: 'Basic {{ (paperlessng_superuser_name + ":" + paperlessng_superuser_password) | b64encode }}' return_content: yes @@ -51,7 +51,7 @@ - name: verify uploaded document has been consumed uri: - url: "http://{{ paperlessng_listen_address }}:{{ paperlessng_listen_port }}/api/logs/" + url: "http://{{ paperlessng_listen_address }}:{{ paperlessng_listen_port }}/api/logs/paperless/" headers: Authorization: 'Basic {{ (paperlessng_superuser_name + ":" + paperlessng_superuser_password) | b64encode }}' return_content: yes From 731418349ff5b812bf5fc67b7ac0a63a6058ab1f Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 7 Feb 2021 18:23:54 +0100 Subject: [PATCH 057/225] added a test case that replicates #511 --- src/documents/sanity_checker.py | 12 ++++++--- src/documents/tests/samples/simple.png | Bin 0 -> 7913 bytes src/documents/tests/test_consumer.py | 35 +++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 4 deletions(-) create mode 100644 src/documents/tests/samples/simple.png diff --git a/src/documents/sanity_checker.py b/src/documents/sanity_checker.py index bc0b689d4..b8fd73f98 100644 --- a/src/documents/sanity_checker.py +++ b/src/documents/sanity_checker.py @@ -56,7 +56,8 @@ def check_sanity(): messages.append(SanityError( f"Thumbnail of document {doc.pk} does not exist.")) else: - present_files.remove(os.path.normpath(doc.thumbnail_path)) + if os.path.normpath(doc.thumbnail_path) in present_files: + present_files.remove(os.path.normpath(doc.thumbnail_path)) try: with doc.thumbnail_file as f: f.read() @@ -71,7 +72,8 @@ def check_sanity(): messages.append(SanityError( f"Original of document {doc.pk} does not exist.")) else: - present_files.remove(os.path.normpath(doc.source_path)) + if os.path.normpath(doc.source_path) in present_files: + present_files.remove(os.path.normpath(doc.source_path)) try: with doc.source_file as f: checksum = hashlib.md5(f.read()).hexdigest() @@ -92,7 +94,8 @@ def check_sanity(): f"Archived version of document {doc.pk} does not exist." )) else: - present_files.remove(os.path.normpath(doc.archive_path)) + if os.path.normpath(doc.archive_path) in present_files: + present_files.remove(os.path.normpath(doc.archive_path)) try: with doc.archive_file as f: checksum = hashlib.md5(f.read()).hexdigest() @@ -103,7 +106,8 @@ def check_sanity(): else: if not checksum == doc.archive_checksum: messages.append(SanityError( - f"Checksum mismatch of archive {doc.pk}. " + f"Checksum mismatch of archived document " + f"{doc.pk}. " f"Stored: {doc.checksum}, actual: {checksum}." )) diff --git a/src/documents/tests/samples/simple.png b/src/documents/tests/samples/simple.png new file mode 100644 index 0000000000000000000000000000000000000000..a3a76840103812c371dd78f2e8bc704ea3eec1ba GIT binary patch literal 7913 zcmd^k_al{m{5A=ttdc?qk!-Ru%a%e&PWC2yk7I^1LXzzyA!P5pl3fZN9eZVd zU+43E{)y*#{BSt8`@Y}z{eEAs>vdh%J3>=knVgKCjDUcETvbIui-6z^G5j5Wjs!lL zq@Mf2ud{A)sygT3$M2j)IDDsbSJZRYcCvE!GIg~iuy%BEu;g_!ceS*1bhB}CUn6Rf zflHC93ioxq({U3>zlYW*r0W_S5x?K0-C$?sVP$=B?z~5SZy2pZ9)|N-lDk@;Zi7uh z;=~e{SySnlr|r`4XY}wVUd$&&d8$7Fb~=Cf%v|F_EnHLP_g?k$JJJl|5)x)}-%<#IA|uI;P7c?SW&9IV- z*E}MgNqTj4bz!80+1}p1kdcf!rS;=0N*HjU3F%eoFivY^WaRAP5?oPnX9b7L)Z{d= zvx|E7j%MX|opx&H!~|nW(=Pq5=OSF(+>a56h{L_LnUmu~m`AyQ0Gaaga_4`WmTCbs z1eBg}x_ifalloF5+QNzA5#inSJgGE|+k4{g!kU>iT zql}D473jWD7& zWn`G9r>9HH%I=pKHLYiy9(DZ@O#WE*b&Tro+??4Owy8Gow7E(?zd|jEk3~cTjWMM{ zDEj(J^rIxLaRT&=QLDFYCY$}6Dho6@r$OTKU4ub+qj4?9R^4d0XYa$+OLF^XdTPkLp|gde5uq z@&EGlOk|%@S6Aoc}4FjAAguYKJ@bi*&O-e*CCAuamFC1<#c- z>!z!#`?0aH5la%v&CMMmml=dYEv9|m-}kQAuIfUw-Tj!!@&r{-TYGKNZ;RX$hY17N z{ciNR41?m=V__hx%%trFeV6!tm+VPq%}|xoBvbq?4N>va-rfiHGr2$tEgF+y0XLPM=gTX^5Tw{`9Nz_zNS2h!9%dXDYKCk5Z>lUfBGy zXzg5e7C}L3*^_-XK))t%LzrB!ZqYN-)*!-bvH?=>k*jyJ%6?haJ=M`M0X&wAM@2<# zjI^9Q(JgVC@86$cKD801)dsY(;%{@bP6U%3+}J3UhknjULndrDe34PYJ*cHc7Anqe zVt1m!tIV{W?p^W6h>E-o$;CnrjNe*SIT_|>?$xMXn`R>1Ldw5N-Upr{k} zHFMw*wXLmfcTdlKAD_DftsNc2j*gD&TU+u;e1=nNYq7Pp;`o8OVi@JMwBD*C$->U8 zwVXEhFT+*k1^J(&A|vHm0*01Ba!o65}y_#!sx7xMS7g+a3)-IXg> zY|x*tz^V&4VWm)RrU9qNUMr(zL_mQm8XB*(@sJ*A5htvHg{5W0;!tkXr$=eMb6LkWBgLp%-<{7`QGnzl zYFb(j0|Nt#p^Nmw-6lL^V`HpRQU;&b78W?`J(f90D6Vq?1BVqA6>YySDRDx{c$r65 z^#070w{(G3FbE9|4F^t%1XAe3BB#TyDnPeE%_Y5m&xfgTaw=ukk%qf(S7&PFD;OB0 zYjV5;!ubsAyD^k!d2<8$TDC2KLm`$)`fCs&k-{qqMhlz| za2$Wj(cs@&z_I!C>@54E5>3v~mX?-l;?5ryhH_)=o1er+UF9cZ2^II-FTE!6{N4KQ zZeo*@on3*K5h=xWD)X)w^uH5=@C5Fsy-S~UxHdNX_@f*r>U$0@6O;v>29){kJkL5t zx3%5JcanEWNl`y!4j+bwh0)V*~3HRS^+$%CRc)lgGPpoA+(0C*(WWqzS{LY z=F4Rd$e@vhH8nNCsaK#`uySxHJxmjQ^e|Oeb$bb|mb)c&xKcqfUR+p6LqS2o;b7m_ z+pFO0T&m+Wn61jn&#$4YoUaqvax}|tH8~0-S@8D7!h+2PJZ>Zq&zQAdrA^$*VTx>E z0BX()hY^WOPUgIH`Ep08X}g63KcZa0mtRouu|2jCVSTW<=?XApc{Bra{PykJ&}T)7 zOW8R&J-RP|rmg^!j@T9y_Z@e~v1t`TJJgc=o+g&$AM^FALR@;fLGCO^C>6BEcf2KW zY|4?qAB#n4C=~Db4SF>)Y`~!d!mQHopX*~Cotp*mad9k4u}r+^in225jg1YG@h1gn zGiW5`wnl-+9xhvRk5Kiw>ZnU4EhqKrTAGGCGTPc&niD@{PwfS6=Q$o9?3CB{_xBI( z(b3S{e0hhCx#|LxnM2rr_>jhKq&NXhPEL+qJcU9QJTR2*XDY}h2+|p|L@ZrW|&@$!|pdu<9?HSb4r?leb_NFrEgXKJB%kKS+)YC6T(K?CEqWag zgE;`|wal>o^EE;#S5V~kXuU^4A@1~KuZ3hBo+81Y^l%qfQ0D08*I4Lat{O@uhvClT zza9Sib>S1OQtKZ~@$oNUIS~_Hlm1UFtgM!b8XU$&QRMz0$M$EFj1?Rmi`9>hj{F2Y zSM(6aK|w+E=8w4&K7yRnnxJM=>b^9&r1bh4i<*klWOGrCk;8DG-Ed(X+QZqIDl9C_ zcX@7pzHcdpL9DpbQ?*5T?X@7=#c024moKwP4)cY!X3D?%CNv4_q2$X%O>MgVmClVZ zEAZNIOa;2iuEmk^{Rl zv9e04su6`TRY?u`z_c54X-*Gj6M6MnSyF!uf_1wyM*UOba2GXCSO08;dl%2TqB9oM!Zn4Bb zPVwi-w6;;iUl$jb^5{@@#GU@TZgVCbkyl8Tr8d4^E-%j^E&N>Wvt>0_R2Pk^-2)t` z{kT9zw%*6z(v5ZEIX3T0n7U$^~ z{pG|#=z^zxd}1OyBo5^4w*@$WlXRP1W?r@(d`}gThoZF92A12_zG`{%x=<;(*ia~L;WqNWaRImS7`(g^1_SM47h z^uFyu?yb_Hl#qtGO?zbC&N~V_3g7i)od&Zb>|;CgwR691GVJW^BqT70nRYX9aS%VN zESW&paBrIG{3zx(s10(N>Fhai5uu4@SGSY&TuI*kjYgxVSSas#QNG$WlxApmEh#K~ z$9^X!CgvOkg{c0aa?JI(_mbk`3-;%p4>lq9=*J0zpon`mX_YYBG=j` zCML2EhRy8kxW{vxdVST94{7~}Y;KhV?zc0)uGViVNlHjau}pT*He=s1(K2Dd5OL!` zxC8_QB;T>7=zlM5my_oXcC6_3ty{|UvIIfy567u+hAS9`qt4RbcZ7s2PJc>xuGmxu z7edctKUBt4JLtzNGow$E431+y5I7(0yCCX7D|&=)Z@5l>Ax4Z{_earJcR8-J+R;h- z3Y(dkt)rhBtCn1$p#cap0bv6bl5(i0heuq#MSncXbG7m<*qFHz9w7YvZ%xJpVEp}0j{b>IlM%j#XG^ zfgqCbc%Y_MX4XLl0!?L})b@5_jCPT(`L|?2{3E%#xj|zJC%!;C1zk%sU#GjjU&+df z11vzU7bik(sM$b3K!A#vn7GP!L6WXGb1(L@X;8s0D=A;|C11YSU4dTo_)v`Yp_}IS z%A^UX1-pq6c{#a48foc^9K^am>)fnCgk2JI{N4Iq8o9nyP_lG{ybB`T>U*kX&N=2% zopfkr9ndOd^BaNld5jvb1qTO{Z_cS<8mbG7Q*9{7s{(}`Fgm)T^H{Q{PoE;{s*$0U z=J~*Bp5wz0su{8lR+oRR3pa|ank!^#nygQ?si>&50+$E|9C?o&GKku@;>K%=XeLez z*2D`YZ|1;&R!!fEWNb1b-lwJM^KaSO9Lpan38OVG7INd#V$pMdD+dR)5SIoD5qHLN z3NQ}FKqV|$en)zrOSIeDvOfglJ^Fk`V9etzCDjt}yVs?>e?~rD6!J=+ot-VOSY~K1 zR`T-l8dLRrkfSx9Tw5m+dXgz^^GHeQGj-#YW)8Ot|w)RGEm&Lha;&s@0czVJqsA!bRw>`cA2<%pPCR|;qx{meAi z9&Py#P1E_yopplSl=v}U9)eY~VsxxNXnm59kRXxNj#fB)+YV|3F$NOF6L$pq$pu=q zgu+44ZY{y{s``EhQDMxn6l4Jp;@;#4+%A0s%(T>$R4acY`0({{4BZv))yVxyZ^`MOXGB_*8c8Wd_#+Fy?O)z#wY( zEx6ykdsp=b`gzqrW*9@(n>SQrMrDfq$W#l=--x3}F-f8trQt0b7?pWvLV}}3jlj3O zNgA@pG?V^&hNwSirW~QWck9$~A_|_Kp2-5HFLY?-OrXHYJ=+oElP(EjSZ^YLl(LG- zIWRddfnA4{y`#^-GD#qJc>+%US%-Ll*v*-5YinEI+bdZsb=MP8he91&#TFLw@EA86 zDO2&((Q)YpcFW&?_IhF>X*K5#C>t8PDfwz>uB{KJH6)md<=T;-0#CLwM21W9@>>7)#OIK|3ki{r zM(X?d`~0+R(;o!`jltG6ThYo!*z7jt}R4WbjKO zw!o%c6Sdb1kun|X=}{FGM}ts$laOHPoSG4EaK7MaNn$UwN})xH)Y#7?cRpr5$oT?R z=S@n=ZDdn+_E``VkRu(-PrOq&6wfG|38>0994YQ@j9s5zPDB^uFmJI8;B1!Ar+3hW(A9kjfHoWWs0c2N22*86N=Zri z?%lf^w{DS+6dTGbDUob$ZMm%ezT4!IY+k3VpwPXlE@s(Fb6wK&BBWZJJUj}(ir^Rp zUH^8!dZvr#EO-G376_->9#lf336G5Y0^Ski7(S@5u(qC=Y6}7X9*^r3*+!Am37rM0 z{G||4aU--f{jQty&e9WnuNahdV>wH4T|B5}GC;q+sxBmi2>fj*G)(m9E`w8JFjR)g zY;OW6#J>_=s|GxV^)#T_6`h?0z_yy0n}@u3aprdu@@h&-in#0Fb8_B?fo;CxF_IG}MrZbwM*6&|=zh%yxL5E?d5aY%Bs= zS0wb)k-IN#hd$B0HidB|hJd@ny|6yY;*ydKxE6xYnwpw@$gpyT@YV=yzyh|Zp-o)W z@wZ;CB@`zwH65Mp_Li@&ZzNb9L7Sg9#+Ds0mDKRYe}Fc9$0y6oUqSv9k9S)GAeE-q z*7(58KPQ3YKrEUEB&W_09;h7ut?u)DeM;ppb4mfO+{@Pl>K^DBd84gp0 zzgzhTxVi;QRY9__?OO<R46EGKlI<6Z3YFODRN&=D2&7tw0=L`{%Ws48Nm*HVuh;Abg6;_L1 zeNK*ipu<+so4dKY(_G~@dbgDO>C@6LI@y56p5w*G=4feR5JLFPn;myXNl)SZX1UCJ z5a6`U{KkwJ)YuM@!2AgwtmUi5bHJX|y?gh92E+>}8{<<`BjGWpfxg5&mS`X+3Wc~I z&kJ~w3@1X)gU2@zu94@Zp>`V{7u~U17d9yGGM5=8s7?~FL%<>Ugb`?#ih+R}rBeeT z0&02GPLiis2Jo>km~#Q2BSL+ucr9Hxf4&X!i5;xt$nY>Nv)?)))_aB=pT8n^SK6Ff zj$h)_OC7G-x;oV^E;is1AgiYK=2W*jN!SJX@m^Z5(Ob}S_4oN4tq71tnGj<@F4u*( zdaFEo`bGLRoscD$d94}wf#y_s^r!^y+GHMH1<~=n(B_ zdJ(dLfdROl6!u@DXS!l%K-1OwABsY?oT(rdnbRA zj04&rNO7L;X9T{;g1L3}@=}ISgzEbB*8l~4jxT~uzbzzm2?$=V%#0ZJY(D64P0!7} zgdy-^*LyC-I@_{HE5FovS}vsQ-~N6p)U;=~lKze)iEdF1*lD{$V}HZ>m6g{r{s$i* zq2fT76oWs(Q`p=1_%gc@zC}rXPR=0ULx{&=+bI+t7|&4i=@+06xGxSW&WrweIfq+b zwp({hISAMTn#NDdN5v>-h$`|mJeP;b!5MVIdrjeRylBb`7upAN)F3WD^DZGFGw}3e zkg2XKE}E*5l%1Md8Vfs4ntFORXof+Nv;Su|uSDkmvpvB7-uwTZoy{X=g#M$ne+jJV Ruvbl>s;I6|ENAlKe*j2SP+b53 literal 0 HcmV?d00001 diff --git a/src/documents/tests/test_consumer.py b/src/documents/tests/test_consumer.py index a6f0cc55a..8ead1ea41 100644 --- a/src/documents/tests/test_consumer.py +++ b/src/documents/tests/test_consumer.py @@ -5,12 +5,14 @@ import tempfile from unittest import mock from unittest.mock import MagicMock +from django.conf import settings from django.test import TestCase, override_settings from .utils import DirectoriesMixin from ..consumer import Consumer, ConsumerError from ..models import FileInfo, Tag, Correspondent, DocumentType, Document from ..parsers import DocumentParser, ParseError +from ..tasks import sanity_check class TestAttributes(TestCase): @@ -181,6 +183,24 @@ class DummyParser(DocumentParser): self.text = "The Text" +class CopyParser(DocumentParser): + + def get_thumbnail(self, document_path, mime_type): + return self.fake_thumb + + def get_optimised_thumbnail(self, document_path, mime_type): + return self.fake_thumb + + def __init__(self, logging_group, progress_callback=None): + super(CopyParser, self).__init__(logging_group, progress_callback) + _, self.fake_thumb = tempfile.mkstemp(suffix=".png", dir=self.tempdir) + + def parse(self, document_path, mime_type, file_name=None): + self.text = "The text" + self.archive_path = os.path.join(self.tempdir, "archive.pdf") + shutil.copy(document_path, self.archive_path) + + class FaultyParser(DocumentParser): def get_thumbnail(self, document_path, mime_type): @@ -203,6 +223,8 @@ def fake_magic_from_file(file, mime=False): if mime: if os.path.splitext(file)[1] == ".pdf": return "application/pdf" + elif os.path.splitext(file)[1] == ".png": + return "image/png" else: return "unknown" else: @@ -516,6 +538,19 @@ class TestConsumer(DirectoriesMixin, TestCase): self._assert_first_last_send_progress(last_status="FAILED") + @mock.patch("documents.parsers.document_consumer_declaration.send") + def test_similar_filenames(self, m): + shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"), os.path.join(settings.CONSUMPTION_DIR, "simple.pdf")) + shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.png"), os.path.join(settings.CONSUMPTION_DIR, "simple.png")) + m.return_value = [(None, { + "parser": CopyParser, + "mime_types": {"application/pdf": ".pdf", "image/png": ".zip"}, + "weight": 0 + })] + doc1 = self.consumer.try_consume_file(os.path.join(settings.CONSUMPTION_DIR, "simple.png")) + doc2 = self.consumer.try_consume_file(os.path.join(settings.CONSUMPTION_DIR, "simple.pdf")) + + sanity_check() class PreConsumeTestCase(TestCase): From 9b64eebd108b1e27238895353298d97425d1fc88 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 7 Feb 2021 18:26:03 +0100 Subject: [PATCH 058/225] revert commit --- src/documents/sanity_checker.py | 12 +++------ src/documents/tests/samples/simple.png | Bin 7913 -> 0 bytes src/documents/tests/test_consumer.py | 35 ------------------------- 3 files changed, 4 insertions(+), 43 deletions(-) delete mode 100644 src/documents/tests/samples/simple.png diff --git a/src/documents/sanity_checker.py b/src/documents/sanity_checker.py index b8fd73f98..bc0b689d4 100644 --- a/src/documents/sanity_checker.py +++ b/src/documents/sanity_checker.py @@ -56,8 +56,7 @@ def check_sanity(): messages.append(SanityError( f"Thumbnail of document {doc.pk} does not exist.")) else: - if os.path.normpath(doc.thumbnail_path) in present_files: - present_files.remove(os.path.normpath(doc.thumbnail_path)) + present_files.remove(os.path.normpath(doc.thumbnail_path)) try: with doc.thumbnail_file as f: f.read() @@ -72,8 +71,7 @@ def check_sanity(): messages.append(SanityError( f"Original of document {doc.pk} does not exist.")) else: - if os.path.normpath(doc.source_path) in present_files: - present_files.remove(os.path.normpath(doc.source_path)) + present_files.remove(os.path.normpath(doc.source_path)) try: with doc.source_file as f: checksum = hashlib.md5(f.read()).hexdigest() @@ -94,8 +92,7 @@ def check_sanity(): f"Archived version of document {doc.pk} does not exist." )) else: - if os.path.normpath(doc.archive_path) in present_files: - present_files.remove(os.path.normpath(doc.archive_path)) + present_files.remove(os.path.normpath(doc.archive_path)) try: with doc.archive_file as f: checksum = hashlib.md5(f.read()).hexdigest() @@ -106,8 +103,7 @@ def check_sanity(): else: if not checksum == doc.archive_checksum: messages.append(SanityError( - f"Checksum mismatch of archived document " - f"{doc.pk}. " + f"Checksum mismatch of archive {doc.pk}. " f"Stored: {doc.checksum}, actual: {checksum}." )) diff --git a/src/documents/tests/samples/simple.png b/src/documents/tests/samples/simple.png deleted file mode 100644 index a3a76840103812c371dd78f2e8bc704ea3eec1ba..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 7913 zcmd^k_al{m{5A=ttdc?qk!-Ru%a%e&PWC2yk7I^1LXzzyA!P5pl3fZN9eZVd zU+43E{)y*#{BSt8`@Y}z{eEAs>vdh%J3>=knVgKCjDUcETvbIui-6z^G5j5Wjs!lL zq@Mf2ud{A)sygT3$M2j)IDDsbSJZRYcCvE!GIg~iuy%BEu;g_!ceS*1bhB}CUn6Rf zflHC93ioxq({U3>zlYW*r0W_S5x?K0-C$?sVP$=B?z~5SZy2pZ9)|N-lDk@;Zi7uh z;=~e{SySnlr|r`4XY}wVUd$&&d8$7Fb~=Cf%v|F_EnHLP_g?k$JJJl|5)x)}-%<#IA|uI;P7c?SW&9IV- z*E}MgNqTj4bz!80+1}p1kdcf!rS;=0N*HjU3F%eoFivY^WaRAP5?oPnX9b7L)Z{d= zvx|E7j%MX|opx&H!~|nW(=Pq5=OSF(+>a56h{L_LnUmu~m`AyQ0Gaaga_4`WmTCbs z1eBg}x_ifalloF5+QNzA5#inSJgGE|+k4{g!kU>iT zql}D473jWD7& zWn`G9r>9HH%I=pKHLYiy9(DZ@O#WE*b&Tro+??4Owy8Gow7E(?zd|jEk3~cTjWMM{ zDEj(J^rIxLaRT&=QLDFYCY$}6Dho6@r$OTKU4ub+qj4?9R^4d0XYa$+OLF^XdTPkLp|gde5uq z@&EGlOk|%@S6Aoc}4FjAAguYKJ@bi*&O-e*CCAuamFC1<#c- z>!z!#`?0aH5la%v&CMMmml=dYEv9|m-}kQAuIfUw-Tj!!@&r{-TYGKNZ;RX$hY17N z{ciNR41?m=V__hx%%trFeV6!tm+VPq%}|xoBvbq?4N>va-rfiHGr2$tEgF+y0XLPM=gTX^5Tw{`9Nz_zNS2h!9%dXDYKCk5Z>lUfBGy zXzg5e7C}L3*^_-XK))t%LzrB!ZqYN-)*!-bvH?=>k*jyJ%6?haJ=M`M0X&wAM@2<# zjI^9Q(JgVC@86$cKD801)dsY(;%{@bP6U%3+}J3UhknjULndrDe34PYJ*cHc7Anqe zVt1m!tIV{W?p^W6h>E-o$;CnrjNe*SIT_|>?$xMXn`R>1Ldw5N-Upr{k} zHFMw*wXLmfcTdlKAD_DftsNc2j*gD&TU+u;e1=nNYq7Pp;`o8OVi@JMwBD*C$->U8 zwVXEhFT+*k1^J(&A|vHm0*01Ba!o65}y_#!sx7xMS7g+a3)-IXg> zY|x*tz^V&4VWm)RrU9qNUMr(zL_mQm8XB*(@sJ*A5htvHg{5W0;!tkXr$=eMb6LkWBgLp%-<{7`QGnzl zYFb(j0|Nt#p^Nmw-6lL^V`HpRQU;&b78W?`J(f90D6Vq?1BVqA6>YySDRDx{c$r65 z^#070w{(G3FbE9|4F^t%1XAe3BB#TyDnPeE%_Y5m&xfgTaw=ukk%qf(S7&PFD;OB0 zYjV5;!ubsAyD^k!d2<8$TDC2KLm`$)`fCs&k-{qqMhlz| za2$Wj(cs@&z_I!C>@54E5>3v~mX?-l;?5ryhH_)=o1er+UF9cZ2^II-FTE!6{N4KQ zZeo*@on3*K5h=xWD)X)w^uH5=@C5Fsy-S~UxHdNX_@f*r>U$0@6O;v>29){kJkL5t zx3%5JcanEWNl`y!4j+bwh0)V*~3HRS^+$%CRc)lgGPpoA+(0C*(WWqzS{LY z=F4Rd$e@vhH8nNCsaK#`uySxHJxmjQ^e|Oeb$bb|mb)c&xKcqfUR+p6LqS2o;b7m_ z+pFO0T&m+Wn61jn&#$4YoUaqvax}|tH8~0-S@8D7!h+2PJZ>Zq&zQAdrA^$*VTx>E z0BX()hY^WOPUgIH`Ep08X}g63KcZa0mtRouu|2jCVSTW<=?XApc{Bra{PykJ&}T)7 zOW8R&J-RP|rmg^!j@T9y_Z@e~v1t`TJJgc=o+g&$AM^FALR@;fLGCO^C>6BEcf2KW zY|4?qAB#n4C=~Db4SF>)Y`~!d!mQHopX*~Cotp*mad9k4u}r+^in225jg1YG@h1gn zGiW5`wnl-+9xhvRk5Kiw>ZnU4EhqKrTAGGCGTPc&niD@{PwfS6=Q$o9?3CB{_xBI( z(b3S{e0hhCx#|LxnM2rr_>jhKq&NXhPEL+qJcU9QJTR2*XDY}h2+|p|L@ZrW|&@$!|pdu<9?HSb4r?leb_NFrEgXKJB%kKS+)YC6T(K?CEqWag zgE;`|wal>o^EE;#S5V~kXuU^4A@1~KuZ3hBo+81Y^l%qfQ0D08*I4Lat{O@uhvClT zza9Sib>S1OQtKZ~@$oNUIS~_Hlm1UFtgM!b8XU$&QRMz0$M$EFj1?Rmi`9>hj{F2Y zSM(6aK|w+E=8w4&K7yRnnxJM=>b^9&r1bh4i<*klWOGrCk;8DG-Ed(X+QZqIDl9C_ zcX@7pzHcdpL9DpbQ?*5T?X@7=#c024moKwP4)cY!X3D?%CNv4_q2$X%O>MgVmClVZ zEAZNIOa;2iuEmk^{Rl zv9e04su6`TRY?u`z_c54X-*Gj6M6MnSyF!uf_1wyM*UOba2GXCSO08;dl%2TqB9oM!Zn4Bb zPVwi-w6;;iUl$jb^5{@@#GU@TZgVCbkyl8Tr8d4^E-%j^E&N>Wvt>0_R2Pk^-2)t` z{kT9zw%*6z(v5ZEIX3T0n7U$^~ z{pG|#=z^zxd}1OyBo5^4w*@$WlXRP1W?r@(d`}gThoZF92A12_zG`{%x=<;(*ia~L;WqNWaRImS7`(g^1_SM47h z^uFyu?yb_Hl#qtGO?zbC&N~V_3g7i)od&Zb>|;CgwR691GVJW^BqT70nRYX9aS%VN zESW&paBrIG{3zx(s10(N>Fhai5uu4@SGSY&TuI*kjYgxVSSas#QNG$WlxApmEh#K~ z$9^X!CgvOkg{c0aa?JI(_mbk`3-;%p4>lq9=*J0zpon`mX_YYBG=j` zCML2EhRy8kxW{vxdVST94{7~}Y;KhV?zc0)uGViVNlHjau}pT*He=s1(K2Dd5OL!` zxC8_QB;T>7=zlM5my_oXcC6_3ty{|UvIIfy567u+hAS9`qt4RbcZ7s2PJc>xuGmxu z7edctKUBt4JLtzNGow$E431+y5I7(0yCCX7D|&=)Z@5l>Ax4Z{_earJcR8-J+R;h- z3Y(dkt)rhBtCn1$p#cap0bv6bl5(i0heuq#MSncXbG7m<*qFHz9w7YvZ%xJpVEp}0j{b>IlM%j#XG^ zfgqCbc%Y_MX4XLl0!?L})b@5_jCPT(`L|?2{3E%#xj|zJC%!;C1zk%sU#GjjU&+df z11vzU7bik(sM$b3K!A#vn7GP!L6WXGb1(L@X;8s0D=A;|C11YSU4dTo_)v`Yp_}IS z%A^UX1-pq6c{#a48foc^9K^am>)fnCgk2JI{N4Iq8o9nyP_lG{ybB`T>U*kX&N=2% zopfkr9ndOd^BaNld5jvb1qTO{Z_cS<8mbG7Q*9{7s{(}`Fgm)T^H{Q{PoE;{s*$0U z=J~*Bp5wz0su{8lR+oRR3pa|ank!^#nygQ?si>&50+$E|9C?o&GKku@;>K%=XeLez z*2D`YZ|1;&R!!fEWNb1b-lwJM^KaSO9Lpan38OVG7INd#V$pMdD+dR)5SIoD5qHLN z3NQ}FKqV|$en)zrOSIeDvOfglJ^Fk`V9etzCDjt}yVs?>e?~rD6!J=+ot-VOSY~K1 zR`T-l8dLRrkfSx9Tw5m+dXgz^^GHeQGj-#YW)8Ot|w)RGEm&Lha;&s@0czVJqsA!bRw>`cA2<%pPCR|;qx{meAi z9&Py#P1E_yopplSl=v}U9)eY~VsxxNXnm59kRXxNj#fB)+YV|3F$NOF6L$pq$pu=q zgu+44ZY{y{s``EhQDMxn6l4Jp;@;#4+%A0s%(T>$R4acY`0({{4BZv))yVxyZ^`MOXGB_*8c8Wd_#+Fy?O)z#wY( zEx6ykdsp=b`gzqrW*9@(n>SQrMrDfq$W#l=--x3}F-f8trQt0b7?pWvLV}}3jlj3O zNgA@pG?V^&hNwSirW~QWck9$~A_|_Kp2-5HFLY?-OrXHYJ=+oElP(EjSZ^YLl(LG- zIWRddfnA4{y`#^-GD#qJc>+%US%-Ll*v*-5YinEI+bdZsb=MP8he91&#TFLw@EA86 zDO2&((Q)YpcFW&?_IhF>X*K5#C>t8PDfwz>uB{KJH6)md<=T;-0#CLwM21W9@>>7)#OIK|3ki{r zM(X?d`~0+R(;o!`jltG6ThYo!*z7jt}R4WbjKO zw!o%c6Sdb1kun|X=}{FGM}ts$laOHPoSG4EaK7MaNn$UwN})xH)Y#7?cRpr5$oT?R z=S@n=ZDdn+_E``VkRu(-PrOq&6wfG|38>0994YQ@j9s5zPDB^uFmJI8;B1!Ar+3hW(A9kjfHoWWs0c2N22*86N=Zri z?%lf^w{DS+6dTGbDUob$ZMm%ezT4!IY+k3VpwPXlE@s(Fb6wK&BBWZJJUj}(ir^Rp zUH^8!dZvr#EO-G376_->9#lf336G5Y0^Ski7(S@5u(qC=Y6}7X9*^r3*+!Am37rM0 z{G||4aU--f{jQty&e9WnuNahdV>wH4T|B5}GC;q+sxBmi2>fj*G)(m9E`w8JFjR)g zY;OW6#J>_=s|GxV^)#T_6`h?0z_yy0n}@u3aprdu@@h&-in#0Fb8_B?fo;CxF_IG}MrZbwM*6&|=zh%yxL5E?d5aY%Bs= zS0wb)k-IN#hd$B0HidB|hJd@ny|6yY;*ydKxE6xYnwpw@$gpyT@YV=yzyh|Zp-o)W z@wZ;CB@`zwH65Mp_Li@&ZzNb9L7Sg9#+Ds0mDKRYe}Fc9$0y6oUqSv9k9S)GAeE-q z*7(58KPQ3YKrEUEB&W_09;h7ut?u)DeM;ppb4mfO+{@Pl>K^DBd84gp0 zzgzhTxVi;QRY9__?OO<R46EGKlI<6Z3YFODRN&=D2&7tw0=L`{%Ws48Nm*HVuh;Abg6;_L1 zeNK*ipu<+so4dKY(_G~@dbgDO>C@6LI@y56p5w*G=4feR5JLFPn;myXNl)SZX1UCJ z5a6`U{KkwJ)YuM@!2AgwtmUi5bHJX|y?gh92E+>}8{<<`BjGWpfxg5&mS`X+3Wc~I z&kJ~w3@1X)gU2@zu94@Zp>`V{7u~U17d9yGGM5=8s7?~FL%<>Ugb`?#ih+R}rBeeT z0&02GPLiis2Jo>km~#Q2BSL+ucr9Hxf4&X!i5;xt$nY>Nv)?)))_aB=pT8n^SK6Ff zj$h)_OC7G-x;oV^E;is1AgiYK=2W*jN!SJX@m^Z5(Ob}S_4oN4tq71tnGj<@F4u*( zdaFEo`bGLRoscD$d94}wf#y_s^r!^y+GHMH1<~=n(B_ zdJ(dLfdROl6!u@DXS!l%K-1OwABsY?oT(rdnbRA zj04&rNO7L;X9T{;g1L3}@=}ISgzEbB*8l~4jxT~uzbzzm2?$=V%#0ZJY(D64P0!7} zgdy-^*LyC-I@_{HE5FovS}vsQ-~N6p)U;=~lKze)iEdF1*lD{$V}HZ>m6g{r{s$i* zq2fT76oWs(Q`p=1_%gc@zC}rXPR=0ULx{&=+bI+t7|&4i=@+06xGxSW&WrweIfq+b zwp({hISAMTn#NDdN5v>-h$`|mJeP;b!5MVIdrjeRylBb`7upAN)F3WD^DZGFGw}3e zkg2XKE}E*5l%1Md8Vfs4ntFORXof+Nv;Su|uSDkmvpvB7-uwTZoy{X=g#M$ne+jJV Ruvbl>s;I6|ENAlKe*j2SP+b53 diff --git a/src/documents/tests/test_consumer.py b/src/documents/tests/test_consumer.py index 8ead1ea41..a6f0cc55a 100644 --- a/src/documents/tests/test_consumer.py +++ b/src/documents/tests/test_consumer.py @@ -5,14 +5,12 @@ import tempfile from unittest import mock from unittest.mock import MagicMock -from django.conf import settings from django.test import TestCase, override_settings from .utils import DirectoriesMixin from ..consumer import Consumer, ConsumerError from ..models import FileInfo, Tag, Correspondent, DocumentType, Document from ..parsers import DocumentParser, ParseError -from ..tasks import sanity_check class TestAttributes(TestCase): @@ -183,24 +181,6 @@ class DummyParser(DocumentParser): self.text = "The Text" -class CopyParser(DocumentParser): - - def get_thumbnail(self, document_path, mime_type): - return self.fake_thumb - - def get_optimised_thumbnail(self, document_path, mime_type): - return self.fake_thumb - - def __init__(self, logging_group, progress_callback=None): - super(CopyParser, self).__init__(logging_group, progress_callback) - _, self.fake_thumb = tempfile.mkstemp(suffix=".png", dir=self.tempdir) - - def parse(self, document_path, mime_type, file_name=None): - self.text = "The text" - self.archive_path = os.path.join(self.tempdir, "archive.pdf") - shutil.copy(document_path, self.archive_path) - - class FaultyParser(DocumentParser): def get_thumbnail(self, document_path, mime_type): @@ -223,8 +203,6 @@ def fake_magic_from_file(file, mime=False): if mime: if os.path.splitext(file)[1] == ".pdf": return "application/pdf" - elif os.path.splitext(file)[1] == ".png": - return "image/png" else: return "unknown" else: @@ -538,19 +516,6 @@ class TestConsumer(DirectoriesMixin, TestCase): self._assert_first_last_send_progress(last_status="FAILED") - @mock.patch("documents.parsers.document_consumer_declaration.send") - def test_similar_filenames(self, m): - shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"), os.path.join(settings.CONSUMPTION_DIR, "simple.pdf")) - shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.png"), os.path.join(settings.CONSUMPTION_DIR, "simple.png")) - m.return_value = [(None, { - "parser": CopyParser, - "mime_types": {"application/pdf": ".pdf", "image/png": ".zip"}, - "weight": 0 - })] - doc1 = self.consumer.try_consume_file(os.path.join(settings.CONSUMPTION_DIR, "simple.png")) - doc2 = self.consumer.try_consume_file(os.path.join(settings.CONSUMPTION_DIR, "simple.pdf")) - - sanity_check() class PreConsumeTestCase(TestCase): From 61143b3ad1adc65845e7f316e5a7253fb0bffafc Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 7 Feb 2021 19:53:08 +0100 Subject: [PATCH 059/225] make the test case fail --- src/documents/tests/test_consumer.py | 1 + 1 file changed, 1 insertion(+) diff --git a/src/documents/tests/test_consumer.py b/src/documents/tests/test_consumer.py index 8ead1ea41..280124f48 100644 --- a/src/documents/tests/test_consumer.py +++ b/src/documents/tests/test_consumer.py @@ -538,6 +538,7 @@ class TestConsumer(DirectoriesMixin, TestCase): self._assert_first_last_send_progress(last_status="FAILED") + @override_settings(PAPERLESS_FILENAME_FORMAT="{title}") @mock.patch("documents.parsers.document_consumer_declaration.send") def test_similar_filenames(self, m): shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"), os.path.join(settings.CONSUMPTION_DIR, "simple.pdf")) From 40106f6fccd85ee7ba1eb24765d9beb3305cf6cd Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 8 Feb 2021 00:10:52 +0100 Subject: [PATCH 060/225] updated documentation regarding execution of management commands with docker fixes #509 --- docs/administration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/administration.rst b/docs/administration.rst index c54323e6e..03a9610d1 100644 --- a/docs/administration.rst +++ b/docs/administration.rst @@ -180,7 +180,7 @@ tasks on your paperless instance. You can invoke these commands either by .. code:: shell-session $ cd /path/to/paperless - $ docker-compose run --rm webserver + $ docker-compose exec -u paperless webserver python3 manage.py or From 816c95a4ae443f552ea793b94b72dc2594d04e83 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 8 Feb 2021 13:18:08 +0100 Subject: [PATCH 061/225] code style --- src/documents/tests/test_date_parsing.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/documents/tests/test_date_parsing.py b/src/documents/tests/test_date_parsing.py index 9cbb19c2b..50543ee75 100644 --- a/src/documents/tests/test_date_parsing.py +++ b/src/documents/tests/test_date_parsing.py @@ -1,7 +1,6 @@ import datetime import os import shutil -from unittest import mock from uuid import uuid4 from dateutil import tz @@ -9,7 +8,6 @@ from django.conf import settings from django.test import TestCase, override_settings from documents.parsers import parse_date -from paperless_tesseract.parsers import RasterisedDocumentParser class TestDate(TestCase): @@ -152,4 +150,4 @@ class TestDate(TestCase): 2018, 2, 13, 0, 0, tzinfo=tz.gettz(settings.TIME_ZONE) ) - ) \ No newline at end of file + ) From 302ebf737e7df91a93b20fc9764346e7c681e073 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 8 Feb 2021 13:18:39 +0100 Subject: [PATCH 062/225] refactor migration test case --- ...rations.py => test_migration_mime_type.py} | 45 +------------------ src/documents/tests/utils.py | 40 ++++++++++++++++- 2 files changed, 41 insertions(+), 44 deletions(-) rename src/documents/tests/{test_migrations.py => test_migration_mime_type.py} (66%) diff --git a/src/documents/tests/test_migrations.py b/src/documents/tests/test_migration_mime_type.py similarity index 66% rename from src/documents/tests/test_migrations.py rename to src/documents/tests/test_migration_mime_type.py index ee7180c63..5e825e89d 100644 --- a/src/documents/tests/test_migrations.py +++ b/src/documents/tests/test_migration_mime_type.py @@ -1,52 +1,11 @@ import os import shutil -from pathlib import Path -from django.apps import apps from django.conf import settings -from django.db import connection -from django.db.migrations.executor import MigrationExecutor -from django.test import TestCase, TransactionTestCase, override_settings +from django.test import override_settings -from documents.models import Document from documents.parsers import get_default_file_extension -from documents.tests.utils import DirectoriesMixin - - -class TestMigrations(TransactionTestCase): - - @property - def app(self): - return apps.get_containing_app_config(type(self).__module__).name - - migrate_from = None - migrate_to = None - - def setUp(self): - super(TestMigrations, self).setUp() - - assert self.migrate_from and self.migrate_to, \ - "TestCase '{}' must define migrate_from and migrate_to properties".format(type(self).__name__) - self.migrate_from = [(self.app, self.migrate_from)] - self.migrate_to = [(self.app, self.migrate_to)] - executor = MigrationExecutor(connection) - old_apps = executor.loader.project_state(self.migrate_from).apps - - # Reverse to the original migration - executor.migrate(self.migrate_from) - - self.setUpBeforeMigration(old_apps) - - # Run the migration to test - executor = MigrationExecutor(connection) - executor.loader.build_graph() # reload. - executor.migrate(self.migrate_to) - - self.apps = executor.loader.project_state(self.migrate_to).apps - - def setUpBeforeMigration(self, apps): - pass - +from documents.tests.utils import DirectoriesMixin, TestMigrations STORAGE_TYPE_UNENCRYPTED = "unencrypted" STORAGE_TYPE_GPG = "gpg" diff --git a/src/documents/tests/utils.py b/src/documents/tests/utils.py index ebf79f6c5..b9be25d70 100644 --- a/src/documents/tests/utils.py +++ b/src/documents/tests/utils.py @@ -4,7 +4,10 @@ import tempfile from collections import namedtuple from contextlib import contextmanager -from django.test import override_settings +from django.apps import apps +from django.db import connection +from django.db.migrations.executor import MigrationExecutor +from django.test import override_settings, TransactionTestCase def setup_directories(): @@ -79,3 +82,38 @@ class DirectoriesMixin: def tearDown(self) -> None: super(DirectoriesMixin, self).tearDown() remove_dirs(self.dirs) + + +class TestMigrations(TransactionTestCase): + + @property + def app(self): + return apps.get_containing_app_config(type(self).__module__).name + + migrate_from = None + migrate_to = None + + def setUp(self): + super(TestMigrations, self).setUp() + + assert self.migrate_from and self.migrate_to, \ + "TestCase '{}' must define migrate_from and migrate_to properties".format(type(self).__name__) + self.migrate_from = [(self.app, self.migrate_from)] + self.migrate_to = [(self.app, self.migrate_to)] + executor = MigrationExecutor(connection) + old_apps = executor.loader.project_state(self.migrate_from).apps + + # Reverse to the original migration + executor.migrate(self.migrate_from) + + self.setUpBeforeMigration(old_apps) + + # Run the migration to test + executor = MigrationExecutor(connection) + executor.loader.build_graph() # reload. + executor.migrate(self.migrate_to) + + self.apps = executor.loader.project_state(self.migrate_to).apps + + def setUpBeforeMigration(self, apps): + pass From c2d8bda83c8bdc214565351329a4643eb6ba998a Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 8 Feb 2021 19:59:14 +0100 Subject: [PATCH 063/225] fix for #511 --- src/documents/file_handling.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/documents/file_handling.py b/src/documents/file_handling.py index 6ee97f5c3..64858de78 100644 --- a/src/documents/file_handling.py +++ b/src/documents/file_handling.py @@ -161,5 +161,8 @@ def generate_filename(doc, counter=0, append_gpg=True): def archive_name_from_filename(filename): - - return os.path.splitext(filename)[0] + ".pdf" + name, ext = os.path.splitext(filename) + if ext == ".pdf": + return filename + else: + return filename + ".pdf" From 0c676b90f24ded60ddee4a95d43658496e99076f Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 8 Feb 2021 20:59:14 +0100 Subject: [PATCH 064/225] migration for #511 --- .../migrations/1012_fix_archive_files.py | 181 ++++++++++++++++++ src/documents/tests/samples/simple.jpg | Bin 0 -> 17740 bytes src/documents/tests/samples/simple.txt | 1 + .../tests/test_migration_archive_files.py | 175 +++++++++++++++++ 4 files changed, 357 insertions(+) create mode 100644 src/documents/migrations/1012_fix_archive_files.py create mode 100644 src/documents/tests/samples/simple.jpg create mode 100644 src/documents/tests/samples/simple.txt create mode 100644 src/documents/tests/test_migration_archive_files.py diff --git a/src/documents/migrations/1012_fix_archive_files.py b/src/documents/migrations/1012_fix_archive_files.py new file mode 100644 index 000000000..e95715265 --- /dev/null +++ b/src/documents/migrations/1012_fix_archive_files.py @@ -0,0 +1,181 @@ +# Generated by Django 3.1.6 on 2021-02-07 22:26 +import hashlib +import logging +import os +import shutil + +from django.conf import settings +from django.db import migrations + + +logger = logging.getLogger("paperless.migrations") + + +def archive_name_from_filename_old(filename): + return os.path.splitext(filename)[0] + ".pdf" + + +def archive_path_old(doc): + if doc.filename: + fname = archive_name_from_filename_old(doc.filename) + else: + fname = "{:07}.pdf".format(doc.pk) + + return os.path.join( + settings.ARCHIVE_DIR, + fname + ) + + +def archive_name_from_filename_new(filename): + name, ext = os.path.splitext(filename) + if ext == ".pdf": + return filename + else: + return filename + ".pdf" + + +def archive_path_new(doc): + if doc.filename: + fname = archive_name_from_filename_new(doc.filename) + else: + fname = "{:07}.pdf".format(doc.pk) + + return os.path.join( + settings.ARCHIVE_DIR, + fname + ) + + +STORAGE_TYPE_GPG = "gpg" + + +def source_path(doc): + if doc.filename: + fname = str(doc.filename) + else: + fname = "{:07}{}".format(doc.pk, doc.file_type) + if doc.storage_type == STORAGE_TYPE_GPG: + fname += ".gpg" # pragma: no cover + + return os.path.join( + settings.ORIGINALS_DIR, + fname + ) + + +def move_old_to_new_locations(apps, schema_editor): + Document = apps.get_model("documents", "Document") + + affected_document_ids = set() + + old_archive_path_to_id = {} + + # check for documents that have incorrect archive versions + for doc in Document.objects.filter(archive_checksum__isnull=False): + old_path = archive_path_old(doc) + + if not os.path.isfile(old_path): + raise ValueError( + f"Archived document of {doc.filename} does not exist at: " + f"{old_path}") + + if old_path in old_archive_path_to_id: + affected_document_ids.add(doc.id) + affected_document_ids.add(old_archive_path_to_id[old_path]) + else: + old_archive_path_to_id[old_path] = doc.id + + # check that we can regenerate these archive versions + for doc_id in affected_document_ids: + from documents.parsers import get_parser_class_for_mime_type + + doc = Document.objects.get(id=doc_id) + parser_class = get_parser_class_for_mime_type(doc.mime_type) + if not parser_class: + raise Exception( + f"document {doc.filename} has an invalid archived document, " + f"but no parsers are available. Cannot migrate.") + + # move files + for doc in Document.objects.filter(archive_checksum__isnull=False): + old_path = archive_path_old(doc) + new_path = archive_path_new(doc) + + if old_path != new_path and not os.path.isfile(new_path): + logger.debug( + f"Moving {old_path} to {new_path}" + ) + shutil.move(old_path, new_path) + + # regenerate archive documents + for doc_id in affected_document_ids: + from documents.parsers import get_parser_class_for_mime_type, \ + DocumentParser, \ + ParseError + + doc = Document.objects.get(id=doc_id) + logger.info( + f"Regenerating archive document for {doc.filename}" + ) + parser_class = get_parser_class_for_mime_type(doc.mime_type) + parser: DocumentParser = parser_class(None, None) + try: + parser.parse(source_path(doc), doc.mime_type, os.path.basename(doc.filename)) + doc.content = parser.get_text() + if parser.archive_path and os.path.isfile(parser.archive_path): + with open(parser.archive_path, "rb") as f: + doc.archive_checksum = hashlib.md5(f.read()).hexdigest() + shutil.copy2(parser.archive_path, archive_path_new(doc)) + else: + doc.archive_checksum = None + if os.path.isfile(archive_path_new(doc)): + os.unlink(archive_path_new(doc)) + doc.save() + except ParseError: + logger.exception( + f"Unable to regenerate archive document for {doc.filename}" + ) + finally: + parser.cleanup() + + +def move_new_to_old_locations(apps, schema_editor): + Document = apps.get_model("documents", "Document") + + old_archive_paths = set() + + for doc in Document.objects.filter(archive_checksum__isnull=False): + new_archive_path = archive_path_new(doc) + old_archive_path = archive_path_old(doc) + if old_archive_path in old_archive_paths: + raise ValueError( + f"Cannot migrate: Archive file name {old_archive_path} of " + f"document {doc.filename} would clash with another archive " + f"filename.") + old_archive_paths.add(old_archive_path) + if new_archive_path != old_archive_path and os.path.isfile(old_archive_path): + raise ValueError( + f"Cannot migrate: Cannot move {new_archive_path} to " + f"{old_archive_path}: file already exists." + ) + + for doc in Document.objects.filter(archive_checksum__isnull=False): + new_archive_path = archive_path_new(doc) + old_archive_path = archive_path_old(doc) + shutil.move(new_archive_path, old_archive_path) + logger.debug(f"Moving {new_archive_path} to {old_archive_path}") + + +class Migration(migrations.Migration): + + dependencies = [ + ('documents', '1011_auto_20210101_2340'), + ] + + operations = [ + migrations.RunPython( + move_old_to_new_locations, + move_new_to_old_locations + ) + ] diff --git a/src/documents/tests/samples/simple.jpg b/src/documents/tests/samples/simple.jpg new file mode 100644 index 0000000000000000000000000000000000000000..a8c58af0df6e4b4e6b72179b92aa9df4e2bbd8d8 GIT binary patch literal 17740 zcmd41WmFx_wl>;Jad&rFxVr>*cMq~~cLGaLXzOTg z_lUngVh-=e0097k`H$S@UzqkE?C>uv@DKLZ*OhzJc^oq>ro;chHva?LI(oZ3j$i*# zpYdNYJR$&~^m)7poJdz!B&Q)Cr;oRTjjy*Ymy_*djQ_h(0X~2(pajSQoB$GVecX_b z$PnOrtbG7)zyYuUd>@at02ko&SpUzy|J%s_7vPHwe7rUQAm`y3!VPZGt#`thpZzaJwQMtX*wA?2rhqf3C$p zy#4100RQHrk3B!%e~CbtvH$>EdwBT81OO0$u^HfoLEGhy&gL@jwEQ1SA6~Kq`<9WB{2!7LWtv0&jsl zpa3WYihyFE6et7AfeN4!s0M0)TA&Vi2fPOwfJUGRXa-tKsV3>d;&fL zy+A)O01N^{zz8r3i~-}o7vL)}348;lff-;HmmF$e}hK`;;s2o;15!T@1`ut7K= zJPt7K8*jfE+>2 zAXkt($OGgB@&WmQ{6T@BU{DAs3=|G}0g3=cfnI}RK(U}Vpaf77CkKsBIRP(A29s1eizY5}!@IzS&m-JnmPUQj=15Ht)L1&xEgfF?mx zpc&8{XaTeYS^=$retR>If4p5!ExYt za1uBLoDR+e=YZdW3&2I-QgAuA3S0xO2fqh@05^l%z@6YO@F#FDcmO;M9tBT;zk;X0 zv*3B~5_lE74&DH7fp@_N;3M!U_#Au*z6Rfd@4ycbFa!ocg`h*QAh-|$2oZ!7LII(M z&_Wm>%n$^G1HuL2g$O`|Az~0oh%`hNq5x5bs6jL#IuLz`A;bh?4zYsRK#&kehzrCW z;sx=A1VDlzp^)d0myjq(G$am^07-_VL9!sZkbFoHq!dyCsfN@+-a|e>S|II^kB}Zn zFJu5R0vU&Vg-k(aAq$XY$U0;LvJKgT970YY=a5UtZ^$3WJroRuK~bR?P#h>eln6=+ zrG(Nz>7h(e1e62H4dsUlK}DgG&?itis1j5assYu3>O+m7W>8D04HOA=gt|gKpgvH4 zXb?0M`WzYweGQF+CO}i5>CkNGTWBG)1X=;DhSozHpv}-W=tpP|v=2H69ff{@euK_J z7oaQ9@6b)?F7yC;0zHRbLT{jVFaQRHp~5g=I4}YjF^n8W4WoxK!4NP`7%xl!CIXXy zJ%P!=lwfKwEtoFM5M~OqgxSFCVNNhNm>0|s76c20y?{l+qG4}fNw73n7VIso5LODS zgw?{{!9KuRVV$rZSRZT%HU|3&n}*H9mSO9#P1r8%5OxZ?fc=L3g#&OX92JfU$AuHZ zN#T@mIye&?0q2DC!Uf@?a4EPfToJAc*M#fAjo@Z*E4VG(5$+22g!{sS;Gysr@F;jJ zJOQ2p&xGf~3*e>jN_Z{20p1L6hj+t!;e+ro_*eK0d;z`!{{i2IAHYxG=kROz9~1xu zih_oMg@TVlj6#7zgTjcyio%J)iz0*~h9Zq3hoX$4fue(Ah+>Ljg<^~1h~kFgh2oF$ z3?&>T5+w#D9wh}O6D1d=5Ty*I3Z)+914=8(N0iSfgD9gYUr}aI7ExAFHc)m^j!@1} zE>Uh#9#EmEXsB4I_^2ePl&Ey5%&6?B+^7PmqNq}+a;VCv8mPLcMyTef)~NQVE~uWU zeyG8y;i!?QF{lZssi;|~d8oyx6{xkS4X7=sov5Es2T(^*zoO2dE~2iZZlUg@o}m6h z{f&Bu20=qb!$QMHBSE7=qeo*w<3Qs@6GoFjlR;BNQ%BQ5Gek2(vqrN=b3yY$^GADz z_5$q{S{zz3S_WD!S`k_~S`AtQS_|4ow9jaRXya(#(B{!r&^FL^(T>s1(SD=dp@Y#; z(Xr48(8U!a%{m#K6ZO#h}Ju#9+hV z#t_61$B@BL!qC9b!!W_H!mz_|#_+`O#|XiAi4l#FfRToggHec4j!}!zh|z}8jnR)W ziZO{Xhp~dOfw7NqigAhY2NQ&ef{BGmfJuf)i^+`1fysv{f+>ZmfT@P5gK30mfoX^7 zjOmH#j~R*?ff<9Dh?#-;7PA<$67wBqGiE1dFXk}j7tC4AWy~L#dzdGfmzaOBKv<|) z*jPkZ6j*dvtXN!Ff>`2MvRKMkT37~H=2$jZPFNmT{#YSc5m+%;NmvsUKjM_9kGZm~hwDA?H8MA#JA^wV{9vI zdu&&1AM9Z47ueC*3E1h_Z?Q|TtFarf+pxQ_2e8Mnr?Ho?e_-!npJHEO-{C-UFmUj3 z$Z%+JSa7&-1aTyABSkv z`G&KAvyQWabBuF|^A{I_i;jzrONL8}%Yw^|D}*bFE03#=tA}ffYlG{A>xmnP8;<)5 zHvu;tHxIWIw+8nEZU=5J?g;KA?gH*Q?k?^L?iKDG9t;l?j}VUnj{%Pjj}K22PX8!i@qXh2_^9|e z_$2r=_$>Hb_(J$n_=@p^ zg76Jt8sS^QGQv8-7Q$}ALBg+u^Mv0C_Xy7jZ-_ud=tTHL6hsU}97F;{l0*tbnnXrK zRz!|Ooenp%}oJCwjTuuCexRbb_c!GG2c%68U_>B0L1VVyILP$bI z!c4+VB0?fVqDrDiVori2aVH5Nc}@~Tl0x#9q>QAVq?P0o$q305$uh|n$uY?_DL{%w zicd;G%1FvZDopx>RE1QR)SMJa>P{L!`hql;G?nx%X*uaT(l*jw(lOE*(lydu(x0Ta zWDqh;GGa0sG6WePnK+pOnI@SrnKhXUnJ-xw*=w?7vK+EfvO2O>vQK2AWYc7;WIJR( z$!^IZ(P^41iQB+VgP;^oZP<*9Wq}Zf5 zrnsgAQDRUMQPNN%DETQRDU~R7D9tF5l%ABql#!GPlv$L;l(m#Cl%FZbC}%0xDfcNa zDDSCIsqm>NsaU9Zsl=%isI;j}sqCmcsDh{>s1m5MsEVoTs9LE$Q;k#2QT?Dgq`IUA zsL`p3sA;I#s0FB{s8y)-s4b}-sePzJsiUb=sq?5SsT--gsE4Shs8^_Wsn4nZ(xA}b z(@@f|(D2bn&?wR9(wNgY(0J2?(7dKep~<7EplPJ(q8X-{qFJTcqxnU1Pm4-RKub-F zpcSB%rd6fYr?sMWruCzJP8&y?L0d#yOWR7@OFKckK)XqMLVH68rNf~kr(>eyp%bT5 zq|>1@r*ok5rVFEsp-ZDHpsS{9ru#%UPB%}tL3d2|n;uG!Lr+f6M9)JnL9axwOK(Z< zMDI%Z7Ife~}6NVc`7$YttB_k^%Kch6G8lxej4Wm0_5MvZ$GUHpuO2#I} z9>y`odB#n~6UJL6I1@e-H4}nKkV%F~gUN&m$>hlt!W7Mv##G2u%hbly&os%j!nDiu zi|K(GgPDYxftj0GoLPxkkJ*aZg*kvZk~xX_EpsJv6Z0qLapnc)E#{xhcPwZuL@abH zoGhX&iY&S;mMqRJ{wxtJi7dG+6)a6GpIF9O7Fo7g&RFhP(OHRE8Cbbl#aWeE^;w^? zy0HeazG6*b&1bD)ZDs9eon&2Q-DkZ*fDt$d3Ir=c03n0WK$s#N5I%_Kh&PCAL^+}n z(SsOAEFiWKXNY??3^r0WMmAnHDK=F$BQ`rWFSby&Shh^IQnm)RF1As&dA2RKGq!tn zbaoPUMs{9yDRwn>V|FCFH+wky8}@AWGWJIH9`TTT>CLQXnPZcYhK6;4A=J5DdoFwQv6Y|e7d z51gMlzi=*d?sHyoLAmg_sJS?}M7fl>^tr6LJh(!*V!1N8%D5W2K5vp^gP@=l00fWCOi&2 zzC15^l6dlYYI)juhInRqHhIo?9(b{M$$43Mg?Qz8b$Oriy7Pwc#`0$Imh(38_VP~h zuJIo6-twXH5%V$f@$o(3)8sSfbLI=;d(D^5SHjo8*TXl#x59V8_nRNiPsq=}&&w~( zufcE5@5~>_|B64Izl6Vm{}cZg{uTa1{u==l0b&700e%4)0WASb0at-%0x<$v0_6hD z0(}Bg0^bEr1?~hf1<3^wg2IA|g8G8Cg5HA91rr7H1ZxF51xExI1a}25g`h$NLUckr zLefH-Lgqp)Lcu~YLYYG4Ld`<`LeoM&gnkM=2xALV33CXG3#$kl3p)t=3r7j336}^r z3V#-!6kZoT5xx__6rm7d6A=|r7BLdB7x5E`6iF2+7HJUqEHWvwE^;DrCyFUbA<8Z) zCaNN8Eb1T{Ao@x)U9?oRNwiONO7w^5nHV63BSs^}B_=7RA!aV-BKAxyRxC%XN~}$6 zSZrQwSL{k0E>0xQBrYH>FRmwUEAB1+Qao9_Nc_F{C-F(~b@5a2dkJg_Y6(sWNeK-J za|u_85Q#X6T!|Wq4vA5TC5Z!x8%Z=tGD(D_sHC!_v81D9pk%aUmSlxwtK^X6yyULr zwG@gJi4=>Ju#}RNk(7f}fYfWLOsNW~R;eMWd8u8gYiSf|5@{A`VQD34BWVZeKb$eyr05qqNg#Po^FlV?xjp1gfh`{d)3i6^U1j-TAiV9QX; zaLY)`XvsX4@sxQXlPps#(S1Q*mHz+qRw=4Hs9!;KHo?TvCUR~Zo-d#Ri zK1sevzEQqcep-G@{!#(1K%&5^AgZ9MV5Z=z5UP-%P@wQ$;j_Y&!luH7B21A)kyTMt zQB~1Q(N!^2F;THlu|cs{aawUp@k$9riBt)pB(9{cWTE7)^js-fsaUB=X+UXCX;0}! z8C{uDnNwL>SzFm!*+)52IYYTZxm|fwc}4kH`9TF&gYPo8w>WJ#H>apsB8m=0>nt+<3nz5Rb+B3Cy zwF0#UwLZ03wOzFvb#!$~buRTM>bmN7>i+7{>N)DQ>fP#->Kp188gLCV4R#Gl4J{38 z4IhmtjZBRyjgJ~%G`?$`YeF?iG!dHOni`r`n%xn(Lb9T2L)wErgc1 zmZsKIEpM$TtxTy)(uQl3YO`xgYHMrTX!~iu*3Qwc)9%rp(%#a()wIn(mn% zRF7B>p(mlIrDvn(rx&f4t5>i0S#L&fSMOFIOP^MsUtdw*MBi0ETt7v>OutQkOn*)P z+yG`kYQSzFWuRk#Gzc_^Gsrh+Fc>gcFgP^0H^et&G88dXGqf`FHjFaNHmo)5F`PEs zF}yXxGNLsSFj6uyGjcb2X_RhMW%SW#(rDA@+8EuK+L+f^!PwZ?)%dw_s&R#Jhw&HV z4dY7_G!rTl9us*JV-pvXaFbM%3X=|#FD4r%SEgvDRHnS93Z^EeuBOjT(@ZN(KblUO zZkhfz!!V;U<2O?>Gc)rri!jSHt1;^_n>O1u`(ut{&R{NVu4ew!+}AwX{H^(W^M3OM z^CJt81(5~9LefIV0%;L!kzi45(PA-Xv2O9p64jE@lE+fs(!|ou@}*^lWwm9G<+SCl zuwuin`K*P+iN>-dt?W)Ber9=dtzr`=WG{lmu^>W*JC$hw{LfkBtRmN zQb;|dBQg}3imXI-A*Yag$UA#{dlq|1dtG}6`%wE7`wIIm`ziZ9`#T4G2UZ732R#Qz zhcJg!hbo6|hZ%=`hX+SOM>a=kM*~M^$LEe2jx~;-9p@a6oWM>bPMl71PR35|PLWR8 zPVbxsoR*z_I>VhQo%x)Voh_VwoMW5|oSU6ToxeL@xnR1`yNI}Gy4bk{yCk`kyL@z+ za@ljacO`UXbA94!=<4eF(lyJq-nHL#$@Qlj+>O$W-%Z8M%FWL$&aKF;&FzcZmfNj6 zo;!=Xq`RKGv-@-R4EH+sKKDiUpB``zN)LVy6_2ML{vK~UN<7*J1!4s<1xf_!2f75l49p2^2pkFg9{4*5Cx|6TD#$R%Jt!(D zFQ_SKJZLlMPcT6+Td-`fX|PvtOmITsT}S+#x(XJTv@#_;C35@SEp&&k@gMo|``RejfY03{pi2F#ANS;XLNbAVp$kfQ%$o|OH$g3!vDAp*MDAOqKsJN)o zsII8FsMA-dujpQhz0!N-`YP&G{;Sqkldtw)gI`m;7I>}k+Tr!{*V(TdUyr@sdVLp7 z63r8>5^WRxEIK{9K6)tnd-QD#K@3NXVvJQxU`%pMO-z5xTFiAUZY*1@T&zW`e{5oG zRcvqUO6+AEP8@5TY@At~UtB_5Wn6FEa@^$`>^H1$WZ#&*@q3f_rt(eio0T`0@i_5_ zc-eUKc>nmM_^SB6_|^F91iS?H1o;HZgusN9gxZ9`gzpKri3Ew9iAssqiO&+#6W=9{ zByJ|&C6OfYCaEPMlfsj7lA4mfB<&@GlPQyhlC_gvlB1FflG~G~lTT7mQ|MD9QjAi( zQsPp|Q$D3ErCg-qq#{z~Q!P^iQ&UsxQioGFQvarrr17Syr`e~yNXt!YNt;YNOoye@ zri-T=q6Pnz5Y;WKv`bW@=};WWLHQ%IwUX z%RI}%%woxs&9cY}$V$no%Noww%(~Ae%NEGi%686<$}Y_A%%00W%fZZH$&t;m%n8a# z%XybGnzNk?8E5MGd5&{8l}a8ig? z$W$m(XkHjtm{$0{aIA2*2vS5-Bvxct3JD;8C#iBnQd8kS#DWd z*-Y8ba?El>xk9;hd02T)c}w|p`Dq141#5+T#nXz=itLJ(im8gzN{mX@O8H9b%FxQ3 z%9hIM%F`;$Dnyk+l}%MxRc=*V)lAh{HFh;ywNkZR_4Dey>W=F9>R&auHJml7H4Zfq zHH9@@HA^+uwS=`iwHmd~wXbVSYd_bn)&8j?trM)%t@EgRQ&&|tShraZ)Kk@q)f?6O z)~D3Js~@l5e+PfZ@b1Yw%XiP-WxZ>DH~sGCd#v|t@0H&p-@klc@V@K)();TMq6WSO z?FP4o*oMl6friaSppm*!ywRl5zcH<`q48_u(FgPoEFTm;*nD{YA^$_?hs6(9O@vK+ zOs??O@mEa&ERI5X31u==Ah<`=BDPU=ASLtEgUT>EeR9i%>!j!u>oo54?@aIf(D|+N=SS?1oFCObI)9A*SpIR~<7OAQi?&O; z%c3i!E4Qn?YoY6^o2Z+=TesV*JF&aId!qZW2fYW;qts*H6V+4N)7$go6Yz=Vlhh~k zPtQK(d}{x+@ag(9(Px3rdY`>NCx3qb`RnJCUaVe@UbSB5-k9FX-l5)|K3E@PpIo0! z-;2J&zE6GYefRxT{Sy6V{m=Sy`rG>#`mYB_280F-2K)xn20jc-4V(|+4e|_X4|)tH z4%QD&3?2_*4si^r4Y>@(4pk404DAo24zms`4Lc0K8ZI9m7~UR%jWCYLkJybwjFgP@ zjckm9N9jjpMy*F*j24Z48vQ;7jM0ug8M7J-A1fH^8CxHF7^fMR9=9A18_yr_9$y>3 zpP-(Qny{P*o5-K&o>-f>|3dRc`is?<@Gk{ldcLfGdH72E^~u+#U!Q+1{QCLpk4exZ z-K6ZK&E(6;;>o_r&2NxzjNjzHA-_d^EBiM1ZD$H)ie*Y^%5f@ss%mOv>R=juntfV* z+I2d9x^DW*^vMj)49|@AjMq%cOykV-%*8C>tl+HSY`|>hY}@SO?Cl)+ocNsiT zT+iJ4JTOl;FFS8LA30w(KRCa;fVzNKP+f3Yc(YKq@MYn25qFVyQFqa2F>SGVac=SV z64{d2lG#$oQvOoU()VT1GX1jrGIIIVa^>>K^1%wm3g?REipNUwO5@7R%H=BYs>rIz z>a*3itKF;XYoImyHTgB<+UvE-wb8Ysb*y#nb?tTU^|bZo_4)Oi@8sVlzFU3||6cUH z@B8)-_z%_}sy|$R#Q%8rNC zIwUa=84xy z+DYrl(#hQ^?Wx?U{b|f;?djLk^PfaNMShz74EtI1bKvLR8O9mcna-K-S=L$S+1fei zobg=w+~qvsyy1NI{P!=4U(&y9entJN`ZfOR=LNxq(1qzm=ta>*|Ha-V#wGWq?xp`_ z_GQ=Q_bccX%az)d$5qNz%hlr5-8J2{{I%oto9p-2)7RI(DSk`+w)y?)clGZtzt3)n zZbWY^ZeHA!-HhBE-QwN~+#278+!o&U-|qdv{KNA{|4-nbw|_qU+4_t6m*cPYU%$WE zf4lzvxP#px?lkVa?=tQ>?$++X_bm5n_a66Y_igtp55NP{gUW;3L&`(T!}7z!zw$~R zvj8Cfqym)81Au-A0H{v@Kz0rQFgrZv5tu(LKd#(^A6KORS(AR;|0z(&KQ#;rg+Spj zIQ+kM6f{)$<0=#!j*5YbhW<}^M3@-pnE#yoW8`05!7vB}hK>S9`7f3KN7ch801p*N zeY60g2f%nB2p;I6AE5fLq#-cmpA4b@F8@RVf}um8kIhPtSq?A=6!fTofd)tUN8=Gg zVQ>^wG(3C)bUH$M-T)#7J`qtF9c%xlW?~XjMt)&2uNE>{K4Daws>)B~5%T7f_jgK?TkZXVdwLG5n+}q@$@?@hhX)`!j^y4w>H+XY z5d+NziJJuSQ`jyBab@b&Sn<%R+e^xFu@|az*bfNnZX|6cS2>2x2=}RN_i5>Ft_;B^ zg$~I%S9;wxHK!!+^y`%0XQGkF3930GMRkZcwg?a#YL0?rZ;ghB+Yjg06O`fc0ECZz z>+4JIsw@CUA@Mwce@x(YJ*H8zqmcmWG-V9EUyqT!3#YOY)#KAqHzmCyWt%x9lW7De z_@BD~`6e@JJX#e#F+eyTCzI?uZCz}kxMWj5dVpv9E}_1iGJQ)Hr!i1BMLs06r|n;T z{=?mqWLEA$Ec6ak+o3n3x9Y{K#(H#tmy3q0v_ct?=MBVs#D|C1f1!*s;f40`(jG>I z$0`Tyb+`+eOxv2H7zuyAm{e@{bYZDwXwHXugbi|o3js^ zZw=s!UfXppXQO_K#X5hx?UJ8)^3s$%O}x39d+`A5xOiOGL|VM~^AUfFblj0Ca(lniQu$TL zysdO8pK)(G8zTGX6=~~dc6uZHmOnWDVxFcsQ%RU9C+=Nw3ZYTLPtPthe@2Fvm{<49 zf3Vz@#&@Hu_iA^BSf*sQ+9+jz_7xXK#3rS;si}TOnmZF#)efZ`H{FiqIg_28l^mb= zYUq!Tyzewb&64<|N`#R3?O-gwHi5%fKfxVpZ{8dYK01l9fmv=&sWEz~pv{Fg%(F)K z(YfvkL#8|DZ_Q{1!+fWsGbY|hwc_$TY6@QM12VT3wB7FCcDH{MuDR~a=w(%?|2&tc zQ7G3L$>yd!z#C6ac9};nGAF*Jt8X_BGkKrRFWo@9s@su#$rX6})2cAsaAPr6s#8)S zBV#)F{kTD#6Vy-B{U zDBe^iu^aD(k{ollU*K5Yw*OKS`@u}rqOd$es(P*7T7*tQX2xPiUtLuzu9{4l$~GI{ zQ7Qvh5Cg+CZl-z@i|_FvxY|jYz~Wk;!%wM53pK3C-lWGh)B<(&YvRc9Knn&I?bv|zp!s-{*ta>`_y}iwhbsVBY zWu`}~Tf!`zp0TQg;W2V%G2_Umn@Xj;TRo>Vaq6LdHNA+Ve*OTID8vh82kX~G_1N2! zhsjr~WL;d79#a=?R;85N2{q>sH>!#u*rXy8(}oeo*xgfx50TKuqv?_QTya)3 zI!?ljS36v?c3e1u=r~sy1-D*Xgr`xBxkod~w`c8J( za$w!C*20X-lAj=v*_#7ZQ_7?ngBJTyn>zD4cGa0;*Y<-{hTQH#Np@p#!a1J$wwC%z zBQFh0gJt?`Z*4@Y{EfiUh|rXsfH!H%pU5UMy15nCuO%_2>Qky(wm5|Vb<23$@<)za zgQam>B$28M0sd(ENMW8EJL9jL-^ZWs7r*I=5lp82sUw&`z&EkUi4)Wkbc?T5GIylM zpYp>?uSFW)OKoORxU*@zpRnH@$1-p**b1FIqw3L!!Y=XJbkfUnp%i*1 z&&RLP&&ZPA-zXAeF2uS1eV4^+b)NA-y@g?+x0xoLW`I{yC2W`)w@Jhz*MznGHu;?1{{P)X zy_mS$U!5wa9&ZcRe&%g*vN85*aEy?xYaN?u*FePJPiPE6f1zIB*H^~XJ^9i=cQcvZizNoC4r)|*Muf+H_a>FGM7j8`8Ljls_XU*$)S`$x|wth7Kb zb@{}6qvm3cSg~2ovl`~Sl*T$k1-|v#Tou!OT-%nYiL5Do#nd1v$76<+A1t)?BB*%$oF8tE5ZJMuRzcsQPgv z#DA*|%G{T^nBz6FO)|C3eS7SOQkrhII}XRRl6Ns<)TB!7QhqSX%UsY(_^Ys{ICH7D z({HJ~)79K2Pwj7~!FKa1rZ`uqO8!-6%0MLE2q|9~wL|ks1+#mtXpK+hCwch$!PR7K z$ry)Mwk11NE(YA>1Alae-40&6jRdpNtNI&NZjo1+<4hc$3cOWMM5jFdg^yO^+7&(Z zE=)G^c{~NROQJrx!zzT$Rj1Z`$Y7ZFSjk>8w*F77Y<}f$h9Z2kH=A+%rpx}2s-(0k z)t|ZSvg(tiDdCclIFSec{q_QS*w@n5-+AGLM3`3*^PfXAou93PT z*C`dUzb> zj@V|AlrWUiy7IW!s|;#Rvg(;9N}oMR_TY?{U~6&k&9X4-PVBUxKi3;CTm5Q(q#@7HTT( zL&&<{9Fm>BFX}$VxNeRgYLzEzB8OA#2cmRRnaQr5c8Ku2bHW&2t)y79s}h@g+SmKL zZ*^xLORtl=_xLOcrWSCGm$@umqHmLLj0jH)`l_aqMRlu-rM_7#YW{tT*HlUo8vR>` zr79{K#NEXd{BeA#b{X4ESY$2rggirHn4h!5z_Nx(Ys5cC`^TttE0sykTcKH|xhZFU zJ~yn2y%e%9^TQdR zNhwc`A>614o7=2?@JlP(Rino)k`w6(Jd%rZZLPxB7Y*COyoAu!h7e z$UDZqt%sJ4Br@%yIqtCKn2P?ev_mkam$VukgLVZ&hhC9%#y~?ys8S z%Km9t7CL2)LL8N5lq>g_?#ru8-9`omK6)O)@j%s-;-hrD<4zMM~K5IpmY1FoZTI2f%^(*VT-SEfb8W`v2^D**8zehS<7CRj*mO3eZnd#-u zol>x@NUxgFD^l;=?v#t8IZ;rsZ$XS&^);TSQ+?1Ik>R*8V(ES3c31A-G%b~TR?a;u zCE3PDaV==}n5M-h89=Y^Fde=9$$BVQKB&)$lI-tCmMw0Z@9?ev`(GHEw?mBlc=OhS zEO~0qLfL|sC&k=$zi!&4-NSiMGY|d$>UtI|39PRWvKcO}r-o~-+Xz+s3@mo?NG`J2 zYKLe8x{Te(=roqOx5`SC>9EHor=Tu*#lxfO2l|t z&a#uRm`xIB#s`^A2(k=+ChM3cD6XB}QNkgYsp!rt6;)Zn#@EfVo#g!et<2r&%iUI{^=vDOQ=HPgaqpd) zro;+y8vc!tcJy`qKh%cu3fOgEH?f=*{?H1f@OLM*gYJRrDK9jaRGEtB1f6X5lBbfq zqOwDod6&esrxhKS=2b_{dRa%sLX#GmcQd(=9P z_eeCbl`@D_33Q7(XT~};X)~fc!!>p7Y_twbhW20Wj6`T@adR&0Z07{A3< zo9h-Qo0`I<)L`muDAcdS?X>fbKh!z@yylxiLYp6}Q*_|(54O~bllsh6)LQYssR=^_ zcMgBh-*ot2omi{1QZ2tz>&*YI;8dcQb|VE-`%~g!P=!A^pqk3tlq1E=mywS5GgAXY zHK_@~nCb-T867Qo3#ML$yVxFsRBhu>RYzPTNuY+06RqzT>e(QMFL5{-h=4 zucN;dm;3V#(@JAh6&P0{&`+XLPA7y9wd9QtMiWCf%I1xNHlKAI^Vx?4mpZ;_N5eOj zH4RMtZgPXAkz%M`8)KotG#~u=qwx3nFF9QClxCfli)~0zc zg#5T8CdE!!Q(0-66Tb@Uy_^(gg!VDnyLVSvd-#L1{eMhinQWTIqXU?j+0tBPE(uDRaL|KtJqsAxKAn189F_u1G5rMigxN6n&s zWuj3%8qZEK_;xg9xd3NPDO*`+N<~eB#ICIjFT?KA+VNFUSstg&a(+o2<$juxUQa=+ zXXC&=br~@?uDMlqMs{qkndk2V=j$8wu@`IC-Sf{ryESwUs;)p>-sssQ&7`?U3)p&G z%#A36gEvD$(#BK9%IL1da+pfzrfciM={u^Z4;b!*-IgJt8mkwdUD~s&vP_rimakL2 z_kP#UR7!}uQmiv>@B#qpE06K+4pBx-h z8Tf4d#ZxXnkVBPu@2)C`rI)T$zB)0N9hzb6ghG&D37s zv=}J(;Q9cRvoSaam$4e;oD5}0S5DM@jSO2)N3@Kk>ao^;5-hl3Nzu74%}$6~opzBN zSy)}BbU$q=d+l!J7YyTLbe<@&s2_S>D$(aL=sBTlfyA)hOZ(ZO0+rQJ%-4#+)6ZOIgD2Y$9ydtfCzG2Se^eImz)%=nSE(r#SMis-mHQ@d?Xd zuekM-S@?{|J6@dtky*bay<{VMgOUXAL;WR5&K>7p`J;-wwuQrh6%4KY>y=CgZCvW51 zLUPi&-5sDTY;3oC!CxO^22{S~HLGpcsnBtmr_hf{%F|V$wn1<=>g1bhFN#ER$*JzC z(yP81*Ty2Zo7g$8SB)~AGtB-uh-e@~o_6RvBNy3PJx6?(bokWUdSfYAS;P>|I&JnT z0%eTOCrO8^YyKEhhNnbg0IXxyOFZBJHDQ%{+j!aYo$ zow;GU(AAUo?iOC7Q4!DDo-lHv(e#&c=)dtM>aMzysEv-LGKD^VR%o;TqdDsFX)1Fz zH14!?1Am@5LJLxL`K_pHR(PSUVTWk8KASUM*jilbP$6u`I*6Wg!m{pTM#c!U=x5h# zyT4jeFHb@kSA{X>XF0RNsNWylaUCvhOm{eXOT4c(agSo7=u+=&+7>EpVvo4>PGum3 zxU7#9RfNCqRQfF-GcA7NmKS`Y)^c>r^VB45$#2YK&grs7mPp~3J;!T$y1_6!P0ui? zaft)%5-v2mgu%4z%8ojp*A)g zBs8vDzH%ypZB0z(2t|2^&rvkE7;`DFD^32WwwCozr9ZJdj68^!C{3HGHcKSA$#qt0 z-QB#zxaHls64x+&*0IlUlVLGA&>3XbEZuR%G{O`mGVT<)lxv2Q5@wrlPBa@DBv8fK zCf`G&0RI%vEU+*!{+bTE@c_+DY$-;?RK1-EFS8*zvxlB=QF`;$;~UsM=K+wFrTS5l zTQEyjI&L91m3~+(p^>_O)g)ANhtu%%FcUsx0#N#i^(#)TGUMec&f7sJ$ z;7wP(Yifk2kvk_O_4ET|)CPvJ%YvhldBLJGucqUQh1G>h{(3Wt6(g}qc0E&fbTuqY znecU~^tv%sgkD$2DI1kdDaOiQ8XxzjoqllCbytb6lDK!TF*OV?%uwY1)VsXkdLBRj zeznBQIG|Gq+ivieX2sK$)0~Vxu6Kj2*28A+*rNw|OJ zSLKSYwSUs99(Qpk)!i3x^ttg~?i;bmaPa9|y>7Xc>B}*kVecO861s4oy@oAi1ou!v zSs8retsm!=kvtO`600Re1)+99F@JYfJx_wVABl6N>p5-+phtX4HwqGmjDtE=wk zws&!_N~VX%jV2?hx5(8mEj|e^!fpa(^ZP^xWG8zS+u$C^1UphB#Uhv(s2r$Y$yDRnY&xUkD7O|q zQy)bRG``Y3v=mMqQ(%Lytp};kZzU4|bi3Bj1 z`z<{7TCw0mOXh7#BS-%G9>O$-ibnMf`GMh!LzlwW8GA% zmWplVZ??<#_HV7R)4oQTs4+05+?F+kMes>`68!llej|1szO0}4gYa8isH7NgxNc|E zfe*HMN85yP%J)`e_pi~u>kihGYALp+SB%U$Rj)%T-3D4eu z8vBISt(7a^+KC(>RP*hFTcGwd+p~y zqp?OXGn;0-tRtT00{`7QE!d8f?>N6uIa8HIlYbqerKg|2&r}&>?ro8Hk+{F62P}5z zU+me{3EFnVYg)kj2MMdGs11-8;+zZL%4MCJC0ZsU5uv)~Ioc_^dkPoSyst__RYke< zh*!V062phEvr~}f*Y=v`zn!?hG%1uz{2IDVa^VV%p0V7qc+tg$JC{+@MB%x2s!6K2 z)=}M#9vN+fH6nGcKrKi%SgQDJcqPEp&l|1J%})6wKT=`z&3#et9cOsb2ej}5^YsJM z!U6KX4*=;-{^LV+pXP!s!rzs*BF|&bMd9kpYe9{QU?OtPcvzY zl(i0iFA?cfwyKr~Lu43rnB`W=NKX9ne1Q6Dm%@uqEN+D>b8^2ZY)YY{DzbdBg8XIF zvN1^Ss*I~`?MiLY5*ANfDXBYZxN3%1Az5=BG_0NS(j?=}K z6Q9*oH&sj4!HWYo3`XsGa<^spN|ixM<%zmGWAr&-FyF7LWZcTZ#9!g#0C{sD)$f>e=eNcUSka^ZywL0lbd$a&Yl=jY1rer| zPOySx0hKqIC3m;Z`Gup3#T3`<@MZuE2>>+B?g;<@ literal 0 HcmV?d00001 diff --git a/src/documents/tests/samples/simple.txt b/src/documents/tests/samples/simple.txt new file mode 100644 index 000000000..6de7b8c69 --- /dev/null +++ b/src/documents/tests/samples/simple.txt @@ -0,0 +1 @@ +This is a test file. diff --git a/src/documents/tests/test_migration_archive_files.py b/src/documents/tests/test_migration_archive_files.py new file mode 100644 index 000000000..534a5b499 --- /dev/null +++ b/src/documents/tests/test_migration_archive_files.py @@ -0,0 +1,175 @@ +import hashlib +import os +import shutil +from pathlib import Path + +from django.conf import settings +from django.test import override_settings + +from documents.sanity_checker import SanityFailedError +from documents.tasks import sanity_check +from documents.tests.utils import DirectoriesMixin, TestMigrations + + +STORAGE_TYPE_GPG = "gpg" + + +def archive_name_from_filename_old(filename): + return os.path.splitext(filename)[0] + ".pdf" + + +def archive_path_old(self): + if self.filename: + fname = archive_name_from_filename_old(self.filename) + else: + fname = "{:07}.pdf".format(self.pk) + + return os.path.join( + settings.ARCHIVE_DIR, + fname + ) + + +def archive_name_from_filename_new(filename): + name, ext = os.path.splitext(filename) + if ext == ".pdf": + return filename + else: + return filename + ".pdf" + + +def archive_path_new(self): + if self.filename: + fname = archive_name_from_filename_new(self.filename) + else: + fname = "{:07}.pdf".format(self.pk) + + return os.path.join( + settings.ARCHIVE_DIR, + fname + ) + + +def source_path(doc): + if doc.filename: + fname = str(doc.filename) + else: + fname = "{:07}{}".format(doc.pk, doc.file_type) + if doc.storage_type == STORAGE_TYPE_GPG: + fname += ".gpg" # pragma: no cover + + return os.path.join( + settings.ORIGINALS_DIR, + fname + ) + + +def thumbnail_path(doc): + file_name = "{:07}.png".format(doc.pk) + if doc.storage_type == STORAGE_TYPE_GPG: + file_name += ".gpg" + + return os.path.join( + settings.THUMBNAIL_DIR, + file_name + ) + + +def make_test_document(document_class, title: str, filename: str, mime_type: str, original: str, archive: str = None, new: bool = False): + doc = document_class() + doc.filename = filename + doc.title = title + doc.mime_type = mime_type + doc.content = "the content, does not matter for this test" + + shutil.copy2(original, source_path(doc)) + with open(original, "rb") as f: + doc.checksum = hashlib.md5(f.read()).hexdigest() + + if archive: + if new: + shutil.copy2(archive, archive_path_new(doc)) + else: + shutil.copy2(archive, archive_path_old(doc)) + with open(archive, "rb") as f: + doc.archive_checksum = hashlib.md5(f.read()).hexdigest() + + doc.save() + + Path(thumbnail_path(doc)).touch() + + return doc + + +@override_settings(PAPERLESS_FILENAME_FORMAT="{title}") +class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): + + migrate_from = '1011_auto_20210101_2340' + migrate_to = '1012_fix_archive_files' + + def setUpBeforeMigration(self, apps): + simple_jpg = os.path.join(os.path.dirname(__file__), "samples", "simple.jpg") + simple_pdf = os.path.join(os.path.dirname(__file__), "samples", "simple.pdf") + simple_pdf2 = os.path.join(os.path.dirname(__file__), "samples", "documents", "originals", "0000002.pdf") + simple_txt = os.path.join(os.path.dirname(__file__), "samples", "simple.txt") + + Document = apps.get_model("documents", "Document") + + self.doc_unrelated = make_test_document(Document, "unrelated", "unrelated.txt", "application/pdf", simple_pdf2, simple_pdf2) + self.doc_no_archive = make_test_document(Document, "no_archive", "no_archive.txt", "text/plain", simple_txt) + self.clashA = make_test_document(Document, "clash", "clash.pdf", "application/pdf", simple_pdf, simple_pdf) + self.clashB = make_test_document(Document, "clash", "clash.jpg", "image/jpeg", simple_jpg, simple_pdf) + + self.assertEqual(archive_path_old(self.clashA), archive_path_old(self.clashB)) + self.assertRaises(SanityFailedError, sanity_check) + + def testArchiveFilesMigrated(self): + Document = self.apps.get_model('documents', 'Document') + + for doc in Document.objects.all(): + self.assertTrue(os.path.isfile(archive_path_new(self.clashB))) + with open(source_path(doc), "rb") as f: + original_checksum = hashlib.md5(f.read()).hexdigest() + self.assertEqual(original_checksum, doc.checksum) + + if doc.archive_checksum: + self.assertTrue(os.path.isfile(archive_path_new(doc))) + with open(archive_path_new(doc), "rb") as f: + archive_checksum = hashlib.md5(f.read()).hexdigest() + self.assertEqual(archive_checksum, doc.archive_checksum) + + # this will raise errors when any inconsistencies remain after migration + sanity_check() + + +class TestMigrateArchiveFilesBackwards(DirectoriesMixin, TestMigrations): + + migrate_from = '1012_fix_archive_files' + migrate_to = '1011_auto_20210101_2340' + + def setUpBeforeMigration(self, apps): + simple_jpg = os.path.join(os.path.dirname(__file__), "samples", "simple.jpg") + simple_pdf = os.path.join(os.path.dirname(__file__), "samples", "simple.pdf") + simple_pdf2 = os.path.join(os.path.dirname(__file__), "samples", "documents", "originals", "0000002.pdf") + simple_txt = os.path.join(os.path.dirname(__file__), "samples", "simple.txt") + + Document = apps.get_model("documents", "Document") + + self.doc_unrelated = make_test_document(Document, "unrelated", "unrelated.txt", "application/pdf", simple_pdf2, simple_pdf2, new=True) + self.doc_no_archive = make_test_document(Document, "no_archive", "no_archive.txt", "text/plain", simple_txt, new=True) + self.clashB = make_test_document(Document, "clash", "clash.jpg", "image/jpeg", simple_jpg, simple_pdf, new=True) + + def testArchiveFilesReverted(self): + Document = self.apps.get_model('documents', 'Document') + + for doc in Document.objects.all(): + self.assertTrue(os.path.isfile(archive_path_old(self.clashB))) + with open(source_path(doc), "rb") as f: + original_checksum = hashlib.md5(f.read()).hexdigest() + self.assertEqual(original_checksum, doc.checksum) + + if doc.archive_checksum: + self.assertTrue(os.path.isfile(archive_path_old(doc))) + with open(archive_path_old(doc), "rb") as f: + archive_checksum = hashlib.md5(f.read()).hexdigest() + self.assertEqual(archive_checksum, doc.archive_checksum) From d90080f325f0217d2f63ff6b58f4d7a80de448c5 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 8 Feb 2021 22:49:01 +0100 Subject: [PATCH 065/225] only move files if necessary --- src/documents/migrations/1012_fix_archive_files.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/documents/migrations/1012_fix_archive_files.py b/src/documents/migrations/1012_fix_archive_files.py index e95715265..98794aad0 100644 --- a/src/documents/migrations/1012_fix_archive_files.py +++ b/src/documents/migrations/1012_fix_archive_files.py @@ -163,8 +163,9 @@ def move_new_to_old_locations(apps, schema_editor): for doc in Document.objects.filter(archive_checksum__isnull=False): new_archive_path = archive_path_new(doc) old_archive_path = archive_path_old(doc) - shutil.move(new_archive_path, old_archive_path) - logger.debug(f"Moving {new_archive_path} to {old_archive_path}") + if new_archive_path != old_archive_path: + logger.debug(f"Moving {new_archive_path} to {old_archive_path}") + shutil.move(new_archive_path, old_archive_path) class Migration(migrations.Migration): From 7621e10840b3cb5ae914e68ceeec5cfa4f9aa1e3 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 8 Feb 2021 23:54:07 +0100 Subject: [PATCH 066/225] only move unaffected files, regenerate affected files --- .../migrations/1012_fix_archive_files.py | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/src/documents/migrations/1012_fix_archive_files.py b/src/documents/migrations/1012_fix_archive_files.py index 98794aad0..ab7e88a6a 100644 --- a/src/documents/migrations/1012_fix_archive_files.py +++ b/src/documents/migrations/1012_fix_archive_files.py @@ -102,11 +102,17 @@ def move_old_to_new_locations(apps, schema_editor): old_path = archive_path_old(doc) new_path = archive_path_new(doc) - if old_path != new_path and not os.path.isfile(new_path): - logger.debug( - f"Moving {old_path} to {new_path}" - ) - shutil.move(old_path, new_path) + if doc.id in affected_document_ids: + # remove affected archive versions + if os.path.isfile(old_path): + os.unlink(old_path) + else: + # move unaffected archive versions + if old_path != new_path and os.path.isfile(old_path) and not os.path.isfile(new_path): + logger.debug( + f"Moving {old_path} to {new_path}" + ) + shutil.move(old_path, new_path) # regenerate archive documents for doc_id in affected_document_ids: From e05735bc0faab955f18c26a6ede763abc4ab15a9 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 00:00:46 +0100 Subject: [PATCH 067/225] fix some test cases --- src/documents/tests/test_api.py | 1 + src/documents/tests/test_file_handling.py | 1 + src/documents/tests/test_management.py | 1 + 3 files changed, 3 insertions(+) diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py index 54d2b292d..c4cca8fdc 100644 --- a/src/documents/tests/test_api.py +++ b/src/documents/tests/test_api.py @@ -146,6 +146,7 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): self.assertEqual(response.status_code, 200) self.assertEqual(response.content, content_thumbnail) + @override_settings(PAPERLESS_FILENAME_FORMAT="") def test_download_with_archive(self): _, filename = tempfile.mkstemp(dir=self.dirs.originals_dir) diff --git a/src/documents/tests/test_file_handling.py b/src/documents/tests/test_file_handling.py index b24f52aa2..ae25c0f12 100644 --- a/src/documents/tests/test_file_handling.py +++ b/src/documents/tests/test_file_handling.py @@ -558,6 +558,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): self.assertTrue(os.path.isfile(doc.source_path)) self.assertTrue(os.path.isfile(doc.archive_path)) + @override_settings(PAPERLESS_FILENAME_FORMAT="") def test_archive_deleted(self): original = os.path.join(settings.ORIGINALS_DIR, "0000001.pdf") archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") diff --git a/src/documents/tests/test_management.py b/src/documents/tests/test_management.py index a71baa8ba..ac119a431 100644 --- a/src/documents/tests/test_management.py +++ b/src/documents/tests/test_management.py @@ -106,6 +106,7 @@ class TestMakeIndex(TestCase): class TestRenamer(DirectoriesMixin, TestCase): + @override_settings(PAPERLESS_FILENAME_FORMAT="") def test_rename(self): doc = Document.objects.create(title="test", mime_type="application/pdf") doc.filename = generate_filename(doc) From 1e5a4181913b7a65a3b59bc0a0f0735988850071 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 00:01:11 +0100 Subject: [PATCH 068/225] more testing #511 --- src/documents/tests/samples/simple-noalpha.png | Bin 0 -> 6414 bytes .../tests/test_migration_archive_files.py | 3 +++ 2 files changed, 3 insertions(+) create mode 100644 src/documents/tests/samples/simple-noalpha.png diff --git a/src/documents/tests/samples/simple-noalpha.png b/src/documents/tests/samples/simple-noalpha.png new file mode 100644 index 0000000000000000000000000000000000000000..9a76e76a0b0f7dc86408dcc09b809f3130201f56 GIT binary patch literal 6414 zcmeHs^;^?l*gr53K|%>(gh-3hT~Y!QRGNWwD2*TrgAqew^b|!qJybU!A|fFJ-#=U> z0q^lv1R?N2;ra0CD^wMs)ORqlj@ z(d8s%1V1d*Hc8gjS5YN)CJD(3Velim@ikIXHTkxNN;Gc9|5~bzn${}sY2eE8X|KfQ z^p+K-&f@Cq9}dMw+tXeZbq9TJpC2*M)7x`%Wvy7*6%uiAA%#UbuHT6N8K9!BPI5qe z;|4_!xw<;f|Efi;Vv=*Wu@XyteSNfY;{|>rieBP_e!b^%&*uuqKP(rY1PpWM=4|rz z)c4>4f5=8S2zZ2W#QtlexEtIS8ZgS;MuMHyL%k4YW)YP)Fv)y~hak0^RX<)&uSm@?po^Gi~ z{mX@pb?!xo$)rt*NvV|!=hj431?92$Wl8EkmG?Yaix(P7WzoFRh!#z^i8UmnJ_kSDT^0+K@Y$Y%#uaq>AU3nIvikevUoAg&a>`%R zRtzg?NVckdU7(dCa)Q`WS6{AAr!}j2K^W3imLuG89?ZGR@$6T^+D2)U4~x%yOSr`k zFFC@h02vt>yP~h24wJCOxZ?Q6d7Tdz*Q*(nHowPAJ)HBV*J9UR%QX>G4L`f#e0|1y z@W%;v^uhK}r%OzQo+1Z36AJ)?ZMEJT{eLPbV_oxfQYY&DDvTN8U)A|+$p@U285^n1 z5iV5I6#a_S1bK8)WJN`H*GCHBZ_008zh34%lp8_I!S^V!=kDm)dEJq0e;s0{`vX6R zoL7!srW(BbCBM9Utxl@E4A_s?QnayPyuOo@)9L;atIUh@vz;dMcBO-V_LoHaOTQf% z|H)m#J1;Ejb}YVo+Tw9*aQz@r@Z3V@TXPLRa(NV-6Bi3r3OLDzy)DY3yXtX78xk5? z_j=8s%tp0P#kJVHM%phm!qHMu(ylF%oc5Nf#%XiSY>+GCKPjybe1Sphm=$7tf?=Lc zRkXHj{byRMnu=(z-*rAEO5Oo-P=4FTk7GrWXJ|Q;)TNTdpKCYO-`JY1suNLEoLW%e zLhQ^u0k`1Rl<|eP!tL+32LIz^QpLE{RVTLqtWkjxglzKF&dmLTA8gY1NygornHJNaGiToBH>Qg69!ffJQ)-1jFuj0h!%(Ns4px`FE4+U$2sk~vUK ziTB?dEjGUfO4=B3il$~2wyX_!$f^P5e!1 z7L!42Ppu8*c}!LVjJituUuh9PSo+Pxz+hBv*M2;Unq7-60mvF`3_L6Q847MkDL__G za1(#H0cz9c_fg2a+Tg1;=NY`zYT5ENDa(&-WmqC(s z@DdBZUxtL|J6*Qyt7*J@@JD$ArAqXx*}=n4(6r2EZSd>HSZPzh>9^h>7HQW} zs>vv|iu(fM;&fzLw?4K-WpgN-N?~NoTkH=*LSzCj3CAuCrw1!Fl&BCShmd9M>wI;2 zpUqEvPgwG1?|>q&^rcGlT>N!RH_q21c8sAcwD|F@;;=0vBL7Gd`1NzDt(PGeJ+`$)-X9U)TIw?<}_PWo&;_`a&o zjc=uGTf!cH7Vj1v4sDNNf>Er>^yr~_KTC9uY!0?X(Z?%4Q>3I{3Ld;;m>X;M?e)Jl z3R3PsX}58`QW%c=}XmEQLalTT^jf2yb^vSd=&7+-(=SumOb<#SbZc*pld{=01 z75ysqm~C9XCMJuzI5$#g@OT4X9KL)2fxm@~1fxGvJ@7`~gm63-;xQ1OA^!CABer(C2#&pzl zdGP@Sd&J=`D5bC;w@{DGjxtc2P>Fk!tNDTNNk5zF*EfSK0C|i-) zd^ubopC;>+;v@QFr9_k>ham>lFce~9BhJUN+5fE9gx`O{Hk)712I+rv_+1mTw=wau zBr9MT`ql6+gQK%s-Bp8{7t_Nta$bv(g=L%%&|S=K-HAfxROwv2`Qf;C3M!0oek4 zw!%EeDBA##csI|y(kWB;@jWL+?8*&3os>xF41CuZuRV=axJz6XI$6{<#IN4)czfEg z_ef^O7eZ~EXPck1*nZsz<<#AwC^z*CC~Ue?;3CeowBcwiFJ1m=7(0A^om)gLuXpd% z%Ls?>rFea#-&@q8>(@I@2p+oro@wH`?2((U8qBaUjK}{GMwjtub>M-H4f$HbO_+`= z)$0m9v=<)87jOk$v3@g(A5MRMylafjclK+#Q$74$t0c{=Spd7vS9-8E@p3S@v z28yUR)aXmCmn>$FW~rGPX_%6k4KUQ1`u-9Ur+tV>PmKNgS;EoIfAKZowEO-yuNFJJ z?pu^ZGMBXI>0?{ukhpxStD-)eriXf^N{U(8-@D06R^1O`m_+2`w7y8YKYSROqB$YK z=DSLU9eb+%@$M2%z^A9Agipw{?5EgI1E3AVm5x%HBp)*N-vrTjSLo9Z(-(rreeja^ z7`+`0o94-?Wf&YiJpBMV;)r0j@)XJgwc+LH2W0@~r`fMXHe|eEG za(28^Sd(;^z9>1B_QV$l$pz$$DQ(e0rvLP8k5Zh|vgfz`a^L?bf+NVu>B{zG4K4K` z@YIK!e~0qMpn;V~u(!A3pb~uG5FhWEY|k{Dtqx>6_NDx=ZM`B{E@D;BOwMlrC^;E} z!O$Q$#l^)5m(ZEref@6JTEBzb-$Ht$$0*+sij z*Ms+LaM#3blti@?1^*AWptn?%qJxS|ey#(ZaiA1Wr(~m>JK|c{o-?yMx_}?P|BzcUBWSf5%50 z$7;QWrIxc;QmxBx5)M6O@}^;fUmtxDd^gn`FlW&mLN*YoIq3&Hqf_E~Z#ghu>!sGG zA&4=hpz{;u%X34WKVYrv9cO<0{W_X$F#=ZJq0kA~B;C;ll1#pNXv%uorym`|>Jln+Ab!&ZTCL`na zTaKgvZG5QkYVIfX+E#Pk1)4xM9&cUQY8>%R57X%*bp&jQ$|z>t*(Q*4zwp zGz2}>7BH6xJV!66zilgeziD7#z`WW`EWJG@&0!08c%wh_C*=>~hRmm9scC7)O7oQC z@&uqVYaVLdwzQ8!ESr6>CeH?bh+W2!%8`VnjB30i*3sw}x=)1ZG-Wr*oW~_3asUG&S1{zxY9Sf3Z814W9Ebo}ORllX}Oi z9&$|}hl3HpV+HX+1D{7wC*3jijXo(R>&dM^HZ)_IGf~mFVEOhOthXWHo#Zqr*Tf-a zxsVXNM}ihO6HkCvHZO!1gJ2+w{%iu(de5WE^3KE! zaPnm>XS=kx8~gXfFmyp&W_Y|?I@QZ$85k0w0;tfm$V_#d!iX>CTDP^dsr zDgi}B3+j9S5P@e$#x?!GyqGm04`mx;Fxv^p9a`4lp-$&))5Qk zj!4su=L)}g0Np+qn^5pxuLMjaH=U=;QQ7-DL4X^@We5DSt*tHods&(#e6$n(pqPn~ zG5on*p7nQ6&R5GlAQdHY*ltZ$yMHL200FSlV>Sp(N2pn)dr}o-bQIa>Kz-Wb?8-ss zK<+@Umo91#eP5uTjb=IB52aO7c^(mV0}73LQ0~y1OwA%$_JjP9Ce2Uu-^~@p}@ix|9k_>mvB928$ z4W+Nd>wUWaJcnG)<`5iAZ*Da?42@oYyNe8<`$2CHMJL;W+&b*fOZt-SY+ zb%;b*DkVRYOgQ-;ave@Q5Zd4UX#62GbXwWj_Ro5(fQJbH@GfKpMp_6IN zKhcgAVZV=L=(IT6{QKmJ8<-_f<4;P0{zCXUlO53Wzh8D`-HYqUjctDVCi$y+6dium z$kZOpX!OZ4Y`=qP5a>2}8sbD@OZgO0ka_>sc%)k-*v9sNYno4K)6Y*E?M%%0xc=JE za&KpXcjQo7g<2dWD)X8bYa;h;C2A2nFRC79)3D3ST9Wy^xrOHQYziXuSmWA>4}jlY zf8}Q(-X(_L^{kx@I_njiA-$^?xTp^?qF>Ejg(QJnppOuCA#e3sJb5I3W%BJUDuaZJ z7pfqZg5baTnw8<%wFA{Xx1(k|z-!K*UIG_UV)af_m2+!t1S&0Z=T1Io_lLV(+xec< z9wj!lDsMDH{Gh&*58`)50FF85C-AqKV|L?nKs%C^ znIha(KBy4{X_IdYnN^B1U@|*Y8r7AlW;IZ{*N2*$o5juV!7ywu0^~4Hx zZC43Dd=VW+qAFIZWdDsADy^2Tr^&{Z!>C%t7m8TrU*BN?L!Car3tM13*x%T}XY29n zUfH+~PKe5$*4IKjve^ubA-;i%?(TY~Op;x@sXlUBqSC&ef``*6cVo2pi!w^7nw)@D zKy2Tp&Y+9ujnWncz6|O3dwG+bRnN~hG@h}n5*U(;lkpEI3Hnb;;eAIkpfz)oijj%9?=Feax09g9v_!8-`JT#Yz@p2YYinZluse_vY23 z_6*g~lww03%YM|PivANZKR&Fr>0kl$EDIL3@La zF6%OQT7q9cb|ml>7ngX%v+GdkTHQ;=g)9GWxE;_K85v2s4B_IU`$QWV`Orq`HW$~M j)8PMG{Qq+yCmbJAiV3Zcx7xrT97I|gI_f2AHX;85-b7+% literal 0 HcmV?d00001 diff --git a/src/documents/tests/test_migration_archive_files.py b/src/documents/tests/test_migration_archive_files.py index 534a5b499..5970a83eb 100644 --- a/src/documents/tests/test_migration_archive_files.py +++ b/src/documents/tests/test_migration_archive_files.py @@ -112,6 +112,7 @@ class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): simple_pdf = os.path.join(os.path.dirname(__file__), "samples", "simple.pdf") simple_pdf2 = os.path.join(os.path.dirname(__file__), "samples", "documents", "originals", "0000002.pdf") simple_txt = os.path.join(os.path.dirname(__file__), "samples", "simple.txt") + simple_png = os.path.join(os.path.dirname(__file__), "samples", "simple-noalpha.png") Document = apps.get_model("documents", "Document") @@ -119,8 +120,10 @@ class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): self.doc_no_archive = make_test_document(Document, "no_archive", "no_archive.txt", "text/plain", simple_txt) self.clashA = make_test_document(Document, "clash", "clash.pdf", "application/pdf", simple_pdf, simple_pdf) self.clashB = make_test_document(Document, "clash", "clash.jpg", "image/jpeg", simple_jpg, simple_pdf) + self.clashC = make_test_document(Document, "clash", "clash.png", "image/png", simple_png, simple_pdf) self.assertEqual(archive_path_old(self.clashA), archive_path_old(self.clashB)) + self.assertEqual(archive_path_old(self.clashA), archive_path_old(self.clashC)) self.assertRaises(SanityFailedError, sanity_check) def testArchiveFilesMigrated(self): From 0ed001c56ee51c9309e886abb0e739725b490ebe Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 00:13:13 +0100 Subject: [PATCH 069/225] validate move before migration --- src/documents/migrations/1012_fix_archive_files.py | 6 ++++++ src/documents/tests/test_migration_archive_files.py | 2 ++ 2 files changed, 8 insertions(+) diff --git a/src/documents/migrations/1012_fix_archive_files.py b/src/documents/migrations/1012_fix_archive_files.py index ab7e88a6a..f75ae431d 100644 --- a/src/documents/migrations/1012_fix_archive_files.py +++ b/src/documents/migrations/1012_fix_archive_files.py @@ -74,12 +74,18 @@ def move_old_to_new_locations(apps, schema_editor): # check for documents that have incorrect archive versions for doc in Document.objects.filter(archive_checksum__isnull=False): old_path = archive_path_old(doc) + new_path = archive_path_new(doc) if not os.path.isfile(old_path): raise ValueError( f"Archived document of {doc.filename} does not exist at: " f"{old_path}") + if old_path != new_path and os.path.isfile(new_path): + raise ValueError( + f"Need to move {old_path} to {new_path}, but target file " + f"already exists") + if old_path in old_archive_path_to_id: affected_document_ids.add(doc.id) affected_document_ids.add(old_archive_path_to_id[old_path]) diff --git a/src/documents/tests/test_migration_archive_files.py b/src/documents/tests/test_migration_archive_files.py index 5970a83eb..69c1b51eb 100644 --- a/src/documents/tests/test_migration_archive_files.py +++ b/src/documents/tests/test_migration_archive_files.py @@ -141,6 +141,8 @@ class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): archive_checksum = hashlib.md5(f.read()).hexdigest() self.assertEqual(archive_checksum, doc.archive_checksum) + self.assertEqual(Document.objects.filter(archive_checksum__isnull=False).count(), 4) + # this will raise errors when any inconsistencies remain after migration sanity_check() From c9511680b338500efdbb2f248ad2e7824de4d770 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 01:36:39 +0100 Subject: [PATCH 070/225] version push --- src-ui/src/environments/environment.prod.ts | 2 +- src/paperless/version.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src-ui/src/environments/environment.prod.ts b/src-ui/src/environments/environment.prod.ts index 4e38813b5..95cd94d29 100644 --- a/src-ui/src/environments/environment.prod.ts +++ b/src-ui/src/environments/environment.prod.ts @@ -2,7 +2,7 @@ export const environment = { production: true, apiBaseUrl: "/api/", appTitle: "Paperless-ng", - version: "1.1.0", + version: "1.1.1", webSocketHost: window.location.host, webSocketProtocol: (window.location.protocol == "https:" ? "wss:" : "ws:") }; diff --git a/src/paperless/version.py b/src/paperless/version.py index 37a1329f2..ae805838c 100644 --- a/src/paperless/version.py +++ b/src/paperless/version.py @@ -1 +1 @@ -__version__ = (1, 1, 0) +__version__ = (1, 1, 1) From 05f59e7d5e982c3d5f7facde4f7d68637a3f3e84 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 02:13:25 +0100 Subject: [PATCH 071/225] another way to make the test case fail --- src/documents/tests/test_consumer.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/documents/tests/test_consumer.py b/src/documents/tests/test_consumer.py index 280124f48..b96e4844b 100644 --- a/src/documents/tests/test_consumer.py +++ b/src/documents/tests/test_consumer.py @@ -543,13 +543,15 @@ class TestConsumer(DirectoriesMixin, TestCase): def test_similar_filenames(self, m): shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"), os.path.join(settings.CONSUMPTION_DIR, "simple.pdf")) shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.png"), os.path.join(settings.CONSUMPTION_DIR, "simple.png")) + shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple-noalpha.png"), os.path.join(settings.CONSUMPTION_DIR, "simple.png.pdf")) m.return_value = [(None, { "parser": CopyParser, - "mime_types": {"application/pdf": ".pdf", "image/png": ".zip"}, + "mime_types": {"application/pdf": ".pdf", "image/png": ".png"}, "weight": 0 })] doc1 = self.consumer.try_consume_file(os.path.join(settings.CONSUMPTION_DIR, "simple.png")) doc2 = self.consumer.try_consume_file(os.path.join(settings.CONSUMPTION_DIR, "simple.pdf")) + doc3 = self.consumer.try_consume_file(os.path.join(settings.CONSUMPTION_DIR, "simple.png.pdf")) sanity_check() From fca8576d80cc99312b15b4d512cd528e618039d5 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 19:46:19 +0100 Subject: [PATCH 072/225] archive filenames are now stored in the database and checked for collisions just as original filenames as well, unified method for archive version checking --- src/documents/consumer.py | 7 +- src/documents/file_handling.py | 31 +-- .../management/commands/document_archiver.py | 14 +- .../management/commands/document_exporter.py | 2 +- .../migrations/1012_fix_archive_files.py | 177 +++++++++++++----- src/documents/models.py | 33 +++- src/documents/sanity_checker.py | 2 +- src/documents/signals/handlers.py | 56 +++--- src/documents/views.py | 12 +- 9 files changed, 229 insertions(+), 105 deletions(-) diff --git a/src/documents/consumer.py b/src/documents/consumer.py index 3baedb5c3..acb3ad33f 100755 --- a/src/documents/consumer.py +++ b/src/documents/consumer.py @@ -292,8 +292,7 @@ class Consumer(LoggingMixin): # After everything is in the database, copy the files into # place. If this fails, we'll also rollback the transaction. with FileLock(settings.MEDIA_LOCK): - document.filename = generate_unique_filename( - document, settings.ORIGINALS_DIR) + document.filename = generate_unique_filename(document) create_source_path_directory(document.source_path) self._write(document.storage_type, @@ -303,6 +302,10 @@ class Consumer(LoggingMixin): thumbnail, document.thumbnail_path) if archive_path and os.path.isfile(archive_path): + document.archive_filename = generate_unique_filename( + document, + archive_filename=True + ) create_source_path_directory(document.archive_path) self._write(document.storage_type, archive_path, document.archive_path) diff --git a/src/documents/file_handling.py b/src/documents/file_handling.py index 64858de78..535aa3d2c 100644 --- a/src/documents/file_handling.py +++ b/src/documents/file_handling.py @@ -79,12 +79,20 @@ def many_to_dictionary(field): return mydictionary -def generate_unique_filename(doc, root): +def generate_unique_filename(doc, archive_filename=False): + if archive_filename: + old_filename = doc.archive_filename + root = settings.ARCHIVE_DIR + else: + old_filename = doc.filename + root = settings.ORIGINALS_DIR + counter = 0 while True: - new_filename = generate_filename(doc, counter) - if new_filename == doc.filename: + new_filename = generate_filename( + doc, counter, archive_filename=archive_filename) + if new_filename == old_filename: # still the same as before. return new_filename @@ -94,7 +102,7 @@ def generate_unique_filename(doc, root): return new_filename -def generate_filename(doc, counter=0, append_gpg=True): +def generate_filename(doc, counter=0, append_gpg=True, archive_filename=False): path = "" try: @@ -148,21 +156,16 @@ def generate_filename(doc, counter=0, append_gpg=True): f"{settings.PAPERLESS_FILENAME_FORMAT}, falling back to default") counter_str = f"_{counter:02}" if counter else "" + + filetype_str = ".pdf" if archive_filename else doc.file_type + if len(path) > 0: - filename = f"{path}{counter_str}{doc.file_type}" + filename = f"{path}{counter_str}{filetype_str}" else: - filename = f"{doc.pk:07}{counter_str}{doc.file_type}" + filename = f"{doc.pk:07}{counter_str}{filetype_str}" # Append .gpg for encrypted files if append_gpg and doc.storage_type == doc.STORAGE_TYPE_GPG: filename += ".gpg" return filename - - -def archive_name_from_filename(filename): - name, ext = os.path.splitext(filename) - if ext == ".pdf": - return filename - else: - return filename + ".pdf" diff --git a/src/documents/management/commands/document_archiver.py b/src/documents/management/commands/document_archiver.py index d2ff9c8c2..fe8c8b530 100644 --- a/src/documents/management/commands/document_archiver.py +++ b/src/documents/management/commands/document_archiver.py @@ -16,7 +16,8 @@ from whoosh.writing import AsyncWriter from documents.models import Document from ... import index -from ...file_handling import create_source_path_directory +from ...file_handling import create_source_path_directory, \ + generate_unique_filename from ...parsers import get_parser_class_for_mime_type @@ -39,13 +40,16 @@ def handle_document(document_id): with transaction.atomic(): with open(parser.get_archive_path(), 'rb') as f: checksum = hashlib.md5(f.read()).hexdigest() - # i'm going to save first so that in case the file move + # I'm going to save first so that in case the file move # fails, the database is rolled back. - # we also don't use save() since that triggers the filehandling + # We also don't use save() since that triggers the filehandling # logic, and we don't want that yet (file not yet in place) + document.archive_filename = generate_unique_filename( + document, archive_filename=True) Document.objects.filter(pk=document.pk).update( archive_checksum=checksum, - content=parser.get_text() + content=parser.get_text(), + archive_filename=document.archive_filename ) with FileLock(settings.MEDIA_LOCK): create_source_path_directory(document.archive_path) @@ -101,7 +105,7 @@ class Command(BaseCommand): document_ids = list(map( lambda doc: doc.id, filter( - lambda d: overwrite or not d.archive_checksum, + lambda d: overwrite or not d.has_archive_version, documents ) )) diff --git a/src/documents/management/commands/document_exporter.py b/src/documents/management/commands/document_exporter.py index 1505b0856..d8f5861a5 100644 --- a/src/documents/management/commands/document_exporter.py +++ b/src/documents/management/commands/document_exporter.py @@ -139,7 +139,7 @@ class Command(BaseCommand): thumbnail_target = os.path.join(self.target, thumbnail_name) document_dict[EXPORTER_THUMBNAIL_NAME] = thumbnail_name - if os.path.exists(document.archive_path): + if document.has_archive_version: archive_name = base_name + "-archive.pdf" archive_target = os.path.join(self.target, archive_name) document_dict[EXPORTER_ARCHIVE_NAME] = archive_name diff --git a/src/documents/migrations/1012_fix_archive_files.py b/src/documents/migrations/1012_fix_archive_files.py index f75ae431d..5db503391 100644 --- a/src/documents/migrations/1012_fix_archive_files.py +++ b/src/documents/migrations/1012_fix_archive_files.py @@ -1,43 +1,27 @@ # Generated by Django 3.1.6 on 2021-02-07 22:26 +import datetime import hashlib import logging import os import shutil +import pathvalidate from django.conf import settings -from django.db import migrations +from django.db import migrations, models +from django.template.defaultfilters import slugify +from documents.file_handling import defaultdictNoStr, many_to_dictionary logger = logging.getLogger("paperless.migrations") -def archive_name_from_filename_old(filename): +def archive_name_from_filename(filename): return os.path.splitext(filename)[0] + ".pdf" def archive_path_old(doc): if doc.filename: - fname = archive_name_from_filename_old(doc.filename) - else: - fname = "{:07}.pdf".format(doc.pk) - - return os.path.join( - settings.ARCHIVE_DIR, - fname - ) - - -def archive_name_from_filename_new(filename): - name, ext = os.path.splitext(filename) - if ext == ".pdf": - return filename - else: - return filename + ".pdf" - - -def archive_path_new(doc): - if doc.filename: - fname = archive_name_from_filename_new(doc.filename) + fname = archive_name_from_filename(doc.filename) else: fname = "{:07}.pdf".format(doc.pk) @@ -50,6 +34,16 @@ def archive_path_new(doc): STORAGE_TYPE_GPG = "gpg" +def archive_path_new(doc): + if doc.archive_filename is not None: + return os.path.join( + settings.ARCHIVE_DIR, + str(doc.archive_filename) + ) + else: + return None + + def source_path(doc): if doc.filename: fname = str(doc.filename) @@ -64,6 +58,98 @@ def source_path(doc): ) +def generate_unique_filename(doc, archive_filename=False): + if archive_filename: + old_filename = doc.archive_filename + root = settings.ARCHIVE_DIR + else: + old_filename = doc.filename + root = settings.ORIGINALS_DIR + + counter = 0 + + while True: + new_filename = generate_filename( + doc, counter, archive_filename=archive_filename) + if new_filename == old_filename: + # still the same as before. + return new_filename + + if os.path.exists(os.path.join(root, new_filename)): + counter += 1 + else: + return new_filename + + +def generate_filename(doc, counter=0, append_gpg=True, archive_filename=False): + path = "" + + try: + if settings.PAPERLESS_FILENAME_FORMAT is not None: + tags = defaultdictNoStr(lambda: slugify(None), + many_to_dictionary(doc.tags)) + + tag_list = pathvalidate.sanitize_filename( + ",".join(sorted( + [tag.name for tag in doc.tags.all()] + )), + replacement_text="-" + ) + + if doc.correspondent: + correspondent = pathvalidate.sanitize_filename( + doc.correspondent.name, replacement_text="-" + ) + else: + correspondent = "none" + + if doc.document_type: + document_type = pathvalidate.sanitize_filename( + doc.document_type.name, replacement_text="-" + ) + else: + document_type = "none" + + path = settings.PAPERLESS_FILENAME_FORMAT.format( + title=pathvalidate.sanitize_filename( + doc.title, replacement_text="-"), + correspondent=correspondent, + document_type=document_type, + created=datetime.date.isoformat(doc.created), + created_year=doc.created.year if doc.created else "none", + created_month=f"{doc.created.month:02}" if doc.created else "none", # NOQA: E501 + created_day=f"{doc.created.day:02}" if doc.created else "none", + added=datetime.date.isoformat(doc.added), + added_year=doc.added.year if doc.added else "none", + added_month=f"{doc.added.month:02}" if doc.added else "none", + added_day=f"{doc.added.day:02}" if doc.added else "none", + tags=tags, + tag_list=tag_list + ).strip() + + path = path.strip(os.sep) + + except (ValueError, KeyError, IndexError): + logger.warning( + f"Invalid PAPERLESS_FILENAME_FORMAT: " + f"{settings.PAPERLESS_FILENAME_FORMAT}, falling back to default") + + counter_str = f"_{counter:02}" if counter else "" + + filetype_str = ".pdf" if archive_filename else doc.file_type + + if len(path) > 0: + filename = f"{path}{counter_str}{filetype_str}" + else: + filename = f"{doc.pk:07}{counter_str}{filetype_str}" + + # Append .gpg for encrypted files + if append_gpg and doc.storage_type == STORAGE_TYPE_GPG: + filename += ".gpg" + + return filename + + def move_old_to_new_locations(apps, schema_editor): Document = apps.get_model("documents", "Document") @@ -74,18 +160,12 @@ def move_old_to_new_locations(apps, schema_editor): # check for documents that have incorrect archive versions for doc in Document.objects.filter(archive_checksum__isnull=False): old_path = archive_path_old(doc) - new_path = archive_path_new(doc) if not os.path.isfile(old_path): raise ValueError( f"Archived document of {doc.filename} does not exist at: " f"{old_path}") - if old_path != new_path and os.path.isfile(new_path): - raise ValueError( - f"Need to move {old_path} to {new_path}, but target file " - f"already exists") - if old_path in old_archive_path_to_id: affected_document_ids.add(doc.id) affected_document_ids.add(old_archive_path_to_id[old_path]) @@ -103,22 +183,19 @@ def move_old_to_new_locations(apps, schema_editor): f"document {doc.filename} has an invalid archived document, " f"but no parsers are available. Cannot migrate.") - # move files for doc in Document.objects.filter(archive_checksum__isnull=False): - old_path = archive_path_old(doc) - new_path = archive_path_new(doc) if doc.id in affected_document_ids: + old_path = archive_path_old(doc) # remove affected archive versions if os.path.isfile(old_path): os.unlink(old_path) else: - # move unaffected archive versions - if old_path != new_path and os.path.isfile(old_path) and not os.path.isfile(new_path): - logger.debug( - f"Moving {old_path} to {new_path}" - ) - shutil.move(old_path, new_path) + # Set archive path for unaffected files + doc.archive_filename = archive_path_old(doc) + Document.objects.filter(id=doc.id).update( + archive_filename=doc.archive_filename + ) # regenerate archive documents for doc_id in affected_document_ids: @@ -135,14 +212,16 @@ def move_old_to_new_locations(apps, schema_editor): try: parser.parse(source_path(doc), doc.mime_type, os.path.basename(doc.filename)) doc.content = parser.get_text() - if parser.archive_path and os.path.isfile(parser.archive_path): - with open(parser.archive_path, "rb") as f: + + if parser.get_archive_path() and os.path.isfile(parser.get_archive_path()): + doc.archive_filename = generate_unique_filename( + doc, archive_filename=True) + with open(parser.get_archive_path(), "rb") as f: doc.archive_checksum = hashlib.md5(f.read()).hexdigest() - shutil.copy2(parser.archive_path, archive_path_new(doc)) + os.makedirs(os.path.dirname(archive_path_new(doc)), exist_ok=True) + shutil.copy2(parser.get_archive_path(), archive_path_new(doc)) else: doc.archive_checksum = None - if os.path.isfile(archive_path_new(doc)): - os.unlink(archive_path_new(doc)) doc.save() except ParseError: logger.exception( @@ -187,8 +266,18 @@ class Migration(migrations.Migration): ] operations = [ + migrations.AddField( + model_name='document', + name='archive_filename', + field=models.FilePathField(default=None, editable=False, help_text='Current archive filename in storage', max_length=1024, null=True, unique=True, verbose_name='archive filename'), + ), + migrations.AlterField( + model_name='document', + name='filename', + field=models.FilePathField(default=None, editable=False, help_text='Current filename in storage', max_length=1024, null=True, unique=True, verbose_name='filename'), + ), migrations.RunPython( move_old_to_new_locations, move_new_to_old_locations - ) + ), ] diff --git a/src/documents/models.py b/src/documents/models.py index 86878dd7e..47433724a 100755 --- a/src/documents/models.py +++ b/src/documents/models.py @@ -16,7 +16,6 @@ from django.utils.timezone import is_aware from django.utils.translation import gettext_lazy as _ -from documents.file_handling import archive_name_from_filename from documents.parsers import get_default_file_extension @@ -208,10 +207,21 @@ class Document(models.Model): max_length=1024, editable=False, default=None, + unique=True, null=True, help_text=_("Current filename in storage") ) + archive_filename = models.FilePathField( + _("archive filename"), + max_length=1024, + editable=False, + default=None, + unique=True, + null=True, + help_text=_("Current archive filename in storage") + ) + archive_serial_number = models.IntegerField( _("archive serial number"), blank=True, @@ -256,16 +266,19 @@ class Document(models.Model): return open(self.source_path, "rb") @property - def archive_path(self): - if self.filename: - fname = archive_name_from_filename(self.filename) - else: - fname = "{:07}.pdf".format(self.pk) + def has_archive_version(self): + return self.archive_filename is not None + + @property + def archive_path(self): + if self.has_archive_version: + return os.path.join( + settings.ARCHIVE_DIR, + str(self.archive_filename) + ) + else: + return None - return os.path.join( - settings.ARCHIVE_DIR, - fname - ) @property def archive_file(self): diff --git a/src/documents/sanity_checker.py b/src/documents/sanity_checker.py index b8fd73f98..0b385e81b 100644 --- a/src/documents/sanity_checker.py +++ b/src/documents/sanity_checker.py @@ -88,7 +88,7 @@ def check_sanity(): )) # Check sanity of the archive file. - if doc.archive_checksum: + if doc.has_archive_version: if not os.path.isfile(doc.archive_path): messages.append(SanityError( f"Archived version of document {doc.pk} does not exist." diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py index 848bcb900..5d46dc431 100755 --- a/src/documents/signals/handlers.py +++ b/src/documents/signals/handlers.py @@ -14,7 +14,7 @@ from filelock import FileLock from .. import index, matching from ..file_handling import delete_empty_directories, \ - create_source_path_directory, archive_name_from_filename, \ + create_source_path_directory, \ generate_unique_filename from ..models import Document, Tag @@ -148,18 +148,18 @@ def set_tags(sender, @receiver(models.signals.post_delete, sender=Document) def cleanup_document_deletion(sender, instance, using, **kwargs): with FileLock(settings.MEDIA_LOCK): - for f in (instance.source_path, - instance.archive_path, - instance.thumbnail_path): - if os.path.isfile(f): + for filename in (instance.source_path, + instance.archive_path, + instance.thumbnail_path): + if filename and os.path.isfile(filename): try: - os.unlink(f) + os.unlink(filename) logger.debug( - f"Deleted file {f}.") + f"Deleted file {filename}.") except OSError as e: logger.warning( f"While deleting document {str(instance)}, the file " - f"{f} could not be deleted: {e}" + f"{filename} could not be deleted: {e}" ) delete_empty_directories( @@ -167,10 +167,11 @@ def cleanup_document_deletion(sender, instance, using, **kwargs): root=settings.ORIGINALS_DIR ) - delete_empty_directories( - os.path.dirname(instance.archive_path), - root=settings.ARCHIVE_DIR - ) + if instance.has_archive_version: + delete_empty_directories( + os.path.dirname(instance.archive_path), + root=settings.ARCHIVE_DIR + ) def validate_move(instance, old_path, new_path): @@ -207,8 +208,7 @@ def update_filename_and_move_files(sender, instance, **kwargs): with FileLock(settings.MEDIA_LOCK): old_filename = instance.filename - new_filename = generate_unique_filename( - instance, settings.ORIGINALS_DIR) + new_filename = generate_unique_filename(instance) if new_filename == instance.filename: # Don't do anything if its the same. @@ -222,8 +222,11 @@ def update_filename_and_move_files(sender, instance, **kwargs): # archive files are optional, archive checksum tells us if we have one, # since this is None for documents without archived files. - if instance.archive_checksum: - new_archive_filename = archive_name_from_filename(new_filename) + if instance.has_archive_version: + old_archive_filename = instance.archive_filename + new_archive_filename = generate_unique_filename( + instance, archive_filename=True + ) old_archive_path = instance.archive_path new_archive_path = os.path.join(settings.ARCHIVE_DIR, new_archive_filename) @@ -233,6 +236,8 @@ def update_filename_and_move_files(sender, instance, **kwargs): create_source_path_directory(new_archive_path) else: + old_archive_filename = None + new_archive_filename = None old_archive_path = None new_archive_path = None @@ -240,22 +245,28 @@ def update_filename_and_move_files(sender, instance, **kwargs): try: os.rename(old_source_path, new_source_path) - if instance.archive_checksum: - os.rename(old_archive_path, new_archive_path) instance.filename = new_filename + if instance.has_archive_version: + os.rename(old_archive_path, new_archive_path) + instance.archive_filename = new_archive_filename + # Don't save() here to prevent infinite recursion. Document.objects.filter(pk=instance.pk).update( - filename=new_filename) + filename=instance.filename, + archive_filename=instance.archive_filename, + ) except OSError as e: instance.filename = old_filename + instance.archive_filename = old_archive_filename # this happens when we can't move a file. If that's the case for # the archive file, we try our best to revert the changes. # no need to save the instance, the update() has not happened yet. try: os.rename(new_source_path, old_source_path) - os.rename(new_archive_path, old_archive_path) + if instance.has_archive_version: + os.rename(new_archive_path, old_archive_path) except Exception as e: # This is fine, since: # A: if we managed to move source from A to B, we will also @@ -271,9 +282,10 @@ def update_filename_and_move_files(sender, instance, **kwargs): # since moving them once succeeded, it's very likely going to # succeed again. os.rename(new_source_path, old_source_path) - if instance.archive_checksum: + if instance.has_archive_version: os.rename(new_archive_path, old_archive_path) instance.filename = old_filename + instance.archive_filename = old_archive_filename # again, no need to save the instance, since the actual update() # operation failed. @@ -283,7 +295,7 @@ def update_filename_and_move_files(sender, instance, **kwargs): delete_empty_directories(os.path.dirname(old_source_path), root=settings.ORIGINALS_DIR) - if old_archive_path and not os.path.isfile(old_archive_path): + if instance.has_archive_version and not os.path.isfile(old_archive_path): # NOQA: E501 delete_empty_directories(os.path.dirname(old_archive_path), root=settings.ARCHIVE_DIR) diff --git a/src/documents/views.py b/src/documents/views.py index 5ab4ca9df..3a840567d 100755 --- a/src/documents/views.py +++ b/src/documents/views.py @@ -192,7 +192,7 @@ class DocumentViewSet(RetrieveModelMixin, def file_response(self, pk, request, disposition): doc = Document.objects.get(id=pk) - if not self.original_requested(request) and os.path.isfile(doc.archive_path): # NOQA: E501 + if not self.original_requested(request) and doc.has_archive_version: # NOQA: E501 file_handle = doc.archive_file filename = doc.get_public_filename(archive=True) mime_type = 'application/pdf' @@ -237,18 +237,18 @@ class DocumentViewSet(RetrieveModelMixin, "original_size": os.stat(doc.source_path).st_size, "original_mime_type": doc.mime_type, "media_filename": doc.filename, - "has_archive_version": os.path.isfile(doc.archive_path), + "has_archive_version": doc.has_archive_version, "original_metadata": self.get_metadata( - doc.source_path, doc.mime_type) + doc.source_path, doc.mime_type), + "archive_checksum": doc.archive_checksum, + "archive_media_filename": doc.archive_filename } - if doc.archive_checksum and os.path.isfile(doc.archive_path): - meta['archive_checksum'] = doc.archive_checksum + if doc.has_archive_version: meta['archive_size'] = os.stat(doc.archive_path).st_size, meta['archive_metadata'] = self.get_metadata( doc.archive_path, "application/pdf") else: - meta['archive_checksum'] = None meta['archive_size'] = None meta['archive_metadata'] = None From a9f1766d1c36240ca8c59bc62d70baa137d6a634 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 19:46:32 +0100 Subject: [PATCH 073/225] todo note --- src/documents/management/commands/document_importer.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/documents/management/commands/document_importer.py b/src/documents/management/commands/document_importer.py index 7e5d32085..9347b0d4c 100644 --- a/src/documents/management/commands/document_importer.py +++ b/src/documents/management/commands/document_importer.py @@ -151,6 +151,9 @@ class Command(BaseCommand): shutil.copy2(thumbnail_path, document.thumbnail_path) if archive_path: create_source_path_directory(document.archive_path) + # TODO: this assumes that the export is valid and + # archive_filename is present on all documents with + # archived files shutil.copy2(archive_path, document.archive_path) document.save() From 2b7424c42a6a1c1f7b7ae4d3c4b0eb38eb3c5cc5 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 19:46:42 +0100 Subject: [PATCH 074/225] imports --- src/documents/signals/handlers.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py index 5d46dc431..17260fa8a 100755 --- a/src/documents/signals/handlers.py +++ b/src/documents/signals/handlers.py @@ -1,6 +1,5 @@ import logging import os -from subprocess import Popen from django.conf import settings from django.contrib.admin.models import ADDITION, LogEntry From 0c40a28ad368b98bfdce33365f1328347f1683b4 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 19:46:59 +0100 Subject: [PATCH 075/225] more sanity checks regarding archive versions --- src/documents/sanity_checker.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/documents/sanity_checker.py b/src/documents/sanity_checker.py index 0b385e81b..61e927c82 100644 --- a/src/documents/sanity_checker.py +++ b/src/documents/sanity_checker.py @@ -87,6 +87,18 @@ def check_sanity(): f"Stored: {doc.checksum}, actual: {checksum}." )) + if doc.archive_checksum and not doc.archive_filename: + messages.append(SanityError( + f"Document {doc.pk} has an archive file checksum, but no " + f"archive filename." + )) + + if not doc.archive_checksum and doc.archive_filename: + messages.append(SanityError( + f"Document {doc.pk} has an archive file, but its checksum is " + f"missing." + )) + # Check sanity of the archive file. if doc.has_archive_version: if not os.path.isfile(doc.archive_path): From 1ba89ddd09158c0f079b44bc503b2e80136de695 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 19:47:50 +0100 Subject: [PATCH 076/225] refactor migration tests to allow testing for exceptions while migrating --- src/documents/tests/utils.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/documents/tests/utils.py b/src/documents/tests/utils.py index b9be25d70..da8d3d429 100644 --- a/src/documents/tests/utils.py +++ b/src/documents/tests/utils.py @@ -92,6 +92,7 @@ class TestMigrations(TransactionTestCase): migrate_from = None migrate_to = None + auto_migrate = True def setUp(self): super(TestMigrations, self).setUp() @@ -108,6 +109,12 @@ class TestMigrations(TransactionTestCase): self.setUpBeforeMigration(old_apps) + self.apps = old_apps + + if self.auto_migrate: + self.performMigration() + + def performMigration(self): # Run the migration to test executor = MigrationExecutor(connection) executor.loader.build_graph() # reload. From 69d7f8c180ee8ef420599a676a4538ecdf25723f Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 19:49:29 +0100 Subject: [PATCH 077/225] testing the updated migration --- .../tests/test_migration_archive_files.py | 132 +++++++++++------- 1 file changed, 81 insertions(+), 51 deletions(-) diff --git a/src/documents/tests/test_migration_archive_files.py b/src/documents/tests/test_migration_archive_files.py index 69c1b51eb..5ae317e37 100644 --- a/src/documents/tests/test_migration_archive_files.py +++ b/src/documents/tests/test_migration_archive_files.py @@ -6,21 +6,19 @@ from pathlib import Path from django.conf import settings from django.test import override_settings -from documents.sanity_checker import SanityFailedError -from documents.tasks import sanity_check from documents.tests.utils import DirectoriesMixin, TestMigrations STORAGE_TYPE_GPG = "gpg" -def archive_name_from_filename_old(filename): +def archive_name_from_filename(filename): return os.path.splitext(filename)[0] + ".pdf" def archive_path_old(self): if self.filename: - fname = archive_name_from_filename_old(self.filename) + fname = archive_name_from_filename(self.filename) else: fname = "{:07}.pdf".format(self.pk) @@ -30,24 +28,14 @@ def archive_path_old(self): ) -def archive_name_from_filename_new(filename): - name, ext = os.path.splitext(filename) - if ext == ".pdf": - return filename - else: - return filename + ".pdf" - - -def archive_path_new(self): - if self.filename: - fname = archive_name_from_filename_new(self.filename) - else: - fname = "{:07}.pdf".format(self.pk) - - return os.path.join( - settings.ARCHIVE_DIR, - fname - ) +def archive_path_new(doc): + if doc.archive_filename is not None: + return os.path.join( + settings.ARCHIVE_DIR, + str(doc.archive_filename) + ) + else: + return None def source_path(doc): @@ -75,22 +63,25 @@ def thumbnail_path(doc): ) -def make_test_document(document_class, title: str, filename: str, mime_type: str, original: str, archive: str = None, new: bool = False): +def make_test_document(document_class, title: str, mime_type: str, original: str, original_filename: str, archive: str = None, archive_filename: str = None): doc = document_class() - doc.filename = filename + doc.filename = original_filename doc.title = title doc.mime_type = mime_type doc.content = "the content, does not matter for this test" + doc.save() shutil.copy2(original, source_path(doc)) with open(original, "rb") as f: doc.checksum = hashlib.md5(f.read()).hexdigest() if archive: - if new: + if archive_filename: + doc.archive_filename = archive_filename shutil.copy2(archive, archive_path_new(doc)) else: shutil.copy2(archive, archive_path_old(doc)) + with open(archive, "rb") as f: doc.archive_checksum = hashlib.md5(f.read()).hexdigest() @@ -101,36 +92,42 @@ def make_test_document(document_class, title: str, filename: str, mime_type: str return doc -@override_settings(PAPERLESS_FILENAME_FORMAT="{title}") +simple_jpg = os.path.join(os.path.dirname(__file__), "samples", "simple.jpg") +simple_pdf = os.path.join(os.path.dirname(__file__), "samples", "simple.pdf") +simple_pdf2 = os.path.join(os.path.dirname(__file__), "samples", "documents", "originals", "0000002.pdf") +simple_txt = os.path.join(os.path.dirname(__file__), "samples", "simple.txt") +simple_png = os.path.join(os.path.dirname(__file__), "samples", "simple-noalpha.png") + + +@override_settings(PAPERLESS_FILENAME_FORMAT="") class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): migrate_from = '1011_auto_20210101_2340' migrate_to = '1012_fix_archive_files' def setUpBeforeMigration(self, apps): - simple_jpg = os.path.join(os.path.dirname(__file__), "samples", "simple.jpg") - simple_pdf = os.path.join(os.path.dirname(__file__), "samples", "simple.pdf") - simple_pdf2 = os.path.join(os.path.dirname(__file__), "samples", "documents", "originals", "0000002.pdf") - simple_txt = os.path.join(os.path.dirname(__file__), "samples", "simple.txt") - simple_png = os.path.join(os.path.dirname(__file__), "samples", "simple-noalpha.png") - Document = apps.get_model("documents", "Document") - self.doc_unrelated = make_test_document(Document, "unrelated", "unrelated.txt", "application/pdf", simple_pdf2, simple_pdf2) - self.doc_no_archive = make_test_document(Document, "no_archive", "no_archive.txt", "text/plain", simple_txt) - self.clashA = make_test_document(Document, "clash", "clash.pdf", "application/pdf", simple_pdf, simple_pdf) - self.clashB = make_test_document(Document, "clash", "clash.jpg", "image/jpeg", simple_jpg, simple_pdf) - self.clashC = make_test_document(Document, "clash", "clash.png", "image/png", simple_png, simple_pdf) + doc_no_archive = make_test_document(Document, "no_archive", "text/plain", simple_txt, "no_archive.txt") + clash1 = make_test_document(Document, "clash", "application/pdf", simple_pdf, "clash.pdf", simple_pdf) + clash2 = make_test_document(Document, "clash", "image/jpeg", simple_jpg, "clash.jpg", simple_pdf) + clash3 = make_test_document(Document, "clash", "image/png", simple_png, "clash.png", simple_pdf) + clash4 = make_test_document(Document, "clash.png", "application/pdf", simple_pdf2, "clash.png.pdf", simple_pdf2) - self.assertEqual(archive_path_old(self.clashA), archive_path_old(self.clashB)) - self.assertEqual(archive_path_old(self.clashA), archive_path_old(self.clashC)) - self.assertRaises(SanityFailedError, sanity_check) + self.assertEqual(archive_path_old(clash1), archive_path_old(clash2)) + self.assertEqual(archive_path_old(clash1), archive_path_old(clash3)) + self.assertNotEqual(archive_path_old(clash1), archive_path_old(clash4)) def testArchiveFilesMigrated(self): Document = self.apps.get_model('documents', 'Document') for doc in Document.objects.all(): - self.assertTrue(os.path.isfile(archive_path_new(self.clashB))) + if doc.archive_checksum: + self.assertIsNotNone(doc.archive_filename) + self.assertTrue(os.path.isfile(archive_path_new(doc))) + else: + self.assertIsNone(doc.archive_filename) + with open(source_path(doc), "rb") as f: original_checksum = hashlib.md5(f.read()).hexdigest() self.assertEqual(original_checksum, doc.checksum) @@ -143,32 +140,32 @@ class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): self.assertEqual(Document.objects.filter(archive_checksum__isnull=False).count(), 4) - # this will raise errors when any inconsistencies remain after migration - sanity_check() + +@override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{title}") +class TestMigrateArchiveFilesWithFilenameFormat(TestMigrateArchiveFiles): + pass +@override_settings(PAPERLESS_FILENAME_FORMAT="") class TestMigrateArchiveFilesBackwards(DirectoriesMixin, TestMigrations): migrate_from = '1012_fix_archive_files' migrate_to = '1011_auto_20210101_2340' def setUpBeforeMigration(self, apps): - simple_jpg = os.path.join(os.path.dirname(__file__), "samples", "simple.jpg") - simple_pdf = os.path.join(os.path.dirname(__file__), "samples", "simple.pdf") - simple_pdf2 = os.path.join(os.path.dirname(__file__), "samples", "documents", "originals", "0000002.pdf") - simple_txt = os.path.join(os.path.dirname(__file__), "samples", "simple.txt") Document = apps.get_model("documents", "Document") - self.doc_unrelated = make_test_document(Document, "unrelated", "unrelated.txt", "application/pdf", simple_pdf2, simple_pdf2, new=True) - self.doc_no_archive = make_test_document(Document, "no_archive", "no_archive.txt", "text/plain", simple_txt, new=True) - self.clashB = make_test_document(Document, "clash", "clash.jpg", "image/jpeg", simple_jpg, simple_pdf, new=True) + doc_unrelated = make_test_document(Document, "unrelated", "application/pdf", simple_pdf2, "unrelated.txt", simple_pdf2, "unrelated.pdf") + doc_no_archive = make_test_document(Document, "no_archive", "text/plain", simple_txt, "no_archive.txt") + clashB = make_test_document(Document, "clash", "image/jpeg", simple_jpg, "clash.jpg", simple_pdf, "clash_02.pdf") def testArchiveFilesReverted(self): Document = self.apps.get_model('documents', 'Document') for doc in Document.objects.all(): - self.assertTrue(os.path.isfile(archive_path_old(self.clashB))) + if doc.archive_checksum: + self.assertTrue(os.path.isfile(archive_path_old(doc))) with open(source_path(doc), "rb") as f: original_checksum = hashlib.md5(f.read()).hexdigest() self.assertEqual(original_checksum, doc.checksum) @@ -178,3 +175,36 @@ class TestMigrateArchiveFilesBackwards(DirectoriesMixin, TestMigrations): with open(archive_path_old(doc), "rb") as f: archive_checksum = hashlib.md5(f.read()).hexdigest() self.assertEqual(archive_checksum, doc.archive_checksum) + + self.assertEqual(Document.objects.filter(archive_checksum__isnull=False).count(), 2) + + +@override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{title}") +class TestMigrateArchiveFilesBackwardsWithFilenameFormat(TestMigrateArchiveFilesBackwards): + pass + + +@override_settings(PAPERLESS_FILENAME_FORMAT="") +class TestMigrateArchiveFilesBackwardsErrors(DirectoriesMixin, TestMigrations): + + migrate_from = '1012_fix_archive_files' + migrate_to = '1011_auto_20210101_2340' + auto_migrate = False + + def test_filename_clash(self): + + Document = self.apps.get_model("documents", "Document") + + self.clashA = make_test_document(Document, "clash", "application/pdf", simple_pdf, "clash.pdf", simple_pdf, "clash_02.pdf") + self.clashB = make_test_document(Document, "clash", "image/jpeg", simple_jpg, "clash.jpg", simple_pdf, "clash_01.pdf") + + self.assertRaisesMessage(ValueError, "would clash with another archive filename", self.performMigration) + + def test_filename_exists(self): + + Document = self.apps.get_model("documents", "Document") + + self.clashA = make_test_document(Document, "clash", "application/pdf", simple_pdf, "clash.pdf", simple_pdf, "clash.pdf") + self.clashB = make_test_document(Document, "clash", "image/jpeg", simple_jpg, "clash.jpg", simple_pdf, "clash_01.pdf") + + self.assertRaisesMessage(ValueError, "file already exists.", self.performMigration) From d13e86a8924f5e3a5a9b8cbbb66c1258bcb27226 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 19:51:16 +0100 Subject: [PATCH 078/225] update all test cases to address the archive filename changes --- src/documents/tests/test_api.py | 18 +++---- src/documents/tests/test_consumer.py | 13 ++++- src/documents/tests/test_file_handling.py | 52 ++++++++++++++----- src/documents/tests/test_management.py | 2 + .../tests/test_management_exporter.py | 2 +- src/documents/tests/test_sanity_check.py | 2 +- 6 files changed, 64 insertions(+), 25 deletions(-) diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py index c4cca8fdc..84b3bc185 100644 --- a/src/documents/tests/test_api.py +++ b/src/documents/tests/test_api.py @@ -149,19 +149,16 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): @override_settings(PAPERLESS_FILENAME_FORMAT="") def test_download_with_archive(self): - _, filename = tempfile.mkstemp(dir=self.dirs.originals_dir) - content = b"This is a test" content_archive = b"This is the same test but archived" - with open(filename, "wb") as f: - f.write(content) - - filename = os.path.basename(filename) - - doc = Document.objects.create(title="none", filename=filename, + doc = Document.objects.create(title="none", filename="my_document.pdf", + archive_filename="archived.pdf", mime_type="application/pdf") + with open(doc.source_path, "wb") as f: + f.write(content) + with open(doc.archive_path, "wb") as f: f.write(content_archive) @@ -578,7 +575,7 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): async_task.assert_not_called() def test_get_metadata(self): - doc = Document.objects.create(title="test", filename="file.pdf", mime_type="image/png", archive_checksum="A") + doc = Document.objects.create(title="test", filename="file.pdf", mime_type="image/png", archive_checksum="A", archive_filename="archive.pdf") shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "documents", "thumbnails", "0000001.png"), doc.source_path) shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"), doc.archive_path) @@ -592,6 +589,8 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): self.assertTrue(meta['has_archive_version']) self.assertEqual(len(meta['original_metadata']), 0) self.assertGreater(len(meta['archive_metadata']), 0) + self.assertEqual(meta['media_filename'], "file.pdf") + self.assertEqual(meta['archive_media_filename'], "archive.pdf") def test_get_metadata_invalid_doc(self): response = self.client.get(f"/api/documents/34576/metadata/") @@ -611,6 +610,7 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): self.assertFalse(meta['has_archive_version']) self.assertGreater(len(meta['original_metadata']), 0) self.assertIsNone(meta['archive_metadata']) + self.assertIsNone(meta['archive_media_filename']) def test_get_empty_suggestions(self): doc = Document.objects.create(title="test", mime_type="application/pdf") diff --git a/src/documents/tests/test_consumer.py b/src/documents/tests/test_consumer.py index b96e4844b..1ed041187 100644 --- a/src/documents/tests/test_consumer.py +++ b/src/documents/tests/test_consumer.py @@ -296,6 +296,7 @@ class TestConsumer(DirectoriesMixin, TestCase): self.assertIsNone(document.correspondent) self.assertIsNone(document.document_type) self.assertEqual(document.filename, "0000001.pdf") + self.assertEqual(document.archive_filename, "0000001.pdf") self.assertTrue(os.path.isfile( document.source_path @@ -454,6 +455,7 @@ class TestConsumer(DirectoriesMixin, TestCase): self.assertEqual(document.title, "new docs") self.assertEqual(document.filename, "none/new docs.pdf") + self.assertEqual(document.archive_filename, "none/new docs.pdf") self._assert_first_last_send_progress() @@ -468,7 +470,7 @@ class TestConsumer(DirectoriesMixin, TestCase): filenames.insert(0, f) return f - m.side_effect = lambda f, root: get_filename() + m.side_effect = lambda f, archive_filename = False: get_filename() filename = self.get_test_file() @@ -479,6 +481,7 @@ class TestConsumer(DirectoriesMixin, TestCase): self.assertEqual(document.title, "new docs") self.assertIsNotNone(os.path.isfile(document.title)) self.assertTrue(os.path.isfile(document.source_path)) + self.assertTrue(os.path.isfile(document.archive_path)) self._assert_first_last_send_progress() @@ -553,8 +556,16 @@ class TestConsumer(DirectoriesMixin, TestCase): doc2 = self.consumer.try_consume_file(os.path.join(settings.CONSUMPTION_DIR, "simple.pdf")) doc3 = self.consumer.try_consume_file(os.path.join(settings.CONSUMPTION_DIR, "simple.png.pdf")) + self.assertEqual(doc1.filename, "simple.png") + self.assertEqual(doc1.archive_filename, "simple.pdf") + self.assertEqual(doc2.filename, "simple.pdf") + self.assertEqual(doc2.archive_filename, "simple_01.pdf") + self.assertEqual(doc3.filename, "simple.png.pdf") + self.assertEqual(doc3.archive_filename, "simple.png.pdf") + sanity_check() + class PreConsumeTestCase(TestCase): @mock.patch("documents.consumer.Popen") diff --git a/src/documents/tests/test_file_handling.py b/src/documents/tests/test_file_handling.py index ae25c0f12..fd74aad7d 100644 --- a/src/documents/tests/test_file_handling.py +++ b/src/documents/tests/test_file_handling.py @@ -448,7 +448,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") Path(original).touch() Path(archive).touch() - doc = Document.objects.create(mime_type="application/pdf", filename="0000001.pdf", checksum="A", archive_checksum="B") + doc = Document.objects.create(mime_type="application/pdf", filename="0000001.pdf", checksum="A", archive_filename="0000001.pdf", archive_checksum="B") self.assertTrue(os.path.isfile(original)) self.assertTrue(os.path.isfile(archive)) @@ -461,7 +461,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") Path(original).touch() Path(archive).touch() - doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B") + doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B", archive_filename="0000001.pdf") self.assertFalse(os.path.isfile(original)) self.assertFalse(os.path.isfile(archive)) @@ -475,7 +475,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): original = os.path.join(settings.ORIGINALS_DIR, "0000001.pdf") archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") Path(original).touch() - doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B") + doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B", archive_filename="0000001.pdf") self.assertTrue(os.path.isfile(original)) self.assertFalse(os.path.isfile(archive)) @@ -486,16 +486,19 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): def test_move_archive_exists(self): original = os.path.join(settings.ORIGINALS_DIR, "0000001.pdf") archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") + existing_archive_file = os.path.join(settings.ARCHIVE_DIR, "none", "my_doc.pdf") Path(original).touch() Path(archive).touch() os.makedirs(os.path.join(settings.ARCHIVE_DIR, "none")) - Path(os.path.join(settings.ARCHIVE_DIR, "none", "my_doc.pdf")).touch() - doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B") + Path(existing_archive_file).touch() + doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B", archive_filename="0000001.pdf") - self.assertTrue(os.path.isfile(original)) - self.assertTrue(os.path.isfile(archive)) + self.assertFalse(os.path.isfile(original)) + self.assertFalse(os.path.isfile(archive)) self.assertTrue(os.path.isfile(doc.source_path)) self.assertTrue(os.path.isfile(doc.archive_path)) + self.assertTrue(os.path.isfile(existing_archive_file)) + self.assertEqual(doc.archive_filename, "none/my_doc_01.pdf") @override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{title}") @mock.patch("documents.signals.handlers.os.rename") @@ -514,7 +517,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") Path(original).touch() Path(archive).touch() - doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B") + doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B", archive_filename="0000001.pdf") self.assertTrue(os.path.isfile(original)) self.assertTrue(os.path.isfile(archive)) @@ -527,7 +530,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") #Path(original).touch() Path(archive).touch() - doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B") + doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", archive_filename="0000001.pdf", checksum="A", archive_checksum="B") self.assertFalse(os.path.isfile(original)) self.assertTrue(os.path.isfile(archive)) @@ -551,7 +554,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") Path(original).touch() Path(archive).touch() - doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B") + doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", archive_filename="0000001.pdf", checksum="A", archive_checksum="B") self.assertTrue(os.path.isfile(original)) self.assertTrue(os.path.isfile(archive)) @@ -564,7 +567,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") Path(original).touch() Path(archive).touch() - doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B") + doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B", archive_filename="0000001.pdf") self.assertTrue(os.path.isfile(original)) self.assertTrue(os.path.isfile(archive)) @@ -578,6 +581,28 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): self.assertFalse(os.path.isfile(doc.source_path)) self.assertFalse(os.path.isfile(doc.archive_path)) + @override_settings(PAPERLESS_FILENAME_FORMAT="{title}") + def test_archive_deleted2(self): + original = os.path.join(settings.ORIGINALS_DIR, "document.png") + original2 = os.path.join(settings.ORIGINALS_DIR, "0000001.pdf") + archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") + Path(original).touch() + Path(original2).touch() + Path(archive).touch() + + doc1 = Document.objects.create(mime_type="image/png", title="document", filename="document.png", checksum="A", archive_checksum="B", archive_filename="0000001.pdf") + doc2 = Document.objects.create(mime_type="application/pdf", title="0000001", filename="0000001.pdf", checksum="C") + + self.assertTrue(os.path.isfile(doc1.source_path)) + self.assertTrue(os.path.isfile(doc1.archive_path)) + self.assertTrue(os.path.isfile(doc2.source_path)) + + doc2.delete() + + self.assertTrue(os.path.isfile(doc1.source_path)) + self.assertTrue(os.path.isfile(doc1.archive_path)) + self.assertFalse(os.path.isfile(doc2.source_path)) + @override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{title}") def test_database_error(self): @@ -585,7 +610,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): archive = os.path.join(settings.ARCHIVE_DIR, "0000001.pdf") Path(original).touch() Path(archive).touch() - doc = Document(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B") + doc = Document(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_filename="0000001.pdf", archive_checksum="B") with mock.patch("documents.signals.handlers.Document.objects.filter") as m: m.side_effect = DatabaseError() doc.save() @@ -595,6 +620,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): self.assertTrue(os.path.isfile(doc.source_path)) self.assertTrue(os.path.isfile(doc.archive_path)) + class TestFilenameGeneration(TestCase): @override_settings( @@ -618,7 +644,7 @@ class TestFilenameGeneration(TestCase): def run(): doc = Document.objects.create(checksum=str(uuid.uuid4()), title=str(uuid.uuid4()), content="wow") - doc.filename = generate_unique_filename(doc, settings.ORIGINALS_DIR) + doc.filename = generate_unique_filename(doc) Path(doc.thumbnail_path).touch() with open(doc.source_path, "w") as f: f.write(str(uuid.uuid4())) diff --git a/src/documents/tests/test_management.py b/src/documents/tests/test_management.py index ac119a431..38425a0aa 100644 --- a/src/documents/tests/test_management.py +++ b/src/documents/tests/test_management.py @@ -20,6 +20,7 @@ from documents.tests.utils import DirectoriesMixin sample_file = os.path.join(os.path.dirname(__file__), "samples", "simple.pdf") +@override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{title}") class TestArchiver(DirectoriesMixin, TestCase): def make_models(self): @@ -45,6 +46,7 @@ class TestArchiver(DirectoriesMixin, TestCase): self.assertTrue(os.path.isfile(doc.archive_path)) self.assertTrue(os.path.isfile(doc.source_path)) self.assertTrue(filecmp.cmp(sample_file, doc.source_path)) + self.assertEqual(doc.archive_filename, "none/A.pdf") class TestDecryptDocuments(TestCase): diff --git a/src/documents/tests/test_management_exporter.py b/src/documents/tests/test_management_exporter.py index 986e458b9..cdc745305 100644 --- a/src/documents/tests/test_management_exporter.py +++ b/src/documents/tests/test_management_exporter.py @@ -22,7 +22,7 @@ class TestExportImport(DirectoriesMixin, TestCase): self.target = tempfile.mkdtemp() self.addCleanup(shutil.rmtree, self.target) - self.d1 = Document.objects.create(content="Content", checksum="42995833e01aea9b3edee44bbfdd7ce1", archive_checksum="62acb0bcbfbcaa62ca6ad3668e4e404b", title="wow1", filename="0000001.pdf", mime_type="application/pdf") + self.d1 = Document.objects.create(content="Content", checksum="42995833e01aea9b3edee44bbfdd7ce1", archive_checksum="62acb0bcbfbcaa62ca6ad3668e4e404b", title="wow1", filename="0000001.pdf", mime_type="application/pdf", archive_filename="0000001.pdf") self.d2 = Document.objects.create(content="Content", checksum="9c9691e51741c1f4f41a20896af31770", title="wow2", filename="0000002.pdf", mime_type="application/pdf") self.d3 = Document.objects.create(content="Content", checksum="d38d7ed02e988e072caf924e0f3fcb76", title="wow2", filename="0000003.pdf", mime_type="application/pdf") self.d4 = Document.objects.create(content="Content", checksum="82186aaa94f0b98697d704b90fd1c072", title="wow_dec", filename="0000004.pdf.gpg", mime_type="application/pdf", storage_type=Document.STORAGE_TYPE_GPG) diff --git a/src/documents/tests/test_sanity_check.py b/src/documents/tests/test_sanity_check.py index 0554cd7cd..d7ab1e07f 100644 --- a/src/documents/tests/test_sanity_check.py +++ b/src/documents/tests/test_sanity_check.py @@ -21,7 +21,7 @@ class TestSanityCheck(DirectoriesMixin, TestCase): shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "documents", "archive", "0000001.pdf"), os.path.join(self.dirs.archive_dir, "0000001.pdf")) shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "documents", "thumbnails", "0000001.png"), os.path.join(self.dirs.thumbnail_dir, "0000001.png")) - return Document.objects.create(title="test", checksum="42995833e01aea9b3edee44bbfdd7ce1", archive_checksum="62acb0bcbfbcaa62ca6ad3668e4e404b", content="test", pk=1, filename="0000001.pdf", mime_type="application/pdf") + return Document.objects.create(title="test", checksum="42995833e01aea9b3edee44bbfdd7ce1", archive_checksum="62acb0bcbfbcaa62ca6ad3668e4e404b", content="test", pk=1, filename="0000001.pdf", mime_type="application/pdf", archive_filename="0000001.pdf") def test_no_docs(self): self.assertEqual(len(check_sanity()), 0) From 9df332b61435f93409b806e8463571190cbf8550 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 19:51:25 +0100 Subject: [PATCH 079/225] test resources --- src/clash.pdf | Bin 0 -> 10862 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 src/clash.pdf diff --git a/src/clash.pdf b/src/clash.pdf new file mode 100644 index 0000000000000000000000000000000000000000..304fbe71785e0320b55414484ff07b4ebe558dd7 GIT binary patch literal 10862 zcmb7q1z40@*ES%Hlpq~LH#5V~5)u+Bor1)`4Ba3pNQ+2!OE=OWARW@(Ez%v*^^fPA z?|i=RcfR-ke=Zm{d+)W^+V{HGVqMR(n3SbuxWL>3*i6mmnRnP=00>}fWPvRr0syMJ z*}(xoB_j(pCu;!kCEUi;$qc~H#{&S$m|Ho)9T2~*piXdUxUnq^u~88q^lQbz(Fwo@ z0f>oVJ32YQq1M>0DWPiCwzD9DyCwtsCG5gr13An@c1C@fkHR{rR%V5TJqj-^a93FE zvyTs4Hh3u)c+IitY7X}6HtL?dSRQ_FhJ@S=@Fjq8BDwIO&j)}Xj&vh8UjR1+Ld0Gh zJkYhsQ(Ys*^dYb`#INSCsO?T2)I$p=ll!LBqRBq?c*Uc1uVA<(m&~<{JVb`Z@^u5L zu4LZFI8_c)D={@iDbq6*hn@feDZ=M84}o=GmTxjov&fQTzNtm-N6Oj`|7bBRDtaEz zZhr{UGbk}}#B|SO0hyVPL0IM>4wd%$4mk@=v1YBldX`QvD?bfL0)eKT& z8oDqu#X^fJ2q(>M*RC}m(or70NTuK_FD3hSM851s+?OlEluC%#nA5q+4vsCdv^!Y!U`%1QXxJ?lj!2LG5w z=85`R*p1w)dG_lV7JfcA^vtEpqmtVw%AX^E#WypiR=O__@zCu*)f>!P zL^po5P9Dem3`!cnXU~EjwMN%y!h+F9uf_Rl>V(X&G7Fl;>$GHVZGCW31fxSvcGJQB zTnZW)xvBCCIMOQqg(+fQoov{HTST>^#JuLVl7}69Sz*-E3|ItyCpg{;O1Nu!*{r?0 zxC5%yO^M5P*4_(N0!!rv#v>*|&HVREpSO%^gfzNcBM-P1UJM~A)t;Gvbqi6@u5b4J zV_XOv?&8xo!)=Us-3YiYKL=QCBp(L7p;#Hl4rjJspzQ)k;Yko&hu+y1N)1Xh|w>i zgMVkd+8ZOMKS-~w@bbU-uK;&~!k|vjzasz)0rC7sZxL}^5R z(<6>_fSCxZO3VCpAmT4kdNU^{J7FNu#l?l&1;TCXUiWM zs4JI^BLhO;ABLD~|1D++{U)nKyz&%m>UHAFGbRDM`H(bJ128no4*9@epC1_ zl86O_CgeP-G1E< zjBpk=2ml1A!yO&r4p3t!fRdCd7a|J-c)7vc;6KgyT9p3H8*`X21Of(u`Am$tpgbmgh+CWRbAb?FT*h#I zL1Pm>BOwT%$$xMY%=m9R?A|z7{XspLF%WJA|E0wd;VJOH8v-*HHnDZEh9bxTwX?G_ zH-`Qq7Z3s3f6#+ih5m}Q--r%1H@E&n->)^GjU&(;_V23xloGasIv|ws{6R&;DZ<9K zHh)-PV+SrwJXx40G zsLjh@a?G&gs-EDEHuPSZMD|wo?6Z4}N3+Nab{iX&7>qfQQ6>lO7ombpP$bc{nR=Qb zWW{sz8K`TkQ^v?-FH}mb2~$+0L(-zWe>IYruo99>;ua~dxv2HJ^Xu~`2dM=mti5lF zH`ZR;COrITH0D)9crdHNY&ET86q$5={E2E6^7{LuWs*}gQGdq0f4C*&w_8fdORKp# zI>D{w5xfF`L5L+)08mN+YNv=u-H6i41}^F51eb>)Lc-k1P40gwtp2RSeu@44KZu8) z|6efDQ|Ym22N5)0bF3%4W@l5t4Wa8ike7 zQH`Ag7*p*l@`(^`321708=F%Sh4v~o+vc#6^vtX4_tqWWEU5CK{rl=O@rBO}(Y`y< z+Bwsd-mnOl;`)5~kz(-ue33q={I2`_nVv_=f$0*V4=u89`!;l7>X7W)LAPgWNSi!s zmoyLkqUIyLXSgPmgSBNsR5k5VwqIs4CPGPERfjI@Xx!c8S)Xhe)r|KN8@hTkf4%v@ z=WLgAiT{Wsvh*Ju!uy*;{uAwZctQUPX=JNSJK$m9fgAQBfY~}lv+(T}w`g%vO(PPr z+M2SjnfvA(u&WR6$y+(v8i|{#jlMZ1g^%Xqy}Qk>VEgA8hsItcLnfqT0Sy!!h8TPl zVFQiNJjFI7fU6?C;*CO~p2DoDeNORcCoc17NL7%F?>eajCd_nqohD+M$ip#!e~atg z@ejc9{XYW7D+u}5fGrWibyA;5Jom7@i+ag(5_Jjh{wg3674b3<%MBR|^;nK4`Y9kW zBIF7128M!c^!UTgr9l^aFL{WxRW)maiHjcILPg)lX=_-Cv{u8Mb4mR~iRP;@2AZKX z$fQmzV^S*z&$*($n?tO3O0mnyA2en9w`p z>+Z1B@O!%-dH!Wsk|9I8U$)Nn{7L#yB!ibI7WoB<1!oV@>DHuzUh{+m?WomKVzF5~ zb-g?AtJASgNwzF+`tCa(jYdOIa`yK(-oA#~HFaTj>%+mGta_ANydDzvAqO z3&?PzH{YJ%6*DdW-V3cff^b$}V_L04A&ij6L&fiNRY7&-r|A1{M$cWAm{__Yu$Zq3 z{BTU@BRYWy&!8S4Fk*_ZC>C>z<(*QL0c4Mj=fh84gA}R}-pk%cbm5VZRfAF;CQ0Y0 zC-y*bfNeYdF_EMp=_TsRRhna-!7HqlNzV!OmstGf;Wk6zMlrew2O*~ebzwicYb)<+SqUkf|RtY2<> zy60Qi8+^GULVHkQulYjWUIpsYV>Y~_jV@P`E-8{-@fd}ahFEIFP&Qo?vo3HBrJ4>* zr4=9{(W$7#=Io>#BbbkEFHzEA1Hz})g0Ied;pU~R$2g`KK2DO0uJLEhFRl8}#`yER z!%k$Dncdg@L)g?v`R6E^Pdz{OVsa%L1{G-QlJMc;;uS=uY3C38IAQadov`>YEhC>Dt5`sxqrs4> z`bA<+ScvoGxFQp6N>vn>eY2Y!6~#Q6+KfbuLPZAb7&-Q&Xu>F2$MYK+sYR1Q$>TZ6 zLWAULM^Bs7L5>%`g{5V|lE>$$6NEs$uN07kRxQ4?TL9Ok&G@&G*ktX zXvWG;@HS1+IOBzRHWg8V+$~G z+WmrBl`1!Nw~y?F(z>T2v69e2{YcI(c_CD3F+C)xV` zaXO5PxQ8@DhI$iHS15B-m1_EGrISmKAC9_1j$~HQnAwSchqB1YjQDOF7AmjPdvR(^ zPq2nfyN^i1k+xYTchDn0d#7n!>+wVjWhxbGqZfk>vo(?eKfChfWL{QR7cw^SG3202 z{d|p;0sG3;h-S@Q&T_^XX_Ci6fjUI~ftMgh!PvRVC`KbNsyt(lo%a0U_qsXm4iPa8 z&EfW7vbkI{=zEgFw6o!5WwYX+ox&cSgK2luqx~|YsMf-I&o-0HI}%t#anaK)3PD)l zu6FirvuFMUT|U#J{v{U0AZT@FE1M?Hd>k_!I!)ACmv#Pxq?xX6ppsu{l6q-?nx(8W zzaRRFNC`E~roIHVx(Htp??!ad?aAL^+HEz=&!QAX!8=IWl# zet%S#CLJan%)S>w_>DMK54ctqIJA%C#vY`mM8Nkz_`oq{LM?oz&P8pawXW+`n<461 zot$hU;cFd~YS-%OmqB*>R=XSrcp$XsP(%tKa~;7WOCezeLGuRqv4 z(pr&uXi}{_KbcALfs3N)Rl7jm5jw7I2LT#Jx+{t&t%}4)Z(J}fjXZQvNko2!X+RBI zpIv?j1G$O}e5FcvOj@@={n>55gxHwrG#^ue-j0fq(*6**8xEdAFRtpRVU#k61->Mf z*0lDqMl~dJj^JsNN^xpM`q0K+fdbndu^;yJT1IVJ%!O1e9 z-B3=`TG*bno-Y*!IBR`K_4Eq~lYS?O*+khu(8LuVrigM?u)E7h+bc|c7jj2J@ayw3phalqa+}*lqKRy{>Ko(5#QlLLrS{C3xaFMfNwU4Wze zecf$;njByAj*0j?M-SYCxYsIK9WuCI9)9L7T~sXxP|-$QBEx2f4ZfY%YMsDU&DF!P zcOpjGj;$bb1N%ndWgFUA=AIpW=Z2|a{nBJYs)l&F^_TmG0#AgVaM|ws{BC6_Gs!qv zR@6T6R`yzTadfa2W6r?(UFmI>bNkaLA|E~-ZXMQvINlfeL4|ft12&vpGx2(8)>nZVh+@O`1u+(~*l_j7|*0 z&Z88zvz95mRKt5nX^%!-CyVI(g4b*s-nGUI3Bl+cu_{K47_VZ{_vt{a8#gXeb2L{5))x-x+_KRFebD)?_|&V0OLVxlXL_c}}*N#kRPzu5yT%ayE^GWIDg zxZv!m{OsO#R~oS;!3GxtxOd2Y{+r5dTt@v3DQ0TTEVKsW!HU1zS#<5p0k{ z+6z0yFe#sSO~3u7*Pte_=xKD)h@mP~D~5TxYub)eluYBke4Mrav}M3bOv9`0CFK^m z7Loy}coS#NRdsgIbdN#r+=m&fWx+f@u1yV09L>1uzO&IFquK-^98wid=VT%$MYEf6 zEMrAOnG^yS$quXvqTG~pVQ7l4FHx}~KrtbM3^uUwRal0X=%F&l>b%nWmtt|*4Ydb$ zsxoeVf~5}j?T>hoJf3YmS$kT-IfcI3)~s0f4d2(`T%RJYSvz;D2is9&+z(-K(BZrI0XIGDoZjs)AUUAK| zNh9g0{(M-#9OFB>)w#S-HN~`Sj!F^OcVo0d+qP7M!@2!+ zuhY1{aqJkf@-h{?+$JYnFX8*7Is>+@&}E@8G7O(aCAP zJTK_ZDKE!nLSy%0Thmh!*M{AgWFfU@0g9mH1!0or?IDhosjM%%R05=;T}M>LCPHBe z*wixA+h#0hL6XHE*vaxy>agv~uu{FdSh#g^M!O7WFR~-MU{|OP#otRD|fEIpJY!{zI-8PY=Dsp zQC@8cdtg1)db>Q>P|J6J9N)JScN%;?<9uL9gQebhX|lBTph%~<=(R|-8>%%;7g@h*y_vuhu1$B>TE`w^a^v-T%PgPK9$_4lcFNOo6vUfbAj?~0!wm5SUCY4pFR zIyEKTzTmy8Bw*Mrev6+6q<3F&N_;CdojGE6d-3ls7IcAWsb2T8z?>zS z0X#i342%zb3T9~U7kxC#mMzUHgsqB8WTDhwPGT>U`jx3iu>p5gIOtXF@pE_U^KkC9 zgKCr?YRj8B-RAL`5n_H;Q<-YF1v_`wdl+4%C+GVLo>#R?;_hlcZY(Y?1mZ8pH&}cN zz6*FQkDYC|Hl~>>eb;Yhz>gR;O8uN()v`4jFVqVnrY52GdaPGUtV_L)GT-c(qT|7C zdDFEe@{VA;C1q*6SlB#d+4pVbhWHrciL`0`S6Lm82UYp~9cHo46bm{Xl0Po=1Vm9s zOA3~fW{3EP=F#;*pH>p_7kq%UE*O)Ve7>PGVZhj2hNd2Lfh%J|5w0NzllJ*Ku?j;8 zz%ry#^)@qr5FQElvE??!yLdntPFt!x6Fn`6KnVpl^`-*!B@D~H2Nt^>*33|BXX>>=YSW(PkZF9)qPzU z6QZ8xIL@=QM#Lo24;T+rG%~loc=Xo57u@Tdh1$nNP+`fb00ePMy!LI{rQm0G^Lg&2 z>x;?_ygu2aVfXN{@m67L$JC?Is+KeCO^NprPc_j`4OnL~(&{)#IOfi?a*yj7svu8- z#Al2krIRPASuRAp7<*3kBeuBYIH{`SB%5kht7(6tCUH(}1moL=zvO_k4ex}5AVuiH zlcf5)N!^T)(Cc_U-YZ?z<{n0ea<$wxqlGAqS7lmk&AWq4BD`U~WDlFW1e0Fmt~SWO zN-XNO;;#=`TdYgi8mx|?Tef=^Bl*5j9fz`rgkQ<+yDzE!F^U+gFp!u2&>FaV+7>~2 z^>uhX03FzW?)yIK2yT45#P@bU|3q@Xr|RdeZRzi_q;xd5Ep`D!pPL-FVwfzwpX*6ZIImx&Hv%PmobhGhFN0?bMLVwO#m3vU z?PUZ8Dr-lzQIfuK;*4s8UK#g&97Y53@Ljm1)U}*d-hckMsko^Z$lR`kOL{~sup-cW zE*}s^KshFNV=R{drQ*sC0rptZwOC2 z$xG?t=RLuF$zJs*gUe3SGfzL=(p%Gkn9G5n8z6Jue84|EF5~~dJuXuOK>qvrEid9V z`tPsE_YzH&)nAc#A9$2Hre`#m^Y9AE#|(R~rZCE?`0O)}Nk+E@Jq&BXq^FlBlqR-o z8KLWDWzQrdKtt;v>jtEU_+Rv)CUB=5@Pwxk?rzq;)9}M#t-L8Z#{noS#(C!HiWRa^Jn)6PE zrtaYpf*(ZikO({ze+bSKpYa^LUMEweH)RrUY$Gp_FF?RpCDf<&wsraesT~GhO+TLlJ8dDwft$Tv z2_vs*;%Os$*Fg8}CAZfX4f>lgYw`C6wuj>Tl@88LBf5PP5M(g>U9UCU1MgD`r0mtw zlwK$jYXkD;<|PNI8T%Rt z)KouiDS>plO*&q@fv4{zKN+Gjw>oL#Y5JWSl33w1wBH@d2vR5pR(=<8rG1S&)t+g=LxGW82DPP| zu`)l$fcrQ|prude!^WM}2@A2`Jeu&?mRgVs&01GB7=cxB>QGvTIX}b5${HcxeCs4s zPq+})6ig$bl(}RKB14yZ=aUm#h@b0^M-T?eV4QsdCmQqYAR%>757K3-A7lIuk4Fd2LT()m4}@d z=x+t?@Vp*1%Mmc-6LDhQ`3tsjOQL1O+QlCFAI8ZsvJ(yjiFIhNC>o(1@g51=r(8rz zGnq&3%0shxN5w|5Mwv#XMs;}-i)Cu)-bbnRTz4z1a;%!J;?wgI>Je%YhIQjbT14VS zLL;Y$MV`ePy_k{i$&FDPRNYrLd3Ka(k$IL$m^t^X=-JSRun&VDM>-yMeC>GD8QhW7 zX%nQNR<6!h$Pg!wBb?Eiu`UWN$~uZ6 z>Qi=tYMAPg>UEBAPTKRV?CNaKY)E#voIz|_Y(}4UcNrWTHrOo^6#%b>dBPoG;XQcW zgMB@HLVZj<8ohdv$+4#0bKT#1;(N}!hggS+i;0aM_Y!lne}x4@tP`>Nu}x?l6;(7x zo`Dt0744PnRpj;9p0a5bC~Ot*NF?S^<-E?FFA~YiOkXLe(8zhV7Rjz3n?sT1E>~^v zMnXZ4OU^o6CvUc@KD)liy~h3H6%lSlV6{wzOkTJN4wV{exETXWS~o*O}J~)=m5G%ZAH?YRzgA z^z)j+8a_i8{Bxq<{_xbCmSFS8HgCqC8eMFc zrOyjCt&8(s$+t`x9j!ANEzse}ZL($UwTpiF9=6$nPV>Q;X`yeIQ8=@m3-IQ?grXFV zx$2%bg9GE%$-1AO^ehZD_TJXLkHc8Xnj<=MIuTjEgf(uvFJ37Em)Ex6wCohr*1J@k z1I5T_Z9iFjB5gu%()GN)HA84jx?Q}UyCl69#~a3!!`Z~mMh!&cN8?5nLXAbo@;&vf zMO(s-#QT7|h20s59>^a!XlplbXV(r(u-EpQ-jq2FO{BE7idu{*+iElET@M;CZ6EnA z@qOU?Cx)-|+YIXrdJLBI8`EdIQoCAIl#1l0ftEd7Yl_ca$;8Si$m+__2Bm!{4`T~Y z46P0?3HBt~c%(&HOK|Y`9laMD9cu;`VLVxyYnbd>=gCz~w{6cnPfT`d%qK=UekHL4 zk#8;=0=ZAwDAnp!xB6(ee0W2v4o17iN=A#u#9^o=DFeiPt8hG+Iec#*sn@wzEX}VM zy7ALSW4_*U=YoIE`psO%UgVsI3+qAm9Jp@GZKFreU+<{FW@CFJZoO%ax~k!c61T1C zVu#D%QuXS`)svHNXWF|3w=}1oyXSr$Q~M?J*VW!?c9qJ_WnT{+2^%)o@zKTAZ_28G z7E6ZZ@<)-6USgHPoSIMXqH~OLW^#xyMDU$rzK)PmYcNbWPgwI+@OfC0;5p*q)0PX3>Tqc3YF}@e?T%o)8GJ*1YH_;u4fJg^ zywMEPYEyhrcyXL{L3{dQ4fVi(Is2feiFqtH*JNgWh2#rU34C3Bx}ZT^_-1;da7<^a z%Zx(=S1~Wu>r(APDa&+z=w}){)%y0;=|+;Xj)w%sc;p0lihPW16MKt7c*FT~?ObNV zE$M=RVv3fCfv3ccq&HMoxxL)_|)ECDAi`EC2V%Ackqx&^f?i{yj~A*W;{3wQZ*U=80evAWj&)eC zyblHN-eFV!Ly0HwTZ#AQOL=WoxXIr|-T&Gd`DcfuGStD^5%8_5fTcs47wM=(FY%_Zezy4i=mr)459~?%*q%)AOv7S uInXN Date: Tue, 9 Feb 2021 20:46:41 +0100 Subject: [PATCH 080/225] codestyle --- src/documents/models.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/documents/models.py b/src/documents/models.py index 47433724a..1b94b3cc9 100755 --- a/src/documents/models.py +++ b/src/documents/models.py @@ -279,7 +279,6 @@ class Document(models.Model): else: return None - @property def archive_file(self): return open(self.archive_path, "rb") From a1293c77b9717e16c95b44902355f544d2dfd64e Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 20:54:02 +0100 Subject: [PATCH 081/225] fix migration and more tests --- .../migrations/1012_fix_archive_files.py | 2 +- .../tests/test_migration_archive_files.py | 33 ++++++++++++++----- 2 files changed, 25 insertions(+), 10 deletions(-) diff --git a/src/documents/migrations/1012_fix_archive_files.py b/src/documents/migrations/1012_fix_archive_files.py index 5db503391..5102522ba 100644 --- a/src/documents/migrations/1012_fix_archive_files.py +++ b/src/documents/migrations/1012_fix_archive_files.py @@ -192,7 +192,7 @@ def move_old_to_new_locations(apps, schema_editor): os.unlink(old_path) else: # Set archive path for unaffected files - doc.archive_filename = archive_path_old(doc) + doc.archive_filename = archive_name_from_filename(doc.filename) Document.objects.filter(id=doc.id).update( archive_filename=doc.archive_filename ) diff --git a/src/documents/tests/test_migration_archive_files.py b/src/documents/tests/test_migration_archive_files.py index 5ae317e37..c4421e8e5 100644 --- a/src/documents/tests/test_migration_archive_files.py +++ b/src/documents/tests/test_migration_archive_files.py @@ -108,15 +108,15 @@ class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): def setUpBeforeMigration(self, apps): Document = apps.get_model("documents", "Document") - doc_no_archive = make_test_document(Document, "no_archive", "text/plain", simple_txt, "no_archive.txt") - clash1 = make_test_document(Document, "clash", "application/pdf", simple_pdf, "clash.pdf", simple_pdf) - clash2 = make_test_document(Document, "clash", "image/jpeg", simple_jpg, "clash.jpg", simple_pdf) - clash3 = make_test_document(Document, "clash", "image/png", simple_png, "clash.png", simple_pdf) - clash4 = make_test_document(Document, "clash.png", "application/pdf", simple_pdf2, "clash.png.pdf", simple_pdf2) + self.doc_no_archive = make_test_document(Document, "no_archive", "text/plain", simple_txt, "no_archive.txt") + self.clash1 = make_test_document(Document, "clash", "application/pdf", simple_pdf, "clash.pdf", simple_pdf) + self.clash2 = make_test_document(Document, "clash", "image/jpeg", simple_jpg, "clash.jpg", simple_pdf) + self.clash3 = make_test_document(Document, "clash", "image/png", simple_png, "clash.png", simple_pdf) + self.clash4 = make_test_document(Document, "clash.png", "application/pdf", simple_pdf2, "clash.png.pdf", simple_pdf2) - self.assertEqual(archive_path_old(clash1), archive_path_old(clash2)) - self.assertEqual(archive_path_old(clash1), archive_path_old(clash3)) - self.assertNotEqual(archive_path_old(clash1), archive_path_old(clash4)) + self.assertEqual(archive_path_old(self.clash1), archive_path_old(self.clash2)) + self.assertEqual(archive_path_old(self.clash1), archive_path_old(self.clash3)) + self.assertNotEqual(archive_path_old(self.clash1), archive_path_old(self.clash4)) def testArchiveFilesMigrated(self): Document = self.apps.get_model('documents', 'Document') @@ -140,10 +140,25 @@ class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): self.assertEqual(Document.objects.filter(archive_checksum__isnull=False).count(), 4) + def test_filenames(self): + Document = self.apps.get_model('documents', 'Document') + self.assertEqual(Document.objects.get(id=self.doc_no_archive.id).archive_filename, None) + self.assertEqual(Document.objects.get(id=self.clash1.id).archive_filename, f"{self.clash1.id:07}.pdf") + self.assertEqual(Document.objects.get(id=self.clash2.id).archive_filename, f"{self.clash2.id:07}.pdf") + self.assertEqual(Document.objects.get(id=self.clash3.id).archive_filename, f"{self.clash3.id:07}.pdf") + self.assertEqual(Document.objects.get(id=self.clash4.id).archive_filename, "clash.png.pdf") + @override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{title}") class TestMigrateArchiveFilesWithFilenameFormat(TestMigrateArchiveFiles): - pass + + def test_filenames(self): + Document = self.apps.get_model('documents', 'Document') + self.assertEqual(Document.objects.get(id=self.doc_no_archive.id).archive_filename, None) + self.assertEqual(Document.objects.get(id=self.clash1.id).archive_filename, "none/clash.pdf") + self.assertEqual(Document.objects.get(id=self.clash2.id).archive_filename, "none/clash_01.pdf") + self.assertEqual(Document.objects.get(id=self.clash3.id).archive_filename, "none/clash_02.pdf") + self.assertEqual(Document.objects.get(id=self.clash4.id).archive_filename, "clash.png.pdf") @override_settings(PAPERLESS_FILENAME_FORMAT="") From 887dd122febbb8506eda7877b8e8ca07f3c3a3c2 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 21:00:04 +0100 Subject: [PATCH 082/225] more info in the admin --- src/documents/admin.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/documents/admin.py b/src/documents/admin.py index 266422d51..7cc33dc20 100755 --- a/src/documents/admin.py +++ b/src/documents/admin.py @@ -50,7 +50,11 @@ class DocumentAdmin(admin.ModelAdmin): "modified", "mime_type", "storage_type", - "filename") + "filename", + "checksum", + "archive_filename", + "archive_checksum" + ) list_display_links = ("title",) From 7d677665088d225fcb3e971702b90eca75da92b3 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 21:53:10 +0100 Subject: [PATCH 083/225] todo note #520 --- src/documents/views.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/documents/views.py b/src/documents/views.py index 3a840567d..9d1d87235 100755 --- a/src/documents/views.py +++ b/src/documents/views.py @@ -291,6 +291,8 @@ class DocumentViewSet(RetrieveModelMixin, handle = GnuPG.decrypted(doc.thumbnail_file) else: handle = doc.thumbnail_file + # TODO: Send ETag information and use that to send new thumbnails + # if available return HttpResponse(handle, content_type='image/png') except (FileNotFoundError, Document.DoesNotExist): From 8d6071e977c023c0f2515f6cea6929633dd86d3d Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 22:12:43 +0100 Subject: [PATCH 084/225] fix a bug with thumbnail generation when TIKA was enabled --- src/documents/consumer.py | 2 +- .../management/commands/document_thumbnails.py | 5 ++++- src/documents/parsers.py | 9 ++++++--- src/documents/tests/test_consumer.py | 12 ++++++------ src/documents/tests/test_parsers.py | 8 ++++---- src/paperless_tesseract/parsers.py | 2 +- src/paperless_text/parsers.py | 2 +- src/paperless_tika/parsers.py | 4 ++-- 8 files changed, 25 insertions(+), 19 deletions(-) diff --git a/src/documents/consumer.py b/src/documents/consumer.py index acb3ad33f..3fd62fabc 100755 --- a/src/documents/consumer.py +++ b/src/documents/consumer.py @@ -241,7 +241,7 @@ class Consumer(LoggingMixin): self._send_progress(70, 100, 'WORKING', MESSAGE_GENERATING_THUMBNAIL) thumbnail = document_parser.get_optimised_thumbnail( - self.path, mime_type) + self.path, mime_type, self.filename) text = document_parser.get_text() date = document_parser.get_date() diff --git a/src/documents/management/commands/document_thumbnails.py b/src/documents/management/commands/document_thumbnails.py index cf2cbeb77..b7f935e3b 100644 --- a/src/documents/management/commands/document_thumbnails.py +++ b/src/documents/management/commands/document_thumbnails.py @@ -22,7 +22,10 @@ def _process_document(doc_in): try: thumb = parser.get_optimised_thumbnail( - document.source_path, document.mime_type) + document.source_path, + document.mime_type, + document.get_public_filename() + ) shutil.move(thumb, document.thumbnail_path) finally: diff --git a/src/documents/parsers.py b/src/documents/parsers.py index 98af4f080..1ed5deb3c 100644 --- a/src/documents/parsers.py +++ b/src/documents/parsers.py @@ -288,14 +288,17 @@ class DocumentParser(LoggingMixin): def get_archive_path(self): return self.archive_path - def get_thumbnail(self, document_path, mime_type): + def get_thumbnail(self, document_path, mime_type, file_name=None): """ Returns the path to a file we can use as a thumbnail for this document. """ raise NotImplementedError() - def get_optimised_thumbnail(self, document_path, mime_type): - thumbnail = self.get_thumbnail(document_path, mime_type) + def get_optimised_thumbnail(self, + document_path, + mime_type, + file_name=None): + thumbnail = self.get_thumbnail(document_path, mime_type, file_name) if settings.OPTIMIZE_THUMBNAILS: out_path = os.path.join(self.tempdir, "thumb_optipng.png") diff --git a/src/documents/tests/test_consumer.py b/src/documents/tests/test_consumer.py index 1ed041187..44effd39e 100644 --- a/src/documents/tests/test_consumer.py +++ b/src/documents/tests/test_consumer.py @@ -167,7 +167,7 @@ class TestFieldPermutations(TestCase): class DummyParser(DocumentParser): - def get_thumbnail(self, document_path, mime_type): + def get_thumbnail(self, document_path, mime_type, file_name=None): # not important during tests raise NotImplementedError() @@ -176,7 +176,7 @@ class DummyParser(DocumentParser): _, self.fake_thumb = tempfile.mkstemp(suffix=".png", dir=scratch_dir) self.archive_path = archive_path - def get_optimised_thumbnail(self, document_path, mime_type): + def get_optimised_thumbnail(self, document_path, mime_type, file_name=None): return self.fake_thumb def parse(self, document_path, mime_type, file_name=None): @@ -185,10 +185,10 @@ class DummyParser(DocumentParser): class CopyParser(DocumentParser): - def get_thumbnail(self, document_path, mime_type): + def get_thumbnail(self, document_path, mime_type, file_name=None): return self.fake_thumb - def get_optimised_thumbnail(self, document_path, mime_type): + def get_optimised_thumbnail(self, document_path, mime_type, file_name=None): return self.fake_thumb def __init__(self, logging_group, progress_callback=None): @@ -203,7 +203,7 @@ class CopyParser(DocumentParser): class FaultyParser(DocumentParser): - def get_thumbnail(self, document_path, mime_type): + def get_thumbnail(self, document_path, mime_type, file_name=None): # not important during tests raise NotImplementedError() @@ -211,7 +211,7 @@ class FaultyParser(DocumentParser): super(FaultyParser, self).__init__(logging_group) _, self.fake_thumb = tempfile.mkstemp(suffix=".png", dir=scratch_dir) - def get_optimised_thumbnail(self, document_path, mime_type): + def get_optimised_thumbnail(self, document_path, mime_type, file_name=None): return self.fake_thumb def parse(self, document_path, mime_type, file_name=None): diff --git a/src/documents/tests/test_parsers.py b/src/documents/tests/test_parsers.py index 8da6470a4..9dd74313f 100644 --- a/src/documents/tests/test_parsers.py +++ b/src/documents/tests/test_parsers.py @@ -68,7 +68,7 @@ class TestParserDiscovery(TestCase): ) -def fake_get_thumbnail(self, path, mimetype): +def fake_get_thumbnail(self, path, mimetype, file_name): return os.path.join(os.path.dirname(__file__), "examples", "no-text.png") @@ -89,15 +89,15 @@ class TestBaseParser(TestCase): def test_get_optimised_thumbnail(self): parser = DocumentParser(None) - parser.get_optimised_thumbnail("any", "not important") + parser.get_optimised_thumbnail("any", "not important", "document.pdf") @mock.patch("documents.parsers.DocumentParser.get_thumbnail", fake_get_thumbnail) @override_settings(OPTIMIZE_THUMBNAILS=False) def test_get_optimised_thumb_disabled(self): parser = DocumentParser(None) - path = parser.get_optimised_thumbnail("any", "not important") - self.assertEqual(path, fake_get_thumbnail(None, None, None)) + path = parser.get_optimised_thumbnail("any", "not important", "document.pdf") + self.assertEqual(path, fake_get_thumbnail(None, None, None, None)) class TestParserAvailability(TestCase): diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py index 0a976b569..271a840df 100644 --- a/src/paperless_tesseract/parsers.py +++ b/src/paperless_tesseract/parsers.py @@ -48,7 +48,7 @@ class RasterisedDocumentParser(DocumentParser): ) return result - def get_thumbnail(self, document_path, mime_type): + def get_thumbnail(self, document_path, mime_type, file_name=None): return make_thumbnail_from_pdf( document_path, self.tempdir, self.logging_group) diff --git a/src/paperless_text/parsers.py b/src/paperless_text/parsers.py index c307bf10b..837f05c9f 100644 --- a/src/paperless_text/parsers.py +++ b/src/paperless_text/parsers.py @@ -13,7 +13,7 @@ class TextDocumentParser(DocumentParser): logging_name = "paperless.parsing.text" - def get_thumbnail(self, document_path, mime_type): + def get_thumbnail(self, document_path, mime_type, file_name=None): def read_text(): with open(document_path, 'r') as src: diff --git a/src/paperless_tika/parsers.py b/src/paperless_tika/parsers.py index b888af820..6b0f62ada 100644 --- a/src/paperless_tika/parsers.py +++ b/src/paperless_tika/parsers.py @@ -16,9 +16,9 @@ class TikaDocumentParser(DocumentParser): logging_name = "paperless.parsing.tika" - def get_thumbnail(self, document_path, mime_type): + def get_thumbnail(self, document_path, mime_type, file_name=None): if not self.archive_path: - self.archive_path = self.convert_to_pdf(document_path) + self.archive_path = self.convert_to_pdf(document_path, file_name) return make_thumbnail_from_pdf( self.archive_path, self.tempdir, self.logging_group) From 1d002149dc9b9b021dcb7d667839b22e2fe39467 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 23:03:07 +0100 Subject: [PATCH 085/225] added ASN to filename format #519 --- docs/advanced_usage.rst | 1 + src/documents/file_handling.py | 6 ++++++ src/documents/tests/test_file_handling.py | 7 +++++++ 3 files changed, 14 insertions(+) diff --git a/docs/advanced_usage.rst b/docs/advanced_usage.rst index 8f6b91b4c..c64df4317 100644 --- a/docs/advanced_usage.rst +++ b/docs/advanced_usage.rst @@ -217,6 +217,7 @@ will create a directory structure as follows: Paperless provides the following placeholders withing filenames: +* ``{asn}``: The archive serial number of the document, or "none". * ``{correspondent}``: The name of the correspondent, or "none". * ``{document_type}``: The name of the document type, or "none". * ``{tag_list}``: A comma separated list of all tags assigned to the document. diff --git a/src/documents/file_handling.py b/src/documents/file_handling.py index 535aa3d2c..8c0e2148b 100644 --- a/src/documents/file_handling.py +++ b/src/documents/file_handling.py @@ -131,6 +131,11 @@ def generate_filename(doc, counter=0, append_gpg=True, archive_filename=False): else: document_type = "none" + if doc.archive_serial_number: + asn = str(doc.archive_serial_number) + else: + asn = "none" + path = settings.PAPERLESS_FILENAME_FORMAT.format( title=pathvalidate.sanitize_filename( doc.title, replacement_text="-"), @@ -144,6 +149,7 @@ def generate_filename(doc, counter=0, append_gpg=True, archive_filename=False): added_year=doc.added.year if doc.added else "none", added_month=f"{doc.added.month:02}" if doc.added else "none", added_day=f"{doc.added.day:02}" if doc.added else "none", + asn=asn, tags=tags, tag_list=tag_list ).strip() diff --git a/src/documents/tests/test_file_handling.py b/src/documents/tests/test_file_handling.py index fd74aad7d..927c865c6 100644 --- a/src/documents/tests/test_file_handling.py +++ b/src/documents/tests/test_file_handling.py @@ -201,6 +201,13 @@ class TestFileHandling(DirectoriesMixin, TestCase): self.assertEqual(generate_filename(d), "my_doc_type - the_doc.pdf") + @override_settings(PAPERLESS_FILENAME_FORMAT="{asn} - {title}") + def test_asn(self): + d1 = Document.objects.create(title="the_doc", mime_type="application/pdf", archive_serial_number=652, checksum="A") + d2 = Document.objects.create(title="the_doc", mime_type="application/pdf", archive_serial_number=None, checksum="B") + self.assertEqual(generate_filename(d1), "652 - the_doc.pdf") + self.assertEqual(generate_filename(d2), "none - the_doc.pdf") + @override_settings(PAPERLESS_FILENAME_FORMAT="{tags[type]}") def test_tags_with_underscore(self): document = Document() From 8330b3598cddd09ea84f55087ed1ac102ae8cc89 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 9 Feb 2021 23:23:11 +0100 Subject: [PATCH 086/225] changelog --- docs/changelog.rst | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 537e898c7..9ab83bc1a 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -5,6 +5,22 @@ Changelog ********* +paperless-ng 1.1.1 +################## + +* Fixed a bug in the sanity checker that would cause it to display "x not in list" errors instead of actual issues. + +* Fixed a bug with filename generation for archive filenames that would cause the archive files of two documents to overlap. + + * This happened when ``PAPERLESS_FILENAME_FORMAT`` is used and the filename for two documents is the same, except for the file extension. + * Paperless will now store the archive filename in the database as well instead of deriving it from the original filename, and use the + same logic for detecting and avoiding filename clashes that's also used for original filenames. + * The migrations will repair any missing archive files. If you're using tika, ensure that tika is running while performing the migration. Docker will take care of that. + +* Fixed a bug with thumbnail regeneration when TIKA integration was used. + +* Added ASN as a placeholder field to the filename format. + paperless-ng 1.1.0 ################## @@ -17,7 +33,7 @@ paperless-ng 1.1.0 or added with one of the mobile apps. * Documents are successfully added to paperless. * Document consumption failed (with error messages) - + * Configuration options to enable/disable individual notifications. * Live updates to document lists and saved views when new documents are added. From 92464116101b833517b6d1c5ad125784e59c86b5 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 00:52:01 +0100 Subject: [PATCH 087/225] better logging for the migration --- src/documents/migrations/1012_fix_archive_files.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/src/documents/migrations/1012_fix_archive_files.py b/src/documents/migrations/1012_fix_archive_files.py index 5102522ba..04670829f 100644 --- a/src/documents/migrations/1012_fix_archive_files.py +++ b/src/documents/migrations/1012_fix_archive_files.py @@ -163,7 +163,7 @@ def move_old_to_new_locations(apps, schema_editor): if not os.path.isfile(old_path): raise ValueError( - f"Archived document of {doc.filename} does not exist at: " + f"Archived document ID:{doc.id} does not exist at: " f"{old_path}") if old_path in old_archive_path_to_id: @@ -180,7 +180,7 @@ def move_old_to_new_locations(apps, schema_editor): parser_class = get_parser_class_for_mime_type(doc.mime_type) if not parser_class: raise Exception( - f"document {doc.filename} has an invalid archived document, " + f"Document ID:{doc.id} has an invalid archived document, " f"but no parsers are available. Cannot migrate.") for doc in Document.objects.filter(archive_checksum__isnull=False): @@ -205,7 +205,7 @@ def move_old_to_new_locations(apps, schema_editor): doc = Document.objects.get(id=doc_id) logger.info( - f"Regenerating archive document for {doc.filename}" + f"Regenerating archive document for document ID:{doc.id}" ) parser_class = get_parser_class_for_mime_type(doc.mime_type) parser: DocumentParser = parser_class(None, None) @@ -222,10 +222,16 @@ def move_old_to_new_locations(apps, schema_editor): shutil.copy2(parser.get_archive_path(), archive_path_new(doc)) else: doc.archive_checksum = None + logger.error( + f"Parser did not return an archive document for document " + f"ID:{doc.id}. Removing archive document." + ) doc.save() except ParseError: logger.exception( - f"Unable to regenerate archive document for {doc.filename}" + f"Unable to regenerate archive document for ID:{doc.id}. You " + f"need to invoke the document_archiver management command " + f"manually for that document." ) finally: parser.cleanup() From 34e84cc75732321ede7d7649767a2bf568857049 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 00:52:18 +0100 Subject: [PATCH 088/225] sanity checker testing --- src/documents/sanity_checker.py | 8 +++----- src/documents/tests/test_sanity_check.py | 17 +++++++++++++++-- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/src/documents/sanity_checker.py b/src/documents/sanity_checker.py index 61e927c82..2d30a59d7 100644 --- a/src/documents/sanity_checker.py +++ b/src/documents/sanity_checker.py @@ -87,20 +87,18 @@ def check_sanity(): f"Stored: {doc.checksum}, actual: {checksum}." )) + # Check sanity of the archive file. if doc.archive_checksum and not doc.archive_filename: messages.append(SanityError( f"Document {doc.pk} has an archive file checksum, but no " f"archive filename." )) - - if not doc.archive_checksum and doc.archive_filename: + elif not doc.archive_checksum and doc.archive_filename: messages.append(SanityError( f"Document {doc.pk} has an archive file, but its checksum is " f"missing." )) - - # Check sanity of the archive file. - if doc.has_archive_version: + elif doc.has_archive_version: if not os.path.isfile(doc.archive_path): messages.append(SanityError( f"Archived version of document {doc.pk} does not exist." diff --git a/src/documents/tests/test_sanity_check.py b/src/documents/tests/test_sanity_check.py index d7ab1e07f..fae18aaca 100644 --- a/src/documents/tests/test_sanity_check.py +++ b/src/documents/tests/test_sanity_check.py @@ -86,6 +86,19 @@ class TestSanityCheck(DirectoriesMixin, TestCase): Path(self.dirs.originals_dir, "orphaned").touch() self.assertEqual(len(check_sanity()), 1) - def test_all(self): - Document.objects.create(title="test", checksum="dgfhj", archive_checksum="dfhg", content="", pk=1, filename="0000001.pdf") + def test_error_tostring(self): + Document.objects.create(title="test", checksum="dgfhj", archive_checksum="dfhg", content="", pk=1, filename="0000001.pdf", archive_filename="0000001.pdf") string = str(SanityFailedError(check_sanity())) + self.assertIsNotNone(string) + + def test_archive_filename_no_checksum(self): + doc = self.make_test_data() + doc.archive_checksum = None + doc.save() + self.assertEqual(len(check_sanity()), 2) + + def test_archive_checksum_no_filename(self): + doc = self.make_test_data() + doc.archive_filename = None + doc.save() + self.assertEqual(len(check_sanity()), 2) From 7082cb9c3626084fdda52394ec2c7f2ad1479a2b Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 01:12:45 +0100 Subject: [PATCH 089/225] document renamer testing --- src/documents/tests/test_management.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/src/documents/tests/test_management.py b/src/documents/tests/test_management.py index 38425a0aa..978dff8b3 100644 --- a/src/documents/tests/test_management.py +++ b/src/documents/tests/test_management.py @@ -110,23 +110,25 @@ class TestRenamer(DirectoriesMixin, TestCase): @override_settings(PAPERLESS_FILENAME_FORMAT="") def test_rename(self): - doc = Document.objects.create(title="test", mime_type="application/pdf") + doc = Document.objects.create(title="test", mime_type="image/jpeg") doc.filename = generate_filename(doc) + doc.archive_filename = generate_filename(doc, archive_filename=True) doc.save() Path(doc.source_path).touch() + Path(doc.archive_path).touch() - old_source_path = doc.source_path - - with override_settings(PAPERLESS_FILENAME_FORMAT="{title}"): + with override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{title}"): call_command("document_renamer") doc2 = Document.objects.get(id=doc.id) - self.assertEqual(doc2.filename, "test.pdf") - self.assertFalse(os.path.isfile(old_source_path)) + self.assertEqual(doc2.filename, "none/test.jpg") + self.assertEqual(doc2.archive_filename, "none/test.pdf") self.assertFalse(os.path.isfile(doc.source_path)) + self.assertFalse(os.path.isfile(doc.archive_path)) self.assertTrue(os.path.isfile(doc2.source_path)) + self.assertTrue(os.path.isfile(doc2.archive_path)) class TestCreateClassifier(TestCase): From 77d745381f8382295203039a8e82167f49fc4916 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 01:31:15 +0100 Subject: [PATCH 090/225] more testing --- .../tests/test_migration_archive_files.py | 27 ++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/src/documents/tests/test_migration_archive_files.py b/src/documents/tests/test_migration_archive_files.py index c4421e8e5..637aaeee0 100644 --- a/src/documents/tests/test_migration_archive_files.py +++ b/src/documents/tests/test_migration_archive_files.py @@ -95,8 +95,10 @@ def make_test_document(document_class, title: str, mime_type: str, original: str simple_jpg = os.path.join(os.path.dirname(__file__), "samples", "simple.jpg") simple_pdf = os.path.join(os.path.dirname(__file__), "samples", "simple.pdf") simple_pdf2 = os.path.join(os.path.dirname(__file__), "samples", "documents", "originals", "0000002.pdf") +simple_pdf3 = os.path.join(os.path.dirname(__file__), "samples", "documents", "originals", "0000003.pdf") simple_txt = os.path.join(os.path.dirname(__file__), "samples", "simple.txt") simple_png = os.path.join(os.path.dirname(__file__), "samples", "simple-noalpha.png") +simple_png2 = os.path.join(os.path.dirname(__file__), "examples", "no-text.png") @override_settings(PAPERLESS_FILENAME_FORMAT="") @@ -108,6 +110,8 @@ class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): def setUpBeforeMigration(self, apps): Document = apps.get_model("documents", "Document") + self.unrelated = make_test_document(Document, "unrelated", "application/pdf", simple_pdf3, "unrelated.pdf", simple_pdf) + self.no_text = make_test_document(Document, "no-text", "image/png", simple_png2, "no-text.png", simple_pdf) self.doc_no_archive = make_test_document(Document, "no_archive", "text/plain", simple_txt, "no_archive.txt") self.clash1 = make_test_document(Document, "clash", "application/pdf", simple_pdf, "clash.pdf", simple_pdf) self.clash2 = make_test_document(Document, "clash", "image/jpeg", simple_jpg, "clash.jpg", simple_pdf) @@ -138,10 +142,12 @@ class TestMigrateArchiveFiles(DirectoriesMixin, TestMigrations): archive_checksum = hashlib.md5(f.read()).hexdigest() self.assertEqual(archive_checksum, doc.archive_checksum) - self.assertEqual(Document.objects.filter(archive_checksum__isnull=False).count(), 4) + self.assertEqual(Document.objects.filter(archive_checksum__isnull=False).count(), 6) def test_filenames(self): Document = self.apps.get_model('documents', 'Document') + self.assertEqual(Document.objects.get(id=self.unrelated.id).archive_filename, "unrelated.pdf") + self.assertEqual(Document.objects.get(id=self.no_text.id).archive_filename, "no-text.pdf") self.assertEqual(Document.objects.get(id=self.doc_no_archive.id).archive_filename, None) self.assertEqual(Document.objects.get(id=self.clash1.id).archive_filename, f"{self.clash1.id:07}.pdf") self.assertEqual(Document.objects.get(id=self.clash2.id).archive_filename, f"{self.clash2.id:07}.pdf") @@ -154,6 +160,8 @@ class TestMigrateArchiveFilesWithFilenameFormat(TestMigrateArchiveFiles): def test_filenames(self): Document = self.apps.get_model('documents', 'Document') + self.assertEqual(Document.objects.get(id=self.unrelated.id).archive_filename, "unrelated.pdf") + self.assertEqual(Document.objects.get(id=self.no_text.id).archive_filename, "no-text.pdf") self.assertEqual(Document.objects.get(id=self.doc_no_archive.id).archive_filename, None) self.assertEqual(Document.objects.get(id=self.clash1.id).archive_filename, "none/clash.pdf") self.assertEqual(Document.objects.get(id=self.clash2.id).archive_filename, "none/clash_01.pdf") @@ -161,6 +169,23 @@ class TestMigrateArchiveFilesWithFilenameFormat(TestMigrateArchiveFiles): self.assertEqual(Document.objects.get(id=self.clash4.id).archive_filename, "clash.png.pdf") +@override_settings(PAPERLESS_FILENAME_FORMAT="") +class TestMigrateArchiveFilesErrors(DirectoriesMixin, TestMigrations): + + migrate_from = '1011_auto_20210101_2340' + migrate_to = '1012_fix_archive_files' + auto_migrate = False + + def test_archive_missing(self): + + Document = self.apps.get_model("documents", "Document") + + doc = make_test_document(Document, "clash", "application/pdf", simple_pdf, "clash.pdf", simple_pdf) + os.unlink(archive_path_old(doc)) + + self.assertRaisesMessage(ValueError, "does not exist at: ", self.performMigration) + + @override_settings(PAPERLESS_FILENAME_FORMAT="") class TestMigrateArchiveFilesBackwards(DirectoriesMixin, TestMigrations): From 1bc961f0c0455b415c343a499ee39f3e54cf1ccc Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 11:50:57 +0100 Subject: [PATCH 091/225] update dependencies --- Pipfile.lock | 80 ++++++++++++++++++++++-------------------------- requirements.txt | 10 +++--- 2 files changed, 41 insertions(+), 49 deletions(-) diff --git a/Pipfile.lock b/Pipfile.lock index c3d57fc7d..f6a672b1b 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -190,24 +190,16 @@ }, "cryptography": { "hashes": [ - "sha256:0003a52a123602e1acee177dc90dd201f9bb1e73f24a070db7d36c588e8f5c7d", - "sha256:0e85aaae861d0485eb5a79d33226dd6248d2a9f133b81532c8f5aae37de10ff7", - "sha256:594a1db4511bc4d960571536abe21b4e5c3003e8750ab8365fafce71c5d86901", - "sha256:69e836c9e5ff4373ce6d3ab311c1a2eed274793083858d3cd4c7d12ce20d5f9c", - "sha256:788a3c9942df5e4371c199d10383f44a105d67d401fb4304178020142f020244", - "sha256:7e177e4bea2de937a584b13645cab32f25e3d96fc0bc4a4cf99c27dc77682be6", - "sha256:83d9d2dfec70364a74f4e7c70ad04d3ca2e6a08b703606993407bf46b97868c5", - "sha256:84ef7a0c10c24a7773163f917f1cb6b4444597efd505a8aed0a22e8c4780f27e", - "sha256:982f661bffc7a24b6d4f8ebe3291f17cf3833a0941c6f4d9d55c790b9aa2cdb3", - "sha256:9e21301f7a1e7c03dbea73e8602905a4ebba641547a462b26dd03451e5769e7c", - "sha256:9f6b0492d111b43de5f70052e24c1f0951cb9e6022188ebcb1cc3a3d301469b0", - "sha256:a69bd3c68b98298f490e84519b954335154917eaab52cf582fa2c5c7efc6e812", - "sha256:b4890d5fb9b7a23e3bf8abf5a8a7da8e228f1e97dc96b30b95685df840b6914a", - "sha256:c366df0401d1ec4e548bebe8f91d55ebcc0ec3137900d214dd7aac8427ef3030", - "sha256:dc42f645f8f3a489c3dd416730a514e7a91a59510ddaadc09d04224c098d3302" + "sha256:287032b6a7d86abc98e8e977b20138c53fea40e5b24e29090d5a675a973dcd10", + "sha256:288c65eea20bd89b11102c47b118bc1e0749386b0a0dfebba414076c5d4c8188", + "sha256:7eed937ad9b53280a5f53570d3a7dc93cb4412b6a3d58d4c6bb78cc26319c729", + "sha256:dab437c2e84628703e3358f0f06555a6259bc5039209d51aa3b05af667ff4fd0", + "sha256:ee5e19f0856b6fbbdbab15c2787ca65d203801d2d65d0b8de6218f424206c848", + "sha256:f21be9ec6b44c223b2024bbe59d394fadc7be320d18a8d595419afadb6cd5620", + "sha256:f6ea140d2736b7e1f0de4f988c43f76b0b3f3d365080e091715429ba218dce28" ], - "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'", - "version": "==3.3.1" + "markers": "python_version >= '3.6'", + "version": "==3.4.4" }, "daphne": { "hashes": [ @@ -243,11 +235,11 @@ }, "django-extensions": { "hashes": [ - "sha256:7cd002495ff0a0e5eb6cdd6be759600905b4e4079232ea27618fc46bdd853651", - "sha256:c7f88625a53f631745d4f2bef9ec4dcb999ed59476393bdbbe99db8596778846" + "sha256:674ad4c3b1587a884881824f40212d51829e662e52f85b012cd83d83fe1271d9", + "sha256:9507f8761ee760748938fd8af766d0608fb2738cf368adfa1b2451f61c15ae35" ], "index": "pypi", - "version": "==3.1.0" + "version": "==3.1.1" }, "django-filter": { "hashes": [ @@ -462,11 +454,11 @@ }, "joblib": { "hashes": [ - "sha256:75ead23f13484a2a414874779d69ade40d4fa1abe62b222a23cd50d4bc822f6f", - "sha256:7ad866067ac1fdec27d51c8678ea760601b70e32ff1881d4dc8e1171f2b64b24" + "sha256:9c17567692206d2f3fb9ecf5e991084254fe631665c450b443761c4186a613f7", + "sha256:feeb1ec69c4d45129954f1b7034954241eedfd6ba39b5e9e4b6883be3332d5e5" ], "markers": "python_version >= '3.6'", - "version": "==1.0.0" + "version": "==1.0.1" }, "langdetect": { "hashes": [ @@ -1113,11 +1105,11 @@ }, "tqdm": { "hashes": [ - "sha256:4621f6823bab46a9cc33d48105753ccbea671b68bab2c50a9f0be23d4065cb5a", - "sha256:fe3d08dd00a526850568d542ff9de9bbc2a09a791da3c334f3213d8d0bbbca65" + "sha256:2874fa525c051177583ec59c0fb4583e91f28ccd3f217ffad2acdb32d2c789ac", + "sha256:ab9b659241d82b8b51b2269ee243ec95286046bf06015c4e15a947cc15914211" ], "index": "pypi", - "version": "==4.56.0" + "version": "==4.56.1" }, "twisted": { "extras": [ @@ -1189,19 +1181,19 @@ }, "uvloop": { "hashes": [ - "sha256:08b109f0213af392150e2fe6f81d33261bb5ce968a288eb698aad4f46eb711bd", - "sha256:123ac9c0c7dd71464f58f1b4ee0bbd81285d96cdda8bc3519281b8973e3a461e", - "sha256:2e57a28567d874afd803b6d8a7aeee7bfa0a34994f1b237837e6d7e525cea9a4", - "sha256:4315d2ec3ca393dd5bc0b0089d23101276778c304d42faff5dc4579cb6caef09", - "sha256:4544dcf77d74f3a84f03dd6278174575c44c67d7165d4c42c71db3fdc3860726", - "sha256:63c0bd965e73a4e8dde6a15635262ec38036d83327230552ada21298ee5fa3b7", - "sha256:afd5513c0ae414ec71d24f6f123614a80f3d27ca655a4fcf6cabe50994cc1891", - "sha256:b4f591aa4b3fa7f32fb51e2ee9fea1b495eb75b0b3c8d0ca52514ad675ae63f7", - "sha256:bcac356d62edd330080aed082e78d4b580ff260a677508718f88016333e2c9c5", - "sha256:e7514d7a48c063226b7d06617cbb12a14278d4323a065a8d46a7962686ce2e95", - "sha256:f07909cd9fc08c52d294b1570bba92186181ca01fe3dc9ffba68955273dd7362" + "sha256:08547ec0d40005da5c5f969f406cc90e6d589b41388958e326d78ea63fb5d2f9", + "sha256:1a503d5b49da6e3dd5607d6e533a5315b1caedbf629901807c65a23a09cad065", + "sha256:1c114eafd3c53cab74024d122742ed9001611e8f341421dba84a8ea7d886cc2f", + "sha256:2d76da5529025f9771855505cfde7bf16b9b0001e2fe05b0abf3e4fc01a5dfdb", + "sha256:9900020313c158cbc7befdd5ad60eb7b5443d7d1e83ad57c310e330fe7b2d858", + "sha256:9ae0b0e86ffcfe11c208fdb1552894911aa0915280e904a3757ebd4505792568", + "sha256:ad5391ce33ac1c745dec6f8a394387b3247c374e93b12a1f0403791f6be4bb8a", + "sha256:aecae1a9edd2fb727277f159fb74abab1f426442ae3c03b3b9e9817c7ffcafd2", + "sha256:cb121716f4ff5c01dd9b9a12b2193e59383243c21d14c916f588fd626bd3c74b", + "sha256:e17cc3f32288daf8476368ddb60cad0f00ab13c979f6f4003ee773f1d90cab58", + "sha256:fab95c88d0c3694089c93a8c46fdab30117c4ce0d1afa2ab7d6f4d0607495857" ], - "version": "==0.14.0" + "version": "==0.15.0" }, "watchdog": { "hashes": [ @@ -1506,11 +1498,11 @@ }, "faker": { "hashes": [ - "sha256:190f0d3ce037866b5d230f0b9fd0f513f07c25dc326dcad6ee019849c68d441c", - "sha256:db7adc3b4755005fc960cf96fb4ed46b54b6eb21413741ab3f31a9595f379905" + "sha256:bf2a9b3f8d00a5dada61fc4a3f80fe0d6795c7f02a138a7d2ef2db5817c7d017", + "sha256:d4aecdb877519d06c2fdc01ffc5ecf70658981acf5e13cd07ded9892994ef5c6" ], "markers": "python_version >= '3.6'", - "version": "==6.0.0" + "version": "==6.1.1" }, "filelock": { "hashes": [ @@ -1713,11 +1705,11 @@ }, "pytest-xdist": { "hashes": [ - "sha256:1d8edbb1a45e8e1f8e44b1260583107fc23f8bc8da6d18cb331ff61d41258ecf", - "sha256:f127e11e84ad37cc1de1088cb2990f3c354630d428af3f71282de589c5bb779b" + "sha256:2447a1592ab41745955fb870ac7023026f20a5f0bfccf1b52a879bd193d46450", + "sha256:718887296892f92683f6a51f25a3ae584993b06f7076ce1e1fd482e59a8220a2" ], "index": "pypi", - "version": "==2.2.0" + "version": "==2.2.1" }, "python-dateutil": { "hashes": [ diff --git a/requirements.txt b/requirements.txt index 5379725b5..29fe08569 100644 --- a/requirements.txt +++ b/requirements.txt @@ -24,11 +24,11 @@ click==7.1.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, coloredlogs==15.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' concurrent-log-handler==0.9.19 constantly==15.1.0 -cryptography==3.3.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5' +cryptography==3.4.4; python_version >= '3.6' daphne==3.0.1; python_version >= '3.6' dateparser==0.7.6 django-cors-headers==3.7.0 -django-extensions==3.1.0 +django-extensions==3.1.1 django-filter==2.4.0 django-picklefield==3.0.1; python_version >= '3' django-q==1.3.4 @@ -49,7 +49,7 @@ img2pdf==0.4.0 incremental==17.5.0 inotify-simple==1.3.5; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' inotifyrecursive==0.3.5 -joblib==1.0.0; python_version >= '3.6' +joblib==1.0.1; python_version >= '3.6' langdetect==1.0.8 lxml==4.6.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' msgpack==1.0.2 @@ -87,13 +87,13 @@ sortedcontainers==2.3.0 sqlparse==0.4.1; python_version >= '3.5' threadpoolctl==2.1.0; python_version >= '3.5' tika==1.24 -tqdm==4.56.0 +tqdm==4.56.1 twisted[tls]==20.3.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' txaio==20.12.1; python_version >= '3.6' tzlocal==2.1 urllib3==1.26.3; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4' uvicorn[standard]==0.13.3 -uvloop==0.14.0 +uvloop==0.15.0 watchdog==1.0.2 watchgod==0.6 wcwidth==0.2.5 From ed84cf26e7ccdff4e6b1e23008278df1d7ffa27e Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 14:31:17 +0100 Subject: [PATCH 092/225] update dependencies --- Pipfile | 2 ++ Pipfile.lock | 27 ++++++++++++++------------- requirements.txt | 2 +- 3 files changed, 17 insertions(+), 14 deletions(-) diff --git a/Pipfile b/Pipfile index 7fcf3de26..d9940d371 100644 --- a/Pipfile +++ b/Pipfile @@ -52,6 +52,8 @@ channels-redis = "*" uvicorn = {extras = ["standard"], version = "*"} concurrent-log-handler = "*" django-redis = "*" +# uvloop 0.15+ incompatible with python 3.6 +uvloop = "~=0.14.0" [dev-packages] coveralls = "*" diff --git a/Pipfile.lock b/Pipfile.lock index f6a672b1b..c05610316 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "0c2003b9d3d95d1af594f749a2740b55079551ea0ae512177ee9524bb327281e" + "sha256": "d5459c3ca4499fcab45844d273d1da636fedd28e6108cb620b982175e81fdce5" }, "pipfile-spec": 6, "requires": {}, @@ -1181,19 +1181,20 @@ }, "uvloop": { "hashes": [ - "sha256:08547ec0d40005da5c5f969f406cc90e6d589b41388958e326d78ea63fb5d2f9", - "sha256:1a503d5b49da6e3dd5607d6e533a5315b1caedbf629901807c65a23a09cad065", - "sha256:1c114eafd3c53cab74024d122742ed9001611e8f341421dba84a8ea7d886cc2f", - "sha256:2d76da5529025f9771855505cfde7bf16b9b0001e2fe05b0abf3e4fc01a5dfdb", - "sha256:9900020313c158cbc7befdd5ad60eb7b5443d7d1e83ad57c310e330fe7b2d858", - "sha256:9ae0b0e86ffcfe11c208fdb1552894911aa0915280e904a3757ebd4505792568", - "sha256:ad5391ce33ac1c745dec6f8a394387b3247c374e93b12a1f0403791f6be4bb8a", - "sha256:aecae1a9edd2fb727277f159fb74abab1f426442ae3c03b3b9e9817c7ffcafd2", - "sha256:cb121716f4ff5c01dd9b9a12b2193e59383243c21d14c916f588fd626bd3c74b", - "sha256:e17cc3f32288daf8476368ddb60cad0f00ab13c979f6f4003ee773f1d90cab58", - "sha256:fab95c88d0c3694089c93a8c46fdab30117c4ce0d1afa2ab7d6f4d0607495857" + "sha256:08b109f0213af392150e2fe6f81d33261bb5ce968a288eb698aad4f46eb711bd", + "sha256:123ac9c0c7dd71464f58f1b4ee0bbd81285d96cdda8bc3519281b8973e3a461e", + "sha256:2e57a28567d874afd803b6d8a7aeee7bfa0a34994f1b237837e6d7e525cea9a4", + "sha256:4315d2ec3ca393dd5bc0b0089d23101276778c304d42faff5dc4579cb6caef09", + "sha256:4544dcf77d74f3a84f03dd6278174575c44c67d7165d4c42c71db3fdc3860726", + "sha256:63c0bd965e73a4e8dde6a15635262ec38036d83327230552ada21298ee5fa3b7", + "sha256:afd5513c0ae414ec71d24f6f123614a80f3d27ca655a4fcf6cabe50994cc1891", + "sha256:b4f591aa4b3fa7f32fb51e2ee9fea1b495eb75b0b3c8d0ca52514ad675ae63f7", + "sha256:bcac356d62edd330080aed082e78d4b580ff260a677508718f88016333e2c9c5", + "sha256:e7514d7a48c063226b7d06617cbb12a14278d4323a065a8d46a7962686ce2e95", + "sha256:f07909cd9fc08c52d294b1570bba92186181ca01fe3dc9ffba68955273dd7362" ], - "version": "==0.15.0" + "index": "pypi", + "version": "==0.14.0" }, "watchdog": { "hashes": [ diff --git a/requirements.txt b/requirements.txt index 29fe08569..097cef78e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -93,7 +93,7 @@ txaio==20.12.1; python_version >= '3.6' tzlocal==2.1 urllib3==1.26.3; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4' uvicorn[standard]==0.13.3 -uvloop==0.15.0 +uvloop==0.14.0 watchdog==1.0.2 watchgod==0.6 wcwidth==0.2.5 From 6c8f010f7a8df84c281c7219ecb4c7c630a9c6f9 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 14:50:20 +0100 Subject: [PATCH 093/225] retries for archive generation --- .../migrations/1012_fix_archive_files.py | 100 ++++++++++++------ 1 file changed, 66 insertions(+), 34 deletions(-) diff --git a/src/documents/migrations/1012_fix_archive_files.py b/src/documents/migrations/1012_fix_archive_files.py index 04670829f..4fa258b8a 100644 --- a/src/documents/migrations/1012_fix_archive_files.py +++ b/src/documents/migrations/1012_fix_archive_files.py @@ -4,6 +4,7 @@ import hashlib import logging import os import shutil +from time import sleep import pathvalidate from django.conf import settings @@ -12,8 +13,12 @@ from django.template.defaultfilters import slugify from documents.file_handling import defaultdictNoStr, many_to_dictionary + logger = logging.getLogger("paperless.migrations") +############################################################################### +# This is code copied straight paperless before the change. +############################################################################### def archive_name_from_filename(filename): return os.path.splitext(filename)[0] + ".pdf" @@ -150,6 +155,65 @@ def generate_filename(doc, counter=0, append_gpg=True, archive_filename=False): return filename +############################################################################### +# This code performs bidirection archive file transformation. +############################################################################### + + +def create_archive_version(doc, retry_count=4): + from documents.parsers import get_parser_class_for_mime_type, \ + DocumentParser, \ + ParseError + + logger.info( + f"Regenerating archive document for document ID:{doc.id}" + ) + parser_class = get_parser_class_for_mime_type(doc.mime_type) + for try_num in range(retry_count): + parser: DocumentParser = parser_class(None, None) + try: + parser.parse(source_path(doc), doc.mime_type, + os.path.basename(doc.filename)) + doc.content = parser.get_text() + + if parser.get_archive_path() and os.path.isfile( + parser.get_archive_path()): + doc.archive_filename = generate_unique_filename( + doc, archive_filename=True) + with open(parser.get_archive_path(), "rb") as f: + doc.archive_checksum = hashlib.md5(f.read()).hexdigest() + os.makedirs(os.path.dirname(archive_path_new(doc)), + exist_ok=True) + shutil.copy2(parser.get_archive_path(), archive_path_new(doc)) + else: + doc.archive_checksum = None + logger.error( + f"Parser did not return an archive document for document " + f"ID:{doc.id}. Removing archive document." + ) + doc.save() + return + except ParseError: + if try_num + 1 == retry_count: + logger.exception( + f"Unable to regenerate archive document for ID:{doc.id}. You " + f"need to invoke the document_archiver management command " + f"manually for that document." + ) + doc.archive_checksum = None + doc.save() + return + else: + # This is mostly here for the tika parser in docker + # environemnts. The servers for parsing need to come up first, + # and the docker setup doesn't ensure that tika is running + # before attempting migrations. + logger.error("Parse error, will try again in 5 seconds...") + sleep(5) + finally: + parser.cleanup() + + def move_old_to_new_locations(apps, schema_editor): Document = apps.get_model("documents", "Document") @@ -199,42 +263,10 @@ def move_old_to_new_locations(apps, schema_editor): # regenerate archive documents for doc_id in affected_document_ids: - from documents.parsers import get_parser_class_for_mime_type, \ - DocumentParser, \ - ParseError - doc = Document.objects.get(id=doc_id) - logger.info( - f"Regenerating archive document for document ID:{doc.id}" - ) - parser_class = get_parser_class_for_mime_type(doc.mime_type) - parser: DocumentParser = parser_class(None, None) - try: - parser.parse(source_path(doc), doc.mime_type, os.path.basename(doc.filename)) - doc.content = parser.get_text() + create_archive_version(doc) + - if parser.get_archive_path() and os.path.isfile(parser.get_archive_path()): - doc.archive_filename = generate_unique_filename( - doc, archive_filename=True) - with open(parser.get_archive_path(), "rb") as f: - doc.archive_checksum = hashlib.md5(f.read()).hexdigest() - os.makedirs(os.path.dirname(archive_path_new(doc)), exist_ok=True) - shutil.copy2(parser.get_archive_path(), archive_path_new(doc)) - else: - doc.archive_checksum = None - logger.error( - f"Parser did not return an archive document for document " - f"ID:{doc.id}. Removing archive document." - ) - doc.save() - except ParseError: - logger.exception( - f"Unable to regenerate archive document for ID:{doc.id}. You " - f"need to invoke the document_archiver management command " - f"manually for that document." - ) - finally: - parser.cleanup() def move_new_to_old_locations(apps, schema_editor): From 04519ee6232d34ac1fbf7e2ad07030c073fb35c2 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 16:58:55 +0100 Subject: [PATCH 094/225] more testing of the migration --- .../migrations/1012_fix_archive_files.py | 33 ++++++--- .../tests/test_migration_archive_files.py | 74 +++++++++++++++++++ 2 files changed, 95 insertions(+), 12 deletions(-) diff --git a/src/documents/migrations/1012_fix_archive_files.py b/src/documents/migrations/1012_fix_archive_files.py index 4fa258b8a..5f5064396 100644 --- a/src/documents/migrations/1012_fix_archive_files.py +++ b/src/documents/migrations/1012_fix_archive_files.py @@ -160,7 +160,12 @@ def generate_filename(doc, counter=0, append_gpg=True, archive_filename=False): ############################################################################### -def create_archive_version(doc, retry_count=4): +def parse_wrapper(parser, path, mime_type, file_name): + # this is here so that I can mock this out for testing. + parser.parse(path, mime_type, file_name) + + +def create_archive_version(doc, retry_count=3): from documents.parsers import get_parser_class_for_mime_type, \ DocumentParser, \ ParseError @@ -172,8 +177,8 @@ def create_archive_version(doc, retry_count=4): for try_num in range(retry_count): parser: DocumentParser = parser_class(None, None) try: - parser.parse(source_path(doc), doc.mime_type, - os.path.basename(doc.filename)) + parse_wrapper(parser, source_path(doc), doc.mime_type, + os.path.basename(doc.filename)) doc.content = parser.get_text() if parser.get_archive_path() and os.path.isfile( @@ -225,25 +230,28 @@ def move_old_to_new_locations(apps, schema_editor): for doc in Document.objects.filter(archive_checksum__isnull=False): old_path = archive_path_old(doc) - if not os.path.isfile(old_path): - raise ValueError( - f"Archived document ID:{doc.id} does not exist at: " - f"{old_path}") - if old_path in old_archive_path_to_id: affected_document_ids.add(doc.id) affected_document_ids.add(old_archive_path_to_id[old_path]) else: old_archive_path_to_id[old_path] = doc.id - # check that we can regenerate these archive versions + # check that archive files of all unaffected documents are in place + for doc in Document.objects.filter(archive_checksum__isnull=False): + old_path = archive_path_old(doc) + if doc.id not in affected_document_ids and not os.path.isfile(old_path): + raise ValueError( + f"Archived document ID:{doc.id} does not exist at: " + f"{old_path}") + + # check that we can regenerate affected archive versions for doc_id in affected_document_ids: from documents.parsers import get_parser_class_for_mime_type doc = Document.objects.get(id=doc_id) parser_class = get_parser_class_for_mime_type(doc.mime_type) if not parser_class: - raise Exception( + raise ValueError( f"Document ID:{doc.id} has an invalid archived document, " f"but no parsers are available. Cannot migrate.") @@ -253,6 +261,9 @@ def move_old_to_new_locations(apps, schema_editor): old_path = archive_path_old(doc) # remove affected archive versions if os.path.isfile(old_path): + logger.debug( + f"Removing {old_path}" + ) os.unlink(old_path) else: # Set archive path for unaffected files @@ -267,8 +278,6 @@ def move_old_to_new_locations(apps, schema_editor): create_archive_version(doc) - - def move_new_to_old_locations(apps, schema_editor): Document = apps.get_model("documents", "Document") diff --git a/src/documents/tests/test_migration_archive_files.py b/src/documents/tests/test_migration_archive_files.py index 637aaeee0..b3f2e916c 100644 --- a/src/documents/tests/test_migration_archive_files.py +++ b/src/documents/tests/test_migration_archive_files.py @@ -2,10 +2,12 @@ import hashlib import os import shutil from pathlib import Path +from unittest import mock from django.conf import settings from django.test import override_settings +from documents.parsers import ParseError from documents.tests.utils import DirectoriesMixin, TestMigrations @@ -169,6 +171,11 @@ class TestMigrateArchiveFilesWithFilenameFormat(TestMigrateArchiveFiles): self.assertEqual(Document.objects.get(id=self.clash4.id).archive_filename, "clash.png.pdf") +def fake_parse_wrapper(parser, path, mime_type, file_name): + parser.archive_path = None + parser.text = "the text" + + @override_settings(PAPERLESS_FILENAME_FORMAT="") class TestMigrateArchiveFilesErrors(DirectoriesMixin, TestMigrations): @@ -185,6 +192,73 @@ class TestMigrateArchiveFilesErrors(DirectoriesMixin, TestMigrations): self.assertRaisesMessage(ValueError, "does not exist at: ", self.performMigration) + def test_parser_missing(self): + Document = self.apps.get_model("documents", "Document") + + doc1 = make_test_document(Document, "document", "invalid/typesss768", simple_png, "document.png", simple_pdf) + doc2 = make_test_document(Document, "document", "invalid/typesss768", simple_jpg, "document.jpg", simple_pdf) + + self.assertRaisesMessage(ValueError, "no parsers are available", self.performMigration) + + @mock.patch("documents.migrations.1012_fix_archive_files.parse_wrapper") + def test_parser_error(self, m): + m.side_effect = ParseError() + Document = self.apps.get_model("documents", "Document") + + doc1 = make_test_document(Document, "document", "image/png", simple_png, "document.png", simple_pdf) + doc2 = make_test_document(Document, "document", "application/pdf", simple_jpg, "document.jpg", simple_pdf) + + self.assertIsNotNone(doc1.archive_checksum) + self.assertIsNotNone(doc2.archive_checksum) + + with self.assertLogs() as capture: + self.performMigration() + + self.assertEqual(m.call_count, 6) + + self.assertEqual( + len(list(filter(lambda log: "Parse error, will try again in 5 seconds" in log, capture.output))), + 4) + + self.assertEqual( + len(list(filter(lambda log: "Unable to regenerate archive document for ID:" in log, capture.output))), + 2) + + Document = self.apps.get_model("documents", "Document") + + doc1 = Document.objects.get(id=doc1.id) + doc2 = Document.objects.get(id=doc2.id) + + self.assertIsNone(doc1.archive_checksum) + self.assertIsNone(doc2.archive_checksum) + self.assertIsNone(doc1.archive_filename) + self.assertIsNone(doc2.archive_filename) + + @mock.patch("documents.migrations.1012_fix_archive_files.parse_wrapper") + def test_parser_no_archive(self, m): + m.side_effect = fake_parse_wrapper + + Document = self.apps.get_model("documents", "Document") + + doc1 = make_test_document(Document, "document", "image/png", simple_png, "document.png", simple_pdf) + doc2 = make_test_document(Document, "document", "application/pdf", simple_jpg, "document.jpg", simple_pdf) + + with self.assertLogs() as capture: + self.performMigration() + + self.assertEqual( + len(list(filter(lambda log: "Parser did not return an archive document for document" in log, capture.output))), + 2) + + Document = self.apps.get_model("documents", "Document") + + doc1 = Document.objects.get(id=doc1.id) + doc2 = Document.objects.get(id=doc2.id) + + self.assertIsNone(doc1.archive_checksum) + self.assertIsNone(doc2.archive_checksum) + self.assertIsNone(doc1.archive_filename) + self.assertIsNone(doc2.archive_filename) @override_settings(PAPERLESS_FILENAME_FORMAT="") class TestMigrateArchiveFilesBackwards(DirectoriesMixin, TestMigrations): From e4fe5bebabb3b6b64912334a75ef7bbb5b5680a9 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 17:09:22 +0100 Subject: [PATCH 095/225] requirements --- Pipfile | 2 ++ Pipfile.lock | 28 ++++++++++++++++++---------- requirements.txt | 2 +- 3 files changed, 21 insertions(+), 11 deletions(-) diff --git a/Pipfile b/Pipfile index d9940d371..aae5f1506 100644 --- a/Pipfile +++ b/Pipfile @@ -54,6 +54,8 @@ concurrent-log-handler = "*" django-redis = "*" # uvloop 0.15+ incompatible with python 3.6 uvloop = "~=0.14.0" +# TODO: keep an eye on piwheel builds and update this once available (https://www.piwheels.org/project/cryptography/) +cryptography = "~=3.3.2" [dev-packages] coveralls = "*" diff --git a/Pipfile.lock b/Pipfile.lock index c05610316..9d1a75b88 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "d5459c3ca4499fcab45844d273d1da636fedd28e6108cb620b982175e81fdce5" + "sha256": "b3bed0a6b8981e8fffc1b6aa3bc35a0b1472f28e6f745c62469eb8045740e57b" }, "pipfile-spec": 6, "requires": {}, @@ -190,16 +190,24 @@ }, "cryptography": { "hashes": [ - "sha256:287032b6a7d86abc98e8e977b20138c53fea40e5b24e29090d5a675a973dcd10", - "sha256:288c65eea20bd89b11102c47b118bc1e0749386b0a0dfebba414076c5d4c8188", - "sha256:7eed937ad9b53280a5f53570d3a7dc93cb4412b6a3d58d4c6bb78cc26319c729", - "sha256:dab437c2e84628703e3358f0f06555a6259bc5039209d51aa3b05af667ff4fd0", - "sha256:ee5e19f0856b6fbbdbab15c2787ca65d203801d2d65d0b8de6218f424206c848", - "sha256:f21be9ec6b44c223b2024bbe59d394fadc7be320d18a8d595419afadb6cd5620", - "sha256:f6ea140d2736b7e1f0de4f988c43f76b0b3f3d365080e091715429ba218dce28" + "sha256:0d7b69674b738068fa6ffade5c962ecd14969690585aaca0a1b1fc9058938a72", + "sha256:1bd0ccb0a1ed775cd7e2144fe46df9dc03eefd722bbcf587b3e0616ea4a81eff", + "sha256:3c284fc1e504e88e51c428db9c9274f2da9f73fdf5d7e13a36b8ecb039af6e6c", + "sha256:49570438e60f19243e7e0d504527dd5fe9b4b967b5a1ff21cc12b57602dd85d3", + "sha256:541dd758ad49b45920dda3b5b48c968f8b2533d8981bcdb43002798d8f7a89ed", + "sha256:5a60d3780149e13b7a6ff7ad6526b38846354d11a15e21068e57073e29e19bed", + "sha256:7951a966613c4211b6612b0352f5bf29989955ee592c4a885d8c7d0f830d0433", + "sha256:922f9602d67c15ade470c11d616f2b2364950602e370c76f0c94c94ae672742e", + "sha256:a0f0b96c572fc9f25c3f4ddbf4688b9b38c69836713fb255f4a2715d93cbaf44", + "sha256:a777c096a49d80f9d2979695b835b0f9c9edab73b59e4ceb51f19724dda887ed", + "sha256:a9a4ac9648d39ce71c2f63fe7dc6db144b9fa567ddfc48b9fde1b54483d26042", + "sha256:aa4969f24d536ae2268c902b2c3d62ab464b5a66bcb247630d208a79a8098e9b", + "sha256:c7390f9b2119b2b43160abb34f63277a638504ef8df99f11cb52c1fda66a2e6f", + "sha256:ddd06e71c449a4fe10d0c60846280ee35d69ce49e3e413ce46d5f129e1468083", + "sha256:e18e6ab84dfb0ab997faf8cca25a86ff15dfea4027b986322026cc99e0a892da" ], - "markers": "python_version >= '3.6'", - "version": "==3.4.4" + "index": "pypi", + "version": "==3.3.2" }, "daphne": { "hashes": [ diff --git a/requirements.txt b/requirements.txt index 097cef78e..cf63b4660 100644 --- a/requirements.txt +++ b/requirements.txt @@ -24,7 +24,7 @@ click==7.1.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, coloredlogs==15.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' concurrent-log-handler==0.9.19 constantly==15.1.0 -cryptography==3.4.4; python_version >= '3.6' +cryptography==3.3.2 daphne==3.0.1; python_version >= '3.6' dateparser==0.7.6 django-cors-headers==3.7.0 From 45dfbf37473afec20cb715317657f784e8c8179a Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 18:27:41 +0100 Subject: [PATCH 096/225] downgrades --- docs/administration.rst | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/administration.rst b/docs/administration.rst index c54323e6e..98548efd4 100644 --- a/docs/administration.rst +++ b/docs/administration.rst @@ -171,6 +171,35 @@ Most of the update process is automated when using the ansible role. $ ansible-playbook playbook.yml +Downgrading Paperless +##################### + +Downgrades are possible. However, some updates also contain database migrations (these change the layout of the database and may move data). +In order to move back from a version that applied database migrations, you'll have to revert the database migration *before* downgrading, +and then downgrade paperless. + +This table lists the most recent database migrations for each versions: + ++---------+-------------------------+ +| Version | Latest migration number | ++---------+-------------------------+ +| 1.0.0 | 1011 | ++---------+-------------------------+ +| 1.1.0 | 1011 | ++---------+-------------------------+ +| 1.1.1 | 1012 | ++---------+-------------------------+ + +Execute the following management command to migrate your database: + +.. code:: shell-session + + $ python3 manage.py migrate documents + +.. note:: + + Some migrations cannot be undone. The command will issue errors if that happens. + Management utilities #################### From 252d4cb513e5281a1a986faf4ed30b40dfe7bb8f Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 18:55:39 +0100 Subject: [PATCH 097/225] update document admin --- src/documents/admin.py | 35 +++++++++-------------------------- 1 file changed, 9 insertions(+), 26 deletions(-) diff --git a/src/documents/admin.py b/src/documents/admin.py index 7cc33dc20..5447b6ef2 100755 --- a/src/documents/admin.py +++ b/src/documents/admin.py @@ -59,16 +59,19 @@ class DocumentAdmin(admin.ModelAdmin): list_display_links = ("title",) list_display = ( - "correspondent", - "title", - "tags_", "created", + "added", + "archive_serial_number", + "title", + "mime_type", + "filename", + "archive_filename" ) list_filter = ( - "document_type", - "tags", - "correspondent" + ("mime_type"), + ("archive_serial_number", admin.EmptyFieldListFilter), + ("archive_filename", admin.EmptyFieldListFilter), ) filter_horizontal = ("tags",) @@ -99,26 +102,6 @@ class DocumentAdmin(admin.ModelAdmin): index.add_or_update_document(obj) super(DocumentAdmin, self).save_model(request, obj, form, change) - @mark_safe - def tags_(self, obj): - r = "" - for tag in obj.tags.all(): - r += self._html_tag( - "span", - tag.name + ", " - ) - return r - - @staticmethod - def _html_tag(kind, inside=None, **kwargs): - attributes = format_html_join(' ', '{}="{}"', kwargs.items()) - - if inside is not None: - return format_html("<{kind} {attributes}>{inside}", - kind=kind, attributes=attributes, inside=inside) - - return format_html("<{} {}/>", kind, attributes) - class RuleInline(admin.TabularInline): model = SavedViewFilterRule From 658fb2f2084b915265794c7793ef80b19e9a7b04 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 20:01:35 +0100 Subject: [PATCH 098/225] remove invalid test cases --- src/documents/tests/test_admin.py | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/src/documents/tests/test_admin.py b/src/documents/tests/test_admin.py index 6915f595d..840747d68 100644 --- a/src/documents/tests/test_admin.py +++ b/src/documents/tests/test_admin.py @@ -23,18 +23,6 @@ class TestDocumentAdmin(DirectoriesMixin, TestCase): self.assertEqual(Document.objects.get(id=doc.id).title, "new title") m.assert_called_once() - def test_tags(self): - doc = Document.objects.create(title="test") - doc.tags.create(name="t1") - doc.tags.create(name="t2") - - self.assertEqual(self.doc_admin.tags_(doc), "t1, t2, ") - - def test_tags_empty(self): - doc = Document.objects.create(title="test") - - self.assertEqual(self.doc_admin.tags_(doc), "") - @mock.patch("documents.admin.index.remove_document") def test_delete_model(self, m): doc = Document.objects.create(title="test") From 21740a9d87e9e5639c0d5ee59680c7a4a6ae8873 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 20:05:02 +0100 Subject: [PATCH 099/225] changelog --- docs/changelog.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 9ab83bc1a..a0ca5e5c9 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -8,6 +8,8 @@ Changelog paperless-ng 1.1.1 ################## +This release contains new database migrations. + * Fixed a bug in the sanity checker that would cause it to display "x not in list" errors instead of actual issues. * Fixed a bug with filename generation for archive filenames that would cause the archive files of two documents to overlap. From dd19ea46fe8dd0b413d9eacca13148b323acba42 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 20:14:55 +0100 Subject: [PATCH 100/225] backup documentation --- docs/administration.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/administration.rst b/docs/administration.rst index f2c84016e..21b6ebcea 100644 --- a/docs/administration.rst +++ b/docs/administration.rst @@ -23,6 +23,12 @@ Options available to any installation of paperless: * The document exporter is also able to update an already existing export. Therefore, incremental backups with ``rsync`` are entirely possible. +.. caution:: + + You cannot import the export generated with one version of paperless in a + different version of paperless. The export contains an exact image of the + database, and migrations may change the database layout. + Options available to docker installations: * Backup the docker volumes. These usually reside within From 857fe3a55c0e24d748544358a8a4acc70f1361af Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 21:34:39 +0100 Subject: [PATCH 101/225] fix one incorrect use of archive_version --- src/documents/serialisers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py index fb0962902..b01f82805 100644 --- a/src/documents/serialisers.py +++ b/src/documents/serialisers.py @@ -129,7 +129,7 @@ class DocumentSerializer(DynamicFieldsModelSerializer): return obj.get_public_filename() def get_archived_file_name(self, obj): - if obj.archive_checksum: + if obj.has_archive_version: return obj.get_public_filename(archive=True) else: return None From 3fa7dcb0cbfed934e35bd7fce995427ed1861098 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 21:34:50 +0100 Subject: [PATCH 102/225] changes to the admin document list --- src/documents/admin.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/documents/admin.py b/src/documents/admin.py index 5447b6ef2..f87ab1ec5 100755 --- a/src/documents/admin.py +++ b/src/documents/admin.py @@ -59,9 +59,7 @@ class DocumentAdmin(admin.ModelAdmin): list_display_links = ("title",) list_display = ( - "created", - "added", - "archive_serial_number", + "id", "title", "mime_type", "filename", @@ -76,7 +74,7 @@ class DocumentAdmin(admin.ModelAdmin): filter_horizontal = ("tags",) - ordering = ["-created"] + ordering = ["-id"] date_hierarchy = "created" From 2f8546110908f70dcb34f971a5c291896ccad6c8 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 10 Feb 2021 23:53:48 +0100 Subject: [PATCH 103/225] added some test cases that I still need to address --- src/documents/tests/test_file_handling.py | 32 +++++++++++++++++++++++ src/documents/tests/test_management.py | 16 ++++++++++++ 2 files changed, 48 insertions(+) diff --git a/src/documents/tests/test_file_handling.py b/src/documents/tests/test_file_handling.py index 927c865c6..a806e1025 100644 --- a/src/documents/tests/test_file_handling.py +++ b/src/documents/tests/test_file_handling.py @@ -507,6 +507,38 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): self.assertTrue(os.path.isfile(existing_archive_file)) self.assertEqual(doc.archive_filename, "none/my_doc_01.pdf") + @override_settings(PAPERLESS_FILENAME_FORMAT="{title}") + def test_move_original_only(self): + original = os.path.join(settings.ORIGINALS_DIR, "document_01.pdf") + archive = os.path.join(settings.ARCHIVE_DIR, "document.pdf") + Path(original).touch() + Path(archive).touch() + + doc = Document.objects.create(mime_type="application/pdf", title="document", filename="document_01.pdf", checksum="A", + archive_checksum="B", archive_filename="document.pdf") + + self.assertEqual(doc.filename, "document.pdf") + self.assertEqual(doc.archive_filename, "document.pdf") + + self.assertTrue(os.path.isfile(doc.source_path)) + self.assertTrue(os.path.isfile(doc.archive_path)) + + @override_settings(PAPERLESS_FILENAME_FORMAT="{title}") + def test_move_archive_only(self): + original = os.path.join(settings.ORIGINALS_DIR, "document.pdf") + archive = os.path.join(settings.ARCHIVE_DIR, "document_01.pdf") + Path(original).touch() + Path(archive).touch() + + doc = Document.objects.create(mime_type="application/pdf", title="document", filename="document.pdf", checksum="A", + archive_checksum="B", archive_filename="document_01.pdf") + + self.assertEqual(doc.filename, "document.pdf") + self.assertEqual(doc.archive_filename, "document.pdf") + + self.assertTrue(os.path.isfile(doc.source_path)) + self.assertTrue(os.path.isfile(doc.archive_path)) + @override_settings(PAPERLESS_FILENAME_FORMAT="{correspondent}/{title}") @mock.patch("documents.signals.handlers.os.rename") def test_move_archive_error(self, m): diff --git a/src/documents/tests/test_management.py b/src/documents/tests/test_management.py index 978dff8b3..1a550a4b4 100644 --- a/src/documents/tests/test_management.py +++ b/src/documents/tests/test_management.py @@ -43,11 +43,27 @@ class TestArchiver(DirectoriesMixin, TestCase): doc = Document.objects.get(id=doc.id) self.assertIsNotNone(doc.checksum) + self.assertIsNotNone(doc.archive_checksum) self.assertTrue(os.path.isfile(doc.archive_path)) self.assertTrue(os.path.isfile(doc.source_path)) self.assertTrue(filecmp.cmp(sample_file, doc.source_path)) self.assertEqual(doc.archive_filename, "none/A.pdf") + @override_settings(PAPERLESS_FILENAME_FORMAT="{title}") + def test_naming_priorities(self): + doc1 = Document.objects.create(checksum="A", title="document", content="first document", mime_type="application/pdf", filename="document.pdf") + doc2 = Document.objects.create(checksum="B", title="document", content="second document", mime_type="application/pdf", filename="document_01.pdf") + shutil.copy(sample_file, os.path.join(self.dirs.originals_dir, f"document.pdf")) + shutil.copy(sample_file, os.path.join(self.dirs.originals_dir, f"document_01.pdf")) + + handle_document(doc2.pk) + handle_document(doc1.pk) + + doc1 = Document.objects.get(id=doc1.id) + doc2 = Document.objects.get(id=doc2.id) + + self.assertEqual(doc1.archive_filename, "document.pdf") + self.assertEqual(doc2.archive_filename, "document_01.pdf") class TestDecryptDocuments(TestCase): From 421115352784e0078c44fda56d7858d65e8b307d Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 11 Feb 2021 13:47:17 +0100 Subject: [PATCH 104/225] update file renaming logic --- src/documents/signals/handlers.py | 69 +++++++++---------- src/documents/tests/test_file_handling.py | 14 ++++ .../tests/test_migration_archive_files.py | 1 + 3 files changed, 46 insertions(+), 38 deletions(-) diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py index 17260fa8a..c5e213949 100755 --- a/src/documents/signals/handlers.py +++ b/src/documents/signals/handlers.py @@ -208,21 +208,16 @@ def update_filename_and_move_files(sender, instance, **kwargs): with FileLock(settings.MEDIA_LOCK): old_filename = instance.filename new_filename = generate_unique_filename(instance) - - if new_filename == instance.filename: - # Don't do anything if its the same. - return - + move_original = old_filename != new_filename old_source_path = instance.source_path new_source_path = os.path.join(settings.ORIGINALS_DIR, new_filename) - if not validate_move(instance, old_source_path, new_source_path): + if move_original and not validate_move(instance, old_source_path, new_source_path): return - # archive files are optional, archive checksum tells us if we have one, - # since this is None for documents without archived files. + old_archive_filename = instance.archive_filename + if instance.has_archive_version: - old_archive_filename = instance.archive_filename new_archive_filename = generate_unique_filename( instance, archive_filename=True ) @@ -230,25 +225,26 @@ def update_filename_and_move_files(sender, instance, **kwargs): new_archive_path = os.path.join(settings.ARCHIVE_DIR, new_archive_filename) - if not validate_move(instance, old_archive_path, new_archive_path): + move_archive = old_archive_filename != new_archive_filename + if move_archive and not validate_move(instance, old_archive_path, new_archive_path): return - - create_source_path_directory(new_archive_path) else: - old_archive_filename = None - new_archive_filename = None - old_archive_path = None - new_archive_path = None + move_archive = False - create_source_path_directory(new_source_path) + if not move_original and not move_archive: + # Don't do anything if filenames did not change. + return try: - os.rename(old_source_path, new_source_path) - instance.filename = new_filename + if move_original: + instance.filename = new_filename + create_source_path_directory(new_source_path) + os.rename(old_source_path, new_source_path) - if instance.has_archive_version: - os.rename(old_archive_path, new_archive_path) + if move_archive: instance.archive_filename = new_archive_filename + create_source_path_directory(new_archive_path) + os.rename(old_archive_path, new_archive_path) # Don't save() here to prevent infinite recursion. Document.objects.filter(pk=instance.pk).update( @@ -256,16 +252,24 @@ def update_filename_and_move_files(sender, instance, **kwargs): archive_filename=instance.archive_filename, ) - except OSError as e: + except Exception as e: + # This happens when either: + # - moving the files failed due to file system errors + # - saving to the database failed due to database errors + # In both cases, we need to revert to the original state. + + # restore old values on the instance instance.filename = old_filename instance.archive_filename = old_archive_filename - # this happens when we can't move a file. If that's the case for - # the archive file, we try our best to revert the changes. - # no need to save the instance, the update() has not happened yet. + + # Try to move files to their original location. try: - os.rename(new_source_path, old_source_path) - if instance.has_archive_version: + if move_original and os.path.isfile(new_source_path): + os.rename(new_source_path, old_source_path) + + if move_archive and os.path.isfile(new_archive_path): os.rename(new_archive_path, old_archive_path) + except Exception as e: # This is fine, since: # A: if we managed to move source from A to B, we will also @@ -276,17 +280,6 @@ def update_filename_and_move_files(sender, instance, **kwargs): # B: if moving the orignal file failed, nothing has changed # anyway. pass - except DatabaseError as e: - # this happens after moving files, so move them back into place. - # since moving them once succeeded, it's very likely going to - # succeed again. - os.rename(new_source_path, old_source_path) - if instance.has_archive_version: - os.rename(new_archive_path, old_archive_path) - instance.filename = old_filename - instance.archive_filename = old_archive_filename - # again, no need to save the instance, since the actual update() - # operation failed. # finally, remove any empty sub folders. This will do nothing if # something has failed above. diff --git a/src/documents/tests/test_file_handling.py b/src/documents/tests/test_file_handling.py index a806e1025..59af7e317 100644 --- a/src/documents/tests/test_file_handling.py +++ b/src/documents/tests/test_file_handling.py @@ -446,6 +446,18 @@ class TestFileHandling(DirectoriesMixin, TestCase): self.assertEqual(document2.filename, "qwe.pdf") + @override_settings(PAPERLESS_FILENAME_FORMAT="{title}") + @mock.patch("documents.signals.handlers.Document.objects.filter") + def test_no_update_without_change(self, m): + doc = Document.objects.create(title="document", filename="document.pdf", archive_filename="document.pdf", checksum="A", archive_checksum="B", mime_type="application/pdf") + Path(doc.source_path).touch() + Path(doc.archive_path).touch() + + doc.save() + + m.assert_not_called() + + class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): @@ -558,6 +570,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): Path(archive).touch() doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", checksum="A", archive_checksum="B", archive_filename="0000001.pdf") + m.assert_called() self.assertTrue(os.path.isfile(original)) self.assertTrue(os.path.isfile(archive)) self.assertTrue(os.path.isfile(doc.source_path)) @@ -595,6 +608,7 @@ class TestFileHandlingWithArchive(DirectoriesMixin, TestCase): Path(archive).touch() doc = Document.objects.create(mime_type="application/pdf", title="my_doc", filename="0000001.pdf", archive_filename="0000001.pdf", checksum="A", archive_checksum="B") + m.assert_called() self.assertTrue(os.path.isfile(original)) self.assertTrue(os.path.isfile(archive)) self.assertTrue(os.path.isfile(doc.source_path)) diff --git a/src/documents/tests/test_migration_archive_files.py b/src/documents/tests/test_migration_archive_files.py index b3f2e916c..6217ae05f 100644 --- a/src/documents/tests/test_migration_archive_files.py +++ b/src/documents/tests/test_migration_archive_files.py @@ -260,6 +260,7 @@ class TestMigrateArchiveFilesErrors(DirectoriesMixin, TestMigrations): self.assertIsNone(doc1.archive_filename) self.assertIsNone(doc2.archive_filename) + @override_settings(PAPERLESS_FILENAME_FORMAT="") class TestMigrateArchiveFilesBackwards(DirectoriesMixin, TestMigrations): From ed0b1fe1151b1f4ec446dacd67c43ebd7e2ab62a Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Thu, 11 Feb 2021 22:16:41 +0100 Subject: [PATCH 105/225] better exception logging --- src/documents/classifier.py | 4 ++-- src/documents/index.py | 4 ++-- src/documents/management/commands/document_archiver.py | 2 +- src/documents/management/commands/document_consumer.py | 6 ++---- src/paperless_mail/tasks.py | 7 ++----- 5 files changed, 9 insertions(+), 14 deletions(-) diff --git a/src/documents/classifier.py b/src/documents/classifier.py index 81f860314..c880e50d6 100755 --- a/src/documents/classifier.py +++ b/src/documents/classifier.py @@ -43,9 +43,9 @@ def load_classifier(): version=version, timeout=86400) except (EOFError, IncompatibleClassifierVersionError) as e: # there's something wrong with the model file. - logger.error( + logger.exception( f"Unrecoverable error while loading document " - f"classification model: {str(e)}, deleting model file." + f"classification model, deleting model file." ) os.unlink(settings.MODEL_FILE) classifier = None diff --git a/src/documents/index.py b/src/documents/index.py index bc17daeb1..ea788f4b3 100644 --- a/src/documents/index.py +++ b/src/documents/index.py @@ -78,8 +78,8 @@ def open_index(recreate=False): try: if exists_in(settings.INDEX_DIR) and not recreate: return open_dir(settings.INDEX_DIR, schema=get_schema()) - except Exception as e: - logger.error(f"Error while opening the index: {e}, recreating.") + except Exception: + logger.exception(f"Error while opening the index, recreating.") if not os.path.isdir(settings.INDEX_DIR): os.makedirs(settings.INDEX_DIR, exist_ok=True) diff --git a/src/documents/management/commands/document_archiver.py b/src/documents/management/commands/document_archiver.py index fe8c8b530..297c95121 100644 --- a/src/documents/management/commands/document_archiver.py +++ b/src/documents/management/commands/document_archiver.py @@ -60,7 +60,7 @@ def handle_document(document_id): index.update_document(writer, document) except Exception as e: - logger.error(f"Error while parsing document {document}: {str(e)}") + logger.exception(f"Error while parsing document {document}") finally: parser.cleanup() diff --git a/src/documents/management/commands/document_consumer.py b/src/documents/management/commands/document_consumer.py index c01743628..aaa644891 100644 --- a/src/documents/management/commands/document_consumer.py +++ b/src/documents/management/commands/document_consumer.py @@ -54,8 +54,7 @@ def _consume(filepath): if settings.CONSUMER_SUBDIRS_AS_TAGS: tag_ids = _tags_from_path(filepath) except Exception as e: - logger.error( - "Error creating tags from path: {}".format(e)) + logger.exception("Error creating tags from path") try: async_task("documents.tasks.consume_file", @@ -66,8 +65,7 @@ def _consume(filepath): # Catch all so that the consumer won't crash. # This is also what the test case is listening for to check for # errors. - logger.error( - "Error while consuming document: {}".format(e)) + logger.exception("Error while consuming document") def _consume_wait_unmodified(file, num_tries=20, wait_time=1): diff --git a/src/paperless_mail/tasks.py b/src/paperless_mail/tasks.py index 086edef7b..c591f04b9 100644 --- a/src/paperless_mail/tasks.py +++ b/src/paperless_mail/tasks.py @@ -13,11 +13,8 @@ def process_mail_accounts(): try: total_new_documents += MailAccountHandler().handle_mail_account( account) - except MailError as e: - logger.error( - f"Error while processing mail account {account}: {e}", - exc_info=True - ) + except MailError: + logger.exception(f"Error while processing mail account {account}") if total_new_documents > 0: return f"Added {total_new_documents} document(s)." From 5b56fad9c7ff3561cfd272dad9483cfea6cae040 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Fri, 12 Feb 2021 01:31:50 +0100 Subject: [PATCH 106/225] fix test case --- src/documents/file_handling.py | 22 +++++++- src/documents/signals/handlers.py | 93 +++++++++++++++---------------- 2 files changed, 66 insertions(+), 49 deletions(-) diff --git a/src/documents/file_handling.py b/src/documents/file_handling.py index 8c0e2148b..d7a42db24 100644 --- a/src/documents/file_handling.py +++ b/src/documents/file_handling.py @@ -79,7 +79,19 @@ def many_to_dictionary(field): return mydictionary -def generate_unique_filename(doc, archive_filename=False): +def generate_unique_filename(doc, + archive_filename=False): + """ + Generates a unique filename for doc in settings.ORIGINALS_DIR. + + The returned filename is guaranteed to be either the current filename + of the document if unchanged, or a new filename that does not correspondent + to any existing files. The function will append _01, _02, etc to the + filename before the extension to avoid conflicts. + + If archive_filename is True, return a unique archive filename instead. + + """ if archive_filename: old_filename = doc.archive_filename root = settings.ARCHIVE_DIR @@ -87,6 +99,14 @@ def generate_unique_filename(doc, archive_filename=False): old_filename = doc.filename root = settings.ORIGINALS_DIR + # If generating archive filenames, try to make a name that is similar to + # the original filename first. + + if archive_filename and doc.filename: + new_filename = os.path.splitext(doc.filename)[0] + ".pdf" + if new_filename == old_filename or not os.path.exists(os.path.join(root, new_filename)): # NOQA: E501 + return new_filename + counter = 0 while True: diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py index c5e213949..eb981661c 100755 --- a/src/documents/signals/handlers.py +++ b/src/documents/signals/handlers.py @@ -173,21 +173,23 @@ def cleanup_document_deletion(sender, instance, using, **kwargs): ) +class CannotMoveFilesException(Exception): + pass + + def validate_move(instance, old_path, new_path): if not os.path.isfile(old_path): # Can't do anything if the old file does not exist anymore. logger.fatal( f"Document {str(instance)}: File {old_path} has gone.") - return False + raise CannotMoveFilesException() if os.path.isfile(new_path): # Can't do anything if the new file already exists. Skip updating file. logger.warning( f"Document {str(instance)}: Cannot rename file " f"since target path {new_path} already exists.") - return False - - return True + raise CannotMoveFilesException() @receiver(models.signals.m2m_changed, sender=Document.tags.through) @@ -206,45 +208,40 @@ def update_filename_and_move_files(sender, instance, **kwargs): return with FileLock(settings.MEDIA_LOCK): - old_filename = instance.filename - new_filename = generate_unique_filename(instance) - move_original = old_filename != new_filename - old_source_path = instance.source_path - new_source_path = os.path.join(settings.ORIGINALS_DIR, new_filename) - - if move_original and not validate_move(instance, old_source_path, new_source_path): - return - - old_archive_filename = instance.archive_filename - - if instance.has_archive_version: - new_archive_filename = generate_unique_filename( - instance, archive_filename=True - ) - old_archive_path = instance.archive_path - new_archive_path = os.path.join(settings.ARCHIVE_DIR, - new_archive_filename) - - move_archive = old_archive_filename != new_archive_filename - if move_archive and not validate_move(instance, old_archive_path, new_archive_path): - return - else: - move_archive = False - - if not move_original and not move_archive: - # Don't do anything if filenames did not change. - return - try: + old_filename = instance.filename + old_source_path = instance.source_path + + instance.filename = generate_unique_filename(instance) + move_original = old_filename != instance.filename + + old_archive_filename = instance.archive_filename + old_archive_path = instance.archive_path + + if instance.has_archive_version: + + instance.archive_filename = generate_unique_filename( + instance, archive_filename=True + ) + + move_archive = old_archive_filename != instance.archive_filename # NOQA: E501 + else: + move_archive = False + + if not move_original and not move_archive: + # Don't do anything if filenames did not change. + return + if move_original: - instance.filename = new_filename - create_source_path_directory(new_source_path) - os.rename(old_source_path, new_source_path) + validate_move(instance, old_source_path, instance.source_path) + create_source_path_directory(instance.source_path) + os.rename(old_source_path, instance.source_path) if move_archive: - instance.archive_filename = new_archive_filename - create_source_path_directory(new_archive_path) - os.rename(old_archive_path, new_archive_path) + validate_move( + instance, old_archive_path, instance.archive_path) + create_source_path_directory(instance.archive_path) + os.rename(old_archive_path, instance.archive_path) # Don't save() here to prevent infinite recursion. Document.objects.filter(pk=instance.pk).update( @@ -252,23 +249,19 @@ def update_filename_and_move_files(sender, instance, **kwargs): archive_filename=instance.archive_filename, ) - except Exception as e: + except (OSError, DatabaseError, CannotMoveFilesException): # This happens when either: # - moving the files failed due to file system errors # - saving to the database failed due to database errors # In both cases, we need to revert to the original state. - # restore old values on the instance - instance.filename = old_filename - instance.archive_filename = old_archive_filename - # Try to move files to their original location. try: - if move_original and os.path.isfile(new_source_path): - os.rename(new_source_path, old_source_path) + if move_original and os.path.isfile(instance.source_path): + os.rename(instance.source_path, old_source_path) - if move_archive and os.path.isfile(new_archive_path): - os.rename(new_archive_path, old_archive_path) + if move_archive and os.path.isfile(instance.archive_path): + os.rename(instance.archive_path, old_archive_path) except Exception as e: # This is fine, since: @@ -281,6 +274,10 @@ def update_filename_and_move_files(sender, instance, **kwargs): # anyway. pass + # restore old values on the instance + instance.filename = old_filename + instance.archive_filename = old_archive_filename + # finally, remove any empty sub folders. This will do nothing if # something has failed above. if not os.path.isfile(old_source_path): From d6c34719091351e0606acd34fd9524bce52346be Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Fri, 12 Feb 2021 16:53:51 +0100 Subject: [PATCH 107/225] reprganized docker file, less layers, new shortcuts for management commands --- Dockerfile | 59 +++++++++++++-------------- docker/install_management_commands.sh | 6 +++ docker/management_script.sh | 15 +++++++ 3 files changed, 50 insertions(+), 30 deletions(-) create mode 100755 docker/install_management_commands.sh create mode 100644 docker/management_script.sh diff --git a/Dockerfile b/Dockerfile index 6247ef009..57f96c39f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -10,10 +10,6 @@ RUN ./configure && make FROM python:3.7-slim -WORKDIR /usr/src/paperless/ - -COPY requirements.txt ./ - # Binary dependencies RUN apt-get update \ && apt-get -y --no-install-recommends install \ @@ -49,16 +45,24 @@ RUN apt-get update \ tesseract-ocr-spa \ unpaper \ zlib1g \ - && rm -rf /var/lib/apt/lists/* # This pulls in updated dependencies from bullseye to fix some issues with file type detection. # TODO: Remove this once bullseye releases. -RUN echo "deb http://deb.debian.org/debian bullseye main" > /etc/apt/sources.list.d/bullseye.list \ + && echo "deb http://deb.debian.org/debian bullseye main" > /etc/apt/sources.list.d/bullseye.list \ && apt-get update \ && apt-get install --no-install-recommends -y file libmagic-dev \ && rm -rf /var/lib/apt/lists/* \ && rm /etc/apt/sources.list.d/bullseye.list +# copy jbig2enc +COPY --from=jbig2enc /usr/src/jbig2enc/src/.libs/libjbig2enc* /usr/local/lib/ +COPY --from=jbig2enc /usr/src/jbig2enc/src/jbig2 /usr/local/bin/ +COPY --from=jbig2enc /usr/src/jbig2enc/src/*.h /usr/local/include/ + +WORKDIR /usr/src/paperless/src/ + +COPY requirements.txt ../ + # Python dependencies RUN apt-get update \ && apt-get -y --no-install-recommends install \ @@ -67,41 +71,36 @@ RUN apt-get update \ libpq-dev \ libqpdf-dev \ && python3 -m pip install --upgrade --no-cache-dir supervisor \ - && python3 -m pip install --no-cache-dir -r requirements.txt \ + && python3 -m pip install --no-cache-dir -r ../requirements.txt \ && apt-get -y purge build-essential libqpdf-dev \ && apt-get -y autoremove --purge \ - && rm -rf /var/lib/apt/lists/* \ - && mkdir /var/log/supervisord /var/run/supervisord + && rm -rf /var/lib/apt/lists/* +# setup docker-specific things +COPY docker/ ./docker/ -# copy scripts -# this fixes issues with imagemagick and PDF -COPY docker/imagemagick-policy.xml /etc/ImageMagick-6/policy.xml - -COPY gunicorn.conf.py ./ -COPY docker/supervisord.conf /etc/supervisord.conf -COPY docker/docker-entrypoint.sh /sbin/docker-entrypoint.sh - -# copy jbig2enc -COPY --from=jbig2enc /usr/src/jbig2enc/src/.libs/libjbig2enc* /usr/local/lib/ -COPY --from=jbig2enc /usr/src/jbig2enc/src/jbig2 /usr/local/bin/ -COPY --from=jbig2enc /usr/src/jbig2enc/src/*.h /usr/local/include/ +RUN cd docker \ + && cp imagemagick-policy.xml /etc/ImageMagick-6/policy.xml \ + && mkdir /var/log/supervisord /var/run/supervisord \ + && cp supervisord.conf /etc/supervisord.conf \ + && cp docker-entrypoint.sh /sbin/docker-entrypoint.sh \ + && chmod 755 /sbin/docker-entrypoint.sh \ + && chmod +x install_management_commands.sh \ + && ./install_management_commands.sh \ + && cd .. \ + && rm docker -rf +COPY gunicorn.conf.py ../ # copy app -COPY src/ ./src/ +COPY src/ ./ # add users, setup scripts RUN addgroup --gid 1000 paperless \ && useradd --uid 1000 --gid paperless --home-dir /usr/src/paperless paperless \ - && chown -R paperless:paperless . \ - && chmod 755 /sbin/docker-entrypoint.sh - -WORKDIR /usr/src/paperless/src/ - -RUN sudo -HEu paperless python3 manage.py collectstatic --clear --no-input - -RUN sudo -HEu paperless python3 manage.py compilemessages + && chown -R paperless:paperless ../ \ + && sudo -HEu paperless python3 manage.py collectstatic --clear --no-input \ + && sudo -HEu paperless python3 manage.py compilemessages VOLUME ["/usr/src/paperless/data", "/usr/src/paperless/media", "/usr/src/paperless/consume", "/usr/src/paperless/export"] ENTRYPOINT ["/sbin/docker-entrypoint.sh"] diff --git a/docker/install_management_commands.sh b/docker/install_management_commands.sh new file mode 100755 index 000000000..6aeca635d --- /dev/null +++ b/docker/install_management_commands.sh @@ -0,0 +1,6 @@ +for command in document_archiver document_exporter document_importer mail_fetcher document_create_classifier document_index document_renamer document_retagger document_thumbnails; +do + echo "installing $command..." + sed "s/management_command/$command/g" management_script.sh > /usr/local/bin/$command + chmod +x /usr/local/bin/$command +done diff --git a/docker/management_script.sh b/docker/management_script.sh new file mode 100644 index 000000000..bffa26e98 --- /dev/null +++ b/docker/management_script.sh @@ -0,0 +1,15 @@ +#!/bin/bash + +set -e + +cd /usr/src/paperless/src/ + +if [[ $(id -u) == 0 ]] ; +then + sudo -HEu paperless python3 manage.py management_command "$@" +elif [[ $(id -un) == "paperless" ]] ; +then + python3 manage.py management_command "$@" +else + echo "Unknown user." +fi From 6ac90181cbc8bf56ac9c9060117c750399a1fda9 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Fri, 12 Feb 2021 16:54:00 +0100 Subject: [PATCH 108/225] documentation and changelog --- docs/administration.rst | 18 +++++++++++++----- docs/changelog.rst | 3 +++ 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/docs/administration.rst b/docs/administration.rst index 21b6ebcea..5773a80cc 100644 --- a/docs/administration.rst +++ b/docs/administration.rst @@ -206,26 +206,34 @@ Execute the following management command to migrate your database: Some migrations cannot be undone. The command will issue errors if that happens. +.. _utilities-management-commands: + Management utilities #################### Paperless comes with some management commands that perform various maintenance -tasks on your paperless instance. You can invoke these commands either by +tasks on your paperless instance. You can invoke these commands in the following way: + +With docker-compose, while paperless is running: .. code:: shell-session $ cd /path/to/paperless - $ docker-compose exec -u paperless webserver python3 manage.py + $ docker-compose exec webserver -or +With docker, while paperless is running: + +.. code:: shell-session + + $ docker exec -it + +Bare metal: .. code:: shell-session $ cd /path/to/paperless/src $ python3 manage.py -depending on whether you use docker or not. - All commands have built-in help, which can be accessed by executing them with the argument ``--help``. diff --git a/docs/changelog.rst b/docs/changelog.rst index a0ca5e5c9..bf07a3c6a 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -23,6 +23,9 @@ This release contains new database migrations. * Added ASN as a placeholder field to the filename format. +* The docker image now comes with built-in shortcuts for most management commands. These are now the recommended way to execute management commands, since these + also ensure that they're always executed as the paperless user and you're less likely to run into permission issues. See :ref:`utilities-management-commands`. + paperless-ng 1.1.0 ################## From 13e91d8c95879b010c080065bf4ddb5f4387e277 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Fri, 12 Feb 2021 18:04:15 +0100 Subject: [PATCH 109/225] changelog --- docs/changelog.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index bf07a3c6a..d0d38cce4 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -14,10 +14,10 @@ This release contains new database migrations. * Fixed a bug with filename generation for archive filenames that would cause the archive files of two documents to overlap. - * This happened when ``PAPERLESS_FILENAME_FORMAT`` is used and the filename for two documents is the same, except for the file extension. + * This happened when ``PAPERLESS_FILENAME_FORMAT`` is used and the filenames of two or more documents are the same, except for the file extension. * Paperless will now store the archive filename in the database as well instead of deriving it from the original filename, and use the same logic for detecting and avoiding filename clashes that's also used for original filenames. - * The migrations will repair any missing archive files. If you're using tika, ensure that tika is running while performing the migration. Docker will take care of that. + * The migrations will repair any missing archive files. If you're using tika, ensure that tika is running while performing the migration. Docker-compose will take care of that. * Fixed a bug with thumbnail regeneration when TIKA integration was used. From ed478a1d739bc1714817d02dd0a8eeb8d1e046a5 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Fri, 12 Feb 2021 18:20:17 +0100 Subject: [PATCH 110/225] change thumbnail display for extra wide images #433 --- docs/changelog.rst | 5 +++++ .../document-card-large/document-card-large.component.scss | 2 +- .../document-card-small/document-card-small.component.scss | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index d0d38cce4..47dd1bc02 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -5,6 +5,11 @@ Changelog ********* +paperless-ng 1.1.2 +################## + +* Always show top left corner of thumbnails, even for extra wide documents. + paperless-ng 1.1.1 ################## diff --git a/src-ui/src/app/components/document-list/document-card-large/document-card-large.component.scss b/src-ui/src/app/components/document-list/document-card-large/document-card-large.component.scss index a549b6f53..c49de93b5 100644 --- a/src-ui/src/app/components/document-list/document-card-large/document-card-large.component.scss +++ b/src-ui/src/app/components/document-list/document-card-large/document-card-large.component.scss @@ -6,7 +6,7 @@ .doc-img { object-fit: cover; - object-position: top; + object-position: top left; height: 100%; position: absolute; mix-blend-mode: multiply; diff --git a/src-ui/src/app/components/document-list/document-card-small/document-card-small.component.scss b/src-ui/src/app/components/document-list/document-card-small/document-card-small.component.scss index a4af1bb11..cbe8fa6f6 100644 --- a/src-ui/src/app/components/document-list/document-card-small/document-card-small.component.scss +++ b/src-ui/src/app/components/document-list/document-card-small/document-card-small.component.scss @@ -2,7 +2,7 @@ .doc-img { object-fit: cover; - object-position: top; + object-position: top left; height: 200px; mix-blend-mode: multiply; } From 8b2965d55b160d5fe2b889ddc0f5b8591e39230c Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 13 Feb 2021 16:39:29 +0100 Subject: [PATCH 111/225] added sanity checker management command for manual execution #534 --- docker/install_management_commands.sh | 2 +- docs/administration.rst | 28 +++++++++++++++ docs/changelog.rst | 2 ++ .../commands/document_sanity_checker.py | 27 ++++++++++++++ src/documents/sanity_checker.py | 10 ++++-- src/documents/tests/test_management.py | 35 +++++++++++++++++++ 6 files changed, 101 insertions(+), 3 deletions(-) create mode 100644 src/documents/management/commands/document_sanity_checker.py diff --git a/docker/install_management_commands.sh b/docker/install_management_commands.sh index 6aeca635d..17fb8f277 100755 --- a/docker/install_management_commands.sh +++ b/docker/install_management_commands.sh @@ -1,4 +1,4 @@ -for command in document_archiver document_exporter document_importer mail_fetcher document_create_classifier document_index document_renamer document_retagger document_thumbnails; +for command in document_archiver document_exporter document_importer mail_fetcher document_create_classifier document_index document_renamer document_retagger document_thumbnails document_sanity_checker; do echo "installing $command..." sed "s/management_command/$command/g" management_script.sh > /usr/local/bin/$command diff --git a/docs/administration.rst b/docs/administration.rst index 5773a80cc..c91f501bd 100644 --- a/docs/administration.rst +++ b/docs/administration.rst @@ -410,6 +410,34 @@ the naming scheme. The command takes no arguments and processes all your documents at once. +.. _utilities-sanity-checker: + +Sanity checker +============== + +Paperless has a built-in sanity checker that inspects your document collection for issues. + +The issues detected by the sanity checker are as follows: + +* Missing original files. +* Missing archive files. +* Inaccessible original files due to improper permissions. +* Inaccessible archive files due to improper permissions. +* Corrupted original documents by comparing their checksum against what is stored in the database. +* Corrupted archive documents by comparing their checksum against what is stored in the database. +* Missing thumbnails. +* Inaccessible thumbnails due to improper permissions. +* Documents without any content (warning). +* Orphaned files in the media directory (warning). These are files that are not referenced by any document im paperless. + + +.. code:: + + document_sanity_checker + +The command takes no arguments. Depending on the size of your document archive, this may take some time. + + Fetching e-mail =============== diff --git a/docs/changelog.rst b/docs/changelog.rst index 47dd1bc02..f15c73605 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -10,6 +10,8 @@ paperless-ng 1.1.2 * Always show top left corner of thumbnails, even for extra wide documents. +* Added a management command for executing the sanity checker directly. See :ref:`utilities-sanity-checker`. + paperless-ng 1.1.1 ################## diff --git a/src/documents/management/commands/document_sanity_checker.py b/src/documents/management/commands/document_sanity_checker.py new file mode 100644 index 000000000..b5982ed90 --- /dev/null +++ b/src/documents/management/commands/document_sanity_checker.py @@ -0,0 +1,27 @@ +import logging +from django.core.management.base import BaseCommand +from documents.sanity_checker import check_sanity, SanityError, SanityWarning + +logger = logging.getLogger("paperless.management.sanity_checker") + + +class Command(BaseCommand): + + help = """ + This command checks your document archive for issues. + """.replace(" ", "") + + def handle(self, *args, **options): + + messages = check_sanity(progress=True) + + if len(messages) == 0: + logger.info("No issues found.") + else: + for msg in messages: + if type(msg) == SanityError: + logger.error(str(msg)) + elif type(msg) == SanityWarning: + logger.warning(str(msg)) + else: + logger.info((str(msg))) diff --git a/src/documents/sanity_checker.py b/src/documents/sanity_checker.py index 2d30a59d7..f42062154 100644 --- a/src/documents/sanity_checker.py +++ b/src/documents/sanity_checker.py @@ -2,6 +2,7 @@ import hashlib import os from django.conf import settings +from tqdm import tqdm from documents.models import Document @@ -38,7 +39,7 @@ class SanityFailedError(Exception): f"{message_string}\n\n===============\n\n") -def check_sanity(): +def check_sanity(progress=False): messages = [] present_files = [] @@ -50,7 +51,12 @@ def check_sanity(): if lockfile in present_files: present_files.remove(lockfile) - for doc in Document.objects.all(): + if progress: + docs = tqdm(Document.objects.all()) + else: + docs = Document.objects.all() + + for doc in docs: # Check sanity of the thumbnail if not os.path.isfile(doc.thumbnail_path): messages.append(SanityError( diff --git a/src/documents/tests/test_management.py b/src/documents/tests/test_management.py index 1a550a4b4..56710c6e5 100644 --- a/src/documents/tests/test_management.py +++ b/src/documents/tests/test_management.py @@ -65,6 +65,7 @@ class TestArchiver(DirectoriesMixin, TestCase): self.assertEqual(doc1.archive_filename, "document.pdf") self.assertEqual(doc2.archive_filename, "document_01.pdf") + class TestDecryptDocuments(TestCase): @override_settings( @@ -154,3 +155,37 @@ class TestCreateClassifier(TestCase): call_command("document_create_classifier") m.assert_called_once() + + +class TestSanityChecker(DirectoriesMixin, TestCase): + + def test_no_errors(self): + with self.assertLogs() as capture: + call_command("document_sanity_checker") + + self.assertEqual(len(capture.output), 1) + self.assertIn("No issues found.", capture.output[0]) + + @mock.patch("documents.management.commands.document_sanity_checker.logger.warning") + @mock.patch("documents.management.commands.document_sanity_checker.logger.error") + def test_warnings(self, error, warning): + doc = Document.objects.create(title="test", filename="test.pdf", checksum="d41d8cd98f00b204e9800998ecf8427e") + Path(doc.source_path).touch() + Path(doc.thumbnail_path).touch() + + call_command("document_sanity_checker") + + error.assert_not_called() + warning.assert_called() + + @mock.patch("documents.management.commands.document_sanity_checker.logger.warning") + @mock.patch("documents.management.commands.document_sanity_checker.logger.error") + def test_errors(self, error, warning): + doc = Document.objects.create(title="test", content="test", filename="test.pdf", checksum="abc") + Path(doc.source_path).touch() + Path(doc.thumbnail_path).touch() + + call_command("document_sanity_checker") + + warning.assert_not_called() + error.assert_called() From 5f0e800f6ed41703b01f31f3b402cefb40a46305 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 13 Feb 2021 16:41:03 +0100 Subject: [PATCH 112/225] metadata tab not showing anything if files are missing #534 --- docs/changelog.rst | 2 ++ src/documents/tests/test_api.py | 24 ++++++++++++++++++++++-- src/documents/views.py | 10 ++++++++-- 3 files changed, 32 insertions(+), 4 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index f15c73605..4c40a71ab 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -11,6 +11,8 @@ paperless-ng 1.1.2 * Always show top left corner of thumbnails, even for extra wide documents. * Added a management command for executing the sanity checker directly. See :ref:`utilities-sanity-checker`. + +* Fixed an issue with the metadata tab not reporting anything in case of missing files. paperless-ng 1.1.1 ################## diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py index 84b3bc185..20eba6628 100644 --- a/src/documents/tests/test_api.py +++ b/src/documents/tests/test_api.py @@ -577,8 +577,11 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): def test_get_metadata(self): doc = Document.objects.create(title="test", filename="file.pdf", mime_type="image/png", archive_checksum="A", archive_filename="archive.pdf") - shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "documents", "thumbnails", "0000001.png"), doc.source_path) - shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"), doc.archive_path) + source_file = os.path.join(os.path.dirname(__file__), "samples", "documents", "thumbnails", "0000001.png") + archive_file = os.path.join(os.path.dirname(__file__), "samples", "simple.pdf") + + shutil.copy(source_file, doc.source_path) + shutil.copy(archive_file, doc.archive_path) response = self.client.get(f"/api/documents/{doc.pk}/metadata/") self.assertEqual(response.status_code, 200) @@ -591,6 +594,8 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): self.assertGreater(len(meta['archive_metadata']), 0) self.assertEqual(meta['media_filename'], "file.pdf") self.assertEqual(meta['archive_media_filename'], "archive.pdf") + self.assertEqual(meta['original_size'], os.stat(source_file).st_size) + self.assertEqual(meta['archive_size'], os.stat(archive_file).st_size) def test_get_metadata_invalid_doc(self): response = self.client.get(f"/api/documents/34576/metadata/") @@ -612,6 +617,21 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): self.assertIsNone(meta['archive_metadata']) self.assertIsNone(meta['archive_media_filename']) + def test_get_metadata_missing_files(self): + doc = Document.objects.create(title="test", filename="file.pdf", mime_type="application/pdf", archive_filename="file.pdf", archive_checksum="B", checksum="A") + + response = self.client.get(f"/api/documents/{doc.pk}/metadata/") + self.assertEqual(response.status_code, 200) + + meta = response.data + + self.assertTrue(meta['has_archive_version']) + self.assertIsNone(meta['original_metadata']) + self.assertIsNone(meta['original_size']) + self.assertIsNone(meta['archive_metadata']) + self.assertIsNone(meta['archive_size']) + + def test_get_empty_suggestions(self): doc = Document.objects.create(title="test", mime_type="application/pdf") diff --git a/src/documents/views.py b/src/documents/views.py index 9d1d87235..b2e5b4cd3 100755 --- a/src/documents/views.py +++ b/src/documents/views.py @@ -225,6 +225,12 @@ class DocumentViewSet(RetrieveModelMixin, else: return [] + def get_filesize(self, filename): + if os.path.isfile(filename): + return os.stat(filename).st_size + else: + return None + @action(methods=['get'], detail=True) def metadata(self, request, pk=None): try: @@ -234,7 +240,7 @@ class DocumentViewSet(RetrieveModelMixin, meta = { "original_checksum": doc.checksum, - "original_size": os.stat(doc.source_path).st_size, + "original_size": self.get_filesize(doc.source_path), "original_mime_type": doc.mime_type, "media_filename": doc.filename, "has_archive_version": doc.has_archive_version, @@ -245,7 +251,7 @@ class DocumentViewSet(RetrieveModelMixin, } if doc.has_archive_version: - meta['archive_size'] = os.stat(doc.archive_path).st_size, + meta['archive_size'] = self.get_filesize(doc.archive_path) meta['archive_metadata'] = self.get_metadata( doc.archive_path, "application/pdf") else: From 64d61ae2fa6fef6ab7f64402d3e95ffcc81d240c Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 13 Feb 2021 18:01:19 +0100 Subject: [PATCH 113/225] version bump --- src-ui/src/environments/environment.prod.ts | 2 +- src/paperless/version.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src-ui/src/environments/environment.prod.ts b/src-ui/src/environments/environment.prod.ts index 95cd94d29..e2c66690e 100644 --- a/src-ui/src/environments/environment.prod.ts +++ b/src-ui/src/environments/environment.prod.ts @@ -2,7 +2,7 @@ export const environment = { production: true, apiBaseUrl: "/api/", appTitle: "Paperless-ng", - version: "1.1.1", + version: "1.1.2", webSocketHost: window.location.host, webSocketProtocol: (window.location.protocol == "https:" ? "wss:" : "ws:") }; diff --git a/src/paperless/version.py b/src/paperless/version.py index ae805838c..728d237e2 100644 --- a/src/paperless/version.py +++ b/src/paperless/version.py @@ -1 +1 @@ -__version__ = (1, 1, 1) +__version__ = (1, 1, 2) From 0a1f264c71f463c19c05a3433a25e25477558a88 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 13 Feb 2021 18:09:00 +0100 Subject: [PATCH 114/225] Gotenberg troubleshooting --- docs/troubleshooting.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst index ae7fe14c7..d2a76e7b3 100644 --- a/docs/troubleshooting.rst +++ b/docs/troubleshooting.rst @@ -94,6 +94,30 @@ If you want to get rid of the warning or actually experience issues with automat the file ``classification_model.pickle`` in the data directory and let paperless recreate it. +504 Server Error: Gateway Timeout when adding Office documents +############################################################## + +You may experience these errors when using the optional TIKA integration: + +.. code:: + + requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office + +Gotenberg is a server that converts Office documents into PDF documents and has a default timeout of 10 seconds. +When conversion takes longer, Gotenberg raises this error. + +You can increase the timeout by configuring an environment variable for gotenberg (see also `here `__). +If using docker-compose, this is achieved by the following configuration change in the ``docker-compose.yml`` file: + +.. code:: yaml + + gotenberg: + image: thecodingmachine/gotenberg + restart: unless-stopped + environment: + DISABLE_GOOGLE_CHROME: 1 + DEFAULT_WAIT_TIMEOUT: 30 + Permission denied errors in the consumption directory ##################################################### From b48e67d71452ac7f04f004ee0af74735eb4b1f8e Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 13 Feb 2021 19:51:04 +0100 Subject: [PATCH 115/225] revert a faulty change that caused memory usage to explode #537 --- src/documents/classifier.py | 39 +++++++++++--------------- src/documents/tests/test_classifier.py | 5 ++-- src/documents/tests/test_tasks.py | 1 - src/paperless/settings.py | 10 ------- 4 files changed, 18 insertions(+), 37 deletions(-) diff --git a/src/documents/classifier.py b/src/documents/classifier.py index c880e50d6..b577997e3 100755 --- a/src/documents/classifier.py +++ b/src/documents/classifier.py @@ -5,7 +5,6 @@ import pickle import re from django.conf import settings -from django.core.cache import cache from documents.models import Document, MatchingModel @@ -31,29 +30,23 @@ def load_classifier(): ) return None - version = os.stat(settings.MODEL_FILE).st_mtime + classifier = DocumentClassifier() + try: + classifier.load() - classifier = cache.get("paperless-classifier", version=version) - - if not classifier: - classifier = DocumentClassifier() - try: - classifier.load() - cache.set("paperless-classifier", classifier, - version=version, timeout=86400) - except (EOFError, IncompatibleClassifierVersionError) as e: - # there's something wrong with the model file. - logger.exception( - f"Unrecoverable error while loading document " - f"classification model, deleting model file." - ) - os.unlink(settings.MODEL_FILE) - classifier = None - except OSError as e: - logger.error( - f"Error while loading document classification model: {str(e)}" - ) - classifier = None + except (EOFError, IncompatibleClassifierVersionError) as e: + # there's something wrong with the model file. + logger.exception( + f"Unrecoverable error while loading document " + f"classification model, deleting model file." + ) + os.unlink(settings.MODEL_FILE) + classifier = None + except OSError as e: + logger.error( + f"Error while loading document classification model: {str(e)}" + ) + classifier = None return classifier diff --git a/src/documents/tests/test_classifier.py b/src/documents/tests/test_classifier.py index 1efe564d1..fcc08f842 100644 --- a/src/documents/tests/test_classifier.py +++ b/src/documents/tests/test_classifier.py @@ -3,6 +3,7 @@ import tempfile from pathlib import Path from unittest import mock +import pytest from django.conf import settings from django.test import TestCase, override_settings @@ -233,7 +234,6 @@ class TestClassifier(DirectoriesMixin, TestCase): self.assertFalse(os.path.exists(settings.MODEL_FILE)) self.assertIsNone(load_classifier()) - @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}) @mock.patch("documents.classifier.DocumentClassifier.load") def test_load_classifier(self, load): Path(settings.MODEL_FILE).touch() @@ -242,6 +242,7 @@ class TestClassifier(DirectoriesMixin, TestCase): @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}) @override_settings(MODEL_FILE=os.path.join(os.path.dirname(__file__), "data", "model.pickle")) + @pytest.mark.skip(reason="Disabled caching due to high memory usage - need to investigate.") def test_load_classifier_cached(self): classifier = load_classifier() self.assertIsNotNone(classifier) @@ -250,7 +251,6 @@ class TestClassifier(DirectoriesMixin, TestCase): classifier2 = load_classifier() load.assert_not_called() - @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}) @mock.patch("documents.classifier.DocumentClassifier.load") def test_load_classifier_incompatible_version(self, load): Path(settings.MODEL_FILE).touch() @@ -260,7 +260,6 @@ class TestClassifier(DirectoriesMixin, TestCase): self.assertIsNone(load_classifier()) self.assertFalse(os.path.exists(settings.MODEL_FILE)) - @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}) @mock.patch("documents.classifier.DocumentClassifier.load") def test_load_classifier_os_error(self, load): Path(settings.MODEL_FILE).touch() diff --git a/src/documents/tests/test_tasks.py b/src/documents/tests/test_tasks.py index ed280441f..8c5a1b5df 100644 --- a/src/documents/tests/test_tasks.py +++ b/src/documents/tests/test_tasks.py @@ -52,7 +52,6 @@ class TestTasks(DirectoriesMixin, TestCase): load_classifier.assert_called_once() self.assertFalse(os.path.isfile(settings.MODEL_FILE)) - @override_settings(CACHES={'default': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'}}) def test_train_classifier(self): c = Correspondent.objects.create(matching_algorithm=Tag.MATCH_AUTO, name="test") doc = Document.objects.create(correspondent=c, content="test", title="test") diff --git a/src/paperless/settings.py b/src/paperless/settings.py index 60e7b45bd..150709bb1 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -169,16 +169,6 @@ CHANNEL_LAYERS = { }, } -CACHES = { - "default": { - "BACKEND": "django_redis.cache.RedisCache", - "LOCATION": os.getenv("PAPERLESS_REDIS", "redis://localhost:6379"), - "OPTIONS": { - "CLIENT_CLASS": "django_redis.client.DefaultClient", - } - } -} - ############################################################################### # Security # ############################################################################### From 12836d4c68b7738ccb4080e7012e363b2efcd385 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sat, 13 Feb 2021 20:25:52 +0100 Subject: [PATCH 116/225] revert django-q configuration --- docs/changelog.rst | 2 ++ src/paperless/settings.py | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 4c40a71ab..52b53dca8 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -14,6 +14,8 @@ paperless-ng 1.1.2 * Fixed an issue with the metadata tab not reporting anything in case of missing files. +* Reverted a change from 1.1.0 that caused huge memory usage due to redis caching. + paperless-ng 1.1.1 ################## diff --git a/src/paperless/settings.py b/src/paperless/settings.py index 150709bb1..718738f71 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -398,7 +398,7 @@ Q_CLUSTER = { 'name': 'paperless', 'catch_up': False, 'workers': TASK_WORKERS, - 'django_redis': 'default' + 'redis': os.getenv("PAPERLESS_REDIS", "redis://localhost:6379") } From 62caeed283a03035f175b1cd6e25c9048883c707 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Sun, 14 Feb 2021 07:05:05 +0000 Subject: [PATCH 117/225] Apply translations in cs translation completed for the source file '/src/locale/en-us/LC_MESSAGES/django.po' on the 'cs' language. --- src/locale/cs/LC_MESSAGES/django.po | 650 ++++++++++++++++++++++++++++ 1 file changed, 650 insertions(+) create mode 100644 src/locale/cs/LC_MESSAGES/django.po diff --git a/src/locale/cs/LC_MESSAGES/django.po b/src/locale/cs/LC_MESSAGES/django.po new file mode 100644 index 000000000..959246aab --- /dev/null +++ b/src/locale/cs/LC_MESSAGES/django.po @@ -0,0 +1,650 @@ +# SOME DESCRIPTIVE TITLE. +# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER +# This file is distributed under the same license as the PACKAGE package. +# FIRST AUTHOR , YEAR. +# +# Translators: +# Štěpán Šebestian , 2021 +# +#, fuzzy +msgid "" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"Report-Msgid-Bugs-To: \n" +"POT-Creation-Date: 2021-01-28 22:02+0100\n" +"PO-Revision-Date: 2020-12-30 19:27+0000\n" +"Last-Translator: Štěpán Šebestian , 2021\n" +"Language-Team: Czech (https://www.transifex.com/paperless/teams/115905/cs/)\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Language: cs\n" +"Plural-Forms: nplurals=4; plural=(n == 1 && n % 1 == 0) ? 0 : (n >= 2 && n <= 4 && n % 1 == 0) ? 1: (n % 1 != 0 ) ? 2 : 3;\n" + +#: documents/apps.py:10 +msgid "Documents" +msgstr "Dokumenty" + +#: documents/models.py:33 +msgid "Any word" +msgstr "Jakékoliv slovo" + +#: documents/models.py:34 +msgid "All words" +msgstr "Všechna slova" + +#: documents/models.py:35 +msgid "Exact match" +msgstr "Přesná shoda" + +#: documents/models.py:36 +msgid "Regular expression" +msgstr "Regulární výraz" + +#: documents/models.py:37 +msgid "Fuzzy word" +msgstr "Fuzzy slovo" + +#: documents/models.py:38 +msgid "Automatic" +msgstr "Automatický" + +#: documents/models.py:42 documents/models.py:352 paperless_mail/models.py:25 +#: paperless_mail/models.py:109 +msgid "name" +msgstr "název" + +#: documents/models.py:46 +msgid "match" +msgstr "shoda" + +#: documents/models.py:50 +msgid "matching algorithm" +msgstr "algoritmus pro shodu" + +#: documents/models.py:56 +msgid "is insensitive" +msgstr "je ignorováno" + +#: documents/models.py:75 documents/models.py:135 +msgid "correspondent" +msgstr "korespondent" + +#: documents/models.py:76 +msgid "correspondents" +msgstr "korespondenti" + +#: documents/models.py:98 +msgid "color" +msgstr "barva" + +#: documents/models.py:102 +msgid "is inbox tag" +msgstr "tag přichozí" + +#: documents/models.py:104 +msgid "" +"Marks this tag as an inbox tag: All newly consumed documents will be tagged " +"with inbox tags." +msgstr "" +"Označí tento tag jako tag pro příchozí: Všechny nově zkonzumované dokumenty " +"budou označeny tagem pro přichozí" + +#: documents/models.py:109 +msgid "tag" +msgstr "tag" + +#: documents/models.py:110 documents/models.py:166 +msgid "tags" +msgstr "tagy" + +#: documents/models.py:116 documents/models.py:148 +msgid "document type" +msgstr "typ dokumentu" + +#: documents/models.py:117 +msgid "document types" +msgstr "typy dokumentu" + +#: documents/models.py:125 +msgid "Unencrypted" +msgstr "Nešifrované" + +#: documents/models.py:126 +msgid "Encrypted with GNU Privacy Guard" +msgstr "Šifrované pomocí GNU Privacy Guard" + +#: documents/models.py:139 +msgid "title" +msgstr "titulek" + +#: documents/models.py:152 +msgid "content" +msgstr "obsah" + +#: documents/models.py:154 +msgid "" +"The raw, text-only data of the document. This field is primarily used for " +"searching." +msgstr "" +"Nezpracovaná, pouze textová data dokumentu. Toto pole je používáno především" +" pro vyhledávání." + +#: documents/models.py:159 +msgid "mime type" +msgstr "mime typ" + +#: documents/models.py:170 +msgid "checksum" +msgstr "kontrolní součet" + +#: documents/models.py:174 +msgid "The checksum of the original document." +msgstr "Kontrolní součet původního dokumentu" + +#: documents/models.py:178 +msgid "archive checksum" +msgstr "kontrolní součet archivu" + +#: documents/models.py:183 +msgid "The checksum of the archived document." +msgstr "Kontrolní součet archivovaného dokumentu." + +#: documents/models.py:187 documents/models.py:330 +msgid "created" +msgstr "vytvořeno" + +#: documents/models.py:191 +msgid "modified" +msgstr "upraveno" + +#: documents/models.py:195 +msgid "storage type" +msgstr "typ úložiště" + +#: documents/models.py:203 +msgid "added" +msgstr "přidáno" + +#: documents/models.py:207 +msgid "filename" +msgstr "název souboru" + +#: documents/models.py:212 +msgid "Current filename in storage" +msgstr "Aktuální název souboru v úložišti" + +#: documents/models.py:216 +msgid "archive serial number" +msgstr "sériové číslo archivu" + +#: documents/models.py:221 +msgid "The position of this document in your physical document archive." +msgstr "Pozice dokumentu ve vašem archivu fyzických dokumentů" + +#: documents/models.py:227 +msgid "document" +msgstr "dokument" + +#: documents/models.py:228 +msgid "documents" +msgstr "dokumenty" + +#: documents/models.py:313 +msgid "debug" +msgstr "debug" + +#: documents/models.py:314 +msgid "information" +msgstr "informace" + +#: documents/models.py:315 +msgid "warning" +msgstr "varování" + +#: documents/models.py:316 +msgid "error" +msgstr "chyba" + +#: documents/models.py:317 +msgid "critical" +msgstr "kritická" + +#: documents/models.py:321 +msgid "group" +msgstr "skupina" + +#: documents/models.py:324 +msgid "message" +msgstr "zpráva" + +#: documents/models.py:327 +msgid "level" +msgstr "úroveň" + +#: documents/models.py:334 +msgid "log" +msgstr "záznam" + +#: documents/models.py:335 +msgid "logs" +msgstr "záznamy" + +#: documents/models.py:346 documents/models.py:396 +msgid "saved view" +msgstr "uložený pohled" + +#: documents/models.py:347 +msgid "saved views" +msgstr "uložené pohledy" + +#: documents/models.py:350 +msgid "user" +msgstr "uživatel" + +#: documents/models.py:356 +msgid "show on dashboard" +msgstr "zobrazit v dashboardu" + +#: documents/models.py:359 +msgid "show in sidebar" +msgstr "zobrazit v postranním menu" + +#: documents/models.py:363 +msgid "sort field" +msgstr "pole na řazení" + +#: documents/models.py:366 +msgid "sort reverse" +msgstr "třídit opačně" + +#: documents/models.py:372 +msgid "title contains" +msgstr "titulek obsahuje" + +#: documents/models.py:373 +msgid "content contains" +msgstr "obsah obsahuje" + +#: documents/models.py:374 +msgid "ASN is" +msgstr "ASN je" + +#: documents/models.py:375 +msgid "correspondent is" +msgstr "korespondent je" + +#: documents/models.py:376 +msgid "document type is" +msgstr "typ dokumentu je" + +#: documents/models.py:377 +msgid "is in inbox" +msgstr "je v příchozích" + +#: documents/models.py:378 +msgid "has tag" +msgstr "má tag" + +#: documents/models.py:379 +msgid "has any tag" +msgstr "má jakýkoliv tag" + +#: documents/models.py:380 +msgid "created before" +msgstr "vytvořeno před" + +#: documents/models.py:381 +msgid "created after" +msgstr "vytvořeno po" + +#: documents/models.py:382 +msgid "created year is" +msgstr "rok vytvoření je" + +#: documents/models.py:383 +msgid "created month is" +msgstr "měsíc vytvoření je" + +#: documents/models.py:384 +msgid "created day is" +msgstr "den vytvoření je" + +#: documents/models.py:385 +msgid "added before" +msgstr "přidáno před" + +#: documents/models.py:386 +msgid "added after" +msgstr "přidáno po" + +#: documents/models.py:387 +msgid "modified before" +msgstr "upraveno před" + +#: documents/models.py:388 +msgid "modified after" +msgstr "upraveno po" + +#: documents/models.py:389 +msgid "does not have tag" +msgstr "nemá tag" + +#: documents/models.py:400 +msgid "rule type" +msgstr "typ pravidla" + +#: documents/models.py:404 +msgid "value" +msgstr "hodnota" + +#: documents/models.py:410 +msgid "filter rule" +msgstr "filtrovací pravidlo" + +#: documents/models.py:411 +msgid "filter rules" +msgstr "filtrovací pravidla" + +#: documents/serialisers.py:383 +#, python-format +msgid "File type %(type)s not supported" +msgstr "Typ souboru %(type)s není podporován" + +#: documents/templates/index.html:20 +msgid "Paperless-ng is loading..." +msgstr "Paperless-ng se načítá..." + +#: documents/templates/registration/logged_out.html:13 +msgid "Paperless-ng signed out" +msgstr "Odhlášeno od Paperless-ng" + +#: documents/templates/registration/logged_out.html:41 +msgid "You have been successfully logged out. Bye!" +msgstr "Byli jste úspěšně odhlášeni. Nashledanou!" + +#: documents/templates/registration/logged_out.html:42 +msgid "Sign in again" +msgstr "Přihlašte se znovu" + +#: documents/templates/registration/login.html:13 +msgid "Paperless-ng sign in" +msgstr "Paperless-ng přihlášení" + +#: documents/templates/registration/login.html:42 +msgid "Please sign in." +msgstr "Prosím přihlaste se." + +#: documents/templates/registration/login.html:45 +msgid "Your username and password didn't match. Please try again." +msgstr "Vaše uživatelské jméno a heslo se neshodují. Prosím, zkuste to znovu." + +#: documents/templates/registration/login.html:48 +msgid "Username" +msgstr "Uživatelské jméno" + +#: documents/templates/registration/login.html:49 +msgid "Password" +msgstr "Heslo" + +#: documents/templates/registration/login.html:54 +msgid "Sign in" +msgstr "Přihlásit se" + +#: paperless/settings.py:286 +msgid "English" +msgstr "Angličtina" + +#: paperless/settings.py:287 +msgid "German" +msgstr "Němčina" + +#: paperless/settings.py:288 +msgid "Dutch" +msgstr "Holandština" + +#: paperless/settings.py:289 +msgid "French" +msgstr "Francouzština" + +#: paperless/urls.py:114 +msgid "Paperless-ng administration" +msgstr "Správa Paperless-ng" + +#: paperless_mail/admin.py:25 +msgid "Filter" +msgstr "Filtr" + +#: paperless_mail/admin.py:27 +msgid "" +"Paperless will only process mails that match ALL of the filters given below." +msgstr "" +"Paperless zpracuje pouze emaily které odpovídají VŠEM níže zadaným filtrům." + +#: paperless_mail/admin.py:37 +msgid "Actions" +msgstr "Akce" + +#: paperless_mail/admin.py:39 +msgid "" +"The action applied to the mail. This action is only performed when documents" +" were consumed from the mail. Mails without attachments will remain entirely" +" untouched." +msgstr "" +"Akce provedena na emailu. Tato akce je provedena jen pokud byly dokumenty " +"zkonzumovány z emailu. Emaily bez příloh zůstanou nedotčeny." + +#: paperless_mail/admin.py:46 +msgid "Metadata" +msgstr "Metadata" + +#: paperless_mail/admin.py:48 +msgid "" +"Assign metadata to documents consumed from this rule automatically. If you " +"do not assign tags, types or correspondents here, paperless will still " +"process all matching rules that you have defined." +msgstr "" +"Automaticky přiřadit metadata dokumentům zkonzumovaných z tohoto pravidla. " +"Pokud zde nepřiřadíte tagy, typy nebo korespondenty, paperless stále " +"zpracuje všechna shodující-se pravidla které jste definovali." + +#: paperless_mail/apps.py:9 +msgid "Paperless mail" +msgstr "Paperless pošta" + +#: paperless_mail/models.py:11 +msgid "mail account" +msgstr "emailový účet" + +#: paperless_mail/models.py:12 +msgid "mail accounts" +msgstr "emailové účty" + +#: paperless_mail/models.py:19 +msgid "No encryption" +msgstr "Žádné šifrování" + +#: paperless_mail/models.py:20 +msgid "Use SSL" +msgstr "Používat SSL" + +#: paperless_mail/models.py:21 +msgid "Use STARTTLS" +msgstr "Používat STARTTLS" + +#: paperless_mail/models.py:29 +msgid "IMAP server" +msgstr "IMAP server" + +#: paperless_mail/models.py:33 +msgid "IMAP port" +msgstr "IMAP port" + +#: paperless_mail/models.py:36 +msgid "" +"This is usually 143 for unencrypted and STARTTLS connections, and 993 for " +"SSL connections." +msgstr "" +"Toto je většinou 143 pro nešifrovaná připojení/připojení používající " +"STARTTLS a 993 pro SSL připojení." + +#: paperless_mail/models.py:40 +msgid "IMAP security" +msgstr "IMAP bezpečnost" + +#: paperless_mail/models.py:46 +msgid "username" +msgstr "uživatelské jméno" + +#: paperless_mail/models.py:50 +msgid "password" +msgstr "heslo" + +#: paperless_mail/models.py:60 +msgid "mail rule" +msgstr "mailové pravidlo" + +#: paperless_mail/models.py:61 +msgid "mail rules" +msgstr "mailová pravidla" + +#: paperless_mail/models.py:67 +msgid "Only process attachments." +msgstr "Zpracovávat jen přílohy" + +#: paperless_mail/models.py:68 +msgid "Process all files, including 'inline' attachments." +msgstr "Zpracovat všechny soubory, včetně vložených příloh" + +#: paperless_mail/models.py:78 +msgid "Mark as read, don't process read mails" +msgstr "Označit jako přečtené, nezpracovávat přečtené emaily" + +#: paperless_mail/models.py:79 +msgid "Flag the mail, don't process flagged mails" +msgstr "Označit email, nezpracovávat označené emaily" + +#: paperless_mail/models.py:80 +msgid "Move to specified folder" +msgstr "Přesunout do specifikované složky" + +#: paperless_mail/models.py:81 +msgid "Delete" +msgstr "Odstranit" + +#: paperless_mail/models.py:88 +msgid "Use subject as title" +msgstr "Použít předmět jako titulek" + +#: paperless_mail/models.py:89 +msgid "Use attachment filename as title" +msgstr "Použít název souboru u přílohy jako titulek" + +#: paperless_mail/models.py:99 +msgid "Do not assign a correspondent" +msgstr "Nepřiřazovat korespondenta" + +#: paperless_mail/models.py:101 +msgid "Use mail address" +msgstr "Použít emailovou adresu" + +#: paperless_mail/models.py:103 +msgid "Use name (or mail address if not available)" +msgstr "Použít jméno (nebo emailovou adresu pokud jméno není dostupné)" + +#: paperless_mail/models.py:105 +msgid "Use correspondent selected below" +msgstr "Použít korespondenta vybraného níže" + +#: paperless_mail/models.py:113 +msgid "order" +msgstr "pořadí" + +#: paperless_mail/models.py:120 +msgid "account" +msgstr "účet" + +#: paperless_mail/models.py:124 +msgid "folder" +msgstr "složka" + +#: paperless_mail/models.py:128 +msgid "filter from" +msgstr "filtrovat z" + +#: paperless_mail/models.py:131 +msgid "filter subject" +msgstr "název filtru" + +#: paperless_mail/models.py:134 +msgid "filter body" +msgstr "tělo filtru" + +#: paperless_mail/models.py:138 +msgid "filter attachment filename" +msgstr "název souboru u přílohy filtru" + +#: paperless_mail/models.py:140 +msgid "" +"Only consume documents which entirely match this filename if specified. " +"Wildcards such as *.pdf or *invoice* are allowed. Case insensitive." +msgstr "" +"Konzumovat jen dokumenty které přesně odpovídají tomuto názvu souboru pokud " +"specifikováno. Zástupné znaky jako *.pdf nebo *invoice* jsou povoleny. " +"Nezáleží na velikosti písmen." + +#: paperless_mail/models.py:146 +msgid "maximum age" +msgstr "maximální stáří" + +#: paperless_mail/models.py:148 +msgid "Specified in days." +msgstr "Specifikováno ve dnech." + +#: paperless_mail/models.py:151 +msgid "attachment type" +msgstr "typ přílohy" + +#: paperless_mail/models.py:154 +msgid "" +"Inline attachments include embedded images, so it's best to combine this " +"option with a filename filter." +msgstr "" +"Vložené přílohy zahrnují vložené obrázky, takže je nejlepší tuto možnost " +"kombinovat s filtrem na název souboru" + +#: paperless_mail/models.py:159 +msgid "action" +msgstr "akce" + +#: paperless_mail/models.py:165 +msgid "action parameter" +msgstr "parametr akce" + +#: paperless_mail/models.py:167 +msgid "" +"Additional parameter for the action selected above, i.e., the target folder " +"of the move to folder action." +msgstr "" +"Další parametr pro výše vybranou akci, napříkad cílová složka akce přesunutí" +" do složky." + +#: paperless_mail/models.py:173 +msgid "assign title from" +msgstr "nastavit titulek z" + +#: paperless_mail/models.py:183 +msgid "assign this tag" +msgstr "přiřadit tento tag" + +#: paperless_mail/models.py:191 +msgid "assign this document type" +msgstr "přiřadit tento typ dokumentu" + +#: paperless_mail/models.py:195 +msgid "assign correspondent from" +msgstr "přiřadit korespondenta z" + +#: paperless_mail/models.py:205 +msgid "assign this correspondent" +msgstr "přiřadit tohoto korespondenta" From 43c729568b434c7aae22c76f4cc335d86275ba19 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 14 Feb 2021 12:29:55 +0100 Subject: [PATCH 118/225] release worker memory after tasks are done. --- src/paperless/settings.py | 1 + 1 file changed, 1 insertion(+) diff --git a/src/paperless/settings.py b/src/paperless/settings.py index 718738f71..59133dd45 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -397,6 +397,7 @@ TASK_WORKERS = int(os.getenv("PAPERLESS_TASK_WORKERS", default_task_workers())) Q_CLUSTER = { 'name': 'paperless', 'catch_up': False, + 'recycle': 1, 'workers': TASK_WORKERS, 'redis': os.getenv("PAPERLESS_REDIS", "redis://localhost:6379") } From 3d0a52c25f2811f07b8987f43ee5b7cfb6c423a6 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 14 Feb 2021 12:50:30 +0100 Subject: [PATCH 119/225] only load channels app if DEBUG is enabled; its only purpose is to monkey-patch the runserver command. --- src/paperless/settings.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/paperless/settings.py b/src/paperless/settings.py index 59133dd45..8d14262d1 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -102,10 +102,11 @@ INSTALLED_APPS = [ "django_q", - "channels", - ] + env_apps +if DEBUG: + INSTALLED_APPS.append("channels") + REST_FRAMEWORK = { 'DEFAULT_AUTHENTICATION_CLASSES': [ 'rest_framework.authentication.BasicAuthentication', From 6e48da41e59ce1346c9ea253de28f17c1bba62eb Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 14 Feb 2021 14:05:42 +0100 Subject: [PATCH 120/225] changelog --- docs/changelog.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 52b53dca8..fcd14c59d 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -16,6 +16,8 @@ paperless-ng 1.1.2 * Reverted a change from 1.1.0 that caused huge memory usage due to redis caching. +* Some memory usage optimizations. + paperless-ng 1.1.1 ################## From df6c59bc4f7ca10f23632dda0f1785e6eadf7791 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 14 Feb 2021 15:38:47 +0100 Subject: [PATCH 121/225] update dependencies --- Pipfile | 3 +- Pipfile.lock | 109 ++++++++++++++++++++++------------------------- requirements.txt | 7 ++- 3 files changed, 54 insertions(+), 65 deletions(-) diff --git a/Pipfile b/Pipfile index aae5f1506..8bacba7a4 100644 --- a/Pipfile +++ b/Pipfile @@ -39,7 +39,7 @@ scikit-learn="==0.24.0" # Prevent scipy updates because 1.6 is incompatible with python 3.6 scipy="~=1.5.4" whitenoise = "~=5.2.0" -watchdog = "*" +watchdog = "~=1.0.0" whoosh="~=2.7.4" inotifyrecursive = "~=0.3.4" ocrmypdf = "~=11.6" @@ -51,7 +51,6 @@ channels = "~=3.0" channels-redis = "*" uvicorn = {extras = ["standard"], version = "*"} concurrent-log-handler = "*" -django-redis = "*" # uvloop 0.15+ incompatible with python 3.6 uvloop = "~=0.14.0" # TODO: keep an eye on piwheel builds and update this once available (https://www.piwheels.org/project/cryptography/) diff --git a/Pipfile.lock b/Pipfile.lock index 9d1a75b88..fee6359da 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "b3bed0a6b8981e8fffc1b6aa3bc35a0b1472f28e6f745c62469eb8045740e57b" + "sha256": "bd8b69979d91f4d8c52cac127c891d750c52959807220a98dcf74fed126bfa26" }, "pipfile-spec": 6, "requires": {}, @@ -60,11 +60,11 @@ }, "autobahn": { "hashes": [ - "sha256:93df8fc9d1821c9dabff9fed52181a9ad6eea5e9989d53102c391607d7c1666e", - "sha256:cceed2121b7a93024daa93c91fae33007f8346f0e522796421f36a6183abea99" + "sha256:41a3a3f89cde48643baf4e105d9491c566295f9abee951379e59121784044b8b", + "sha256:7e6b1bf95196b733978bab2d54a7ab8899c16ce11be369dc58422c07b7eea726" ], "markers": "python_version >= '3.6'", - "version": "==21.1.1" + "version": "==21.2.1" }, "automat": { "hashes": [ @@ -90,47 +90,47 @@ }, "cffi": { "hashes": [ - "sha256:00a1ba5e2e95684448de9b89888ccd02c98d512064b4cb987d48f4b40aa0421e", - "sha256:00e28066507bfc3fe865a31f325c8391a1ac2916219340f87dfad602c3e48e5d", - "sha256:045d792900a75e8b1e1b0ab6787dd733a8190ffcf80e8c8ceb2fb10a29ff238a", - "sha256:0638c3ae1a0edfb77c6765d487fee624d2b1ee1bdfeffc1f0b58c64d149e7eec", - "sha256:105abaf8a6075dc96c1fe5ae7aae073f4696f2905fde6aeada4c9d2926752362", - "sha256:155136b51fd733fa94e1c2ea5211dcd4c8879869008fc811648f16541bf99668", - "sha256:1a465cbe98a7fd391d47dce4b8f7e5b921e6cd805ef421d04f5f66ba8f06086c", - "sha256:1d2c4994f515e5b485fd6d3a73d05526aa0fcf248eb135996b088d25dfa1865b", - "sha256:23f318bf74b170c6e9adb390e8bd282457f6de46c19d03b52f3fd042b5e19654", - "sha256:2c24d61263f511551f740d1a065eb0212db1dbbbbd241db758f5244281590c06", - "sha256:51a8b381b16ddd370178a65360ebe15fbc1c71cf6f584613a7ea08bfad946698", - "sha256:594234691ac0e9b770aee9fcdb8fa02c22e43e5c619456efd0d6c2bf276f3eb2", - "sha256:5cf4be6c304ad0b6602f5c4e90e2f59b47653ac1ed9c662ed379fe48a8f26b0c", - "sha256:64081b3f8f6f3c3de6191ec89d7dc6c86a8a43911f7ecb422c60e90c70be41c7", - "sha256:6bc25fc545a6b3d57b5f8618e59fc13d3a3a68431e8ca5fd4c13241cd70d0009", - "sha256:798caa2a2384b1cbe8a2a139d80734c9db54f9cc155c99d7cc92441a23871c03", - "sha256:7c6b1dece89874d9541fc974917b631406233ea0440d0bdfbb8e03bf39a49b3b", - "sha256:7ef7d4ced6b325e92eb4d3502946c78c5367bc416398d387b39591532536734e", - "sha256:840793c68105fe031f34d6a086eaea153a0cd5c491cde82a74b420edd0a2b909", - "sha256:8d6603078baf4e11edc4168a514c5ce5b3ba6e3e9c374298cb88437957960a53", - "sha256:9cc46bc107224ff5b6d04369e7c595acb700c3613ad7bcf2e2012f62ece80c35", - "sha256:9f7a31251289b2ab6d4012f6e83e58bc3b96bd151f5b5262467f4bb6b34a7c26", - "sha256:9ffb888f19d54a4d4dfd4b3f29bc2c16aa4972f1c2ab9c4ab09b8ab8685b9c2b", - "sha256:a5ed8c05548b54b998b9498753fb9cadbfd92ee88e884641377d8a8b291bcc01", - "sha256:a7711edca4dcef1a75257b50a2fbfe92a65187c47dab5a0f1b9b332c5919a3fb", - "sha256:af5c59122a011049aad5dd87424b8e65a80e4a6477419c0c1015f73fb5ea0293", - "sha256:b18e0a9ef57d2b41f5c68beefa32317d286c3d6ac0484efd10d6e07491bb95dd", - "sha256:b4e248d1087abf9f4c10f3c398896c87ce82a9856494a7155823eb45a892395d", - "sha256:ba4e9e0ae13fc41c6b23299545e5ef73055213e466bd107953e4a013a5ddd7e3", - "sha256:be8661bcee1bc2fc4b033a6ab65bd1f87ce5008492601695d0b9a4e820c3bde5", - "sha256:c6332685306b6417a91b1ff9fae889b3ba65c2292d64bd9245c093b1b284809d", - "sha256:d5ff0621c88ce83a28a10d2ce719b2ee85635e85c515f12bac99a95306da4b2e", - "sha256:d9efd8b7a3ef378dd61a1e77367f1924375befc2eba06168b6ebfa903a5e59ca", - "sha256:df5169c4396adc04f9b0a05f13c074df878b6052430e03f50e68adf3a57aa28d", - "sha256:ebb253464a5d0482b191274f1c8bf00e33f7e0b9c66405fbffc61ed2c839c775", - "sha256:ec80dc47f54e6e9a78181ce05feb71a0353854cc26999db963695f950b5fb375", - "sha256:f032b34669220030f905152045dfa27741ce1a6db3324a5bc0b96b6c7420c87b", - "sha256:f60567825f791c6f8a592f3c6e3bd93dd2934e3f9dac189308426bd76b00ef3b", - "sha256:f803eaa94c2fcda012c047e62bc7a51b0bdabda1cad7a92a522694ea2d76e49f" + "sha256:005a36f41773e148deac64b08f233873a4d0c18b053d37da83f6af4d9087b813", + "sha256:0857f0ae312d855239a55c81ef453ee8fd24136eaba8e87a2eceba644c0d4c06", + "sha256:1071534bbbf8cbb31b498d5d9db0f274f2f7a865adca4ae429e147ba40f73dea", + "sha256:158d0d15119b4b7ff6b926536763dc0714313aa59e320ddf787502c70c4d4bee", + "sha256:1f436816fc868b098b0d63b8920de7d208c90a67212546d02f84fe78a9c26396", + "sha256:2894f2df484ff56d717bead0a5c2abb6b9d2bf26d6960c4604d5c48bbc30ee73", + "sha256:29314480e958fd8aab22e4a58b355b629c59bf5f2ac2492b61e3dc06d8c7a315", + "sha256:34eff4b97f3d982fb93e2831e6750127d1355a923ebaeeb565407b3d2f8d41a1", + "sha256:35f27e6eb43380fa080dccf676dece30bef72e4a67617ffda586641cd4508d49", + "sha256:3d3dd4c9e559eb172ecf00a2a7517e97d1e96de2a5e610bd9b68cea3925b4892", + "sha256:43e0b9d9e2c9e5d152946b9c5fe062c151614b262fda2e7b201204de0b99e482", + "sha256:48e1c69bbacfc3d932221851b39d49e81567a4d4aac3b21258d9c24578280058", + "sha256:51182f8927c5af975fece87b1b369f722c570fe169f9880764b1ee3bca8347b5", + "sha256:5560dbf8deedbffb638d8a2da31da91094db361cc07f8a501a339b2daae2cbcc", + "sha256:58e3f59d583d413809d60779492342801d6e82fefb89c86a38e040c16883be53", + "sha256:5de7970188bb46b7bf9858eb6890aad302577a5f6f75091fd7cdd3ef13ef3045", + "sha256:65fa59693c62cf06e45ddbb822165394a288edce9e276647f0046e1ec26920f3", + "sha256:69e395c24fc60aad6bb4fa7e583698ea6cc684648e1ffb7fe85e3c1ca131a7d5", + "sha256:6c97d7350133666fbb5cf4abdc1178c812cb205dc6f41d174a7b0f18fb93337e", + "sha256:6e4714cc64f474e4d6e37cfff31a814b509a35cb17de4fb1999907575684479c", + "sha256:72d8d3ef52c208ee1c7b2e341f7d71c6fd3157138abf1a95166e6165dd5d4369", + "sha256:8ae6299f6c68de06f136f1f9e69458eae58f1dacf10af5c17353eae03aa0d827", + "sha256:8b198cec6c72df5289c05b05b8b0969819783f9418e0409865dac47288d2a053", + "sha256:9338beed13d880320450d95c9e07ccf839faa3ea7b75d788f4ed46d845044a71", + "sha256:99cd03ae7988a93dd00bcd9d0b75e1f6c426063d6f03d2f90b89e29b25b82dfa", + "sha256:9cf8022fb8d07a97c178b02327b284521c7708d7c71a9c9c355c178ac4bbd3d4", + "sha256:9de2e279153a443c656f2defd67769e6d1e4163952b3c622dcea5b08a6405322", + "sha256:9e93e79c2551ff263400e1e4be085a1210e12073a31c2011dbbda14bda0c6132", + "sha256:9ff227395193126d82e60319a673a037d5de84633f11279e336f9c0f189ecc62", + "sha256:a465da611f6fa124963b91bf432d960a555563efe4ed1cc403ba5077b15370aa", + "sha256:ad17025d226ee5beec591b52800c11680fca3df50b8b29fe51d882576e039ee0", + "sha256:afb29c1ba2e5a3736f1c301d9d0abe3ec8b86957d04ddfa9d7a6a42b9367e396", + "sha256:b85eb46a81787c50650f2392b9b4ef23e1f126313b9e0e9013b35c15e4288e2e", + "sha256:bb89f306e5da99f4d922728ddcd6f7fcebb3241fc40edebcb7284d7514741991", + "sha256:cbde590d4faaa07c72bf979734738f328d239913ba3e043b1e98fe9a39f8b2b6", + "sha256:cd2868886d547469123fadc46eac7ea5253ea7fcb139f12e1dfc2bbd406427d1", + "sha256:d42b11d692e11b6634f7613ad8df5d6d5f8875f5d48939520d351007b3c13406", + "sha256:f2d45f97ab6bb54753eab54fffe75aaf3de4ff2341c9daee1987ee1837636f1d", + "sha256:fd78e5fee591709f32ef6edb9a015b4aa1a5022598e36227500c8f4e02328d9c" ], - "version": "==1.14.4" + "version": "==1.14.5" }, "channels": { "hashes": [ @@ -273,15 +273,6 @@ "index": "pypi", "version": "==1.3.4" }, - "django-redis": { - "hashes": [ - "sha256:1133b26b75baa3664164c3f44b9d5d133d1b8de45d94d79f38d1adc5b1d502e5", - "sha256:306589c7021e6468b2656edc89f62b8ba67e8d5a1c8877e2688042263daa7a63", - "sha256:f2b25b62cc95b63b7059aaf8e81710e7eea94678e545d31c46e47a6f4af99e56" - ], - "index": "pypi", - "version": "==4.12.1" - }, "djangorestframework": { "hashes": [ "sha256:0209bafcb7b5010fdfec784034f059d512256424de2a0f084cb82b096d6dd6a7", @@ -1113,11 +1104,11 @@ }, "tqdm": { "hashes": [ - "sha256:2874fa525c051177583ec59c0fb4583e91f28ccd3f217ffad2acdb32d2c789ac", - "sha256:ab9b659241d82b8b51b2269ee243ec95286046bf06015c4e15a947cc15914211" + "sha256:11d544652edbdfc9cc41aa4c8a5c166513e279f3f2d9f1a9e1c89935b51de6ff", + "sha256:a89be573bfddb81bb0b395a416d5e55e3ecc73ce95a368a4f6360bedea33195e" ], "index": "pypi", - "version": "==4.56.1" + "version": "==4.56.2" }, "twisted": { "extras": [ @@ -1649,11 +1640,11 @@ }, "pygments": { "hashes": [ - "sha256:bc9591213a8f0e0ca1a5e68a479b4887fdc3e75d0774e5c71c31920c427de435", - "sha256:df49d09b498e83c1a73128295860250b0b7edd4c723a32e9bc0d295c7c2ec337" + "sha256:37a13ba168a02ac54cc5891a42b1caec333e59b66addb7fa633ea8a6d73445c0", + "sha256:b21b072d0ccdf29297a82a2363359d99623597b8a265b8081760e4d0f7153c88" ], "markers": "python_version >= '3.5'", - "version": "==2.7.4" + "version": "==2.8.0" }, "pyparsing": { "hashes": [ diff --git a/requirements.txt b/requirements.txt index cf63b4660..0a0ae75c0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -12,11 +12,11 @@ arrow==0.17.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, asgiref==3.3.1; python_version >= '3.5' async-timeout==3.0.1; python_full_version >= '3.5.3' attrs==20.3.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3' -autobahn==21.1.1; python_version >= '3.6' +autobahn==21.2.1; python_version >= '3.6' automat==20.2.0 blessed==1.17.12 certifi==2020.12.5 -cffi==1.14.4 +cffi==1.14.5 channels-redis==3.2.0 channels==3.0.3 chardet==4.0.0; python_version >= '3.1' @@ -32,7 +32,6 @@ django-extensions==3.1.1 django-filter==2.4.0 django-picklefield==3.0.1; python_version >= '3' django-q==1.3.4 -django-redis==4.12.1 django==3.1.6 djangorestframework==3.12.2 filelock==3.0.12 @@ -87,7 +86,7 @@ sortedcontainers==2.3.0 sqlparse==0.4.1; python_version >= '3.5' threadpoolctl==2.1.0; python_version >= '3.5' tika==1.24 -tqdm==4.56.1 +tqdm==4.56.2 twisted[tls]==20.3.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' txaio==20.12.1; python_version >= '3.6' tzlocal==2.1 From 98b147b62249d9fbb96764785740869cb48cfe62 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 14 Feb 2021 17:08:29 +0100 Subject: [PATCH 122/225] better sanity checker that logs messages in the log files and does not fail on warnings. --- docs/changelog.rst | 6 +- .../commands/document_sanity_checker.py | 16 +-- src/documents/sanity_checker.py | 116 ++++++++--------- src/documents/tasks.py | 14 ++- src/documents/tests/test_management.py | 27 ++-- src/documents/tests/test_sanity_check.py | 119 +++++++++++++++--- src/documents/tests/test_tasks.py | 36 ++++-- 7 files changed, 212 insertions(+), 122 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index fcd14c59d..e5048a097 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -10,8 +10,10 @@ paperless-ng 1.1.2 * Always show top left corner of thumbnails, even for extra wide documents. -* Added a management command for executing the sanity checker directly. See :ref:`utilities-sanity-checker`. - +* Added a management command for executing the sanity checker directly. + See :ref:`utilities-sanity-checker`. + The sanity checker will also report errors in the log files. + * Fixed an issue with the metadata tab not reporting anything in case of missing files. * Reverted a change from 1.1.0 that caused huge memory usage due to redis caching. diff --git a/src/documents/management/commands/document_sanity_checker.py b/src/documents/management/commands/document_sanity_checker.py index b5982ed90..c04a5efef 100644 --- a/src/documents/management/commands/document_sanity_checker.py +++ b/src/documents/management/commands/document_sanity_checker.py @@ -1,8 +1,5 @@ -import logging from django.core.management.base import BaseCommand -from documents.sanity_checker import check_sanity, SanityError, SanityWarning - -logger = logging.getLogger("paperless.management.sanity_checker") +from documents.sanity_checker import check_sanity class Command(BaseCommand): @@ -15,13 +12,4 @@ class Command(BaseCommand): messages = check_sanity(progress=True) - if len(messages) == 0: - logger.info("No issues found.") - else: - for msg in messages: - if type(msg) == SanityError: - logger.error(str(msg)) - elif type(msg) == SanityWarning: - logger.warning(str(msg)) - else: - logger.info((str(msg))) + messages.log_messages() diff --git a/src/documents/sanity_checker.py b/src/documents/sanity_checker.py index f42062154..2e1777841 100644 --- a/src/documents/sanity_checker.py +++ b/src/documents/sanity_checker.py @@ -1,4 +1,5 @@ import hashlib +import logging import os from django.conf import settings @@ -7,40 +8,48 @@ from tqdm import tqdm from documents.models import Document -class SanityMessage: - message = None +class SanityCheckMessages: + + def __init__(self): + self._messages = [] + + def error(self, message): + self._messages.append({"level": logging.ERROR, "message": message}) + + def warning(self, message): + self._messages.append({"level": logging.WARNING, "message": message}) + + def info(self, message): + self._messages.append({"level": logging.INFO, "message": message}) + + def log_messages(self): + logger = logging.getLogger("paperless.sanity_checker") + + if len(self._messages) == 0: + logger.info("Sanity checker detected no issues.") + else: + for msg in self._messages: + logger.log(msg['level'], msg['message']) + + def __len__(self): + return len(self._messages) + + def __getitem__(self, item): + return self._messages[item] + + def has_error(self): + return any([msg['level'] == logging.ERROR for msg in self._messages]) + + def has_warning(self): + return any([msg['level'] == logging.WARNING for msg in self._messages]) -class SanityWarning(SanityMessage): - def __init__(self, message): - self.message = message - - def __str__(self): - return f"Warning: {self.message}" - - -class SanityError(SanityMessage): - def __init__(self, message): - self.message = message - - def __str__(self): - return f"ERROR: {self.message}" - - -class SanityFailedError(Exception): - - def __init__(self, messages): - self.messages = messages - - def __str__(self): - message_string = "\n".join([str(m) for m in self.messages]) - return ( - f"The following issuse were found by the sanity checker:\n" - f"{message_string}\n\n===============\n\n") +class SanityCheckFailedException(Exception): + pass def check_sanity(progress=False): - messages = [] + messages = SanityCheckMessages() present_files = [] for root, subdirs, files in os.walk(settings.MEDIA_ROOT): @@ -59,8 +68,7 @@ def check_sanity(progress=False): for doc in docs: # Check sanity of the thumbnail if not os.path.isfile(doc.thumbnail_path): - messages.append(SanityError( - f"Thumbnail of document {doc.pk} does not exist.")) + messages.error(f"Thumbnail of document {doc.pk} does not exist.") else: if os.path.normpath(doc.thumbnail_path) in present_files: present_files.remove(os.path.normpath(doc.thumbnail_path)) @@ -68,15 +76,14 @@ def check_sanity(progress=False): with doc.thumbnail_file as f: f.read() except OSError as e: - messages.append(SanityError( + messages.error( f"Cannot read thumbnail file of document {doc.pk}: {e}" - )) + ) # Check sanity of the original file # TODO: extract method if not os.path.isfile(doc.source_path): - messages.append(SanityError( - f"Original of document {doc.pk} does not exist.")) + messages.error(f"Original of document {doc.pk} does not exist.") else: if os.path.normpath(doc.source_path) in present_files: present_files.remove(os.path.normpath(doc.source_path)) @@ -84,31 +91,31 @@ def check_sanity(progress=False): with doc.source_file as f: checksum = hashlib.md5(f.read()).hexdigest() except OSError as e: - messages.append(SanityError( - f"Cannot read original file of document {doc.pk}: {e}")) + messages.error( + f"Cannot read original file of document {doc.pk}: {e}") else: if not checksum == doc.checksum: - messages.append(SanityError( + messages.error( f"Checksum mismatch of document {doc.pk}. " f"Stored: {doc.checksum}, actual: {checksum}." - )) + ) # Check sanity of the archive file. if doc.archive_checksum and not doc.archive_filename: - messages.append(SanityError( + messages.error( f"Document {doc.pk} has an archive file checksum, but no " f"archive filename." - )) + ) elif not doc.archive_checksum and doc.archive_filename: - messages.append(SanityError( + messages.error( f"Document {doc.pk} has an archive file, but its checksum is " f"missing." - )) + ) elif doc.has_archive_version: if not os.path.isfile(doc.archive_path): - messages.append(SanityError( + messages.error( f"Archived version of document {doc.pk} does not exist." - )) + ) else: if os.path.normpath(doc.archive_path) in present_files: present_files.remove(os.path.normpath(doc.archive_path)) @@ -116,26 +123,23 @@ def check_sanity(progress=False): with doc.archive_file as f: checksum = hashlib.md5(f.read()).hexdigest() except OSError as e: - messages.append(SanityError( + messages.error( f"Cannot read archive file of document {doc.pk}: {e}" - )) + ) else: if not checksum == doc.archive_checksum: - messages.append(SanityError( + messages.error( f"Checksum mismatch of archived document " f"{doc.pk}. " - f"Stored: {doc.checksum}, actual: {checksum}." - )) + f"Stored: {doc.archive_checksum}, " + f"actual: {checksum}." + ) # other document checks if not doc.content: - messages.append(SanityWarning( - f"Document {doc.pk} has no content." - )) + messages.info(f"Document {doc.pk} has no content.") for extra_file in present_files: - messages.append(SanityWarning( - f"Orphaned file in media dir: {extra_file}" - )) + messages.warning(f"Orphaned file in media dir: {extra_file}") return messages diff --git a/src/documents/tasks.py b/src/documents/tasks.py index f74a3d420..ddfd2eac3 100644 --- a/src/documents/tasks.py +++ b/src/documents/tasks.py @@ -9,8 +9,7 @@ from documents import index, sanity_checker from documents.classifier import DocumentClassifier, load_classifier from documents.consumer import Consumer, ConsumerError from documents.models import Document, Tag, DocumentType, Correspondent -from documents.sanity_checker import SanityFailedError - +from documents.sanity_checker import SanityCheckFailedException logger = logging.getLogger("paperless.tasks") @@ -94,8 +93,15 @@ def consume_file(path, def sanity_check(): messages = sanity_checker.check_sanity() - if len(messages) > 0: - raise SanityFailedError(messages) + messages.log_messages() + + if messages.has_error(): + raise SanityCheckFailedException( + "Sanity check failed with errors. See log.") + elif messages.has_warning(): + return "Sanity check exited with warnings. See log." + elif len(messages) > 0: + return "Sanity check exited with infos. See log." else: return "No issues detected." diff --git a/src/documents/tests/test_management.py b/src/documents/tests/test_management.py index 56710c6e5..40eaaf277 100644 --- a/src/documents/tests/test_management.py +++ b/src/documents/tests/test_management.py @@ -159,33 +159,20 @@ class TestCreateClassifier(TestCase): class TestSanityChecker(DirectoriesMixin, TestCase): - def test_no_errors(self): + def test_no_issues(self): with self.assertLogs() as capture: call_command("document_sanity_checker") self.assertEqual(len(capture.output), 1) - self.assertIn("No issues found.", capture.output[0]) + self.assertIn("Sanity checker detected no issues.", capture.output[0]) - @mock.patch("documents.management.commands.document_sanity_checker.logger.warning") - @mock.patch("documents.management.commands.document_sanity_checker.logger.error") - def test_warnings(self, error, warning): - doc = Document.objects.create(title="test", filename="test.pdf", checksum="d41d8cd98f00b204e9800998ecf8427e") - Path(doc.source_path).touch() - Path(doc.thumbnail_path).touch() - - call_command("document_sanity_checker") - - error.assert_not_called() - warning.assert_called() - - @mock.patch("documents.management.commands.document_sanity_checker.logger.warning") - @mock.patch("documents.management.commands.document_sanity_checker.logger.error") - def test_errors(self, error, warning): + def test_errors(self): doc = Document.objects.create(title="test", content="test", filename="test.pdf", checksum="abc") Path(doc.source_path).touch() Path(doc.thumbnail_path).touch() - call_command("document_sanity_checker") + with self.assertLogs() as capture: + call_command("document_sanity_checker") - warning.assert_not_called() - error.assert_called() + self.assertEqual(len(capture.output), 1) + self.assertIn("Checksum mismatch of document", capture.output[0]) diff --git a/src/documents/tests/test_sanity_check.py b/src/documents/tests/test_sanity_check.py index fae18aaca..6f73e4962 100644 --- a/src/documents/tests/test_sanity_check.py +++ b/src/documents/tests/test_sanity_check.py @@ -1,3 +1,4 @@ +import logging import os import shutil from pathlib import Path @@ -7,10 +8,59 @@ from django.conf import settings from django.test import TestCase from documents.models import Document -from documents.sanity_checker import check_sanity, SanityFailedError +from documents.sanity_checker import check_sanity, SanityCheckMessages from documents.tests.utils import DirectoriesMixin +class TestSanityCheckMessages(TestCase): + + def test_no_messages(self): + messages = SanityCheckMessages() + self.assertEqual(len(messages), 0) + self.assertFalse(messages.has_error()) + self.assertFalse(messages.has_warning()) + with self.assertLogs() as capture: + messages.log_messages() + self.assertEqual(len(capture.output), 1) + self.assertEqual(capture.records[0].levelno, logging.INFO) + self.assertEqual(capture.records[0].message, "Sanity checker detected no issues.") + + def test_info(self): + messages = SanityCheckMessages() + messages.info("Something might be wrong") + self.assertEqual(len(messages), 1) + self.assertFalse(messages.has_error()) + self.assertFalse(messages.has_warning()) + with self.assertLogs() as capture: + messages.log_messages() + self.assertEqual(len(capture.output), 1) + self.assertEqual(capture.records[0].levelno, logging.INFO) + self.assertEqual(capture.records[0].message, "Something might be wrong") + + def test_warning(self): + messages = SanityCheckMessages() + messages.warning("Something is wrong") + self.assertEqual(len(messages), 1) + self.assertFalse(messages.has_error()) + self.assertTrue(messages.has_warning()) + with self.assertLogs() as capture: + messages.log_messages() + self.assertEqual(len(capture.output), 1) + self.assertEqual(capture.records[0].levelno, logging.WARNING) + self.assertEqual(capture.records[0].message, "Something is wrong") + + def test_error(self): + messages = SanityCheckMessages() + messages.error("Something is seriously wrong") + self.assertEqual(len(messages), 1) + self.assertTrue(messages.has_error()) + self.assertFalse(messages.has_warning()) + with self.assertLogs() as capture: + messages.log_messages() + self.assertEqual(len(capture.output), 1) + self.assertEqual(capture.records[0].levelno, logging.ERROR) + self.assertEqual(capture.records[0].message, "Something is seriously wrong") + class TestSanityCheck(DirectoriesMixin, TestCase): def make_test_data(self): @@ -33,72 +83,105 @@ class TestSanityCheck(DirectoriesMixin, TestCase): def test_no_thumbnail(self): doc = self.make_test_data() os.remove(doc.thumbnail_path) - self.assertEqual(len(check_sanity()), 1) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Thumbnail of document .* does not exist") def test_thumbnail_no_access(self): doc = self.make_test_data() os.chmod(doc.thumbnail_path, 0o000) - self.assertEqual(len(check_sanity()), 1) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Cannot read thumbnail file of document") os.chmod(doc.thumbnail_path, 0o777) def test_no_original(self): doc = self.make_test_data() os.remove(doc.source_path) - self.assertEqual(len(check_sanity()), 1) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Original of document .* does not exist.") def test_original_no_access(self): doc = self.make_test_data() os.chmod(doc.source_path, 0o000) - self.assertEqual(len(check_sanity()), 1) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Cannot read original file of document") os.chmod(doc.source_path, 0o777) def test_original_checksum_mismatch(self): doc = self.make_test_data() doc.checksum = "WOW" doc.save() - self.assertEqual(len(check_sanity()), 1) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Checksum mismatch of document") def test_no_archive(self): doc = self.make_test_data() os.remove(doc.archive_path) - self.assertEqual(len(check_sanity()), 1) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Archived version of document .* does not exist.") def test_archive_no_access(self): doc = self.make_test_data() os.chmod(doc.archive_path, 0o000) - self.assertEqual(len(check_sanity()), 1) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Cannot read archive file of document") os.chmod(doc.archive_path, 0o777) def test_archive_checksum_mismatch(self): doc = self.make_test_data() doc.archive_checksum = "WOW" doc.save() - self.assertEqual(len(check_sanity()), 1) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Checksum mismatch of archived document") def test_empty_content(self): doc = self.make_test_data() doc.content = "" doc.save() - self.assertEqual(len(check_sanity()), 1) + messages = check_sanity() + self.assertFalse(messages.has_error()) + self.assertFalse(messages.has_warning()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Document .* has no content.") def test_orphaned_file(self): doc = self.make_test_data() Path(self.dirs.originals_dir, "orphaned").touch() - self.assertEqual(len(check_sanity()), 1) - - def test_error_tostring(self): - Document.objects.create(title="test", checksum="dgfhj", archive_checksum="dfhg", content="", pk=1, filename="0000001.pdf", archive_filename="0000001.pdf") - string = str(SanityFailedError(check_sanity())) - self.assertIsNotNone(string) + messages = check_sanity() + self.assertFalse(messages.has_error()) + self.assertTrue(messages.has_warning()) + self.assertEqual(len(messages), 1) + self.assertRegex(messages[0]['message'], "Orphaned file in media dir") def test_archive_filename_no_checksum(self): doc = self.make_test_data() doc.archive_checksum = None doc.save() - self.assertEqual(len(check_sanity()), 2) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 2) + self.assertRegex(messages[0]['message'], "has an archive file, but its checksum is missing.") def test_archive_checksum_no_filename(self): doc = self.make_test_data() doc.archive_filename = None doc.save() - self.assertEqual(len(check_sanity()), 2) + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertEqual(len(messages), 2) + self.assertRegex(messages[0]['message'], "has an archive file checksum, but no archive filename.") diff --git a/src/documents/tests/test_tasks.py b/src/documents/tests/test_tasks.py index 8c5a1b5df..dab8ebfb9 100644 --- a/src/documents/tests/test_tasks.py +++ b/src/documents/tests/test_tasks.py @@ -2,12 +2,12 @@ import os from unittest import mock from django.conf import settings -from django.test import TestCase, override_settings +from django.test import TestCase from django.utils import timezone from documents import tasks from documents.models import Document, Tag, Correspondent, DocumentType -from documents.sanity_checker import SanityError, SanityFailedError +from documents.sanity_checker import SanityCheckMessages, SanityCheckFailedException from documents.tests.utils import DirectoriesMixin @@ -74,13 +74,33 @@ class TestTasks(DirectoriesMixin, TestCase): self.assertNotEqual(mtime2, mtime3) @mock.patch("documents.tasks.sanity_checker.check_sanity") - def test_sanity_check(self, m): - m.return_value = [] - tasks.sanity_check() + def test_sanity_check_success(self, m): + m.return_value = SanityCheckMessages() + self.assertEqual(tasks.sanity_check(), "No issues detected.") m.assert_called_once() - m.reset_mock() - m.return_value = [SanityError("")] - self.assertRaises(SanityFailedError, tasks.sanity_check) + + @mock.patch("documents.tasks.sanity_checker.check_sanity") + def test_sanity_check_error(self, m): + messages = SanityCheckMessages() + messages.error("Some error") + m.return_value = messages + self.assertRaises(SanityCheckFailedException, tasks.sanity_check) + m.assert_called_once() + + @mock.patch("documents.tasks.sanity_checker.check_sanity") + def test_sanity_check_warning(self, m): + messages = SanityCheckMessages() + messages.warning("Some warning") + m.return_value = messages + self.assertEqual(tasks.sanity_check(), "Sanity check exited with warnings. See log.") + m.assert_called_once() + + @mock.patch("documents.tasks.sanity_checker.check_sanity") + def test_sanity_check_info(self, m): + messages = SanityCheckMessages() + messages.info("Some info") + m.return_value = messages + self.assertEqual(tasks.sanity_check(), "Sanity check exited with infos. See log.") m.assert_called_once() def test_bulk_update_documents(self): From 5e669534f2301e3467f1a954c946a68b48a5cd96 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Sun, 14 Feb 2021 17:24:31 +0100 Subject: [PATCH 123/225] reorganized test case --- docs/changelog.rst | 3 +- src/documents/tests/test_sanity_check.py | 55 +++++++----------------- 2 files changed, 17 insertions(+), 41 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index e5048a097..f51360258 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -12,7 +12,8 @@ paperless-ng 1.1.2 * Added a management command for executing the sanity checker directly. See :ref:`utilities-sanity-checker`. - The sanity checker will also report errors in the log files. + +* The weekly sanity check now reports messages in the log files. * Fixed an issue with the metadata tab not reporting anything in case of missing files. diff --git a/src/documents/tests/test_sanity_check.py b/src/documents/tests/test_sanity_check.py index 6f73e4962..b07ba930a 100644 --- a/src/documents/tests/test_sanity_check.py +++ b/src/documents/tests/test_sanity_check.py @@ -73,6 +73,11 @@ class TestSanityCheck(DirectoriesMixin, TestCase): return Document.objects.create(title="test", checksum="42995833e01aea9b3edee44bbfdd7ce1", archive_checksum="62acb0bcbfbcaa62ca6ad3668e4e404b", content="test", pk=1, filename="0000001.pdf", mime_type="application/pdf", archive_filename="0000001.pdf") + def assertSanityError(self, messageRegex): + messages = check_sanity() + self.assertTrue(messages.has_error()) + self.assertRegex(messages[0]['message'], messageRegex) + def test_no_docs(self): self.assertEqual(len(check_sanity()), 0) @@ -83,71 +88,47 @@ class TestSanityCheck(DirectoriesMixin, TestCase): def test_no_thumbnail(self): doc = self.make_test_data() os.remove(doc.thumbnail_path) - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 1) - self.assertRegex(messages[0]['message'], "Thumbnail of document .* does not exist") + self.assertSanityError("Thumbnail of document .* does not exist") def test_thumbnail_no_access(self): doc = self.make_test_data() os.chmod(doc.thumbnail_path, 0o000) - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 1) - self.assertRegex(messages[0]['message'], "Cannot read thumbnail file of document") + self.assertSanityError("Cannot read thumbnail file of document") os.chmod(doc.thumbnail_path, 0o777) def test_no_original(self): doc = self.make_test_data() os.remove(doc.source_path) - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 1) - self.assertRegex(messages[0]['message'], "Original of document .* does not exist.") + self.assertSanityError("Original of document .* does not exist.") def test_original_no_access(self): doc = self.make_test_data() os.chmod(doc.source_path, 0o000) - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 1) - self.assertRegex(messages[0]['message'], "Cannot read original file of document") + self.assertSanityError("Cannot read original file of document") os.chmod(doc.source_path, 0o777) def test_original_checksum_mismatch(self): doc = self.make_test_data() doc.checksum = "WOW" doc.save() - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 1) - self.assertRegex(messages[0]['message'], "Checksum mismatch of document") + self.assertSanityError("Checksum mismatch of document") def test_no_archive(self): doc = self.make_test_data() os.remove(doc.archive_path) - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 1) - self.assertRegex(messages[0]['message'], "Archived version of document .* does not exist.") + self.assertSanityError("Archived version of document .* does not exist.") def test_archive_no_access(self): doc = self.make_test_data() os.chmod(doc.archive_path, 0o000) - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 1) - self.assertRegex(messages[0]['message'], "Cannot read archive file of document") + self.assertSanityError("Cannot read archive file of document") os.chmod(doc.archive_path, 0o777) def test_archive_checksum_mismatch(self): doc = self.make_test_data() doc.archive_checksum = "WOW" doc.save() - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 1) - self.assertRegex(messages[0]['message'], "Checksum mismatch of archived document") + self.assertSanityError("Checksum mismatch of archived document") def test_empty_content(self): doc = self.make_test_data() @@ -172,16 +153,10 @@ class TestSanityCheck(DirectoriesMixin, TestCase): doc = self.make_test_data() doc.archive_checksum = None doc.save() - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 2) - self.assertRegex(messages[0]['message'], "has an archive file, but its checksum is missing.") + self.assertSanityError("has an archive file, but its checksum is missing.") def test_archive_checksum_no_filename(self): doc = self.make_test_data() doc.archive_filename = None doc.save() - messages = check_sanity() - self.assertTrue(messages.has_error()) - self.assertEqual(len(messages), 2) - self.assertRegex(messages[0]['message'], "has an archive file checksum, but no archive filename.") + self.assertSanityError("has an archive file checksum, but no archive filename.") From 7e88085377ec68edced2ce528eb0b347059abab6 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 11:25:25 +0100 Subject: [PATCH 124/225] load sklearn modules only when training data has changed --- src/documents/classifier.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/documents/classifier.py b/src/documents/classifier.py index b577997e3..47dd7dfc7 100755 --- a/src/documents/classifier.py +++ b/src/documents/classifier.py @@ -95,9 +95,6 @@ class DocumentClassifier(object): pickle.dump(self.document_type_classifier, f) def train(self): - from sklearn.feature_extraction.text import CountVectorizer - from sklearn.neural_network import MLPClassifier - from sklearn.preprocessing import MultiLabelBinarizer, LabelBinarizer data = list() labels_tags = list() @@ -162,6 +159,10 @@ class DocumentClassifier(object): ) ) + from sklearn.feature_extraction.text import CountVectorizer + from sklearn.neural_network import MLPClassifier + from sklearn.preprocessing import MultiLabelBinarizer, LabelBinarizer + # Step 2: vectorize data logger.debug("Vectorizing data...") self.data_vectorizer = CountVectorizer( From c330cca2c9cdcd4328415504cfcee974b831e9cd Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 11:26:13 +0100 Subject: [PATCH 125/225] remove unused imports --- src/documents/admin.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/documents/admin.py b/src/documents/admin.py index f87ab1ec5..47e6c7f89 100755 --- a/src/documents/admin.py +++ b/src/documents/admin.py @@ -1,10 +1,8 @@ from django.contrib import admin -from django.utils.html import format_html, format_html_join -from django.utils.safestring import mark_safe from whoosh.writing import AsyncWriter from . import index -from .models import Correspondent, Document, DocumentType, Log, Tag, \ +from .models import Correspondent, Document, DocumentType, Tag, \ SavedView, SavedViewFilterRule From 416101d557797517da202c405ca6b4b99f9e9955 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 11:52:46 +0100 Subject: [PATCH 126/225] only import dateparser when required --- src/documents/parsers.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/documents/parsers.py b/src/documents/parsers.py index 1ed5deb3c..b2714f6a3 100644 --- a/src/documents/parsers.py +++ b/src/documents/parsers.py @@ -6,7 +6,6 @@ import shutil import subprocess import tempfile -import dateparser import magic from django.conf import settings from django.utils import timezone @@ -200,6 +199,8 @@ def parse_date(filename, text): """ Call dateparser.parse with a particular date ordering """ + import dateparser + return dateparser.parse( ds, settings={ From 56bd966c02a4347bf764215d33897583fd5d5111 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 12:18:10 +0100 Subject: [PATCH 127/225] local import of ocrmypdf so that the webserver does not load that --- src/paperless_tesseract/parsers.py | 10 ++++++---- src/paperless_tesseract/tests/test_parser.py | 11 +++-------- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/src/paperless_tesseract/parsers.py b/src/paperless_tesseract/parsers.py index 271a840df..78c335ac3 100644 --- a/src/paperless_tesseract/parsers.py +++ b/src/paperless_tesseract/parsers.py @@ -2,12 +2,8 @@ import json import os import re -import ocrmypdf -import pdftotext -import pikepdf from PIL import Image from django.conf import settings -from ocrmypdf import InputFileError, EncryptedPdfError from documents.parsers import DocumentParser, ParseError, \ make_thumbnail_from_pdf @@ -22,6 +18,8 @@ class RasterisedDocumentParser(DocumentParser): logging_name = "paperless.parsing.tesseract" def extract_metadata(self, document_path, mime_type): + import pikepdf + namespace_pattern = re.compile(r"\{(.*)\}(.*)") result = [] @@ -91,6 +89,9 @@ class RasterisedDocumentParser(DocumentParser): return None def parse(self, document_path, mime_type, file_name=None): + import ocrmypdf + from ocrmypdf import InputFileError, EncryptedPdfError + mode = settings.OCR_MODE text_original = get_text_from_pdf(document_path) @@ -223,6 +224,7 @@ def strip_excess_whitespace(text): def get_text_from_pdf(pdf_file): + import pdftotext if not os.path.isfile(pdf_file): return None diff --git a/src/paperless_tesseract/tests/test_parser.py b/src/paperless_tesseract/tests/test_parser.py index 644587de0..4fd314667 100644 --- a/src/paperless_tesseract/tests/test_parser.py +++ b/src/paperless_tesseract/tests/test_parser.py @@ -164,17 +164,12 @@ class TestParser(DirectoriesMixin, TestCase): self.assertRaises(ParseError, f) - @mock.patch("paperless_tesseract.parsers.ocrmypdf.ocr") - def test_image_calc_a4_dpi(self, m): + def test_image_calc_a4_dpi(self): parser = RasterisedDocumentParser(None) - parser.parse(os.path.join(self.SAMPLE_FILES, "simple-no-dpi.png"), "image/png") + dpi = parser.calculate_a4_dpi(os.path.join(self.SAMPLE_FILES, "simple-no-dpi.png")) - m.assert_called_once() - - args, kwargs = m.call_args - - self.assertEqual(kwargs['image_dpi'], 62) + self.assertEqual(dpi, 62) @mock.patch("paperless_tesseract.parsers.RasterisedDocumentParser.calculate_a4_dpi") def test_image_dpi_fail(self, m): From 8bf4241b1656a8f55a84362985ad7b56335e0839 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 13:26:36 +0100 Subject: [PATCH 128/225] some search index optimizations --- src/documents/admin.py | 10 ++++---- src/documents/bulk_edit.py | 7 +++--- src/documents/index.py | 22 +++++++++++++---- src/documents/signals/handlers.py | 4 +++- src/documents/tests/test_admin.py | 40 +++++++++++++++++++++---------- src/documents/views.py | 22 ++++++++--------- 6 files changed, 69 insertions(+), 36 deletions(-) diff --git a/src/documents/admin.py b/src/documents/admin.py index 47e6c7f89..aee45b49d 100755 --- a/src/documents/admin.py +++ b/src/documents/admin.py @@ -1,7 +1,5 @@ from django.contrib import admin -from whoosh.writing import AsyncWriter -from . import index from .models import Correspondent, Document, DocumentType, Tag, \ SavedView, SavedViewFilterRule @@ -84,17 +82,21 @@ class DocumentAdmin(admin.ModelAdmin): created_.short_description = "Created" def delete_queryset(self, request, queryset): - ix = index.open_index() - with AsyncWriter(ix) as writer: + from documents import index + + with index.open_index_writer() as writer: for o in queryset: index.remove_document(writer, o) + super(DocumentAdmin, self).delete_queryset(request, queryset) def delete_model(self, request, obj): + from documents import index index.remove_document_from_index(obj) super(DocumentAdmin, self).delete_model(request, obj) def save_model(self, request, obj, form, change): + from documents import index index.add_or_update_document(obj) super(DocumentAdmin, self).save_model(request, obj, form, change) diff --git a/src/documents/bulk_edit.py b/src/documents/bulk_edit.py index c0c80a795..7503eafc5 100644 --- a/src/documents/bulk_edit.py +++ b/src/documents/bulk_edit.py @@ -2,9 +2,7 @@ import itertools from django.db.models import Q from django_q.tasks import async_task -from whoosh.writing import AsyncWriter -from documents import index from documents.models import Document, Correspondent, DocumentType @@ -99,8 +97,9 @@ def modify_tags(doc_ids, add_tags, remove_tags): def delete(doc_ids): Document.objects.filter(id__in=doc_ids).delete() - ix = index.open_index() - with AsyncWriter(ix) as writer: + from documents import index + + with index.open_index_writer() as writer: for id in doc_ids: index.remove_document_by_id(writer, id) diff --git a/src/documents/index.py b/src/documents/index.py index ea788f4b3..89e56e930 100644 --- a/src/documents/index.py +++ b/src/documents/index.py @@ -86,6 +86,22 @@ def open_index(recreate=False): return create_in(settings.INDEX_DIR, get_schema()) +@contextmanager +def open_index_writer(ix=None, optimize=False): + if ix: + writer = AsyncWriter(ix) + else: + writer = AsyncWriter(open_index()) + + try: + yield writer + except Exception as e: + logger.exception(str(e)) + writer.cancel() + finally: + writer.commit(optimize=optimize) + + def update_document(writer, doc): tags = ",".join([t.name for t in doc.tags.all()]) writer.update_document( @@ -110,14 +126,12 @@ def remove_document_by_id(writer, doc_id): def add_or_update_document(document): - ix = open_index() - with AsyncWriter(ix) as writer: + with open_index_writer() as writer: update_document(writer, document) def remove_document_from_index(document): - ix = open_index() - with AsyncWriter(ix) as writer: + with open_index_writer() as writer: remove_document(writer, document) diff --git a/src/documents/signals/handlers.py b/src/documents/signals/handlers.py index eb981661c..ad63bf301 100755 --- a/src/documents/signals/handlers.py +++ b/src/documents/signals/handlers.py @@ -11,7 +11,7 @@ from django.dispatch import receiver from django.utils import timezone from filelock import FileLock -from .. import index, matching +from .. import matching from ..file_handling import delete_empty_directories, \ create_source_path_directory, \ generate_unique_filename @@ -305,4 +305,6 @@ def set_log_entry(sender, document=None, logging_group=None, **kwargs): def add_to_index(sender, document, **kwargs): + from documents import index + index.add_or_update_document(document) diff --git a/src/documents/tests/test_admin.py b/src/documents/tests/test_admin.py index 840747d68..ce00a0698 100644 --- a/src/documents/tests/test_admin.py +++ b/src/documents/tests/test_admin.py @@ -4,6 +4,7 @@ from django.contrib.admin.sites import AdminSite from django.test import TestCase from django.utils import timezone +from documents import index from documents.admin import DocumentAdmin from documents.models import Document from documents.tests.utils import DirectoriesMixin @@ -11,37 +12,52 @@ from documents.tests.utils import DirectoriesMixin class TestDocumentAdmin(DirectoriesMixin, TestCase): + def get_document_from_index(self, doc): + ix = index.open_index() + with ix.searcher() as searcher: + return searcher.document(id=doc.id) + def setUp(self) -> None: super(TestDocumentAdmin, self).setUp() self.doc_admin = DocumentAdmin(model=Document, admin_site=AdminSite()) - @mock.patch("documents.admin.index.add_or_update_document") - def test_save_model(self, m): + def test_save_model(self): doc = Document.objects.create(title="test") + doc.title = "new title" self.doc_admin.save_model(None, doc, None, None) self.assertEqual(Document.objects.get(id=doc.id).title, "new title") - m.assert_called_once() + self.assertEqual(self.get_document_from_index(doc)['title'], "new title") - @mock.patch("documents.admin.index.remove_document") - def test_delete_model(self, m): + def test_delete_model(self): doc = Document.objects.create(title="test") - self.doc_admin.delete_model(None, doc) - self.assertRaises(Document.DoesNotExist, Document.objects.get, id=doc.id) - m.assert_called_once() + index.add_or_update_document(doc) + self.assertIsNotNone(self.get_document_from_index(doc)) - @mock.patch("documents.admin.index.remove_document") - def test_delete_queryset(self, m): + self.doc_admin.delete_model(None, doc) + + self.assertRaises(Document.DoesNotExist, Document.objects.get, id=doc.id) + self.assertIsNone(self.get_document_from_index(doc)) + + def test_delete_queryset(self): + docs = [] for i in range(42): - Document.objects.create(title="Many documents with the same title", checksum=f"{i:02}") + doc = Document.objects.create(title="Many documents with the same title", checksum=f"{i:02}") + docs.append(doc) + index.add_or_update_document(doc) self.assertEqual(Document.objects.count(), 42) + for doc in docs: + self.assertIsNotNone(self.get_document_from_index(doc)) + self.doc_admin.delete_queryset(None, Document.objects.all()) - self.assertEqual(m.call_count, 42) self.assertEqual(Document.objects.count(), 0) + for doc in docs: + self.assertIsNone(self.get_document_from_index(doc)) + def test_created(self): doc = Document.objects.create(title="test", created=timezone.datetime(2020, 4, 12)) self.assertEqual(self.doc_admin.created_(doc), "2020-04-12") diff --git a/src/documents/views.py b/src/documents/views.py index b2e5b4cd3..9d7b04508 100755 --- a/src/documents/views.py +++ b/src/documents/views.py @@ -32,7 +32,6 @@ from rest_framework.viewsets import ( ViewSet ) -import documents.index as index from paperless.db import GnuPG from paperless.views import StandardPagination from .classifier import load_classifier @@ -176,10 +175,12 @@ class DocumentViewSet(RetrieveModelMixin, def update(self, request, *args, **kwargs): response = super(DocumentViewSet, self).update( request, *args, **kwargs) + from documents import index index.add_or_update_document(self.get_object()) return response def destroy(self, request, *args, **kwargs): + from documents import index index.remove_document_from_index(self.get_object()) return super(DocumentViewSet, self).destroy(request, *args, **kwargs) @@ -501,10 +502,6 @@ class SearchView(APIView): permission_classes = (IsAuthenticated,) - def __init__(self, *args, **kwargs): - super(SearchView, self).__init__(*args, **kwargs) - self.ix = index.open_index() - def add_infos_to_hit(self, r): try: doc = Document.objects.get(id=r['id']) @@ -525,6 +522,7 @@ class SearchView(APIView): } def get(self, request, format=None): + from documents import index if 'query' in request.query_params: query = request.query_params['query'] @@ -554,8 +552,10 @@ class SearchView(APIView): if page < 1: page = 1 + ix = index.open_index() + try: - with index.query_page(self.ix, page, query, more_like_id, more_like_content) as (result_page, corrected_query): # NOQA: E501 + with index.query_page(ix, page, query, more_like_id, more_like_content) as (result_page, corrected_query): # NOQA: E501 return Response( {'count': len(result_page), 'page': result_page.pagenum, @@ -570,10 +570,6 @@ class SearchAutoCompleteView(APIView): permission_classes = (IsAuthenticated,) - def __init__(self, *args, **kwargs): - super(SearchAutoCompleteView, self).__init__(*args, **kwargs) - self.ix = index.open_index() - def get(self, request, format=None): if 'term' in request.query_params: term = request.query_params['term'] @@ -587,7 +583,11 @@ class SearchAutoCompleteView(APIView): else: limit = 10 - return Response(index.autocomplete(self.ix, term, limit)) + from documents import index + + ix = index.open_index() + + return Response(index.autocomplete(ix, term, limit)) class StatisticsView(APIView): From 2fcf4842299afa54f46257866554fe10f63b2818 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 14:52:47 +0100 Subject: [PATCH 129/225] bugfix dismissing wrong status messages --- src-ui/src/app/services/consumer-status.service.ts | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src-ui/src/app/services/consumer-status.service.ts b/src-ui/src/app/services/consumer-status.service.ts index 23df0871f..e03282175 100644 --- a/src-ui/src/app/services/consumer-status.service.ts +++ b/src-ui/src/app/services/consumer-status.service.ts @@ -169,7 +169,12 @@ export class ConsumerStatusService { } dismiss(status: FileStatus) { - let index = this.consumerStatus.findIndex(s => s.filename == status.filename) + let index + if (status.taskId != null) { + index = this.consumerStatus.findIndex(s => s.taskId == status.taskId) + } else { + index = this.consumerStatus.findIndex(s => s.filename == status.filename) + } if (index > -1) { this.consumerStatus.splice(index, 1) From f07a6b45869431cae3730b7caf5cbc87acc0cfc5 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 16:27:35 +0100 Subject: [PATCH 130/225] PAPERLESS_WEBSERVER_WORKERS option --- docs/changelog.rst | 8 ++++++ docs/configuration.rst | 62 ++++++++++++++++++++++++++++++++++++++++++ docs/setup.rst | 5 +++- gunicorn.conf.py | 4 ++- 4 files changed, 77 insertions(+), 2 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index f51360258..1b3434117 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -5,6 +5,14 @@ Changelog ********* +paperless-ng 1.1.3 +################## + +* Added a docker-specific configuration option to adjust the number of + worker processes of the web server. See :ref:`configuration-docker`. + +* Some more memory usage optimizations. + paperless-ng 1.1.2 ################## diff --git a/docs/configuration.rst b/docs/configuration.rst index a307cc25a..95914e836 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -555,3 +555,65 @@ PAPERLESS_GS_BINARY= PAPERLESS_OPTIPNG_BINARY= Defaults to "/usr/bin/optipng". + + +.. _configuration-docker: + +Docker-specific options +####################### + +These options don't have any effect in ``paperless.conf``. These options adjust +the behavior of the docker container. Configure these in `docker-compose.env`. + +PAPERLESS_WEBSERVER_WORKERS= + The number of worker processes the webserver should spawn. More worker processes + usually result in the front end to load data much quicker. However, each worker process + also loads the entire application into memory separately, so increasing this value + will increase RAM usage. + + Consider configuring this to 1 on low power devices with limited amount of RAM. + + Defaults to 2. + +USERMAP_UID= + The ID of the paperless user in the container. Set this to your actual user ID on the + host system, which you can get by executing + + .. code:: shell-session + + $ id -u + + Paperless will change ownership on its folders to this user, so you need to get this right + in order to be able to write to the consumption directory. + + Defaults to 1000. + +USERMAP_GID= + The ID of the paperless Group in the container. Set this to your actual group ID on the + host system, which you can get by executing + + .. code:: shell-session + + $ id -g + + Paperless will change ownership on its folders to this group, so you need to get this right + in order to be able to write to the consumption directory. + + Defaults to 1000. + +PAPERLESS_OCR_LANGUAGES= + Additional OCR languages to install. By default, paperless comes with + English, German, Italian, Spanish and French. If your language is not in this list, install + additional languages with this configuration option: + + .. code:: bash + + PAPERLESS_OCR_LANGUAGES=tur ces + + To actually use these languages, also set the default OCR language of paperless: + + .. code:: bash + + PAPERLESS_OCR_LANGUAGE=tur + + Defaults to none, which does not install any additional languages. diff --git a/docs/setup.rst b/docs/setup.rst index 4025b6721..8ebb6f83c 100644 --- a/docs/setup.rst +++ b/docs/setup.rst @@ -763,7 +763,8 @@ configuring some options in paperless can help improve performance immensely: * Stick with SQLite to save some resources. * Consider setting ``PAPERLESS_OCR_PAGES`` to 1, so that paperless will only OCR - the first page of your documents. + the first page of your documents. In most cases, this page contains enough + information to be able to find it. * ``PAPERLESS_TASK_WORKERS`` and ``PAPERLESS_THREADS_PER_WORKER`` are configured to use all cores. The Raspberry Pi models 3 and up have 4 cores, meaning that paperless will use 2 workers and 2 threads per worker. This may result in @@ -776,6 +777,8 @@ configuring some options in paperless can help improve performance immensely: file generation for already ocr'ed documents entirely. * Set ``PAPERLESS_OPTIMIZE_THUMBNAILS`` to 'false' if you want faster consumption times. Thumbnails will be about 20% larger. +* If using docker, consider setting ``PAPERLESS_WEBSERVER_WORKERS`` to + 1. This will save some memory. For details, refer to :ref:`configuration`. diff --git a/gunicorn.conf.py b/gunicorn.conf.py index 6fcbe6b0e..995b4fb9d 100644 --- a/gunicorn.conf.py +++ b/gunicorn.conf.py @@ -1,5 +1,7 @@ +import os + bind = '0.0.0.0:8000' -workers = 2 +workers = int(os.getenv("PAPERLESS_WEBSERVER_WORKERS", 2)) worker_class = 'uvicorn.workers.UvicornWorker' timeout = 120 From 43fdf634f26703525f1ab77a2082cc972bc287f1 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 16:37:44 +0100 Subject: [PATCH 131/225] added a note regarding python 3.6 --- docs/changelog.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 1b3434117..978a2e966 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -13,6 +13,16 @@ paperless-ng 1.1.3 * Some more memory usage optimizations. +.. note:: + + Many packages that paperless depends on are slowly dropping Python 3.6 + support one after another, including the web server. + + At some point, paperless will drop Python 3.6 support. If using a bare + metal installation upgrade to python 3.7 or newer if you're still on 3.6. + + If using docker, this does not affect you. + paperless-ng 1.1.2 ################## From 831db6ab878dc9cbe712faa959cae6b9765a0b52 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 16:46:06 +0100 Subject: [PATCH 132/225] note regarding Python 3.6 --- docs/changelog.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 978a2e966..fa1be64fd 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -15,11 +15,12 @@ paperless-ng 1.1.3 .. note:: - Many packages that paperless depends on are slowly dropping Python 3.6 - support one after another, including the web server. + Some packages that paperless depends on are slowly dropping Python 3.6 + support one after another, including the web server. Supporting Python + 3.6 means that I cannot update these packages anymore. At some point, paperless will drop Python 3.6 support. If using a bare - metal installation upgrade to python 3.7 or newer if you're still on 3.6. + metal installation and you're still on Python 3.6, upgrade to 3.7 or newer. If using docker, this does not affect you. From 3f03d51b244c457912e4fbf0cbd6f859c85ab863 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 16:52:45 +0100 Subject: [PATCH 133/225] version bump --- src-ui/src/environments/environment.prod.ts | 2 +- src/paperless/version.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src-ui/src/environments/environment.prod.ts b/src-ui/src/environments/environment.prod.ts index e2c66690e..b2ffebc09 100644 --- a/src-ui/src/environments/environment.prod.ts +++ b/src-ui/src/environments/environment.prod.ts @@ -2,7 +2,7 @@ export const environment = { production: true, apiBaseUrl: "/api/", appTitle: "Paperless-ng", - version: "1.1.2", + version: "1.1.3", webSocketHost: window.location.host, webSocketProtocol: (window.location.protocol == "https:" ? "wss:" : "ws:") }; diff --git a/src/paperless/version.py b/src/paperless/version.py index 728d237e2..08f427169 100644 --- a/src/paperless/version.py +++ b/src/paperless/version.py @@ -1 +1 @@ -__version__ = (1, 1, 2) +__version__ = (1, 1, 3) From cb308fae7b8ae1ff2763c09a0ecf942acede99fc Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 23:14:54 +0100 Subject: [PATCH 134/225] only show inbox statistics if inbox tags are defined --- .../statistics-widget.component.html | 4 ++-- src/documents/tests/test_api.py | 7 +++++++ src/documents/views.py | 12 +++++++++--- 3 files changed, 18 insertions(+), 5 deletions(-) diff --git a/src-ui/src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html b/src-ui/src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html index 6ab996675..a29b50f78 100644 --- a/src-ui/src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html +++ b/src-ui/src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html @@ -1,6 +1,6 @@ -

    Documents in inbox: {{statistics.documents_inbox}}

    -

    Total documents: {{statistics.documents_total}}

    +

    Documents in inbox: {{statistics?.documents_inbox}}

    +

    Total documents: {{statistics?.documents_total}}

         diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py index 20eba6628..01e7210a5 100644 --- a/src/documents/tests/test_api.py +++ b/src/documents/tests/test_api.py @@ -442,6 +442,13 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): self.assertEqual(response.data['documents_total'], 3) self.assertEqual(response.data['documents_inbox'], 1) + def test_statistics_no_inbox_tag(self): + Document.objects.create(title="none1", checksum="A") + + response = self.client.get("/api/statistics/") + self.assertEqual(response.status_code, 200) + self.assertEqual(response.data['documents_inbox'], None) + @mock.patch("documents.views.async_task") def test_upload(self, m): diff --git a/src/documents/views.py b/src/documents/views.py index 9d7b04508..68d6e3c77 100755 --- a/src/documents/views.py +++ b/src/documents/views.py @@ -595,8 +595,14 @@ class StatisticsView(APIView): permission_classes = (IsAuthenticated,) def get(self, request, format=None): - return Response({ - 'documents_total': Document.objects.all().count(), - 'documents_inbox': Document.objects.filter( + documents_total = Document.objects.all().count() + if Tag.objects.filter(is_inbox_tag=True).exists(): + documents_inbox = Document.objects.filter( tags__is_inbox_tag=True).distinct().count() + else: + documents_inbox = None + + return Response({ + 'documents_total': documents_total, + 'documents_inbox': documents_inbox, }) From 2c4e34dd0cc0e21478304a99e6344f1564fdd2ec Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Mon, 15 Feb 2021 23:44:48 +0100 Subject: [PATCH 135/225] changelog --- docs/changelog.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index fa1be64fd..8b87e642e 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -13,12 +13,14 @@ paperless-ng 1.1.3 * Some more memory usage optimizations. +* Don't show inbox statistics if no inbox tag is defined. + .. note:: Some packages that paperless depends on are slowly dropping Python 3.6 support one after another, including the web server. Supporting Python 3.6 means that I cannot update these packages anymore. - + At some point, paperless will drop Python 3.6 support. If using a bare metal installation and you're still on Python 3.6, upgrade to 3.7 or newer. From 1e4ec7e29ef746febf958da7b0e2881b83dbbacf Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Tue, 16 Feb 2021 14:54:18 +0100 Subject: [PATCH 136/225] added en-GB language --- src-ui/angular.json | 3 +- src-ui/src/app/app.module.ts | 2 + .../manage/settings/settings.component.html | 2 +- src-ui/src/app/services/settings.service.ts | 1 + src-ui/src/locale/messages.en_GB.xlf | 2 +- src/locale/en_GB/LC_MESSAGES/django.po | 6 +- .../{en-us => en_US}/LC_MESSAGES/django.po | 178 ++++++++++-------- src/paperless/settings.py | 3 +- 8 files changed, 107 insertions(+), 90 deletions(-) rename src/locale/{en-us => en_US}/LC_MESSAGES/django.po (81%) diff --git a/src-ui/angular.json b/src-ui/angular.json index 829a482af..a26dd8713 100644 --- a/src-ui/angular.json +++ b/src-ui/angular.json @@ -18,7 +18,8 @@ "locales": { "de": "src/locale/messages.de.xlf", "nl-NL": "src/locale/messages.nl_NL.xlf", - "fr": "src/locale/messages.fr.xlf" + "fr": "src/locale/messages.fr.xlf", + "en-GB": "src/locale/messages.en_GB.xlf" } }, "architect": { diff --git a/src-ui/src/app/app.module.ts b/src-ui/src/app/app.module.ts index 49e937c0b..54c19a216 100644 --- a/src-ui/src/app/app.module.ts +++ b/src-ui/src/app/app.module.ts @@ -64,10 +64,12 @@ import { CustomDatePipe } from './pipes/custom-date.pipe'; import localeFr from '@angular/common/locales/fr'; import localeNl from '@angular/common/locales/nl'; import localeDe from '@angular/common/locales/de'; +import localeEnGb from '@angular/common/locales/en-GB'; registerLocaleData(localeFr) registerLocaleData(localeNl) registerLocaleData(localeDe) +registerLocaleData(localeEnGb) @NgModule({ declarations: [ diff --git a/src-ui/src/app/components/manage/settings/settings.component.html b/src-ui/src/app/components/manage/settings/settings.component.html index affe2f0b0..aa7ea05d3 100644 --- a/src-ui/src/app/components/manage/settings/settings.component.html +++ b/src-ui/src/app/components/manage/settings/settings.component.html @@ -167,7 +167,7 @@
    • -
    +
    diff --git a/src-ui/src/app/services/settings.service.ts b/src-ui/src/app/services/settings.service.ts index f3e437ada..a145ae5a7 100644 --- a/src-ui/src/app/services/settings.service.ts +++ b/src-ui/src/app/services/settings.service.ts @@ -80,6 +80,7 @@ export class SettingsService { getLanguageOptions(): LanguageOption[] { return [ {code: "en-US", name: $localize`English (US)`, englishName: "English (US)"}, + {code: "en-GB", name: $localize`English (GB)`, englishName: "English (GB)"}, {code: "de", name: $localize`German`, englishName: "German"}, {code: "nl", name: $localize`Dutch`, englishName: "Dutch"}, {code: "fr", name: $localize`French`, englishName: "French"} diff --git a/src-ui/src/locale/messages.en_GB.xlf b/src-ui/src/locale/messages.en_GB.xlf index 4df278c69..49baa77ed 100644 --- a/src-ui/src/locale/messages.en_GB.xlf +++ b/src-ui/src/locale/messages.en_GB.xlf @@ -515,7 +515,7 @@ Color - Color + Colour src/app/components/manage/tag-list/tag-list.component.html 20 diff --git a/src/locale/en_GB/LC_MESSAGES/django.po b/src/locale/en_GB/LC_MESSAGES/django.po index 5f6901f7f..de13e5d34 100644 --- a/src/locale/en_GB/LC_MESSAGES/django.po +++ b/src/locale/en_GB/LC_MESSAGES/django.po @@ -2,10 +2,10 @@ # Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER # This file is distributed under the same license as the PACKAGE package. # FIRST AUTHOR , YEAR. -# +# # Translators: # Ali Bates , 2021 -# +# #, fuzzy msgid "" msgstr "" @@ -76,7 +76,7 @@ msgstr "correspondents" #: documents/models.py:98 msgid "color" -msgstr "color" +msgstr "colour" #: documents/models.py:102 msgid "is inbox tag" diff --git a/src/locale/en-us/LC_MESSAGES/django.po b/src/locale/en_US/LC_MESSAGES/django.po similarity index 81% rename from src/locale/en-us/LC_MESSAGES/django.po rename to src/locale/en_US/LC_MESSAGES/django.po index fdf3fd809..cb31bbc8c 100644 --- a/src/locale/en-us/LC_MESSAGES/django.po +++ b/src/locale/en_US/LC_MESSAGES/django.po @@ -8,7 +8,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-01-28 22:02+0100\n" +"POT-Creation-Date: 2021-02-16 14:52+0100\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" @@ -21,144 +21,144 @@ msgstr "" msgid "Documents" msgstr "" -#: documents/models.py:33 +#: documents/models.py:32 msgid "Any word" msgstr "" -#: documents/models.py:34 +#: documents/models.py:33 msgid "All words" msgstr "" -#: documents/models.py:35 +#: documents/models.py:34 msgid "Exact match" msgstr "" -#: documents/models.py:36 +#: documents/models.py:35 msgid "Regular expression" msgstr "" -#: documents/models.py:37 +#: documents/models.py:36 msgid "Fuzzy word" msgstr "" -#: documents/models.py:38 +#: documents/models.py:37 msgid "Automatic" msgstr "" -#: documents/models.py:42 documents/models.py:352 paperless_mail/models.py:25 +#: documents/models.py:41 documents/models.py:364 paperless_mail/models.py:25 #: paperless_mail/models.py:109 msgid "name" msgstr "" -#: documents/models.py:46 +#: documents/models.py:45 msgid "match" msgstr "" -#: documents/models.py:50 +#: documents/models.py:49 msgid "matching algorithm" msgstr "" -#: documents/models.py:56 +#: documents/models.py:55 msgid "is insensitive" msgstr "" -#: documents/models.py:75 documents/models.py:135 +#: documents/models.py:74 documents/models.py:134 msgid "correspondent" msgstr "" -#: documents/models.py:76 +#: documents/models.py:75 msgid "correspondents" msgstr "" -#: documents/models.py:98 +#: documents/models.py:97 msgid "color" msgstr "" -#: documents/models.py:102 +#: documents/models.py:101 msgid "is inbox tag" msgstr "" -#: documents/models.py:104 +#: documents/models.py:103 msgid "" "Marks this tag as an inbox tag: All newly consumed documents will be tagged " "with inbox tags." msgstr "" -#: documents/models.py:109 +#: documents/models.py:108 msgid "tag" msgstr "" -#: documents/models.py:110 documents/models.py:166 +#: documents/models.py:109 documents/models.py:165 msgid "tags" msgstr "" -#: documents/models.py:116 documents/models.py:148 +#: documents/models.py:115 documents/models.py:147 msgid "document type" msgstr "" -#: documents/models.py:117 +#: documents/models.py:116 msgid "document types" msgstr "" -#: documents/models.py:125 +#: documents/models.py:124 msgid "Unencrypted" msgstr "" -#: documents/models.py:126 +#: documents/models.py:125 msgid "Encrypted with GNU Privacy Guard" msgstr "" -#: documents/models.py:139 +#: documents/models.py:138 msgid "title" msgstr "" -#: documents/models.py:152 +#: documents/models.py:151 msgid "content" msgstr "" -#: documents/models.py:154 +#: documents/models.py:153 msgid "" "The raw, text-only data of the document. This field is primarily used for " "searching." msgstr "" -#: documents/models.py:159 +#: documents/models.py:158 msgid "mime type" msgstr "" -#: documents/models.py:170 +#: documents/models.py:169 msgid "checksum" msgstr "" -#: documents/models.py:174 +#: documents/models.py:173 msgid "The checksum of the original document." msgstr "" -#: documents/models.py:178 +#: documents/models.py:177 msgid "archive checksum" msgstr "" -#: documents/models.py:183 +#: documents/models.py:182 msgid "The checksum of the archived document." msgstr "" -#: documents/models.py:187 documents/models.py:330 +#: documents/models.py:186 documents/models.py:342 msgid "created" msgstr "" -#: documents/models.py:191 +#: documents/models.py:190 msgid "modified" msgstr "" -#: documents/models.py:195 +#: documents/models.py:194 msgid "storage type" msgstr "" -#: documents/models.py:203 +#: documents/models.py:202 msgid "added" msgstr "" -#: documents/models.py:207 +#: documents/models.py:206 msgid "filename" msgstr "" @@ -167,178 +167,186 @@ msgid "Current filename in storage" msgstr "" #: documents/models.py:216 +msgid "archive filename" +msgstr "" + +#: documents/models.py:222 +msgid "Current archive filename in storage" +msgstr "" + +#: documents/models.py:226 msgid "archive serial number" msgstr "" -#: documents/models.py:221 +#: documents/models.py:231 msgid "The position of this document in your physical document archive." msgstr "" -#: documents/models.py:227 +#: documents/models.py:237 msgid "document" msgstr "" -#: documents/models.py:228 +#: documents/models.py:238 msgid "documents" msgstr "" -#: documents/models.py:313 +#: documents/models.py:325 msgid "debug" msgstr "" -#: documents/models.py:314 +#: documents/models.py:326 msgid "information" msgstr "" -#: documents/models.py:315 +#: documents/models.py:327 msgid "warning" msgstr "" -#: documents/models.py:316 +#: documents/models.py:328 msgid "error" msgstr "" -#: documents/models.py:317 +#: documents/models.py:329 msgid "critical" msgstr "" -#: documents/models.py:321 +#: documents/models.py:333 msgid "group" msgstr "" -#: documents/models.py:324 +#: documents/models.py:336 msgid "message" msgstr "" -#: documents/models.py:327 +#: documents/models.py:339 msgid "level" msgstr "" -#: documents/models.py:334 +#: documents/models.py:346 msgid "log" msgstr "" -#: documents/models.py:335 +#: documents/models.py:347 msgid "logs" msgstr "" -#: documents/models.py:346 documents/models.py:396 +#: documents/models.py:358 documents/models.py:408 msgid "saved view" msgstr "" -#: documents/models.py:347 +#: documents/models.py:359 msgid "saved views" msgstr "" -#: documents/models.py:350 +#: documents/models.py:362 msgid "user" msgstr "" -#: documents/models.py:356 +#: documents/models.py:368 msgid "show on dashboard" msgstr "" -#: documents/models.py:359 +#: documents/models.py:371 msgid "show in sidebar" msgstr "" -#: documents/models.py:363 +#: documents/models.py:375 msgid "sort field" msgstr "" -#: documents/models.py:366 +#: documents/models.py:378 msgid "sort reverse" msgstr "" -#: documents/models.py:372 +#: documents/models.py:384 msgid "title contains" msgstr "" -#: documents/models.py:373 +#: documents/models.py:385 msgid "content contains" msgstr "" -#: documents/models.py:374 +#: documents/models.py:386 msgid "ASN is" msgstr "" -#: documents/models.py:375 +#: documents/models.py:387 msgid "correspondent is" msgstr "" -#: documents/models.py:376 +#: documents/models.py:388 msgid "document type is" msgstr "" -#: documents/models.py:377 +#: documents/models.py:389 msgid "is in inbox" msgstr "" -#: documents/models.py:378 +#: documents/models.py:390 msgid "has tag" msgstr "" -#: documents/models.py:379 +#: documents/models.py:391 msgid "has any tag" msgstr "" -#: documents/models.py:380 +#: documents/models.py:392 msgid "created before" msgstr "" -#: documents/models.py:381 +#: documents/models.py:393 msgid "created after" msgstr "" -#: documents/models.py:382 +#: documents/models.py:394 msgid "created year is" msgstr "" -#: documents/models.py:383 +#: documents/models.py:395 msgid "created month is" msgstr "" -#: documents/models.py:384 +#: documents/models.py:396 msgid "created day is" msgstr "" -#: documents/models.py:385 +#: documents/models.py:397 msgid "added before" msgstr "" -#: documents/models.py:386 +#: documents/models.py:398 msgid "added after" msgstr "" -#: documents/models.py:387 +#: documents/models.py:399 msgid "modified before" msgstr "" -#: documents/models.py:388 +#: documents/models.py:400 msgid "modified after" msgstr "" -#: documents/models.py:389 +#: documents/models.py:401 msgid "does not have tag" msgstr "" -#: documents/models.py:400 +#: documents/models.py:412 msgid "rule type" msgstr "" -#: documents/models.py:404 +#: documents/models.py:416 msgid "value" msgstr "" -#: documents/models.py:410 +#: documents/models.py:422 msgid "filter rule" msgstr "" -#: documents/models.py:411 +#: documents/models.py:423 msgid "filter rules" msgstr "" -#: documents/serialisers.py:383 +#: documents/serialisers.py:370 #, python-format msgid "File type %(type)s not supported" msgstr "" @@ -383,19 +391,23 @@ msgstr "" msgid "Sign in" msgstr "" -#: paperless/settings.py:286 -msgid "English" +#: paperless/settings.py:291 +msgid "English (US)" msgstr "" -#: paperless/settings.py:287 +#: paperless/settings.py:292 +msgid "English (GB)" +msgstr "" + +#: paperless/settings.py:293 msgid "German" msgstr "" -#: paperless/settings.py:288 +#: paperless/settings.py:294 msgid "Dutch" msgstr "" -#: paperless/settings.py:289 +#: paperless/settings.py:295 msgid "French" msgstr "" diff --git a/src/paperless/settings.py b/src/paperless/settings.py index 8d14262d1..42ddec88d 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -288,7 +288,8 @@ if os.getenv("PAPERLESS_DBHOST"): LANGUAGE_CODE = 'en-us' LANGUAGES = [ - ("en-us", _("English")), + ("en-us", _("English (US)")), + ("en-gb", _("English (GB)")), ("de", _("German")), ("nl-nl", _("Dutch")), ("fr", _("French")) From 1c81d8801355e149b1b6631e93ea7107f66561ed Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 17 Feb 2021 12:15:22 +0100 Subject: [PATCH 137/225] add support for iso 8601 date display --- .../manage/settings/settings.component.html | 2 +- .../manage/settings/settings.component.ts | 7 +++++-- src-ui/src/app/pipes/custom-date.pipe.ts | 17 ++++++++++++++--- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/src-ui/src/app/components/manage/settings/settings.component.html b/src-ui/src/app/components/manage/settings/settings.component.html index aa7ea05d3..3674bbcb9 100644 --- a/src-ui/src/app/components/manage/settings/settings.component.html +++ b/src-ui/src/app/components/manage/settings/settings.component.html @@ -34,7 +34,7 @@
    diff --git a/src-ui/src/app/components/manage/settings/settings.component.ts b/src-ui/src/app/components/manage/settings/settings.component.ts index 47c7b8d7b..15cfb8a19 100644 --- a/src-ui/src/app/components/manage/settings/settings.component.ts +++ b/src-ui/src/app/components/manage/settings/settings.component.ts @@ -35,7 +35,7 @@ export class SettingsComponent implements OnInit { savedViews: PaperlessSavedView[] get computedDateLocale(): string { - return this.settingsForm.value.dateLocale || this.settingsForm.value.displayLanguage + return this.settingsForm.value.dateLocale || this.settingsForm.value.displayLanguage || this.currentLocale } constructor( @@ -92,7 +92,10 @@ export class SettingsComponent implements OnInit { } get dateLocaleOptions(): LanguageOption[] { - return [{code: "", name: $localize`Use date format of display language`}].concat(this.settings.getLanguageOptions()) + return [ + {code: "", name: $localize`Use date format of display language`}, + {code: "iso-8601", name: $localize`ISO 8601`} + ].concat(this.settings.getLanguageOptions()) } get today() { diff --git a/src-ui/src/app/pipes/custom-date.pipe.ts b/src-ui/src/app/pipes/custom-date.pipe.ts index 19989fbdd..cf5a931cb 100644 --- a/src-ui/src/app/pipes/custom-date.pipe.ts +++ b/src-ui/src/app/pipes/custom-date.pipe.ts @@ -2,18 +2,29 @@ import { DatePipe } from '@angular/common'; import { Inject, LOCALE_ID, Pipe, PipeTransform } from '@angular/core'; import { SettingsService, SETTINGS_KEYS } from '../services/settings.service'; +const FORMAT_TO_ISO_FORMAT = { + "longDate": "y-MM-dd", + "mediumDate": "yy-MM-dd", + "shortDate": "yy-MM-dd" +} + @Pipe({ name: 'customDate' }) export class CustomDatePipe extends DatePipe implements PipeTransform { constructor(@Inject(LOCALE_ID) locale: string, private settings: SettingsService) { - super(settings.get(SETTINGS_KEYS.DATE_LOCALE) || locale) - + super(locale) } transform(value: any, format?: string, timezone?: string, locale?: string): string | null { - return super.transform(value, format || this.settings.get(SETTINGS_KEYS.DATE_FORMAT), timezone, locale) + let l = locale || this.settings.get(SETTINGS_KEYS.DATE_LOCALE) + let f = format || this.settings.get(SETTINGS_KEYS.DATE_FORMAT) + if (l == "iso-8601") { + return super.transform(value, FORMAT_TO_ISO_FORMAT[f], timezone) + } else { + return super.transform(value, format || this.settings.get(SETTINGS_KEYS.DATE_FORMAT), timezone, locale) + } } } From a1162d6d5ab8caaf9cc8d7318c40e4cdc08c3e4f Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 17 Feb 2021 12:25:34 +0100 Subject: [PATCH 138/225] update requirements --- Pipfile.lock | 18 +++++++++--------- requirements.txt | 4 ++-- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/Pipfile.lock b/Pipfile.lock index fee6359da..f75949d20 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -591,11 +591,11 @@ }, "ocrmypdf": { "hashes": [ - "sha256:a54634d017a2f44aa2115b0b6ae5aa41a7cec018f5c53d16ad3abec1e70b3db7", - "sha256:d0e2da48d4abd90f48f0937b2cd4ba57503b56c603f5e3aa91e20e3b21a036cd" + "sha256:0f624456a50be0b0bc8c0b59704d159f637616c093a1cabe8bb383706561bcf7", + "sha256:b829ad640a6160423162012e094ee2f7cd074ec99efadd7f7486954ec9182985" ], "index": "pypi", - "version": "==11.6.0" + "version": "==11.6.2" }, "pathvalidate": { "hashes": [ @@ -841,11 +841,11 @@ }, "python-magic": { "hashes": [ - "sha256:356efa93c8899047d1eb7d3eb91e871ba2f5b1376edbaf4cc305e3c872207355", - "sha256:b757db2a5289ea3f1ced9e60f072965243ea43a2221430048fd8cacab17be0ce" + "sha256:8551e804c09a3398790bd9e392acb26554ae2609f29c72abb0b9dee9a5571eae", + "sha256:ca884349f2c92ce830e3f498c5b7c7051fe2942c3ee4332f65213b8ebff15a62" ], "index": "pypi", - "version": "==0.4.18" + "version": "==0.4.22" }, "pytz": { "hashes": [ @@ -1837,11 +1837,11 @@ }, "tox": { "hashes": [ - "sha256:65d0e90ceb816638a50d64f4b47b11da767b284c0addda2294cb3cd69bd72425", - "sha256:cf7fef81a3a2434df4d7af2a6d1bf606d2970220addfbe7dea2615bd4bb2c252" + "sha256:89afa9c59c04beb55eda789c7a65feb1a70fde117f85f1bd1c27c66758456e60", + "sha256:ed1e650cf6368bcbc4a071eeeba363c480920e0ed8a9ad1793c7caaa5ad33d49" ], "index": "pypi", - "version": "==3.21.4" + "version": "==3.22.0" }, "urllib3": { "hashes": [ diff --git a/requirements.txt b/requirements.txt index 0a0ae75c0..7100ba953 100644 --- a/requirements.txt +++ b/requirements.txt @@ -53,7 +53,7 @@ langdetect==1.0.8 lxml==4.6.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' msgpack==1.0.2 numpy==1.19.5 -ocrmypdf==11.6.0 +ocrmypdf==11.6.2 pathvalidate==2.3.2 pdfminer.six==20201018; python_version >= '3.4' pdftotext==2.1.5 @@ -71,7 +71,7 @@ python-dateutil==2.8.1 python-dotenv==0.15.0 python-gnupg==0.4.6 python-levenshtein==0.12.2 -python-magic==0.4.18 +python-magic==0.4.22 pytz==2021.1 pyyaml==5.4.1 redis==3.5.3 From 08a44cf468631c029c5469205f860adc89139e40 Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 17 Feb 2021 12:31:19 +0100 Subject: [PATCH 139/225] changelog and version --- docs/changelog.rst | 19 +++++++++++++------ src-ui/src/environments/environment.prod.ts | 2 +- src/paperless/version.py | 2 +- 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 8b87e642e..ad45becab 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -5,15 +5,12 @@ Changelog ********* -paperless-ng 1.1.3 +paperless-ng 1.1.4 ################## -* Added a docker-specific configuration option to adjust the number of - worker processes of the web server. See :ref:`configuration-docker`. +* Added English (GB) locale. -* Some more memory usage optimizations. - -* Don't show inbox statistics if no inbox tag is defined. +* Added ISO-8601 date display option. .. note:: @@ -26,6 +23,16 @@ paperless-ng 1.1.3 If using docker, this does not affect you. +paperless-ng 1.1.3 +################## + +* Added a docker-specific configuration option to adjust the number of + worker processes of the web server. See :ref:`configuration-docker`. + +* Some more memory usage optimizations. + +* Don't show inbox statistics if no inbox tag is defined. + paperless-ng 1.1.2 ################## diff --git a/src-ui/src/environments/environment.prod.ts b/src-ui/src/environments/environment.prod.ts index b2ffebc09..6df8ec565 100644 --- a/src-ui/src/environments/environment.prod.ts +++ b/src-ui/src/environments/environment.prod.ts @@ -2,7 +2,7 @@ export const environment = { production: true, apiBaseUrl: "/api/", appTitle: "Paperless-ng", - version: "1.1.3", + version: "1.1.4", webSocketHost: window.location.host, webSocketProtocol: (window.location.protocol == "https:" ? "wss:" : "ws:") }; diff --git a/src/paperless/version.py b/src/paperless/version.py index 08f427169..f336d581e 100644 --- a/src/paperless/version.py +++ b/src/paperless/version.py @@ -1 +1 @@ -__version__ = (1, 1, 3) +__version__ = (1, 1, 4) From b7f1b9f8ad0f3a6e8ee11cbe4a20732353913245 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 17 Feb 2021 11:50:01 +0000 Subject: [PATCH 140/225] Apply translations in de translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'de' language. --- src/locale/de/LC_MESSAGES/django.po | 182 +++++++++++++++------------- 1 file changed, 97 insertions(+), 85 deletions(-) diff --git a/src/locale/de/LC_MESSAGES/django.po b/src/locale/de/LC_MESSAGES/django.po index c53985275..358c9b721 100644 --- a/src/locale/de/LC_MESSAGES/django.po +++ b/src/locale/de/LC_MESSAGES/django.po @@ -11,8 +11,8 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-01-28 22:02+0100\n" -"PO-Revision-Date: 2020-12-30 19:27+0000\n" +"POT-Creation-Date: 2021-02-16 14:52+0100\n" +"PO-Revision-Date: 2021-02-16 18:37+0000\n" "Last-Translator: Jonas Winkler, 2021\n" "Language-Team: German (https://www.transifex.com/paperless/teams/115905/de/)\n" "MIME-Version: 1.0\n" @@ -25,64 +25,64 @@ msgstr "" msgid "Documents" msgstr "Dokumente" -#: documents/models.py:33 +#: documents/models.py:32 msgid "Any word" msgstr "Irgendein Wort" -#: documents/models.py:34 +#: documents/models.py:33 msgid "All words" msgstr "Alle Wörter" -#: documents/models.py:35 +#: documents/models.py:34 msgid "Exact match" msgstr "Exakte Übereinstimmung" -#: documents/models.py:36 +#: documents/models.py:35 msgid "Regular expression" msgstr "Regulärer Ausdruck" -#: documents/models.py:37 +#: documents/models.py:36 msgid "Fuzzy word" msgstr "Ungenaues Wort" -#: documents/models.py:38 +#: documents/models.py:37 msgid "Automatic" msgstr "Automatisch" -#: documents/models.py:42 documents/models.py:352 paperless_mail/models.py:25 +#: documents/models.py:41 documents/models.py:364 paperless_mail/models.py:25 #: paperless_mail/models.py:109 msgid "name" msgstr "Name" -#: documents/models.py:46 +#: documents/models.py:45 msgid "match" msgstr "Zuweisungsmuster" -#: documents/models.py:50 +#: documents/models.py:49 msgid "matching algorithm" msgstr "Zuweisungsalgorithmus" -#: documents/models.py:56 +#: documents/models.py:55 msgid "is insensitive" msgstr "Groß-/Kleinschreibung irrelevant" -#: documents/models.py:75 documents/models.py:135 +#: documents/models.py:74 documents/models.py:134 msgid "correspondent" msgstr "Korrespondent" -#: documents/models.py:76 +#: documents/models.py:75 msgid "correspondents" msgstr "Korrespondenten" -#: documents/models.py:98 +#: documents/models.py:97 msgid "color" msgstr "Farbe" -#: documents/models.py:102 +#: documents/models.py:101 msgid "is inbox tag" msgstr "Posteingangs-Tag" -#: documents/models.py:104 +#: documents/models.py:103 msgid "" "Marks this tag as an inbox tag: All newly consumed documents will be tagged " "with inbox tags." @@ -90,39 +90,39 @@ msgstr "" "Markiert das Tag als Posteingangs-Tag. Neue Dokumente werden immer mit " "diesem Tag versehen." -#: documents/models.py:109 +#: documents/models.py:108 msgid "tag" msgstr "Tag" -#: documents/models.py:110 documents/models.py:166 +#: documents/models.py:109 documents/models.py:165 msgid "tags" msgstr "Tags" -#: documents/models.py:116 documents/models.py:148 +#: documents/models.py:115 documents/models.py:147 msgid "document type" msgstr "Dokumenttyp" -#: documents/models.py:117 +#: documents/models.py:116 msgid "document types" msgstr "Dokumenttypen" -#: documents/models.py:125 +#: documents/models.py:124 msgid "Unencrypted" msgstr "Nicht verschlüsselt" -#: documents/models.py:126 +#: documents/models.py:125 msgid "Encrypted with GNU Privacy Guard" msgstr "Verschlüsselt mit GNU Privacy Guard" -#: documents/models.py:139 +#: documents/models.py:138 msgid "title" msgstr "Titel" -#: documents/models.py:152 +#: documents/models.py:151 msgid "content" msgstr "Inhalt" -#: documents/models.py:154 +#: documents/models.py:153 msgid "" "The raw, text-only data of the document. This field is primarily used for " "searching." @@ -130,43 +130,43 @@ msgstr "" "Der Inhalt des Dokuments in Textform. Dieses Feld wird primär für die Suche " "verwendet." -#: documents/models.py:159 +#: documents/models.py:158 msgid "mime type" msgstr "MIME-Typ" -#: documents/models.py:170 +#: documents/models.py:169 msgid "checksum" msgstr "Prüfsumme" -#: documents/models.py:174 +#: documents/models.py:173 msgid "The checksum of the original document." msgstr "Die Prüfsumme des originalen Dokuments." -#: documents/models.py:178 +#: documents/models.py:177 msgid "archive checksum" msgstr "Archiv-Prüfsumme" -#: documents/models.py:183 +#: documents/models.py:182 msgid "The checksum of the archived document." msgstr "Die Prüfsumme des archivierten Dokuments." -#: documents/models.py:187 documents/models.py:330 +#: documents/models.py:186 documents/models.py:342 msgid "created" msgstr "Ausgestellt" -#: documents/models.py:191 +#: documents/models.py:190 msgid "modified" msgstr "Geändert" -#: documents/models.py:195 +#: documents/models.py:194 msgid "storage type" msgstr "Speichertyp" -#: documents/models.py:203 +#: documents/models.py:202 msgid "added" msgstr "Hinzugefügt" -#: documents/models.py:207 +#: documents/models.py:206 msgid "filename" msgstr "Dateiname" @@ -175,178 +175,186 @@ msgid "Current filename in storage" msgstr "Aktueller Dateiname im Datenspeicher" #: documents/models.py:216 +msgid "archive filename" +msgstr "Archiv-Dateiname" + +#: documents/models.py:222 +msgid "Current archive filename in storage" +msgstr "Aktueller Dateiname im Archiv" + +#: documents/models.py:226 msgid "archive serial number" msgstr "Archiv-Seriennummer" -#: documents/models.py:221 +#: documents/models.py:231 msgid "The position of this document in your physical document archive." msgstr "Die Position dieses Dokuments in Ihrem physischen Dokumentenarchiv." -#: documents/models.py:227 +#: documents/models.py:237 msgid "document" msgstr "Dokument" -#: documents/models.py:228 +#: documents/models.py:238 msgid "documents" msgstr "Dokumente" -#: documents/models.py:313 +#: documents/models.py:325 msgid "debug" msgstr "Debug" -#: documents/models.py:314 +#: documents/models.py:326 msgid "information" msgstr "Information" -#: documents/models.py:315 +#: documents/models.py:327 msgid "warning" msgstr "Warnung" -#: documents/models.py:316 +#: documents/models.py:328 msgid "error" msgstr "Fehler" -#: documents/models.py:317 +#: documents/models.py:329 msgid "critical" msgstr "Kritisch" -#: documents/models.py:321 +#: documents/models.py:333 msgid "group" msgstr "Gruppe" -#: documents/models.py:324 +#: documents/models.py:336 msgid "message" msgstr "Nachricht" -#: documents/models.py:327 +#: documents/models.py:339 msgid "level" msgstr "Level" -#: documents/models.py:334 +#: documents/models.py:346 msgid "log" msgstr "Protokoll" -#: documents/models.py:335 +#: documents/models.py:347 msgid "logs" msgstr "Protokoll" -#: documents/models.py:346 documents/models.py:396 +#: documents/models.py:358 documents/models.py:408 msgid "saved view" msgstr "Gespeicherte Ansicht" -#: documents/models.py:347 +#: documents/models.py:359 msgid "saved views" msgstr "Gespeicherte Ansichten" -#: documents/models.py:350 +#: documents/models.py:362 msgid "user" msgstr "Benutzer" -#: documents/models.py:356 +#: documents/models.py:368 msgid "show on dashboard" msgstr "Auf Startseite zeigen" -#: documents/models.py:359 +#: documents/models.py:371 msgid "show in sidebar" msgstr "In Seitenleiste zeigen" -#: documents/models.py:363 +#: documents/models.py:375 msgid "sort field" msgstr "Sortierfeld" -#: documents/models.py:366 +#: documents/models.py:378 msgid "sort reverse" msgstr "Umgekehrte Sortierung" -#: documents/models.py:372 +#: documents/models.py:384 msgid "title contains" msgstr "Titel enthält" -#: documents/models.py:373 +#: documents/models.py:385 msgid "content contains" msgstr "Inhalt enthält" -#: documents/models.py:374 +#: documents/models.py:386 msgid "ASN is" msgstr "ASN ist" -#: documents/models.py:375 +#: documents/models.py:387 msgid "correspondent is" msgstr "Korrespondent ist" -#: documents/models.py:376 +#: documents/models.py:388 msgid "document type is" msgstr "Dokumenttyp ist" -#: documents/models.py:377 +#: documents/models.py:389 msgid "is in inbox" msgstr "Ist im Posteingang" -#: documents/models.py:378 +#: documents/models.py:390 msgid "has tag" msgstr "Hat Tag" -#: documents/models.py:379 +#: documents/models.py:391 msgid "has any tag" msgstr "Hat irgendein Tag" -#: documents/models.py:380 +#: documents/models.py:392 msgid "created before" msgstr "Ausgestellt vor" -#: documents/models.py:381 +#: documents/models.py:393 msgid "created after" msgstr "Ausgestellt nach" -#: documents/models.py:382 +#: documents/models.py:394 msgid "created year is" msgstr "Ausgestellt im Jahr" -#: documents/models.py:383 +#: documents/models.py:395 msgid "created month is" msgstr "Ausgestellt im Monat" -#: documents/models.py:384 +#: documents/models.py:396 msgid "created day is" msgstr "Ausgestellt am Tag" -#: documents/models.py:385 +#: documents/models.py:397 msgid "added before" msgstr "Hinzugefügt vor" -#: documents/models.py:386 +#: documents/models.py:398 msgid "added after" msgstr "Hinzugefügt nach" -#: documents/models.py:387 +#: documents/models.py:399 msgid "modified before" msgstr "Geändert vor" -#: documents/models.py:388 +#: documents/models.py:400 msgid "modified after" msgstr "Geändert nach" -#: documents/models.py:389 +#: documents/models.py:401 msgid "does not have tag" msgstr "Hat nicht folgendes Tag" -#: documents/models.py:400 +#: documents/models.py:412 msgid "rule type" msgstr "Regeltyp" -#: documents/models.py:404 +#: documents/models.py:416 msgid "value" msgstr "Wert" -#: documents/models.py:410 +#: documents/models.py:422 msgid "filter rule" msgstr "Filterregel" -#: documents/models.py:411 +#: documents/models.py:423 msgid "filter rules" msgstr "Filterregeln" -#: documents/serialisers.py:383 +#: documents/serialisers.py:370 #, python-format msgid "File type %(type)s not supported" msgstr "Dateityp %(type)s nicht unterstützt" @@ -393,19 +401,23 @@ msgstr "Passwort" msgid "Sign in" msgstr "Anmelden" -#: paperless/settings.py:286 -msgid "English" -msgstr "Englisch" +#: paperless/settings.py:291 +msgid "English (US)" +msgstr "Englisch (US)" -#: paperless/settings.py:287 +#: paperless/settings.py:292 +msgid "English (GB)" +msgstr "Englisch (UK)" + +#: paperless/settings.py:293 msgid "German" msgstr "Deutsch" -#: paperless/settings.py:288 +#: paperless/settings.py:294 msgid "Dutch" msgstr "Niederländisch" -#: paperless/settings.py:289 +#: paperless/settings.py:295 msgid "French" msgstr "Französisch" From 6ff99945f3de64859517c204f13fc0409cf69020 Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 17 Feb 2021 13:18:19 +0100 Subject: [PATCH 141/225] Update Crowdin configuration file --- crowdin.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crowdin.yml b/crowdin.yml index f5a7765dd..a121e80b8 100644 --- a/crowdin.yml +++ b/crowdin.yml @@ -1,5 +1,5 @@ files: - - source: /src/locale/en-us/LC_MESSAGES/django.po + - source: /src/locale/en_US/LC_MESSAGES/django.po translation: /src/locale/%two_letters_code%/LC_MESSAGES/django.po - source: /src-ui/messages.xlf translation: /src-ui/src/locale/messages.%two_letters_code%.xlf From 4e4d6e806cad6cd871ab732d9c50396f0aa3b567 Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 17 Feb 2021 13:22:45 +0100 Subject: [PATCH 142/225] Update Crowdin configuration file --- crowdin.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/crowdin.yml b/crowdin.yml index a121e80b8..0e6a46a31 100644 --- a/crowdin.yml +++ b/crowdin.yml @@ -1,5 +1,5 @@ files: - source: /src/locale/en_US/LC_MESSAGES/django.po - translation: /src/locale/%two_letters_code%/LC_MESSAGES/django.po + translation: /src/locale/%locale_with_underscore%/LC_MESSAGES/django.po - source: /src-ui/messages.xlf - translation: /src-ui/src/locale/messages.%two_letters_code%.xlf + translation: /src-ui/src/locale/messages.%locale_with_underscore%.xlf From 58df3d5767105944937eb0792b72204cd137805e Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 17 Feb 2021 12:57:48 +0000 Subject: [PATCH 143/225] Apply translations in nl_NL translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'nl_NL' language. --- src/locale/nl_NL/LC_MESSAGES/django.po | 186 +++++++++++++------------ 1 file changed, 99 insertions(+), 87 deletions(-) diff --git a/src/locale/nl_NL/LC_MESSAGES/django.po b/src/locale/nl_NL/LC_MESSAGES/django.po index 63391da15..eefba7fbb 100644 --- a/src/locale/nl_NL/LC_MESSAGES/django.po +++ b/src/locale/nl_NL/LC_MESSAGES/django.po @@ -4,17 +4,17 @@ # FIRST AUTHOR , YEAR. # # Translators: +# Jonas Winkler, 2021 # Jo Vandeginste , 2021 -# Ben , 2021 # #, fuzzy msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-01-28 22:02+0100\n" -"PO-Revision-Date: 2020-12-30 19:27+0000\n" -"Last-Translator: Ben , 2021\n" +"POT-Creation-Date: 2021-02-16 14:52+0100\n" +"PO-Revision-Date: 2021-02-16 18:37+0000\n" +"Last-Translator: Jo Vandeginste , 2021\n" "Language-Team: Dutch (Netherlands) (https://www.transifex.com/paperless/teams/115905/nl_NL/)\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" @@ -26,64 +26,64 @@ msgstr "" msgid "Documents" msgstr "Documenten" -#: documents/models.py:33 +#: documents/models.py:32 msgid "Any word" msgstr "Eender welk woord" -#: documents/models.py:34 +#: documents/models.py:33 msgid "All words" msgstr "Alle woorden" -#: documents/models.py:35 +#: documents/models.py:34 msgid "Exact match" msgstr "Exacte overeenkomst" -#: documents/models.py:36 +#: documents/models.py:35 msgid "Regular expression" msgstr "Reguliere expressie" -#: documents/models.py:37 +#: documents/models.py:36 msgid "Fuzzy word" msgstr "Gelijkaardig woord" -#: documents/models.py:38 +#: documents/models.py:37 msgid "Automatic" msgstr "Automatisch" -#: documents/models.py:42 documents/models.py:352 paperless_mail/models.py:25 +#: documents/models.py:41 documents/models.py:364 paperless_mail/models.py:25 #: paperless_mail/models.py:109 msgid "name" msgstr "naam" -#: documents/models.py:46 +#: documents/models.py:45 msgid "match" msgstr "Overeenkomst" -#: documents/models.py:50 +#: documents/models.py:49 msgid "matching algorithm" msgstr "Algoritme voor het bepalen van de overeenkomst" -#: documents/models.py:56 +#: documents/models.py:55 msgid "is insensitive" msgstr "is niet hoofdlettergevoelig" -#: documents/models.py:75 documents/models.py:135 +#: documents/models.py:74 documents/models.py:134 msgid "correspondent" msgstr "correspondent" -#: documents/models.py:76 +#: documents/models.py:75 msgid "correspondents" msgstr "correspondenten" -#: documents/models.py:98 +#: documents/models.py:97 msgid "color" msgstr "Kleur" -#: documents/models.py:102 +#: documents/models.py:101 msgid "is inbox tag" msgstr "is \"Postvak in\"-etiket" -#: documents/models.py:104 +#: documents/models.py:103 msgid "" "Marks this tag as an inbox tag: All newly consumed documents will be tagged " "with inbox tags." @@ -91,39 +91,39 @@ msgstr "" "Markeer dit etiket als een \"Postvak in\"-etiket: alle nieuw verwerkte " "documenten krijgen de \"Postvak in\"-etiketten." -#: documents/models.py:109 +#: documents/models.py:108 msgid "tag" msgstr "etiket" -#: documents/models.py:110 documents/models.py:166 +#: documents/models.py:109 documents/models.py:165 msgid "tags" msgstr "etiketten" -#: documents/models.py:116 documents/models.py:148 +#: documents/models.py:115 documents/models.py:147 msgid "document type" msgstr "documenttype" -#: documents/models.py:117 +#: documents/models.py:116 msgid "document types" msgstr "documenttypen" -#: documents/models.py:125 +#: documents/models.py:124 msgid "Unencrypted" msgstr "Niet versleuteld" -#: documents/models.py:126 +#: documents/models.py:125 msgid "Encrypted with GNU Privacy Guard" msgstr "Versleuteld met GNU Privacy Guard" -#: documents/models.py:139 +#: documents/models.py:138 msgid "title" msgstr "titel" -#: documents/models.py:152 +#: documents/models.py:151 msgid "content" msgstr "inhoud" -#: documents/models.py:154 +#: documents/models.py:153 msgid "" "The raw, text-only data of the document. This field is primarily used for " "searching." @@ -131,43 +131,43 @@ msgstr "" "De onbewerkte gegevens van het document. Dit veld wordt voornamelijk " "gebruikt om te zoeken." -#: documents/models.py:159 +#: documents/models.py:158 msgid "mime type" msgstr "mimetype" -#: documents/models.py:170 +#: documents/models.py:169 msgid "checksum" msgstr "checksum" -#: documents/models.py:174 +#: documents/models.py:173 msgid "The checksum of the original document." msgstr "Het controlecijfer van het originele document." -#: documents/models.py:178 +#: documents/models.py:177 msgid "archive checksum" msgstr "archief checksum" -#: documents/models.py:183 +#: documents/models.py:182 msgid "The checksum of the archived document." msgstr "De checksum van het gearchiveerde document." -#: documents/models.py:187 documents/models.py:330 +#: documents/models.py:186 documents/models.py:342 msgid "created" msgstr "aangemaakt" -#: documents/models.py:191 +#: documents/models.py:190 msgid "modified" msgstr "gewijzigd" -#: documents/models.py:195 +#: documents/models.py:194 msgid "storage type" msgstr "type opslag" -#: documents/models.py:203 +#: documents/models.py:202 msgid "added" msgstr "toegevoegd" -#: documents/models.py:207 +#: documents/models.py:206 msgid "filename" msgstr "bestandsnaam" @@ -176,178 +176,186 @@ msgid "Current filename in storage" msgstr "Huidige bestandsnaam in opslag" #: documents/models.py:216 +msgid "archive filename" +msgstr "Bestandsnaam in archief" + +#: documents/models.py:222 +msgid "Current archive filename in storage" +msgstr "Huidige bestandsnaam in archief" + +#: documents/models.py:226 msgid "archive serial number" msgstr "serienummer in archief" -#: documents/models.py:221 +#: documents/models.py:231 msgid "The position of this document in your physical document archive." msgstr "De positie van dit document in je fysieke documentenarchief." -#: documents/models.py:227 +#: documents/models.py:237 msgid "document" msgstr "document" -#: documents/models.py:228 +#: documents/models.py:238 msgid "documents" msgstr "documenten" -#: documents/models.py:313 +#: documents/models.py:325 msgid "debug" msgstr "debug" -#: documents/models.py:314 +#: documents/models.py:326 msgid "information" msgstr "informatie" -#: documents/models.py:315 +#: documents/models.py:327 msgid "warning" msgstr "waarschuwing" -#: documents/models.py:316 +#: documents/models.py:328 msgid "error" msgstr "fout" -#: documents/models.py:317 +#: documents/models.py:329 msgid "critical" msgstr "kritisch" -#: documents/models.py:321 +#: documents/models.py:333 msgid "group" msgstr "groep" -#: documents/models.py:324 +#: documents/models.py:336 msgid "message" msgstr "bericht" -#: documents/models.py:327 +#: documents/models.py:339 msgid "level" msgstr "niveau" -#: documents/models.py:334 +#: documents/models.py:346 msgid "log" msgstr "bericht" -#: documents/models.py:335 +#: documents/models.py:347 msgid "logs" msgstr "berichten" -#: documents/models.py:346 documents/models.py:396 +#: documents/models.py:358 documents/models.py:408 msgid "saved view" msgstr "opgeslagen view" -#: documents/models.py:347 +#: documents/models.py:359 msgid "saved views" msgstr "opgeslagen views" -#: documents/models.py:350 +#: documents/models.py:362 msgid "user" msgstr "gebruiker" -#: documents/models.py:356 +#: documents/models.py:368 msgid "show on dashboard" msgstr "weergeven op dashboard" -#: documents/models.py:359 +#: documents/models.py:371 msgid "show in sidebar" msgstr "weergeven in zijbalk" -#: documents/models.py:363 +#: documents/models.py:375 msgid "sort field" msgstr "sorteerveld" -#: documents/models.py:366 +#: documents/models.py:378 msgid "sort reverse" msgstr "omgekeerd sorteren" -#: documents/models.py:372 +#: documents/models.py:384 msgid "title contains" msgstr "titel bevat" -#: documents/models.py:373 +#: documents/models.py:385 msgid "content contains" msgstr "inhoud bevat" -#: documents/models.py:374 +#: documents/models.py:386 msgid "ASN is" msgstr "ASN is" -#: documents/models.py:375 +#: documents/models.py:387 msgid "correspondent is" msgstr "correspondent is" -#: documents/models.py:376 +#: documents/models.py:388 msgid "document type is" msgstr "documenttype is" -#: documents/models.py:377 +#: documents/models.py:389 msgid "is in inbox" msgstr "zit in \"Postvak in\"" -#: documents/models.py:378 +#: documents/models.py:390 msgid "has tag" msgstr "heeft etiket" -#: documents/models.py:379 +#: documents/models.py:391 msgid "has any tag" msgstr "heeft één van de etiketten" -#: documents/models.py:380 +#: documents/models.py:392 msgid "created before" msgstr "aangemaakt voor" -#: documents/models.py:381 +#: documents/models.py:393 msgid "created after" msgstr "aangemaakt na" -#: documents/models.py:382 +#: documents/models.py:394 msgid "created year is" msgstr "aangemaakt jaar is" -#: documents/models.py:383 +#: documents/models.py:395 msgid "created month is" msgstr "aangemaakte maand is" -#: documents/models.py:384 +#: documents/models.py:396 msgid "created day is" msgstr "aangemaakte dag is" -#: documents/models.py:385 +#: documents/models.py:397 msgid "added before" msgstr "toegevoegd voor" -#: documents/models.py:386 +#: documents/models.py:398 msgid "added after" msgstr "toegevoegd na" -#: documents/models.py:387 +#: documents/models.py:399 msgid "modified before" msgstr "gewijzigd voor" -#: documents/models.py:388 +#: documents/models.py:400 msgid "modified after" msgstr "gewijzigd na" -#: documents/models.py:389 +#: documents/models.py:401 msgid "does not have tag" msgstr "heeft geen etiket" -#: documents/models.py:400 +#: documents/models.py:412 msgid "rule type" msgstr "type regel" -#: documents/models.py:404 +#: documents/models.py:416 msgid "value" msgstr "waarde" -#: documents/models.py:410 +#: documents/models.py:422 msgid "filter rule" msgstr "filterregel" -#: documents/models.py:411 +#: documents/models.py:423 msgid "filter rules" msgstr "filterregels" -#: documents/serialisers.py:383 +#: documents/serialisers.py:370 #, python-format msgid "File type %(type)s not supported" msgstr "Bestandstype %(type)s niet ondersteund" @@ -392,19 +400,23 @@ msgstr "Wachtwoord" msgid "Sign in" msgstr "Aanmelden" -#: paperless/settings.py:286 -msgid "English" -msgstr "Engels" +#: paperless/settings.py:291 +msgid "English (US)" +msgstr "Engels (US)" -#: paperless/settings.py:287 +#: paperless/settings.py:292 +msgid "English (GB)" +msgstr "Engels (Brits)" + +#: paperless/settings.py:293 msgid "German" msgstr "Duits" -#: paperless/settings.py:288 +#: paperless/settings.py:294 msgid "Dutch" msgstr "Nederlands" -#: paperless/settings.py:289 +#: paperless/settings.py:295 msgid "French" msgstr "Frans" From 0d78e58d77907ec9037bd459608602f787fddcda Mon Sep 17 00:00:00 2001 From: jonaswinkler Date: Wed, 17 Feb 2021 14:26:06 +0100 Subject: [PATCH 144/225] fixed paperless not properly selecting en-gb --- src-ui/src/app/services/settings.service.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src-ui/src/app/services/settings.service.ts b/src-ui/src/app/services/settings.service.ts index a145ae5a7..d511fa65f 100644 --- a/src-ui/src/app/services/settings.service.ts +++ b/src-ui/src/app/services/settings.service.ts @@ -79,8 +79,8 @@ export class SettingsService { getLanguageOptions(): LanguageOption[] { return [ - {code: "en-US", name: $localize`English (US)`, englishName: "English (US)"}, - {code: "en-GB", name: $localize`English (GB)`, englishName: "English (GB)"}, + {code: "en-us", name: $localize`English (US)`, englishName: "English (US)"}, + {code: "en-gb", name: $localize`English (GB)`, englishName: "English (GB)"}, {code: "de", name: $localize`German`, englishName: "German"}, {code: "nl", name: $localize`Dutch`, englishName: "Dutch"}, {code: "fr", name: $localize`French`, englishName: "French"} From 7bc557a9999e86e6fdcb0627372b50721325f28a Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 17 Feb 2021 13:36:15 +0000 Subject: [PATCH 145/225] Apply translations in en_GB translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'en_GB' language. --- src/locale/en_GB/LC_MESSAGES/django.po | 191 +++++++++++++------------ 1 file changed, 102 insertions(+), 89 deletions(-) diff --git a/src/locale/en_GB/LC_MESSAGES/django.po b/src/locale/en_GB/LC_MESSAGES/django.po index de13e5d34..f0c5b0381 100644 --- a/src/locale/en_GB/LC_MESSAGES/django.po +++ b/src/locale/en_GB/LC_MESSAGES/django.po @@ -2,18 +2,19 @@ # Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER # This file is distributed under the same license as the PACKAGE package. # FIRST AUTHOR , YEAR. -# +# # Translators: -# Ali Bates , 2021 -# +# Ali Bates, 2021 +# Jonas Winkler, 2021 +# #, fuzzy msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-01-28 22:02+0100\n" -"PO-Revision-Date: 2020-12-30 19:27+0000\n" -"Last-Translator: Ali Bates , 2021\n" +"POT-Creation-Date: 2021-02-16 14:52+0100\n" +"PO-Revision-Date: 2021-02-16 18:37+0000\n" +"Last-Translator: Jonas Winkler, 2021\n" "Language-Team: English (United Kingdom) (https://www.transifex.com/paperless/teams/115905/en_GB/)\n" "MIME-Version: 1.0\n" "Content-Type: text/plain; charset=UTF-8\n" @@ -25,64 +26,64 @@ msgstr "" msgid "Documents" msgstr "Documents" -#: documents/models.py:33 +#: documents/models.py:32 msgid "Any word" msgstr "Any word" -#: documents/models.py:34 +#: documents/models.py:33 msgid "All words" msgstr "All words" -#: documents/models.py:35 +#: documents/models.py:34 msgid "Exact match" msgstr "Exact match" -#: documents/models.py:36 +#: documents/models.py:35 msgid "Regular expression" msgstr "Regular expression" -#: documents/models.py:37 +#: documents/models.py:36 msgid "Fuzzy word" msgstr "Fuzzy word" -#: documents/models.py:38 +#: documents/models.py:37 msgid "Automatic" msgstr "Automatic" -#: documents/models.py:42 documents/models.py:352 paperless_mail/models.py:25 +#: documents/models.py:41 documents/models.py:364 paperless_mail/models.py:25 #: paperless_mail/models.py:109 msgid "name" msgstr "name" -#: documents/models.py:46 +#: documents/models.py:45 msgid "match" msgstr "match" -#: documents/models.py:50 +#: documents/models.py:49 msgid "matching algorithm" msgstr "matching algorithm" -#: documents/models.py:56 +#: documents/models.py:55 msgid "is insensitive" msgstr "is insensitive" -#: documents/models.py:75 documents/models.py:135 +#: documents/models.py:74 documents/models.py:134 msgid "correspondent" msgstr "correspondent" -#: documents/models.py:76 +#: documents/models.py:75 msgid "correspondents" msgstr "correspondents" -#: documents/models.py:98 +#: documents/models.py:97 msgid "color" msgstr "colour" -#: documents/models.py:102 +#: documents/models.py:101 msgid "is inbox tag" msgstr "is inbox tag" -#: documents/models.py:104 +#: documents/models.py:103 msgid "" "Marks this tag as an inbox tag: All newly consumed documents will be tagged " "with inbox tags." @@ -90,39 +91,39 @@ msgstr "" "Marks this tag as an inbox tag: All newly consumed documents will be tagged " "with inbox tags." -#: documents/models.py:109 +#: documents/models.py:108 msgid "tag" msgstr "tag" -#: documents/models.py:110 documents/models.py:166 +#: documents/models.py:109 documents/models.py:165 msgid "tags" msgstr "tags" -#: documents/models.py:116 documents/models.py:148 +#: documents/models.py:115 documents/models.py:147 msgid "document type" msgstr "document type" -#: documents/models.py:117 +#: documents/models.py:116 msgid "document types" msgstr "document types" -#: documents/models.py:125 +#: documents/models.py:124 msgid "Unencrypted" msgstr "Unencrypted" -#: documents/models.py:126 +#: documents/models.py:125 msgid "Encrypted with GNU Privacy Guard" msgstr "Encrypted with GNU Privacy Guard" -#: documents/models.py:139 +#: documents/models.py:138 msgid "title" msgstr "title" -#: documents/models.py:152 +#: documents/models.py:151 msgid "content" msgstr "content" -#: documents/models.py:154 +#: documents/models.py:153 msgid "" "The raw, text-only data of the document. This field is primarily used for " "searching." @@ -130,43 +131,43 @@ msgstr "" "The raw, text-only data of the document. This field is primarily used for " "searching." -#: documents/models.py:159 +#: documents/models.py:158 msgid "mime type" msgstr "mime type" -#: documents/models.py:170 +#: documents/models.py:169 msgid "checksum" msgstr "checksum" -#: documents/models.py:174 +#: documents/models.py:173 msgid "The checksum of the original document." msgstr "The checksum of the original document." -#: documents/models.py:178 +#: documents/models.py:177 msgid "archive checksum" msgstr "archive checksum" -#: documents/models.py:183 +#: documents/models.py:182 msgid "The checksum of the archived document." msgstr "The checksum of the archived document." -#: documents/models.py:187 documents/models.py:330 +#: documents/models.py:186 documents/models.py:342 msgid "created" msgstr "created" -#: documents/models.py:191 +#: documents/models.py:190 msgid "modified" msgstr "modified" -#: documents/models.py:195 +#: documents/models.py:194 msgid "storage type" msgstr "storage type" -#: documents/models.py:203 +#: documents/models.py:202 msgid "added" msgstr "added" -#: documents/models.py:207 +#: documents/models.py:206 msgid "filename" msgstr "filename" @@ -175,178 +176,186 @@ msgid "Current filename in storage" msgstr "Current filename in storage" #: documents/models.py:216 +msgid "archive filename" +msgstr "archive filename" + +#: documents/models.py:222 +msgid "Current archive filename in storage" +msgstr "Current archive filename in storage" + +#: documents/models.py:226 msgid "archive serial number" msgstr "archive serial number" -#: documents/models.py:221 +#: documents/models.py:231 msgid "The position of this document in your physical document archive." msgstr "The position of this document in your physical document archive." -#: documents/models.py:227 +#: documents/models.py:237 msgid "document" msgstr "document" -#: documents/models.py:228 +#: documents/models.py:238 msgid "documents" msgstr "documents" -#: documents/models.py:313 +#: documents/models.py:325 msgid "debug" msgstr "debug" -#: documents/models.py:314 +#: documents/models.py:326 msgid "information" msgstr "information" -#: documents/models.py:315 +#: documents/models.py:327 msgid "warning" msgstr "warning" -#: documents/models.py:316 +#: documents/models.py:328 msgid "error" msgstr "error" -#: documents/models.py:317 +#: documents/models.py:329 msgid "critical" msgstr "critical" -#: documents/models.py:321 +#: documents/models.py:333 msgid "group" msgstr "group" -#: documents/models.py:324 +#: documents/models.py:336 msgid "message" msgstr "message" -#: documents/models.py:327 +#: documents/models.py:339 msgid "level" msgstr "level" -#: documents/models.py:334 +#: documents/models.py:346 msgid "log" msgstr "log" -#: documents/models.py:335 +#: documents/models.py:347 msgid "logs" msgstr "logs" -#: documents/models.py:346 documents/models.py:396 +#: documents/models.py:358 documents/models.py:408 msgid "saved view" msgstr "saved view" -#: documents/models.py:347 +#: documents/models.py:359 msgid "saved views" msgstr "saved views" -#: documents/models.py:350 +#: documents/models.py:362 msgid "user" msgstr "user" -#: documents/models.py:356 +#: documents/models.py:368 msgid "show on dashboard" msgstr "show on dashboard" -#: documents/models.py:359 +#: documents/models.py:371 msgid "show in sidebar" msgstr "show in sidebar" -#: documents/models.py:363 +#: documents/models.py:375 msgid "sort field" msgstr "sort field" -#: documents/models.py:366 +#: documents/models.py:378 msgid "sort reverse" msgstr "sort reverse" -#: documents/models.py:372 +#: documents/models.py:384 msgid "title contains" msgstr "title contains" -#: documents/models.py:373 +#: documents/models.py:385 msgid "content contains" msgstr "content contains" -#: documents/models.py:374 +#: documents/models.py:386 msgid "ASN is" msgstr "ASN is" -#: documents/models.py:375 +#: documents/models.py:387 msgid "correspondent is" msgstr "correspondent is" -#: documents/models.py:376 +#: documents/models.py:388 msgid "document type is" msgstr "document type is" -#: documents/models.py:377 +#: documents/models.py:389 msgid "is in inbox" msgstr "is in inbox" -#: documents/models.py:378 +#: documents/models.py:390 msgid "has tag" msgstr "has tag" -#: documents/models.py:379 +#: documents/models.py:391 msgid "has any tag" msgstr "has any tag" -#: documents/models.py:380 +#: documents/models.py:392 msgid "created before" msgstr "created before" -#: documents/models.py:381 +#: documents/models.py:393 msgid "created after" msgstr "created after" -#: documents/models.py:382 +#: documents/models.py:394 msgid "created year is" msgstr "created year is" -#: documents/models.py:383 +#: documents/models.py:395 msgid "created month is" msgstr "created month is" -#: documents/models.py:384 +#: documents/models.py:396 msgid "created day is" msgstr "created day is" -#: documents/models.py:385 +#: documents/models.py:397 msgid "added before" msgstr "added before" -#: documents/models.py:386 +#: documents/models.py:398 msgid "added after" msgstr "added after" -#: documents/models.py:387 +#: documents/models.py:399 msgid "modified before" msgstr "modified before" -#: documents/models.py:388 +#: documents/models.py:400 msgid "modified after" msgstr "modified after" -#: documents/models.py:389 +#: documents/models.py:401 msgid "does not have tag" msgstr "does not have tag" -#: documents/models.py:400 +#: documents/models.py:412 msgid "rule type" msgstr "rule type" -#: documents/models.py:404 +#: documents/models.py:416 msgid "value" msgstr "value" -#: documents/models.py:410 +#: documents/models.py:422 msgid "filter rule" msgstr "filter rule" -#: documents/models.py:411 +#: documents/models.py:423 msgid "filter rules" msgstr "filter rules" -#: documents/serialisers.py:383 +#: documents/serialisers.py:370 #, python-format msgid "File type %(type)s not supported" msgstr "File type %(type)s not supported" @@ -391,19 +400,23 @@ msgstr "Password" msgid "Sign in" msgstr "Sign in" -#: paperless/settings.py:286 -msgid "English" -msgstr "English" +#: paperless/settings.py:291 +msgid "English (US)" +msgstr "English (US)" -#: paperless/settings.py:287 +#: paperless/settings.py:292 +msgid "English (GB)" +msgstr "English (GB)" + +#: paperless/settings.py:293 msgid "German" msgstr "German" -#: paperless/settings.py:288 +#: paperless/settings.py:294 msgid "Dutch" msgstr "Dutch" -#: paperless/settings.py:289 +#: paperless/settings.py:295 msgid "French" msgstr "French" From 0e237fa45908f2e60d9801422b06a40f3475946e Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Thu, 18 Feb 2021 17:11:47 +0100 Subject: [PATCH 146/225] messages --- src-ui/messages.xlf | 66 +++++++++++-------- .../filter-editor/filter-editor.component.ts | 4 +- 2 files changed, 43 insertions(+), 27 deletions(-) diff --git a/src-ui/messages.xlf b/src-ui/messages.xlf index a80bd082f..731fdff7f 100644 --- a/src-ui/messages.xlf +++ b/src-ui/messages.xlf @@ -52,17 +52,17 @@     - View "" saved successfully. + View "" saved successfully. src/app/components/document-list/document-list.component.ts - 109 + 115 View "" created successfully. src/app/components/document-list/document-list.component.ts - 130 + 136 @@ -114,8 +114,8 @@ 72 - - Save "" + + Save "" src/app/components/document-list/document-list.component.html 71 @@ -513,13 +513,6 @@ 1 - - Filter - - src/app/components/manage/logs/logs.component.html - 7 - - Saved view "" deleted. @@ -545,14 +538,21 @@ Use date format of display language src/app/components/manage/settings/settings.component.ts - 95 + 96 + + + + ISO 8601 + + src/app/components/manage/settings/settings.component.ts + 97 Error while storing settings on server: src/app/components/manage/settings/settings.component.ts - 111 + 114 @@ -1081,6 +1081,13 @@ 46 + + Title: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 50 + + Filter tags @@ -1414,20 +1421,20 @@ 1 - - Documents in inbox: - - src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html - 3 - - - - Total documents: + + Total documents: src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html 4 + + Documents in inbox: + + src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html + 3 + + Processing: @@ -1619,25 +1626,32 @@ 82 + + English (GB) + + src/app/services/settings.service.ts + 83 + + German src/app/services/settings.service.ts - 83 + 84 Dutch src/app/services/settings.service.ts - 84 + 85 French src/app/services/settings.service.ts - 85 + 86 diff --git a/src-ui/src/app/components/document-list/filter-editor/filter-editor.component.ts b/src-ui/src/app/components/document-list/filter-editor/filter-editor.component.ts index a9111bc9f..a6506efde 100644 --- a/src-ui/src/app/components/document-list/filter-editor/filter-editor.component.ts +++ b/src-ui/src/app/components/document-list/filter-editor/filter-editor.component.ts @@ -46,6 +46,8 @@ export class FilterEditorComponent implements OnInit, OnDestroy { return $localize`Without any tag` } + case FILTER_TITLE: + return $localize`Title: ${rule.value}` } } @@ -117,7 +119,7 @@ export class FilterEditorComponent implements OnInit, OnDestroy { }) } - get filterRules() { + get filterRules(): FilterRule[] { let filterRules: FilterRule[] = [] if (this._titleFilter) { filterRules.push({rule_type: FILTER_TITLE, value: this._titleFilter}) From 0f80eee54e7b7e3a29f9f03be6e8330f12a5eed8 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Thu, 18 Feb 2021 17:29:21 +0100 Subject: [PATCH 147/225] refactored most of the list view; fixes #147, much snappier UX when switching between views --- .../saved-view-widget.component.ts | 2 +- .../document-detail.component.ts | 4 +- .../bulk-editor/bulk-editor.component.ts | 2 +- .../document-list.component.html | 6 +- .../document-list/document-list.component.ts | 38 +-- .../services/document-list-view.service.ts | 236 ++++++++++-------- 6 files changed, 163 insertions(+), 125 deletions(-) diff --git a/src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.ts b/src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.ts index 36feeea1e..20bc26cdc 100644 --- a/src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.ts +++ b/src-ui/src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.ts @@ -48,7 +48,7 @@ export class SavedViewWidgetComponent implements OnInit, OnDestroy { if (this.savedView.show_in_sidebar) { this.router.navigate(['view', this.savedView.id]) } else { - this.list.load(this.savedView) + this.list.loadSavedView(this.savedView, true) this.router.navigate(["documents"]) } } diff --git a/src-ui/src/app/components/document-detail/document-detail.component.ts b/src-ui/src/app/components/document-detail/document-detail.component.ts index 071c8e249..af98a6f7f 100644 --- a/src-ui/src/app/components/document-detail/document-detail.component.ts +++ b/src-ui/src/app/components/document-detail/document-detail.component.ts @@ -191,8 +191,8 @@ export class DocumentDetailComponent implements OnInit { close() { this.openDocumentService.closeDocument(this.document) - if (this.documentListViewService.savedViewId) { - this.router.navigate(['view', this.documentListViewService.savedViewId]) + if (this.documentListViewService.activeSavedViewId) { + this.router.navigate(['view', this.documentListViewService.activeSavedViewId]) } else { this.router.navigate(['documents']) } diff --git a/src-ui/src/app/components/document-list/bulk-editor/bulk-editor.component.ts b/src-ui/src/app/components/document-list/bulk-editor/bulk-editor.component.ts index 04fc2a978..ebc7a8c11 100644 --- a/src-ui/src/app/components/document-list/bulk-editor/bulk-editor.component.ts +++ b/src-ui/src/app/components/document-list/bulk-editor/bulk-editor.component.ts @@ -137,7 +137,7 @@ export class BulkEditorComponent { } else { modal.componentInstance.message = $localize`This operation will add the tags ${this._localizeList(changedTags.itemsToAdd)} and remove the tags ${this._localizeList(changedTags.itemsToRemove)} on ${this.list.selected.size} selected document(s).` } - + modal.componentInstance.btnClass = "btn-warning" modal.componentInstance.btnCaption = $localize`Confirm` modal.componentInstance.confirmClicked.subscribe(() => { diff --git a/src-ui/src/app/components/document-list/document-list.component.html b/src-ui/src/app/components/document-list/document-list.component.html index cfc2e655d..75a9804a2 100644 --- a/src-ui/src/app/components/document-list/document-list.component.html +++ b/src-ui/src/app/components/document-list/document-list.component.html @@ -63,12 +63,12 @@
    - +
    - +
    @@ -86,7 +86,7 @@ {list.collectionSize, plural, =1 {One document} other {{{list.collectionSize || 0}} documents}} (filtered)

    + [rotate]="true" aria-label="Default pagination">
    diff --git a/src-ui/src/app/components/document-list/document-list.component.ts b/src-ui/src/app/components/document-list/document-list.component.ts index 0d1562c24..cf7afb845 100644 --- a/src-ui/src/app/components/document-list/document-list.component.ts +++ b/src-ui/src/app/components/document-list/document-list.component.ts @@ -1,4 +1,4 @@ -import { AfterViewInit, Component, OnDestroy, OnInit, QueryList, ViewChild, ViewChildren } from '@angular/core'; +import { Component, OnDestroy, OnInit, QueryList, ViewChild, ViewChildren } from '@angular/core'; import { ActivatedRoute, Router } from '@angular/router'; import { NgbModal } from '@ng-bootstrap/ng-bootstrap'; import { Subscription } from 'rxjs'; @@ -9,7 +9,7 @@ import { ConsumerStatusService } from 'src/app/services/consumer-status.service' import { DocumentListViewService } from 'src/app/services/document-list-view.service'; import { DOCUMENT_SORT_FIELDS } from 'src/app/services/rest/document.service'; import { SavedViewService } from 'src/app/services/rest/saved-view.service'; -import { Toast, ToastService } from 'src/app/services/toast.service'; +import { ToastService } from 'src/app/services/toast.service'; import { FilterEditorComponent } from './filter-editor/filter-editor.component'; import { SaveViewConfigDialogComponent } from './save-view-config-dialog/save-view-config-dialog.component'; @@ -46,7 +46,7 @@ export class DocumentListComponent implements OnInit, OnDestroy { } getTitle() { - return this.list.savedViewTitle || $localize`Documents` + return this.list.activeSavedViewTitle || $localize`Documents` } getSortFields() { @@ -73,19 +73,18 @@ export class DocumentListComponent implements OnInit, OnDestroy { this.list.reload() }) this.route.paramMap.subscribe(params => { - this.list.clear() if (params.has('id')) { this.savedViewService.getCached(+params.get('id')).subscribe(view => { if (!view) { this.router.navigate(["404"]) return } - this.list.savedView = view + this.list.activateSavedView(view) this.list.reload() this.rulesChanged() }) } else { - this.list.savedView = null + this.list.activateSavedView(null) this.list.reload() this.rulesChanged() } @@ -99,16 +98,23 @@ export class DocumentListComponent implements OnInit, OnDestroy { } loadViewConfig(view: PaperlessSavedView) { - this.list.load(view) + this.list.loadSavedView(view) this.list.reload() this.rulesChanged() } saveViewConfig() { - this.savedViewService.update(this.list.savedView).subscribe(result => { - this.toastService.showInfo($localize`View "${this.list.savedView.name}" saved successfully.`) - }) - + if (this.list.activeSavedViewId != null) { + let savedView: PaperlessSavedView = { + id: this.list.activeSavedViewId, + filter_rules: this.list.filterRules, + sort_field: this.list.sortField, + sort_reverse: this.list.sortReverse + } + this.savedViewService.patch(savedView).subscribe(result => { + this.toastService.showInfo($localize`View "${this.list.activeSavedViewTitle}" saved successfully.`) + }) + } } saveViewConfigAs() { @@ -116,7 +122,7 @@ export class DocumentListComponent implements OnInit, OnDestroy { modal.componentInstance.defaultName = this.filterEditor.generateFilterName() modal.componentInstance.saveClicked.subscribe(formValue => { modal.componentInstance.buttonsEnabled = false - let savedView = { + let savedView: PaperlessSavedView = { name: formValue.name, show_on_dashboard: formValue.showOnDashboard, show_in_sidebar: formValue.showInSideBar, @@ -137,8 +143,8 @@ export class DocumentListComponent implements OnInit, OnDestroy { resetFilters(): void { this.filterRulesModified = false - if (this.list.savedViewId) { - this.savedViewService.getCached(this.list.savedViewId).subscribe(viewUntouched => { + if (this.list.activeSavedViewId) { + this.savedViewService.getCached(this.list.activeSavedViewId).subscribe(viewUntouched => { this.list.filterRules = viewUntouched.filter_rules this.list.reload() }) @@ -150,11 +156,11 @@ export class DocumentListComponent implements OnInit, OnDestroy { rulesChanged() { let modified = false - if (this.list.savedView == null) { + if (this.list.activeSavedViewId == null) { modified = this.list.filterRules.length > 0 // documents list is modified if it has any filters } else { // compare savedView current filters vs original - this.savedViewService.getCached(this.list.savedViewId).subscribe(view => { + this.savedViewService.getCached(this.list.activeSavedViewId).subscribe(view => { let filterRulesInitial = view.filter_rules if (this.list.filterRules.length !== filterRulesInitial.length) modified = true diff --git a/src-ui/src/app/services/document-list-view.service.ts b/src-ui/src/app/services/document-list-view.service.ts index 4eb0c276c..334706a3c 100644 --- a/src-ui/src/app/services/document-list-view.service.ts +++ b/src-ui/src/app/services/document-list-view.service.ts @@ -8,6 +8,23 @@ import { DOCUMENT_LIST_SERVICE } from '../data/storage-keys'; import { DocumentService } from './rest/document.service'; import { SettingsService, SETTINGS_KEYS } from './settings.service'; +interface ListViewState { + + title?: string + + documents?: PaperlessDocument[] + + currentPage: number + collectionSize: number + + sortField: string + sortReverse: boolean + + filterRules: FilterRule[] + + selected?: Set + +} /** * This service manages the document list which is displayed using the document list view. @@ -20,156 +37,174 @@ import { SettingsService, SETTINGS_KEYS } from './settings.service'; }) export class DocumentListViewService { - static DEFAULT_SORT_FIELD = 'created' - isReloading: boolean = false - documents: PaperlessDocument[] = [] - currentPage = 1 - currentPageSize: number = this.settings.get(SETTINGS_KEYS.DOCUMENT_LIST_SIZE) - collectionSize: number + rangeSelectionAnchorIndex: number lastRangeSelectionToIndex: number - /** - * This is the current config for the document list. The service will always remember the last settings used for the document list. - */ - private _documentListViewConfig: PaperlessSavedView - /** - * Optionally, this is the currently selected saved view, which might be null. - */ - private _savedViewConfig: PaperlessSavedView + currentPageSize: number = this.settings.get(SETTINGS_KEYS.DOCUMENT_LIST_SIZE) - get savedView(): PaperlessSavedView { - return this._savedViewConfig + private listViewStates: Map = new Map() + + private _activeSavedViewId: number = null + + get activeSavedViewId() { + return this._activeSavedViewId } - set savedView(value: PaperlessSavedView) { - if (value && !this._savedViewConfig || value && value.id != this._savedViewConfig.id) { - //saved view inactive and should be active now, or saved view active, but a different view is requested - //this is here so that we don't modify value, which might be the actual instance of the saved view. - this.selectNone() - this._savedViewConfig = Object.assign({}, value) - } else if (this._savedViewConfig && !value) { - //saved view active, but document list requested - this.selectNone() - this._savedViewConfig = null + get activeSavedViewTitle() { + return this.activeListViewState.title + } + + private defaultListViewState(): ListViewState { + return { + title: null, + documents: [], + currentPage: 1, + collectionSize: null, + sortField: "created", + sortReverse: true, + filterRules: [], + selected: new Set() } } - get savedViewId() { - return this.savedView?.id + private get activeListViewState() { + if (!this.listViewStates.has(this._activeSavedViewId)) { + this.listViewStates.set(this._activeSavedViewId, this.defaultListViewState()) + } + return this.listViewStates.get(this._activeSavedViewId) } - get savedViewTitle() { - return this.savedView?.name - } - - get documentListView() { - return this._documentListViewConfig - } - - set documentListView(value) { - if (value) { - this._documentListViewConfig = Object.assign({}, value) - this.saveDocumentListView() + activateSavedView(view: PaperlessSavedView) { + this.rangeSelectionAnchorIndex = this.lastRangeSelectionToIndex = null + if (view) { + this._activeSavedViewId = view.id + this.loadSavedView(view) + } else { + this._activeSavedViewId = null } } - /** - * This is what switches between the saved views and the document list view. Everything on the document list uses - * this property to determine the settings for the currently displayed document list. - */ - get view() { - return this.savedView || this.documentListView - } - - load(view: PaperlessSavedView) { - this.documentListView.filter_rules = cloneFilterRules(view.filter_rules) - this.documentListView.sort_reverse = view.sort_reverse - this.documentListView.sort_field = view.sort_field - this.saveDocumentListView() - } - - clear() { - this.collectionSize = null - this.documents = [] - this.currentPage = 1 + loadSavedView(view: PaperlessSavedView, closeCurrentView: boolean = false) { + if (closeCurrentView) { + this._activeSavedViewId = null + } + this.activeListViewState.filterRules = cloneFilterRules(view.filter_rules) + this.activeListViewState.sortField = view.sort_field + this.activeListViewState.sortReverse = view.sort_reverse + if (this._activeSavedViewId) { + this.activeListViewState.title = view.name + } + this.reduceSelectionToFilter() } reload(onFinish?) { this.isReloading = true + let activeListViewState = this.activeListViewState + this.documentService.listFiltered( - this.currentPage, + activeListViewState.currentPage, this.currentPageSize, - this.view.sort_field, - this.view.sort_reverse, - this.view.filter_rules).subscribe( + activeListViewState.sortField, + activeListViewState.sortReverse, + activeListViewState.filterRules).subscribe( result => { - this.collectionSize = result.count - this.documents = result.results + this.isReloading = false + activeListViewState.collectionSize = result.count + activeListViewState.documents = result.results if (onFinish) { onFinish() } this.rangeSelectionAnchorIndex = this.lastRangeSelectionToIndex = null - this.isReloading = false }, error => { - if (this.currentPage != 1 && error.status == 404) { + this.isReloading = false + if (activeListViewState.currentPage != 1 && error.status == 404) { // this happens when applying a filter: the current page might not be available anymore due to the reduced result set. - this.currentPage = 1 + activeListViewState.currentPage = 1 this.reload() } - this.isReloading = false }) } set filterRules(filterRules: FilterRule[]) { - //we're going to clone the filterRules object, since we don't - //want changes in the filter editor to propagate into here right away. - this.view.filter_rules = filterRules + this.activeListViewState.filterRules = filterRules this.reload() this.reduceSelectionToFilter() this.saveDocumentListView() } get filterRules(): FilterRule[] { - return this.view.filter_rules + return this.activeListViewState.filterRules } set sortField(field: string) { - this.view.sort_field = field - this.saveDocumentListView() + this.activeListViewState.sortField = field this.reload() + this.saveDocumentListView() } get sortField(): string { - return this.view.sort_field + return this.activeListViewState.sortField } set sortReverse(reverse: boolean) { - this.view.sort_reverse = reverse - this.saveDocumentListView() + this.activeListViewState.sortReverse = reverse this.reload() + this.saveDocumentListView() } get sortReverse(): boolean { - return this.view.sort_reverse + return this.activeListViewState.sortReverse + } + + get collectionSize(): number { + return this.activeListViewState.collectionSize + } + + get currentPage(): number { + return this.activeListViewState.currentPage + } + + set currentPage(page: number) { + this.activeListViewState.currentPage = page + this.reload() + this.saveDocumentListView() + } + + get documents(): PaperlessDocument[] { + return this.activeListViewState.documents + } + + get selected(): Set { + return this.activeListViewState.selected } setSort(field: string, reverse: boolean) { - this.view.sort_field = field - this.view.sort_reverse = reverse - this.saveDocumentListView() + this.activeListViewState.sortField = field + this.activeListViewState.sortReverse = reverse this.reload() + this.saveDocumentListView() } private saveDocumentListView() { - sessionStorage.setItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG, JSON.stringify(this.documentListView)) + if (this._activeSavedViewId == null) { + let savedState: ListViewState = { + collectionSize: this.activeListViewState.collectionSize, + currentPage: this.activeListViewState.currentPage, + filterRules: this.activeListViewState.filterRules, + sortField: this.activeListViewState.sortField, + sortReverse: this.activeListViewState.sortReverse + } + sessionStorage.setItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG, JSON.stringify(savedState)) + } } quickFilter(filterRules: FilterRule[]) { - this.savedView = null - this.view.filter_rules = filterRules + this._activeSavedViewId = null + this.activeListViewState.filterRules = filterRules + this.activeListViewState.currentPage = 1 this.reduceSelectionToFilter() this.saveDocumentListView() this.router.navigate(["documents"]) @@ -217,8 +252,6 @@ export class DocumentListViewService { } } - selected = new Set() - selectNone() { this.selected.clear() this.rangeSelectionAnchorIndex = this.lastRangeSelectionToIndex = null @@ -227,13 +260,11 @@ export class DocumentListViewService { reduceSelectionToFilter() { if (this.selected.size > 0) { this.documentService.listAllFilteredIds(this.filterRules).subscribe(ids => { - let subset = new Set() - for (let id of ids) { - if (this.selected.has(id)) { - subset.add(id) + for (let id of this.selected) { + if (!ids.includes(id)) { + this.selected.delete(id) } } - this.selected = subset }) } } @@ -287,20 +318,21 @@ export class DocumentListViewService { } constructor(private documentService: DocumentService, private settings: SettingsService, private router: Router) { - let documentListViewConfigJson = sessionStorage.getItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG) + let documentListViewConfigJson = sessionStorage.getItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG) if (documentListViewConfigJson) { try { - this.documentListView = JSON.parse(documentListViewConfigJson) + let savedState: ListViewState = JSON.parse(documentListViewConfigJson) + // Remove null elements from the restored state + Object.keys(savedState).forEach(k => { + if (savedState[k] == null) { + delete savedState[k] + } + }) + //only use restored state attributes instead of defaults if they are not null + let newState = Object.assign(this.defaultListViewState(), savedState) + this.listViewStates.set(null, newState) } catch (e) { sessionStorage.removeItem(DOCUMENT_LIST_SERVICE.CURRENT_VIEW_CONFIG) - this.documentListView = null - } - } - if (!this.documentListView || this.documentListView.filter_rules == null || this.documentListView.sort_reverse == null || this.documentListView.sort_field == null) { - this.documentListView = { - filter_rules: [], - sort_reverse: true, - sort_field: 'created' } } } From b5a8106a6ab1da778e0f2a0bf09e7f0257224bce Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Thu, 18 Feb 2021 16:43:41 +0000 Subject: [PATCH 148/225] Translate /src-ui/messages.xlf in de translation completed for the source file '/src-ui/messages.xlf' on the 'de' language. --- src-ui/src/locale/messages.de.xlf | 78 +++++++++++++++++++------------ 1 file changed, 47 insertions(+), 31 deletions(-) diff --git a/src-ui/src/locale/messages.de.xlf b/src-ui/src/locale/messages.de.xlf index cc5e1d68a..aa57afdab 100644 --- a/src-ui/src/locale/messages.de.xlf +++ b/src-ui/src/locale/messages.de.xlf @@ -58,11 +58,11 @@ - View "" saved successfully. - Ansicht "" erfolgreich gespeichert. + View "" saved successfully. + Ansicht "" erfolgreich gespeichert. src/app/components/document-list/document-list.component.ts - 109 + 115 @@ -70,7 +70,7 @@ Ansicht "" erfolgreich erstellt. src/app/components/document-list/document-list.component.ts - 130 + 136 @@ -129,9 +129,9 @@ 72 - - Save "" - "" speichern + + Save "" + "" speichern src/app/components/document-list/document-list.component.html 71 @@ -585,14 +585,6 @@ 1 - - Filter - Filtern - - src/app/components/manage/logs/logs.component.html - 7 - - Saved view "" deleted. Gespeicherte Ansicht "" gelöscht. @@ -622,7 +614,15 @@ Benutze Datumsformat der Anzeigesprache src/app/components/manage/settings/settings.component.ts - 95 + 96 + + + + ISO 8601 + ISO 8601 + + src/app/components/manage/settings/settings.component.ts + 97 @@ -630,7 +630,7 @@ Fehler beim Speichern der Einstellungen auf dem Server: src/app/components/manage/settings/settings.component.ts - 111 + 114 @@ -1234,6 +1234,14 @@ 46 + + Title: + Titel: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 50 + + Filter tags Tags filtern @@ -1614,22 +1622,22 @@ 1 - - Documents in inbox: - Dokumente im Posteingang: - - src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html - 3 - - - - Total documents: - Anzahl Dokumente gesamt: + + Total documents: + Anzahl Dokumente gesamt: src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html 4 + + Documents in inbox: + Dokumente im Posteingang: + + src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html + 3 + + Processing: Verarbeite: @@ -1848,12 +1856,20 @@ 82 + + English (GB) + Englisch (UK) + + src/app/services/settings.service.ts + 83 + + German Deutsch src/app/services/settings.service.ts - 83 + 84 @@ -1861,7 +1877,7 @@ Niederländisch src/app/services/settings.service.ts - 84 + 85 @@ -1869,7 +1885,7 @@ Französisch src/app/services/settings.service.ts - 85 + 86 From efb49af7ac44285fe260d698652f1af817dd5ce2 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Thu, 18 Feb 2021 16:44:55 +0000 Subject: [PATCH 149/225] Translate /src-ui/messages.xlf in en_GB translation completed for the source file '/src-ui/messages.xlf' on the 'en_GB' language. --- src-ui/src/locale/messages.en_GB.xlf | 78 +++++++++++++++++----------- 1 file changed, 47 insertions(+), 31 deletions(-) diff --git a/src-ui/src/locale/messages.en_GB.xlf b/src-ui/src/locale/messages.en_GB.xlf index 49baa77ed..14ab56659 100644 --- a/src-ui/src/locale/messages.en_GB.xlf +++ b/src-ui/src/locale/messages.en_GB.xlf @@ -58,11 +58,11 @@ - View "" saved successfully. - View &quot;&quot; saved successfully. + View "" saved successfully. + View &quot;&quot; saved successfully. src/app/components/document-list/document-list.component.ts - 109 + 115 @@ -70,7 +70,7 @@ View &quot;&quot; created successfully. src/app/components/document-list/document-list.component.ts - 130 + 136 @@ -129,9 +129,9 @@ 72 - - Save "" - Save &quot;&quot; + + Save "" + Save &quot;&quot; src/app/components/document-list/document-list.component.html 71 @@ -585,14 +585,6 @@ 1 - - Filter - Filter - - src/app/components/manage/logs/logs.component.html - 7 - - Saved view "" deleted. Saved view &quot;&quot; deleted. @@ -622,7 +614,15 @@ Use date format of display language src/app/components/manage/settings/settings.component.ts - 95 + 96 + + + + ISO 8601 + ISO 8601 + + src/app/components/manage/settings/settings.component.ts + 97 @@ -630,7 +630,7 @@ Error while storing settings on server: src/app/components/manage/settings/settings.component.ts - 111 + 114 @@ -1234,6 +1234,14 @@ 46 + + Title: + Title: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 50 + + Filter tags Filter tags @@ -1614,22 +1622,22 @@ 1 - - Documents in inbox: - Documents in inbox: - - src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html - 3 - - - - Total documents: - Total documents: + + Total documents: + Total documents: src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html 4 + + Documents in inbox: + Documents in inbox: + + src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html + 3 + + Processing: Processing: @@ -1848,12 +1856,20 @@ 82 + + English (GB) + English (GB) + + src/app/services/settings.service.ts + 83 + + German German src/app/services/settings.service.ts - 83 + 84 @@ -1861,7 +1877,7 @@ Dutch src/app/services/settings.service.ts - 84 + 85 @@ -1869,7 +1885,7 @@ French src/app/services/settings.service.ts - 85 + 86 From 1140a878b47066107436e0996f8966a3bc258ee7 Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Fri, 19 Feb 2021 11:22:43 +0100 Subject: [PATCH 150/225] Update README.md --- README.md | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index c90be0912..ea47248dc 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,16 @@ # Paperless-ng -[Paperless](https://github.com/the-paperless-project/paperless) is an application by Daniel Quinn and contributors that indexes your scanned documents and allows you to easily search for documents and store metadata alongside your documents. +[Paperless (click me)](https://github.com/the-paperless-project/paperless) is an application by Daniel Quinn and contributors that indexes your scanned documents and allows you to easily search for documents and store metadata alongside your documents. -Paperless-ng is a fork of the original project, adding a new interface and many other changes under the hood. For a detailed list of changes, have a look at the [change log](https://paperless-ng.readthedocs.io/en/latest/changelog.html) in the documentation. +Paperless-ng is a fork of the original project, adding a new interface and many other changes under the hood. These key points should help you decide whether Paperless-ng is something you would prefer over Paperless: + +* Interface: The new front end is the main interface for paperless-ng, the old interface still exists but most customizations (such as thumbnails for the document list) have been removed. +* Encryption: Paperless-ng does not support GnuPG anymore, since storing your data on encrypted file systems (that you optionally mount on demand) achieves about the same result. +* Resource usage: Paperless-ng does use a bit more resources than Paperless. Running the web server requires about 300MB of RAM, depending on the configuration. It still runs on Pi (many users do that), but it has been generally geared to better use the resources of more powerful systems. +* API changes: If you rely on the REST API of paperless, some of its functionality has been changed. + +For a detailed list of changes, have a look at the [change log](https://paperless-ng.readthedocs.io/en/latest/changelog.html) in the documentation. # How it Works @@ -32,8 +39,8 @@ Here's what you get: * Performs OCR on your documents, adds selectable text to image only documents and adds tags, correspondents and document types to your documents. * Supports PDF documents, images, plain text files, and Office documents (Word, Excel, Powerpoint, and LibreOffice equivalents). * Office document support is optional and provided by Apache Tika (see [configuration](https://paperless-ng.readthedocs.io/en/latest/configuration.html#tika-settings)) -* Paperless stores your documents plain on disk. Filenames and folders are managed by paperless and can be configured freely. -* Single page application front end. Should be pretty snappy. Will be mobile friendly in the future. +* Paperless stores your documents plain on disk. Filenames and folders are managed by paperless and their format can be configured freely. +* Single page application front end. * Includes a dashboard that shows basic statistics and has document upload. * Filtering by tags, correspondents, types, and more. * Customizable views can be saved and displayed on the dashboard. @@ -44,14 +51,13 @@ Here's what you get: * Searching for similar documents ("More like this") * Email processing: Paperless adds documents from your email accounts. * Configure multiple accounts and filters for each account. - * When adding documents from mails, paperless can move these mails to a new folder, mark them as read, flag them or delete them. + * When adding documents from mails, paperless can move these mails to a new folder, mark them as read, flag them as important or delete them. * Machine learning powered document matching. * Paperless learns from your documents and will be able to automatically assign tags, correspondents and types to documents once you've stored a few documents in paperless. -* A task processor that processes documents in parallel and also tells you when something goes wrong. On modern multi core systems, consumption is blazing fast. +* Optimized for multi core systems: Paperless-ng consumes multiple documents in parallel. +* The integrated sanity checker makes sure that your document archive is in good health. -If you want to see some screenshots of paperless-ng in action, [some are available in the documentation](https://paperless-ng.readthedocs.io/en/latest/screenshots.html). However, some parts of the UI have changed since I took these. - -For a complete list of changes from paperless, check out the [changelog](https://paperless-ng.readthedocs.io/en/latest/changelog.html) +If you want to see some screenshots of paperless-ng in action, [some are available in the documentation](https://paperless-ng.readthedocs.io/en/latest/screenshots.html). # Getting started @@ -103,4 +109,4 @@ These projects also exist, but their status and compatibility with paperless-ng # Important Note -Document scanners are typically used to scan sensitive documents. Things like your social insurance number, tax records, invoices, etc. Everything is stored in the clear without encryption by default (it needs to be searchable, so if someone has ideas on how to do that on encrypted data, I'm all ears). This means that Paperless should never be run on an untrusted host. Instead, I recommend that if you do want to use it, run it locally on a server in your own home. +Document scanners are typically used to scan sensitive documents. Things like your social insurance number, tax records, invoices, etc. Everything is stored in the clear without encryption. This means that Paperless should never be run on an untrusted host. Instead, I recommend that if you do want to use it, run it locally on a server in your own home. From 44936dc5f0663d086dfc45d0d3caa8ad3c0f4e05 Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Fri, 19 Feb 2021 11:24:21 +0100 Subject: [PATCH 151/225] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ea47248dc..745c3b8f9 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Paperless-ng is a fork of the original project, adding a new interface and many * Interface: The new front end is the main interface for paperless-ng, the old interface still exists but most customizations (such as thumbnails for the document list) have been removed. * Encryption: Paperless-ng does not support GnuPG anymore, since storing your data on encrypted file systems (that you optionally mount on demand) achieves about the same result. -* Resource usage: Paperless-ng does use a bit more resources than Paperless. Running the web server requires about 300MB of RAM, depending on the configuration. It still runs on Pi (many users do that), but it has been generally geared to better use the resources of more powerful systems. +* Resource usage: Paperless-ng does use a bit more resources than Paperless. Running the web server requires about 300MB of RAM or more, depending on the configuration. It still runs on Pi (many users do that), but it has been generally geared to better use the resources of more powerful systems. * API changes: If you rely on the REST API of paperless, some of its functionality has been changed. For a detailed list of changes, have a look at the [change log](https://paperless-ng.readthedocs.io/en/latest/changelog.html) in the documentation. From 950bb468278856afd51c18d18f7f4c8cbf6a331b Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Fri, 19 Feb 2021 11:31:14 +0100 Subject: [PATCH 152/225] version bump --- src-ui/src/environments/environment.prod.ts | 2 +- src/paperless/version.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src-ui/src/environments/environment.prod.ts b/src-ui/src/environments/environment.prod.ts index 6df8ec565..51beb3fba 100644 --- a/src-ui/src/environments/environment.prod.ts +++ b/src-ui/src/environments/environment.prod.ts @@ -2,7 +2,7 @@ export const environment = { production: true, apiBaseUrl: "/api/", appTitle: "Paperless-ng", - version: "1.1.4", + version: "1.1.5", webSocketHost: window.location.host, webSocketProtocol: (window.location.protocol == "https:" ? "wss:" : "ws:") }; diff --git a/src/paperless/version.py b/src/paperless/version.py index f336d581e..b21fef930 100644 --- a/src/paperless/version.py +++ b/src/paperless/version.py @@ -1 +1 @@ -__version__ = (1, 1, 4) +__version__ = (1, 1, 5) From 4de4789605469dfbf01fd5c274e4cb1e8e900185 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Fri, 19 Feb 2021 11:34:51 +0100 Subject: [PATCH 153/225] this took way too much time --- resources/logo.txt | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 resources/logo.txt diff --git a/resources/logo.txt b/resources/logo.txt new file mode 100644 index 000000000..75a1d2c75 --- /dev/null +++ b/resources/logo.txt @@ -0,0 +1,16 @@ + 9w + {@@N + Q@@@@H + G@@@@@@@\ + SilN@@@@@@@ + *Q *@@@@@@@@S /= = = = = = = = = = = = = = = = = =\ + *@ B@@@@@@@@N || || + N R$ A@@@@@@@@@@ || PAPERLESS-NG || +x@@ $U B@@@@@@@@@R || || +N@@N^ @ N@@@@@@@@@* \= = = = = = = = = = = = = = = = = =/ +|@@@u @ E@@@@@@@@l + Q@@@ \ Px@@@@@@P + 1@@S` @@@o' + z$ ; + v + / From 0ae9aecdef1cd0c40a10f83ca02e3afb3d8b37f2 Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Fri, 19 Feb 2021 11:51:59 +0100 Subject: [PATCH 154/225] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 745c3b8f9..83ef1c1ed 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Paperless-ng is a fork of the original project, adding a new interface and many * Interface: The new front end is the main interface for paperless-ng, the old interface still exists but most customizations (such as thumbnails for the document list) have been removed. * Encryption: Paperless-ng does not support GnuPG anymore, since storing your data on encrypted file systems (that you optionally mount on demand) achieves about the same result. -* Resource usage: Paperless-ng does use a bit more resources than Paperless. Running the web server requires about 300MB of RAM or more, depending on the configuration. It still runs on Pi (many users do that), but it has been generally geared to better use the resources of more powerful systems. +* Resource usage: Paperless-ng does use a bit more resources than Paperless. Running the web server requires about 300MB of RAM or more, depending on the configuration. While adding documents, it requires about 300MB additional RAM, depending on the document. It still runs on Pi (many users do that), but it has been generally geared to better use the resources of more powerful systems. * API changes: If you rely on the REST API of paperless, some of its functionality has been changed. For a detailed list of changes, have a look at the [change log](https://paperless-ng.readthedocs.io/en/latest/changelog.html) in the documentation. From 31f03ef1d301a6e23c276408e88d53aadf6886f2 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Sat, 20 Feb 2021 16:09:29 +0100 Subject: [PATCH 155/225] API support for downloading compressed archives of multiple documents --- src/documents/bulk_download.py | 60 +++++++++++++++++ src/documents/serialisers.py | 65 ++++++++++++------- src/documents/tests/test_api.py | 111 ++++++++++++++++++++++++++++++++ src/documents/views.py | 58 ++++++++++++++++- src/paperless/urls.py | 6 +- 5 files changed, 274 insertions(+), 26 deletions(-) create mode 100644 src/documents/bulk_download.py diff --git a/src/documents/bulk_download.py b/src/documents/bulk_download.py new file mode 100644 index 000000000..8c675b4b5 --- /dev/null +++ b/src/documents/bulk_download.py @@ -0,0 +1,60 @@ +from zipfile import ZipFile + +from documents.models import Document + + +class BulkArchiveStrategy: + + def __init__(self, zipf: ZipFile): + self.zipf = zipf + + def make_unique_filename(self, + doc: Document, + archive: bool = False, + folder: str = ""): + counter = 0 + while True: + filename = folder + doc.get_public_filename(archive, counter) + if filename in self.zipf.namelist(): + counter += 1 + else: + return filename + + def add_document(self, doc: Document): + raise NotImplementedError() # pragma: no cover + + +class OriginalsOnlyStrategy(BulkArchiveStrategy): + + def add_document(self, doc: Document): + self.zipf.write(doc.source_path, self.make_unique_filename(doc)) + + +class ArchiveOnlyStrategy(BulkArchiveStrategy): + + def __init__(self, zipf): + super(ArchiveOnlyStrategy, self).__init__(zipf) + + def add_document(self, doc: Document): + if doc.has_archive_version: + self.zipf.write(doc.archive_path, + self.make_unique_filename(doc, archive=True)) + else: + self.zipf.write(doc.source_path, + self.make_unique_filename(doc)) + + +class OriginalAndArchiveStrategy(BulkArchiveStrategy): + + def add_document(self, doc: Document): + if doc.has_archive_version: + self.zipf.write( + doc.archive_path, self.make_unique_filename( + doc, archive=True, folder="archive/" + ) + ) + + self.zipf.write( + doc.source_path, + self.make_unique_filename(doc, folder="originals/") + ) diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py index b01f82805..66736bdbf 100644 --- a/src/documents/serialisers.py +++ b/src/documents/serialisers.py @@ -192,14 +192,34 @@ class SavedViewSerializer(serializers.ModelSerializer): return saved_view -class BulkEditSerializer(serializers.Serializer): +class DocumentListSerializer(serializers.Serializer): documents = serializers.ListField( - child=serializers.IntegerField(), + required=True, label="Documents", - write_only=True + write_only=True, + child=serializers.IntegerField() ) + def _validate_document_id_list(self, documents, name="documents"): + if not type(documents) == list: + raise serializers.ValidationError(f"{name} must be a list") + if not all([type(i) == int for i in documents]): + raise serializers.ValidationError( + f"{name} must be a list of integers") + count = Document.objects.filter(id__in=documents).count() + if not count == len(documents): + raise serializers.ValidationError( + f"Some documents in {name} don't exist or were " + f"specified twice.") + + def validate_documents(self, documents): + self._validate_document_id_list(documents) + return documents + + +class BulkEditSerializer(DocumentListSerializer): + method = serializers.ChoiceField( choices=[ "set_correspondent", @@ -215,18 +235,6 @@ class BulkEditSerializer(serializers.Serializer): parameters = serializers.DictField(allow_empty=True) - def _validate_document_id_list(self, documents, name="documents"): - if not type(documents) == list: - raise serializers.ValidationError(f"{name} must be a list") - if not all([type(i) == int for i in documents]): - raise serializers.ValidationError( - f"{name} must be a list of integers") - count = Document.objects.filter(id__in=documents).count() - if not count == len(documents): - raise serializers.ValidationError( - f"Some documents in {name} don't exist or were " - f"specified twice.") - def _validate_tag_id_list(self, tags, name="tags"): if not type(tags) == list: raise serializers.ValidationError(f"{name} must be a list") @@ -238,10 +246,6 @@ class BulkEditSerializer(serializers.Serializer): raise serializers.ValidationError( f"Some tags in {name} don't exist or were specified twice.") - def validate_documents(self, documents): - self._validate_document_id_list(documents) - return documents - def validate_method(self, method): if method == "set_correspondent": return bulk_edit.set_correspondent @@ -392,9 +396,24 @@ class PostDocumentSerializer(serializers.Serializer): return None -class SelectionDataSerializer(serializers.Serializer): +class BulkDownloadSerializer(DocumentListSerializer): - documents = serializers.ListField( - required=True, - child=serializers.IntegerField() + content = serializers.ChoiceField( + choices=["archive", "originals", "both"], + default="archive" ) + + compression = serializers.ChoiceField( + choices=["none", "deflated", "bzip2", "lzma"], + default="none" + ) + + def validate_compression(self, compression): + import zipfile + + return { + "none": zipfile.ZIP_STORED, + "deflated": zipfile.ZIP_DEFLATED, + "bzip2": zipfile.ZIP_BZIP2, + "lzma": zipfile.ZIP_LZMA + }[compression] diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py index 01e7210a5..7486154e1 100644 --- a/src/documents/tests/test_api.py +++ b/src/documents/tests/test_api.py @@ -1,7 +1,10 @@ +import datetime +import io import json import os import shutil import tempfile +import zipfile from unittest import mock from django.conf import settings @@ -1123,6 +1126,113 @@ class TestBulkEdit(DirectoriesMixin, APITestCase): self.assertCountEqual(response.data['selected_document_types'], [{"id": self.c1.id, "document_count": 1}, {"id": self.c2.id, "document_count": 0}]) +class TestBulkDownload(DirectoriesMixin, APITestCase): + + def setUp(self): + super(TestBulkDownload, self).setUp() + + user = User.objects.create_superuser(username="temp_admin") + self.client.force_login(user=user) + + self.doc1 = Document.objects.create(title="unrelated", checksum="A") + self.doc2 = Document.objects.create(title="document A", filename="docA.pdf", mime_type="application/pdf", checksum="B", created=datetime.datetime(2021, 1, 1)) + self.doc2b = Document.objects.create(title="document A", filename="docA2.pdf", mime_type="application/pdf", checksum="D", created=datetime.datetime(2021, 1, 1)) + self.doc3 = Document.objects.create(title="document B", filename="docB.jpg", mime_type="image/jpeg", checksum="C", created=datetime.datetime(2020, 3, 21), archive_filename="docB.pdf", archive_checksum="D") + + shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.pdf"), self.doc2.source_path) + shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.png"), self.doc2b.source_path) + shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "simple.jpg"), self.doc3.source_path) + shutil.copy(os.path.join(os.path.dirname(__file__), "samples", "test_with_bom.pdf"), self.doc3.archive_path) + + def test_download_originals(self): + response = self.client.post("/api/documents/bulk_download/", json.dumps({ + "documents": [self.doc2.id, self.doc3.id], + "content": "originals" + }), content_type='application/json') + + self.assertEqual(response.status_code, 200) + self.assertEqual(response['Content-Type'], 'application/zip') + + with zipfile.ZipFile(io.BytesIO(response.content)) as zipf: + self.assertEqual(len(zipf.filelist), 2) + self.assertIn("2021-01-01 document A.pdf", zipf.namelist()) + self.assertIn("2020-03-21 document B.jpg", zipf.namelist()) + + with self.doc2.source_file as f: + self.assertEqual(f.read(), zipf.read("2021-01-01 document A.pdf")) + + with self.doc3.source_file as f: + self.assertEqual(f.read(), zipf.read("2020-03-21 document B.jpg")) + + def test_download_default(self): + response = self.client.post("/api/documents/bulk_download/", json.dumps({ + "documents": [self.doc2.id, self.doc3.id] + }), content_type='application/json') + + self.assertEqual(response.status_code, 200) + self.assertEqual(response['Content-Type'], 'application/zip') + + with zipfile.ZipFile(io.BytesIO(response.content)) as zipf: + self.assertEqual(len(zipf.filelist), 2) + self.assertIn("2021-01-01 document A.pdf", zipf.namelist()) + self.assertIn("2020-03-21 document B.pdf", zipf.namelist()) + + with self.doc2.source_file as f: + self.assertEqual(f.read(), zipf.read("2021-01-01 document A.pdf")) + + with self.doc3.archive_file as f: + self.assertEqual(f.read(), zipf.read("2020-03-21 document B.pdf")) + + def test_download_both(self): + response = self.client.post("/api/documents/bulk_download/", json.dumps({ + "documents": [self.doc2.id, self.doc3.id], + "content": "both" + }), content_type='application/json') + + self.assertEqual(response.status_code, 200) + self.assertEqual(response['Content-Type'], 'application/zip') + + with zipfile.ZipFile(io.BytesIO(response.content)) as zipf: + self.assertEqual(len(zipf.filelist), 3) + self.assertIn("originals/2021-01-01 document A.pdf", zipf.namelist()) + self.assertIn("archive/2020-03-21 document B.pdf", zipf.namelist()) + self.assertIn("originals/2020-03-21 document B.jpg", zipf.namelist()) + + with self.doc2.source_file as f: + self.assertEqual(f.read(), zipf.read("originals/2021-01-01 document A.pdf")) + + with self.doc3.archive_file as f: + self.assertEqual(f.read(), zipf.read("archive/2020-03-21 document B.pdf")) + + with self.doc3.source_file as f: + self.assertEqual(f.read(), zipf.read("originals/2020-03-21 document B.jpg")) + + def test_filename_clashes(self): + response = self.client.post("/api/documents/bulk_download/", json.dumps({ + "documents": [self.doc2.id, self.doc2b.id] + }), content_type='application/json') + + self.assertEqual(response.status_code, 200) + self.assertEqual(response['Content-Type'], 'application/zip') + + with zipfile.ZipFile(io.BytesIO(response.content)) as zipf: + self.assertEqual(len(zipf.filelist), 2) + + self.assertIn("2021-01-01 document A.pdf", zipf.namelist()) + self.assertIn("2021-01-01 document A_01.pdf", zipf.namelist()) + + with self.doc2.source_file as f: + self.assertEqual(f.read(), zipf.read("2021-01-01 document A.pdf")) + + with self.doc2b.source_file as f: + self.assertEqual(f.read(), zipf.read("2021-01-01 document A_01.pdf")) + + def test_compression(self): + response = self.client.post("/api/documents/bulk_download/", json.dumps({ + "documents": [self.doc2.id, self.doc2b.id], + "compression": "lzma" + }), content_type='application/json') + class TestApiAuth(APITestCase): def test_auth_required(self): @@ -1146,4 +1256,5 @@ class TestApiAuth(APITestCase): self.assertEqual(self.client.get("/api/search/").status_code, 401) self.assertEqual(self.client.get("/api/search/auto_complete/").status_code, 401) self.assertEqual(self.client.get("/api/documents/bulk_edit/").status_code, 401) + self.assertEqual(self.client.get("/api/documents/bulk_download/").status_code, 401) self.assertEqual(self.client.get("/api/documents/selection_data/").status_code, 401) diff --git a/src/documents/views.py b/src/documents/views.py index 68d6e3c77..d886324ae 100755 --- a/src/documents/views.py +++ b/src/documents/views.py @@ -2,6 +2,7 @@ import logging import os import tempfile import uuid +import zipfile from datetime import datetime from time import mktime @@ -34,6 +35,8 @@ from rest_framework.viewsets import ( from paperless.db import GnuPG from paperless.views import StandardPagination +from .bulk_download import OriginalAndArchiveStrategy, OriginalsOnlyStrategy, \ + ArchiveOnlyStrategy from .classifier import load_classifier from .filters import ( CorrespondentFilterSet, @@ -51,7 +54,9 @@ from .serialisers import ( DocumentTypeSerializer, PostDocumentSerializer, SavedViewSerializer, - BulkEditSerializer, SelectionDataSerializer + BulkEditSerializer, + DocumentListSerializer, + BulkDownloadSerializer ) @@ -444,7 +449,7 @@ class PostDocumentView(APIView): class SelectionDataView(APIView): permission_classes = (IsAuthenticated,) - serializer_class = SelectionDataSerializer + serializer_class = DocumentListSerializer parser_classes = (parsers.MultiPartParser, parsers.JSONParser) def get_serializer_context(self): @@ -606,3 +611,52 @@ class StatisticsView(APIView): 'documents_total': documents_total, 'documents_inbox': documents_inbox, }) + + +class BulkDownloadView(APIView): + + permission_classes = (IsAuthenticated,) + serializer_class = BulkDownloadSerializer + parser_classes = (parsers.JSONParser,) + + def get_serializer_context(self): + return { + 'request': self.request, + 'format': self.format_kwarg, + 'view': self + } + + def get_serializer(self, *args, **kwargs): + kwargs['context'] = self.get_serializer_context() + return self.serializer_class(*args, **kwargs) + + def post(self, request, format=None): + serializer = self.get_serializer(data=request.data) + serializer.is_valid(raise_exception=True) + + ids = serializer.validated_data.get('documents') + compression = serializer.validated_data.get('compression') + content = serializer.validated_data.get('content') + + os.makedirs(settings.SCRATCH_DIR, exist_ok=True) + temp = tempfile.NamedTemporaryFile(dir=settings.SCRATCH_DIR, suffix="-compressed-archive", delete=False) + + if content == 'both': + strategy_class = OriginalAndArchiveStrategy + elif content == 'originals': + strategy_class = OriginalsOnlyStrategy + else: + strategy_class = ArchiveOnlyStrategy + + with zipfile.ZipFile(temp.name, "w", compression) as zipf: + strategy = strategy_class(zipf) + for id in ids: + doc = Document.objects.get(id=id) + strategy.add_document(doc) + + with open(temp.name, "rb") as f: + response = HttpResponse(f, content_type="application/zip") + response["Content-Disposition"] = '{}; filename="{}"'.format( + "attachment", "documents.zip") + + return response diff --git a/src/paperless/urls.py b/src/paperless/urls.py index 40f4bd754..4e0b8f191 100755 --- a/src/paperless/urls.py +++ b/src/paperless/urls.py @@ -23,7 +23,8 @@ from documents.views import ( PostDocumentView, SavedViewViewSet, BulkEditView, - SelectionDataView + SelectionDataView, + BulkDownloadView ) from paperless.views import FaviconView @@ -63,6 +64,9 @@ urlpatterns = [ re_path(r"^documents/selection_data/", SelectionDataView.as_view(), name="selection_data"), + re_path(r"^documents/bulk_download/", BulkDownloadView.as_view(), + name="bulk_download"), + path('token/', views.obtain_auth_token) ] + api_router.urls)), From 203bc162cd8807968f964bb35f75747397b8856c Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Sat, 20 Feb 2021 16:10:50 +0100 Subject: [PATCH 156/225] front end support for downloading multiple documents --- src-ui/messages.xlf | 49 +++++++++++-------- src-ui/package-lock.json | 5 ++ src-ui/package.json | 1 + .../bulk-editor/bulk-editor.component.html | 14 ++++++ .../bulk-editor/bulk-editor.component.ts | 7 +++ .../src/app/services/rest/document.service.ts | 4 ++ 6 files changed, 59 insertions(+), 21 deletions(-) diff --git a/src-ui/messages.xlf b/src-ui/messages.xlf index 731fdff7f..95e0fad15 100644 --- a/src-ui/messages.xlf +++ b/src-ui/messages.xlf @@ -1226,21 +1226,21 @@ Error executing bulk operation: src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 73 + 74 "" src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 112 + 113 "" and "" src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 114 + 115 This is for messages like 'modify "tag1" and "tag2"' @@ -1248,7 +1248,7 @@ , src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 116 + 117 this is used to separate enumerations and should probably be a comma and a whitespace in most languages @@ -1256,7 +1256,7 @@ and "" src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 117 + 118 this is for messages like 'modify "tag1", "tag2" and "tag3"' @@ -1264,112 +1264,112 @@ Confirm tags assignment src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 126 + 127 This operation will add the tag "" to selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 129 + 130 This operation will add the tags to selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 131 + 132 This operation will remove the tag "" from selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 134 + 135 This operation will remove the tags from selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 136 + 137 This operation will add the tags and remove the tags on selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 138 + 139 Confirm correspondent assignment src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 158 + 159 This operation will assign the correspondent "" to selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 160 + 161 This operation will remove the correspondent from selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 162 + 163 Confirm document type assignment src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 181 + 182 This operation will assign the document type "" to selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 183 + 184 This operation will remove the document type from selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 185 + 186 Delete confirm src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 200 + 201 This operation will permanently delete selected document(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 201 + 202 This operation cannot be undone. src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 202 + 203 Delete document(s) src/app/components/document-list/bulk-editor/bulk-editor.component.ts - 204 + 205 @@ -1393,6 +1393,13 @@ 27 + + Download originals + + src/app/components/document-list/bulk-editor/bulk-editor.component.html + 68 + + Suggestions: diff --git a/src-ui/package-lock.json b/src-ui/package-lock.json index c3827f80c..148bc72eb 100644 --- a/src-ui/package-lock.json +++ b/src-ui/package-lock.json @@ -5545,6 +5545,11 @@ "schema-utils": "^2.6.5" } }, + "file-saver": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/file-saver/-/file-saver-2.0.5.tgz", + "integrity": "sha512-P9bmyZ3h/PRG+Nzga+rbdI4OEpNDzAVyy74uVO9ATgzLK6VtAsYybF/+TOCvrc0MO793d6+42lLyZTw7/ArVzA==" + }, "file-uri-to-path": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz", diff --git a/src-ui/package.json b/src-ui/package.json index 14d828483..d6082c6b9 100644 --- a/src-ui/package.json +++ b/src-ui/package.json @@ -23,6 +23,7 @@ "@ng-bootstrap/ng-bootstrap": "^8.0.0", "@ng-select/ng-select": "^5.0.9", "bootstrap": "^4.5.0", + "file-saver": "^2.0.5", "ng-bootstrap": "^1.6.3", "ng2-pdf-viewer": "^6.3.2", "ngx-cookie-service": "^10.1.1", diff --git a/src-ui/src/app/components/document-list/bulk-editor/bulk-editor.component.html b/src-ui/src/app/components/document-list/bulk-editor/bulk-editor.component.html index 053126445..98716e2f9 100644 --- a/src-ui/src/app/components/document-list/bulk-editor/bulk-editor.component.html +++ b/src-ui/src/app/components/document-list/bulk-editor/bulk-editor.component.html @@ -56,6 +56,20 @@
    +
    + +
    + +
    + +
    +
    +
    +
    -
    +
    From 99a18516b283737cf81dbb28d69c07fca1694766 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Mon, 22 Feb 2021 00:17:16 +0100 Subject: [PATCH 183/225] tests --- src/paperless_tesseract/tests/test_parser.py | 41 ++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/src/paperless_tesseract/tests/test_parser.py b/src/paperless_tesseract/tests/test_parser.py index b64abb382..c389d446f 100644 --- a/src/paperless_tesseract/tests/test_parser.py +++ b/src/paperless_tesseract/tests/test_parser.py @@ -300,6 +300,47 @@ class TestParser(DirectoriesMixin, TestCase): "If you read this, it’s your own fault. Also check your screen orientation." ]) + def test_ocrmypdf_parameters(self): + parser = RasterisedDocumentParser(None) + params = parser.construct_ocrmypdf_parameters(input_file="input.pdf", output_file="output.pdf", + sidecar_file="sidecar.txt", mime_type="application/pdf", + safe_fallback=False) + + self.assertEqual(params['input_file'], "input.pdf") + self.assertEqual(params['output_file'], "output.pdf") + self.assertEqual(params['sidecar'], "sidecar.txt") + + with override_settings(OCR_CLEAN="none"): + params = parser.construct_ocrmypdf_parameters("", "", "", "") + self.assertNotIn("clean", params) + self.assertNotIn("clean_final", params) + + with override_settings(OCR_CLEAN="clean"): + params = parser.construct_ocrmypdf_parameters("", "", "", "") + self.assertTrue(params['clean']) + self.assertNotIn("clean_final", params) + + with override_settings(OCR_CLEAN="clean-final", OCR_MODE="skip"): + params = parser.construct_ocrmypdf_parameters("", "", "", "") + self.assertTrue(params['clean_final']) + self.assertNotIn("clean", params) + + with override_settings(OCR_CLEAN="clean-final", OCR_MODE="redo"): + params = parser.construct_ocrmypdf_parameters("", "", "", "") + self.assertTrue(params['clean']) + self.assertNotIn("clean_final", params) + + with override_settings(OCR_DESKEW=True, OCR_MODE="skip"): + params = parser.construct_ocrmypdf_parameters("", "", "", "") + self.assertTrue(params['deskew']) + + with override_settings(OCR_DESKEW=True, OCR_MODE="redo"): + params = parser.construct_ocrmypdf_parameters("", "", "", "") + self.assertNotIn('deskew', params) + + with override_settings(OCR_DESKEW=False, OCR_MODE="skip"): + params = parser.construct_ocrmypdf_parameters("", "", "", "") + self.assertNotIn('deskew', params) class TestParserFileTypes(DirectoriesMixin, TestCase): From d64818b46cc29e08da16a84fd944bb323e498c58 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Mon, 22 Feb 2021 11:11:04 +0100 Subject: [PATCH 184/225] fixes #591 --- .../management/commands/document_archiver.py | 12 +++++++++--- src/documents/tests/test_management.py | 15 +++++++++++++++ 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/src/documents/management/commands/document_archiver.py b/src/documents/management/commands/document_archiver.py index 0d3a5e82d..6a6056ed7 100644 --- a/src/documents/management/commands/document_archiver.py +++ b/src/documents/management/commands/document_archiver.py @@ -31,6 +31,11 @@ def handle_document(document_id): parser_class = get_parser_class_for_mime_type(mime_type) + if not parser_class: + logger.error(f"No parser found for mime type {mime_type}, cannot " + f"archive document {document} (ID: {document_id})") + return + parser = parser_class(logging_group=uuid.uuid4()) try: @@ -66,11 +71,12 @@ def handle_document(document_id): document.archive_path) shutil.move(thumbnail, document.thumbnail_path) - with AsyncWriter(index.open_index()) as writer: - index.update_document(writer, document) + with index.open_index_writer() as writer: + index.update_document(writer, document) except Exception as e: - logger.exception(f"Error while parsing document {document}") + logger.exception(f"Error while parsing document {document} " + f"(ID: {document_id})") finally: parser.cleanup() diff --git a/src/documents/tests/test_management.py b/src/documents/tests/test_management.py index 40eaaf277..f7beb8907 100644 --- a/src/documents/tests/test_management.py +++ b/src/documents/tests/test_management.py @@ -49,6 +49,21 @@ class TestArchiver(DirectoriesMixin, TestCase): self.assertTrue(filecmp.cmp(sample_file, doc.source_path)) self.assertEqual(doc.archive_filename, "none/A.pdf") + def test_unknown_mime_type(self): + doc = self.make_models() + doc.mime_type = "sdgfh" + doc.save() + shutil.copy(sample_file, doc.source_path) + + handle_document(doc.pk) + + doc = Document.objects.get(id=doc.id) + + self.assertIsNotNone(doc.checksum) + self.assertIsNone(doc.archive_checksum) + self.assertIsNone(doc.archive_filename) + self.assertTrue(os.path.isfile(doc.source_path)) + @override_settings(PAPERLESS_FILENAME_FORMAT="{title}") def test_naming_priorities(self): doc1 = Document.objects.create(checksum="A", title="document", content="first document", mime_type="application/pdf", filename="document.pdf") From aa3d91a338a66b39774acdfdc092be5f1672ff61 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Mon, 22 Feb 2021 11:38:16 +0100 Subject: [PATCH 185/225] associate error messages with documents --- .../management/commands/document_archiver.py | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/documents/management/commands/document_archiver.py b/src/documents/management/commands/document_archiver.py index 6a6056ed7..497e8fe7e 100644 --- a/src/documents/management/commands/document_archiver.py +++ b/src/documents/management/commands/document_archiver.py @@ -4,7 +4,9 @@ import multiprocessing import logging import os import shutil +import sys import uuid +from io import TextIOBase import tqdm from django import db @@ -12,7 +14,6 @@ from django.conf import settings from django.core.management.base import BaseCommand from django.db import transaction from filelock import FileLock -from whoosh.writing import AsyncWriter from documents.models import Document from ... import index @@ -24,11 +25,28 @@ from ...parsers import get_parser_class_for_mime_type logger = logging.getLogger("paperless.management.archiver") +class LoggerWriter(TextIOBase): + + def __init__(self, doc: Document): + self.doc = doc + + def write(self, message): + # if statement reduces the amount of newlines that are + # printed to the logger + if message != '\n': + logger.error( + f"Document {self.doc} (ID: {self.doc.pk}): {message}" + ) + + def handle_document(document_id): document = Document.objects.get(id=document_id) mime_type = document.mime_type + # redirect errors to the log and associate them with the current document + sys.stderr = LoggerWriter(document) + parser_class = get_parser_class_for_mime_type(mime_type) if not parser_class: From 6d934da5dd3e14d6cd6120574790cfbf75bdeb5a Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Mon, 22 Feb 2021 11:52:54 +0100 Subject: [PATCH 186/225] Revert "associate error messages with documents" This reverts commit aa3d91a3 --- .../management/commands/document_archiver.py | 20 +------------------ 1 file changed, 1 insertion(+), 19 deletions(-) diff --git a/src/documents/management/commands/document_archiver.py b/src/documents/management/commands/document_archiver.py index 497e8fe7e..6a6056ed7 100644 --- a/src/documents/management/commands/document_archiver.py +++ b/src/documents/management/commands/document_archiver.py @@ -4,9 +4,7 @@ import multiprocessing import logging import os import shutil -import sys import uuid -from io import TextIOBase import tqdm from django import db @@ -14,6 +12,7 @@ from django.conf import settings from django.core.management.base import BaseCommand from django.db import transaction from filelock import FileLock +from whoosh.writing import AsyncWriter from documents.models import Document from ... import index @@ -25,28 +24,11 @@ from ...parsers import get_parser_class_for_mime_type logger = logging.getLogger("paperless.management.archiver") -class LoggerWriter(TextIOBase): - - def __init__(self, doc: Document): - self.doc = doc - - def write(self, message): - # if statement reduces the amount of newlines that are - # printed to the logger - if message != '\n': - logger.error( - f"Document {self.doc} (ID: {self.doc.pk}): {message}" - ) - - def handle_document(document_id): document = Document.objects.get(id=document_id) mime_type = document.mime_type - # redirect errors to the log and associate them with the current document - sys.stderr = LoggerWriter(document) - parser_class = get_parser_class_for_mime_type(mime_type) if not parser_class: From 3b553f645523efdbfdf3ef879a7d3ad6bc64856e Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Mon, 22 Feb 2021 11:53:13 +0100 Subject: [PATCH 187/225] changelog --- docs/changelog.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index abe667400..b23ca5137 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -25,6 +25,8 @@ paperless-ng 1.2.0 with polling from 1 to 5 seconds. This will decrease the likelihood of paperless consuming partially written files. +* Fixed a crash of the document archiver management command when trying to process documents with unknown mime types. + * Paperless no longer depends on ``libpoppler-cpp-dev``. .. note:: From 127d30918d5dcb04f7ac9877f893a2a26df0775c Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Mon, 22 Feb 2021 12:03:07 +0100 Subject: [PATCH 188/225] lets hope this works! --- .github/workflows/ci.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 87b527cad..d816687ca 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -112,6 +112,13 @@ jobs: uses: actions/setup-node@v2 with: node-version: '15' + - + name: Configure version on dev branches + if: startsWith(github.ref, 'refs/heads/feature-') || github.ref == 'refs/heads/dev' + run: | + git_hash=$(git rev-parse --short "$GITHUB_SHA") + git_branch=${GITHUB_REF#refs/heads/} + sed -i -E 's/version: "(.*)"/version: "$git_branch $git_hash"/g' src-ui/src/environments/environment.prod.ts - name: Build frontend run: ./compile-frontend.sh From 8da85d360955857d969df0afbc29b0a21d360a37 Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Mon, 22 Feb 2021 13:25:21 +0100 Subject: [PATCH 189/225] Update ci.yml --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d816687ca..dbb718cc8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -118,7 +118,7 @@ jobs: run: | git_hash=$(git rev-parse --short "$GITHUB_SHA") git_branch=${GITHUB_REF#refs/heads/} - sed -i -E 's/version: "(.*)"/version: "$git_branch $git_hash"/g' src-ui/src/environments/environment.prod.ts + sed -i -E "s/version: \"(.*)\"/version: \"${git_branch} ${git_hash}\"/g" src-ui/src/environments/environment.prod.ts - name: Build frontend run: ./compile-frontend.sh From 8797a58efc76fa3da7dc708fdd4899034d9abbf5 Mon Sep 17 00:00:00 2001 From: Joel Nordell Date: Mon, 22 Feb 2021 22:20:41 -0600 Subject: [PATCH 190/225] Add apple-touch-icon for iOS devices "Add to Home Screen" --- src-ui/angular.json | 2 ++ src-ui/src/apple-touch-icon.png | Bin 0 -> 6272 bytes src-ui/src/index.html | 1 + 3 files changed, 3 insertions(+) create mode 100644 src-ui/src/apple-touch-icon.png diff --git a/src-ui/angular.json b/src-ui/angular.json index a26dd8713..edf428fc7 100644 --- a/src-ui/angular.json +++ b/src-ui/angular.json @@ -36,6 +36,7 @@ "aot": true, "assets": [ "src/favicon.ico", + "src/apple-touch-icon.png", "src/assets", "src/manifest.webmanifest", { "glob": "pdf.worker.min.js", @@ -112,6 +113,7 @@ "karmaConfig": "karma.conf.js", "assets": [ "src/favicon.ico", + "src/apple-touch-icon.png", "src/assets", "src/manifest.webmanifest" ], diff --git a/src-ui/src/apple-touch-icon.png b/src-ui/src/apple-touch-icon.png new file mode 100644 index 0000000000000000000000000000000000000000..46388a873d15b42999b0a173c8d33a224b80ca51 GIT binary patch literal 6272 zcmb_h^;cBU7QQe#ghQh;lz@tWbW3+jch^uuNew-SfFPw35&{AO0y2OINT-y5!cY=J zNq494uJ4!k54^k9y?4!7_ndY1*?WKc`}V|WYbuiv(-H#!K%%OmpbNfR|J{U8@ToiP zN&voYc+08k6M|PTp?x&?o5)ke#2Wx^n*F=+p1*!64-Ue86pelKJRE#bHeLt-g+g&V zyCc2rY&;R%9$t=_+Y+<@aQCmOf~>y(`>iZT8R~J`sM#5WiFE-7i!gbRtoF!+5UWck zV^apreVQQsyouX!T+oWxhe~lm$DN-#Cv3^aob&}JDijp*Ke0t`e%bu{+||6c0^#78 zdX{aqnJ(ann+~#E>1UsPw7<4i1eGPE<{QT+3|H>~B0exf$it&xGK94J0JGEod7$Ez z0_VJGxK0mn19^IJn(L-Qhe6Op!elDCJ~7_c{!z|4&L1ZSWirlfL_Ill8X__HVK8s{ z>fiT|6^`S=Mr2>PER}k3eP^==!k>kg-pEC5WCLLwOZfhOnKhtHG>R;#)`N4uGc6U zYw=!RjKMB<_j4936NEgDxAa=@26SX7XrR^}PFKpuzz^66&19lvTWUgJmM2Fn5c2QI z(9w0fwhGaF-(dCgNtO7s*LZ}ndk4kQLW@T2y+gfDk~~IQ(TbOwE|>a>yNp^!TCVTE z+$oq=zhh#=lG-}@eH2>K{}LvX7T++LX$! zB!q$+Qi+t?oa|pn2{1Rwk6$Pgk-5g=tUc?oH16E)oNQkh10m$B^NS|5Q!O|rofQJ@ zKQH->e$j=oQKIijyS_4wRYP2IxyvuJ*=dr$0N2hRDW{D)qC(54itv<9K3gcpb0Sor z`RJXLaJshf)DRU4c)s1o0*6p0AMzLl8k{d1>j}Jj0zXydf$c|zLNHiNlRZ=Khj!xo zIE^GA*@d6OWSl-(=F%F*na)}%qhjmo5Ornh zJP){z0$l4}Cuw%%G0CJCmtrgdoUAU@nw(#TPv!w-?hZlD!bYf`cDsp)AoI9HkAG|R z4bT^x^wK%6(xe6N46F-+K`%SZKm6ZGVOcD9V)lL$K2wK3TP;_b2vVYyiZHo@gXU{rf?b=$j3sVtd>%k}_%a&Ffd z#-gqX-)&UC$(0%2KWcLWGcW?y>J9oq$?slNxfCN_ap%?5PT=V#g~tgVo;_fxSFT2$ z8fPBXhk$%bwdE7RxMifJw7jp(f{_W~#Suw_4rR$z&!zJL)dU%L3Mw0It{D0L=w9emcX^bG~R zDR=;RY-)AyIvaEO9Cqdd4mYiW59~M-%q61Y^~UlN&|In1rb?hUEq?;w{sxKMMi{-L zf{})&vWRTT$COkRb43r?)|MkJs%koByAu^tQlIUOA#mIHT%e^C3A)M}TjIT&o)3(6 z3YlzGAH3Z$SFGMQsC?+c;F0tz>-+*XPWcM_5-(Khb|t@8v5ARca)|`zG-Kj-@l?6u zK6#buzzNX+5|87{jYW+<6U;+uB6w>Tm;t1>THfD6)vdwx+zV&gD-?Kj7@|z_Gjt-6sm+>v%Qf(-|Qub3mMT&G`(x(R2FJp>A z<%>QgR*RvzT4*PDovUYjLIF{)J<)yr0P`QbM1hbBnH5%sN}oUYRitH|*I($XM7m_l zu6Pnc!Qhyn{hMzhpcHI9`x^po1I`b0__z;K{Ae+~K|qCiRx;|+KTZAu(O~G5auJx) zw!C`8wq*I1B>nT7(*EzyrO_*RB@6IG_A)7Viev%K;Aa8!V> z{g=MnTgrRY`x`FTkGRSqB%*g$*Ki%g%sj;SMrqz=QBtJv2RB$0#NUHK2D zp+djvRN4M$UdDv6CiPW>E^s6Z*=3`*Ya)%`d{ZlG^wqxAe{M_rV%=5C<=m}d;_8yy zZx0&1Dn)Pmh6Z%JV8HtUVQfeKz^kFEy?!8cokiPNlry9#;*Vh=X}IQzXqX z{tn3;SU(tMU0J-npDAJ|T*G_O#Uim?hToU~GR!Mbi^ znm6z}G?;+VQEJ{aN^y4*&?R7(x~@nxGyy?abcU7})lMDQ6QcZBHm>iB0Mc#4!(zZ+ei-ae6BSK?2;=>jP{QYHfr^1wb-! z>YplHygj3Q!u!Og1c&r~*_k#?I{U=;2$o=V^#TbuY%_xd8@G`B(TqB2CEGrY9B6-j z8kiz$Y=(6R0rQ5Y**_U~dq~PHaPdAP#=%T&dhEX3ij3}j$Q>w^sE zToP*j6@>_ECv;NMucsCQRCRskl5P?CI`L2q^~JcEN9gbA$=>EatZ!J-eo(X)NZLr< zJ8av;!U0qlxWRJ~a;u4ycmC7Rg`ohUq< zRa4{+mF(?rwUMNF-LlC~qf1cy`{$qgC-;hMrcaXUQrkcy!fJ!qyCzHM$fE<~2dusp zYuB$vyT%luu`xP-wk%&&RS|0>74>zzi*1o}IKG7`s36{fG5PXJe)!D(kGpq>k@0UZ z>d8@VOQW=lTvdwj8h>Ds><2Ym9Kur!ac%#+GN!JEKG)tr>CeVE8PrDho)~d1ki1UH zd?Ec-%O$0*$NbmNxEK4u4nNaFj86|JF^Gv9jI=VgzI5o>hNxR2JTBFBwmyx$benze z_$_z9Lggdep`%F ze5j|U2;2CYeVAk8CM{sa_S64-pS^t%db?rN(r$MUD8F4l6*HjtfO7r^MxOkR2qBMU z(@64oDP@H7-$Lqu76sSKM)&gmId_>@{(ZHX-&V;nh>)=@(Ton7tJ%cTd;PUbBJJl?3-kMD&L z=Nn9!nWGCqi_-!jfQ+>agOBqtg-&VFqT#vQ)4GOf5)hMVG>qLi-}pKh7LHw_wU_EY z8Q0_6Ww<@$0V^4zCnjQ&=W37X< ztfa8#Gr|!w0==$ZR#yrFfAZcU!oiuAayH^>?$T%z?2Nib$lpy^wdYB{bEg2sc6eGh zA8)l2`cb5GCIX%->@b*m^VAPZk3o2nUc)?hXcwP25HzVuP&5Tb8NN0ZxogyvYoIZ6 zwhBV6%4lGJ^jGG}%oQ8lv7-5pxtomwt;p}l=}`ZdL&D;ME-#63OV3t@EFN4IX1%F& z{IM;j%o`sPEv|NLru+|FFv87XIaVmkPPi%k-J2^?1oSPaYLcs=pr+svi@*a?FTdcI zds#VSViau^j0YPFz{oITM2x1`XO~1#2=a&^K!qJ`LVVLNdqmm{CD{OpLOT^s;ZILwMoYX~$-wwVVvUd2kE zA2%}rJ%dA1t;HhYeew zdOlY(wqv$dySH+5D^6luUl;3Pr*P@?m3D@-;VoqV-;}=j{H$&vMHgE7CQ0c_UP>9G z?FY(+nl*y~mLZQ@rxLCrUb}4G3s$Z7B7S^w85ZJ5rdw1?;dcN9i19@fIpGk4=DzhB zuM}d@AMdOCllK;iaUStjZz+9ip7KV5nrH03bnez^!m0n{qtmVa(Dh5i~6VTg;y%(KqiSu;`rI`YL-15j{mk zR+XZ{f7pCG-%=b5EtT2D7kaWT(bqrSDnOm;O|`HdcWp);W+hp9*KmGl)%|{>ArmST z0lKk5(JI}Sde>n4u>vPNzxF^}g}%abf<9|7CnJ<@PyC%gUiw_NtsU^e+Ft;~ASe%w z&G2jc^Lh`Mk_OFK069KneKlE2Ev;`dkH#kGRSF*Q?~r|?ubIDouh!w*xQDCGTjcbq=X_!?CKKi60fX5}nOstzS6uT%tsz;a{*aYau(Yz#0YMA$O`BHef-b4*H zr&gOBt(`<^4atVs6R#0Z7=I0Q*eX{F%P_ywOB%BA5^oB;$E2+@5!!^;Y z{h;eBP~A2=8r*Lq3Fig0D~zlEPLKjJVIB{Y_YQ~ZxhZ3KJ)B@*brCz!v^pVivtcVF z>;eGKB3FP0}nE!j9ERY=4ZMKYdOO5Hcle5MzdQ_x+u z`RBzsT4TjIMSb#z>u_PL@&uzS8~`*6s3S;RZ;!}PjoF*Fu!GW|Ko2$0d~|xjc%V~4 zbY*=K^t^uwCCJ6z3bU<(57G~Ei$4dea;`SV9GmY*T__0nK4~NIgUiG~TL=VbjTGr%3dynp zN%8um$;jEwa3&J%Q-%+TZp!Gw({p^e*HHg`IW+9=yC)`TM|N@XqvYl+DiGKvIi@!F zszy<4C@*Fqeh#)9>ZyLCr5FD|SY+5;m(m%JMHCXer<0~>(kyn*t|jwELH5>=!k&^g z6l`GrBbl0K=a5WKu_aD|SA-b$8UkC`Gw=Yx4AAGeZJ&_P6MTek)eny|7h7gixlu=c zEQ}d^xSISfjADdb<=)6yWc(b745d$}<1} literal 0 HcmV?d00001 diff --git a/src-ui/src/index.html b/src-ui/src/index.html index b05f9123b..39ecd824c 100644 --- a/src-ui/src/index.html +++ b/src-ui/src/index.html @@ -9,6 +9,7 @@ + From f397c5472c014dead0650aa82fd7df0fff18554b Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Tue, 23 Feb 2021 12:35:24 +0100 Subject: [PATCH 191/225] validation for regular expressions on matching models #605 --- src/documents/serialisers.py | 39 +++++++++++++------------- src/documents/tests/test_api.py | 37 +++++++++++++++++++++++- src/locale/en_US/LC_MESSAGES/django.po | 10 +++++-- 3 files changed, 63 insertions(+), 23 deletions(-) diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py index 66736bdbf..652e5fe30 100644 --- a/src/documents/serialisers.py +++ b/src/documents/serialisers.py @@ -1,11 +1,13 @@ +import re + import magic from django.utils.text import slugify from rest_framework import serializers from rest_framework.fields import SerializerMethodField from . import bulk_edit -from .models import Correspondent, Tag, Document, Log, DocumentType, \ - SavedView, SavedViewFilterRule +from .models import Correspondent, Tag, Document, DocumentType, \ + SavedView, SavedViewFilterRule, MatchingModel from .parsers import is_mime_type_supported from django.utils.translation import gettext as _ @@ -33,16 +35,27 @@ class DynamicFieldsModelSerializer(serializers.ModelSerializer): self.fields.pop(field_name) -class CorrespondentSerializer(serializers.ModelSerializer): +class MatchingModelSerializer(serializers.ModelSerializer): document_count = serializers.IntegerField(read_only=True) - last_correspondence = serializers.DateTimeField(read_only=True) - def get_slug(self, obj): return slugify(obj.name) slug = SerializerMethodField() + def validate_match(self, match): + if 'matching_algorithm' in self.initial_data and self.initial_data['matching_algorithm'] == MatchingModel.MATCH_REGEX: # NOQA: E501 + try: + re.compile(match) + except Exception as e: + raise serializers.ValidationError(_("Invalid regular expresssion: ") + str(e)) + return match + + +class CorrespondentSerializer(MatchingModelSerializer): + + last_correspondence = serializers.DateTimeField(read_only=True) + class Meta: model = Correspondent fields = ( @@ -57,13 +70,7 @@ class CorrespondentSerializer(serializers.ModelSerializer): ) -class DocumentTypeSerializer(serializers.ModelSerializer): - - document_count = serializers.IntegerField(read_only=True) - - def get_slug(self, obj): - return slugify(obj.name) - slug = SerializerMethodField() +class DocumentTypeSerializer(MatchingModelSerializer): class Meta: model = DocumentType @@ -78,13 +85,7 @@ class DocumentTypeSerializer(serializers.ModelSerializer): ) -class TagSerializer(serializers.ModelSerializer): - - document_count = serializers.IntegerField(read_only=True) - - def get_slug(self, obj): - return slugify(obj.name) - slug = SerializerMethodField() +class TagSerializer(MatchingModelSerializer): class Meta: model = Tag diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py index 7486154e1..cbd10ac71 100644 --- a/src/documents/tests/test_api.py +++ b/src/documents/tests/test_api.py @@ -14,7 +14,7 @@ from rest_framework.test import APITestCase from whoosh.writing import AsyncWriter from documents import index, bulk_edit -from documents.models import Document, Correspondent, DocumentType, Tag, SavedView +from documents.models import Document, Correspondent, DocumentType, Tag, SavedView, MatchingModel from documents.tests.utils import DirectoriesMixin @@ -772,6 +772,41 @@ class TestDocumentApi(DirectoriesMixin, APITestCase): self.assertEqual(response.status_code, 200) self.assertListEqual(response.data, ["test", "test2"]) + def test_invalid_regex_other_algorithm(self): + for endpoint in ['correspondents', 'tags', 'document_types']: + response = self.client.post(f"/api/{endpoint}/", { + "name": "test", + "matching_algorithm": MatchingModel.MATCH_ANY, + "match": "[" + }, format='json') + self.assertEqual(response.status_code, 201, endpoint) + + def test_invalid_regex(self): + for endpoint in ['correspondents', 'tags', 'document_types']: + response = self.client.post(f"/api/{endpoint}/", { + "name": "test", + "matching_algorithm": MatchingModel.MATCH_REGEX, + "match": "[" + }, format='json') + self.assertEqual(response.status_code, 400, endpoint) + + def test_valid_regex(self): + for endpoint in ['correspondents', 'tags', 'document_types']: + response = self.client.post(f"/api/{endpoint}/", { + "name": "test", + "matching_algorithm": MatchingModel.MATCH_REGEX, + "match": "[0-9]" + }, format='json') + self.assertEqual(response.status_code, 201, endpoint) + + def test_regex_no_algorithm(self): + for endpoint in ['correspondents', 'tags', 'document_types']: + response = self.client.post(f"/api/{endpoint}/", { + "name": "test", + "match": "[0-9]" + }, format='json') + self.assertEqual(response.status_code, 201, endpoint) + class TestBulkEdit(DirectoriesMixin, APITestCase): diff --git a/src/locale/en_US/LC_MESSAGES/django.po b/src/locale/en_US/LC_MESSAGES/django.po index cb31bbc8c..c234d5c2a 100644 --- a/src/locale/en_US/LC_MESSAGES/django.po +++ b/src/locale/en_US/LC_MESSAGES/django.po @@ -8,7 +8,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-16 14:52+0100\n" +"POT-Creation-Date: 2021-02-23 12:30+0100\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" @@ -346,7 +346,11 @@ msgstr "" msgid "filter rules" msgstr "" -#: documents/serialisers.py:370 +#: documents/serialisers.py:52 +msgid "Invalid regular expresssion: " +msgstr "" + +#: documents/serialisers.py:376 #, python-format msgid "File type %(type)s not supported" msgstr "" @@ -411,7 +415,7 @@ msgstr "" msgid "French" msgstr "" -#: paperless/urls.py:114 +#: paperless/urls.py:118 msgid "Paperless-ng administration" msgstr "" From 25366af4bb29892bcd8804658170e80b075de542 Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Tue, 23 Feb 2021 13:03:20 +0100 Subject: [PATCH 192/225] Update CONTRIBUTING.md --- CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a8fb1f8e9..0bbd6c8ea 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -9,11 +9,11 @@ If you want to implement something big: Please start a discussion about that in ## Python -Use python 3.6 for development. Paperless supports python 3.6, 3.7 and 3.8. +Paperless supports python 3.6, 3.7, 3.8 and 3.9. ## Branches -master always reflects the latest release. +master always reflects the latest release. Apart from changes to the documentation or readme, absolutely no functional changes on this branch in between releases. dev contains all changes that will be part of the next release. Use this branch to start making your changes. From 5d94a983d2da4b1e180e91ae4c9290a7ebd91eb1 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Tue, 23 Feb 2021 13:09:41 +0100 Subject: [PATCH 193/225] move codestyle checks into separate job --- .github/workflows/ci.yml | 44 ++++++++++++++++++++++++++++++++-------- 1 file changed, 36 insertions(+), 8 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dbb718cc8..cb61b753b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -20,7 +20,7 @@ jobs: name: Set up Python uses: actions/setup-python@v2 with: - python-version: 3.8 + python-version: 3.7 - name: Get pip cache dir id: pip-cache @@ -49,6 +49,39 @@ jobs: name: documentation path: docs/_build/html/ + codestyle: + runs-on: ubuntu-20.04 + steps: + - + name: Checkout + uses: actions/checkout@v2 + - + name: Set up Python + uses: actions/setup-python@v2 + with: + python-version: 3.7 + - + name: Get pip cache dir + id: pip-cache + run: | + echo "::set-output name=dir::$(pip cache dir)" + - + name: Persistent Github pip cache + uses: actions/cache@v2 + with: + path: ${{ steps.pip-cache.outputs.dir }} + key: ${{ runner.os }}-pip${{ matrix.python-version }} + - + name: Prepare tests + run: | + pip install --upgrade pipenv + pipenv install --system --dev --ignore-pipfile + - + name: Codestyle + run: | + cd src/ + pycodestyle + tests: runs-on: ubuntu-20.04 strategy: @@ -87,11 +120,6 @@ jobs: run: | cd src/ pytest - - - name: Codestyle - run: | - cd src/ - pycodestyle - name: Publish coverage results if: matrix.python-version == '3.8' @@ -130,7 +158,7 @@ jobs: path: src/documents/static/frontend/ build-release: - needs: [frontend, documentation, tests] + needs: [frontend, documentation, tests, codestyle] runs-on: ubuntu-20.04 steps: - @@ -240,7 +268,7 @@ jobs: build-docker-image: if: github.event_name == 'push' && (startsWith(github.ref, 'refs/heads/feature-') || github.ref == 'refs/heads/dev' || startsWith(github.ref, 'refs/tags/ng-')) runs-on: ubuntu-latest - needs: [frontend, tests] + needs: [frontend, tests, codestyle] steps: - name: Prepare From 5ce73574c8ba1cadcac0d4450fd58479bacb17a1 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Tue, 23 Feb 2021 12:12:00 +0000 Subject: [PATCH 194/225] Apply translations in de translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'de' language. --- src/locale/de/LC_MESSAGES/django.po | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/locale/de/LC_MESSAGES/django.po b/src/locale/de/LC_MESSAGES/django.po index 358c9b721..d1a07d18f 100644 --- a/src/locale/de/LC_MESSAGES/django.po +++ b/src/locale/de/LC_MESSAGES/django.po @@ -11,7 +11,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-16 14:52+0100\n" +"POT-Creation-Date: 2021-02-23 12:30+0100\n" "PO-Revision-Date: 2021-02-16 18:37+0000\n" "Last-Translator: Jonas Winkler, 2021\n" "Language-Team: German (https://www.transifex.com/paperless/teams/115905/de/)\n" @@ -354,7 +354,11 @@ msgstr "Filterregel" msgid "filter rules" msgstr "Filterregeln" -#: documents/serialisers.py:370 +#: documents/serialisers.py:52 +msgid "Invalid regular expresssion: " +msgstr "Ungültiger regulärer Ausdruck:" + +#: documents/serialisers.py:376 #, python-format msgid "File type %(type)s not supported" msgstr "Dateityp %(type)s nicht unterstützt" @@ -421,7 +425,7 @@ msgstr "Niederländisch" msgid "French" msgstr "Französisch" -#: paperless/urls.py:114 +#: paperless/urls.py:118 msgid "Paperless-ng administration" msgstr "Paperless-ng Administration" From 065ff6eaf5c212bc2a19adf1abb78e1c2aab8fc3 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Tue, 23 Feb 2021 13:13:12 +0100 Subject: [PATCH 195/225] rename some steps --- .github/workflows/ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cb61b753b..85db2d482 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -72,7 +72,7 @@ jobs: path: ${{ steps.pip-cache.outputs.dir }} key: ${{ runner.os }}-pip${{ matrix.python-version }} - - name: Prepare tests + name: Install dependencies run: | pip install --upgrade pipenv pipenv install --system --dev --ignore-pipfile @@ -109,7 +109,7 @@ jobs: path: ${{ steps.pip-cache.outputs.dir }} key: ${{ runner.os }}-pip${{ matrix.python-version }} - - name: Prepare tests + name: Install dependencies run: | sudo apt-get update -qq sudo apt-get install -qq --no-install-recommends unpaper tesseract-ocr imagemagick ghostscript optipng From f888647b12deec0a63bd3b35a7bd3c288a10290e Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Tue, 23 Feb 2021 13:13:38 +0100 Subject: [PATCH 196/225] front end support for displaying error messages about regular expressions. --- .../correspondent-edit-dialog.component.html | 2 +- .../document-type-edit-dialog.component.html | 2 +- .../tag-list/tag-edit-dialog/tag-edit-dialog.component.html | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src-ui/src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html b/src-ui/src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html index bbc2c8453..26a6dc219 100644 --- a/src-ui/src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html +++ b/src-ui/src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html @@ -8,7 +8,7 @@
    - +
    diff --git a/src-ui/src/app/components/manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component.html b/src-ui/src/app/components/manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component.html index ed9f4f5af..55cd3de5b 100644 --- a/src-ui/src/app/components/manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component.html +++ b/src-ui/src/app/components/manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component.html @@ -9,7 +9,7 @@ - +
    diff --git a/src-ui/src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.html b/src-ui/src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.html index fd4c3b7f0..418a5c281 100644 --- a/src-ui/src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.html +++ b/src-ui/src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.html @@ -20,7 +20,7 @@ - +
    From 49b23aafa34a821f04c936f6b70ff9dcbcd0efe6 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Tue, 23 Feb 2021 13:16:56 +0100 Subject: [PATCH 197/225] documentation typos --- docs/configuration.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/configuration.rst b/docs/configuration.rst index 56537035c..e26180382 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -278,7 +278,7 @@ PAPERLESS_OCR_DESKEW= PAPERLESS_OCR_ROTATE_PAGES= Tells paperless to correct page rotation (90°, 180° and 270° rotation). - If you notice that paperless is not rotating pages incorrectly rotated + If you notice that paperless is not rotating incorrectly rotated pages (or vice versa), try adjusting the threshold up or down (see below). Defaults to ``true``, which enables this feature. @@ -287,7 +287,7 @@ PAPERLESS_OCR_ROTATE_PAGES= PAPERLESS_OCR_ROTATE_PAGES_THRESHOLD= Adjust the threshold for automatic page rotation by ``PAPERLESS_OCR_ROTATE_PAGES``. This is an arbitrary value reported by tesseract. "15" is a very conservative value, - whereas "2" is a very aggressive option and will often result correctly rotated pages + whereas "2" is a very aggressive option and will often result in correctly rotated pages being rotated as well. Defaults to "12". @@ -338,7 +338,7 @@ PAPERLESS_OCR_USER_ARGS= the API of OCRmyPDF, you have to specify these in a format that can be passed to the API. See `the API reference of OCRmyPDF `_ for valid parameters. All command line options are supported, but they - use underscores instead of dashed. + use underscores instead of dashes. .. caution:: From 7a73e1859660c68b3ca254cd879aa188e7d561c5 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Tue, 23 Feb 2021 13:21:11 +0100 Subject: [PATCH 198/225] fix pycodestyle and messages --- src/documents/serialisers.py | 5 ++++- src/locale/en_US/LC_MESSAGES/django.po | 7 ++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py index 652e5fe30..adda7117e 100644 --- a/src/documents/serialisers.py +++ b/src/documents/serialisers.py @@ -48,7 +48,10 @@ class MatchingModelSerializer(serializers.ModelSerializer): try: re.compile(match) except Exception as e: - raise serializers.ValidationError(_("Invalid regular expresssion: ") + str(e)) + raise serializers.ValidationError( + _("Invalid regular expresssion: %(error)s") % + {'error': str(e)} + ) return match diff --git a/src/locale/en_US/LC_MESSAGES/django.po b/src/locale/en_US/LC_MESSAGES/django.po index c234d5c2a..27d628455 100644 --- a/src/locale/en_US/LC_MESSAGES/django.po +++ b/src/locale/en_US/LC_MESSAGES/django.po @@ -8,7 +8,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-23 12:30+0100\n" +"POT-Creation-Date: 2021-02-23 13:20+0100\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" @@ -347,10 +347,11 @@ msgid "filter rules" msgstr "" #: documents/serialisers.py:52 -msgid "Invalid regular expresssion: " +#, python-format +msgid "Invalid regular expresssion: %(error)s" msgstr "" -#: documents/serialisers.py:376 +#: documents/serialisers.py:378 #, python-format msgid "File type %(type)s not supported" msgstr "" From 3a46cc33eef5bd1cbcfd565b1fc228771ef5349d Mon Sep 17 00:00:00 2001 From: Bishop Clark Date: Tue, 23 Feb 2021 22:09:12 -0800 Subject: [PATCH 199/225] Update README.md Like 'stuff' and 'traffic', 'mail' gets no 's'. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 83ef1c1ed..aebd774e0 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Here's what you get: * Searching for similar documents ("More like this") * Email processing: Paperless adds documents from your email accounts. * Configure multiple accounts and filters for each account. - * When adding documents from mails, paperless can move these mails to a new folder, mark them as read, flag them as important or delete them. + * When adding documents from mail, paperless can move these mail to a new folder, mark them as read, flag them as important or delete them. * Machine learning powered document matching. * Paperless learns from your documents and will be able to automatically assign tags, correspondents and types to documents once you've stored a few documents in paperless. * Optimized for multi core systems: Paperless-ng consumes multiple documents in parallel. From 440abcda7c993806646020239509deff46e81d75 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 09:09:47 +0000 Subject: [PATCH 200/225] Apply translations in fr translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'fr' language. --- src/locale/fr/LC_MESSAGES/django.po | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/src/locale/fr/LC_MESSAGES/django.po b/src/locale/fr/LC_MESSAGES/django.po index 244d45fb5..6acd81649 100644 --- a/src/locale/fr/LC_MESSAGES/django.po +++ b/src/locale/fr/LC_MESSAGES/django.po @@ -12,7 +12,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-16 14:52+0100\n" +"POT-Creation-Date: 2021-02-23 13:20+0100\n" "PO-Revision-Date: 2021-02-16 18:37+0000\n" "Last-Translator: Philmo67, 2021\n" "Language-Team: French (https://www.transifex.com/paperless/teams/115905/fr/)\n" @@ -356,7 +356,12 @@ msgstr "règle de filtrage" msgid "filter rules" msgstr "règles de filtrage" -#: documents/serialisers.py:370 +#: documents/serialisers.py:52 +#, python-format +msgid "Invalid regular expresssion: %(error)s" +msgstr "Expression régulière incorrecte : %(error)s" + +#: documents/serialisers.py:378 #, python-format msgid "File type %(type)s not supported" msgstr "Type de fichier %(type)s non pris en charge" @@ -423,7 +428,7 @@ msgstr "Néerlandais" msgid "French" msgstr "Français" -#: paperless/urls.py:114 +#: paperless/urls.py:118 msgid "Paperless-ng administration" msgstr "Administration de Paperless-ng" From 1328fa9e5ec486ff92e5ea4885a5c89fe586f375 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 11:52:24 +0000 Subject: [PATCH 201/225] Apply translations in de translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'de' language. --- src/locale/de/LC_MESSAGES/django.po | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/src/locale/de/LC_MESSAGES/django.po b/src/locale/de/LC_MESSAGES/django.po index d1a07d18f..c031ebc85 100644 --- a/src/locale/de/LC_MESSAGES/django.po +++ b/src/locale/de/LC_MESSAGES/django.po @@ -11,7 +11,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-23 12:30+0100\n" +"POT-Creation-Date: 2021-02-23 13:20+0100\n" "PO-Revision-Date: 2021-02-16 18:37+0000\n" "Last-Translator: Jonas Winkler, 2021\n" "Language-Team: German (https://www.transifex.com/paperless/teams/115905/de/)\n" @@ -355,10 +355,11 @@ msgid "filter rules" msgstr "Filterregeln" #: documents/serialisers.py:52 -msgid "Invalid regular expresssion: " -msgstr "Ungültiger regulärer Ausdruck:" +#, python-format +msgid "Invalid regular expresssion: %(error)s" +msgstr "Ungültiger regulärer Ausdruck: %(error)s" -#: documents/serialisers.py:376 +#: documents/serialisers.py:378 #, python-format msgid "File type %(type)s not supported" msgstr "Dateityp %(type)s nicht unterstützt" From 39505c9764221b278c90d30c8b1ba5436a85b9d0 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 13:33:41 +0000 Subject: [PATCH 202/225] Translate /src-ui/messages.xlf in pt_BR translation completed for the source file '/src-ui/messages.xlf' on the 'pt_BR' language. --- src-ui/src/locale/messages.pt_BR.xlf | 2337 ++++++++++++++++++++++++++ 1 file changed, 2337 insertions(+) create mode 100644 src-ui/src/locale/messages.pt_BR.xlf diff --git a/src-ui/src/locale/messages.pt_BR.xlf b/src-ui/src/locale/messages.pt_BR.xlf new file mode 100644 index 000000000..ac3b557ad --- /dev/null +++ b/src-ui/src/locale/messages.pt_BR.xlf @@ -0,0 +1,2337 @@ + + + + + Document added + Documento adicionado + + src/app/app.component.ts + 51 + + + + Document was added to paperless. + Documento foi adicionado ao paperless. + + src/app/app.component.ts + 51 + + + + Open document + Abrir documento + + src/app/app.component.ts + 51 + + + + Could not add : + Não foi possível adicionar : + + src/app/app.component.ts + 59 + + + + New document detected + Novo documento detectado + + src/app/app.component.ts + 65 + + + + Document is being processed by paperless. + Documento está sendo processado pelo paperless. + + src/app/app.component.ts + 65 + + + + Documents + Documentos + + src/app/components/document-list/document-list.component.ts + 49 + + + + View "" saved successfully. + Visualização &quot;&quot; salva com sucesso. + + src/app/components/document-list/document-list.component.ts + 115 + + + + View "" created successfully. + Visualização &quot;&quot; criada com sucesso. + + src/app/components/document-list/document-list.component.ts + 136 + + + + Select + Selecionar + + src/app/components/document-list/document-list.component.html + 7 + + + + Select none + Selecionar nenhum + + src/app/components/document-list/document-list.component.html + 10 + + + + Select page + Selecionar página + + src/app/components/document-list/document-list.component.html + 11 + + + + Select all + Selecionar todos + + src/app/components/document-list/document-list.component.html + 12 + + + + Sort + Ordenar + + src/app/components/document-list/document-list.component.html + 39 + + + + Views + Visualizações + + src/app/components/document-list/document-list.component.html + 64 + + + + Save as... + Salvar como... + + src/app/components/document-list/document-list.component.html + 72 + + + + Save "" + Salvar &quot;&quot; + + src/app/components/document-list/document-list.component.html + 71 + + + + {VAR_PLURAL, plural, =1 {Selected of one document} other {Selected of documents}} + {VAR_PLURAL, plural, =1 {Selecionado de um documento} other {Selecionado of documentos}} + + src/app/components/document-list/document-list.component.html + 85 + + + + {VAR_PLURAL, plural, =1 {One document} other { documents}} + {VAR_PLURAL, plural, =1 {Um documento} other { documentos}} + + src/app/components/document-list/document-list.component.html + 86 + + + + (filtered) + (filtrado) + + src/app/components/document-list/document-list.component.html + 86 + + + + ASN + NSA + + src/app/components/document-list/document-list.component.html + 105 + + + + Correspondent + Correspondente + + src/app/components/document-list/document-list.component.html + 111 + + + + Title + Título + + src/app/components/document-list/document-list.component.html + 117 + + + + Document type + Tipo de Documento + + src/app/components/document-list/document-list.component.html + 123 + + + + Created + Criado + + src/app/components/document-list/document-list.component.html + 129 + + + + Added + Adicionado + + src/app/components/document-list/document-list.component.html + 135 + + + + Confirm delete + Confirmar exclusão + + src/app/components/document-detail/document-detail.component.ts + 203 + + + + Do you really want to delete document ""? + Você realmente deseja excluir o documento &quot;&quot;? + + src/app/components/document-detail/document-detail.component.ts + 204 + + + + The files for this document will be deleted permanently. This operation cannot be undone. + Os arquivos desse documento serão excluídos permanentemente. Essa operação não pode ser revertida. + + src/app/components/document-detail/document-detail.component.ts + 205 + + + + Delete document + Excluir documento + + src/app/components/document-detail/document-detail.component.ts + 207 + + + + Error deleting document: + Erro ao excluir documento: + + src/app/components/document-detail/document-detail.component.ts + 214 + + + + Delete + Excluir + + src/app/components/document-detail/document-detail.component.html + 15 + + + + Download + Baixar + + src/app/components/document-detail/document-detail.component.html + 23 + + + + More like this + Mais como este + + src/app/components/document-detail/document-detail.component.html + 38 + + + + Close + Fechar + + src/app/components/document-detail/document-detail.component.html + 44 + + + + Details + Detalhes + + src/app/components/document-detail/document-detail.component.html + 56 + + + + Content + Conteúdo + + src/app/components/document-detail/document-detail.component.html + 72 + + + + Metadata + Metadados + + src/app/components/document-detail/document-detail.component.html + 81 + + + + Discard + Descartar + + src/app/components/document-detail/document-detail.component.html + 130 + + + + Save + Salvar + + src/app/components/document-detail/document-detail.component.html + 132 + + + + Page + Página + + src/app/components/document-detail/document-detail.component.html + 4 + + + + of + de + + src/app/components/document-detail/document-detail.component.html + 8 + + + + Download original + Baixar original + + src/app/components/document-detail/document-detail.component.html + 29 + + + + Archive serial number + Número de série de arquivamento + + src/app/components/document-detail/document-detail.component.html + 60 + + + + Date created + Data de criação + + src/app/components/document-detail/document-detail.component.html + 61 + + + + Date modified + Data de modificação + + src/app/components/document-detail/document-detail.component.html + 87 + + + + Date added + Data de adição + + src/app/components/document-detail/document-detail.component.html + 91 + + + + Media filename + Nome do arquivo + + src/app/components/document-detail/document-detail.component.html + 95 + + + + Original MD5 checksum + Soma de verificação MD5 original + + src/app/components/document-detail/document-detail.component.html + 99 + + + + Original file size + Tamanho do arquivo original + + src/app/components/document-detail/document-detail.component.html + 103 + + + + Original mime type + Tipo mime original + + src/app/components/document-detail/document-detail.component.html + 107 + + + + Archive MD5 checksum + Soma de verificação MD5 de arquivamento + + src/app/components/document-detail/document-detail.component.html + 111 + + + + Archive file size + Tamanho arquivado + + src/app/components/document-detail/document-detail.component.html + 115 + + + + Original document metadata + Metadados do documento original + + src/app/components/document-detail/document-detail.component.html + 121 + + + + Archived document metadata + Metadados do documento arquivado + + src/app/components/document-detail/document-detail.component.html + 122 + + + + Save & next + Salvar & próximo + + src/app/components/document-detail/document-detail.component.html + 131 + + + + Hello , welcome to Paperless-ng! + Olá , bem-vindo ao Paperless-ng! + + src/app/components/dashboard/dashboard.component.ts + 33 + + + + Welcome to Paperless-ng! + Bem-vindo ao Paperless-ng! + + src/app/components/dashboard/dashboard.component.ts + 35 + + + + Dashboard + Painel de controle + + src/app/components/dashboard/dashboard.component.html + 1 + + + + Do you really want to delete the tag ""? + Você realmente deseja excluir a etiqueta &quot;&quot;? + + src/app/components/manage/tag-list/tag-list.component.ts + 30 + + + + Tags + Etiquetas + + src/app/components/manage/tag-list/tag-list.component.html + 1 + + + + Create + Criar + + src/app/components/manage/tag-list/tag-list.component.html + 2 + + + + Filter by: + Filtrar por: + + src/app/components/manage/tag-list/tag-list.component.html + 8 + + + + Name + Nome + + src/app/components/manage/tag-list/tag-list.component.html + 9 + + + + Color + Cor + + src/app/components/manage/tag-list/tag-list.component.html + 20 + + + + Matching + Detecção + + src/app/components/manage/tag-list/tag-list.component.html + 21 + + + + Document count + Número de documentos + + src/app/components/manage/tag-list/tag-list.component.html + 22 + + + + Actions + Ações + + src/app/components/manage/tag-list/tag-list.component.html + 23 + + + + Documents + Documentos + + src/app/components/manage/tag-list/tag-list.component.html + 38 + + + + Edit + Editar + + src/app/components/manage/tag-list/tag-list.component.html + 43 + + + + Do you really want to delete the document type ""? + Você realmente deseja excluir o tipo de documento &quot;&quot;? + + src/app/components/manage/document-type-list/document-type-list.component.ts + 26 + + + + Document types + Tipos de documento + + src/app/components/manage/document-type-list/document-type-list.component.html + 1 + + + + Logs + Logs + + src/app/components/manage/logs/logs.component.html + 1 + + + + Saved view "" deleted. + Visualização &quot;&quot; excluída. + + src/app/components/manage/settings/settings.component.ts + 67 + + + + Settings saved successfully. + Configurações salvas com sucesso. + + src/app/components/manage/settings/settings.component.ts + 87 + + + + Use system language + Usar linguagem do sistema + + src/app/components/manage/settings/settings.component.ts + 91 + + + + Use date format of display language + Usar formato de data da linguagem de exibição + + src/app/components/manage/settings/settings.component.ts + 96 + + + + ISO 8601 + ISO 8601 + + src/app/components/manage/settings/settings.component.ts + 97 + + + + Error while storing settings on server: + Erro ao salvar configurações: + + src/app/components/manage/settings/settings.component.ts + 114 + + + + Settings + Configurações + + src/app/components/manage/settings/settings.component.html + 1 + + + + General settings + Configurações gerais + + src/app/components/manage/settings/settings.component.html + 10 + + + + Notifications + Notificações + + src/app/components/manage/settings/settings.component.html + 115 + + + + Saved views + Visualizações + + src/app/components/manage/settings/settings.component.html + 133 + + + + Appearance + Aparência + + src/app/components/manage/settings/settings.component.html + 13 + + + + Display language + Linguagem de exibição + + src/app/components/manage/settings/settings.component.html + 17 + + + + You need to reload the page after applying a new language. + Você precisar recarregar a página depois de escolher uma nova linguagem. + + src/app/components/manage/settings/settings.component.html + 25 + + + + Date display + Exibição de data + + src/app/components/manage/settings/settings.component.html + 32 + + + + Date format + Formato de data + + src/app/components/manage/settings/settings.component.html + 45 + + + + Short: + Curto: + + src/app/components/manage/settings/settings.component.html + 51 + + + + Medium: + Médio: + + src/app/components/manage/settings/settings.component.html + 55 + + + + Long: + Longo: + + src/app/components/manage/settings/settings.component.html + 59 + + + + Items per page + Itens por página + + src/app/components/manage/settings/settings.component.html + 67 + + + + Document editor + Editor de documentos + + src/app/components/manage/settings/settings.component.html + 83 + + + + Use PDF viewer provided by the browser + Usar visualizador de PDF do navegador + + src/app/components/manage/settings/settings.component.html + 87 + + + + This is usually faster for displaying large PDF documents, but it might not work on some browsers. + Isso é geralmente mais rápido para exibir grandes documentos PDF, mas poderá não funcionar em alguns navegadores. + + src/app/components/manage/settings/settings.component.html + 87 + + + + Dark mode + Modo noturno + + src/app/components/manage/settings/settings.component.html + 94 + + + + Use system settings + Usar configurações do sistema + + src/app/components/manage/settings/settings.component.html + 97 + + + + Enable dark mode + Habilitar modo noturno + + src/app/components/manage/settings/settings.component.html + 98 + + + + Bulk editing + Edição em massa + + src/app/components/manage/settings/settings.component.html + 102 + + + + Show confirmation dialogs + Mostrar janelas de confirmação + + src/app/components/manage/settings/settings.component.html + 106 + + + + Deleting documents will always ask for confirmation. + Ao excluir um documento, sempre será pedido uma confirmação. + + src/app/components/manage/settings/settings.component.html + 106 + + + + Apply on close + Aplicar ao fechar + + src/app/components/manage/settings/settings.component.html + 107 + + + + Document processing + Processamento de documentos + + src/app/components/manage/settings/settings.component.html + 118 + + + + Show notifications when new documents are detected + Exibir notificações quando novos documentos forem detectados + + src/app/components/manage/settings/settings.component.html + 122 + + + + Show notifications when document processing completes successfully + Exibir notificações quando o processamento de um documento concluir com sucesso + + src/app/components/manage/settings/settings.component.html + 123 + + + + Show notifications when document processing fails + Exibir notificações quando o processamento de um documento falhar + + src/app/components/manage/settings/settings.component.html + 124 + + + + Suppress notifications on dashboard + Não exibir notificações no painel de controle + + src/app/components/manage/settings/settings.component.html + 125 + + + + This will suppress all messages about document processing status on the dashboard. + Isso esconderá todas as mensagens sobre o status de processamento de documentos no painel de controle. + + src/app/components/manage/settings/settings.component.html + 125 + + + + Appears on + Aparece em + + src/app/components/manage/settings/settings.component.html + 145 + + + + Show on dashboard + Exibir no painel de controle + + src/app/components/manage/settings/settings.component.html + 148 + + + + Show in sidebar + Mostrar na navegação lateral + + src/app/components/manage/settings/settings.component.html + 152 + + + + No saved views defined. + Nenhuma visualização definida. + + src/app/components/manage/settings/settings.component.html + 162 + + + + 404 Not Found + 404 Não Encontrado + + src/app/components/not-found/not-found.component.html + 7 + + + + Do you really want to delete the correspondent ""? + Você realmente deseja excluir o correspondente &quot;&quot;? + + src/app/components/manage/correspondent-list/correspondent-list.component.ts + 26 + + + + Correspondents + Correspondentes + + src/app/components/manage/correspondent-list/correspondent-list.component.html + 1 + + + + Last correspondence + Última correspondência + + src/app/components/manage/correspondent-list/correspondent-list.component.html + 22 + + + + Confirmation + Confirmação + + src/app/components/common/confirm-dialog/confirm-dialog.component.ts + 17 + + + + Confirm + Confirmar + + src/app/components/common/confirm-dialog/confirm-dialog.component.ts + 29 + + + + Cancel + Cancelar + + src/app/components/common/confirm-dialog/confirm-dialog.component.html + 12 + + + + Create new correspondent + Criar novo correspondente + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.ts + 21 + + + + Edit correspondent + Editar correspondente + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.ts + 25 + + + + Matching algorithm + Algoritmo de detecção + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html + 10 + + + + Matching pattern + Padrão de detecção + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html + 11 + + + + Case insensitive + Não diferenciar maiúsculas de minúsculas + + src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html + 12 + + + + Create new tag + Criar nova etiqueta + + src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.ts + 21 + + + + Edit tag + Editar etiqueta + + src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.ts + 25 + + + + Inbox tag + Etiqueta caixa de entrada + + src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.html + 21 + + + + Inbox tags are automatically assigned to all consumed documents. + Etiquetas de caixa de entrada são atribuídas automaticamente para todos os documentos consumidos. + + src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.html + 21 + + + + Create new document type + Criar novo tipo de documento + + src/app/components/manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component.ts + 21 + + + + Edit document type + Editar tipo de documento + + src/app/components/manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component.ts + 25 + + + + Search results + Resultados da busca + + src/app/components/search/search.component.html + 1 + + + + Invalid search query: + Termo de pesquisa inválido: + + src/app/components/search/search.component.html + 4 + + + + Showing documents similar to + Exibindo documentos similares a + + src/app/components/search/search.component.html + 7 + + + + Search query: + Termo de pesquisa: + + src/app/components/search/search.component.html + 11 + + + + Did you mean ""? + Você quis dizer &quot;&quot;? + + src/app/components/search/search.component.html + 13 + + + + {VAR_PLURAL, plural, =0 {No results} =1 {One result} other { results}} + {VAR_PLURAL, plural, =0 {Sem resultados} =1 {Um resultado} other { resultados}} + + src/app/components/search/search.component.html + 18 + + + + Paperless-ng + Paperless-ng + + src/app/components/app-frame/app-frame.component.html + 11 + + app title + + + Search documents + Procurar documentos + + src/app/components/app-frame/app-frame.component.html + 15 + + + + Logout + Encerrar sessão + + src/app/components/app-frame/app-frame.component.html + 45 + + + + Manage + Gerenciar + + src/app/components/app-frame/app-frame.component.html + 107 + + + + Admin + Admin + + src/app/components/app-frame/app-frame.component.html + 149 + + + + Info + Informação + + src/app/components/app-frame/app-frame.component.html + 155 + + + + Documentation + Documentação + + src/app/components/app-frame/app-frame.component.html + 162 + + + + GitHub + GitHub + + src/app/components/app-frame/app-frame.component.html + 170 + + + + Suggest an idea + Sugerir uma idéia + + src/app/components/app-frame/app-frame.component.html + 176 + + + + Logged in as + Sessão iniciada como + + src/app/components/app-frame/app-frame.component.html + 34 + + + + Open documents + Abrir documentos + + src/app/components/app-frame/app-frame.component.html + 87 + + + + Close all + Fechar todos + + src/app/components/app-frame/app-frame.component.html + 101 + + + + Correspondent: + Correspondente: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 29 + + + + Without correspondent + Sem correspondente + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 31 + + + + Type: + Tipo: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 36 + + + + Without document type + Sem tipo de documento + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 38 + + + + Tag: + Etiqueta: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 42 + + + + Without any tag + Sem etiquetas + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 46 + + + + Title: + Título: + + src/app/components/document-list/filter-editor/filter-editor.component.ts + 50 + + + + Filter tags + Filtrar etiquetas + + src/app/components/document-list/filter-editor/filter-editor.component.html + 12 + + + + Filter correspondents + Filtrar correspondentes + + src/app/components/document-list/filter-editor/filter-editor.component.html + 20 + + + + Filter document types + Filtrar tipos de documento + + src/app/components/document-list/filter-editor/filter-editor.component.html + 27 + + + + Reset filters + Limpar filtros + + src/app/components/document-list/filter-editor/filter-editor.component.html + 50 + + + + Not assigned + Não atribuído + + src/app/components/common/filterable-dropdown/filterable-dropdown.component.ts + 166 + + Filter drop down element to filter for documents with no correspondent/type/tag assigned + + + Apply + Aplicar + + src/app/components/common/filterable-dropdown/filterable-dropdown.component.html + 26 + + + + Last 7 days + Últimos 7 dias + + src/app/components/common/date-dropdown/date-dropdown.component.ts + 24 + + + + Last month + Último mês + + src/app/components/common/date-dropdown/date-dropdown.component.ts + 25 + + + + Last 3 months + Últimos 3 meses + + src/app/components/common/date-dropdown/date-dropdown.component.ts + 26 + + + + Last year + Último ano + + src/app/components/common/date-dropdown/date-dropdown.component.ts + 27 + + + + After + Antes + + src/app/components/common/date-dropdown/date-dropdown.component.html + 13 + + + + Before + Depois + + src/app/components/common/date-dropdown/date-dropdown.component.html + 29 + + + + Clear + Limpar + + src/app/components/common/date-dropdown/date-dropdown.component.html + 18 + + + + View + Ver + + src/app/components/document-list/document-card-large/document-card-large.component.html + 50 + + + + Created: + Criado: + + src/app/components/document-list/document-card-large/document-card-large.component.html + 67 + + + + Filter by correspondent + Filtrar por correspondente + + src/app/components/document-list/document-card-large/document-card-large.component.html + 20 + + + + Filter by tag + Filtrar por etiqueta + + src/app/components/document-list/document-card-large/document-card-large.component.html + 24 + + + + Score: + Nota: + + src/app/components/document-list/document-card-large/document-card-large.component.html + 62 + + + + View in browser + Visualizar no navegador + + src/app/components/document-list/document-card-small/document-card-small.component.html + 40 + + + + Error executing bulk operation: + Erro ao executar operação em massa: + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 74 + + + + "" + &quot;&quot; + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 113 + + + + "" and "" + &quot;&quot; e &quot;&quot; + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 115 + + This is for messages like 'modify "tag1" and "tag2"' + + + , + , + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 117 + + this is used to separate enumerations and should probably be a comma and a whitespace in most languages + + + and "" + e &quot;&quot; + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 118 + + this is for messages like 'modify "tag1", "tag2" and "tag3"' + + + Confirm tags assignment + Confirmar atribuição de etiqueta + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 127 + + + + This operation will add the tag "" to selected document(s). + Essa operação irá adicionar a etiqueta &quot;&quot; em documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 130 + + + + This operation will add the tags to selected document(s). + Essa operação irá adicionar as etiquetas em documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 132 + + + + This operation will remove the tag "" from selected document(s). + Essa operação irá remover a etiqueta &quot;&quot; de documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 135 + + + + This operation will remove the tags from selected document(s). + Essa operação irá remover as etiquetas de documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 137 + + + + This operation will add the tags and remove the tags on selected document(s). + Essa operação irá adicionar as etiquetas e remover as etiquetas de documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 139 + + + + Confirm correspondent assignment + Confirmar atribuição de correspondente + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 159 + + + + This operation will assign the correspondent "" to selected document(s). + Essa operação irá atribuir o correspondente &quot;&quot; para documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 161 + + + + This operation will remove the correspondent from selected document(s). + Essa operação irá remover o correspondente de documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 163 + + + + Confirm document type assignment + Confirmar atribuição de tipo de documento + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 182 + + + + This operation will assign the document type "" to selected document(s). + Essa operação irá atribuir o tipo de documento &quot;&quot; para documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 184 + + + + This operation will remove the document type from selected document(s). + Essa operação irá remover o tipo de documento de documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 186 + + + + Delete confirm + Confirmar exlcusão + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 201 + + + + This operation will permanently delete selected document(s). + Essa operação irá excluir permanentemente documento(s) selecionado(s). + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 202 + + + + This operation cannot be undone. + Essa operação não pode ser revertida. + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 203 + + + + Delete document(s) + Apagar documento(s) + + src/app/components/document-list/bulk-editor/bulk-editor.component.ts + 205 + + + + Select: + Selecionar: + + src/app/components/document-list/bulk-editor/bulk-editor.component.html + 10 + + + + All + Todos + + src/app/components/document-list/bulk-editor/bulk-editor.component.html + 20 + + + + Edit: + Editar: + + src/app/components/document-list/bulk-editor/bulk-editor.component.html + 27 + + + + Download originals + Descarregar originais + + src/app/components/document-list/bulk-editor/bulk-editor.component.html + 68 + + + + Suggestions: + Sugestões: + + src/app/components/common/input/select/select.component.html + 26 + + + + Save current view + Salvar visualização atual + + src/app/components/document-list/save-view-config-dialog/save-view-config-dialog.component.html + 3 + + + + Show all + Mostrar todos + + src/app/components/dashboard/widgets/saved-view-widget/saved-view-widget.component.html + 3 + + + + Statistics + Estatísticas + + src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html + 1 + + + + Total documents: + Total de documentos: + + src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html + 4 + + + + Documents in inbox: + Documentos na caixa de entrada: + + src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html + 3 + + + + Processing: + Processando: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 32 + + + + Failed: + Falha: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 35 + + + + Added: + Adicionado: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 38 + + + + Connecting... + Conectando... + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 118 + + + + Uploading... + Enviando... + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 123 + + + + Upload complete, waiting... + Envio concluído, esperando... + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 126 + + + + HTTP error: + Erro HTTP: + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.ts + 136 + + + + Upload new documents + Enviar novos documentos + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 1 + + + + Drop documents here or + Solte documentos aqui ou + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 13 + + + + Browse files + Navegar arquivos + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 13 + + + + Dismiss completed + Esconder completos + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 4 + + This button dismisses all status messages about processed documents on the dashboard (failed and successful) + + + {VAR_PLURAL, plural, =1 {One more document} other { more documents}} + {VAR_PLURAL, plural, =1 {Mais um documento} other {Mais documentos} + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 25 + + This is shown as a summary line when there are more than 5 document in the processing pipeline. + + + Open document + Abrir documento + + src/app/components/dashboard/widgets/upload-file-widget/upload-file-widget.component.html + 45 + + + + First steps + Primeiros passos + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 1 + + + + Paperless is running! :) + O Paperless está a correr! :) + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 5 + + + + You can start uploading documents by dropping them in the file upload box to the right or by dropping them in the configured consumption folder and they'll start showing up in the documents list. After you've added some metadata to your documents, use the filtering mechanisms of paperless to create custom views (such as 'Recently added', 'Tagged TODO') and they will appear on the dashboard instead of this message. + Pode começar a carregar documentos deixando-os na caixa de carregamento de ficheiros à direita, ou deixando-os na pasta de consumo pré-configurada, que eles depois aparecem na lista de documentos. Depois de ter adicionado alguns meta-dados aos seus documentos, utilize os mecanismos de filtragem do Paperless para criar visualizações personalizadas (tais como 'Adicionados recentemente', 'Etiquetados com PARAFAZER'). Posteriormente estes aparecerão no painel em vez desta mensagem. + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 6,7 + + + + Paperless offers some more features that try to make your life easier: + O Paperless oferece algumas funcionalidades para tentar tornar a sua vida mais fácil: + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 8 + + + + Once you've got a couple documents in paperless and added metadata to them, paperless can assign that metadata to new documents automatically. + Assim que já tenha alguns documentos previamente adicionados com os respetivos meta-dados, o Paperless consegue adicionar automaticamente meta-dados aos novos documentos. + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 10 + + + + You can configure paperless to read your mails and add documents from attached files. + Pode configurar o Paperless para ler os seus e-mails e adicionar os documentos em anexo à plataforma. + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 11 + + + + Consult the documentation on how to use these features. The section on basic usage also has some information on how to use paperless in general. + Consulte a documentação para saber como utilizar estas funcionalidades. A secção sobre utilização básica têm informação para o ajudar a utilizar a plataforma de uma forma geral. + + src/app/components/dashboard/widgets/welcome-widget/welcome-widget.component.html + 13 + + + + Metadata + Metadados + + src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts + 18 + + + + Select + Selecione + + src/app/components/common/select-dialog/select-dialog.component.ts + 18 + + + + Please select an object + Por favor selecione um objeto + + src/app/components/common/select-dialog/select-dialog.component.ts + 21 + + + + Yes + Sim + + src/app/pipes/yes-no.pipe.ts + 9 + + + + No + Não + + src/app/pipes/yes-no.pipe.ts + 9 + + + + (no title) + (sem título) + + src/app/pipes/document-title.pipe.ts + 12 + + + + English (US) + Inglês (US) + + src/app/services/settings.service.ts + 82 + + + + English (GB) + Inglês (GB) + + src/app/services/settings.service.ts + 83 + + + + German + Alemão + + src/app/services/settings.service.ts + 84 + + + + Dutch + Holandês + + src/app/services/settings.service.ts + 85 + + + + French + Francês + + src/app/services/settings.service.ts + 86 + + + + Document already exists. + Documento já existente. + + src/app/services/consumer-status.service.ts + 15 + + + + File not found. + Arquivo não encontrado. + + src/app/services/consumer-status.service.ts + 16 + + + + Pre-consume script does not exist. + Script pré-consumo não existe. + + src/app/services/consumer-status.service.ts + 17 + + Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation + + + Error while executing pre-consume script. + Erro ao executar script pré-consumo. + + src/app/services/consumer-status.service.ts + 18 + + Pre-Consume is a term that appears like that in the documentation as well and does not need a specific translation + + + Post-consume script does not exist. + Script pós-consumo não existe. + + src/app/services/consumer-status.service.ts + 19 + + Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation + + + Error while executing post-consume script. + Erro ao executar script pós-consumo. + + src/app/services/consumer-status.service.ts + 20 + + Post-Consume is a term that appears like that in the documentation as well and does not need a specific translation + + + Received new file. + Novo arquivo recebido. + + src/app/services/consumer-status.service.ts + 21 + + + + File type not supported. + Tipo de arquivo não suportado. + + src/app/services/consumer-status.service.ts + 22 + + + + Processing document... + Processando documento... + + src/app/services/consumer-status.service.ts + 23 + + + + Generating thumbnail... + Gerando imagem... + + src/app/services/consumer-status.service.ts + 24 + + + + Retrieving date from document... + Buscando data do documento... + + src/app/services/consumer-status.service.ts + 25 + + + + Saving document... + Salvando documento... + + src/app/services/consumer-status.service.ts + 26 + + + + Finished. + Encerrado. + + src/app/services/consumer-status.service.ts + 27 + + + + Error + Erro + + src/app/services/toast.service.ts + 35 + + + + Information + Informação + + src/app/services/toast.service.ts + 39 + + + + ASN + NSA + + src/app/services/rest/document.service.ts + 17 + + + + Correspondent + Correspondente + + src/app/services/rest/document.service.ts + 18 + + + + Title + Título + + src/app/services/rest/document.service.ts + 19 + + + + Document type + Tipo de Documento + + src/app/services/rest/document.service.ts + 20 + + + + Created + Criado + + src/app/services/rest/document.service.ts + 21 + + + + Added + Adicionado + + src/app/services/rest/document.service.ts + 22 + + + + Modified + Modificado + + src/app/services/rest/document.service.ts + 23 + + + + Light blue + Azul claro + + src/app/data/paperless-tag.ts + 6 + + + + Blue + Azul + + src/app/data/paperless-tag.ts + 7 + + + + Light green + Verde claro + + src/app/data/paperless-tag.ts + 8 + + + + Green + Verde + + src/app/data/paperless-tag.ts + 9 + + + + Light red + Vermelho claro + + src/app/data/paperless-tag.ts + 10 + + + + Red + Vermelho + + src/app/data/paperless-tag.ts + 11 + + + + Light orange + Laranja claro + + src/app/data/paperless-tag.ts + 12 + + + + Orange + Laranja + + src/app/data/paperless-tag.ts + 13 + + + + Light violet + Violeta claro + + src/app/data/paperless-tag.ts + 14 + + + + Violet + Violeta + + src/app/data/paperless-tag.ts + 15 + + + + Brown + Marrom + + src/app/data/paperless-tag.ts + 16 + + + + Black + Preto + + src/app/data/paperless-tag.ts + 17 + + + + Light grey + Cinza claro + + src/app/data/paperless-tag.ts + 18 + + + + Create new item + Criar novo item + + src/app/components/common/edit-dialog/edit-dialog.component.ts + 50 + + + + Edit item + Editar item + + src/app/components/common/edit-dialog/edit-dialog.component.ts + 54 + + + + Could not save element: + Não podemos salvar elemento: + + src/app/components/common/edit-dialog/edit-dialog.component.ts + 58 + + + + Automatic + Automático + + src/app/components/manage/generic-list/generic-list.component.ts + 39 + + + + Do you really want to delete this element? + Você realmente quer excluir esse elemento? + + src/app/components/manage/generic-list/generic-list.component.ts + 97 + + + + Associated documents will not be deleted. + Documentos associados não serão excluidos. + + src/app/components/manage/generic-list/generic-list.component.ts + 104 + + + + Delete + Excluir + + src/app/components/manage/generic-list/generic-list.component.ts + 106 + + + + Error while deleting element: + Erro ao excluir elemento: + + src/app/components/manage/generic-list/generic-list.component.ts + 114 + + + + Any word + Qualquer palavra + + src/app/data/matching-model.ts + 12 + + + + Any: Document contains any of these words (space separated) + Qualquer: Documento contém qualquer uma dessas palavras (separadas por espaço) + + src/app/data/matching-model.ts + 12 + + + + All words + Todas as palavras + + src/app/data/matching-model.ts + 13 + + + + All: Document contains all of these words (space separated) + Todas: Documento contém todas essas palavras (separadas por espaço) + + src/app/data/matching-model.ts + 13 + + + + Exact match + Detecção exata + + src/app/data/matching-model.ts + 14 + + + + Exact: Document contains this string + Exata: Documento contém essa palavra + + src/app/data/matching-model.ts + 14 + + + + Regular expression + Expressão regular + + src/app/data/matching-model.ts + 15 + + + + Regular expression: Document matches this regular expression + Expressão regular: Documento condiz com essa expressão regular + + src/app/data/matching-model.ts + 15 + + + + Fuzzy word + Palavra difusa + + src/app/data/matching-model.ts + 16 + + + + Fuzzy: Document contains a word similar to this word + Fuzzy: Documento contém uma palavra similar à essa. + + src/app/data/matching-model.ts + 16 + + + + Auto: Learn matching automatically + Auto: Aprender detecção automaticamente + + src/app/data/matching-model.ts + 17 + + + + + \ No newline at end of file From c1ddff4ee5c2f817fb4fdcfade5c360f3993b5e6 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 15:00:14 +0000 Subject: [PATCH 203/225] Apply translations in pt_BR translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'pt_BR' language. --- src/locale/pt_BR/LC_MESSAGES/django.po | 668 +++++++++++++++++++++++++ 1 file changed, 668 insertions(+) create mode 100644 src/locale/pt_BR/LC_MESSAGES/django.po diff --git a/src/locale/pt_BR/LC_MESSAGES/django.po b/src/locale/pt_BR/LC_MESSAGES/django.po new file mode 100644 index 000000000..e1a74b104 --- /dev/null +++ b/src/locale/pt_BR/LC_MESSAGES/django.po @@ -0,0 +1,668 @@ +# SOME DESCRIPTIVE TITLE. +# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER +# This file is distributed under the same license as the PACKAGE package. +# FIRST AUTHOR , YEAR. +# +# Translators: +# Jonas Winkler, 2021 +# Rodrigo A , 2021 +# +#, fuzzy +msgid "" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"Report-Msgid-Bugs-To: \n" +"POT-Creation-Date: 2021-02-23 13:20+0100\n" +"PO-Revision-Date: 2021-02-16 18:37+0000\n" +"Last-Translator: Rodrigo A , 2021\n" +"Language-Team: Portuguese (Brazil) (https://www.transifex.com/paperless/teams/115905/pt_BR/)\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Language: pt_BR\n" +"Plural-Forms: nplurals=2; plural=(n > 1);\n" + +#: documents/apps.py:10 +msgid "Documents" +msgstr "Documentos" + +#: documents/models.py:32 +msgid "Any word" +msgstr "Qualquer palavra" + +#: documents/models.py:33 +msgid "All words" +msgstr "Todas as palavras" + +#: documents/models.py:34 +msgid "Exact match" +msgstr "Detecção exata" + +#: documents/models.py:35 +msgid "Regular expression" +msgstr "Expressão regular" + +#: documents/models.py:36 +msgid "Fuzzy word" +msgstr "Palavra difusa (fuzzy)" + +#: documents/models.py:37 +msgid "Automatic" +msgstr "Automático" + +#: documents/models.py:41 documents/models.py:364 paperless_mail/models.py:25 +#: paperless_mail/models.py:109 +msgid "name" +msgstr "nome" + +#: documents/models.py:45 +msgid "match" +msgstr "detecção" + +#: documents/models.py:49 +msgid "matching algorithm" +msgstr "algoritmo de detecção" + +#: documents/models.py:55 +msgid "is insensitive" +msgstr "diferencia maiúsculas de minúsculas" + +#: documents/models.py:74 documents/models.py:134 +msgid "correspondent" +msgstr "correspondente" + +#: documents/models.py:75 +msgid "correspondents" +msgstr "correspondentes" + +#: documents/models.py:97 +msgid "color" +msgstr "cor" + +#: documents/models.py:101 +msgid "is inbox tag" +msgstr "é etiqueta caixa de entrada" + +#: documents/models.py:103 +msgid "" +"Marks this tag as an inbox tag: All newly consumed documents will be tagged " +"with inbox tags." +msgstr "" +"Marca essa etiqueta como caixa de entrada: Todos os novos documentos " +"consumidos terão as etiquetas de caixa de entrada." + +#: documents/models.py:108 +msgid "tag" +msgstr "etiqueta" + +#: documents/models.py:109 documents/models.py:165 +msgid "tags" +msgstr "etiquetas" + +#: documents/models.py:115 documents/models.py:147 +msgid "document type" +msgstr "tipo de documento" + +#: documents/models.py:116 +msgid "document types" +msgstr "tipos de documento" + +#: documents/models.py:124 +msgid "Unencrypted" +msgstr "Não encriptado" + +#: documents/models.py:125 +msgid "Encrypted with GNU Privacy Guard" +msgstr "Encriptado com GNU Privacy Guard" + +#: documents/models.py:138 +msgid "title" +msgstr "título" + +#: documents/models.py:151 +msgid "content" +msgstr "conteúdo" + +#: documents/models.py:153 +msgid "" +"The raw, text-only data of the document. This field is primarily used for " +"searching." +msgstr "" +"O conteúdo de texto bruto do documento. Esse campo é usado principalmente " +"para busca." + +#: documents/models.py:158 +msgid "mime type" +msgstr "tipo mime" + +#: documents/models.py:169 +msgid "checksum" +msgstr "some de verificação" + +#: documents/models.py:173 +msgid "The checksum of the original document." +msgstr "A soma de verificação original do documento." + +#: documents/models.py:177 +msgid "archive checksum" +msgstr "Soma de verificação de arquivamento." + +#: documents/models.py:182 +msgid "The checksum of the archived document." +msgstr "A soma de verificação do documento arquivado." + +#: documents/models.py:186 documents/models.py:342 +msgid "created" +msgstr "criado" + +#: documents/models.py:190 +msgid "modified" +msgstr "modificado" + +#: documents/models.py:194 +msgid "storage type" +msgstr "tipo de armazenamento" + +#: documents/models.py:202 +msgid "added" +msgstr "adicionado" + +#: documents/models.py:206 +msgid "filename" +msgstr "nome do arquivo" + +#: documents/models.py:212 +msgid "Current filename in storage" +msgstr "Nome do arquivo atual armazenado" + +#: documents/models.py:216 +msgid "archive filename" +msgstr "nome do arquivo para arquivamento" + +#: documents/models.py:222 +msgid "Current archive filename in storage" +msgstr "Nome do arquivo para arquivamento armazenado" + +#: documents/models.py:226 +msgid "archive serial number" +msgstr "número de sério de arquivamento" + +#: documents/models.py:231 +msgid "The position of this document in your physical document archive." +msgstr "A posição deste documento no seu arquivamento físico." + +#: documents/models.py:237 +msgid "document" +msgstr "documento" + +#: documents/models.py:238 +msgid "documents" +msgstr "documentos" + +#: documents/models.py:325 +msgid "debug" +msgstr "debug" + +#: documents/models.py:326 +msgid "information" +msgstr "informação" + +#: documents/models.py:327 +msgid "warning" +msgstr "aviso" + +#: documents/models.py:328 +msgid "error" +msgstr "erro" + +#: documents/models.py:329 +msgid "critical" +msgstr "crítico" + +#: documents/models.py:333 +msgid "group" +msgstr "grupo" + +#: documents/models.py:336 +msgid "message" +msgstr "mensagem" + +#: documents/models.py:339 +msgid "level" +msgstr "nível" + +#: documents/models.py:346 +msgid "log" +msgstr "log" + +#: documents/models.py:347 +msgid "logs" +msgstr "logs" + +#: documents/models.py:358 documents/models.py:408 +msgid "saved view" +msgstr "visualização" + +#: documents/models.py:359 +msgid "saved views" +msgstr "visualizações" + +#: documents/models.py:362 +msgid "user" +msgstr "usuário" + +#: documents/models.py:368 +msgid "show on dashboard" +msgstr "exibir no painel de controle" + +#: documents/models.py:371 +msgid "show in sidebar" +msgstr "exibir no painel lateral" + +#: documents/models.py:375 +msgid "sort field" +msgstr "ordenar campo" + +#: documents/models.py:378 +msgid "sort reverse" +msgstr "odernar reverso" + +#: documents/models.py:384 +msgid "title contains" +msgstr "título contém" + +#: documents/models.py:385 +msgid "content contains" +msgstr "conteúdo contém" + +#: documents/models.py:386 +msgid "ASN is" +msgstr "NSA é" + +#: documents/models.py:387 +msgid "correspondent is" +msgstr "correspondente é" + +#: documents/models.py:388 +msgid "document type is" +msgstr "tipo de documento é" + +#: documents/models.py:389 +msgid "is in inbox" +msgstr "é caixa de entrada" + +#: documents/models.py:390 +msgid "has tag" +msgstr "contém etiqueta" + +#: documents/models.py:391 +msgid "has any tag" +msgstr "contém qualquer etiqueta" + +#: documents/models.py:392 +msgid "created before" +msgstr "criado antes de" + +#: documents/models.py:393 +msgid "created after" +msgstr "criado depois de" + +#: documents/models.py:394 +msgid "created year is" +msgstr "ano de criação é" + +#: documents/models.py:395 +msgid "created month is" +msgstr "mês de criação é" + +#: documents/models.py:396 +msgid "created day is" +msgstr "dia de criação é" + +#: documents/models.py:397 +msgid "added before" +msgstr "adicionado antes de" + +#: documents/models.py:398 +msgid "added after" +msgstr "adicionado depois de" + +#: documents/models.py:399 +msgid "modified before" +msgstr "modificado antes de" + +#: documents/models.py:400 +msgid "modified after" +msgstr "modificado depois de" + +#: documents/models.py:401 +msgid "does not have tag" +msgstr "não tem etiqueta" + +#: documents/models.py:412 +msgid "rule type" +msgstr "tipo de regra" + +#: documents/models.py:416 +msgid "value" +msgstr "valor" + +#: documents/models.py:422 +msgid "filter rule" +msgstr "regra de filtragem" + +#: documents/models.py:423 +msgid "filter rules" +msgstr "regras de filtragem" + +#: documents/serialisers.py:52 +#, python-format +msgid "Invalid regular expresssion: %(error)s" +msgstr "Expressão regular inválida: %(error)s" + +#: documents/serialisers.py:378 +#, python-format +msgid "File type %(type)s not supported" +msgstr "Tipo de arquivo %(type)s não suportado" + +#: documents/templates/index.html:20 +msgid "Paperless-ng is loading..." +msgstr "Paperless-ng está carregando..." + +#: documents/templates/registration/logged_out.html:13 +msgid "Paperless-ng signed out" +msgstr "Paperless-ng saiu" + +#: documents/templates/registration/logged_out.html:41 +msgid "You have been successfully logged out. Bye!" +msgstr "Sua sessão foi encerrada com sucesso. Até mais!" + +#: documents/templates/registration/logged_out.html:42 +msgid "Sign in again" +msgstr "Entre novamente" + +#: documents/templates/registration/login.html:13 +msgid "Paperless-ng sign in" +msgstr "Entrar no Paperless-ng" + +#: documents/templates/registration/login.html:42 +msgid "Please sign in." +msgstr "Por favor, entre na sua conta" + +#: documents/templates/registration/login.html:45 +msgid "Your username and password didn't match. Please try again." +msgstr "Seu usuário e senha estão incorretos. Por favor, tente novamente." + +#: documents/templates/registration/login.html:48 +msgid "Username" +msgstr "Usuário" + +#: documents/templates/registration/login.html:49 +msgid "Password" +msgstr "Senha" + +#: documents/templates/registration/login.html:54 +msgid "Sign in" +msgstr "Entrar" + +#: paperless/settings.py:291 +msgid "English (US)" +msgstr "Inglês (EUA)" + +#: paperless/settings.py:292 +msgid "English (GB)" +msgstr "Inglês (GB)" + +#: paperless/settings.py:293 +msgid "German" +msgstr "Alemão" + +#: paperless/settings.py:294 +msgid "Dutch" +msgstr "Holandês" + +#: paperless/settings.py:295 +msgid "French" +msgstr "Francês" + +#: paperless/urls.py:118 +msgid "Paperless-ng administration" +msgstr "Administração do Paperless-ng" + +#: paperless_mail/admin.py:25 +msgid "Filter" +msgstr "Filtro" + +#: paperless_mail/admin.py:27 +msgid "" +"Paperless will only process mails that match ALL of the filters given below." +msgstr "" +"Paperless processará somente e-mails que se encaixam em TODOS os filtros " +"abaixo." + +#: paperless_mail/admin.py:37 +msgid "Actions" +msgstr "Ações" + +#: paperless_mail/admin.py:39 +msgid "" +"The action applied to the mail. This action is only performed when documents" +" were consumed from the mail. Mails without attachments will remain entirely" +" untouched." +msgstr "" +"A ação se aplica ao e-mail. Essa ação só é executada quando documentos foram" +" consumidos do e-mail. E-mails sem anexos permanecerão intactos." + +#: paperless_mail/admin.py:46 +msgid "Metadata" +msgstr "Metadados" + +#: paperless_mail/admin.py:48 +msgid "" +"Assign metadata to documents consumed from this rule automatically. If you " +"do not assign tags, types or correspondents here, paperless will still " +"process all matching rules that you have defined." +msgstr "" +"Atribua metadados aos documentos consumidos por esta regra automaticamente. " +"Se você não atribuir etiquetas, tipos ou correspondentes aqui, paperless " +"ainda sim processará todas as regras de detecção que você definiu." + +#: paperless_mail/apps.py:9 +msgid "Paperless mail" +msgstr "Paperless mail" + +#: paperless_mail/models.py:11 +msgid "mail account" +msgstr "conta de e-mail" + +#: paperless_mail/models.py:12 +msgid "mail accounts" +msgstr "contas de e-mail" + +#: paperless_mail/models.py:19 +msgid "No encryption" +msgstr "Sem encriptação" + +#: paperless_mail/models.py:20 +msgid "Use SSL" +msgstr "Usar SSL" + +#: paperless_mail/models.py:21 +msgid "Use STARTTLS" +msgstr "Usar STARTTLS" + +#: paperless_mail/models.py:29 +msgid "IMAP server" +msgstr "Servidor IMAP" + +#: paperless_mail/models.py:33 +msgid "IMAP port" +msgstr "Porta IMAP" + +#: paperless_mail/models.py:36 +msgid "" +"This is usually 143 for unencrypted and STARTTLS connections, and 993 for " +"SSL connections." +msgstr "" +"É geralmente 143 para não encriptado e conexões STARTTLS, e 993 para " +"conexões SSL." + +#: paperless_mail/models.py:40 +msgid "IMAP security" +msgstr "segurança IMAP" + +#: paperless_mail/models.py:46 +msgid "username" +msgstr "usuário" + +#: paperless_mail/models.py:50 +msgid "password" +msgstr "senha" + +#: paperless_mail/models.py:60 +msgid "mail rule" +msgstr "regra de e-mail" + +#: paperless_mail/models.py:61 +msgid "mail rules" +msgstr "regras de e-mail" + +#: paperless_mail/models.py:67 +msgid "Only process attachments." +msgstr "Processar somente anexos." + +#: paperless_mail/models.py:68 +msgid "Process all files, including 'inline' attachments." +msgstr "Processar todos os arquivos, incluindo anexos 'inline'." + +#: paperless_mail/models.py:78 +msgid "Mark as read, don't process read mails" +msgstr "Marcar como lido, não processar e-mails lidos" + +#: paperless_mail/models.py:79 +msgid "Flag the mail, don't process flagged mails" +msgstr "Sinalizar o e-mail, não processar e-mails sinalizados" + +#: paperless_mail/models.py:80 +msgid "Move to specified folder" +msgstr "Mover para pasta especificada" + +#: paperless_mail/models.py:81 +msgid "Delete" +msgstr "Excluir" + +#: paperless_mail/models.py:88 +msgid "Use subject as title" +msgstr "Usar assunto como título" + +#: paperless_mail/models.py:89 +msgid "Use attachment filename as title" +msgstr "Usar nome do arquivo anexo como título" + +#: paperless_mail/models.py:99 +msgid "Do not assign a correspondent" +msgstr "Não atribuir um correspondente" + +#: paperless_mail/models.py:101 +msgid "Use mail address" +msgstr "Usar endereço de e-mail" + +#: paperless_mail/models.py:103 +msgid "Use name (or mail address if not available)" +msgstr "Usar nome (ou endereço de e-mail se não disponível)" + +#: paperless_mail/models.py:105 +msgid "Use correspondent selected below" +msgstr "Usar correspondente selecionado abaixo" + +#: paperless_mail/models.py:113 +msgid "order" +msgstr "ordem" + +#: paperless_mail/models.py:120 +msgid "account" +msgstr "conta" + +#: paperless_mail/models.py:124 +msgid "folder" +msgstr "pasta" + +#: paperless_mail/models.py:128 +msgid "filter from" +msgstr "filtrar de" + +#: paperless_mail/models.py:131 +msgid "filter subject" +msgstr "filtrar assunto" + +#: paperless_mail/models.py:134 +msgid "filter body" +msgstr "filtrar corpo" + +#: paperless_mail/models.py:138 +msgid "filter attachment filename" +msgstr "filtrar nome do arquivo anexo" + +#: paperless_mail/models.py:140 +msgid "" +"Only consume documents which entirely match this filename if specified. " +"Wildcards such as *.pdf or *invoice* are allowed. Case insensitive." +msgstr "" +"Consumir somente documentos que correspondem a este nome de arquivo se especificado.\n" +"Curingas como *.pdf ou *invoice* são permitidos. Sem diferenciação de maiúsculas e minúsculas." + +#: paperless_mail/models.py:146 +msgid "maximum age" +msgstr "idade máxima" + +#: paperless_mail/models.py:148 +msgid "Specified in days." +msgstr "Especificada em dias." + +#: paperless_mail/models.py:151 +msgid "attachment type" +msgstr "tipo de anexo" + +#: paperless_mail/models.py:154 +msgid "" +"Inline attachments include embedded images, so it's best to combine this " +"option with a filename filter." +msgstr "" +"Anexos inline incluem imagens inseridas, por isso é melhor combinar essa " +"opção com um filtro de nome de arquivo." + +#: paperless_mail/models.py:159 +msgid "action" +msgstr "ação" + +#: paperless_mail/models.py:165 +msgid "action parameter" +msgstr "parâmetro da ação" + +#: paperless_mail/models.py:167 +msgid "" +"Additional parameter for the action selected above, i.e., the target folder " +"of the move to folder action." +msgstr "" +"Parâmetro adicional para a ação selecionada acima, por exemplo: a pasta de " +"destino da ação de mover pasta." + +#: paperless_mail/models.py:173 +msgid "assign title from" +msgstr "atribuir título de" + +#: paperless_mail/models.py:183 +msgid "assign this tag" +msgstr "atribuir esta etiqueta" + +#: paperless_mail/models.py:191 +msgid "assign this document type" +msgstr "atribuir este tipo de documento" + +#: paperless_mail/models.py:195 +msgid "assign correspondent from" +msgstr "atribuir correspondente de" + +#: paperless_mail/models.py:205 +msgid "assign this correspondent" +msgstr "atribuir este correspondente" From 191d9a7b2b5bcdb91d1323275259e69b26c5abc1 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 16:39:41 +0100 Subject: [PATCH 204/225] add configuration for pt-br --- src-ui/angular.json | 3 ++- src-ui/src/app/app.module.ts | 2 ++ src-ui/src/app/services/settings.service.ts | 3 ++- src/paperless/settings.py | 3 ++- 4 files changed, 8 insertions(+), 3 deletions(-) diff --git a/src-ui/angular.json b/src-ui/angular.json index edf428fc7..2877408d2 100644 --- a/src-ui/angular.json +++ b/src-ui/angular.json @@ -19,7 +19,8 @@ "de": "src/locale/messages.de.xlf", "nl-NL": "src/locale/messages.nl_NL.xlf", "fr": "src/locale/messages.fr.xlf", - "en-GB": "src/locale/messages.en_GB.xlf" + "en-GB": "src/locale/messages.en_GB.xlf", + "pt-BR": "src/locale/messages.pt_BR.xlf" } }, "architect": { diff --git a/src-ui/src/app/app.module.ts b/src-ui/src/app/app.module.ts index 54c19a216..0773a9ace 100644 --- a/src-ui/src/app/app.module.ts +++ b/src-ui/src/app/app.module.ts @@ -64,11 +64,13 @@ import { CustomDatePipe } from './pipes/custom-date.pipe'; import localeFr from '@angular/common/locales/fr'; import localeNl from '@angular/common/locales/nl'; import localeDe from '@angular/common/locales/de'; +import localePt from '@angular/common/locales/pt-PT'; import localeEnGb from '@angular/common/locales/en-GB'; registerLocaleData(localeFr) registerLocaleData(localeNl) registerLocaleData(localeDe) +registerLocaleData(localePt, "pt-BR") registerLocaleData(localeEnGb) @NgModule({ diff --git a/src-ui/src/app/services/settings.service.ts b/src-ui/src/app/services/settings.service.ts index d511fa65f..393417e31 100644 --- a/src-ui/src/app/services/settings.service.ts +++ b/src-ui/src/app/services/settings.service.ts @@ -83,7 +83,8 @@ export class SettingsService { {code: "en-gb", name: $localize`English (GB)`, englishName: "English (GB)"}, {code: "de", name: $localize`German`, englishName: "German"}, {code: "nl", name: $localize`Dutch`, englishName: "Dutch"}, - {code: "fr", name: $localize`French`, englishName: "French"} + {code: "fr", name: $localize`French`, englishName: "French"}, + {code: "pt-br", name: $localize`Portuguese (Brazil)`, englishName: "Portuguese (Brazil)"} ] } diff --git a/src/paperless/settings.py b/src/paperless/settings.py index 7c4e36391..4c860351f 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings.py @@ -292,7 +292,8 @@ LANGUAGES = [ ("en-gb", _("English (GB)")), ("de", _("German")), ("nl-nl", _("Dutch")), - ("fr", _("French")) + ("fr", _("French")), + ("pt-br", _("Portuguese (Brazil)")) ] LOCALE_PATHS = [ From d836756e1920b0279f479c006580fee481a04fb6 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 16:43:55 +0100 Subject: [PATCH 205/225] fix up quoting (hope this gets pushed into transifex --- src-ui/src/locale/messages.pt_BR.xlf | 32 ++++++++++++++-------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/src-ui/src/locale/messages.pt_BR.xlf b/src-ui/src/locale/messages.pt_BR.xlf index ac3b557ad..0efe12ad0 100644 --- a/src-ui/src/locale/messages.pt_BR.xlf +++ b/src-ui/src/locale/messages.pt_BR.xlf @@ -59,7 +59,7 @@ View "" saved successfully. - Visualização &quot;&quot; salva com sucesso. + Visualização "" salva com sucesso. src/app/components/document-list/document-list.component.ts 115 @@ -67,7 +67,7 @@ View "" created successfully. - Visualização &quot;&quot; criada com sucesso. + Visualização "" criada com sucesso. src/app/components/document-list/document-list.component.ts 136 @@ -131,7 +131,7 @@ Save "" - Salvar &quot;&quot; + Salvar "" src/app/components/document-list/document-list.component.html 71 @@ -219,7 +219,7 @@ Do you really want to delete document ""? - Você realmente deseja excluir o documento &quot;&quot;? + Você realmente deseja excluir o documento ""? src/app/components/document-detail/document-detail.component.ts 204 @@ -475,7 +475,7 @@ Do you really want to delete the tag ""? - Você realmente deseja excluir a etiqueta &quot;&quot;? + Você realmente deseja excluir a etiqueta ""? src/app/components/manage/tag-list/tag-list.component.ts 30 @@ -563,7 +563,7 @@ Do you really want to delete the document type ""? - Você realmente deseja excluir o tipo de documento &quot;&quot;? + Você realmente deseja excluir o tipo de documento ""? src/app/components/manage/document-type-list/document-type-list.component.ts 26 @@ -587,7 +587,7 @@ Saved view "" deleted. - Visualização &quot;&quot; excluída. + Visualização "" excluída. src/app/components/manage/settings/settings.component.ts 67 @@ -907,7 +907,7 @@ Do you really want to delete the correspondent ""? - Você realmente deseja excluir o correspondente &quot;&quot;? + Você realmente deseja excluir o correspondente ""? src/app/components/manage/correspondent-list/correspondent-list.component.ts 26 @@ -1075,7 +1075,7 @@ Did you mean ""? - Você quis dizer &quot;&quot;? + Você quis dizer ""? src/app/components/search/search.component.html 13 @@ -1405,7 +1405,7 @@ "" - &quot;&quot; + "" src/app/components/document-list/bulk-editor/bulk-editor.component.ts 113 @@ -1413,7 +1413,7 @@ "" and "" - &quot;&quot; e &quot;&quot; + "" e "" src/app/components/document-list/bulk-editor/bulk-editor.component.ts 115 @@ -1431,7 +1431,7 @@ and "" - e &quot;&quot; + e "" src/app/components/document-list/bulk-editor/bulk-editor.component.ts 118 @@ -1448,7 +1448,7 @@ This operation will add the tag "" to selected document(s). - Essa operação irá adicionar a etiqueta &quot;&quot; em documento(s) selecionado(s). + Essa operação irá adicionar a etiqueta "" em documento(s) selecionado(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts 130 @@ -1464,7 +1464,7 @@ This operation will remove the tag "" from selected document(s). - Essa operação irá remover a etiqueta &quot;&quot; de documento(s) selecionado(s). + Essa operação irá remover a etiqueta "" de documento(s) selecionado(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts 135 @@ -1496,7 +1496,7 @@ This operation will assign the correspondent "" to selected document(s). - Essa operação irá atribuir o correspondente &quot;&quot; para documento(s) selecionado(s). + Essa operação irá atribuir o correspondente "" para documento(s) selecionado(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts 161 @@ -1520,7 +1520,7 @@ This operation will assign the document type "" to selected document(s). - Essa operação irá atribuir o tipo de documento &quot;&quot; para documento(s) selecionado(s). + Essa operação irá atribuir o tipo de documento "" para documento(s) selecionado(s). src/app/components/document-list/bulk-editor/bulk-editor.component.ts 184 From 15861ea41a47e3bc89bf5276974f7cb59e2167da Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 16:50:05 +0100 Subject: [PATCH 206/225] add translation strings for portuguese --- src-ui/messages.xlf | 37 +++++++++++++++----------- src/locale/en_US/LC_MESSAGES/django.po | 6 ++++- 2 files changed, 27 insertions(+), 16 deletions(-) diff --git a/src-ui/messages.xlf b/src-ui/messages.xlf index 95e0fad15..e1747ba03 100644 --- a/src-ui/messages.xlf +++ b/src-ui/messages.xlf @@ -531,28 +531,21 @@ Use system language src/app/components/manage/settings/settings.component.ts - 91 + 92 Use date format of display language src/app/components/manage/settings/settings.component.ts - 96 - - - - ISO 8601 - - src/app/components/manage/settings/settings.component.ts - 97 + 98 Error while storing settings on server: src/app/components/manage/settings/settings.component.ts - 114 + 115 @@ -1630,35 +1623,49 @@ English (US) src/app/services/settings.service.ts - 82 + 84 English (GB) src/app/services/settings.service.ts - 83 + 85 German src/app/services/settings.service.ts - 84 + 86 Dutch src/app/services/settings.service.ts - 85 + 87 French src/app/services/settings.service.ts - 86 + 88 + + + + Portuguese (Brazil) + + src/app/services/settings.service.ts + 89 + + + + ISO 8601 + + src/app/services/settings.service.ts + 95 diff --git a/src/locale/en_US/LC_MESSAGES/django.po b/src/locale/en_US/LC_MESSAGES/django.po index 27d628455..c4e8ee97b 100644 --- a/src/locale/en_US/LC_MESSAGES/django.po +++ b/src/locale/en_US/LC_MESSAGES/django.po @@ -8,7 +8,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-23 13:20+0100\n" +"POT-Creation-Date: 2021-02-24 16:49+0100\n" "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n" "Last-Translator: FULL NAME \n" "Language-Team: LANGUAGE \n" @@ -416,6 +416,10 @@ msgstr "" msgid "French" msgstr "" +#: paperless/settings.py:296 +msgid "Portuguese (Brazil)" +msgstr "" + #: paperless/urls.py:118 msgid "Paperless-ng administration" msgstr "" From 6b20177b09745f3197017873727deaf1338ae3a9 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 17:41:32 +0100 Subject: [PATCH 207/225] update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 83ef1c1ed..c935d90a8 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,7 @@ The documentation for Paperless-ng is available on [ReadTheDocs](https://paperle # Translation -Paperless is currently available in English, German, Dutch and French. Translation is coordinated at transifex: https://www.transifex.com/paperless/paperless-ng +Paperless is currently available in English, German, Dutch, French, and Portuguese. Translation is coordinated at transifex: https://www.transifex.com/paperless/paperless-ng If you want to see paperless in your own language, request that language at transifex and you can start translating after I approve the language. From 035b0a449b240a2615cb0599ff242604a7864e83 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 18:00:26 +0100 Subject: [PATCH 208/225] use ng-bootstrap date selector, with proper formatting/parsing according to the current locale #177 --- src-ui/src/app/app.module.ts | 14 +++-- .../input/date-time/date-time.component.html | 13 ---- .../input/date-time/date-time.component.ts | 61 ------------------- .../common/input/date/date.component.html | 15 +++++ .../date.component.scss} | 0 .../date.component.spec.ts} | 12 ++-- .../common/input/date/date.component.ts | 32 ++++++++++ .../document-detail.component.html | 2 +- .../manage/settings/settings.component.ts | 9 +-- src-ui/src/app/services/settings.service.ts | 36 ++++++++--- src-ui/src/app/utils/ngb-date-adapter.ts | 23 +++++++ .../app/utils/ngb-date-parser-formatter.ts | 59 ++++++++++++++++++ 12 files changed, 176 insertions(+), 100 deletions(-) delete mode 100644 src-ui/src/app/components/common/input/date-time/date-time.component.html delete mode 100644 src-ui/src/app/components/common/input/date-time/date-time.component.ts create mode 100644 src-ui/src/app/components/common/input/date/date.component.html rename src-ui/src/app/components/common/input/{date-time/date-time.component.scss => date/date.component.scss} (100%) rename src-ui/src/app/components/common/input/{date-time/date-time.component.spec.ts => date/date.component.spec.ts} (55%) create mode 100644 src-ui/src/app/components/common/input/date/date.component.ts create mode 100644 src-ui/src/app/utils/ngb-date-adapter.ts create mode 100644 src-ui/src/app/utils/ngb-date-parser-formatter.ts diff --git a/src-ui/src/app/app.module.ts b/src-ui/src/app/app.module.ts index 0773a9ace..9e0dc56de 100644 --- a/src-ui/src/app/app.module.ts +++ b/src-ui/src/app/app.module.ts @@ -3,7 +3,7 @@ import { NgModule } from '@angular/core'; import { AppRoutingModule } from './app-routing.module'; import { AppComponent } from './app.component'; -import { NgbModule } from '@ng-bootstrap/ng-bootstrap'; +import { NgbDateAdapter, NgbDateParserFormatter, NgbModule } from '@ng-bootstrap/ng-bootstrap'; import { HttpClientModule, HTTP_INTERCEPTORS } from '@angular/common/http'; import { DocumentListComponent } from './components/document-list/document-list.component'; import { DocumentDetailComponent } from './components/document-detail/document-detail.component'; @@ -39,7 +39,6 @@ import { SelectComponent } from './components/common/input/select/select.compone import { CheckComponent } from './components/common/input/check/check.component'; import { SaveViewConfigDialogComponent } from './components/document-list/save-view-config-dialog/save-view-config-dialog.component'; import { InfiniteScrollModule } from 'ngx-infinite-scroll'; -import { DateTimeComponent } from './components/common/input/date-time/date-time.component'; import { TagsComponent } from './components/common/input/tags/tags.component'; import { SortableDirective } from './directives/sortable.directive'; import { CookieService } from 'ngx-cookie-service'; @@ -60,6 +59,9 @@ import { NgSelectModule } from '@ng-select/ng-select'; import { NumberComponent } from './components/common/input/number/number.component'; import { SafePipe } from './pipes/safe.pipe'; import { CustomDatePipe } from './pipes/custom-date.pipe'; +import { DateComponent } from './components/common/input/date/date.component'; +import { ISODateAdapter } from './utils/ngb-date-adapter'; +import { LocalizedDateParserFormatter } from './utils/ngb-date-parser-formatter'; import localeFr from '@angular/common/locales/fr'; import localeNl from '@angular/common/locales/nl'; @@ -106,7 +108,6 @@ registerLocaleData(localeEnGb) SelectComponent, CheckComponent, SaveViewConfigDialogComponent, - DateTimeComponent, TagsComponent, SortableDirective, SavedViewWidgetComponent, @@ -122,7 +123,8 @@ registerLocaleData(localeEnGb) SelectDialogComponent, NumberComponent, SafePipe, - CustomDatePipe + CustomDatePipe, + DateComponent ], imports: [ BrowserModule, @@ -144,7 +146,9 @@ registerLocaleData(localeEnGb) multi: true }, FilterPipe, - DocumentTitlePipe + DocumentTitlePipe, + {provide: NgbDateAdapter, useClass: ISODateAdapter}, + {provide: NgbDateParserFormatter, useClass: LocalizedDateParserFormatter} ], bootstrap: [AppComponent] }) diff --git a/src-ui/src/app/components/common/input/date-time/date-time.component.html b/src-ui/src/app/components/common/input/date-time/date-time.component.html deleted file mode 100644 index 7c002db1b..000000000 --- a/src-ui/src/app/components/common/input/date-time/date-time.component.html +++ /dev/null @@ -1,13 +0,0 @@ -
    -
    - - -
    -
    - - -
    -
    - - - \ No newline at end of file diff --git a/src-ui/src/app/components/common/input/date-time/date-time.component.ts b/src-ui/src/app/components/common/input/date-time/date-time.component.ts deleted file mode 100644 index bce208ec8..000000000 --- a/src-ui/src/app/components/common/input/date-time/date-time.component.ts +++ /dev/null @@ -1,61 +0,0 @@ -import { formatDate } from '@angular/common'; -import { Component, forwardRef, Input, OnInit } from '@angular/core'; -import { ControlValueAccessor, NG_VALUE_ACCESSOR } from '@angular/forms'; - -@Component({ - providers: [{ - provide: NG_VALUE_ACCESSOR, - useExisting: forwardRef(() => DateTimeComponent), - multi: true - }], - selector: 'app-input-date-time', - templateUrl: './date-time.component.html', - styleUrls: ['./date-time.component.scss'] -}) -export class DateTimeComponent implements OnInit,ControlValueAccessor { - - constructor() { - } - - onChange = (newValue: any) => {}; - - onTouched = () => {}; - - writeValue(newValue: any): void { - this.dateValue = formatDate(newValue, 'yyyy-MM-dd', "en-US") - this.timeValue = formatDate(newValue, 'HH:mm:ss', 'en-US') - } - registerOnChange(fn: any): void { - this.onChange = fn; - } - registerOnTouched(fn: any): void { - this.onTouched = fn; - } - setDisabledState?(isDisabled: boolean): void { - this.disabled = isDisabled; - } - - @Input() - titleDate: string = "Date" - - @Input() - titleTime: string - - @Input() - disabled: boolean = false - - @Input() - hint: string - - timeValue - - dateValue - - ngOnInit(): void { - } - - dateOrTimeChanged() { - this.onChange(formatDate(this.dateValue + "T" + this.timeValue,"yyyy-MM-ddTHH:mm:ssZZZZZ", "en-us", "UTC")) - } - -} diff --git a/src-ui/src/app/components/common/input/date/date.component.html b/src-ui/src/app/components/common/input/date/date.component.html new file mode 100644 index 000000000..ca9ccc040 --- /dev/null +++ b/src-ui/src/app/components/common/input/date/date.component.html @@ -0,0 +1,15 @@ +
    + +
    + +
    + +
    +
    Invalid date.
    +
    +
    diff --git a/src-ui/src/app/components/common/input/date-time/date-time.component.scss b/src-ui/src/app/components/common/input/date/date.component.scss similarity index 100% rename from src-ui/src/app/components/common/input/date-time/date-time.component.scss rename to src-ui/src/app/components/common/input/date/date.component.scss diff --git a/src-ui/src/app/components/common/input/date-time/date-time.component.spec.ts b/src-ui/src/app/components/common/input/date/date.component.spec.ts similarity index 55% rename from src-ui/src/app/components/common/input/date-time/date-time.component.spec.ts rename to src-ui/src/app/components/common/input/date/date.component.spec.ts index 0657768bd..ea92c7b30 100644 --- a/src-ui/src/app/components/common/input/date-time/date-time.component.spec.ts +++ b/src-ui/src/app/components/common/input/date/date.component.spec.ts @@ -1,20 +1,20 @@ import { ComponentFixture, TestBed } from '@angular/core/testing'; -import { DateTimeComponent } from './date-time.component'; +import { DateComponent } from './date.component'; -describe('DateTimeComponent', () => { - let component: DateTimeComponent; - let fixture: ComponentFixture; +describe('DateComponent', () => { + let component: DateComponent; + let fixture: ComponentFixture; beforeEach(async () => { await TestBed.configureTestingModule({ - declarations: [ DateTimeComponent ] + declarations: [ DateComponent ] }) .compileComponents(); }); beforeEach(() => { - fixture = TestBed.createComponent(DateTimeComponent); + fixture = TestBed.createComponent(DateComponent); component = fixture.componentInstance; fixture.detectChanges(); }); diff --git a/src-ui/src/app/components/common/input/date/date.component.ts b/src-ui/src/app/components/common/input/date/date.component.ts new file mode 100644 index 000000000..b3b863581 --- /dev/null +++ b/src-ui/src/app/components/common/input/date/date.component.ts @@ -0,0 +1,32 @@ +import { Component, forwardRef, Input, OnInit, ViewChild } from '@angular/core'; +import { ControlValueAccessor, NG_VALUE_ACCESSOR } from '@angular/forms'; +import { NgbDateAdapter, NgbDateParserFormatter, NgbDatepickerContent } from '@ng-bootstrap/ng-bootstrap'; +import { SettingsService } from 'src/app/services/settings.service'; +import { v4 as uuidv4 } from 'uuid'; +import { AbstractInputComponent } from '../abstract-input'; + + +@Component({ + providers: [{ + provide: NG_VALUE_ACCESSOR, + useExisting: forwardRef(() => DateComponent), + multi: true + }], + selector: 'app-input-date', + templateUrl: './date.component.html', + styleUrls: ['./date.component.scss'] +}) +export class DateComponent extends AbstractInputComponent implements OnInit { + + constructor(private settings: SettingsService) { + super() + } + + ngOnInit(): void { + super.ngOnInit() + this.placeholder = this.settings.getLocalizedDateInputFormat() + } + + placeholder: string + +} diff --git a/src-ui/src/app/components/document-detail/document-detail.component.html b/src-ui/src/app/components/document-detail/document-detail.component.html index 2814a1242..f9b87aee3 100644 --- a/src-ui/src/app/components/document-detail/document-detail.component.html +++ b/src-ui/src/app/components/document-detail/document-detail.component.html @@ -58,7 +58,7 @@ - + o.code == dateLocale)?.dateInputFormat || "yyyy-mm-dd" + } + get(key: string): any { let setting = SETTINGS.find(s => s.key == key) diff --git a/src-ui/src/app/utils/ngb-date-adapter.ts b/src-ui/src/app/utils/ngb-date-adapter.ts new file mode 100644 index 000000000..19711319d --- /dev/null +++ b/src-ui/src/app/utils/ngb-date-adapter.ts @@ -0,0 +1,23 @@ +import { Injectable } from "@angular/core"; +import { NgbDateAdapter, NgbDateStruct } from "@ng-bootstrap/ng-bootstrap"; + +@Injectable() +export class ISODateAdapter extends NgbDateAdapter { + + fromModel(value: string | null): NgbDateStruct | null { + if (value) { + let date = new Date(value) + return { + day : date.getDate(), + month : date.getMonth() + 1, + year : date.getFullYear() + } + } else { + return null + } + } + + toModel(date: NgbDateStruct | null): string | null { + return date ? new Date(date.year, date.month - 1, date.day).toISOString() : null + } +} diff --git a/src-ui/src/app/utils/ngb-date-parser-formatter.ts b/src-ui/src/app/utils/ngb-date-parser-formatter.ts new file mode 100644 index 000000000..7d819a6bf --- /dev/null +++ b/src-ui/src/app/utils/ngb-date-parser-formatter.ts @@ -0,0 +1,59 @@ +import { Injectable } from "@angular/core" +import { NgbDateParserFormatter, NgbDateStruct } from "@ng-bootstrap/ng-bootstrap" +import { SettingsService } from "../services/settings.service" + +@Injectable() +export class LocalizedDateParserFormatter extends NgbDateParserFormatter { + + constructor(private settings: SettingsService) { + super() + } + + private getDateInputFormat() { + return this.settings.getLocalizedDateInputFormat() + } + + /** + * This constructs a regular expression from a date input format which is then + * used to parse dates. + */ + private getDateParseRegex() { + return new RegExp( + "^" + this.getDateInputFormat() + .replace('dd', '(?[0-9]+)') + .replace('mm', '(?[0-9]+)') + .replace('yyyy', '(?[0-9]+)') + .split('.').join('\\.\\s*') + "$" // allow whitespace(s) after dot (specific for German) + ) + } + + parse(value: string): NgbDateStruct | null { + let match = this.getDateParseRegex().exec(value) + if (match) { + let dateStruct = { + day: +match.groups.day, + month: +match.groups.month, + year: +match.groups.year + } + if (dateStruct.year <= (new Date().getFullYear() - 2000)) { + dateStruct.year += 2000 + } else if (dateStruct.year < 100) { + dateStruct.year += 1900 + } + return dateStruct + } else { + return null + } + } + + format(date: NgbDateStruct | null): string { + if (date) { + return this.getDateInputFormat() + .replace('dd', date.day.toString().padStart(2, '0')) + .replace('mm', date.month.toString().padStart(2, '0')) + .replace('yyyy', date.year.toString().padStart(4, '0')) + } else { + return null + } + } +} \ No newline at end of file From 96b8d35a000f0b1ebb32dbe045fbcc969704520c Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 18:00:35 +0100 Subject: [PATCH 209/225] update dependencies --- src-ui/package-lock.json | 6 +++--- src-ui/package.json | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src-ui/package-lock.json b/src-ui/package-lock.json index 148bc72eb..fd6d69538 100644 --- a/src-ui/package-lock.json +++ b/src-ui/package-lock.json @@ -2055,9 +2055,9 @@ } }, "@ng-bootstrap/ng-bootstrap": { - "version": "8.0.0", - "resolved": "https://registry.npmjs.org/@ng-bootstrap/ng-bootstrap/-/ng-bootstrap-8.0.0.tgz", - "integrity": "sha512-v77Gfd8xHH+exq0WqIqVRlxbUEHdA/2+RUJenUP2IDTQN9E1rWl7O461/kosr+0XPuxPArHQJxhh/WsCYckcNg==", + "version": "8.0.4", + "resolved": "https://registry.npmjs.org/@ng-bootstrap/ng-bootstrap/-/ng-bootstrap-8.0.4.tgz", + "integrity": "sha512-EdxTwOPOtlvfnwrglPniulmzdnXdXH3lTGaGAY1HrYRvdtGg6wicRvl+BvwVE/3Qik5NPkOWMVghUHpv3evIYg==", "requires": { "tslib": "^2.0.0" } diff --git a/src-ui/package.json b/src-ui/package.json index d6082c6b9..a4d014284 100644 --- a/src-ui/package.json +++ b/src-ui/package.json @@ -20,7 +20,7 @@ "@angular/platform-browser": "~10.1.5", "@angular/platform-browser-dynamic": "~10.1.5", "@angular/router": "~10.1.5", - "@ng-bootstrap/ng-bootstrap": "^8.0.0", + "@ng-bootstrap/ng-bootstrap": "^8.0.4", "@ng-select/ng-select": "^5.0.9", "bootstrap": "^4.5.0", "file-saver": "^2.0.5", From d2719b530992ad15e0fcdb8ed8015d6358715f68 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 18:08:55 +0100 Subject: [PATCH 210/225] added a missing string --- src-ui/messages.xlf | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/src-ui/messages.xlf b/src-ui/messages.xlf index e1747ba03..d04d5a6f8 100644 --- a/src-ui/messages.xlf +++ b/src-ui/messages.xlf @@ -1598,6 +1598,13 @@ 21     + + Invalid date. + + src/app/components/common/input/date/date.component.html + 13 + + Yes @@ -1623,49 +1630,49 @@ English (US) src/app/services/settings.service.ts - 84 + 88 English (GB) src/app/services/settings.service.ts - 85 + 89 German src/app/services/settings.service.ts - 86 + 90 Dutch src/app/services/settings.service.ts - 87 + 91 French src/app/services/settings.service.ts - 88 + 92 Portuguese (Brazil) src/app/services/settings.service.ts - 89 + 93 ISO 8601 src/app/services/settings.service.ts - 95 + 98 From fc75e88cb790b5eb48a76b38c59840a1bf7530b1 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 17:10:20 +0000 Subject: [PATCH 211/225] Apply translations in de translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'de' language. --- src/locale/de/LC_MESSAGES/django.po | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/locale/de/LC_MESSAGES/django.po b/src/locale/de/LC_MESSAGES/django.po index c031ebc85..0343c5a94 100644 --- a/src/locale/de/LC_MESSAGES/django.po +++ b/src/locale/de/LC_MESSAGES/django.po @@ -11,7 +11,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-23 13:20+0100\n" +"POT-Creation-Date: 2021-02-24 16:49+0100\n" "PO-Revision-Date: 2021-02-16 18:37+0000\n" "Last-Translator: Jonas Winkler, 2021\n" "Language-Team: German (https://www.transifex.com/paperless/teams/115905/de/)\n" @@ -426,6 +426,10 @@ msgstr "Niederländisch" msgid "French" msgstr "Französisch" +#: paperless/settings.py:296 +msgid "Portuguese (Brazil)" +msgstr "Portugiesisch (Brasilien)" + #: paperless/urls.py:118 msgid "Paperless-ng administration" msgstr "Paperless-ng Administration" From 56204f8497539ca2a04297ddafd7fa1f60bb8bfe Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 17:10:24 +0000 Subject: [PATCH 212/225] Translate /src-ui/messages.xlf in de translation completed for the source file '/src-ui/messages.xlf' on the 'de' language. --- src-ui/src/locale/messages.de.xlf | 48 ++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 16 deletions(-) diff --git a/src-ui/src/locale/messages.de.xlf b/src-ui/src/locale/messages.de.xlf index 6802f8dce..a9762c286 100644 --- a/src-ui/src/locale/messages.de.xlf +++ b/src-ui/src/locale/messages.de.xlf @@ -606,7 +606,7 @@ Benutze Systemsprache src/app/components/manage/settings/settings.component.ts - 91 + 92 @@ -614,15 +614,7 @@ Benutze Datumsformat der Anzeigesprache src/app/components/manage/settings/settings.component.ts - 96 - - - - ISO 8601 - ISO 8601 - - src/app/components/manage/settings/settings.component.ts - 97 + 98 @@ -630,7 +622,7 @@ Fehler beim Speichern der Einstellungen auf dem Server: src/app/components/manage/settings/settings.component.ts - 114 + 115 @@ -1832,6 +1824,14 @@ 21 + + Invalid date. + Ungültiges Datum. + + src/app/components/common/input/date/date.component.html + 13 + + Yes Ja @@ -1861,7 +1861,7 @@ Englisch (US) src/app/services/settings.service.ts - 82 + 88 @@ -1869,7 +1869,7 @@ Englisch (UK) src/app/services/settings.service.ts - 83 + 89 @@ -1877,7 +1877,7 @@ Deutsch src/app/services/settings.service.ts - 84 + 90 @@ -1885,7 +1885,7 @@ Niederländisch src/app/services/settings.service.ts - 85 + 91 @@ -1893,7 +1893,23 @@ Französisch src/app/services/settings.service.ts - 86 + 92 + + + + Portuguese (Brazil) + Portugiesisch (Brasilien) + + src/app/services/settings.service.ts + 93 + + + + ISO 8601 + ISO 8601 + + src/app/services/settings.service.ts + 98 From 0db14d6f744f3045731996c3b2a42a4362ec7788 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 17:18:07 +0000 Subject: [PATCH 213/225] Apply translations in pt_BR translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'pt_BR' language. --- src/locale/pt_BR/LC_MESSAGES/django.po | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/locale/pt_BR/LC_MESSAGES/django.po b/src/locale/pt_BR/LC_MESSAGES/django.po index e1a74b104..7800b0c88 100644 --- a/src/locale/pt_BR/LC_MESSAGES/django.po +++ b/src/locale/pt_BR/LC_MESSAGES/django.po @@ -12,7 +12,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-23 13:20+0100\n" +"POT-Creation-Date: 2021-02-24 16:49+0100\n" "PO-Revision-Date: 2021-02-16 18:37+0000\n" "Last-Translator: Rodrigo A , 2021\n" "Language-Team: Portuguese (Brazil) (https://www.transifex.com/paperless/teams/115905/pt_BR/)\n" @@ -425,6 +425,10 @@ msgstr "Holandês" msgid "French" msgstr "Francês" +#: paperless/settings.py:296 +msgid "Portuguese (Brazil)" +msgstr "Português (Brasil)" + #: paperless/urls.py:118 msgid "Paperless-ng administration" msgstr "Administração do Paperless-ng" From bec5365ac9378ac7e8309b48f0be85d84d2cfc33 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 17:18:26 +0000 Subject: [PATCH 214/225] Translate /src-ui/messages.xlf in pt_BR translation completed for the source file '/src-ui/messages.xlf' on the 'pt_BR' language. --- src-ui/src/locale/messages.pt_BR.xlf | 48 ++++++++++++++++++---------- 1 file changed, 32 insertions(+), 16 deletions(-) diff --git a/src-ui/src/locale/messages.pt_BR.xlf b/src-ui/src/locale/messages.pt_BR.xlf index 0efe12ad0..ff438073d 100644 --- a/src-ui/src/locale/messages.pt_BR.xlf +++ b/src-ui/src/locale/messages.pt_BR.xlf @@ -606,7 +606,7 @@ Usar linguagem do sistema src/app/components/manage/settings/settings.component.ts - 91 + 92 @@ -614,15 +614,7 @@ Usar formato de data da linguagem de exibição src/app/components/manage/settings/settings.component.ts - 96 - - - - ISO 8601 - ISO 8601 - - src/app/components/manage/settings/settings.component.ts - 97 + 98 @@ -630,7 +622,7 @@ Erro ao salvar configurações: src/app/components/manage/settings/settings.component.ts - 114 + 115 @@ -1832,6 +1824,14 @@ 21 + + Invalid date. + Data inválida. + + src/app/components/common/input/date/date.component.html + 13 + + Yes Sim @@ -1861,7 +1861,7 @@ Inglês (US) src/app/services/settings.service.ts - 82 + 88 @@ -1869,7 +1869,7 @@ Inglês (GB) src/app/services/settings.service.ts - 83 + 89 @@ -1877,7 +1877,7 @@ Alemão src/app/services/settings.service.ts - 84 + 90 @@ -1885,7 +1885,7 @@ Holandês src/app/services/settings.service.ts - 85 + 91 @@ -1893,7 +1893,23 @@ Francês src/app/services/settings.service.ts - 86 + 92 + + + + Portuguese (Brazil) + Português (Brasil) + + src/app/services/settings.service.ts + 93 + + + + ISO 8601 + ISO 8601 + + src/app/services/settings.service.ts + 98 From 1896aa7da1fdc417b1b290db52ecfb2c3edd0570 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 18:45:05 +0100 Subject: [PATCH 215/225] use new date inputs for the date filter as well --- src-ui/src/app/app.module.ts | 4 +-- .../date-dropdown.component.html | 22 +++++++++++++-- .../date-dropdown/date-dropdown.component.ts | 16 +++++++++-- src-ui/src/app/utils/ngb-iso-date-adapter.ts | 27 +++++++++++++++++++ ...dapter.ts => ngb-iso-date-time-adapter.ts} | 2 +- 5 files changed, 64 insertions(+), 7 deletions(-) create mode 100644 src-ui/src/app/utils/ngb-iso-date-adapter.ts rename src-ui/src/app/utils/{ngb-date-adapter.ts => ngb-iso-date-time-adapter.ts} (89%) diff --git a/src-ui/src/app/app.module.ts b/src-ui/src/app/app.module.ts index 9e0dc56de..46d8df9a1 100644 --- a/src-ui/src/app/app.module.ts +++ b/src-ui/src/app/app.module.ts @@ -60,7 +60,7 @@ import { NumberComponent } from './components/common/input/number/number.compone import { SafePipe } from './pipes/safe.pipe'; import { CustomDatePipe } from './pipes/custom-date.pipe'; import { DateComponent } from './components/common/input/date/date.component'; -import { ISODateAdapter } from './utils/ngb-date-adapter'; +import { ISODateTimeAdapter } from './utils/ngb-iso-date-time-adapter'; import { LocalizedDateParserFormatter } from './utils/ngb-date-parser-formatter'; import localeFr from '@angular/common/locales/fr'; @@ -147,7 +147,7 @@ registerLocaleData(localeEnGb) }, FilterPipe, DocumentTitlePipe, - {provide: NgbDateAdapter, useClass: ISODateAdapter}, + {provide: NgbDateAdapter, useClass: ISODateTimeAdapter}, {provide: NgbDateParserFormatter, useClass: LocalizedDateParserFormatter} ], bootstrap: [AppComponent] diff --git a/src-ui/src/app/components/common/date-dropdown/date-dropdown.component.html b/src-ui/src/app/components/common/date-dropdown/date-dropdown.component.html index b70a04103..ee8e6e6f4 100644 --- a/src-ui/src/app/components/common/date-dropdown/date-dropdown.component.html +++ b/src-ui/src/app/components/common/date-dropdown/date-dropdown.component.html @@ -20,8 +20,17 @@
    - + +
    + +
    +
    - + +
    + +
    + diff --git a/src-ui/src/app/components/common/date-dropdown/date-dropdown.component.ts b/src-ui/src/app/components/common/date-dropdown/date-dropdown.component.ts index 27472bdc7..e21b28312 100644 --- a/src-ui/src/app/components/common/date-dropdown/date-dropdown.component.ts +++ b/src-ui/src/app/components/common/date-dropdown/date-dropdown.component.ts @@ -1,7 +1,10 @@ import { formatDate } from '@angular/common'; import { Component, EventEmitter, Input, Output, OnInit, OnDestroy } from '@angular/core'; +import { NgbDateAdapter } from '@ng-bootstrap/ng-bootstrap'; import { Subject, Subscription } from 'rxjs'; -import { debounceTime, distinctUntilChanged } from 'rxjs/operators'; +import { debounceTime } from 'rxjs/operators'; +import { SettingsService } from 'src/app/services/settings.service'; +import { ISODateAdapter } from 'src/app/utils/ngb-iso-date-adapter'; export interface DateSelection { before?: string @@ -16,10 +19,17 @@ const LAST_YEAR = 3 @Component({ selector: 'app-date-dropdown', templateUrl: './date-dropdown.component.html', - styleUrls: ['./date-dropdown.component.scss'] + styleUrls: ['./date-dropdown.component.scss'], + providers: [ + {provide: NgbDateAdapter, useClass: ISODateAdapter}, + ] }) export class DateDropdownComponent implements OnInit, OnDestroy { + constructor(settings: SettingsService) { + this.datePlaceHolder = settings.getLocalizedDateInputFormat() + } + quickFilters = [ {id: LAST_7_DAYS, name: $localize`Last 7 days`}, {id: LAST_MONTH, name: $localize`Last month`}, @@ -27,6 +37,8 @@ export class DateDropdownComponent implements OnInit, OnDestroy { {id: LAST_YEAR, name: $localize`Last year`} ] + datePlaceHolder: string + @Input() dateBefore: string diff --git a/src-ui/src/app/utils/ngb-iso-date-adapter.ts b/src-ui/src/app/utils/ngb-iso-date-adapter.ts new file mode 100644 index 000000000..e43602a16 --- /dev/null +++ b/src-ui/src/app/utils/ngb-iso-date-adapter.ts @@ -0,0 +1,27 @@ +import { Injectable } from "@angular/core"; +import { NgbDateAdapter, NgbDateStruct } from "@ng-bootstrap/ng-bootstrap"; + +@Injectable() +export class ISODateAdapter extends NgbDateAdapter { + + fromModel(value: string | null): NgbDateStruct | null { + if (value) { + let date = new Date(value) + return { + day : date.getDate(), + month : date.getMonth() + 1, + year : date.getFullYear() + } + } else { + return null + } + } + + toModel(date: NgbDateStruct | null): string | null { + if (date) { + return date.year.toString().padStart(4, '0') + "-" + date.month.toString().padStart(2, '0') + "-" + date.day.toString().padStart(2, '0') + } else { + return null + } + } +} diff --git a/src-ui/src/app/utils/ngb-date-adapter.ts b/src-ui/src/app/utils/ngb-iso-date-time-adapter.ts similarity index 89% rename from src-ui/src/app/utils/ngb-date-adapter.ts rename to src-ui/src/app/utils/ngb-iso-date-time-adapter.ts index 19711319d..21a97a19a 100644 --- a/src-ui/src/app/utils/ngb-date-adapter.ts +++ b/src-ui/src/app/utils/ngb-iso-date-time-adapter.ts @@ -2,7 +2,7 @@ import { Injectable } from "@angular/core"; import { NgbDateAdapter, NgbDateStruct } from "@ng-bootstrap/ng-bootstrap"; @Injectable() -export class ISODateAdapter extends NgbDateAdapter { +export class ISODateTimeAdapter extends NgbDateAdapter { fromModel(value: string | null): NgbDateStruct | null { if (value) { From a5ba14e4464dbbc9e4167c98f644a7a5b34721c2 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 19:03:03 +0100 Subject: [PATCH 216/225] version bump --- src-ui/src/environments/environment.prod.ts | 2 +- src/paperless/version.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src-ui/src/environments/environment.prod.ts b/src-ui/src/environments/environment.prod.ts index d170a0f02..6c97ac1cb 100644 --- a/src-ui/src/environments/environment.prod.ts +++ b/src-ui/src/environments/environment.prod.ts @@ -2,7 +2,7 @@ export const environment = { production: true, apiBaseUrl: "/api/", appTitle: "Paperless-ng", - version: "1.2.0", + version: "1.2.1", webSocketHost: window.location.host, webSocketProtocol: (window.location.protocol == "https:" ? "wss:" : "ws:") }; diff --git a/src/paperless/version.py b/src/paperless/version.py index 09ac5f672..d93cd02b0 100644 --- a/src/paperless/version.py +++ b/src/paperless/version.py @@ -1 +1 @@ -__version__ = (1, 2, 0) +__version__ = (1, 2, 1) From f038bb90bb2c936064f5997bf627997056f7405d Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 19:03:21 +0100 Subject: [PATCH 217/225] fixes for #600 --- src-ui/src/index.html | 2 +- src/documents/templates/index.html | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src-ui/src/index.html b/src-ui/src/index.html index 39ecd824c..4308a6b16 100644 --- a/src-ui/src/index.html +++ b/src-ui/src/index.html @@ -9,7 +9,7 @@ - + diff --git a/src/documents/templates/index.html b/src/documents/templates/index.html index 285e48d3f..b71aaaeb6 100644 --- a/src/documents/templates/index.html +++ b/src/documents/templates/index.html @@ -15,6 +15,7 @@ + {% translate "Paperless-ng is loading..." %} From 6a64bf6fc16e5b806faf1aa5a91866913ecc6051 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 19:05:06 +0100 Subject: [PATCH 218/225] changelog --- docs/changelog.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index b23ca5137..d2ec7ec69 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -5,6 +5,17 @@ Changelog ********* +paperless-ng 1.2.1 +################## + +* Added translation into Portuguese (Brazil). + +* The date input fields now respect the currently selected date format. + +* Added a fancy icon when adding paperless to the home screen on iOS devices. + +* When using regular expression matching, the regular expression is now validated before saving the tag/correspondent/type. + paperless-ng 1.2.0 ################## From 6ae65e0ab43b90122b0d94871fc912f022693149 Mon Sep 17 00:00:00 2001 From: jonaswinkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 19:21:15 +0100 Subject: [PATCH 219/225] changelog --- docs/changelog.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index d2ec7ec69..137fa2fd2 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -8,11 +8,11 @@ Changelog paperless-ng 1.2.1 ################## -* Added translation into Portuguese (Brazil). +* `Rodrigo Avelino `_ translated Paperless into Portuguese (Brazil). * The date input fields now respect the currently selected date format. -* Added a fancy icon when adding paperless to the home screen on iOS devices. +* Added a fancy icon when adding paperless to the home screen on iOS devices. Thanks to `Joel Nordell `_. * When using regular expression matching, the regular expression is now validated before saving the tag/correspondent/type. From 8465b8cfca7ca8c36c02047166bd86e50e8feabc Mon Sep 17 00:00:00 2001 From: Jonas Winkler <17569239+jonaswinkler@users.noreply.github.com> Date: Wed, 24 Feb 2021 19:27:16 +0100 Subject: [PATCH 220/225] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 01ac30fa1..b21173ba5 100644 --- a/README.md +++ b/README.md @@ -77,9 +77,9 @@ The documentation for Paperless-ng is available on [ReadTheDocs](https://paperle # Translation -Paperless is currently available in English, German, Dutch, French, and Portuguese. Translation is coordinated at transifex: https://www.transifex.com/paperless/paperless-ng +Paperless is currently available in English, German, Dutch, French, and Portuguese. -If you want to see paperless in your own language, request that language at transifex and you can start translating after I approve the language. +There's an active translation project at transifex! If you want to help out by translating paperless into your language, please head over to https://github.com/jonaswinkler/paperless-ng/issues/212 for details. # Feature Requests From c22af0a7825fad1db9bc23075e06449a182e63b8 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 18:43:39 +0000 Subject: [PATCH 221/225] Apply translations in en_GB translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'en_GB' language. --- src/locale/en_GB/LC_MESSAGES/django.po | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/src/locale/en_GB/LC_MESSAGES/django.po b/src/locale/en_GB/LC_MESSAGES/django.po index f0c5b0381..dbdd4b2a5 100644 --- a/src/locale/en_GB/LC_MESSAGES/django.po +++ b/src/locale/en_GB/LC_MESSAGES/django.po @@ -12,7 +12,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-16 14:52+0100\n" +"POT-Creation-Date: 2021-02-24 16:49+0100\n" "PO-Revision-Date: 2021-02-16 18:37+0000\n" "Last-Translator: Jonas Winkler, 2021\n" "Language-Team: English (United Kingdom) (https://www.transifex.com/paperless/teams/115905/en_GB/)\n" @@ -355,7 +355,12 @@ msgstr "filter rule" msgid "filter rules" msgstr "filter rules" -#: documents/serialisers.py:370 +#: documents/serialisers.py:52 +#, python-format +msgid "Invalid regular expresssion: %(error)s" +msgstr "Invalid regular expresssion: %(error)s" + +#: documents/serialisers.py:378 #, python-format msgid "File type %(type)s not supported" msgstr "File type %(type)s not supported" @@ -420,7 +425,11 @@ msgstr "Dutch" msgid "French" msgstr "French" -#: paperless/urls.py:114 +#: paperless/settings.py:296 +msgid "Portuguese (Brazil)" +msgstr "Portuguese (Brazil)" + +#: paperless/urls.py:118 msgid "Paperless-ng administration" msgstr "Paperless-ng administration" From 800f4deb909ee2b661e234743d1fa2fbf1698c87 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 18:43:40 +0000 Subject: [PATCH 222/225] Translate /src-ui/messages.xlf in en_GB translation completed for the source file '/src-ui/messages.xlf' on the 'en_GB' language. --- src-ui/src/locale/messages.en_GB.xlf | 48 ++++++++++++++++++---------- 1 file changed, 32 insertions(+), 16 deletions(-) diff --git a/src-ui/src/locale/messages.en_GB.xlf b/src-ui/src/locale/messages.en_GB.xlf index f5e711c39..b413a50a1 100644 --- a/src-ui/src/locale/messages.en_GB.xlf +++ b/src-ui/src/locale/messages.en_GB.xlf @@ -606,7 +606,7 @@ Use system language src/app/components/manage/settings/settings.component.ts - 91 + 92     @@ -614,15 +614,7 @@ Use date format of display language src/app/components/manage/settings/settings.component.ts - 96 - - - - ISO 8601 - ISO 8601 - - src/app/components/manage/settings/settings.component.ts - 97 + 98 @@ -630,7 +622,7 @@ Error while storing settings on server: src/app/components/manage/settings/settings.component.ts - 114 + 115 @@ -1832,6 +1824,14 @@ 21 + + Invalid date. + Invalid date. + + src/app/components/common/input/date/date.component.html + 13 + + Yes Yes @@ -1861,7 +1861,7 @@ English (US) src/app/services/settings.service.ts - 82 + 88 @@ -1869,7 +1869,7 @@ English (GB) src/app/services/settings.service.ts - 83 + 89 @@ -1877,7 +1877,7 @@ German src/app/services/settings.service.ts - 84 + 90 @@ -1885,7 +1885,7 @@ Dutch src/app/services/settings.service.ts - 85 + 91 @@ -1893,7 +1893,23 @@ French src/app/services/settings.service.ts - 86 + 92 + + + + Portuguese (Brazil) + Portuguese (Brazil) + + src/app/services/settings.service.ts + 93 + + + + ISO 8601 + ISO 8601 + + src/app/services/settings.service.ts + 98 From 2f2b5b90ea0812e3d9212169f4847b53b93b5f2e Mon Sep 17 00:00:00 2001 From: Michael Shamoon <4887959+nikonratm@users.noreply.github.com> Date: Sun, 17 Jan 2021 19:46:44 +0100 Subject: [PATCH 223/225] Dark mode compatibility --- src-ui/src/theme_dark.scss | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/src-ui/src/theme_dark.scss b/src-ui/src/theme_dark.scss index 4e850f017..751ff2b15 100644 --- a/src-ui/src/theme_dark.scss +++ b/src-ui/src/theme_dark.scss @@ -246,6 +246,16 @@ $border-color-dark-mode: #47494f; } } + .btn-light:not(:disabled):not(.disabled) { + background-color: $bg-dark-mode; + color: $text-color-dark-mode-accent; + + &:hover { + background-color: $text-color-dark-mode; + color: $bg-dark-mode; + } + } + .btn-link:not(:disabled):not(.disabled) { color: $primary-dark-mode; } @@ -366,6 +376,12 @@ $border-color-dark-mode: #47494f; .progress-bar.bg-primary { background-color: darken($primary-dark-mode, 5%) !important; } + + .ngb-dp-header, + .ngb-dp-weekdays, + .ngb-dp-month { + background-color: $bg-light-dark-mode; + } } body.color-scheme-dark { From e43ab23a45e7065f8cb29417445ba00c5e33cfbd Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 19:29:33 +0000 Subject: [PATCH 224/225] Apply translations in fr translation completed for the source file '/src/locale/en_US/LC_MESSAGES/django.po' on the 'fr' language. --- src/locale/fr/LC_MESSAGES/django.po | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/locale/fr/LC_MESSAGES/django.po b/src/locale/fr/LC_MESSAGES/django.po index 6acd81649..3b4fbce3f 100644 --- a/src/locale/fr/LC_MESSAGES/django.po +++ b/src/locale/fr/LC_MESSAGES/django.po @@ -12,7 +12,7 @@ msgid "" msgstr "" "Project-Id-Version: PACKAGE VERSION\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2021-02-23 13:20+0100\n" +"POT-Creation-Date: 2021-02-24 16:49+0100\n" "PO-Revision-Date: 2021-02-16 18:37+0000\n" "Last-Translator: Philmo67, 2021\n" "Language-Team: French (https://www.transifex.com/paperless/teams/115905/fr/)\n" @@ -428,6 +428,10 @@ msgstr "Néerlandais" msgid "French" msgstr "Français" +#: paperless/settings.py:296 +msgid "Portuguese (Brazil)" +msgstr "Portugais (Brésil)" + #: paperless/urls.py:118 msgid "Paperless-ng administration" msgstr "Administration de Paperless-ng" From d6cbea97f9b5993f12b1d1ea02fc99ca3a811587 Mon Sep 17 00:00:00 2001 From: "transifex-integration[bot]" <43880903+transifex-integration[bot]@users.noreply.github.com> Date: Wed, 24 Feb 2021 19:30:01 +0000 Subject: [PATCH 225/225] Translate /src-ui/messages.xlf in fr translation completed for the source file '/src-ui/messages.xlf' on the 'fr' language. --- src-ui/src/locale/messages.fr.xlf | 50 ++++++++++++++++++++----------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/src-ui/src/locale/messages.fr.xlf b/src-ui/src/locale/messages.fr.xlf index 79045b3b4..05b9fdab7 100644 --- a/src-ui/src/locale/messages.fr.xlf +++ b/src-ui/src/locale/messages.fr.xlf @@ -606,7 +606,7 @@ Utiliser la langue du système src/app/components/manage/settings/settings.component.ts - 91 + 92 @@ -614,15 +614,7 @@ Utiliser le format de date de la langue d'affichage src/app/components/manage/settings/settings.component.ts - 96 - - - - ISO 8601 - ISO 8601 - - src/app/components/manage/settings/settings.component.ts - 97 + 98 @@ -630,7 +622,7 @@ Une erreur s'est produite lors de l'enregistrement des paramètres sur le serveur : src/app/components/manage/settings/settings.component.ts - 114 + 115 @@ -819,7 +811,7 @@ Document processing - Traitement de document + Traitement de documents src/app/components/manage/settings/settings.component.html 118 @@ -1832,6 +1824,14 @@ 21 + + Invalid date. + Date incorrecte. + + src/app/components/common/input/date/date.component.html + 13 + + Yes Oui @@ -1861,7 +1861,7 @@ Anglais (US) src/app/services/settings.service.ts - 82 + 88 @@ -1869,7 +1869,7 @@ Anglais (GB) src/app/services/settings.service.ts - 83 + 89 @@ -1877,7 +1877,7 @@ Allemand src/app/services/settings.service.ts - 84 + 90 @@ -1885,7 +1885,7 @@ Néerlandais src/app/services/settings.service.ts - 85 + 91 @@ -1893,7 +1893,23 @@ Français src/app/services/settings.service.ts - 86 + 92 + + + + Portuguese (Brazil) + Portugais (Brésil) + + src/app/services/settings.service.ts + 93 + + + + ISO 8601 + ISO 8601 + + src/app/services/settings.service.ts + 98