Merge branch 'dev' into mail_rework

docs/changelog.rst

@@ -38,6 +38,19 @@ paperless-ng 0.9.0
   multi user solution, however, it allows more than one user to access the website
   and set some basic permissions / renew passwords.
 
+* **Modified:** Changes to the consumer:
+
+  * Now uses the excellent watchdog library that should make sure files are
+    discovered no matter what the platform is.
+  * The consumer now uses a task scheduler to run consumption processes in parallel.
+    This means that consuming many documents should be much faster on systems with
+    many cores.
+  * Concurrency is controlled with the new settings ``PAPERLESS_TASK_WORKERS``
+    and ``PAPERLESS_THREADS_PER_WORKER``. See TODO for details on concurrency.
+  * The consumer no longer blocks the database for extended periods of time.
+  * An issue with tesseract running multiple threads per page and slowing down
+    the consumer was fixed.
+
 * **Modified [breaking]:** REST Api changes:
 
   * New filters added, other filters removed (case sensitive filters, slug filters)
@@ -64,8 +77,8 @@ paperless-ng 0.9.0
   * Rework of the code of the tesseract parser. This is now a lot cleaner.
   * Rework of the filename handling code. It was a mess.
   * Fixed some issues with the document exporter not exporting all documents when encountering duplicate filenames.
-  * Consumer rework: now uses the excellent watchdog library, lots of code removed.
-  * Added a task scheduler that takes care of checking mail, training the classifier and maintaining the document search index.
+  * Added a task scheduler that takes care of checking mail, training the classifier, maintaining the document search index
+    and consuming documents.
   * Updated dependencies. Now uses Pipenv all around.
   * Updated Dockerfile and docker-compose. Now uses ``supervisord`` to run everything paperless-related in a single container.
 
@@ -77,6 +90,8 @@ paperless-ng 0.9.0
   * ``PAPERLESS_DEBUG`` defaults to ``false``.
   * The presence of ``PAPERLESS_DBHOST`` now determines whether to use PostgreSQL or
     sqlite.
+  * ``PAPERLESS_OCR_THREADS`` is gone and replaced with ``PAPERLESS_TASK_WORKERS`` and
+    ``PAPERLESS_THREADS_PER_WORKER``. See TODO for details.
 
 * Many more small changes here and there. The usual stuff.
 

docs/troubleshooting.rst

@@ -2,9 +2,38 @@
 Troubleshooting
 ***************
 
-.. warning::
+No files are added by the consumer
+##################################
+
+Check for the following issues:
+
+*   Ensure that the directory you're putting your documents in is the folder
+    paperless is watching. With docker, this setting is performed in the
+    ``docker-compose.yml`` file. Without docker, look at the ``CONSUMPTION_DIR``
+    setting. Don't adjust this setting if you're using docker.
+*   Ensure that redis is up and running. Paperless does its task processing
+    asynchronously, and for documents to arrive at the task processor, it needs
+    redis to run.
+*   Ensure that the task processor is running. Docker does this automatically.
+    Manually invoke the task processor by executing
+
+    .. code:: shell-session
+
+        $ python3 manage.py qcluster
+
+*   Look at the output of paperless and inspect it for any errors.
+*   Go to the admin interface, and check if there are failed tasks. If so, the
+    tasks will contain an error message.
+
+
+Consumer fails to pickup any new files
+######################################
+
+If you notice, that the consumer will only pickup files in the consumption
+directory at startup, but won't find any other files added later, check out
+the configuration file and enable filesystem polling with the setting
+``PAPERLESS_CONSUMER_POLLING``.
 
-    This section is not updated to paperless-ng yet.
 
 Consumer warns ``OCR for XX failed``
 ####################################