Merge branch 'dev'

docs/advanced_usage.md

@@ -136,6 +136,11 @@ script can access the following relevant environment variables set:
     be triggered, leading to failures as two tasks work on the
     same document path
 
+!!! warning
+
+    If your script modifies `DOCUMENT_WORKING_PATH` in a non-deterministic
+    way, this may allow duplicate documents to be stored
+
 A simple but common example for this would be creating a simple script
 like this:
 

docs/api.md

@@ -8,7 +8,6 @@ most of the available filters and ordering fields.
 
 The API provides the following main endpoints:
 
-- `/api/consumption_templates/`: Full CRUD support.
 - `/api/correspondents/`: Full CRUD support.
 - `/api/custom_fields/`: Full CRUD support.
 - `/api/documents/`: Full CRUD support, except POSTing new documents.
@@ -24,6 +23,7 @@ The API provides the following main endpoints:
 - `/api/tags/`: Full CRUD support.
 - `/api/tasks/`: Read-only.
 - `/api/users/`: Full CRUD support.
+- `/api/workflows/`: Full CRUD support.
 
 All of these endpoints except for the logging endpoint allow you to
 fetch (and edit and delete where appropriate) individual objects by
@@ -274,6 +274,7 @@ The endpoint supports the following optional form fields:
 - `correspondent`: Specify the ID of a correspondent that the consumer
   should use for the document.
 - `document_type`: Similar to correspondent.
+- `storage_path`: Similar to correspondent.
 - `tags`: Similar to correspondent. Specify this multiple times to
   have multiple tags added to the document.
 - `archive_serial_number`: An optional archive serial number to set.

docs/configuration.md

@@ -3,6 +3,11 @@
 Paperless provides a wide range of customizations. Depending on how you
 run paperless, these settings have to be defined in different places.
 
+Certain configuration options may be set via the UI. This currently includes
+common [OCR](#ocr) related settings. If set, these will take preference over the
+settings via environment variables. If not set, the environment setting or applicable
+default will be utilized instead.
+
 - If you run paperless on docker, `paperless.conf` is not used.
   Rather, configure paperless by copying necessary options to
   `docker-compose.env`.

docs/index.md

@@ -18,6 +18,7 @@ physical documents into a searchable online archive so you can keep, well, _less
 ## Features
 
 -   **Organize and index** your scanned documents with tags, correspondents, types, and more.
+-   _Your_ data is stored locally on _your_ server and is never transmitted or shared in any way.
 -   Performs **OCR** on your documents, adding searchable and selectable text, even to documents scanned with only images.
 -   Utilizes the open-source Tesseract engine to recognize more than 100 languages.
 -   Documents are saved as PDF/A format which is designed for long term storage, alongside the unaltered originals.
@@ -41,7 +42,7 @@ physical documents into a searchable online archive so you can keep, well, _less
     -   Configure multiple accounts and rules for each account.
     -   After processing, paperless can perform actions on the messages such as marking as read, deleting and more.
 -   A built-in robust **multi-user permissions** system that supports 'global' permissions as well as per document or object.
--   A powerful templating system that gives you more control over the consumption pipeline.
+-   A powerful workflow system that gives you even more control.
 -   **Optimized** for multi core systems: Paperless-ngx consumes multiple documents in parallel.
 -   The integrated sanity checker makes sure that your document archive is in good health.
 
@@ -156,9 +157,9 @@ Tag, correspondent, document type and storage path editing.
 
 </div>
 <div class="grid-half-right" markdown>
-  Consumption templates provide finer control over the document pipeline.
+  Workflows provide finer control over the document pipeline and trigger actions.
 
-![image](assets/screenshots/consumption_template.png)
+![image](assets/screenshots/workflow.png)
 
 </div>
 <div class="clear"></div>

docs/setup.md

@@ -28,7 +28,8 @@ steps described in [Docker setup](#docker_hub) automatically.
 1.  Make sure that Docker and Docker Compose are installed.
 
     !!! tip
-    See the Docker installation instructions at https://docs.docker.com/engine/install/
+
+        See the Docker installation instructions at https://docs.docker.com/engine/install/
 
 2.  Download and run the installation script:
 
@@ -72,7 +73,7 @@ steps described in [Docker setup](#docker_hub) automatically.
 
         If you want to use the included `docker-compose.*.yml` file, you
         need to have at least Docker version **17.09.0** and Docker Compose
-        version **v2**. To check do: `docker compose -v` or `docker -v`
+        version **v2**. To check do: `docker compose version` or `docker -v`
 
         See the [Docker installation guide](https://docs.docker.com/engine/install/) on how to install the current
         version of Docker for your operating system or Linux distribution of

docs/usage.md

@@ -238,7 +238,7 @@ do not have an owner set.
 
 ### Default permissions
 
-Default permissions for documents can be set using consumption templates.
+Default permissions for documents can be set using workflows.
 
 For objects created via the web UI (tags, doc types, etc.) the default is to set the current user
 as owner and no extra permissions, but you explicitly set these under Settings > Permissions.
@@ -255,29 +255,80 @@ permissions can be granted to limit access to certain parts of the UI (and corre
 In order to enable the password reset feature you will need to setup an SMTP backend, see
 [`PAPERLESS_EMAIL_HOST`](configuration.md#PAPERLESS_EMAIL_HOST)
 
-## Consumption templates
+## Workflows
 
-Consumption templates were introduced in v2.0 and allow for finer control over what metadata (tags, doc
-types) and permissions (owner, privileges) are assigned to documents during consumption. In general,
-templates are applied sequentially (by sort order) but subsequent templates will never override an
-assignment from a preceding template. The same is true for mail rules, e.g. if you set the correspondent
-in a mail rule any subsequent consumption templates that are applied _will not_ overwrite this. The
-exception to this is assignments that can be multiple e.g. tags and permissions, which will be merged.
+!!! note
 
-Consumption templates allow you to filter by:
+    v2.3 added "Workflows" and existing "Consumption Templates" were converted automatically to the new more powerful format.
+
+Workflows allow hooking into the Paperless-ngx document pipeline, for example to alter what metadata (tags, doc types) and
+permissions (owner, privileges) are assigned to documents. Workflows can have multiple 'triggers' and 'actions'. Triggers
+are events (with optional filtering rules) that will cause the workflow to be run and actions are the set of sequential
+actions to apply.
+
+In general, workflows and any actions they contain are applied sequentially by sort order. For "assignment" actions, subsequent
+workflow actions will override previous assignments, except for assignments that accept multiple items e.g. tags, custom
+fields and permissions, which will be merged.
+
+### Workflow Triggers
+
+Currently, there are three events that correspond to workflow trigger 'types':
+
+1. **Consumption Started**: _before_ a document is consumed, so events can include filters by source (mail, consumption
+   folder or API), file path, file name, mail rule
+2. **Document Added**: _after_ a document is added. At this time, file path and source information is no longer available,
+   but the document content has been extracted and metadata such as document type, tags, etc. have been set, so these can now
+   be used for filtering.
+3. **Document Updated**: when a document is updated. Similar to 'added' events, triggers can include filtering by content matching,
+   tags, doc type, or correspondent.
+
+The following flow diagram illustrates the three trigger types:
+
+```mermaid
+flowchart TD
+    consumption{"Matching
+    'Consumption'
+    trigger(s)"}
+
+    added{"Matching
+    'Added'
+    trigger(s)"}
+
+    updated{"Matching
+    'Updated'
+    trigger(s)"}
+
+    A[New Document] --> consumption
+    consumption --> |Yes| C[Workflow Actions Run]
+    consumption --> |No| D
+    C --> D[Document Added]
+    D -- Paperless-ngx 'matching' of tags, etc. --> added
+    added --> |Yes| F[Workflow Actions Run]
+    added --> |No| G
+    F --> G[Document Finalized]
+    H[Existing Document Changed] --> updated
+    updated --> |Yes| J[Workflow Actions Run]
+    updated --> |No| K
+    J --> K[Document Saved]
+```
+
+#### Filters {#workflow-trigger-filters}
+
+Workflows allow you to filter by:
 
 - Source, e.g. documents uploaded via consume folder, API (& the web UI) and mail fetch
 - File name, including wildcards e.g. \*.pdf will apply to all pdfs
 - File path, including wildcards. Note that enabling `PAPERLESS_CONSUMER_RECURSIVE` would allow, for
   example, automatically assigning documents to different owners based on the upload directory.
-- Mail rule. Choosing this option will force 'mail fetch' to be the template source.
+- Mail rule. Choosing this option will force 'mail fetch' to be the workflow source.
+- Content matching (`Added` and `Updated` triggers only). Filter document content using the matching settings.
+- Tags (`Added` and `Updated` triggers only). Filter for documents with any of the specified tags
+- Document type (`Added` and `Updated` triggers only). Filter documents with this doc type
+- Correspondent (`Added` and `Updated` triggers only). Filter documents with this correspondent
 
-!!! note
+### Workflow Actions
 
-    You must include a file name filter, a path filter or a mail rule filter. Use * for either to apply
-    to all files.
-
-Consumption templates can assign:
+There is currently one type of workflow action, "Assignment", which can assign:
 
 - Title, see [title placeholders](usage.md#title-placeholders) below
 - Tags, correspondent, document types
@@ -285,21 +336,11 @@ Consumption templates can assign:
 - View and / or edit permissions to users or groups
 - Custom fields. Note that no value for the field will be set
 
-### Consumption template permissions
+#### Title placeholders
 
-All users who have application permissions for editing consumption templates can see the same set
-of templates. In other words, templates themselves intentionally do not have an owner or permissions.
-
-Given their potentially far-reaching capabilities, you may want to restrict access to templates.
-
-Upon migration, existing installs will grant access to consumption templates to users who can add
-documents (and superusers who can always access all parts of the app).
-
-### Title placeholders
-
-Consumption template titles can include placeholders, _only for items that are assigned within the template_.
-This is because at the time of consumption (when the title is to be set), no automatic tags etc. have been
-applied. You can use the following placeholders:
+Workflow titles can include placeholders but the available options differ depending on the type of
+workflow trigger. This is because at the time of consumption (when the title is to be set), no automatic tags etc. have been
+applied. You can use the following placeholders with any trigger type:
 
 - `{correspondent}`: assigned correspondent name
 - `{document_type}`: assigned document type name
@@ -314,6 +355,27 @@ applied. You can use the following placeholders:
 - `{added_time}`: added time in HH:MM format
 - `{original_filename}`: original file name without extension
 
+The following placeholders are only available for "added" or "updated" triggers
+
+- `{created}`: created datetime
+- `{created_year}`: created year
+- `{created_year_short}`: created year
+- `{created_month}`: created month
+- `{created_month_name}`: created month name
+- `{created_month_name_short}`: created month short name
+- `{created_day}`: created day
+- `{created_time}`: created time in HH:MM format
+
+### Workflow permissions
+
+All users who have application permissions for editing workflows can see the same set
+of workflows. In other words, workflows themselves intentionally do not have an owner or permissions.
+
+Given their potentially far-reaching capabilities, you may want to restrict access to workflows.
+
+Upon migration, existing installs will grant access to workflows to users who can add
+documents (and superusers who can always access all parts of the app).
+
 ## Custom Fields {#custom-fields}
 
 Paperless-ngx supports the use of custom fields for documents as of v2.0, allowing a user