paradizelost/paperless-ngx
1
0
Fork 0
mirror of https://github.com/paperless-ngx/paperless-ngx.git synced 2025-10-30 03:56:23 -05:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'dev' into travis-multiarch-builds

Browse Source
This commit is contained in:
jonaswinkler
2020-12-21 21:29:45 +01:00
parent dd6aa2f775 ae3e2a7063
commit b3a13cec17
13 changed files with 95 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files

79
docs/advanced_usage.rst
View File

@@ -5,85 +5,6 @@ Advanced topics
Paperless offers a couple features that automate certain tasks and make your life
easier.
Guesswork
#########
Any document you put into the consumption directory will be consumed, but if
you name the file right, it'll automatically set some values in the database
for you. This is is the logic the consumer follows:
1. Try to find the correspondent, title, and tags in the file name following
the pattern: ``Date - Correspondent - Title - tag,tag,tag.pdf``. Note that
the format of the date is **rigidly defined** as ``YYYYMMDDHHMMSSZ`` or
``YYYYMMDDZ``. The ``Z`` refers "Zulu time" AKA "UTC".
The tags are optional, so the format ``Date - Correspondent - Title.pdf``
works as well.
2. If that doesn't work, we skip the date and try this pattern:
``Correspondent - Title - tag,tag,tag.pdf``.
3. If that doesn't work, we try to find the correspondent and title in the file
name following the pattern: ``Correspondent - Title.pdf``.
4. If that doesn't work, just assume that the name of the file is the title.
So given the above, the following examples would work as you'd expect:
* ``20150314000700Z - Some Company Name - Invoice 2016-01-01 - money,invoices.pdf``
* ``20150314Z - Some Company Name - Invoice 2016-01-01 - money,invoices.pdf``
* ``Some Company Name - Invoice 2016-01-01 - money,invoices.pdf``
* ``Another Company - Letter of Reference.jpg``
* ``Dad's Recipe for Pancakes.png``
These however wouldn't work:
* ``2015-03-14 00:07:00 UTC - Some Company Name, Invoice 2016-01-01, money, invoices.pdf``
* ``2015-03-14 - Some Company Name, Invoice 2016-01-01, money, invoices.pdf``
* ``Some Company Name, Invoice 2016-01-01, money, invoices.pdf``
* ``Another Company- Letter of Reference.jpg``
Do I have to be so strict about naming?
=======================================
Rather than using the strict document naming rules, one can also set the option
``PAPERLESS_FILENAME_DATE_ORDER`` in ``paperless.conf`` to any date order
that is accepted by dateparser_. Doing so will cause ``paperless`` to default
to any date format that is found in the title, instead of a date pulled from
the document's text, without requiring the strict formatting of the document
filename as described above.
.. _dateparser: https://github.com/scrapinghub/dateparser/blob/v0.7.0/docs/usage.rst#settings
.. _advanced-transforming_filenames:
Transforming filenames for parsing
==================================
Some devices can't produce filenames that can be parsed by the default
parser. By configuring the option ``PAPERLESS_FILENAME_PARSE_TRANSFORMS`` in
``paperless.conf`` one can add transformations that are applied to the filename
before it's parsed.
The option contains a list of dictionaries of regular expressions (key:
``pattern``) and replacements (key: ``repl``) in JSON format, which are
applied in order by passing them to ``re.subn``. Transformation stops
after the first match, so at most one transformation is applied. The general
syntax is
.. code:: python
[{"pattern":"pattern1", "repl":"repl1"}, {"pattern":"pattern2", "repl":"repl2"}, ..., {"pattern":"patternN", "repl":"replN"}]
The example below is for a Brother ADS-2400N, a scanner that allows
different names to different hardware buttons (useful for handling
multiple entities in one instance), but insists on adding ``_<count>``
to the filename.
.. code:: python
# Brother profile configuration, support "Name_Date_Count" (the default
# setting) and "Name_Count" (use "Name" as tag and "Count" as title).
PAPERLESS_FILENAME_PARSE_TRANSFORMS=[{"pattern":"^([a-z]+)_(\\d{8})_(\\d{6})_([0-9]+)\\.", "repl":"\\2\\3Z - \\4 - \\1."}, {"pattern":"^([a-z]+)_([0-9]+)\\.", "repl":" - \\2 - \\1."}]
.. _advanced-matching:
Matching tags, correspondents and document types

5
docs/configuration.rst
View File

@@ -400,11 +400,6 @@ PAPERLESS_FILENAME_DATE_ORDER=<format>
Defaults to none, which disables this feature.
PAPERLESS_FILENAME_PARSE_TRANSFORMS
Transforms filenames before they are processed by paperless. See
:ref:`advanced-transforming_filenames` for details.
Defaults to none, which disables this feature.
Binaries
########

2
src-ui/src/app/components/common/input/select/select.component.html
View File

@@ -1,7 +1,7 @@
<div class="form-group paperless-input-select">
<label [for]="inputId">{{title}}</label>
<div [class.input-group]="showPlusButton()">
<ng-select name="correspondent" [(ngModel)]="value"
<ng-select name="inputId" [(ngModel)]="value"
[disabled]="disabled"
[style.color]="textColor"
[style.background]="backgroundColor"

2
src-ui/src/app/components/dashboard/dashboard.component.html
View File

@@ -1,4 +1,4 @@
<app-page-header title="Dashboard" subTitle="Welcome to paperless-ng!">
<app-page-header title="Dashboard" [subTitle]="subtitle">
<img src="assets/logo.svg" height="80" class="m-2 d-none d-md-block">
</app-page-header>

24
src-ui/src/app/components/dashboard/dashboard.component.ts
View File

@@ -1,4 +1,5 @@
import { Component, OnInit } from '@angular/core';
import { Meta } from '@angular/platform-browser';
import { PaperlessSavedView } from 'src/app/data/paperless-saved-view';
import { SavedViewService } from 'src/app/services/rest/saved-view.service';
@@ -11,8 +12,29 @@ import { SavedViewService } from 'src/app/services/rest/saved-view.service';
export class DashboardComponent implements OnInit {
constructor(
private savedViewService: SavedViewService) { }
private savedViewService: SavedViewService,
private meta: Meta
) { }
get displayName() {
let tagFullName = this.meta.getTag('name=full_name')
let tagUsername = this.meta.getTag('name=username')
if (tagFullName && tagFullName.content) {
return tagFullName.content
} else if (tagUsername && tagUsername.content) {
return tagUsername.content
} else {
return null
}
}
get subtitle() {
if (this.displayName) {
return `Hello ${this.displayName}, welcome to Paperless-ng!`
} else {
return `Welcome to Paperless-ng!`
}
}
savedViews: PaperlessSavedView[] = []

5
src-ui/src/app/components/manage/correspondent-list/correspondent-edit-dialog/correspondent-edit-dialog.component.html
View File

@@ -8,10 +8,9 @@
<div class="modal-body">
<app-input-text title="Name" formControlName="name"></app-input-text>
<app-input-text title="Match" formControlName="match"></app-input-text>
<app-input-select title="Matching algorithm" [items]="getMatchingAlgorithms()" formControlName="matching_algorithm"></app-input-select>
<app-input-check title="Case insensitive" formControlName="is_insensitive"></app-input-check>
<app-input-text title="Match" formControlName="match" hint="Auto matching does not require you to fill in this field."></app-input-text>
<app-input-check title="Case insensitive" formControlName="is_insensitive" hint="Auto matching ignores this option."></app-input-check>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-outline-dark" (click)="cancel()">Cancel</button>

4
src-ui/src/app/components/manage/document-type-list/document-type-edit-dialog/document-type-edit-dialog.component.html
View File

@@ -8,9 +8,9 @@
<div class="modal-body">
<app-input-text title="Name" formControlName="name"></app-input-text>
<app-input-text title="Match" formControlName="match"></app-input-text>
<app-input-select title="Matching algorithm" [items]="getMatchingAlgorithms()" formControlName="matching_algorithm"></app-input-select>
<app-input-check title="Case insensitive" formControlName="is_insensitive"></app-input-check>
<app-input-text title="Match" formControlName="match" hint="Auto matching does not require you to fill in this field."></app-input-text>
<app-input-check title="Case insensitive" formControlName="is_insensitive" hint="Auto matching ignores this option."></app-input-check>
</div>
<div class="modal-footer">

16
src-ui/src/app/components/manage/tag-list/tag-edit-dialog/tag-edit-dialog.component.html
View File

@@ -7,11 +7,21 @@
</div>
<div class="modal-body">
<app-input-text title="Name" formControlName="name"></app-input-text>
<app-input-select title="Colour" [items]="getColours()" formControlName="colour" [textColor]="getColor(objectForm.value.colour).textColor" [backgroundColor]="getColor(objectForm.value.colour).value"></app-input-select>
<div class="form-group paperless-input-select">
<label for="colour">Colour</label>
<ng-select name="colour" formControlName="colour" [items]="getColours()" bindValue="id" bindLabel="name" [clearable]="false">
<ng-template ng-option-tmp ng-label-tmp let-item="item">
<span class="badge" [style.background]="item.value" [style.color]="item.textColor">{{item.name}}</span>
</ng-template>
</ng-select>
</div>
<app-input-check title="Inbox tag" formControlName="is_inbox_tag" hint="Inbox tags are automatically assigned to all consumed documents."></app-input-check>
<app-input-text title="Match" formControlName="match"></app-input-text>
<app-input-select title="Matching algorithm" [items]="getMatchingAlgorithms()" formControlName="matching_algorithm"></app-input-select>
<app-input-check title="Case insensitive" formControlName="is_insensitive"></app-input-check>
<app-input-text title="Match" formControlName="match" hint="Auto matching does not require you to fill in this field."></app-input-text>
<app-input-check title="Case insensitive" formControlName="is_insensitive" hint="Auto matching ignores this option."></app-input-check>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-outline-dark" (click)="cancel()">Cancel</button>

4
src-ui/src/app/data/paperless-tag.ts
View File

@@ -6,14 +6,14 @@ export const TAG_COLOURS = [
{id: 1, value: "#a6cee3", name: "Light Blue", textColor: "#000000"},
{id: 2, value: "#1f78b4", name: "Blue", textColor: "#ffffff"},
{id: 3, value: "#b2df8a", name: "Light Green", textColor: "#000000"},
{id: 4, value: "#33a02c", name: "Green", textColor: "#000000"},
{id: 4, value: "#33a02c", name: "Green", textColor: "#ffffff"},
{id: 5, value: "#fb9a99", name: "Light Red", textColor: "#000000"},
{id: 6, value: "#e31a1c", name: "Red ", textColor: "#ffffff"},
{id: 7, value: "#fdbf6f", name: "Light Orange", textColor: "#000000"},
{id: 8, value: "#ff7f00", name: "Orange", textColor: "#000000"},
{id: 9, value: "#cab2d6", name: "Light Violet", textColor: "#000000"},
{id: 10, value: "#6a3d9a", name: "Violet", textColor: "#ffffff"},
{id: 11, value: "#b15928", name: "Brown", textColor: "#000000"},
{id: 11, value: "#b15928", name: "Brown", textColor: "#ffffff"},
{id: 12, value: "#000000", name: "Black", textColor: "#ffffff"},
{id: 13, value: "#cccccc", name: "Light Grey", textColor: "#000000"}
]

25
src/documents/management/commands/document_importer.py
View File

@@ -1,18 +1,29 @@
import json
import os
import shutil
from contextlib import contextmanager
from django.conf import settings
from django.core.management import call_command
from django.core.management.base import BaseCommand, CommandError
from django.db.models.signals import post_save, m2m_changed
from filelock import FileLock
from documents.models import Document
from documents.settings import EXPORTER_FILE_NAME, EXPORTER_THUMBNAIL_NAME, \
EXPORTER_ARCHIVE_NAME
from ...file_handling import create_source_path_directory, \
generate_unique_filename
from ...file_handling import create_source_path_directory
from ...mixins import Renderable
from ...signals.handlers import update_filename_and_move_files
@contextmanager
def disable_signal(sig, receiver, sender):
try:
sig.disconnect(receiver=receiver, sender=sender)
yield
finally:
sig.connect(receiver=receiver, sender=sender)
class Command(Renderable, BaseCommand):
@@ -47,7 +58,12 @@ class Command(Renderable, BaseCommand):
self.manifest = json.load(f)
self._check_manifest()
with disable_signal(post_save,
receiver=update_filename_and_move_files,
sender=Document):
with disable_signal(m2m_changed,
receiver=update_filename_and_move_files,
sender=Document.tags.through):
# Fill up the database with whatever is in the manifest
call_command("loaddata", manifest_path)
@@ -117,9 +133,6 @@ class Command(Renderable, BaseCommand):
document.storage_type = Document.STORAGE_TYPE_UNENCRYPTED
with FileLock(settings.MEDIA_LOCK):
document.filename = generate_unique_filename(
document, settings.ORIGINALS_DIR)
if os.path.isfile(document.source_path):
raise FileExistsError(document.source_path)

2
src/documents/templates/index.html
View File

@@ -8,6 +8,8 @@
<title>Paperless-ng</title>
<base href="/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="username" content="{{username}}">
<meta name="full_name" content="{{full_name}}">
<meta name="cookie_prefix" content="{{cookie_prefix}}">
<link rel="icon" type="image/x-icon" href="favicon.ico">
<link rel="stylesheet" href="{% static 'frontend/styles.css' %}"></head>

30
src/documents/tests/test_management_exporter.py
View File

@@ -24,11 +24,17 @@ class TestExportImport(DirectoriesMixin, TestCase):
file = os.path.join(self.dirs.originals_dir, "0000001.pdf")
Document.objects.create(content="Content", checksum="42995833e01aea9b3edee44bbfdd7ce1", archive_checksum="62acb0bcbfbcaa62ca6ad3668e4e404b", title="wow", filename="0000001.pdf", mime_type="application/pdf")
Document.objects.create(content="Content", checksum="9c9691e51741c1f4f41a20896af31770", title="wow", filename="0000002.pdf.gpg", mime_type="application/pdf", storage_type=Document.STORAGE_TYPE_GPG)
Tag.objects.create(name="t")
DocumentType.objects.create(name="dt")
Correspondent.objects.create(name="c")
d1 = Document.objects.create(content="Content", checksum="42995833e01aea9b3edee44bbfdd7ce1", archive_checksum="62acb0bcbfbcaa62ca6ad3668e4e404b", title="wow", filename="0000001.pdf", mime_type="application/pdf")
d2 = Document.objects.create(content="Content", checksum="9c9691e51741c1f4f41a20896af31770", title="wow", filename="0000002.pdf.gpg", mime_type="application/pdf", storage_type=Document.STORAGE_TYPE_GPG)
t1 = Tag.objects.create(name="t")
dt1 = DocumentType.objects.create(name="dt")
c1 = Correspondent.objects.create(name="c")
d1.tags.add(t1)
d1.correspondents = c1
d1.document_type = dt1
d1.save()
d2.save()
target = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, target)
@@ -59,11 +65,25 @@ class TestExportImport(DirectoriesMixin, TestCase):
self.assertEqual(checksum, element['fields']['archive_checksum'])
with paperless_environment() as dirs:
self.assertEqual(Document.objects.count(), 2)
Document.objects.all().delete()
Correspondent.objects.all().delete()
DocumentType.objects.all().delete()
Tag.objects.all().delete()
self.assertEqual(Document.objects.count(), 0)
call_command('document_importer', target)
self.assertEqual(Document.objects.count(), 2)
messages = check_sanity()
# everything is alright after the test
self.assertEqual(len(messages), 0, str([str(m) for m in messages]))
@override_settings(
PAPERLESS_FILENAME_FORMAT="{title}"
)
def test_exporter_with_filename_format(self):
self.test_exporter()
def test_export_missing_files(self):
target = tempfile.mkdtemp()

2
src/documents/views.py
View File

@@ -57,6 +57,8 @@ class IndexView(TemplateView):
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
context['cookie_prefix'] = settings.COOKIE_PREFIX
context['username'] = self.request.user.username
context['full_name'] = self.request.user.get_full_name()
return context