mirror of
				https://github.com/paperless-ngx/paperless-ngx.git
				synced 2025-11-03 03:16:10 -06:00 
			
		
		
		
	s/Sender/Correspondent & reworked the (im|ex)porter
This commit is contained in:
		@@ -44,10 +44,10 @@ Any document you put into the consumption directory will be consumed, but if you
 | 
			
		||||
name the file right, it'll automatically set some values in the database for
 | 
			
		||||
you.  This is is the logic the consumer follows:
 | 
			
		||||
 | 
			
		||||
1. Try to find the sender, title, and tags in the file name following the
 | 
			
		||||
   pattern: ``Sender - Title - tag,tag,tag.pdf``.
 | 
			
		||||
2. If that doesn't work, try to find the sender and title in the file name
 | 
			
		||||
   following the pattern:  ``Sender - Title.pdf``.
 | 
			
		||||
1. Try to find the correspondent, title, and tags in the file name following
 | 
			
		||||
   the pattern: ``Correspondent - Title - tag,tag,tag.pdf``.
 | 
			
		||||
2. If that doesn't work, try to find the correspondent and title in the file
 | 
			
		||||
   name following the pattern:  ``Correspondent - Title.pdf``.
 | 
			
		||||
3. If that doesn't work, just assume that the name of the file is the title.
 | 
			
		||||
 | 
			
		||||
So given the above, the following examples would work as you'd expect:
 | 
			
		||||
@@ -97,9 +97,9 @@ So, with all that in mind, here's what you do to get it running:
 | 
			
		||||
   the configured email account every 10 minutes for something new and pull down
 | 
			
		||||
   whatever it finds.
 | 
			
		||||
4. Send yourself an email!  Note that the subject is treated as the file name,
 | 
			
		||||
   so if you set the subject to ``Sender - Title - tag,tag,tag``, you'll get
 | 
			
		||||
   what you expect.  Also, you must include the aforementioned secret string in
 | 
			
		||||
   every email so the fetcher knows that it's safe to import.
 | 
			
		||||
   so if you set the subject to ``Correspondent - Title - tag,tag,tag``, you'll
 | 
			
		||||
   get what you expect.  Also, you must include the aforementioned secret
 | 
			
		||||
   string in every email so the fetcher knows that it's safe to import.
 | 
			
		||||
5. After a few minutes, the consumer will poll your mailbox, pull down the
 | 
			
		||||
   message, and place the attachment in the consumption directory with the
 | 
			
		||||
   appropriate name.  A few minutes later, the consumer will import it like any
 | 
			
		||||
@@ -118,16 +118,16 @@ a real API, it's just a URL that accepts an HTTP POST.
 | 
			
		||||
To push your document to *Paperless*, send an HTTP POST to the server with the
 | 
			
		||||
following name/value pairs:
 | 
			
		||||
 | 
			
		||||
* ``sender``: The name of the document's sender.  Note that there are
 | 
			
		||||
  restrictions on what characters you can use here.  Specifically, alphanumeric
 | 
			
		||||
  characters, `-`, `,`, `.`, and `'` are ok, everything else it out.  You also
 | 
			
		||||
  can't use the sequence ` - ` (space, dash, space).
 | 
			
		||||
* ``correspondent``: The name of the document's correspondent.  Note that there
 | 
			
		||||
  are restrictions on what characters you can use here.  Specifically,
 | 
			
		||||
  alphanumeric characters, `-`, `,`, `.`, and `'` are ok, everything else it
 | 
			
		||||
  out.  You also can't use the sequence ` - ` (space, dash, space).
 | 
			
		||||
* ``title``: The title of the document.  The rules for characters is the same
 | 
			
		||||
  here as the sender.
 | 
			
		||||
* ``signature``: For security reasons, we have the sender send a signature using
 | 
			
		||||
  a "shared secret" method to make sure that random strangers don't start
 | 
			
		||||
  uploading stuff to your server.  The means of generating this signature is
 | 
			
		||||
  defined below.
 | 
			
		||||
  here as the correspondent.
 | 
			
		||||
* ``signature``: For security reasons, we have the correspondent send a
 | 
			
		||||
  signature using a "shared secret" method to make sure that random strangers
 | 
			
		||||
  don't start uploading stuff to your server.  The means of generating this
 | 
			
		||||
  signature is defined below.
 | 
			
		||||
 | 
			
		||||
Specify ``enctype="multipart/form-data"``, and then POST your file with:::
 | 
			
		||||
 | 
			
		||||
@@ -146,12 +146,12 @@ verification.
 | 
			
		||||
 | 
			
		||||
In the case of *Paperless*, you configure the server with the secret by setting
 | 
			
		||||
``UPLOAD_SHARED_SECRET``.  Then on your client, you generate your signature by
 | 
			
		||||
concatenating the sender, title, and the secret, and then using sha256 to
 | 
			
		||||
generate a hexdigest.
 | 
			
		||||
concatenating the correspondent, title, and the secret, and then using sha256
 | 
			
		||||
to generate a hexdigest.
 | 
			
		||||
 | 
			
		||||
If you're using Python, this is what that looks like:
 | 
			
		||||
 | 
			
		||||
.. code:: python
 | 
			
		||||
 | 
			
		||||
    from hashlib import sha256
 | 
			
		||||
    signature = sha256(sender + title + secret).hexdigest()
 | 
			
		||||
    signature = sha256(correspondent + title + secret).hexdigest()
 | 
			
		||||
 
 | 
			
		||||
@@ -4,10 +4,68 @@ Migrating, Updates, and Backups
 | 
			
		||||
===============================
 | 
			
		||||
 | 
			
		||||
As *Paperless* is still under active development, there's a lot that can change
 | 
			
		||||
as software updates roll out.  The thing you just need to remember for all of
 | 
			
		||||
this is that for the most part, **the database is expendable** so long as you
 | 
			
		||||
have your files.  This is because the file name of the exported files includes
 | 
			
		||||
the name of the sender, the title, and the tags (if any) on each file.
 | 
			
		||||
as software updates roll out.  You should backup often, so if anything goes
 | 
			
		||||
wrong during an update, you at least have a means of restoring to something
 | 
			
		||||
usable.  Thankfully, there are automated ways of backing up, restoring, and
 | 
			
		||||
updating the software.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _migrating-backup:
 | 
			
		||||
 | 
			
		||||
Backing Up
 | 
			
		||||
----------
 | 
			
		||||
 | 
			
		||||
So you're bored of this whole project, or you want to make a remote backup of
 | 
			
		||||
the unencrypted files for whatever reason.  This is easy to do, simply use the
 | 
			
		||||
:ref:`exporter <utilities-exporter>` to dump your documents and database out
 | 
			
		||||
into an arbitrary directory.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _migrating-restoring:
 | 
			
		||||
 | 
			
		||||
Restoring
 | 
			
		||||
---------
 | 
			
		||||
 | 
			
		||||
Restoring your data is just as easy, since nearly all of your data exists either
 | 
			
		||||
in the file names, or in the contents of the files themselves.  You just need to
 | 
			
		||||
create an empty database (just follow the
 | 
			
		||||
:ref:`installation instructions <setup-installation>` again) and then import the
 | 
			
		||||
``tags.json`` file you created as part of your backup.  Lastly, copy your
 | 
			
		||||
exported documents into the consumption directory and start up the consumer.
 | 
			
		||||
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
    $ cd /path/to/project
 | 
			
		||||
    $ rm data/db.sqlite3  # Delete the database
 | 
			
		||||
    $ cd src
 | 
			
		||||
    $ ./manage.py migrate  # Create the database
 | 
			
		||||
    $ ./manage.py createsuperuser
 | 
			
		||||
    $ ./manage.py loaddata /path/to/arbitrary/place/tags.json
 | 
			
		||||
    $ cp /path/to/exported/docs/* /path/to/consumption/dir/
 | 
			
		||||
    $ ./manage.py document_consumer
 | 
			
		||||
 | 
			
		||||
Importing your data if you are :ref:`using Docker <setup-installation-docker>`
 | 
			
		||||
is almost as simple:
 | 
			
		||||
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
    # Stop and remove your current containers
 | 
			
		||||
    $ docker-compose stop
 | 
			
		||||
    $ docker-compose rm -f
 | 
			
		||||
 | 
			
		||||
    # Recreate them, add the superuser
 | 
			
		||||
    $ docker-compose up -d
 | 
			
		||||
    $ docker-compose run --rm webserver createsuperuser
 | 
			
		||||
 | 
			
		||||
    # Load the tags
 | 
			
		||||
    $ cat /path/to/arbitrary/place/tags.json | docker-compose run --rm webserver loaddata_stdin -
 | 
			
		||||
 | 
			
		||||
    # Load your exported documents into the consumption directory
 | 
			
		||||
    # (How you do this highly depends on how you have set this up)
 | 
			
		||||
    $ cp /path/to/exported/docs/* /path/to/mounted/consumption/dir/
 | 
			
		||||
 | 
			
		||||
After loading the documents into the consumption directory the consumer will
 | 
			
		||||
immediately start consuming the documents.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _migrating-updates:
 | 
			
		||||
@@ -20,7 +78,7 @@ on the directory containing the project files, and then use Django's ``migrate``
 | 
			
		||||
command to execute any database schema updates that might have been rolled in
 | 
			
		||||
as part of the update:
 | 
			
		||||
 | 
			
		||||
.. code:: bash
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
    $ cd /path/to/project
 | 
			
		||||
    $ git pull
 | 
			
		||||
@@ -43,112 +101,3 @@ requires only one additional step:
 | 
			
		||||
 | 
			
		||||
If ``git pull`` doesn't report any changes, there is no need to continue with
 | 
			
		||||
the remaining steps.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _migrating-backup:
 | 
			
		||||
 | 
			
		||||
Backing Up
 | 
			
		||||
----------
 | 
			
		||||
 | 
			
		||||
So you're bored of this whole project, or you want to make a remote backup of
 | 
			
		||||
the unencrypted files for whatever reason.  This is easy to do, simply use the
 | 
			
		||||
:ref:`exporter <utilities-exporter>` to dump your documents out into an
 | 
			
		||||
arbitrary directory.
 | 
			
		||||
 | 
			
		||||
Additionally however, you'll need to back up the tags themselves.  The file
 | 
			
		||||
names contain the tag names, but you still need to define the tags and their
 | 
			
		||||
matching algorithms in the database for things to work properly.  We do this
 | 
			
		||||
with Django's ``dumpdata`` command, which produces JSON output.
 | 
			
		||||
 | 
			
		||||
.. code:: bash
 | 
			
		||||
 | 
			
		||||
    $ cd /path/to/project
 | 
			
		||||
    $ cd src
 | 
			
		||||
    $ ./manage.py document_export /path/to/arbitrary/place/
 | 
			
		||||
    $ ./manage.py dumpdata documents.Tag > /path/to/arbitrary/place/tags.json
 | 
			
		||||
 | 
			
		||||
If you are :ref:`using Docker <setup-installation-docker>`, exporting your tags
 | 
			
		||||
as JSON is almost as easy:
 | 
			
		||||
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
    $ docker-compose run --rm webserver dumpdata documents.Tag > /path/to/arbitrary/place/tags.json
 | 
			
		||||
 | 
			
		||||
To export the documents you can either use ``docker run`` directly, specifying all
 | 
			
		||||
the commandline options by hand, or (more simply) mount a second volume for export.
 | 
			
		||||
 | 
			
		||||
To mount a volume for exports, follow the instructions in the
 | 
			
		||||
``docker-compose.yml.example`` file for the ``/export`` volume (making the changes
 | 
			
		||||
in your own ``docker-compose.yml`` file, of course). Once you have the
 | 
			
		||||
volume mounted, the command to run an export is:
 | 
			
		||||
 | 
			
		||||
.. code-block:: console
 | 
			
		||||
 | 
			
		||||
   $ docker-compose run --rm consumer document_exporter /export
 | 
			
		||||
 | 
			
		||||
If you prefer to use ``docker run`` directly, supplying the necessary commandline
 | 
			
		||||
options:
 | 
			
		||||
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
   $ # Identify your containers
 | 
			
		||||
   $ docker-compose ps
 | 
			
		||||
           Name                       Command                State     Ports
 | 
			
		||||
   -------------------------------------------------------------------------
 | 
			
		||||
   paperless_consumer_1    /sbin/docker-entrypoint.sh ...   Exit 0
 | 
			
		||||
   paperless_webserver_1   /sbin/docker-entrypoint.sh ...   Exit 0
 | 
			
		||||
 | 
			
		||||
   $ # Make sure to replace your passphrase and remove or adapt the id mapping
 | 
			
		||||
   $ docker run --rm \
 | 
			
		||||
       --volumes-from paperless_data_1 \
 | 
			
		||||
       --volume /path/to/arbitrary/place:/export \
 | 
			
		||||
       -e PAPERLESS_PASSPHRASE=YOUR_PASSPHRASE \
 | 
			
		||||
       -e USERMAP_UID=1000 -e USERMAP_GID=1000 \
 | 
			
		||||
       paperless document_exporter /export
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _migrating-restoring:
 | 
			
		||||
 | 
			
		||||
Restoring
 | 
			
		||||
---------
 | 
			
		||||
 | 
			
		||||
Restoring your data is just as easy, since nearly all of your data exists either
 | 
			
		||||
in the file names, or in the contents of the files themselves.  You just need to
 | 
			
		||||
create an empty database (just follow the
 | 
			
		||||
:ref:`installation instructions <setup-installation>` again) and then import the
 | 
			
		||||
``tags.json`` file you created as part of your backup.  Lastly, copy your
 | 
			
		||||
exported documents into the consumption directory and start up the consumer.
 | 
			
		||||
 | 
			
		||||
.. code:: bash
 | 
			
		||||
 | 
			
		||||
    $ cd /path/to/project
 | 
			
		||||
    $ rm data/db.sqlite3  # Delete the database
 | 
			
		||||
    $ cd src
 | 
			
		||||
    $ ./manage.py migrate  # Create the database
 | 
			
		||||
    $ ./manage.py createsuperuser
 | 
			
		||||
    $ ./manage.py loaddata /path/to/arbitrary/place/tags.json
 | 
			
		||||
    $ cp /path/to/exported/docs/* /path/to/consumption/dir/
 | 
			
		||||
    $ ./manage.py document_consumer
 | 
			
		||||
 | 
			
		||||
Importing your data if you are :ref:`using Docker <setup-installation-docker>`
 | 
			
		||||
is almost as simple:
 | 
			
		||||
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
    $ # Stop and remove your current containers
 | 
			
		||||
    $ docker-compose stop
 | 
			
		||||
    $ docker-compose rm -f
 | 
			
		||||
 | 
			
		||||
    $ # Recreate them, add the superuser
 | 
			
		||||
    $ docker-compose up -d
 | 
			
		||||
    $ docker-compose run --rm webserver createsuperuser
 | 
			
		||||
 | 
			
		||||
    $ # Load the tags
 | 
			
		||||
    $ cat /path/to/arbitrary/place/tags.json | docker-compose run --rm webserver loaddata_stdin -
 | 
			
		||||
 | 
			
		||||
    $ # Load your exported documents into the consumption directory
 | 
			
		||||
    $ # (How you do this highly depends on how you have set this up)
 | 
			
		||||
    $ cp /path/to/exported/docs/* /path/to/mounted/consumption/dir/
 | 
			
		||||
 | 
			
		||||
After loading the documents into the consumption directory the consumer will
 | 
			
		||||
immediately start consuming the documents.
 | 
			
		||||
 
 | 
			
		||||
@@ -26,7 +26,7 @@ How to Use It
 | 
			
		||||
 | 
			
		||||
The webserver is started via the ``manage.py`` script:
 | 
			
		||||
 | 
			
		||||
.. code:: bash
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
    $ /path/to/paperless/src/manage.py runserver
 | 
			
		||||
 | 
			
		||||
@@ -64,7 +64,7 @@ How to Use It
 | 
			
		||||
 | 
			
		||||
The consumer is started via the ``manage.py`` script:
 | 
			
		||||
 | 
			
		||||
.. code:: bash
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
    $ /path/to/paperless/src/manage.py document_consumer
 | 
			
		||||
 | 
			
		||||
@@ -95,16 +95,86 @@ How to Use It
 | 
			
		||||
 | 
			
		||||
This too is done via the ``manage.py`` script:
 | 
			
		||||
 | 
			
		||||
.. code:: bash
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
    $ /path/to/paperless/src/manage.py document_exporter /path/to/somewhere
 | 
			
		||||
    $ /path/to/paperless/src/manage.py document_exporter /path/to/somewhere/
 | 
			
		||||
 | 
			
		||||
This will dump all of your PDFs into ``/path/to/somewhere`` for you to do with
 | 
			
		||||
as you please.  The naming scheme on export is identical to that used for
 | 
			
		||||
import, so should you can now safely delete the entire project directly,
 | 
			
		||||
database, encrypted PDFs and all, and later create it all again simply by
 | 
			
		||||
running the consumer again and dumping all of these files into
 | 
			
		||||
``CONSUMPTION_DIR``.
 | 
			
		||||
This will dump all of your unencrypted PDFs into ``/path/to/somewhere`` for you
 | 
			
		||||
to do with as you please.  The files are accompanied with a special file,
 | 
			
		||||
``manifest.json`` which can be used to
 | 
			
		||||
:ref:`import the files <utilities-importer>` at a later date if you wish.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _utilities-exporter-howto-docker:
 | 
			
		||||
 | 
			
		||||
Docker
 | 
			
		||||
______
 | 
			
		||||
 | 
			
		||||
If you are :ref:`using Docker <setup-installation-docker>`, running the
 | 
			
		||||
expoorter is almost as easy.  To mount a volume for exports, follow the
 | 
			
		||||
instructions in the ``docker-compose.yml.example`` file for the ``/export``
 | 
			
		||||
volume (making the changes in your own ``docker-compose.yml`` file, of course).
 | 
			
		||||
Once you have the volume mounted, the command to run an export is:
 | 
			
		||||
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
   $ docker-compose run --rm consumer document_exporter /export
 | 
			
		||||
 | 
			
		||||
If you prefer to use ``docker run`` directly, supplying the necessary commandline
 | 
			
		||||
options:
 | 
			
		||||
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
   $ # Identify your containers
 | 
			
		||||
   $ docker-compose ps
 | 
			
		||||
           Name                       Command                State     Ports
 | 
			
		||||
   -------------------------------------------------------------------------
 | 
			
		||||
   paperless_consumer_1    /sbin/docker-entrypoint.sh ...   Exit 0
 | 
			
		||||
   paperless_webserver_1   /sbin/docker-entrypoint.sh ...   Exit 0
 | 
			
		||||
 | 
			
		||||
   $ # Make sure to replace your passphrase and remove or adapt the id mapping
 | 
			
		||||
   $ docker run --rm \
 | 
			
		||||
       --volumes-from paperless_data_1 \
 | 
			
		||||
       --volume /path/to/arbitrary/place:/export \
 | 
			
		||||
       -e PAPERLESS_PASSPHRASE=YOUR_PASSPHRASE \
 | 
			
		||||
       -e USERMAP_UID=1000 -e USERMAP_GID=1000 \
 | 
			
		||||
       paperless document_exporter /export
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _utilities-importer:
 | 
			
		||||
 | 
			
		||||
The Importer
 | 
			
		||||
------------
 | 
			
		||||
 | 
			
		||||
Looking to transfer Paperless data from one instance to another, or just want
 | 
			
		||||
to restore from a backup?  This is your go-to toy.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _utilities-importer-howto:
 | 
			
		||||
 | 
			
		||||
How to Use It
 | 
			
		||||
.............
 | 
			
		||||
 | 
			
		||||
The importer works just like the exporter.  You point it at a directory, and
 | 
			
		||||
the script does the rest of the work:
 | 
			
		||||
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
    $ /path/to/paperless/src/manage.py document_importer /path/to/somewhere/
 | 
			
		||||
 | 
			
		||||
Docker
 | 
			
		||||
______
 | 
			
		||||
 | 
			
		||||
Assuming that you've already gone through the steps above in the
 | 
			
		||||
:ref:`export <utilities-exporter-howto-docker>` section, then the easiest thing
 | 
			
		||||
to do is just re-use the ``/export`` path you already setup:
 | 
			
		||||
 | 
			
		||||
.. code-block:: shell-session
 | 
			
		||||
 | 
			
		||||
   $ docker-compose run --rm consumer document_importer /export
 | 
			
		||||
 | 
			
		||||
Similarly, if you're not using docker-compose, you can adjust the export
 | 
			
		||||
instructions above to do the import.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
.. _utilities-retagger:
 | 
			
		||||
 
 | 
			
		||||
@@ -3,7 +3,7 @@ from django.contrib.auth.models import User, Group
 | 
			
		||||
from django.core.urlresolvers import reverse
 | 
			
		||||
from django.templatetags.static import static
 | 
			
		||||
 | 
			
		||||
from .models import Sender, Tag, Document, Log
 | 
			
		||||
from .models import Correspondent, Tag, Document, Log
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
class MonthListFilter(admin.SimpleListFilter):
 | 
			
		||||
@@ -107,7 +107,7 @@ class LogAdmin(admin.ModelAdmin):
 | 
			
		||||
    list_filter = ("level", "component",)
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
admin.site.register(Sender)
 | 
			
		||||
admin.site.register(Correspondent)
 | 
			
		||||
admin.site.register(Tag, TagAdmin)
 | 
			
		||||
admin.site.register(Document, DocumentAdmin)
 | 
			
		||||
admin.site.register(Log, LogAdmin)
 | 
			
		||||
 
 | 
			
		||||
@@ -24,7 +24,7 @@ from pyocr.tesseract import TesseractError
 | 
			
		||||
 | 
			
		||||
from paperless.db import GnuPG
 | 
			
		||||
 | 
			
		||||
from .models import Sender, Tag, Document, Log
 | 
			
		||||
from .models import Correspondent, Tag, Document, Log
 | 
			
		||||
from .languages import ISO639
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
@@ -246,7 +246,7 @@ class Consumer(object):
 | 
			
		||||
        """
 | 
			
		||||
 | 
			
		||||
        def get_sender(sender_name):
 | 
			
		||||
            return Sender.objects.get_or_create(
 | 
			
		||||
            return Correspondent.objects.get_or_create(
 | 
			
		||||
                name=sender_name, defaults={"slug": slugify(sender_name)})[0]
 | 
			
		||||
 | 
			
		||||
        def get_tags(tags):
 | 
			
		||||
 
 | 
			
		||||
@@ -8,7 +8,7 @@ from time import mktime
 | 
			
		||||
from django import forms
 | 
			
		||||
from django.conf import settings
 | 
			
		||||
 | 
			
		||||
from .models import Document, Sender
 | 
			
		||||
from .models import Document, Correspondent
 | 
			
		||||
from .consumer import Consumer
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
@@ -24,7 +24,9 @@ class UploadForm(forms.Form):
 | 
			
		||||
    }
 | 
			
		||||
 | 
			
		||||
    sender = forms.CharField(
 | 
			
		||||
        max_length=Sender._meta.get_field("name").max_length, required=False)
 | 
			
		||||
        max_length=Correspondent._meta.get_field("name").max_length,
 | 
			
		||||
        required=False
 | 
			
		||||
    )
 | 
			
		||||
    title = forms.CharField(
 | 
			
		||||
        max_length=Document._meta.get_field("title").max_length,
 | 
			
		||||
        required=False
 | 
			
		||||
@@ -41,7 +43,7 @@ class UploadForm(forms.Form):
 | 
			
		||||
        sender = self.cleaned_data.get("sender")
 | 
			
		||||
        if not sender:
 | 
			
		||||
            return None
 | 
			
		||||
        if not Sender.SAFE_REGEX.match(sender) or " - " in sender:
 | 
			
		||||
        if not Correspondent.SAFE_REGEX.match(sender) or " - " in sender:
 | 
			
		||||
            raise forms.ValidationError("That sender name is suspicious.")
 | 
			
		||||
        return sender
 | 
			
		||||
 | 
			
		||||
@@ -49,7 +51,7 @@ class UploadForm(forms.Form):
 | 
			
		||||
        title = self.cleaned_data.get("title")
 | 
			
		||||
        if not title:
 | 
			
		||||
            return None
 | 
			
		||||
        if not Sender.SAFE_REGEX.match(title) or " - " in title:
 | 
			
		||||
        if not Correspondent.SAFE_REGEX.match(title) or " - " in title:
 | 
			
		||||
            raise forms.ValidationError("That title is suspicious.")
 | 
			
		||||
 | 
			
		||||
    def clean_document(self):
 | 
			
		||||
 
 | 
			
		||||
@@ -14,7 +14,7 @@ from dateutil import parser
 | 
			
		||||
from django.conf import settings
 | 
			
		||||
 | 
			
		||||
from .consumer import Consumer
 | 
			
		||||
from .models import Sender, Log
 | 
			
		||||
from .models import Correspondent, Log
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
class MailFetcherError(Exception):
 | 
			
		||||
@@ -103,7 +103,7 @@ class Message(Loggable):
 | 
			
		||||
    def check_subject(self):
 | 
			
		||||
        if self.subject is None:
 | 
			
		||||
            raise InvalidMessageError("Message does not have a subject")
 | 
			
		||||
        if not Sender.SAFE_REGEX.match(self.subject):
 | 
			
		||||
        if not Correspondent.SAFE_REGEX.match(self.subject):
 | 
			
		||||
            raise InvalidMessageError("Message subject is unsafe: {}".format(
 | 
			
		||||
                self.subject))
 | 
			
		||||
 | 
			
		||||
 
 | 
			
		||||
@@ -1,10 +1,12 @@
 | 
			
		||||
import json
 | 
			
		||||
import os
 | 
			
		||||
import time
 | 
			
		||||
 | 
			
		||||
from django.conf import settings
 | 
			
		||||
from django.core.management.base import BaseCommand, CommandError
 | 
			
		||||
from django.core import serializers
 | 
			
		||||
 | 
			
		||||
from documents.models import Document
 | 
			
		||||
from documents.models import Document, Correspondent, Tag
 | 
			
		||||
from paperless.db import GnuPG
 | 
			
		||||
 | 
			
		||||
from ...mixins import Renderable
 | 
			
		||||
@@ -14,21 +16,19 @@ class Command(Renderable, BaseCommand):
 | 
			
		||||
 | 
			
		||||
    help = """
 | 
			
		||||
        Decrypt and rename all files in our collection into a given target
 | 
			
		||||
        directory.  Note that we don't export any of the parsed data since
 | 
			
		||||
        that can always be re-collected via the consumer.
 | 
			
		||||
        directory.  And include a manifest file containing document data for
 | 
			
		||||
        easy import.
 | 
			
		||||
    """.replace("    ", "")
 | 
			
		||||
 | 
			
		||||
    def add_arguments(self, parser):
 | 
			
		||||
        parser.add_argument("target")
 | 
			
		||||
 | 
			
		||||
    def __init__(self, *args, **kwargs):
 | 
			
		||||
        self.verbosity = 0
 | 
			
		||||
        self.target = None
 | 
			
		||||
        BaseCommand.__init__(self, *args, **kwargs)
 | 
			
		||||
        self.target = None
 | 
			
		||||
 | 
			
		||||
    def handle(self, *args, **options):
 | 
			
		||||
 | 
			
		||||
        self.verbosity = options["verbosity"]
 | 
			
		||||
        self.target = options["target"]
 | 
			
		||||
 | 
			
		||||
        if not os.path.exists(self.target):
 | 
			
		||||
@@ -40,9 +40,15 @@ class Command(Renderable, BaseCommand):
 | 
			
		||||
        if not settings.PASSPHRASE:
 | 
			
		||||
            settings.PASSPHRASE = input("Please enter the passphrase: ")
 | 
			
		||||
 | 
			
		||||
        for document in Document.objects.all():
 | 
			
		||||
        documents = Document.objects.all()
 | 
			
		||||
        document_map = {d.pk: d for d in documents}
 | 
			
		||||
        manifest = json.loads(serializers.serialize("json", documents))
 | 
			
		||||
        for document_dict in manifest:
 | 
			
		||||
 | 
			
		||||
            document = document_map[document_dict["pk"]]
 | 
			
		||||
 | 
			
		||||
            target = os.path.join(self.target, document.file_name)
 | 
			
		||||
            document_dict["__exported_file_name__"] = target
 | 
			
		||||
 | 
			
		||||
            print("Exporting: {}".format(target))
 | 
			
		||||
 | 
			
		||||
@@ -50,3 +56,12 @@ class Command(Renderable, BaseCommand):
 | 
			
		||||
                f.write(GnuPG.decrypted(document.source_file))
 | 
			
		||||
                t = int(time.mktime(document.created.timetuple()))
 | 
			
		||||
                os.utime(target, times=(t, t))
 | 
			
		||||
 | 
			
		||||
        manifest += json.loads(
 | 
			
		||||
            serializers.serialize("json", Correspondent.objects.all()))
 | 
			
		||||
 | 
			
		||||
        manifest += json.loads(serializers.serialize(
 | 
			
		||||
            "json", Tag.objects.all()))
 | 
			
		||||
 | 
			
		||||
        with open(os.path.join(self.target, "manifest.json"), "w") as f:
 | 
			
		||||
            json.dump(manifest, f, indent=2)
 | 
			
		||||
 
 | 
			
		||||
							
								
								
									
										110
									
								
								src/documents/management/commands/document_importer.py
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										110
									
								
								src/documents/management/commands/document_importer.py
									
									
									
									
									
										Normal file
									
								
							@@ -0,0 +1,110 @@
 | 
			
		||||
import json
 | 
			
		||||
import os
 | 
			
		||||
 | 
			
		||||
from django.conf import settings
 | 
			
		||||
from django.core.management.base import BaseCommand, CommandError
 | 
			
		||||
from django.core.management import call_command
 | 
			
		||||
 | 
			
		||||
from documents.models import Document
 | 
			
		||||
from paperless.db import GnuPG
 | 
			
		||||
 | 
			
		||||
from ...mixins import Renderable
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
class Command(Renderable, BaseCommand):
 | 
			
		||||
 | 
			
		||||
    help = """
 | 
			
		||||
        Using a manifest.json file, load the data from there, and import the
 | 
			
		||||
        documents it refers to.
 | 
			
		||||
    """.replace("    ", "")
 | 
			
		||||
 | 
			
		||||
    def add_arguments(self, parser):
 | 
			
		||||
        parser.add_argument("source")
 | 
			
		||||
        parser.add_argument(
 | 
			
		||||
            '--ignore-absent',
 | 
			
		||||
            action='store_true',
 | 
			
		||||
            default=False,
 | 
			
		||||
            help="If the manifest refers to a document that doesn't exist, "
 | 
			
		||||
                 "ignore it and attempt to import what it can"
 | 
			
		||||
        )
 | 
			
		||||
 | 
			
		||||
    def __init__(self, *args, **kwargs):
 | 
			
		||||
        BaseCommand.__init__(self, *args, **kwargs)
 | 
			
		||||
        self.source = None
 | 
			
		||||
        self.manifest = None
 | 
			
		||||
 | 
			
		||||
    def handle(self, *args, **options):
 | 
			
		||||
 | 
			
		||||
        self.source = options["source"]
 | 
			
		||||
 | 
			
		||||
        if not os.path.exists(self.source):
 | 
			
		||||
            raise CommandError("That path doesn't exist")
 | 
			
		||||
 | 
			
		||||
        if not os.access(self.source, os.R_OK):
 | 
			
		||||
            raise CommandError("That path doesn't appear to be readable")
 | 
			
		||||
 | 
			
		||||
        manifest_path = os.path.join(self.source, "manifest.json")
 | 
			
		||||
        self._check_manifest_exists(manifest_path)
 | 
			
		||||
 | 
			
		||||
        with open(manifest_path) as f:
 | 
			
		||||
            self.manifest = json.load(f)
 | 
			
		||||
 | 
			
		||||
        self._check_manifest()
 | 
			
		||||
 | 
			
		||||
        if not settings.PASSPHRASE:
 | 
			
		||||
            raise CommandError(
 | 
			
		||||
                "You need to define a passphrase before continuing.  Please "
 | 
			
		||||
                "consult the documentation for setting up Paperless."
 | 
			
		||||
            )
 | 
			
		||||
 | 
			
		||||
        # Fill up the database with whatever is in the manifest
 | 
			
		||||
        call_command("loaddata", manifest_path)
 | 
			
		||||
 | 
			
		||||
        self._import_files_from_manifest()
 | 
			
		||||
 | 
			
		||||
    @staticmethod
 | 
			
		||||
    def _check_manifest_exists(path):
 | 
			
		||||
        if not os.path.exists(path):
 | 
			
		||||
            raise CommandError(
 | 
			
		||||
                "That directory doesn't appear to contain a manifest.json "
 | 
			
		||||
                "file."
 | 
			
		||||
            )
 | 
			
		||||
 | 
			
		||||
    def _check_manifest(self):
 | 
			
		||||
 | 
			
		||||
        for record in self.manifest:
 | 
			
		||||
 | 
			
		||||
            if not record["model"] == "documents.document":
 | 
			
		||||
                continue
 | 
			
		||||
 | 
			
		||||
            if "__exported_file_name__" not in record:
 | 
			
		||||
                raise CommandError(
 | 
			
		||||
                    'The manifest file contains a record which does not '
 | 
			
		||||
                    'refer to an actual document file.  If you want to import '
 | 
			
		||||
                    'the rest anyway (skipping such references) call the '
 | 
			
		||||
                    'importer with --ignore-absent'
 | 
			
		||||
                )
 | 
			
		||||
 | 
			
		||||
            doc_file = record["__exported_file_name__"]
 | 
			
		||||
            if not os.path.exists(os.path.join(self.source, doc_file)):
 | 
			
		||||
                raise CommandError(
 | 
			
		||||
                    'The manifest file refers to "{}" which does not '
 | 
			
		||||
                    'appear to be in the source directory.  If you want to '
 | 
			
		||||
                    'import the rest anyway (skipping such references) call '
 | 
			
		||||
                    'the importer with --ignore-absent'.format(doc_file)
 | 
			
		||||
                )
 | 
			
		||||
 | 
			
		||||
    def _import_files_from_manifest(self):
 | 
			
		||||
 | 
			
		||||
        for record in self.manifest:
 | 
			
		||||
 | 
			
		||||
            if not record["model"] == "documents.document":
 | 
			
		||||
                continue
 | 
			
		||||
 | 
			
		||||
            doc_file = record["__exported_file_name__"]
 | 
			
		||||
            document = Document.objects.get(pk=record["pk"])
 | 
			
		||||
            with open(doc_file, "rb") as unencrypted:
 | 
			
		||||
                with open(document.source_path, "wb") as encrypted:
 | 
			
		||||
                    print("Encrypting {} and saving it to {}".format(
 | 
			
		||||
                        doc_file, document.source_path))
 | 
			
		||||
                    encrypted.write(GnuPG.encrypted(unencrypted))
 | 
			
		||||
							
								
								
									
										19
									
								
								src/documents/migrations/0011_auto_20160303_1929.py
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										19
									
								
								src/documents/migrations/0011_auto_20160303_1929.py
									
									
									
									
									
										Normal file
									
								
							@@ -0,0 +1,19 @@
 | 
			
		||||
# -*- coding: utf-8 -*-
 | 
			
		||||
# Generated by Django 1.9.2 on 2016-03-03 19:29
 | 
			
		||||
from __future__ import unicode_literals
 | 
			
		||||
 | 
			
		||||
from django.db import migrations
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
class Migration(migrations.Migration):
 | 
			
		||||
 | 
			
		||||
    dependencies = [
 | 
			
		||||
        ('documents', '0010_log'),
 | 
			
		||||
    ]
 | 
			
		||||
 | 
			
		||||
    operations = [
 | 
			
		||||
        migrations.RenameModel(
 | 
			
		||||
            old_name='Sender',
 | 
			
		||||
            new_name='Correspondent',
 | 
			
		||||
        ),
 | 
			
		||||
    ]
 | 
			
		||||
@@ -28,7 +28,7 @@ class SluggedModel(models.Model):
 | 
			
		||||
        return self.name
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
class Sender(SluggedModel):
 | 
			
		||||
class Correspondent(SluggedModel):
 | 
			
		||||
 | 
			
		||||
    # This regex is probably more restrictive than it needs to be, but it's
 | 
			
		||||
    # better safe than sorry.
 | 
			
		||||
@@ -141,7 +141,7 @@ class Document(models.Model):
 | 
			
		||||
    TYPES = (TYPE_PDF, TYPE_PNG, TYPE_JPG, TYPE_GIF, TYPE_TIF,)
 | 
			
		||||
 | 
			
		||||
    sender = models.ForeignKey(
 | 
			
		||||
        Sender, blank=True, null=True, related_name="documents")
 | 
			
		||||
        Correspondent, blank=True, null=True, related_name="documents")
 | 
			
		||||
    title = models.CharField(max_length=128, blank=True, db_index=True)
 | 
			
		||||
    content = models.TextField(db_index=True)
 | 
			
		||||
    file_type = models.CharField(
 | 
			
		||||
@@ -158,9 +158,9 @@ class Document(models.Model):
 | 
			
		||||
        ordering = ("sender", "title")
 | 
			
		||||
 | 
			
		||||
    def __str__(self):
 | 
			
		||||
        created = self.created.strftime("%Y-%m-%d")
 | 
			
		||||
        created = self.created.strftime("%Y%m%d%H%M%S")
 | 
			
		||||
        if self.sender and self.title:
 | 
			
		||||
            return "{}: {}, {}".format(created, self.sender, self.title)
 | 
			
		||||
            return "{}: {} - {}".format(created, self.sender, self.title)
 | 
			
		||||
        if self.sender or self.title:
 | 
			
		||||
            return "{}: {}".format(created, self.sender or self.title)
 | 
			
		||||
        return str(created)
 | 
			
		||||
@@ -179,13 +179,7 @@ class Document(models.Model):
 | 
			
		||||
 | 
			
		||||
    @property
 | 
			
		||||
    def file_name(self):
 | 
			
		||||
        if self.sender and self.title:
 | 
			
		||||
            tags = ",".join([t.slug for t in self.tags.all()])
 | 
			
		||||
            if tags:
 | 
			
		||||
                return "{} - {} - {}.{}".format(
 | 
			
		||||
                    self.sender, self.title, tags, self.file_type)
 | 
			
		||||
            return "{} - {}.{}".format(self.sender, self.title, self.file_type)
 | 
			
		||||
        return os.path.basename(self.source_path)
 | 
			
		||||
        return slugify(str(self)) + "." + self.file_type
 | 
			
		||||
 | 
			
		||||
    @property
 | 
			
		||||
    def download_url(self):
 | 
			
		||||
 
 | 
			
		||||
@@ -1,12 +1,12 @@
 | 
			
		||||
from rest_framework import serializers
 | 
			
		||||
 | 
			
		||||
from .models import Sender, Tag, Document, Log
 | 
			
		||||
from .models import Correspondent, Tag, Document, Log
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
class SenderSerializer(serializers.HyperlinkedModelSerializer):
 | 
			
		||||
class CorrespondentSerializer(serializers.HyperlinkedModelSerializer):
 | 
			
		||||
 | 
			
		||||
    class Meta(object):
 | 
			
		||||
        model = Sender
 | 
			
		||||
        model = Correspondent
 | 
			
		||||
        fields = ("id", "slug", "name")
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
 
 | 
			
		||||
@@ -1,6 +1,5 @@
 | 
			
		||||
from django.contrib.auth.mixins import LoginRequiredMixin
 | 
			
		||||
from django.http import HttpResponse
 | 
			
		||||
from django.template.defaultfilters import slugify
 | 
			
		||||
from django.views.decorators.csrf import csrf_exempt
 | 
			
		||||
from django.views.generic import FormView, DetailView, TemplateView
 | 
			
		||||
 | 
			
		||||
@@ -14,9 +13,9 @@ from rest_framework.viewsets import (
 | 
			
		||||
from paperless.db import GnuPG
 | 
			
		||||
 | 
			
		||||
from .forms import UploadForm
 | 
			
		||||
from .models import Sender, Tag, Document, Log
 | 
			
		||||
from .models import Correspondent, Tag, Document, Log
 | 
			
		||||
from .serialisers import (
 | 
			
		||||
    SenderSerializer, TagSerializer, DocumentSerializer, LogSerializer)
 | 
			
		||||
    CorrespondentSerializer, TagSerializer, DocumentSerializer, LogSerializer)
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
class IndexView(TemplateView):
 | 
			
		||||
@@ -52,7 +51,7 @@ class FetchView(LoginRequiredMixin, DetailView):
 | 
			
		||||
            content_type=content_types[self.object.file_type]
 | 
			
		||||
        )
 | 
			
		||||
        response["Content-Disposition"] = 'attachment; filename="{}"'.format(
 | 
			
		||||
            slugify(str(self.object)) + "." + self.object.file_type)
 | 
			
		||||
            self.object.file_name)
 | 
			
		||||
 | 
			
		||||
        return response
 | 
			
		||||
 | 
			
		||||
@@ -81,10 +80,10 @@ class StandardPagination(PageNumberPagination):
 | 
			
		||||
    max_page_size = 100000
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
class SenderViewSet(ModelViewSet):
 | 
			
		||||
    model = Sender
 | 
			
		||||
    queryset = Sender.objects.all()
 | 
			
		||||
    serializer_class = SenderSerializer
 | 
			
		||||
class CorrespondentViewSet(ModelViewSet):
 | 
			
		||||
    model = Correspondent
 | 
			
		||||
    queryset = Correspondent.objects.all()
 | 
			
		||||
    serializer_class = CorrespondentSerializer
 | 
			
		||||
    pagination_class = StandardPagination
 | 
			
		||||
    permission_classes = (IsAuthenticated,)
 | 
			
		||||
 | 
			
		||||
 
 | 
			
		||||
@@ -22,11 +22,11 @@ from rest_framework.routers import DefaultRouter
 | 
			
		||||
 | 
			
		||||
from documents.views import (
 | 
			
		||||
    IndexView, FetchView, PushView,
 | 
			
		||||
    SenderViewSet, TagViewSet, DocumentViewSet, LogViewSet
 | 
			
		||||
    CorrespondentViewSet, TagViewSet, DocumentViewSet, LogViewSet
 | 
			
		||||
)
 | 
			
		||||
 | 
			
		||||
router = DefaultRouter()
 | 
			
		||||
router.register(r'senders', SenderViewSet)
 | 
			
		||||
router.register(r'senders', CorrespondentViewSet)
 | 
			
		||||
router.register(r'tags', TagViewSet)
 | 
			
		||||
router.register(r'documents', DocumentViewSet)
 | 
			
		||||
router.register(r'logs', LogViewSet)
 | 
			
		||||
 
 | 
			
		||||
		Reference in New Issue
	
	Block a user