Enhancement: support import of zipped export (#10073)

This reverts commit 9c9a0e4496.

.github/workflows/ci.yml

@@ -162,7 +162,7 @@ jobs:
       - name: Install pnpm
         uses: pnpm/action-setup@v4
         with:
-          package_json_file: 'src-ui/package.json'
+          version: 10
       - name: Use Node.js 20
         uses: actions/setup-node@v4
         with:
@@ -195,7 +195,7 @@ jobs:
       - name: Install pnpm
         uses: pnpm/action-setup@v4
         with:
-          package_json_file: 'src-ui/package.json'
+          version: 10
       - name: Use Node.js 20
         uses: actions/setup-node@v4
         with:
@@ -245,7 +245,7 @@ jobs:
       - name: Install pnpm
         uses: pnpm/action-setup@v4
         with:
-          package_json_file: 'src-ui/package.json'
+          version: 10
       - name: Use Node.js 20
         uses: actions/setup-node@v4
         with:
@@ -288,7 +288,7 @@ jobs:
       - name: Install pnpm
         uses: pnpm/action-setup@v4
         with:
-          package_json_file: 'src-ui/package.json'
+          version: 10
       - name: Use Node.js 20
         uses: actions/setup-node@v4
         with:

.github/workflows/crowdin.yml

@@ -14,6 +14,8 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.PNGX_BOT_PAT }}
       - name: crowdin action
         uses: crowdin/github-action@v2
         with:

docs/administration.md

@@ -333,7 +333,7 @@ must be provided to import. If this value is lost, the export cannot be imported
 The document importer takes the export produced by the [Document
 exporter](#exporter) and imports it into paperless.
 
-The importer works just like the exporter. You point it at a directory,
+The importer works just like the exporter. You point it at a directory or the generated .zip file,
 and the script does the rest of the work:
 
 ```shell
@@ -351,9 +351,6 @@ When you use the provided docker compose script, put the export inside
 the `export` folder in your paperless source directory. Specify
 `../export` as the `source`.
 
-Note that .zip files (as can be generated from the exporter) are not supported. You must unzip them into
-the target directory first.
-
 !!! note
 
     Importing from a previous version of Paperless may work, but for best

docs/changelog.md

@@ -1,5 +1,61 @@
 # Changelog
 
+## paperless-ngx 2.16.3
+
+### Features / Enhancements
+
+-   Performance: pre-filter document list in scheduled workflow checks [@shamoon](https://github.com/shamoon) ([#10031](https://github.com/paperless-ngx/paperless-ngx/pull/10031))
+-   Chore: refactor consumer plugin checks to a pre-flight plugin [@shamoon](https://github.com/shamoon) ([#9994](https://github.com/paperless-ngx/paperless-ngx/pull/9994))
+-   Enhancement: include DOCUMENT_TYPE to post consume scripts [@matthesrieke](https://github.com/matthesrieke) ([#9977](https://github.com/paperless-ngx/paperless-ngx/pull/9977))
+
+### Bug Fixes
+
+-   Fix: handle whoosh query correction errors [@shamoon](https://github.com/shamoon) ([#10121](https://github.com/paperless-ngx/paperless-ngx/pull/10121))
+-   Fix: handle favicon with non-default static dir [@shamoon](https://github.com/shamoon) ([#10107](https://github.com/paperless-ngx/paperless-ngx/pull/10107))
+-   Fixhancement: cleanup user or group references in settings upon deletion [@shamoon](https://github.com/shamoon) ([#10049](https://github.com/paperless-ngx/paperless-ngx/pull/10049))
+-   Fix: Add exception to in [@Freilichtbuehne](https://github.com/Freilichtbuehne) ([#10070](https://github.com/paperless-ngx/paperless-ngx/pull/10070))
+-   Fix: include base href when opening global search result in new window [@shamoon](https://github.com/shamoon) ([#10066](https://github.com/paperless-ngx/paperless-ngx/pull/10066))
+
+### Dependencies
+
+<details>
+<summary>10 changes</summary>
+
+-   Chore(deps): Bump the small-changes group across 1 directory with 3 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#10085](https://github.com/paperless-ngx/paperless-ngx/pull/10085))
+-   Chore(deps): Update granian[uvloop] requirement from ~=2.2.0 to ~=2.3.2 @[dependabot[bot]](https://github.com/apps/dependabot) ([#10055](https://github.com/paperless-ngx/paperless-ngx/pull/10055))
+-   Chore(deps): Bump the frontend-angular-dependencies group in /src-ui with 18 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#10099](https://github.com/paperless-ngx/paperless-ngx/pull/10099))
+-   Chore(deps-dev): Bump the frontend-eslint-dependencies group in /src-ui with 4 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#10100](https://github.com/paperless-ngx/paperless-ngx/pull/10100))
+-   Chore(deps): Bump bootstrap from 5.3.3 to 5.3.6 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#10091](https://github.com/paperless-ngx/paperless-ngx/pull/10091))
+-   Chore(deps-dev): Bump @codecov/webpack-plugin from 1.9.0 to 1.9.1 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#10090](https://github.com/paperless-ngx/paperless-ngx/pull/10090))
+-   Chore(deps-dev): Bump @types/node from 22.15.3 to 22.15.29 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#10089](https://github.com/paperless-ngx/paperless-ngx/pull/10089))
+-   Chore(deps): Bump zone.js from 0.15.0 to 0.15.1 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#10088](https://github.com/paperless-ngx/paperless-ngx/pull/10088))
+-   docker(deps): bump astral-sh/uv from 0.7.8-python3.12-bookworm-slim to 0.7.9-python3.12-bookworm-slim @[dependabot[bot]](https://github.com/apps/dependabot) ([#10084](https://github.com/paperless-ngx/paperless-ngx/pull/10084))
+-   docker(deps): Bump astral-sh/uv from 0.6.16-python3.12-bookworm-slim to 0.7.8-python3.12-bookworm-slim @[dependabot[bot]](https://github.com/apps/dependabot) ([#10056](https://github.com/paperless-ngx/paperless-ngx/pull/10056))
+</details>
+
+### All App Changes
+
+<details>
+<summary>16 changes</summary>
+
+-   Fix: handle whoosh query correction errors [@shamoon](https://github.com/shamoon) ([#10121](https://github.com/paperless-ngx/paperless-ngx/pull/10121))
+-   Performance: pre-filter document list in scheduled workflow checks [@shamoon](https://github.com/shamoon) ([#10031](https://github.com/paperless-ngx/paperless-ngx/pull/10031))
+-   Chore(deps): Bump the small-changes group across 1 directory with 3 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#10085](https://github.com/paperless-ngx/paperless-ngx/pull/10085))
+-   Chore: refactor consumer plugin checks to a pre-flight plugin [@shamoon](https://github.com/shamoon) ([#9994](https://github.com/paperless-ngx/paperless-ngx/pull/9994))
+-   Chore(deps): Update granian[uvloop] requirement from ~=2.2.0 to ~=2.3.2 @[dependabot[bot]](https://github.com/apps/dependabot) ([#10055](https://github.com/paperless-ngx/paperless-ngx/pull/10055))
+-   Fix: handle favicon with non-default static dir [@shamoon](https://github.com/shamoon) ([#10107](https://github.com/paperless-ngx/paperless-ngx/pull/10107))
+-   Chore(deps): Bump the frontend-angular-dependencies group in /src-ui with 18 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#10099](https://github.com/paperless-ngx/paperless-ngx/pull/10099))
+-   Chore(deps-dev): Bump the frontend-eslint-dependencies group in /src-ui with 4 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#10100](https://github.com/paperless-ngx/paperless-ngx/pull/10100))
+-   Chore(deps): Bump bootstrap from 5.3.3 to 5.3.6 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#10091](https://github.com/paperless-ngx/paperless-ngx/pull/10091))
+-   Chore(deps-dev): Bump @codecov/webpack-plugin from 1.9.0 to 1.9.1 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#10090](https://github.com/paperless-ngx/paperless-ngx/pull/10090))
+-   Chore(deps-dev): Bump @types/node from 22.15.3 to 22.15.29 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#10089](https://github.com/paperless-ngx/paperless-ngx/pull/10089))
+-   Chore(deps): Bump zone.js from 0.15.0 to 0.15.1 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#10088](https://github.com/paperless-ngx/paperless-ngx/pull/10088))
+-   Fixhancement: cleanup user or group references in settings upon deletion [@shamoon](https://github.com/shamoon) ([#10049](https://github.com/paperless-ngx/paperless-ngx/pull/10049))
+-   Enhancement: include DOCUMENT_TYPE to post consume scripts [@matthesrieke](https://github.com/matthesrieke) ([#9977](https://github.com/paperless-ngx/paperless-ngx/pull/9977))
+-   Fix: Add exception to in [@Freilichtbuehne](https://github.com/Freilichtbuehne) ([#10070](https://github.com/paperless-ngx/paperless-ngx/pull/10070))
+-   Fix: include base href when opening global search result in new window [@shamoon](https://github.com/shamoon) ([#10066](https://github.com/paperless-ngx/paperless-ngx/pull/10066))
+</details>
+
 ## paperless-ngx 2.16.2
 
 ### Bug Fixes

src-ui/package.json

@@ -69,13 +69,6 @@
     "ts-node": "~10.9.1",
     "typescript": "^5.5.4"
   },
-  "packageManager": "pnpm@10.11.1",
-  "devEngines": {
-    "packageManager": {
-      "name": "pnpm",
-      "version": "10.11.1"
-    }
-  },
   "pnpm": {
     "onlyBuiltDependencies": [
       "@parcel/watcher",

src/documents/management/commands/document_importer.py

@@ -1,9 +1,12 @@
 import json
 import logging
 import os
+import tempfile
 from collections.abc import Generator
 from contextlib import contextmanager
 from pathlib import Path
+from zipfile import ZipFile
+from zipfile import is_zipfile
 
 import tqdm
 from django.conf import settings
@@ -234,14 +237,19 @@ class Command(CryptMixin, BaseCommand):
         self.manifest_paths = []
         self.manifest = []
 
+        # Create a temporary directory for extracting a zip file into it, even if supplied source is no zip file to keep code cleaner.
+        with tempfile.TemporaryDirectory() as tmp_dir:
+            if is_zipfile(self.source):
+                with ZipFile(self.source) as zf:
+                    zf.extractall(tmp_dir)
+                self.source = Path(tmp_dir)
+            self._run_import()
+
+    def _run_import(self):
         self.pre_check()
-
         self.load_metadata()
-
         self.load_manifest_files()
-
         self.check_manifest_validity()
-
         self.decrypt_secret_fields()
 
         # see /src/documents/signals/handlers.py

src/documents/tests/test_management_importer.py

@@ -2,6 +2,7 @@ import json
 import tempfile
 from io import StringIO
 from pathlib import Path
+from zipfile import ZipFile
 
 from django.contrib.auth.models import User
 from django.core.management import call_command
@@ -335,3 +336,42 @@ class TestCommandImport(
 
         self.assertIn("Version mismatch:", stdout_str)
         self.assertIn("importing 2.8.1", stdout_str)
+
+    def test_import_zipped_export(self):
+        """
+        GIVEN:
+            - A zip file with correct content (manifest.json and version.json inside)
+        WHEN:
+            - An import is attempted using the zip file as the source
+        THEN:
+            - The command reads from the zip without warnings or errors
+        """
+
+        stdout = StringIO()
+        zip_path = self.dirs.scratch_dir / "export.zip"
+
+        # Create manifest.json and version.json in a temp dir
+        with tempfile.TemporaryDirectory() as temp_dir:
+            temp_dir_path = Path(temp_dir)
+
+            (temp_dir_path / "manifest.json").touch()
+            (temp_dir_path / "version.json").touch()
+
+            # Create the zip file
+            with ZipFile(zip_path, "w") as zf:
+                zf.write(temp_dir_path / "manifest.json", arcname="manifest.json")
+                zf.write(temp_dir_path / "version.json", arcname="version.json")
+
+        # Try to import from the zip file
+        with self.assertRaises(json.decoder.JSONDecodeError):
+            call_command(
+                "document_importer",
+                "--no-progress-bar",
+                str(zip_path),
+                stdout=stdout,
+            )
+        stdout.seek(0)
+        stdout_str = str(stdout.read())
+
+        # There should be no error or warnings. Therefore the output should be empty.
+        self.assertEqual(stdout_str, "")

src/documents/views.py

@@ -145,6 +145,7 @@ from documents.serialisers import CustomFieldSerializer
 from documents.serialisers import DocumentListSerializer
 from documents.serialisers import DocumentSerializer
 from documents.serialisers import DocumentTypeSerializer
+from documents.serialisers import NotesSerializer
 from documents.serialisers import PostDocumentSerializer
 from documents.serialisers import RunTaskViewSerializer
 from documents.serialisers import SavedViewSerializer
@@ -433,27 +434,24 @@ class DocumentTypeViewSet(ModelViewSet, PermissionsAwareDocumentCountMixin):
     ),
     notes=extend_schema(
         description="View, add, or delete notes for the document",
-        responses={
-            200: {
-                "type": "array",
-                "items": {
-                    "type": "object",
-                    "properties": {
-                        "id": {"type": "integer"},
-                        "note": {"type": "string"},
-                        "created": {"type": "string", "format": "date-time"},
-                        "user": {
-                            "type": "object",
-                            "properties": {
-                                "id": {"type": "integer"},
-                                "username": {"type": "string"},
-                                "first_name": {"type": "string"},
-                                "last_name": {"type": "string"},
-                            },
-                        },
-                    },
-                },
+        methods=["GET", "POST", "DELETE"],
+        request=inline_serializer(
+            name="NoteCreateRequest",
+            fields={
+                "note": serializers.CharField(),
             },
+        ),
+        parameters=[
+            OpenApiParameter(
+                name="id",
+                type=OpenApiTypes.INT,
+                location=OpenApiParameter.QUERY,
+                required=False,
+                description="Note ID to delete (used only for DELETE requests)",
+            ),
+        ],
+        responses={
+            200: NotesSerializer(many=True),
             400: None,
             403: None,
             404: None,
@@ -519,6 +517,28 @@ class DocumentTypeViewSet(ModelViewSet, PermissionsAwareDocumentCountMixin):
             404: None,
         },
     ),
+    email=extend_schema(
+        description="Email the document to one or more recipients as an attachment.",
+        request=inline_serializer(
+            name="EmailRequest",
+            fields={
+                "addresses": serializers.CharField(),
+                "subject": serializers.CharField(),
+                "message": serializers.CharField(),
+                "use_archive_version": serializers.BooleanField(default=True),
+            },
+        ),
+        responses={
+            200: inline_serializer(
+                name="EmailResponse",
+                fields={"message": serializers.CharField()},
+            ),
+            400: None,
+            403: None,
+            404: None,
+            500: None,
+        },
+    ),
 )
 class DocumentViewSet(
     PassUserMixin,
@@ -1079,6 +1099,12 @@ class DocumentViewSet(
             200: DocumentSerializer(many=True, all_fields=True),
         },
     ),
+    next_asn=extend_schema(
+        description="Get the next available Archive Serial Number (ASN) for a new document",
+        responses={
+            200: OpenApiTypes.INT,
+        },
+    ),
 )
 class UnifiedSearchViewSet(DocumentViewSet):
     def __init__(self, *args, **kwargs):

src/paperless_mail/views.py

@@ -35,6 +35,7 @@ from paperless_mail.tasks import process_mail_accounts
 @extend_schema_view(
     test=extend_schema(
         operation_id="mail_account_test",
+        request=MailAccountSerializer,
         description="Test a mail account",
         responses={
             200: inline_serializer(
@@ -44,6 +45,17 @@ from paperless_mail.tasks import process_mail_accounts
             400: OpenApiTypes.STR,
         },
     ),
+    process=extend_schema(
+        operation_id="mail_account_process",
+        description="Manually process the selected mail account for new messages.",
+        responses={
+            200: inline_serializer(
+                name="MailAccountProcessResponse",
+                fields={"result": serializers.CharField(default="OK")},
+            ),
+            404: None,
+        },
+    ),
 )
 class MailAccountViewSet(ModelViewSet, PassUserMixin):
     model = MailAccount