Compare commits

..

7 Commits

Author SHA1 Message Date
shamoon
d6cfd87cc0 Revert "Documentation: add note about logo file visibility and exif data"
This reverts commit 43b4f36026.
2025-09-21 16:08:02 -07:00
shamoon
7a287e7479 Merge branch 'main' into fix-strip-exif 2025-09-21 16:07:56 -07:00
shamoon
43b4f36026 Documentation: add note about logo file visibility and exif data 2025-09-21 16:07:29 -07:00
shamoon
76a81adcb5 Fix: remove extraneous exif from logo images 2025-09-21 09:52:18 -07:00
shamoon
6b868a5ecb Fix: restore str celery beat schedule filename (#10893) 2025-09-20 18:54:56 -07:00
shamoon
3e4aa87cc5 Fix formatting 2025-09-12 15:55:22 -07:00
shamoon
fc95d42b35 Documentation: add guidance for feature PRs in CONTRIBUTING.md 2025-09-12 15:51:49 -07:00
22 changed files with 203 additions and 828 deletions

View File

@@ -2,9 +2,11 @@
If you feel like contributing to the project, please do! Bug fixes and improvements are always welcome.
⚠️ Please note: Pull requests that implement a new feature or enhancement _should almost always target an existing feature request_ with evidence of community interest and discussion. This is in order to balance the work of implementing and maintaining new features / enhancements. Pull requests that are opened without meeting this requirement may not be merged.
If you want to implement something big:
- Please start a discussion about that in the issues! Maybe something similar is already in development and we can make it happen together.
- As above, please start with a discussion! Maybe something similar is already in development and we can make it happen together.
- When making additions to the project, consider if the majority of users will benefit from your change. If not, you're probably better of forking the project.
- Also consider if your change will get in the way of other users. A good change is a change that enhances the experience of some users who want that change and does not affect users who do not care about the change.
- Please see the [paperless-ngx merge process](#merging-prs) below.

View File

@@ -261,10 +261,6 @@ different means. These are as follows:
Paperless is set up to check your mails every 10 minutes. This can be
configured via [`PAPERLESS_EMAIL_TASK_CRON`](configuration.md#PAPERLESS_EMAIL_TASK_CRON)
#### Processed Mail
Paperless keeps track of emails it has processed in order to avoid processing the same mail multiple times. This uses the message `UID` provided by the mail server, which should be unique for each message. You can view and manage processed mails from the web UI under Mail > Processed Mails. If you need to re-process a message, you can delete the corresponding processed mail entry, which will allow Paperless-ngx to process the email again the next time the mail fetch task runs.
#### OAuth Email Setup
Paperless-ngx supports OAuth2 authentication for Gmail and Outlook email accounts. To set up an email account with OAuth2, you will need to create a 'developer' app with the respective provider and obtain the client ID and client secret and set the appropriate [configuration variables](configuration.md#email_oauth). You will also need to set either [`PAPERLESS_OAUTH_CALLBACK_BASE_URL`](configuration.md#PAPERLESS_OAUTH_CALLBACK_BASE_URL) or [`PAPERLESS_URL`](configuration.md#PAPERLESS_URL) to the correct value for the OAuth2 flow to work correctly.

View File

@@ -109,11 +109,10 @@
<li class="list-group-item">
<div class="row">
<div class="col" i18n>Name</div>
<div class="col-1 d-none d-sm-block" i18n>Sort Order</div>
<div class="col-2" i18n>Account</div>
<div class="col-2 d-none d-sm-block" i18n>Status</div>
<div class="col d-none d-sm-block" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.ProcessedMail }">Processed Mail</div>
<div class="col-3" i18n>Actions</div>
<div class="col d-none d-sm-block" i18n>Sort Order</div>
<div class="col" i18n>Account</div>
<div class="col d-none d-sm-block" i18n>Status</div>
<div class="col" i18n>Actions</div>
</div>
</li>
@@ -128,9 +127,9 @@
<li class="list-group-item">
<div class="row fade" [class.show]="showRules">
<div class="col d-flex align-items-center"><button class="btn btn-link p-0 text-start" type="button" (click)="editMailRule(rule)" [disabled]="!permissionsService.currentUserCan(PermissionAction.Change, PermissionType.MailRule) || !userCanEdit(rule)">{{rule.name}}</button></div>
<div class="col-1 d-flex align-items-center d-none d-sm-flex">{{rule.order}}</div>
<div class="col-2 d-flex align-items-center">{{(mailAccountService.getCached(rule.account) | async)?.name}}</div>
<div class="col-2 d-flex align-items-center d-none d-sm-flex">
<div class="col d-flex align-items-center d-none d-sm-flex">{{rule.order}}</div>
<div class="col d-flex align-items-center">{{(mailAccountService.getCached(rule.account) | async)?.name}}</div>
<div class="col d-flex align-items-center d-none d-sm-flex">
<div class="form-check form-switch mb-0">
<input #inputField type="checkbox" class="form-check-input cursor-pointer" [id]="rule.id+'_enable'" [(ngModel)]="rule.enabled" (change)="onMailRuleEnableToggled(rule)" *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.MailRule }">
<label class="form-check-label cursor-pointer" [for]="rule.id+'_enable'">
@@ -138,12 +137,7 @@
</label>
</div>
</div>
<div class="col d-flex align-items-center d-none d-sm-flex" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.ProcessedMail }">
<button class="btn btn-sm btn-outline-secondary" type="button" (click)="viewProcessedMail(rule)">
<i-bs width="1em" height="1em" name="clock-history"></i-bs>&nbsp;<ng-container i18n>View Processed Mail</ng-container>
</button>
</div>
<div class="col-3">
<div class="col">
<div class="btn-group d-block d-sm-none">
<div ngbDropdown container="body" class="d-inline-block">
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle>

View File

@@ -409,13 +409,4 @@ describe('MailComponent', () => {
jest.advanceTimersByTime(200)
expect(editSpy).toHaveBeenCalled()
})
it('should open processed mails dialog', () => {
completeSetup()
let modal: NgbModalRef
modalService.activeInstances.subscribe((refs) => (modal = refs[0]))
component.viewProcessedMail(mailRules[0] as MailRule)
const dialog = modal.componentInstance as any
expect(dialog.rule).toEqual(mailRules[0])
})
})

View File

@@ -27,7 +27,6 @@ import { MailRuleEditDialogComponent } from '../../common/edit-dialog/mail-rule-
import { PageHeaderComponent } from '../../common/page-header/page-header.component'
import { PermissionsDialogComponent } from '../../common/permissions-dialog/permissions-dialog.component'
import { ComponentWithPermissions } from '../../with-permissions/with-permissions.component'
import { ProcessedMailDialogComponent } from './processed-mail-dialog/processed-mail-dialog.component'
@Component({
selector: 'pngx-mail',
@@ -348,14 +347,6 @@ export class MailComponent
)
}
viewProcessedMail(rule: MailRule) {
const modal = this.modalService.open(ProcessedMailDialogComponent, {
backdrop: 'static',
size: 'xl',
})
modal.componentInstance.rule = rule
}
userCanEdit(obj: ObjectWithPermissions): boolean {
return this.permissionsService.currentUserHasObjectPermissions(
PermissionAction.Change,

View File

@@ -1,107 +0,0 @@
<div class="modal-header">
<h6 class="modal-title" id="modal-basic-title" i18n>Processed Mail for <em>{{ rule.name }}</em></h6>
<button class="btn btn-sm btn-link text-muted me-auto p-0 p-md-2" title="What's this?" i18n-title type="button" [ngbPopover]="infoPopover" [autoClose]="true">
<i-bs name="question-circle"></i-bs>
</button>
<ng-template #infoPopover>
<a href="https://docs.paperless-ngx.com/usage#processed-mail" target="_blank" referrerpolicy="noopener noreferrer" i18n>Read more</a>
<i-bs class="ms-1" width=".8em" height=".8em" name="box-arrow-up-right"></i-bs>
</ng-template>
<button type="button" class="btn-close" aria-label="Close" (click)="close()"></button>
</div>
<div class="modal-body">
@if (loading) {
<div class="text-center my-5">
<div class="spinner-border" role="status">
<span class="visually-hidden" i18n>Loading...</span>
</div>
</div>
} @else if (processedMails.length === 0) {
<span i18n>No processed email messages found.</span>
} @else {
<div class="table-responsive">
<table class="table table-hover table-sm align-middle">
<thead>
<tr>
<th scope="col" style="width: 40px;">
<div class="form-check m-0 ms-2 me-n2">
<input type="checkbox" class="form-check-input" id="all-objects" [(ngModel)]="toggleAllEnabled" [disabled]="processedMails.length === 0" (click)="toggleAll($event); $event.stopPropagation();">
<label class="form-check-label" for="all-objects"></label>
</div>
</th>
<th scope="col" i18n>Subject</th>
<th scope="col" i18n>Received</th>
<th scope="col" i18n>Processed</th>
<th scope="col" i18n>Status</th>
<th scope="col" i18n>Error</th>
</tr>
</thead>
<tbody>
@for (mail of processedMails; track mail.id) {
<ng-template #statusTooltip>
<div class="small text-light font-monospace">
{{mail.status}}
</div>
</ng-template>
<tr>
<td>
<div class="form-check m-0 ms-2 me-n2">
<input type="checkbox" class="form-check-input" [id]="mail.id" [checked]="selectedMailIds.has(mail.id)" (click)="toggleSelected(mail); $event.stopPropagation();">
<label class="form-check-label" [for]="mail.id"></label>
</div>
</td>
<td>{{ mail.subject }}</td>
<td>{{ mail.received | customDate:'longDate' }}</td>
<td>{{ mail.processed | customDate:'longDate' }}</td>
<td>
@switch (mail.status) {
@case ('SUCCESS') {
<i-bs name="check-circle" title="SUCCESS" class="text-success" [ngbTooltip]="statusTooltip"></i-bs>
}
@case ('FAILED') {
<i-bs name="exclamation-triangle" title="FAILED" class="text-danger" [ngbTooltip]="statusTooltip"></i-bs>
}
@default {
<i-bs name="slash-circle" title="{{ mail.status }}" class="text-muted" [ngbTooltip]="statusTooltip"></i-bs>
}
}
</td>
<td>
<ng-template #errorPopover>
<pre class="small text-light">
{{ mail.error }}
</pre>
</ng-template>
@if (mail.error) {
<span class="text-danger" triggers="mouseenter:mouseleave" [ngbPopover]="errorPopover">{{ mail.error | slice:0:20 }}</span>
}
</td>
</tr>
}
</tbody>
</table>
</div>
<div class="btn-toolbar">
<button type="button" class="btn btn-outline-secondary me-2" (click)="clearSelection()" [disabled]="selectedMailIds.size === 0" i18n>Clear</button>
<pngx-confirm-button
label="Delete selected"
i18n-label
title="Delete selected"
i18n-title
buttonClasses="btn-outline-danger"
iconName="trash"
[disabled]="selectedMailIds.size === 0"
(confirm)="deleteSelected()">
</pngx-confirm-button>
<div class="ms-auto">
<ngb-pagination
[collectionSize]="processedMails.length"
[(page)]="page"
[pageSize]="50"
[maxSize]="5"
(pageChange)="loadProcessedMails()">
</ngb-pagination>
</div>
</div>
}
</div>

View File

@@ -1,8 +0,0 @@
::ng-deep .popover {
max-width: 350px;
pre {
white-space: pre-wrap;
word-break: break-word;
}
}

View File

@@ -1,150 +0,0 @@
import { DatePipe } from '@angular/common'
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import {
HttpTestingController,
provideHttpClientTesting,
} from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { FormsModule } from '@angular/forms'
import { By } from '@angular/platform-browser'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { ToastService } from 'src/app/services/toast.service'
import { environment } from 'src/environments/environment'
import { ProcessedMailDialogComponent } from './processed-mail-dialog.component'
describe('ProcessedMailDialogComponent', () => {
let component: ProcessedMailDialogComponent
let fixture: ComponentFixture<ProcessedMailDialogComponent>
let httpTestingController: HttpTestingController
let toastService: ToastService
const rule: any = { id: 10, name: 'Mail Rule' } // minimal rule object for tests
const mails = [
{
id: 1,
rule: rule.id,
folder: 'INBOX',
uid: 111,
subject: 'A',
received: new Date().toISOString(),
processed: new Date().toISOString(),
status: 'SUCCESS',
error: null,
},
{
id: 2,
rule: rule.id,
folder: 'INBOX',
uid: 222,
subject: 'B',
received: new Date().toISOString(),
processed: new Date().toISOString(),
status: 'FAILED',
error: 'Oops',
},
]
beforeEach(async () => {
await TestBed.configureTestingModule({
imports: [
ProcessedMailDialogComponent,
FormsModule,
NgxBootstrapIconsModule.pick(allIcons),
],
providers: [
DatePipe,
NgbActiveModal,
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
}).compileComponents()
httpTestingController = TestBed.inject(HttpTestingController)
toastService = TestBed.inject(ToastService)
fixture = TestBed.createComponent(ProcessedMailDialogComponent)
component = fixture.componentInstance
component.rule = rule
})
afterEach(() => {
httpTestingController.verify()
})
function expectListRequest(ruleId: number) {
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}processed_mail/?page=1&page_size=50&ordering=-processed_at&rule=${ruleId}`
)
expect(req.request.method).toEqual('GET')
return req
}
it('should load processed mails on init', () => {
fixture.detectChanges()
const req = expectListRequest(rule.id)
req.flush({ count: 2, results: mails })
expect(component.loading).toBeFalsy()
expect(component.processedMails).toEqual(mails)
})
it('should delete selected mails and reload', () => {
fixture.detectChanges()
// initial load
const initialReq = expectListRequest(rule.id)
initialReq.flush({ count: 0, results: [] })
// select a couple of mails and delete
component.selectedMailIds.add(5)
component.selectedMailIds.add(6)
const toastInfoSpy = jest.spyOn(toastService, 'showInfo')
component.deleteSelected()
const delReq = httpTestingController.expectOne(
`${environment.apiBaseUrl}processed_mail/bulk_delete/`
)
expect(delReq.request.method).toEqual('POST')
expect(delReq.request.body).toEqual({ mail_ids: [5, 6] })
delReq.flush({})
// reload after delete
const reloadReq = expectListRequest(rule.id)
reloadReq.flush({ count: 0, results: [] })
expect(toastInfoSpy).toHaveBeenCalled()
})
it('should toggle all, toggle selected, and clear selection', () => {
fixture.detectChanges()
// initial load with two mails
const req = expectListRequest(rule.id)
req.flush({ count: 2, results: mails })
fixture.detectChanges()
// toggle all via header checkbox
const inputs = fixture.debugElement.queryAll(
By.css('input.form-check-input')
)
const header = inputs[0].nativeElement as HTMLInputElement
header.dispatchEvent(new Event('click'))
header.checked = true
header.dispatchEvent(new Event('click'))
expect(component.selectedMailIds.size).toEqual(mails.length)
// toggle a single mail
component.toggleSelected(mails[0] as any)
expect(component.selectedMailIds.has(mails[0].id)).toBeFalsy()
component.toggleSelected(mails[0] as any)
expect(component.selectedMailIds.has(mails[0].id)).toBeTruthy()
// clear selection
component.clearSelection()
expect(component.selectedMailIds.size).toEqual(0)
expect(component.toggleAllEnabled).toBeFalsy()
})
it('should close the dialog', () => {
const activeModal = TestBed.inject(NgbActiveModal)
const closeSpy = jest.spyOn(activeModal, 'close')
component.close()
expect(closeSpy).toHaveBeenCalled()
})
})

View File

@@ -1,96 +0,0 @@
import { SlicePipe } from '@angular/common'
import { Component, inject, Input, OnInit } from '@angular/core'
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
import {
NgbActiveModal,
NgbPagination,
NgbPopoverModule,
NgbTooltipModule,
} from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { ConfirmButtonComponent } from 'src/app/components/common/confirm-button/confirm-button.component'
import { MailRule } from 'src/app/data/mail-rule'
import { ProcessedMail } from 'src/app/data/processed-mail'
import { CustomDatePipe } from 'src/app/pipes/custom-date.pipe'
import { ProcessedMailService } from 'src/app/services/rest/processed-mail.service'
import { ToastService } from 'src/app/services/toast.service'
@Component({
selector: 'pngx-processed-mail-dialog',
imports: [
ConfirmButtonComponent,
CustomDatePipe,
NgbPagination,
NgbPopoverModule,
NgbTooltipModule,
NgxBootstrapIconsModule,
FormsModule,
ReactiveFormsModule,
SlicePipe,
],
templateUrl: './processed-mail-dialog.component.html',
styleUrl: './processed-mail-dialog.component.scss',
})
export class ProcessedMailDialogComponent implements OnInit {
private readonly activeModal = inject(NgbActiveModal)
private readonly processedMailService = inject(ProcessedMailService)
private readonly toastService = inject(ToastService)
public processedMails: ProcessedMail[] = []
public loading: boolean = true
public toggleAllEnabled: boolean = false
public readonly selectedMailIds: Set<number> = new Set<number>()
public page: number = 1
@Input() rule: MailRule
ngOnInit(): void {
this.loadProcessedMails()
}
public close() {
this.activeModal.close()
}
private loadProcessedMails(): void {
this.loading = true
this.clearSelection()
this.processedMailService
.list(this.page, 50, 'processed_at', true, { rule: this.rule.id })
.subscribe((result) => {
this.processedMails = result.results
this.loading = false
})
}
public deleteSelected(): void {
this.processedMailService
.bulk_delete(Array.from(this.selectedMailIds))
.subscribe(() => {
this.toastService.showInfo($localize`Processed mail(s) deleted`)
this.loadProcessedMails()
})
}
public toggleAll(event: PointerEvent) {
if ((event.target as HTMLInputElement).checked) {
this.selectedMailIds.clear()
this.processedMails.forEach((mail) => this.selectedMailIds.add(mail.id))
} else {
this.clearSelection()
}
}
public clearSelection() {
this.toggleAllEnabled = false
this.selectedMailIds.clear()
}
public toggleSelected(mail: ProcessedMail) {
this.selectedMailIds.has(mail.id)
? this.selectedMailIds.delete(mail.id)
: this.selectedMailIds.add(mail.id)
}
}

View File

@@ -1,12 +0,0 @@
import { ObjectWithId } from './object-with-id'
export interface ProcessedMail extends ObjectWithId {
rule: number // MailRule.id
folder: string
uid: number
subject: string
received: Date
processed: Date
status: string
error: string
}

View File

@@ -28,7 +28,6 @@ export enum PermissionType {
ShareLink = '%s_sharelink',
CustomField = '%s_customfield',
Workflow = '%s_workflow',
ProcessedMail = '%s_processedmail',
}
@Injectable({

View File

@@ -1,39 +0,0 @@
import { HttpTestingController } from '@angular/common/http/testing'
import { TestBed } from '@angular/core/testing'
import { Subscription } from 'rxjs'
import { environment } from 'src/environments/environment'
import { commonAbstractPaperlessServiceTests } from './abstract-paperless-service.spec'
import { ProcessedMailService } from './processed-mail.service'
let httpTestingController: HttpTestingController
let service: ProcessedMailService
let subscription: Subscription
const endpoint = 'processed_mail'
// run common tests
commonAbstractPaperlessServiceTests(endpoint, ProcessedMailService)
describe('Additional service tests for ProcessedMailService', () => {
beforeEach(() => {
// Dont need to setup again
httpTestingController = TestBed.inject(HttpTestingController)
service = TestBed.inject(ProcessedMailService)
})
afterEach(() => {
subscription?.unsubscribe()
httpTestingController.verify()
})
it('should call appropriate api endpoint for bulk delete', () => {
const ids = [1, 2, 3]
subscription = service.bulk_delete(ids).subscribe()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}${endpoint}/bulk_delete/`
)
expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({ mail_ids: ids })
req.flush({})
})
})

View File

@@ -1,19 +0,0 @@
import { Injectable } from '@angular/core'
import { ProcessedMail } from 'src/app/data/processed-mail'
import { AbstractPaperlessService } from './abstract-paperless-service'
@Injectable({
providedIn: 'root',
})
export class ProcessedMailService extends AbstractPaperlessService<ProcessedMail> {
constructor() {
super()
this.resourceName = 'processed_mail'
}
public bulk_delete(mailIds: number[]) {
return this.http.post(`${this.getResourceUrl()}bulk_delete/`, {
mail_ids: mailIds,
})
}
}

View File

@@ -51,7 +51,6 @@ import {
check,
check2All,
checkAll,
checkCircle,
checkCircleFill,
checkLg,
chevronDoubleLeft,
@@ -61,7 +60,6 @@ import {
clipboardCheck,
clipboardCheckFill,
clipboardFill,
clockHistory,
dash,
dashCircle,
diagram3,
@@ -265,7 +263,6 @@ const icons = {
check,
check2All,
checkAll,
checkCircle,
checkCircleFill,
checkLg,
chevronDoubleLeft,
@@ -275,7 +272,6 @@ const icons = {
clipboardCheck,
clipboardCheckFill,
clipboardFill,
clockHistory,
dash,
dashCircle,
diagram3,

View File

@@ -1,4 +1,6 @@
import json
from fractions import Fraction
from io import BytesIO
from pathlib import Path
from django.contrib.auth.models import User
@@ -6,6 +8,11 @@ from django.core.files.uploadedfile import SimpleUploadedFile
from rest_framework import status
from rest_framework.test import APITestCase
try:
from PIL import Image
except ModuleNotFoundError: # pragma: no cover - Pillow is required in production
Image = None # type: ignore[assignment]
from documents.tests.utils import DirectoriesMixin
from paperless.models import ApplicationConfiguration
from paperless.models import ColorConvertChoices
@@ -190,6 +197,74 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
)
self.assertFalse(Path(old_logo.path).exists())
def test_api_strips_metadata_from_logo_upload(self):
"""
GIVEN:
- An image file containing EXIF metadata including GPS coordinates
WHEN:
- Uploaded via PATCH to app config
THEN:
- Stored logo no longer contains EXIF metadata
"""
if Image is None:
self.skipTest("Pillow is not installed")
if not hasattr(Image, "Exif"):
self.skipTest("Current Pillow version cannot create EXIF metadata")
assert Image is not None
exif = Image.Exif()
exif[0x010E] = "Test description" # ImageDescription
exif[0x8825] = {
1: "N", # GPSLatitudeRef
2: (Fraction(51, 1), Fraction(30, 1), Fraction(0, 1)),
3: "E", # GPSLongitudeRef
4: (Fraction(0, 1), Fraction(7, 1), Fraction(0, 1)),
}
buffer = BytesIO()
Image.new("RGB", (8, 8), "white").save(buffer, format="JPEG", exif=exif)
buffer.seek(0)
with Image.open(BytesIO(buffer.getvalue())) as uploaded_image:
self.assertGreater(len(uploaded_image.getexif()), 0)
response = self.client.patch(
f"{self.ENDPOINT}1/",
{
"app_logo": SimpleUploadedFile(
name="with_exif.jpg",
content=buffer.getvalue(),
content_type="image/jpeg",
),
},
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config = ApplicationConfiguration.objects.first()
stored_logo = Path(config.app_logo.path)
self.assertTrue(stored_logo.exists())
with Image.open(stored_logo) as sanitized:
sanitized_exif = sanitized.getexif()
self.assertNotEqual(sanitized_exif.get(0x010E), "Test description")
gps_ifd = None
if hasattr(sanitized_exif, "get_ifd"):
try:
gps_ifd = sanitized_exif.get_ifd(0x8825)
except KeyError:
gps_ifd = None
else:
gps_ifd = sanitized_exif.get(0x8825)
if gps_ifd is not None:
self.assertEqual(len(gps_ifd), 0, "GPS metadata should be cleared")
self.assertNotIn("exif", sanitized.info)
def test_api_rejects_malicious_svg_logo(self):
"""
GIVEN:

View File

@@ -1,4 +1,5 @@
import logging
from io import BytesIO
import magic
from allauth.mfa.adapter import get_adapter as get_mfa_adapter
@@ -9,6 +10,10 @@ from allauth.socialaccount.models import SocialApp
from django.contrib.auth.models import Group
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.core.files.uploadedfile import SimpleUploadedFile
from PIL import Image
from PIL import ImageOps
from PIL import UnidentifiedImageError
from rest_framework import serializers
from rest_framework.authtoken.serializers import AuthTokenSerializer
@@ -19,6 +24,102 @@ from paperless_mail.serialisers import ObfuscatedPasswordField
logger = logging.getLogger("paperless.settings")
def strip_image_metadata(uploaded_file, mime_type: str | None):
"""Return a copy of ``uploaded_file`` with EXIF/ICC metadata removed."""
if uploaded_file is None:
return uploaded_file
original_position = uploaded_file.tell() if hasattr(uploaded_file, "tell") else None
image = None
sanitized = None
try:
if hasattr(uploaded_file, "seek"):
uploaded_file.seek(0)
image = Image.open(uploaded_file)
image.load()
except (UnidentifiedImageError, OSError):
if hasattr(uploaded_file, "seek") and original_position is not None:
uploaded_file.seek(original_position)
return uploaded_file
try:
image_format = (image.format or "").upper()
image = ImageOps.exif_transpose(image)
if image_format not in {"JPEG", "JPG", "PNG"}:
if hasattr(uploaded_file, "seek") and original_position is not None:
uploaded_file.seek(original_position)
return uploaded_file
if hasattr(image, "info"):
image.info.pop("exif", None)
image.info.pop("icc_profile", None)
image.info.pop("comment", None)
if image_format in {"JPEG", "JPG"}:
sanitized = image.convert("RGB")
save_kwargs = {
"format": "JPEG",
"quality": 95,
"subsampling": 0,
"optimize": True,
"exif": b"",
}
else: # PNG
target_mode = (
"RGBA"
if ("A" in image.mode or image.info.get("transparency"))
else "RGB"
)
sanitized = image.convert(target_mode)
save_kwargs = {
"format": "PNG",
"optimize": True,
}
buffer = BytesIO()
try:
sanitized.save(buffer, **save_kwargs)
except (OSError, ValueError):
buffer = BytesIO()
if image_format in {"JPEG", "JPG"}:
sanitized.save(
buffer,
format="JPEG",
quality=90,
subsampling=0,
exif=b"",
)
else:
sanitized.save(
buffer,
format="PNG",
)
buffer.seek(0)
if hasattr(uploaded_file, "close"):
try:
uploaded_file.close()
except Exception:
pass
content_type = getattr(uploaded_file, "content_type", None) or mime_type
return SimpleUploadedFile(
name=getattr(uploaded_file, "name", "logo"),
content=buffer.getvalue(),
content_type=content_type,
)
finally:
if sanitized is not None:
sanitized.close()
if image is not None:
image.close()
class PaperlessAuthTokenSerializer(AuthTokenSerializer):
code = serializers.CharField(
label="MFA Code",
@@ -209,9 +310,22 @@ class ApplicationConfigurationSerializer(serializers.ModelSerializer):
return super().update(instance, validated_data)
def validate_app_logo(self, file):
if file and magic.from_buffer(file.read(2048), mime=True) == "image/svg+xml":
if not file:
return file
if hasattr(file, "seek"):
file.seek(0)
mime_type = magic.from_buffer(file.read(2048), mime=True)
if hasattr(file, "seek"):
file.seek(0)
if mime_type == "image/svg+xml":
reject_dangerous_svg(file)
return file
if hasattr(file, "seek"):
file.seek(0)
return file
return strip_image_metadata(file, mime_type)
class Meta:
model = ApplicationConfiguration

View File

@@ -922,7 +922,7 @@ CELERY_ACCEPT_CONTENT = ["application/json", "application/x-python-serialize"]
CELERY_BEAT_SCHEDULE = _parse_beat_schedule()
# https://docs.celeryq.dev/en/stable/userguide/configuration.html#beat-schedule-filename
CELERY_BEAT_SCHEDULE_FILENAME = DATA_DIR / "celerybeat-schedule.db"
CELERY_BEAT_SCHEDULE_FILENAME = str(DATA_DIR / "celerybeat-schedule.db")
# Cachalot: Database read cache.

View File

@@ -57,7 +57,6 @@ from paperless.views import UserViewSet
from paperless_mail.views import MailAccountViewSet
from paperless_mail.views import MailRuleViewSet
from paperless_mail.views import OauthCallbackView
from paperless_mail.views import ProcessedMailViewSet
api_router = DefaultRouter()
api_router.register(r"correspondents", CorrespondentViewSet)
@@ -78,7 +77,6 @@ api_router.register(r"workflow_actions", WorkflowActionViewSet)
api_router.register(r"workflows", WorkflowViewSet)
api_router.register(r"custom_fields", CustomFieldViewSet)
api_router.register(r"config", ApplicationConfigurationViewSet)
api_router.register(r"processed_mail", ProcessedMailViewSet)
urlpatterns = [

View File

@@ -1,12 +0,0 @@
from django_filters import FilterSet
from paperless_mail.models import ProcessedMail
class ProcessedMailFilterSet(FilterSet):
class Meta:
model = ProcessedMail
fields = {
"rule": ["exact"],
"status": ["exact"],
}

View File

@@ -6,7 +6,6 @@ from documents.serialisers import OwnedObjectSerializer
from documents.serialisers import TagsField
from paperless_mail.models import MailAccount
from paperless_mail.models import MailRule
from paperless_mail.models import ProcessedMail
class ObfuscatedPasswordField(serializers.CharField):
@@ -131,20 +130,3 @@ class MailRuleSerializer(OwnedObjectSerializer):
if value > 36500: # ~100 years
raise serializers.ValidationError("Maximum mail age is unreasonably large.")
return value
class ProcessedMailSerializer(OwnedObjectSerializer):
class Meta:
model = ProcessedMail
fields = [
"id",
"owner",
"rule",
"folder",
"uid",
"subject",
"received",
"processed",
"status",
"error",
]

View File

@@ -3,7 +3,6 @@ from unittest import mock
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.utils import timezone
from guardian.shortcuts import assign_perm
from rest_framework import status
from rest_framework.test import APITestCase
@@ -14,7 +13,6 @@ from documents.models import Tag
from documents.tests.utils import DirectoriesMixin
from paperless_mail.models import MailAccount
from paperless_mail.models import MailRule
from paperless_mail.models import ProcessedMail
from paperless_mail.tests.test_mail import BogusMailBox
@@ -723,285 +721,3 @@ class TestAPIMailRules(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn("maximum_age", response.data)
class TestAPIProcessedMails(DirectoriesMixin, APITestCase):
ENDPOINT = "/api/processed_mail/"
def setUp(self):
super().setUp()
self.user = User.objects.create_user(username="temp_admin")
self.user.user_permissions.add(*Permission.objects.all())
self.user.save()
self.client.force_authenticate(user=self.user)
def test_get_processed_mails_owner_aware(self):
"""
GIVEN:
- Configured processed mails with different users
WHEN:
- API call is made to get processed mails
THEN:
- Only unowned, owned by user or granted processed mails are provided
"""
user2 = User.objects.create_user(username="temp_admin2")
account = MailAccount.objects.create(
name="Email1",
username="username1",
password="password1",
imap_server="server.example.com",
imap_port=443,
imap_security=MailAccount.ImapSecurity.SSL,
character_set="UTF-8",
)
rule = MailRule.objects.create(
name="Rule1",
account=account,
folder="INBOX",
filter_from="from@example.com",
order=0,
)
pm1 = ProcessedMail.objects.create(
rule=rule,
folder="INBOX",
uid="1",
subject="Subj1",
received=timezone.now(),
processed=timezone.now(),
status="SUCCESS",
error=None,
)
pm2 = ProcessedMail.objects.create(
rule=rule,
folder="INBOX",
uid="2",
subject="Subj2",
received=timezone.now(),
processed=timezone.now(),
status="FAILED",
error="err",
owner=self.user,
)
ProcessedMail.objects.create(
rule=rule,
folder="INBOX",
uid="3",
subject="Subj3",
received=timezone.now(),
processed=timezone.now(),
status="SUCCESS",
error=None,
owner=user2,
)
pm4 = ProcessedMail.objects.create(
rule=rule,
folder="INBOX",
uid="4",
subject="Subj4",
received=timezone.now(),
processed=timezone.now(),
status="SUCCESS",
error=None,
)
pm4.owner = user2
pm4.save()
assign_perm("view_processedmail", self.user, pm4)
response = self.client.get(self.ENDPOINT)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 3)
returned_ids = {r["id"] for r in response.data["results"]}
self.assertSetEqual(returned_ids, {pm1.id, pm2.id, pm4.id})
def test_get_processed_mails_filter_by_rule(self):
"""
GIVEN:
- Processed mails belonging to two different rules
WHEN:
- API call is made with rule filter
THEN:
- Only processed mails for that rule are returned
"""
account = MailAccount.objects.create(
name="Email1",
username="username1",
password="password1",
imap_server="server.example.com",
imap_port=443,
imap_security=MailAccount.ImapSecurity.SSL,
character_set="UTF-8",
)
rule1 = MailRule.objects.create(
name="Rule1",
account=account,
folder="INBOX",
filter_from="from1@example.com",
order=0,
)
rule2 = MailRule.objects.create(
name="Rule2",
account=account,
folder="INBOX",
filter_from="from2@example.com",
order=1,
)
pm1 = ProcessedMail.objects.create(
rule=rule1,
folder="INBOX",
uid="r1-1",
subject="R1-A",
received=timezone.now(),
processed=timezone.now(),
status="SUCCESS",
error=None,
owner=self.user,
)
pm2 = ProcessedMail.objects.create(
rule=rule1,
folder="INBOX",
uid="r1-2",
subject="R1-B",
received=timezone.now(),
processed=timezone.now(),
status="FAILED",
error="e",
)
ProcessedMail.objects.create(
rule=rule2,
folder="INBOX",
uid="r2-1",
subject="R2-A",
received=timezone.now(),
processed=timezone.now(),
status="SUCCESS",
error=None,
)
response = self.client.get(f"{self.ENDPOINT}?rule={rule1.pk}")
self.assertEqual(response.status_code, status.HTTP_200_OK)
returned_ids = {r["id"] for r in response.data["results"]}
self.assertSetEqual(returned_ids, {pm1.id, pm2.id})
def test_bulk_delete_processed_mails(self):
"""
GIVEN:
- Processed mails belonging to two different rules and different users
WHEN:
- API call is made to bulk delete some of the processed mails
THEN:
- Only the specified processed mails are deleted, respecting ownership and permissions
"""
user2 = User.objects.create_user(username="temp_admin2")
account = MailAccount.objects.create(
name="Email1",
username="username1",
password="password1",
imap_server="server.example.com",
imap_port=443,
imap_security=MailAccount.ImapSecurity.SSL,
character_set="UTF-8",
)
rule = MailRule.objects.create(
name="Rule1",
account=account,
folder="INBOX",
filter_from="from@example.com",
order=0,
)
# unowned and owned by self, and one with explicit object perm
pm_unowned = ProcessedMail.objects.create(
rule=rule,
folder="INBOX",
uid="u1",
subject="Unowned",
received=timezone.now(),
processed=timezone.now(),
status="SUCCESS",
error=None,
)
pm_owned = ProcessedMail.objects.create(
rule=rule,
folder="INBOX",
uid="u2",
subject="Owned",
received=timezone.now(),
processed=timezone.now(),
status="FAILED",
error="e",
owner=self.user,
)
pm_granted = ProcessedMail.objects.create(
rule=rule,
folder="INBOX",
uid="u3",
subject="Granted",
received=timezone.now(),
processed=timezone.now(),
status="SUCCESS",
error=None,
owner=user2,
)
assign_perm("delete_processedmail", self.user, pm_granted)
pm_forbidden = ProcessedMail.objects.create(
rule=rule,
folder="INBOX",
uid="u4",
subject="Forbidden",
received=timezone.now(),
processed=timezone.now(),
status="SUCCESS",
error=None,
owner=user2,
)
# Success for allowed items
response = self.client.post(
f"{self.ENDPOINT}bulk_delete/",
data={
"mail_ids": [pm_unowned.id, pm_owned.id, pm_granted.id],
},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["result"], "OK")
self.assertSetEqual(
set(response.data["deleted_mail_ids"]),
{pm_unowned.id, pm_owned.id, pm_granted.id},
)
self.assertFalse(ProcessedMail.objects.filter(id=pm_unowned.id).exists())
self.assertFalse(ProcessedMail.objects.filter(id=pm_owned.id).exists())
self.assertFalse(ProcessedMail.objects.filter(id=pm_granted.id).exists())
self.assertTrue(ProcessedMail.objects.filter(id=pm_forbidden.id).exists())
# 403 and not deleted
response = self.client.post(
f"{self.ENDPOINT}bulk_delete/",
data={
"mail_ids": [pm_forbidden.id],
},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
self.assertTrue(ProcessedMail.objects.filter(id=pm_forbidden.id).exists())
# missing mail_ids
response = self.client.post(
f"{self.ENDPOINT}bulk_delete/",
data={"mail_ids": "not-a-list"},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)

View File

@@ -3,10 +3,8 @@ import logging
from datetime import timedelta
from django.http import HttpResponseBadRequest
from django.http import HttpResponseForbidden
from django.http import HttpResponseRedirect
from django.utils import timezone
from django_filters.rest_framework import DjangoFilterBackend
from drf_spectacular.types import OpenApiTypes
from drf_spectacular.utils import extend_schema
from drf_spectacular.utils import extend_schema_view
@@ -14,29 +12,23 @@ from drf_spectacular.utils import inline_serializer
from httpx_oauth.oauth2 import GetAccessTokenError
from rest_framework import serializers
from rest_framework.decorators import action
from rest_framework.filters import OrderingFilter
from rest_framework.generics import GenericAPIView
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from rest_framework.viewsets import ModelViewSet
from rest_framework.viewsets import ReadOnlyModelViewSet
from documents.filters import ObjectOwnedOrGrantedPermissionsFilter
from documents.permissions import PaperlessObjectPermissions
from documents.permissions import has_perms_owner_aware
from documents.views import PassUserMixin
from paperless.views import StandardPagination
from paperless_mail.filters import ProcessedMailFilterSet
from paperless_mail.mail import MailError
from paperless_mail.mail import get_mailbox
from paperless_mail.mail import mailbox_login
from paperless_mail.models import MailAccount
from paperless_mail.models import MailRule
from paperless_mail.models import ProcessedMail
from paperless_mail.oauth import PaperlessMailOAuth2Manager
from paperless_mail.serialisers import MailAccountSerializer
from paperless_mail.serialisers import MailRuleSerializer
from paperless_mail.serialisers import ProcessedMailSerializer
from paperless_mail.tasks import process_mail_accounts
@@ -134,34 +126,6 @@ class MailAccountViewSet(ModelViewSet, PassUserMixin):
return Response({"result": "OK"})
class ProcessedMailViewSet(ReadOnlyModelViewSet, PassUserMixin):
permission_classes = (IsAuthenticated, PaperlessObjectPermissions)
serializer_class = ProcessedMailSerializer
pagination_class = StandardPagination
filter_backends = (
DjangoFilterBackend,
OrderingFilter,
ObjectOwnedOrGrantedPermissionsFilter,
)
filterset_class = ProcessedMailFilterSet
queryset = ProcessedMail.objects.all().order_by("-processed")
@action(methods=["post"], detail=False)
def bulk_delete(self, request):
mail_ids = request.data.get("mail_ids", [])
if not isinstance(mail_ids, list) or not all(
isinstance(i, int) for i in mail_ids
):
return HttpResponseBadRequest("mail_ids must be a list of integers")
mails = ProcessedMail.objects.filter(id__in=mail_ids)
for mail in mails:
if not has_perms_owner_aware(request.user, "delete_processedmail", mail):
return HttpResponseForbidden("Insufficient permissions")
mail.delete()
return Response({"result": "OK", "deleted_mail_ids": mail_ids})
class MailRuleViewSet(ModelViewSet, PassUserMixin):
model = MailRule