mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2025-08-18 00:46:25 +00:00
Compare commits
25 Commits
v2.18.0
...
feature-ai
Author | SHA1 | Date | |
---|---|---|---|
![]() |
9f3946d938 | ||
![]() |
a3eed49638 | ||
![]() |
e14f508327 | ||
![]() |
3cbea6cc51 | ||
![]() |
5cd54f833a | ||
![]() |
a59fe0cb3c | ||
![]() |
3fc9992f4d | ||
![]() |
778f6c8162 | ||
![]() |
ddd2428d9c | ||
![]() |
c872e50739 | ||
![]() |
b938d0aeba | ||
![]() |
dd78c5d496 | ||
![]() |
131ae28794 | ||
![]() |
16d7d95517 | ||
![]() |
53c0b6e1c7 | ||
![]() |
168ad3f9a2 | ||
![]() |
f9ffc97970 | ||
![]() |
89671eb303 | ||
![]() |
7f37832ea0 | ||
![]() |
d31b4ced63 | ||
![]() |
e3f0a37f91 | ||
![]() |
8c0a61dbc6 | ||
![]() |
f3c7c95c69 | ||
![]() |
7e3ec32580 | ||
![]() |
84da2ce145 |
@@ -1700,3 +1700,48 @@ password. All of these options come from their similarly-named [Django settings]
|
|||||||
#### [`PAPERLESS_EMAIL_USE_SSL=<bool>`](#PAPERLESS_EMAIL_USE_SSL) {#PAPERLESS_EMAIL_USE_SSL}
|
#### [`PAPERLESS_EMAIL_USE_SSL=<bool>`](#PAPERLESS_EMAIL_USE_SSL) {#PAPERLESS_EMAIL_USE_SSL}
|
||||||
|
|
||||||
: Defaults to false.
|
: Defaults to false.
|
||||||
|
|
||||||
|
## AI {#ai}
|
||||||
|
|
||||||
|
#### [`PAPERLESS_ENABLE_AI=<bool>`](#PAPERLESS_ENABLE_AI) {#PAPERLESS_ENABLE_AI}
|
||||||
|
|
||||||
|
: Enables the AI features in Paperless. This includes the AI-based
|
||||||
|
suggestions. This setting is required to be set to true in order to use the AI features.
|
||||||
|
|
||||||
|
Defaults to false.
|
||||||
|
|
||||||
|
#### [`PAPERLESS_AI_BACKEND=<str>`](#PAPERLESS_AI_BACKEND) {#PAPERLESS_AI_BACKEND}
|
||||||
|
|
||||||
|
: The AI backend to use. This can be either "openai" or "ollama". If set to "ollama", the AI
|
||||||
|
features will be run locally on your machine. If set to "openai", the AI features will be run
|
||||||
|
using the OpenAI API. This setting is required to be set to use the AI features.
|
||||||
|
|
||||||
|
Defaults to None.
|
||||||
|
|
||||||
|
!!! note
|
||||||
|
|
||||||
|
The OpenAI API is a paid service. You will need to set up an OpenAI account and
|
||||||
|
will be charged for usage incurred by Paperless-ngx features and your document data
|
||||||
|
will (of course) be shared with OpenAI. Paperless-ngx does not endorse the use of the
|
||||||
|
OpenAI API in any way.
|
||||||
|
|
||||||
|
Refer to the OpenAI terms of service, and use at your own risk.
|
||||||
|
|
||||||
|
#### [`PAPERLESS_LLM_MODEL=<str>`](#PAPERLESS_LLM_MODEL) {#PAPERLESS_LLM_MODEL}
|
||||||
|
|
||||||
|
: The model to use for the AI backend, i.e. "gpt-3.5-turbo", "gpt-4" or any of the models supported by the
|
||||||
|
current backend. This setting is required to be set to use the AI features.
|
||||||
|
|
||||||
|
Defaults to None.
|
||||||
|
|
||||||
|
#### [`PAPERLESS_LLM_API_KEY=<str>`](#PAPERLESS_LLM_API_KEY) {#PAPERLESS_LLM_API_KEY}
|
||||||
|
|
||||||
|
: The API key to use for the AI backend. This is required for the OpenAI backend only.
|
||||||
|
|
||||||
|
Defaults to None.
|
||||||
|
|
||||||
|
#### [`PAPERLESS_LLM_URL=<str>`](#PAPERLESS_LLM_URL) {#PAPERLESS_LLM_URL}
|
||||||
|
|
||||||
|
: The URL to use for the AI backend. This is required for the Ollama backend only.
|
||||||
|
|
||||||
|
Defaults to None.
|
||||||
|
@@ -25,11 +25,12 @@ physical documents into a searchable online archive so you can keep, well, _less
|
|||||||
## Features
|
## Features
|
||||||
|
|
||||||
- **Organize and index** your scanned documents with tags, correspondents, types, and more.
|
- **Organize and index** your scanned documents with tags, correspondents, types, and more.
|
||||||
- _Your_ data is stored locally on _your_ server and is never transmitted or shared in any way.
|
- _Your_ data is stored locally on _your_ server and is never transmitted or shared in any way, unless you explicitly choose to do so.
|
||||||
- Performs **OCR** on your documents, adding searchable and selectable text, even to documents scanned with only images.
|
- Performs **OCR** on your documents, adding searchable and selectable text, even to documents scanned with only images.
|
||||||
- Utilizes the open-source Tesseract engine to recognize more than 100 languages.
|
- Utilizes the open-source Tesseract engine to recognize more than 100 languages.
|
||||||
- Documents are saved as PDF/A format which is designed for long term storage, alongside the unaltered originals.
|
- Documents are saved as PDF/A format which is designed for long term storage, alongside the unaltered originals.
|
||||||
- Uses machine-learning to automatically add tags, correspondents and document types to your documents.
|
- Uses machine-learning to automatically add tags, correspondents and document types to your documents.
|
||||||
|
- **New**: Paperless-ngx can now leverage AI (Large Language Models or LLMs) for document suggestions. This is an optional feature that can be enabled (and is disabled by default).
|
||||||
- Supports PDF documents, images, plain text files, Office documents (Word, Excel, Powerpoint, and LibreOffice equivalents)[^1] and more.
|
- Supports PDF documents, images, plain text files, Office documents (Word, Excel, Powerpoint, and LibreOffice equivalents)[^1] and more.
|
||||||
- Paperless stores your documents plain on disk. Filenames and folders are managed by paperless and their format can be configured freely with different configurations assigned to different documents.
|
- Paperless stores your documents plain on disk. Filenames and folders are managed by paperless and their format can be configured freely with different configurations assigned to different documents.
|
||||||
- **Beautiful, modern web application** that features:
|
- **Beautiful, modern web application** that features:
|
||||||
|
@@ -260,6 +260,14 @@ Once setup, navigating to the email settings page in Paperless-ngx will allow yo
|
|||||||
You can also submit a document using the REST API, see [POSTing documents](api.md#file-uploads)
|
You can also submit a document using the REST API, see [POSTing documents](api.md#file-uploads)
|
||||||
for details.
|
for details.
|
||||||
|
|
||||||
|
## Document Suggestions
|
||||||
|
|
||||||
|
Paperless-ngx can suggest tags, correspondents, document types and storage paths for documents based on the content of the document. This is done using a machine learning model that is trained on the documents in your database. The suggestions are shown in the document detail page and can be accepted or rejected by the user.
|
||||||
|
|
||||||
|
### AI-Enhanced Suggestions
|
||||||
|
|
||||||
|
If enabled, Paperless-ngx can use an AI LLM model to suggest document titles, dates, tags, correspondents and document types for documents. This feature will always be "opt-in" and does not disable the existing suggestion system. Currently, both remote (via the OpenAI API) and local (via Ollama) models are supported, see [configuration](configuration.md#ai) for details.
|
||||||
|
|
||||||
## Sharing documents from Paperless-ngx
|
## Sharing documents from Paperless-ngx
|
||||||
|
|
||||||
Paperless-ngx supports sharing documents with other users by assigning them [permissions](#object-permissions)
|
Paperless-ngx supports sharing documents with other users by assigning them [permissions](#object-permissions)
|
||||||
|
@@ -35,6 +35,7 @@
|
|||||||
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
|
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
|
||||||
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
|
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
|
||||||
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
|
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
|
||||||
|
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
|
||||||
}
|
}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
@@ -29,6 +29,7 @@ import { SettingsService } from 'src/app/services/settings.service'
|
|||||||
import { ToastService } from 'src/app/services/toast.service'
|
import { ToastService } from 'src/app/services/toast.service'
|
||||||
import { FileComponent } from '../../common/input/file/file.component'
|
import { FileComponent } from '../../common/input/file/file.component'
|
||||||
import { NumberComponent } from '../../common/input/number/number.component'
|
import { NumberComponent } from '../../common/input/number/number.component'
|
||||||
|
import { PasswordComponent } from '../../common/input/password/password.component'
|
||||||
import { SelectComponent } from '../../common/input/select/select.component'
|
import { SelectComponent } from '../../common/input/select/select.component'
|
||||||
import { SwitchComponent } from '../../common/input/switch/switch.component'
|
import { SwitchComponent } from '../../common/input/switch/switch.component'
|
||||||
import { TextComponent } from '../../common/input/text/text.component'
|
import { TextComponent } from '../../common/input/text/text.component'
|
||||||
@@ -46,6 +47,7 @@ import { LoadingComponentWithPermissions } from '../../loading-component/loading
|
|||||||
TextComponent,
|
TextComponent,
|
||||||
NumberComponent,
|
NumberComponent,
|
||||||
FileComponent,
|
FileComponent,
|
||||||
|
PasswordComponent,
|
||||||
AsyncPipe,
|
AsyncPipe,
|
||||||
NgbNavModule,
|
NgbNavModule,
|
||||||
FormsModule,
|
FormsModule,
|
||||||
|
@@ -1,7 +1,7 @@
|
|||||||
<div ngbDropdown #fieldDropdown="ngbDropdown" (openChange)="onOpenClose($event)" [popperOptions]="popperOptions" placement="bottom-end">
|
<div ngbDropdown #fieldDropdown="ngbDropdown" (openChange)="onOpenClose($event)" [popperOptions]="popperOptions">
|
||||||
<button class="btn btn-sm btn-outline-primary" id="customFieldsDropdown" [disabled]="disabled" ngbDropdownToggle>
|
<button type="button" class="btn btn-sm btn-outline-primary" id="customFieldsDropdown" [disabled]="disabled" ngbDropdownToggle>
|
||||||
<i-bs name="ui-radios"></i-bs>
|
<i-bs name="ui-radios"></i-bs>
|
||||||
<div class="d-none d-sm-inline"> <ng-container i18n>Custom Fields</ng-container></div>
|
<div class="d-none d-lg-inline"> <ng-container i18n>Custom Fields</ng-container></div>
|
||||||
</button>
|
</button>
|
||||||
<div ngbDropdownMenu aria-labelledby="customFieldsDropdown" class="shadow custom-fields-dropdown">
|
<div ngbDropdownMenu aria-labelledby="customFieldsDropdown" class="shadow custom-fields-dropdown">
|
||||||
<div class="list-group list-group-flush" (keydown)="listKeyDown($event)">
|
<div class="list-group list-group-flush" (keydown)="listKeyDown($event)">
|
||||||
|
@@ -1,17 +1,24 @@
|
|||||||
<div class="mb-3">
|
<div class="mb-3" [class.pb-3]="error">
|
||||||
<label class="form-label" [for]="inputId">{{title}}</label>
|
<div class="row">
|
||||||
<div class="input-group" [class.is-invalid]="error">
|
<div class="d-flex align-items-center position-relative hidden-button-container" [class.col-md-3]="horizontal">
|
||||||
<input #inputField [type]="showReveal && textVisible ? 'text' : 'password'" class="form-control" [class.is-invalid]="error" [id]="inputId" [(ngModel)]="value" (focus)="onFocus()" (focusout)="onFocusOut()" (change)="onChange(value)" [disabled]="disabled" [autocomplete]="autocomplete">
|
@if (title) {
|
||||||
@if (showReveal) {
|
<label class="form-label" [class.mb-md-0]="horizontal" [for]="inputId">{{title}}</label>
|
||||||
<button type="button" class="btn btn-outline-secondary" (click)="toggleVisibility()" i18n-title title="Show password" [disabled]="disabled || disableRevealToggle">
|
}
|
||||||
<i-bs name="eye"></i-bs>
|
</div>
|
||||||
</button>
|
<div class="position-relative" [class.col-md-9]="horizontal">
|
||||||
|
<div class="input-group" [class.is-invalid]="error">
|
||||||
|
<input #inputField [type]="showReveal && textVisible ? 'text' : 'password'" class="form-control" [class.is-invalid]="error" [id]="inputId" [(ngModel)]="value" (focus)="onFocus()" (focusout)="onFocusOut()" (change)="onChange(value)" [disabled]="disabled" [autocomplete]="autocomplete">
|
||||||
|
@if (showReveal) {
|
||||||
|
<button type="button" class="btn btn-outline-secondary" (click)="toggleVisibility()" i18n-title title="Show password" [disabled]="disabled || disableRevealToggle">
|
||||||
|
<i-bs name="eye"></i-bs>
|
||||||
|
</button>
|
||||||
|
}
|
||||||
|
</div>
|
||||||
|
<div class="invalid-feedback">
|
||||||
|
{{error}}
|
||||||
|
</div>
|
||||||
|
@if (hint) {
|
||||||
|
<small class="form-text text-muted" [innerHTML]="hint | safeHtml"></small>
|
||||||
}
|
}
|
||||||
</div>
|
</div>
|
||||||
<div class="invalid-feedback">
|
|
||||||
{{error}}
|
|
||||||
</div>
|
|
||||||
@if (hint) {
|
|
||||||
<small class="form-text text-muted" [innerHTML]="hint | safeHtml"></small>
|
|
||||||
}
|
|
||||||
</div>
|
</div>
|
||||||
|
@@ -15,6 +15,12 @@
|
|||||||
@if (hint) {
|
@if (hint) {
|
||||||
<small class="form-text text-muted" [innerHTML]="hint | safeHtml"></small>
|
<small class="form-text text-muted" [innerHTML]="hint | safeHtml"></small>
|
||||||
}
|
}
|
||||||
|
@if (getSuggestion()?.length > 0) {
|
||||||
|
<small>
|
||||||
|
<span i18n>Suggestion:</span>
|
||||||
|
<a (click)="applySuggestion(s)" [routerLink]="[]">{{getSuggestion()}}</a>
|
||||||
|
</small>
|
||||||
|
}
|
||||||
<div class="invalid-feedback position-absolute top-100">
|
<div class="invalid-feedback position-absolute top-100">
|
||||||
{{error}}
|
{{error}}
|
||||||
</div>
|
</div>
|
||||||
|
@@ -26,10 +26,20 @@ describe('TextComponent', () => {
|
|||||||
|
|
||||||
it('should support use of input field', () => {
|
it('should support use of input field', () => {
|
||||||
expect(component.value).toBeUndefined()
|
expect(component.value).toBeUndefined()
|
||||||
// TODO: why doesn't this work?
|
input.value = 'foo'
|
||||||
// input.value = 'foo'
|
input.dispatchEvent(new Event('input'))
|
||||||
// input.dispatchEvent(new Event('change'))
|
fixture.detectChanges()
|
||||||
// fixture.detectChanges()
|
expect(component.value).toBe('foo')
|
||||||
// expect(component.value).toEqual('foo')
|
})
|
||||||
|
|
||||||
|
it('should support suggestion', () => {
|
||||||
|
component.value = 'foo'
|
||||||
|
component.suggestion = 'foo'
|
||||||
|
expect(component.getSuggestion()).toBe('')
|
||||||
|
component.value = 'bar'
|
||||||
|
expect(component.getSuggestion()).toBe('foo')
|
||||||
|
component.applySuggestion()
|
||||||
|
fixture.detectChanges()
|
||||||
|
expect(component.value).toBe('foo')
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
@@ -4,6 +4,7 @@ import {
|
|||||||
NG_VALUE_ACCESSOR,
|
NG_VALUE_ACCESSOR,
|
||||||
ReactiveFormsModule,
|
ReactiveFormsModule,
|
||||||
} from '@angular/forms'
|
} from '@angular/forms'
|
||||||
|
import { RouterLink } from '@angular/router'
|
||||||
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
|
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
|
||||||
import { SafeHtmlPipe } from 'src/app/pipes/safehtml.pipe'
|
import { SafeHtmlPipe } from 'src/app/pipes/safehtml.pipe'
|
||||||
import { AbstractInputComponent } from '../abstract-input'
|
import { AbstractInputComponent } from '../abstract-input'
|
||||||
@@ -24,6 +25,7 @@ import { AbstractInputComponent } from '../abstract-input'
|
|||||||
ReactiveFormsModule,
|
ReactiveFormsModule,
|
||||||
SafeHtmlPipe,
|
SafeHtmlPipe,
|
||||||
NgxBootstrapIconsModule,
|
NgxBootstrapIconsModule,
|
||||||
|
RouterLink,
|
||||||
],
|
],
|
||||||
})
|
})
|
||||||
export class TextComponent extends AbstractInputComponent<string> {
|
export class TextComponent extends AbstractInputComponent<string> {
|
||||||
@@ -33,7 +35,19 @@ export class TextComponent extends AbstractInputComponent<string> {
|
|||||||
@Input()
|
@Input()
|
||||||
placeholder: string = ''
|
placeholder: string = ''
|
||||||
|
|
||||||
|
@Input()
|
||||||
|
suggestion: string = ''
|
||||||
|
|
||||||
constructor() {
|
constructor() {
|
||||||
super()
|
super()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
getSuggestion() {
|
||||||
|
return this.value !== this.suggestion ? this.suggestion : ''
|
||||||
|
}
|
||||||
|
|
||||||
|
applySuggestion() {
|
||||||
|
this.value = this.suggestion
|
||||||
|
this.onChange(this.value)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
@@ -0,0 +1,49 @@
|
|||||||
|
<div class="btn-group">
|
||||||
|
<button type="button" class="btn btn-sm btn-outline-primary" (click)="clickSuggest()" [disabled]="loading || (suggestions && !aiEnabled)">
|
||||||
|
@if (loading) {
|
||||||
|
<div class="spinner-border spinner-border-sm" role="status"></div>
|
||||||
|
} @else {
|
||||||
|
<i-bs width="1.2em" height="1.2em" name="stars"></i-bs>
|
||||||
|
}
|
||||||
|
<span class="d-none d-lg-inline ps-1" i18n>Suggest</span>
|
||||||
|
@if (totalSuggestions > 0) {
|
||||||
|
<span class="badge bg-primary ms-2">{{ totalSuggestions }}</span>
|
||||||
|
}
|
||||||
|
</button>
|
||||||
|
|
||||||
|
@if (aiEnabled) {
|
||||||
|
<div class="btn-group" ngbDropdown #dropdown="ngbDropdown" [popperOptions]="popperOptions">
|
||||||
|
<button type="button" class="btn btn-sm btn-outline-primary" ngbDropdownToggle [disabled]="loading || !suggestions" aria-expanded="false" aria-controls="suggestionsDropdown" aria-label="Suggestions dropdown">
|
||||||
|
<span class="visually-hidden" i18n>Show suggestions</span>
|
||||||
|
</button>
|
||||||
|
|
||||||
|
<div ngbDropdownMenu aria-labelledby="suggestionsDropdown" class="shadow suggestions-dropdown">
|
||||||
|
<div class="list-group list-group-flush small pb-0">
|
||||||
|
@if (!suggestions?.suggested_tags && !suggestions?.suggested_document_types && !suggestions?.suggested_correspondents) {
|
||||||
|
<div class="list-group-item text-muted fst-italic">
|
||||||
|
<small class="text-muted small fst-italic" i18n>No novel suggestions</small>
|
||||||
|
</div>
|
||||||
|
}
|
||||||
|
@if (suggestions?.suggested_tags.length > 0) {
|
||||||
|
<small class="list-group-item text-uppercase text-muted small">Tags</small>
|
||||||
|
@for (tag of suggestions.suggested_tags; track tag) {
|
||||||
|
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addTag.emit(tag)" i18n>{{ tag }}</button>
|
||||||
|
}
|
||||||
|
}
|
||||||
|
@if (suggestions?.suggested_document_types.length > 0) {
|
||||||
|
<div class="list-group-item text-uppercase text-muted small">Document Types</div>
|
||||||
|
@for (type of suggestions.suggested_document_types; track type) {
|
||||||
|
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addDocumentType.emit(type)" i18n>{{ type }}</button>
|
||||||
|
}
|
||||||
|
}
|
||||||
|
@if (suggestions?.suggested_correspondents.length > 0) {
|
||||||
|
<div class="list-group-item text-uppercase text-muted small">Correspondents</div>
|
||||||
|
@for (correspondent of suggestions.suggested_correspondents; track correspondent) {
|
||||||
|
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addCorrespondent.emit(correspondent)" i18n>{{ correspondent }}</button>
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
}
|
||||||
|
</div>
|
@@ -0,0 +1,3 @@
|
|||||||
|
.suggestions-dropdown {
|
||||||
|
min-width: 250px;
|
||||||
|
}
|
@@ -0,0 +1,51 @@
|
|||||||
|
import { ComponentFixture, TestBed } from '@angular/core/testing'
|
||||||
|
import { NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
|
||||||
|
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
|
||||||
|
import { SuggestionsDropdownComponent } from './suggestions-dropdown.component'
|
||||||
|
|
||||||
|
describe('SuggestionsDropdownComponent', () => {
|
||||||
|
let component: SuggestionsDropdownComponent
|
||||||
|
let fixture: ComponentFixture<SuggestionsDropdownComponent>
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
TestBed.configureTestingModule({
|
||||||
|
imports: [
|
||||||
|
NgbDropdownModule,
|
||||||
|
NgxBootstrapIconsModule.pick(allIcons),
|
||||||
|
SuggestionsDropdownComponent,
|
||||||
|
],
|
||||||
|
providers: [],
|
||||||
|
})
|
||||||
|
fixture = TestBed.createComponent(SuggestionsDropdownComponent)
|
||||||
|
component = fixture.componentInstance
|
||||||
|
fixture.detectChanges()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should calculate totalSuggestions', () => {
|
||||||
|
component.suggestions = {
|
||||||
|
suggested_correspondents: ['John Doe'],
|
||||||
|
suggested_tags: ['Tag1', 'Tag2'],
|
||||||
|
suggested_document_types: ['Type1'],
|
||||||
|
}
|
||||||
|
expect(component.totalSuggestions).toBe(4)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should emit getSuggestions when clickSuggest is called and suggestions are null', () => {
|
||||||
|
jest.spyOn(component.getSuggestions, 'emit')
|
||||||
|
component.suggestions = null
|
||||||
|
component.clickSuggest()
|
||||||
|
expect(component.getSuggestions.emit).toHaveBeenCalled()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should toggle dropdown when clickSuggest is called and suggestions are not null', () => {
|
||||||
|
component.aiEnabled = true
|
||||||
|
fixture.detectChanges()
|
||||||
|
component.suggestions = {
|
||||||
|
suggested_correspondents: [],
|
||||||
|
suggested_tags: [],
|
||||||
|
suggested_document_types: [],
|
||||||
|
}
|
||||||
|
component.clickSuggest()
|
||||||
|
expect(component.dropdown.open).toBeTruthy()
|
||||||
|
})
|
||||||
|
})
|
@@ -0,0 +1,64 @@
|
|||||||
|
import {
|
||||||
|
Component,
|
||||||
|
EventEmitter,
|
||||||
|
Input,
|
||||||
|
Output,
|
||||||
|
ViewChild,
|
||||||
|
} from '@angular/core'
|
||||||
|
import { NgbDropdown, NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
|
||||||
|
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
|
||||||
|
import { DocumentSuggestions } from 'src/app/data/document-suggestions'
|
||||||
|
import { pngxPopperOptions } from 'src/app/utils/popper-options'
|
||||||
|
|
||||||
|
@Component({
|
||||||
|
selector: 'pngx-suggestions-dropdown',
|
||||||
|
imports: [NgbDropdownModule, NgxBootstrapIconsModule],
|
||||||
|
templateUrl: './suggestions-dropdown.component.html',
|
||||||
|
styleUrl: './suggestions-dropdown.component.scss',
|
||||||
|
})
|
||||||
|
export class SuggestionsDropdownComponent {
|
||||||
|
public popperOptions = pngxPopperOptions
|
||||||
|
|
||||||
|
@ViewChild('dropdown') dropdown: NgbDropdown
|
||||||
|
|
||||||
|
@Input()
|
||||||
|
suggestions: DocumentSuggestions = null
|
||||||
|
|
||||||
|
@Input()
|
||||||
|
aiEnabled: boolean = false
|
||||||
|
|
||||||
|
@Input()
|
||||||
|
loading: boolean = false
|
||||||
|
|
||||||
|
@Input()
|
||||||
|
disabled: boolean = false
|
||||||
|
|
||||||
|
@Output()
|
||||||
|
getSuggestions: EventEmitter<SuggestionsDropdownComponent> =
|
||||||
|
new EventEmitter()
|
||||||
|
|
||||||
|
@Output()
|
||||||
|
addTag: EventEmitter<string> = new EventEmitter()
|
||||||
|
|
||||||
|
@Output()
|
||||||
|
addDocumentType: EventEmitter<string> = new EventEmitter()
|
||||||
|
|
||||||
|
@Output()
|
||||||
|
addCorrespondent: EventEmitter<string> = new EventEmitter()
|
||||||
|
|
||||||
|
public clickSuggest(): void {
|
||||||
|
if (!this.suggestions) {
|
||||||
|
this.getSuggestions.emit(this)
|
||||||
|
} else {
|
||||||
|
this.dropdown?.toggle()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
get totalSuggestions(): number {
|
||||||
|
return (
|
||||||
|
this.suggestions?.suggested_correspondents?.length +
|
||||||
|
this.suggestions?.suggested_tags?.length +
|
||||||
|
this.suggestions?.suggested_document_types?.length || 0
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
@@ -72,16 +72,6 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<pngx-custom-fields-dropdown
|
|
||||||
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
|
|
||||||
[documentId]="documentId"
|
|
||||||
[disabled]="!userCanEdit"
|
|
||||||
[existingFields]="document?.custom_fields"
|
|
||||||
(created)="refreshCustomFields()"
|
|
||||||
(added)="addField($event)">
|
|
||||||
</pngx-custom-fields-dropdown>
|
|
||||||
|
|
||||||
|
|
||||||
<div class="ms-auto" ngbDropdown>
|
<div class="ms-auto" ngbDropdown>
|
||||||
<button class="btn btn-sm btn-outline-primary" id="sendDropdown" ngbDropdownToggle>
|
<button class="btn btn-sm btn-outline-primary" id="sendDropdown" ngbDropdownToggle>
|
||||||
<i-bs name="send"></i-bs>
|
<i-bs name="send"></i-bs>
|
||||||
@@ -102,7 +92,7 @@
|
|||||||
</pngx-page-header>
|
</pngx-page-header>
|
||||||
|
|
||||||
<div class="row">
|
<div class="row">
|
||||||
<div class="col-md-6 col-xl-4 mb-4">
|
<div class="col-md-6 col-xl-5 mb-4">
|
||||||
|
|
||||||
<form [formGroup]='documentForm' (ngSubmit)="save()">
|
<form [formGroup]='documentForm' (ngSubmit)="save()">
|
||||||
|
|
||||||
@@ -119,6 +109,32 @@
|
|||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
||||||
|
<div class="btn-group pb-3 ms-auto">
|
||||||
|
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
|
||||||
|
[disabled]="!userCanEdit || suggestionsLoading"
|
||||||
|
[loading]="suggestionsLoading"
|
||||||
|
[suggestions]="suggestions"
|
||||||
|
[aiEnabled]="aiEnabled"
|
||||||
|
(getSuggestions)="getSuggestions()"
|
||||||
|
(addTag)="createTag($event)"
|
||||||
|
(addDocumentType)="createDocumentType($event)"
|
||||||
|
(addCorrespondent)="createCorrespondent($event)">
|
||||||
|
</pngx-suggestions-dropdown>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="btn-group pb-3 ms-2">
|
||||||
|
<pngx-custom-fields-dropdown
|
||||||
|
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
|
||||||
|
[documentId]="documentId"
|
||||||
|
[disabled]="!userCanEdit"
|
||||||
|
[existingFields]="document?.custom_fields"
|
||||||
|
(created)="refreshCustomFields()"
|
||||||
|
(added)="addField($event)">
|
||||||
|
</pngx-custom-fields-dropdown>
|
||||||
|
</div>
|
||||||
|
</ng-container>
|
||||||
|
|
||||||
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
@@ -127,7 +143,7 @@
|
|||||||
<a ngbNavLink i18n>Details</a>
|
<a ngbNavLink i18n>Details</a>
|
||||||
<ng-template ngbNavContent>
|
<ng-template ngbNavContent>
|
||||||
<div>
|
<div>
|
||||||
<pngx-input-text #inputTitle i18n-title title="Title" formControlName="title" [horizontal]="true" (keyup)="titleKeyUp($event)" [error]="error?.title"></pngx-input-text>
|
<pngx-input-text #inputTitle i18n-title title="Title" formControlName="title" [horizontal]="true" [suggestion]="suggestions?.title" (keyup)="titleKeyUp($event)" [error]="error?.title"></pngx-input-text>
|
||||||
<pngx-input-number i18n-title title="Archive serial number" [error]="error?.archive_serial_number" [horizontal]="true" formControlName='archive_serial_number'></pngx-input-number>
|
<pngx-input-number i18n-title title="Archive serial number" [error]="error?.archive_serial_number" [horizontal]="true" formControlName='archive_serial_number'></pngx-input-number>
|
||||||
<pngx-input-date i18n-title title="Date created" formControlName="created_date" [suggestions]="suggestions?.dates" [showFilter]="true" [horizontal]="true" (filterDocuments)="filterDocuments($event)"
|
<pngx-input-date i18n-title title="Date created" formControlName="created_date" [suggestions]="suggestions?.dates" [showFilter]="true" [horizontal]="true" (filterDocuments)="filterDocuments($event)"
|
||||||
[error]="error?.created_date"></pngx-input-date>
|
[error]="error?.created_date"></pngx-input-date>
|
||||||
@@ -137,7 +153,7 @@
|
|||||||
(createNew)="createDocumentType($event)" [hideAddButton]="createDisabled(DataType.DocumentType)" [suggestions]="suggestions?.document_types" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.DocumentType }"></pngx-input-select>
|
(createNew)="createDocumentType($event)" [hideAddButton]="createDisabled(DataType.DocumentType)" [suggestions]="suggestions?.document_types" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.DocumentType }"></pngx-input-select>
|
||||||
<pngx-input-select [items]="storagePaths" i18n-title title="Storage path" formControlName="storage_path" [allowNull]="true" [showFilter]="true" [horizontal]="true" (filterDocuments)="filterDocuments($event, DataType.StoragePath)"
|
<pngx-input-select [items]="storagePaths" i18n-title title="Storage path" formControlName="storage_path" [allowNull]="true" [showFilter]="true" [horizontal]="true" (filterDocuments)="filterDocuments($event, DataType.StoragePath)"
|
||||||
(createNew)="createStoragePath($event)" [hideAddButton]="createDisabled(DataType.StoragePath)" [suggestions]="suggestions?.storage_paths" i18n-placeholder placeholder="Default" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.StoragePath }"></pngx-input-select>
|
(createNew)="createStoragePath($event)" [hideAddButton]="createDisabled(DataType.StoragePath)" [suggestions]="suggestions?.storage_paths" i18n-placeholder placeholder="Default" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.StoragePath }"></pngx-input-select>
|
||||||
<pngx-input-tags formControlName="tags" [suggestions]="suggestions?.tags" [showFilter]="true" [horizontal]="true" (filterDocuments)="filterDocuments($event, DataType.Tag)" [hideAddButton]="createDisabled(DataType.Tag)" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Tag }"></pngx-input-tags>
|
<pngx-input-tags #tagsInput formControlName="tags" [suggestions]="suggestions?.tags" [showFilter]="true" [horizontal]="true" (filterDocuments)="filterDocuments($event, DataType.Tag)" [hideAddButton]="createDisabled(DataType.Tag)" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Tag }"></pngx-input-tags>
|
||||||
@for (fieldInstance of document?.custom_fields; track fieldInstance.field; let i = $index) {
|
@for (fieldInstance of document?.custom_fields; track fieldInstance.field; let i = $index) {
|
||||||
<div [formGroup]="customFieldFormFields.controls[i]">
|
<div [formGroup]="customFieldFormFields.controls[i]">
|
||||||
@switch (getCustomFieldFromInstance(fieldInstance)?.data_type) {
|
@switch (getCustomFieldFromInstance(fieldInstance)?.data_type) {
|
||||||
@@ -351,14 +367,14 @@
|
|||||||
</form>
|
</form>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="col-md-6 col-xl-8 mb-3 d-none d-md-block position-relative" #pdfPreview>
|
<div class="col-md-6 col-xl-7 mb-3 d-none d-md-block position-relative" #pdfPreview>
|
||||||
<ng-container *ngTemplateOutlet="previewContent"></ng-container>
|
<ng-container *ngTemplateOutlet="previewContent"></ng-container>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<ng-template #saveButtons>
|
<ng-template #saveButtons>
|
||||||
<div class="btn-group pb-3 ms-auto">
|
<div class="btn-group pb-3 ms-4">
|
||||||
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
||||||
<button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive || (isDirty$ | async) !== true">Save</button>
|
<button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive || (isDirty$ | async) !== true">Save</button>
|
||||||
@if (hasNext()) {
|
@if (hasNext()) {
|
||||||
|
@@ -156,6 +156,16 @@ describe('DocumentDetailComponent', () => {
|
|||||||
{
|
{
|
||||||
provide: TagService,
|
provide: TagService,
|
||||||
useValue: {
|
useValue: {
|
||||||
|
getCachedMany: (ids: number[]) =>
|
||||||
|
of(
|
||||||
|
ids.map((id) => ({
|
||||||
|
id,
|
||||||
|
name: `Tag${id}`,
|
||||||
|
is_inbox_tag: true,
|
||||||
|
color: '#ff0000',
|
||||||
|
text_color: '#000000',
|
||||||
|
}))
|
||||||
|
),
|
||||||
listAll: () =>
|
listAll: () =>
|
||||||
of({
|
of({
|
||||||
count: 3,
|
count: 3,
|
||||||
@@ -382,8 +392,32 @@ describe('DocumentDetailComponent', () => {
|
|||||||
currentUserCan = true
|
currentUserCan = true
|
||||||
})
|
})
|
||||||
|
|
||||||
it('should support creating document type', () => {
|
it('should support creating tag, remove from suggestions', () => {
|
||||||
initNormally()
|
initNormally()
|
||||||
|
component.suggestions = {
|
||||||
|
suggested_tags: ['Tag1', 'NewTag12'],
|
||||||
|
}
|
||||||
|
let openModal: NgbModalRef
|
||||||
|
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
|
||||||
|
const modalSpy = jest.spyOn(modalService, 'open')
|
||||||
|
component.createTag('NewTag12')
|
||||||
|
expect(modalSpy).toHaveBeenCalled()
|
||||||
|
openModal.componentInstance.succeeded.next({
|
||||||
|
id: 12,
|
||||||
|
name: 'NewTag12',
|
||||||
|
is_inbox_tag: true,
|
||||||
|
color: '#ff0000',
|
||||||
|
text_color: '#000000',
|
||||||
|
})
|
||||||
|
expect(component.documentForm.get('tags').value).toContain(12)
|
||||||
|
expect(component.suggestions.suggested_tags).not.toContain('NewTag12')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should support creating document type, remove from suggestions', () => {
|
||||||
|
initNormally()
|
||||||
|
component.suggestions = {
|
||||||
|
suggested_document_types: ['DocumentType1', 'NewDocType2'],
|
||||||
|
}
|
||||||
let openModal: NgbModalRef
|
let openModal: NgbModalRef
|
||||||
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
|
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
|
||||||
const modalSpy = jest.spyOn(modalService, 'open')
|
const modalSpy = jest.spyOn(modalService, 'open')
|
||||||
@@ -391,10 +425,16 @@ describe('DocumentDetailComponent', () => {
|
|||||||
expect(modalSpy).toHaveBeenCalled()
|
expect(modalSpy).toHaveBeenCalled()
|
||||||
openModal.componentInstance.succeeded.next({ id: 12, name: 'NewDocType12' })
|
openModal.componentInstance.succeeded.next({ id: 12, name: 'NewDocType12' })
|
||||||
expect(component.documentForm.get('document_type').value).toEqual(12)
|
expect(component.documentForm.get('document_type').value).toEqual(12)
|
||||||
|
expect(component.suggestions.suggested_document_types).not.toContain(
|
||||||
|
'NewDocType2'
|
||||||
|
)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('should support creating correspondent', () => {
|
it('should support creating correspondent, remove from suggestions', () => {
|
||||||
initNormally()
|
initNormally()
|
||||||
|
component.suggestions = {
|
||||||
|
suggested_correspondents: ['Correspondent1', 'NewCorrrespondent12'],
|
||||||
|
}
|
||||||
let openModal: NgbModalRef
|
let openModal: NgbModalRef
|
||||||
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
|
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
|
||||||
const modalSpy = jest.spyOn(modalService, 'open')
|
const modalSpy = jest.spyOn(modalService, 'open')
|
||||||
@@ -405,6 +445,9 @@ describe('DocumentDetailComponent', () => {
|
|||||||
name: 'NewCorrrespondent12',
|
name: 'NewCorrrespondent12',
|
||||||
})
|
})
|
||||||
expect(component.documentForm.get('correspondent').value).toEqual(12)
|
expect(component.documentForm.get('correspondent').value).toEqual(12)
|
||||||
|
expect(component.suggestions.suggested_correspondents).not.toContain(
|
||||||
|
'NewCorrrespondent12'
|
||||||
|
)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('should support creating storage path', () => {
|
it('should support creating storage path', () => {
|
||||||
@@ -983,7 +1026,7 @@ describe('DocumentDetailComponent', () => {
|
|||||||
expect(component.document.custom_fields).toHaveLength(initialLength - 1)
|
expect(component.document.custom_fields).toHaveLength(initialLength - 1)
|
||||||
expect(component.customFieldFormFields).toHaveLength(initialLength - 1)
|
expect(component.customFieldFormFields).toHaveLength(initialLength - 1)
|
||||||
expect(
|
expect(
|
||||||
fixture.debugElement.query(By.css('form')).nativeElement.textContent
|
fixture.debugElement.query(By.css('form ul')).nativeElement.textContent
|
||||||
).not.toContain('Field 1')
|
).not.toContain('Field 1')
|
||||||
const updateSpy = jest.spyOn(documentService, 'update')
|
const updateSpy = jest.spyOn(documentService, 'update')
|
||||||
component.save(true)
|
component.save(true)
|
||||||
@@ -1020,10 +1063,22 @@ describe('DocumentDetailComponent', () => {
|
|||||||
|
|
||||||
it('should get suggestions', () => {
|
it('should get suggestions', () => {
|
||||||
const suggestionsSpy = jest.spyOn(documentService, 'getSuggestions')
|
const suggestionsSpy = jest.spyOn(documentService, 'getSuggestions')
|
||||||
suggestionsSpy.mockReturnValue(of({ tags: [42, 43] }))
|
suggestionsSpy.mockReturnValue(
|
||||||
|
of({
|
||||||
|
tags: [42, 43],
|
||||||
|
suggested_tags: [],
|
||||||
|
suggested_document_types: [],
|
||||||
|
suggested_correspondents: [],
|
||||||
|
})
|
||||||
|
)
|
||||||
initNormally()
|
initNormally()
|
||||||
expect(suggestionsSpy).toHaveBeenCalled()
|
expect(suggestionsSpy).toHaveBeenCalled()
|
||||||
expect(component.suggestions).toEqual({ tags: [42, 43] })
|
expect(component.suggestions).toEqual({
|
||||||
|
tags: [42, 43],
|
||||||
|
suggested_tags: [],
|
||||||
|
suggested_document_types: [],
|
||||||
|
suggested_correspondents: [],
|
||||||
|
})
|
||||||
})
|
})
|
||||||
|
|
||||||
it('should show error if needed for get suggestions', () => {
|
it('should show error if needed for get suggestions', () => {
|
||||||
|
@@ -74,6 +74,7 @@ import { CustomFieldsService } from 'src/app/services/rest/custom-fields.service
|
|||||||
import { DocumentTypeService } from 'src/app/services/rest/document-type.service'
|
import { DocumentTypeService } from 'src/app/services/rest/document-type.service'
|
||||||
import { DocumentService } from 'src/app/services/rest/document.service'
|
import { DocumentService } from 'src/app/services/rest/document.service'
|
||||||
import { StoragePathService } from 'src/app/services/rest/storage-path.service'
|
import { StoragePathService } from 'src/app/services/rest/storage-path.service'
|
||||||
|
import { TagService } from 'src/app/services/rest/tag.service'
|
||||||
import { UserService } from 'src/app/services/rest/user.service'
|
import { UserService } from 'src/app/services/rest/user.service'
|
||||||
import { SettingsService } from 'src/app/services/settings.service'
|
import { SettingsService } from 'src/app/services/settings.service'
|
||||||
import { ToastService } from 'src/app/services/toast.service'
|
import { ToastService } from 'src/app/services/toast.service'
|
||||||
@@ -89,6 +90,7 @@ import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspo
|
|||||||
import { DocumentTypeEditDialogComponent } from '../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
|
import { DocumentTypeEditDialogComponent } from '../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
|
||||||
import { EditDialogMode } from '../common/edit-dialog/edit-dialog.component'
|
import { EditDialogMode } from '../common/edit-dialog/edit-dialog.component'
|
||||||
import { StoragePathEditDialogComponent } from '../common/edit-dialog/storage-path-edit-dialog/storage-path-edit-dialog.component'
|
import { StoragePathEditDialogComponent } from '../common/edit-dialog/storage-path-edit-dialog/storage-path-edit-dialog.component'
|
||||||
|
import { TagEditDialogComponent } from '../common/edit-dialog/tag-edit-dialog/tag-edit-dialog.component'
|
||||||
import { EmailDocumentDialogComponent } from '../common/email-document-dialog/email-document-dialog.component'
|
import { EmailDocumentDialogComponent } from '../common/email-document-dialog/email-document-dialog.component'
|
||||||
import { CheckComponent } from '../common/input/check/check.component'
|
import { CheckComponent } from '../common/input/check/check.component'
|
||||||
import { DateComponent } from '../common/input/date/date.component'
|
import { DateComponent } from '../common/input/date/date.component'
|
||||||
@@ -102,6 +104,7 @@ import { TextComponent } from '../common/input/text/text.component'
|
|||||||
import { UrlComponent } from '../common/input/url/url.component'
|
import { UrlComponent } from '../common/input/url/url.component'
|
||||||
import { PageHeaderComponent } from '../common/page-header/page-header.component'
|
import { PageHeaderComponent } from '../common/page-header/page-header.component'
|
||||||
import { ShareLinksDialogComponent } from '../common/share-links-dialog/share-links-dialog.component'
|
import { ShareLinksDialogComponent } from '../common/share-links-dialog/share-links-dialog.component'
|
||||||
|
import { SuggestionsDropdownComponent } from '../common/suggestions-dropdown/suggestions-dropdown.component'
|
||||||
import { DocumentHistoryComponent } from '../document-history/document-history.component'
|
import { DocumentHistoryComponent } from '../document-history/document-history.component'
|
||||||
import { DocumentNotesComponent } from '../document-notes/document-notes.component'
|
import { DocumentNotesComponent } from '../document-notes/document-notes.component'
|
||||||
import { ComponentWithPermissions } from '../with-permissions/with-permissions.component'
|
import { ComponentWithPermissions } from '../with-permissions/with-permissions.component'
|
||||||
@@ -158,6 +161,7 @@ export enum ZoomSetting {
|
|||||||
NumberComponent,
|
NumberComponent,
|
||||||
MonetaryComponent,
|
MonetaryComponent,
|
||||||
UrlComponent,
|
UrlComponent,
|
||||||
|
SuggestionsDropdownComponent,
|
||||||
CustomDatePipe,
|
CustomDatePipe,
|
||||||
FileSizePipe,
|
FileSizePipe,
|
||||||
IfPermissionsDirective,
|
IfPermissionsDirective,
|
||||||
@@ -179,6 +183,8 @@ export class DocumentDetailComponent
|
|||||||
@ViewChild('inputTitle')
|
@ViewChild('inputTitle')
|
||||||
titleInput: TextComponent
|
titleInput: TextComponent
|
||||||
|
|
||||||
|
@ViewChild('tagsInput') tagsInput: TagsComponent
|
||||||
|
|
||||||
expandOriginalMetadata = false
|
expandOriginalMetadata = false
|
||||||
expandArchivedMetadata = false
|
expandArchivedMetadata = false
|
||||||
|
|
||||||
@@ -190,6 +196,7 @@ export class DocumentDetailComponent
|
|||||||
document: Document
|
document: Document
|
||||||
metadata: DocumentMetadata
|
metadata: DocumentMetadata
|
||||||
suggestions: DocumentSuggestions
|
suggestions: DocumentSuggestions
|
||||||
|
suggestionsLoading: boolean = false
|
||||||
users: User[]
|
users: User[]
|
||||||
|
|
||||||
title: string
|
title: string
|
||||||
@@ -262,6 +269,7 @@ export class DocumentDetailComponent
|
|||||||
constructor(
|
constructor(
|
||||||
private documentsService: DocumentService,
|
private documentsService: DocumentService,
|
||||||
private route: ActivatedRoute,
|
private route: ActivatedRoute,
|
||||||
|
private tagService: TagService,
|
||||||
private correspondentService: CorrespondentService,
|
private correspondentService: CorrespondentService,
|
||||||
private documentTypeService: DocumentTypeService,
|
private documentTypeService: DocumentTypeService,
|
||||||
private router: Router,
|
private router: Router,
|
||||||
@@ -291,6 +299,10 @@ export class DocumentDetailComponent
|
|||||||
return this.settings.get(SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER)
|
return this.settings.get(SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
get aiEnabled(): boolean {
|
||||||
|
return this.settings.get(SETTINGS_KEYS.AI_ENABLED)
|
||||||
|
}
|
||||||
|
|
||||||
get archiveContentRenderType(): ContentRenderType {
|
get archiveContentRenderType(): ContentRenderType {
|
||||||
return this.document?.archived_file_name
|
return this.document?.archived_file_name
|
||||||
? this.getRenderType('application/pdf')
|
? this.getRenderType('application/pdf')
|
||||||
@@ -645,25 +657,12 @@ export class DocumentDetailComponent
|
|||||||
PermissionType.Document
|
PermissionType.Document
|
||||||
)
|
)
|
||||||
) {
|
) {
|
||||||
this.documentsService
|
this.tagService.getCachedMany(doc.tags).subscribe((tags) => {
|
||||||
.getSuggestions(doc.id)
|
// only show suggestions if document has inbox tags
|
||||||
.pipe(
|
if (tags.some((tag) => tag.is_inbox_tag)) {
|
||||||
first(),
|
this.getSuggestions()
|
||||||
takeUntil(this.unsubscribeNotifier),
|
}
|
||||||
takeUntil(this.docChangeNotifier)
|
})
|
||||||
)
|
|
||||||
.subscribe({
|
|
||||||
next: (result) => {
|
|
||||||
this.suggestions = result
|
|
||||||
},
|
|
||||||
error: (error) => {
|
|
||||||
this.suggestions = null
|
|
||||||
this.toastService.showError(
|
|
||||||
$localize`Error retrieving suggestions.`,
|
|
||||||
error
|
|
||||||
)
|
|
||||||
},
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
this.title = this.documentTitlePipe.transform(doc.title)
|
this.title = this.documentTitlePipe.transform(doc.title)
|
||||||
const docFormValues = Object.assign({}, doc)
|
const docFormValues = Object.assign({}, doc)
|
||||||
@@ -680,6 +679,56 @@ export class DocumentDetailComponent
|
|||||||
return this.documentForm.get('custom_fields') as FormArray
|
return this.documentForm.get('custom_fields') as FormArray
|
||||||
}
|
}
|
||||||
|
|
||||||
|
getSuggestions() {
|
||||||
|
this.suggestionsLoading = true
|
||||||
|
this.documentsService
|
||||||
|
.getSuggestions(this.documentId)
|
||||||
|
.pipe(
|
||||||
|
first(),
|
||||||
|
takeUntil(this.unsubscribeNotifier),
|
||||||
|
takeUntil(this.docChangeNotifier)
|
||||||
|
)
|
||||||
|
.subscribe({
|
||||||
|
next: (result) => {
|
||||||
|
this.suggestions = result
|
||||||
|
this.suggestionsLoading = false
|
||||||
|
},
|
||||||
|
error: (error) => {
|
||||||
|
this.suggestions = null
|
||||||
|
this.suggestionsLoading = false
|
||||||
|
this.toastService.showError(
|
||||||
|
$localize`Error retrieving suggestions.`,
|
||||||
|
error
|
||||||
|
)
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
createTag(newName: string) {
|
||||||
|
var modal = this.modalService.open(TagEditDialogComponent, {
|
||||||
|
backdrop: 'static',
|
||||||
|
})
|
||||||
|
modal.componentInstance.dialogMode = EditDialogMode.CREATE
|
||||||
|
if (newName) modal.componentInstance.object = { name: newName }
|
||||||
|
modal.componentInstance.succeeded
|
||||||
|
.pipe(
|
||||||
|
switchMap((newTag) => {
|
||||||
|
return this.tagService
|
||||||
|
.listAll()
|
||||||
|
.pipe(map((tags) => ({ newTag, tags })))
|
||||||
|
})
|
||||||
|
)
|
||||||
|
.pipe(takeUntil(this.unsubscribeNotifier))
|
||||||
|
.subscribe(({ newTag, tags }) => {
|
||||||
|
this.tagsInput.tags = tags.results
|
||||||
|
this.tagsInput.addTag(newTag.id)
|
||||||
|
if (this.suggestions) {
|
||||||
|
this.suggestions.suggested_tags =
|
||||||
|
this.suggestions.suggested_tags.filter((tag) => tag !== newName)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
createDocumentType(newName: string) {
|
createDocumentType(newName: string) {
|
||||||
var modal = this.modalService.open(DocumentTypeEditDialogComponent, {
|
var modal = this.modalService.open(DocumentTypeEditDialogComponent, {
|
||||||
backdrop: 'static',
|
backdrop: 'static',
|
||||||
@@ -698,6 +747,12 @@ export class DocumentDetailComponent
|
|||||||
.subscribe(({ newDocumentType, documentTypes }) => {
|
.subscribe(({ newDocumentType, documentTypes }) => {
|
||||||
this.documentTypes = documentTypes.results
|
this.documentTypes = documentTypes.results
|
||||||
this.documentForm.get('document_type').setValue(newDocumentType.id)
|
this.documentForm.get('document_type').setValue(newDocumentType.id)
|
||||||
|
if (this.suggestions) {
|
||||||
|
this.suggestions.suggested_document_types =
|
||||||
|
this.suggestions.suggested_document_types.filter(
|
||||||
|
(dt) => dt !== newName
|
||||||
|
)
|
||||||
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -721,6 +776,12 @@ export class DocumentDetailComponent
|
|||||||
.subscribe(({ newCorrespondent, correspondents }) => {
|
.subscribe(({ newCorrespondent, correspondents }) => {
|
||||||
this.correspondents = correspondents.results
|
this.correspondents = correspondents.results
|
||||||
this.documentForm.get('correspondent').setValue(newCorrespondent.id)
|
this.documentForm.get('correspondent').setValue(newCorrespondent.id)
|
||||||
|
if (this.suggestions) {
|
||||||
|
this.suggestions.suggested_correspondents =
|
||||||
|
this.suggestions.suggested_correspondents.filter(
|
||||||
|
(c) => c !== newName
|
||||||
|
)
|
||||||
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@@ -1,11 +1,17 @@
|
|||||||
export interface DocumentSuggestions {
|
export interface DocumentSuggestions {
|
||||||
|
title?: string
|
||||||
|
|
||||||
tags?: number[]
|
tags?: number[]
|
||||||
|
suggested_tags?: string[]
|
||||||
|
|
||||||
correspondents?: number[]
|
correspondents?: number[]
|
||||||
|
suggested_correspondents?: string[]
|
||||||
|
|
||||||
document_types?: number[]
|
document_types?: number[]
|
||||||
|
suggested_document_types?: string[]
|
||||||
|
|
||||||
storage_paths?: number[]
|
storage_paths?: number[]
|
||||||
|
suggested_storage_paths?: string[]
|
||||||
|
|
||||||
dates?: string[] // ISO-formatted date string e.g. 2022-11-03
|
dates?: string[] // ISO-formatted date string e.g. 2022-11-03
|
||||||
}
|
}
|
||||||
|
@@ -44,11 +44,18 @@ export enum ConfigOptionType {
|
|||||||
Boolean = 'boolean',
|
Boolean = 'boolean',
|
||||||
JSON = 'json',
|
JSON = 'json',
|
||||||
File = 'file',
|
File = 'file',
|
||||||
|
Password = 'password',
|
||||||
}
|
}
|
||||||
|
|
||||||
export const ConfigCategory = {
|
export const ConfigCategory = {
|
||||||
General: $localize`General Settings`,
|
General: $localize`General Settings`,
|
||||||
OCR: $localize`OCR Settings`,
|
OCR: $localize`OCR Settings`,
|
||||||
|
AI: $localize`AI Settings`,
|
||||||
|
}
|
||||||
|
|
||||||
|
export const LLMBackendConfig = {
|
||||||
|
OPENAI: 'openai',
|
||||||
|
OLLAMA: 'ollama',
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ConfigOption {
|
export interface ConfigOption {
|
||||||
@@ -180,6 +187,42 @@ export const PaperlessConfigOptions: ConfigOption[] = [
|
|||||||
config_key: 'PAPERLESS_APP_TITLE',
|
config_key: 'PAPERLESS_APP_TITLE',
|
||||||
category: ConfigCategory.General,
|
category: ConfigCategory.General,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
key: 'ai_enabled',
|
||||||
|
title: $localize`AI Enabled`,
|
||||||
|
type: ConfigOptionType.Boolean,
|
||||||
|
config_key: 'PAPERLESS_AI_ENABLED',
|
||||||
|
category: ConfigCategory.AI,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'llm_backend',
|
||||||
|
title: $localize`LLM Backend`,
|
||||||
|
type: ConfigOptionType.Select,
|
||||||
|
choices: mapToItems(LLMBackendConfig),
|
||||||
|
config_key: 'PAPERLESS_LLM_BACKEND',
|
||||||
|
category: ConfigCategory.AI,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'llm_model',
|
||||||
|
title: $localize`LLM Model`,
|
||||||
|
type: ConfigOptionType.String,
|
||||||
|
config_key: 'PAPERLESS_LLM_MODEL',
|
||||||
|
category: ConfigCategory.AI,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'llm_api_key',
|
||||||
|
title: $localize`LLM API Key`,
|
||||||
|
type: ConfigOptionType.Password,
|
||||||
|
config_key: 'PAPERLESS_LLM_API_KEY',
|
||||||
|
category: ConfigCategory.AI,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'llm_url',
|
||||||
|
title: $localize`LLM URL`,
|
||||||
|
type: ConfigOptionType.String,
|
||||||
|
config_key: 'PAPERLESS_LLM_URL',
|
||||||
|
category: ConfigCategory.AI,
|
||||||
|
},
|
||||||
]
|
]
|
||||||
|
|
||||||
export interface PaperlessConfig extends ObjectWithId {
|
export interface PaperlessConfig extends ObjectWithId {
|
||||||
@@ -198,4 +241,9 @@ export interface PaperlessConfig extends ObjectWithId {
|
|||||||
user_args: object
|
user_args: object
|
||||||
app_logo: string
|
app_logo: string
|
||||||
app_title: string
|
app_title: string
|
||||||
|
ai_enabled: boolean
|
||||||
|
llm_backend: string
|
||||||
|
llm_model: string
|
||||||
|
llm_api_key: string
|
||||||
|
llm_url: string
|
||||||
}
|
}
|
||||||
|
@@ -73,6 +73,7 @@ export const SETTINGS_KEYS = {
|
|||||||
GMAIL_OAUTH_URL: 'gmail_oauth_url',
|
GMAIL_OAUTH_URL: 'gmail_oauth_url',
|
||||||
OUTLOOK_OAUTH_URL: 'outlook_oauth_url',
|
OUTLOOK_OAUTH_URL: 'outlook_oauth_url',
|
||||||
EMAIL_ENABLED: 'email_enabled',
|
EMAIL_ENABLED: 'email_enabled',
|
||||||
|
AI_ENABLED: 'ai_enabled',
|
||||||
}
|
}
|
||||||
|
|
||||||
export const SETTINGS: UiSetting[] = [
|
export const SETTINGS: UiSetting[] = [
|
||||||
@@ -276,4 +277,9 @@ export const SETTINGS: UiSetting[] = [
|
|||||||
type: 'string',
|
type: 'string',
|
||||||
default: 'page-width', // ZoomSetting from 'document-detail.component'
|
default: 'page-width', // ZoomSetting from 'document-detail.component'
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
key: SETTINGS_KEYS.AI_ENABLED,
|
||||||
|
type: 'boolean',
|
||||||
|
default: false,
|
||||||
|
},
|
||||||
]
|
]
|
||||||
|
@@ -118,6 +118,7 @@ import {
|
|||||||
sliders2Vertical,
|
sliders2Vertical,
|
||||||
sortAlphaDown,
|
sortAlphaDown,
|
||||||
sortAlphaUpAlt,
|
sortAlphaUpAlt,
|
||||||
|
stars,
|
||||||
tag,
|
tag,
|
||||||
tagFill,
|
tagFill,
|
||||||
tags,
|
tags,
|
||||||
@@ -324,6 +325,7 @@ const icons = {
|
|||||||
sliders2Vertical,
|
sliders2Vertical,
|
||||||
sortAlphaDown,
|
sortAlphaDown,
|
||||||
sortAlphaUpAlt,
|
sortAlphaUpAlt,
|
||||||
|
stars,
|
||||||
tagFill,
|
tagFill,
|
||||||
tag,
|
tag,
|
||||||
tags,
|
tags,
|
||||||
|
@@ -115,6 +115,56 @@ def refresh_suggestions_cache(
|
|||||||
cache.touch(doc_key, timeout)
|
cache.touch(doc_key, timeout)
|
||||||
|
|
||||||
|
|
||||||
|
def get_llm_suggestion_cache(
|
||||||
|
document_id: int,
|
||||||
|
backend: str,
|
||||||
|
) -> SuggestionCacheData | None:
|
||||||
|
doc_key = get_suggestion_cache_key(document_id)
|
||||||
|
data: SuggestionCacheData = cache.get(doc_key)
|
||||||
|
|
||||||
|
if data and data.classifier_hash == backend:
|
||||||
|
return data
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def set_llm_suggestions_cache(
|
||||||
|
document_id: int,
|
||||||
|
suggestions: dict,
|
||||||
|
*,
|
||||||
|
backend: str,
|
||||||
|
timeout: int = CACHE_50_MINUTES,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
Cache LLM-generated suggestions using a backend-specific identifier (e.g. 'openai:gpt-4').
|
||||||
|
"""
|
||||||
|
from documents.caching import SuggestionCacheData
|
||||||
|
|
||||||
|
doc_key = get_suggestion_cache_key(document_id)
|
||||||
|
cache.set(
|
||||||
|
doc_key,
|
||||||
|
SuggestionCacheData(
|
||||||
|
classifier_version=1000, # Unique marker for LLM-based suggestion
|
||||||
|
classifier_hash=backend,
|
||||||
|
suggestions=suggestions,
|
||||||
|
),
|
||||||
|
timeout,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def invalidate_llm_suggestions_cache(
|
||||||
|
document_id: int,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
Invalidate the LLM suggestions cache for a specific document and backend.
|
||||||
|
"""
|
||||||
|
doc_key = get_suggestion_cache_key(document_id)
|
||||||
|
data: SuggestionCacheData = cache.get(doc_key)
|
||||||
|
|
||||||
|
if data:
|
||||||
|
cache.delete(doc_key)
|
||||||
|
|
||||||
|
|
||||||
def get_metadata_cache_key(document_id: int) -> str:
|
def get_metadata_cache_key(document_id: int) -> str:
|
||||||
"""
|
"""
|
||||||
Returns the basic key for a document's metadata
|
Returns the basic key for a document's metadata
|
||||||
|
@@ -25,6 +25,7 @@ from guardian.shortcuts import remove_perm
|
|||||||
|
|
||||||
from documents import matching
|
from documents import matching
|
||||||
from documents.caching import clear_document_caches
|
from documents.caching import clear_document_caches
|
||||||
|
from documents.caching import invalidate_llm_suggestions_cache
|
||||||
from documents.file_handling import create_source_path_directory
|
from documents.file_handling import create_source_path_directory
|
||||||
from documents.file_handling import delete_empty_directories
|
from documents.file_handling import delete_empty_directories
|
||||||
from documents.file_handling import generate_unique_filename
|
from documents.file_handling import generate_unique_filename
|
||||||
@@ -524,6 +525,15 @@ def update_filename_and_move_files(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@receiver(models.signals.post_save, sender=Document)
|
||||||
|
def update_llm_suggestions_cache(sender, instance, **kwargs):
|
||||||
|
"""
|
||||||
|
Invalidate the LLM suggestions cache when a document is saved.
|
||||||
|
"""
|
||||||
|
# Invalidate the cache for the document
|
||||||
|
invalidate_llm_suggestions_cache(instance.pk)
|
||||||
|
|
||||||
|
|
||||||
# should be disabled in /src/documents/management/commands/document_importer.py handle
|
# should be disabled in /src/documents/management/commands/document_importer.py handle
|
||||||
@receiver(models.signals.post_save, sender=CustomField)
|
@receiver(models.signals.post_save, sender=CustomField)
|
||||||
def check_paths_and_prune_custom_fields(sender, instance: CustomField, **kwargs):
|
def check_paths_and_prune_custom_fields(sender, instance: CustomField, **kwargs):
|
||||||
|
@@ -31,29 +31,32 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
|
|||||||
response = self.client.get(self.ENDPOINT, format="json")
|
response = self.client.get(self.ENDPOINT, format="json")
|
||||||
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.maxDiff = None
|
||||||
self.assertEqual(
|
self.assertDictEqual(
|
||||||
json.dumps(response.data[0]),
|
response.data[0],
|
||||||
json.dumps(
|
{
|
||||||
{
|
"id": 1,
|
||||||
"id": 1,
|
"user_args": None,
|
||||||
"user_args": None,
|
"output_type": None,
|
||||||
"output_type": None,
|
"pages": None,
|
||||||
"pages": None,
|
"language": None,
|
||||||
"language": None,
|
"mode": None,
|
||||||
"mode": None,
|
"skip_archive_file": None,
|
||||||
"skip_archive_file": None,
|
"image_dpi": None,
|
||||||
"image_dpi": None,
|
"unpaper_clean": None,
|
||||||
"unpaper_clean": None,
|
"deskew": None,
|
||||||
"deskew": None,
|
"rotate_pages": None,
|
||||||
"rotate_pages": None,
|
"rotate_pages_threshold": None,
|
||||||
"rotate_pages_threshold": None,
|
"max_image_pixels": None,
|
||||||
"max_image_pixels": None,
|
"color_conversion_strategy": None,
|
||||||
"color_conversion_strategy": None,
|
"app_title": None,
|
||||||
"app_title": None,
|
"app_logo": None,
|
||||||
"app_logo": None,
|
"ai_enabled": False,
|
||||||
},
|
"llm_backend": None,
|
||||||
),
|
"llm_model": None,
|
||||||
|
"llm_api_key": None,
|
||||||
|
"llm_url": None,
|
||||||
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_api_get_ui_settings_with_config(self):
|
def test_api_get_ui_settings_with_config(self):
|
||||||
|
@@ -47,6 +47,7 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase):
|
|||||||
"backend_setting": "default",
|
"backend_setting": "default",
|
||||||
},
|
},
|
||||||
"email_enabled": False,
|
"email_enabled": False,
|
||||||
|
"ai_enabled": False,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@@ -1,6 +1,8 @@
|
|||||||
import tempfile
|
import tempfile
|
||||||
from datetime import timedelta
|
from datetime import timedelta
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
from django.conf import settings
|
from django.conf import settings
|
||||||
from django.contrib.auth.models import Permission
|
from django.contrib.auth.models import Permission
|
||||||
@@ -10,8 +12,15 @@ from django.test import override_settings
|
|||||||
from django.utils import timezone
|
from django.utils import timezone
|
||||||
from rest_framework import status
|
from rest_framework import status
|
||||||
|
|
||||||
|
from documents.caching import get_llm_suggestion_cache
|
||||||
|
from documents.caching import set_llm_suggestions_cache
|
||||||
|
from documents.models import Correspondent
|
||||||
from documents.models import Document
|
from documents.models import Document
|
||||||
|
from documents.models import DocumentType
|
||||||
from documents.models import ShareLink
|
from documents.models import ShareLink
|
||||||
|
from documents.models import StoragePath
|
||||||
|
from documents.models import Tag
|
||||||
|
from documents.signals.handlers import update_llm_suggestions_cache
|
||||||
from documents.tests.utils import DirectoriesMixin
|
from documents.tests.utils import DirectoriesMixin
|
||||||
from paperless.models import ApplicationConfiguration
|
from paperless.models import ApplicationConfiguration
|
||||||
|
|
||||||
@@ -154,3 +163,104 @@ class TestViews(DirectoriesMixin, TestCase):
|
|||||||
response.render()
|
response.render()
|
||||||
self.assertEqual(response.request["PATH_INFO"], "/accounts/login/")
|
self.assertEqual(response.request["PATH_INFO"], "/accounts/login/")
|
||||||
self.assertContains(response, b"Share link has expired")
|
self.assertContains(response, b"Share link has expired")
|
||||||
|
|
||||||
|
|
||||||
|
class TestAISuggestions(DirectoriesMixin, TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.user = User.objects.create_superuser(username="testuser")
|
||||||
|
self.document = Document.objects.create(
|
||||||
|
title="Test Document",
|
||||||
|
filename="test.pdf",
|
||||||
|
mime_type="application/pdf",
|
||||||
|
)
|
||||||
|
self.tag1 = Tag.objects.create(name="tag1")
|
||||||
|
self.correspondent1 = Correspondent.objects.create(name="correspondent1")
|
||||||
|
self.document_type1 = DocumentType.objects.create(name="type1")
|
||||||
|
self.path1 = StoragePath.objects.create(name="path1")
|
||||||
|
super().setUp()
|
||||||
|
|
||||||
|
@patch("documents.views.get_llm_suggestion_cache")
|
||||||
|
@patch("documents.views.refresh_suggestions_cache")
|
||||||
|
@override_settings(
|
||||||
|
AI_ENABLED=True,
|
||||||
|
LLM_BACKEND="mock_backend",
|
||||||
|
)
|
||||||
|
def test_suggestions_with_cached_llm(self, mock_refresh_cache, mock_get_cache):
|
||||||
|
mock_get_cache.return_value = MagicMock(suggestions={"tags": ["tag1", "tag2"]})
|
||||||
|
|
||||||
|
self.client.force_login(user=self.user)
|
||||||
|
response = self.client.get(f"/api/documents/{self.document.pk}/suggestions/")
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(response.json(), {"tags": ["tag1", "tag2"]})
|
||||||
|
mock_refresh_cache.assert_called_once_with(self.document.pk)
|
||||||
|
|
||||||
|
@patch("documents.views.get_ai_document_classification")
|
||||||
|
@override_settings(
|
||||||
|
AI_ENABLED=True,
|
||||||
|
LLM_BACKEND="mock_backend",
|
||||||
|
)
|
||||||
|
def test_suggestions_with_ai_enabled(
|
||||||
|
self,
|
||||||
|
mock_get_ai_classification,
|
||||||
|
):
|
||||||
|
mock_get_ai_classification.return_value = {
|
||||||
|
"title": "AI Title",
|
||||||
|
"tags": ["tag1", "tag2"],
|
||||||
|
"correspondents": ["correspondent1"],
|
||||||
|
"document_types": ["type1"],
|
||||||
|
"storage_paths": ["path1"],
|
||||||
|
"dates": ["2023-01-01"],
|
||||||
|
}
|
||||||
|
|
||||||
|
self.client.force_login(user=self.user)
|
||||||
|
response = self.client.get(f"/api/documents/{self.document.pk}/suggestions/")
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
self.assertEqual(
|
||||||
|
response.json(),
|
||||||
|
{
|
||||||
|
"title": "AI Title",
|
||||||
|
"tags": [self.tag1.pk],
|
||||||
|
"suggested_tags": ["tag2"],
|
||||||
|
"correspondents": [self.correspondent1.pk],
|
||||||
|
"suggested_correspondents": [],
|
||||||
|
"document_types": [self.document_type1.pk],
|
||||||
|
"suggested_document_types": [],
|
||||||
|
"storage_paths": [self.path1.pk],
|
||||||
|
"suggested_storage_paths": [],
|
||||||
|
"dates": ["2023-01-01"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_invalidate_suggestions_cache(self):
|
||||||
|
self.client.force_login(user=self.user)
|
||||||
|
suggestions = {
|
||||||
|
"title": "AI Title",
|
||||||
|
"tags": ["tag1", "tag2"],
|
||||||
|
"correspondents": ["correspondent1"],
|
||||||
|
"document_types": ["type1"],
|
||||||
|
"storage_paths": ["path1"],
|
||||||
|
"dates": ["2023-01-01"],
|
||||||
|
}
|
||||||
|
set_llm_suggestions_cache(
|
||||||
|
self.document.pk,
|
||||||
|
suggestions,
|
||||||
|
backend="mock_backend",
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
get_llm_suggestion_cache(
|
||||||
|
self.document.pk,
|
||||||
|
backend="mock_backend",
|
||||||
|
).suggestions,
|
||||||
|
suggestions,
|
||||||
|
)
|
||||||
|
# post_save signal triggered
|
||||||
|
update_llm_suggestions_cache(
|
||||||
|
sender=None,
|
||||||
|
instance=self.document,
|
||||||
|
)
|
||||||
|
self.assertIsNone(
|
||||||
|
get_llm_suggestion_cache(
|
||||||
|
self.document.pk,
|
||||||
|
backend="mock_backend",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
@@ -80,10 +80,12 @@ from documents import index
|
|||||||
from documents.bulk_download import ArchiveOnlyStrategy
|
from documents.bulk_download import ArchiveOnlyStrategy
|
||||||
from documents.bulk_download import OriginalAndArchiveStrategy
|
from documents.bulk_download import OriginalAndArchiveStrategy
|
||||||
from documents.bulk_download import OriginalsOnlyStrategy
|
from documents.bulk_download import OriginalsOnlyStrategy
|
||||||
|
from documents.caching import get_llm_suggestion_cache
|
||||||
from documents.caching import get_metadata_cache
|
from documents.caching import get_metadata_cache
|
||||||
from documents.caching import get_suggestion_cache
|
from documents.caching import get_suggestion_cache
|
||||||
from documents.caching import refresh_metadata_cache
|
from documents.caching import refresh_metadata_cache
|
||||||
from documents.caching import refresh_suggestions_cache
|
from documents.caching import refresh_suggestions_cache
|
||||||
|
from documents.caching import set_llm_suggestions_cache
|
||||||
from documents.caching import set_metadata_cache
|
from documents.caching import set_metadata_cache
|
||||||
from documents.caching import set_suggestions_cache
|
from documents.caching import set_suggestions_cache
|
||||||
from documents.classifier import load_classifier
|
from documents.classifier import load_classifier
|
||||||
@@ -168,7 +170,14 @@ from documents.tasks import sanity_check
|
|||||||
from documents.tasks import train_classifier
|
from documents.tasks import train_classifier
|
||||||
from documents.templating.filepath import validate_filepath_template_and_render
|
from documents.templating.filepath import validate_filepath_template_and_render
|
||||||
from paperless import version
|
from paperless import version
|
||||||
|
from paperless.ai.ai_classifier import get_ai_document_classification
|
||||||
|
from paperless.ai.matching import extract_unmatched_names
|
||||||
|
from paperless.ai.matching import match_correspondents_by_name
|
||||||
|
from paperless.ai.matching import match_document_types_by_name
|
||||||
|
from paperless.ai.matching import match_storage_paths_by_name
|
||||||
|
from paperless.ai.matching import match_tags_by_name
|
||||||
from paperless.celery import app as celery_app
|
from paperless.celery import app as celery_app
|
||||||
|
from paperless.config import AIConfig
|
||||||
from paperless.config import GeneralConfig
|
from paperless.config import GeneralConfig
|
||||||
from paperless.db import GnuPG
|
from paperless.db import GnuPG
|
||||||
from paperless.serialisers import GroupSerializer
|
from paperless.serialisers import GroupSerializer
|
||||||
@@ -730,37 +739,103 @@ class DocumentViewSet(
|
|||||||
):
|
):
|
||||||
return HttpResponseForbidden("Insufficient permissions")
|
return HttpResponseForbidden("Insufficient permissions")
|
||||||
|
|
||||||
document_suggestions = get_suggestion_cache(doc.pk)
|
ai_config = AIConfig()
|
||||||
|
|
||||||
if document_suggestions is not None:
|
if ai_config.ai_enabled:
|
||||||
refresh_suggestions_cache(doc.pk)
|
cached_llm_suggestions = get_llm_suggestion_cache(
|
||||||
return Response(document_suggestions.suggestions)
|
doc.pk,
|
||||||
|
backend=ai_config.llm_backend,
|
||||||
classifier = load_classifier()
|
|
||||||
|
|
||||||
dates = []
|
|
||||||
if settings.NUMBER_OF_SUGGESTED_DATES > 0:
|
|
||||||
gen = parse_date_generator(doc.filename, doc.content)
|
|
||||||
dates = sorted(
|
|
||||||
{i for i in itertools.islice(gen, settings.NUMBER_OF_SUGGESTED_DATES)},
|
|
||||||
)
|
)
|
||||||
|
|
||||||
resp_data = {
|
if cached_llm_suggestions:
|
||||||
"correspondents": [
|
refresh_suggestions_cache(doc.pk)
|
||||||
c.id for c in match_correspondents(doc, classifier, request.user)
|
return Response(cached_llm_suggestions.suggestions)
|
||||||
],
|
|
||||||
"tags": [t.id for t in match_tags(doc, classifier, request.user)],
|
|
||||||
"document_types": [
|
|
||||||
dt.id for dt in match_document_types(doc, classifier, request.user)
|
|
||||||
],
|
|
||||||
"storage_paths": [
|
|
||||||
dt.id for dt in match_storage_paths(doc, classifier, request.user)
|
|
||||||
],
|
|
||||||
"dates": [date.strftime("%Y-%m-%d") for date in dates if date is not None],
|
|
||||||
}
|
|
||||||
|
|
||||||
# Cache the suggestions and the classifier hash for later
|
llm_suggestions = get_ai_document_classification(doc)
|
||||||
set_suggestions_cache(doc.pk, resp_data, classifier)
|
|
||||||
|
matched_tags = match_tags_by_name(
|
||||||
|
llm_suggestions.get("tags", []),
|
||||||
|
request.user,
|
||||||
|
)
|
||||||
|
matched_correspondents = match_correspondents_by_name(
|
||||||
|
llm_suggestions.get("correspondents", []),
|
||||||
|
request.user,
|
||||||
|
)
|
||||||
|
matched_types = match_document_types_by_name(
|
||||||
|
llm_suggestions.get("document_types", []),
|
||||||
|
request.user,
|
||||||
|
)
|
||||||
|
matched_paths = match_storage_paths_by_name(
|
||||||
|
llm_suggestions.get("storage_paths", []),
|
||||||
|
request.user,
|
||||||
|
)
|
||||||
|
|
||||||
|
resp_data = {
|
||||||
|
"title": llm_suggestions.get("title"),
|
||||||
|
"tags": [t.id for t in matched_tags],
|
||||||
|
"suggested_tags": extract_unmatched_names(
|
||||||
|
llm_suggestions.get("tags", []),
|
||||||
|
matched_tags,
|
||||||
|
),
|
||||||
|
"correspondents": [c.id for c in matched_correspondents],
|
||||||
|
"suggested_correspondents": extract_unmatched_names(
|
||||||
|
llm_suggestions.get("correspondents", []),
|
||||||
|
matched_correspondents,
|
||||||
|
),
|
||||||
|
"document_types": [d.id for d in matched_types],
|
||||||
|
"suggested_document_types": extract_unmatched_names(
|
||||||
|
llm_suggestions.get("document_types", []),
|
||||||
|
matched_types,
|
||||||
|
),
|
||||||
|
"storage_paths": [s.id for s in matched_paths],
|
||||||
|
"suggested_storage_paths": extract_unmatched_names(
|
||||||
|
llm_suggestions.get("storage_paths", []),
|
||||||
|
matched_paths,
|
||||||
|
),
|
||||||
|
"dates": llm_suggestions.get("dates", []),
|
||||||
|
}
|
||||||
|
|
||||||
|
set_llm_suggestions_cache(doc.pk, resp_data, backend=ai_config.llm_backend)
|
||||||
|
else:
|
||||||
|
document_suggestions = get_suggestion_cache(doc.pk)
|
||||||
|
|
||||||
|
if document_suggestions is not None:
|
||||||
|
refresh_suggestions_cache(doc.pk)
|
||||||
|
return Response(document_suggestions.suggestions)
|
||||||
|
|
||||||
|
classifier = load_classifier()
|
||||||
|
|
||||||
|
dates = []
|
||||||
|
if settings.NUMBER_OF_SUGGESTED_DATES > 0:
|
||||||
|
gen = parse_date_generator(doc.filename, doc.content)
|
||||||
|
dates = sorted(
|
||||||
|
{
|
||||||
|
i
|
||||||
|
for i in itertools.islice(
|
||||||
|
gen,
|
||||||
|
settings.NUMBER_OF_SUGGESTED_DATES,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
resp_data = {
|
||||||
|
"correspondents": [
|
||||||
|
c.id for c in match_correspondents(doc, classifier, request.user)
|
||||||
|
],
|
||||||
|
"tags": [t.id for t in match_tags(doc, classifier, request.user)],
|
||||||
|
"document_types": [
|
||||||
|
dt.id for dt in match_document_types(doc, classifier, request.user)
|
||||||
|
],
|
||||||
|
"storage_paths": [
|
||||||
|
dt.id for dt in match_storage_paths(doc, classifier, request.user)
|
||||||
|
],
|
||||||
|
"dates": [
|
||||||
|
date.strftime("%Y-%m-%d") for date in dates if date is not None
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cache the suggestions and the classifier hash for later
|
||||||
|
set_suggestions_cache(doc.pk, resp_data, classifier)
|
||||||
|
|
||||||
return Response(resp_data)
|
return Response(resp_data)
|
||||||
|
|
||||||
@@ -2149,6 +2224,10 @@ class UiSettingsView(GenericAPIView):
|
|||||||
|
|
||||||
ui_settings["email_enabled"] = settings.EMAIL_ENABLED
|
ui_settings["email_enabled"] = settings.EMAIL_ENABLED
|
||||||
|
|
||||||
|
ai_config = AIConfig()
|
||||||
|
|
||||||
|
ui_settings["ai_enabled"] = ai_config.ai_enabled
|
||||||
|
|
||||||
user_resp = {
|
user_resp = {
|
||||||
"id": user.id,
|
"id": user.id,
|
||||||
"username": user.username,
|
"username": user.username,
|
||||||
|
0
src/paperless/ai/__init__.py
Normal file
0
src/paperless/ai/__init__.py
Normal file
86
src/paperless/ai/ai_classifier.py
Normal file
86
src/paperless/ai/ai_classifier.py
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
import json
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from documents.models import Document
|
||||||
|
from paperless.ai.client import AIClient
|
||||||
|
|
||||||
|
logger = logging.getLogger("paperless.ai.ai_classifier")
|
||||||
|
|
||||||
|
|
||||||
|
def get_ai_document_classification(document: Document) -> dict:
|
||||||
|
"""
|
||||||
|
Returns classification suggestions for a given document using an LLM.
|
||||||
|
Output schema matches the API's expected DocumentClassificationSuggestions format.
|
||||||
|
"""
|
||||||
|
filename = document.filename or ""
|
||||||
|
content = document.content or ""
|
||||||
|
|
||||||
|
prompt = f"""
|
||||||
|
You are an assistant that extracts structured information from documents.
|
||||||
|
Only respond with the JSON object as described below.
|
||||||
|
Never ask for further information, additional content or ask questions. Never include any other text.
|
||||||
|
Suggested tags and document types must be strictly based on the content of the document.
|
||||||
|
Do not change the field names or the JSON structure, only provide the values. Use double quotes and proper JSON syntax.
|
||||||
|
|
||||||
|
The JSON object must contain the following fields:
|
||||||
|
- title: A short, descriptive title
|
||||||
|
- tags: A list of simple tags like ["insurance", "medical", "receipts"]
|
||||||
|
- correspondents: A list of names or organizations mentioned in the document
|
||||||
|
- document_types: The type/category of the document (e.g. "invoice", "medical record")
|
||||||
|
- storage_paths: Suggested folder paths (e.g. "Medical/Insurance")
|
||||||
|
- dates: List up to 3 relevant dates in YYYY-MM-DD format
|
||||||
|
|
||||||
|
The format of the JSON object is as follows:
|
||||||
|
{{
|
||||||
|
"title": "xxxxx",
|
||||||
|
"tags": ["xxxx", "xxxx"],
|
||||||
|
"correspondents": ["xxxx", "xxxx"],
|
||||||
|
"document_types": ["xxxx", "xxxx"],
|
||||||
|
"storage_paths": ["xxxx", "xxxx"],
|
||||||
|
"dates": ["YYYY-MM-DD", "YYYY-MM-DD", "YYYY-MM-DD"],
|
||||||
|
}}
|
||||||
|
---
|
||||||
|
|
||||||
|
FILENAME:
|
||||||
|
{filename}
|
||||||
|
|
||||||
|
CONTENT:
|
||||||
|
{content[:8000]} # Trim to safe size
|
||||||
|
"""
|
||||||
|
|
||||||
|
try:
|
||||||
|
client = AIClient()
|
||||||
|
result = client.run_llm_query(prompt)
|
||||||
|
suggestions = parse_ai_classification_response(result)
|
||||||
|
return suggestions or {}
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Error during LLM classification: %s", exc_info=True)
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def parse_ai_classification_response(text: str) -> dict:
|
||||||
|
"""
|
||||||
|
Parses LLM output and ensures it conforms to expected schema.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
raw = json.loads(text)
|
||||||
|
return {
|
||||||
|
"title": raw.get("title"),
|
||||||
|
"tags": raw.get("tags", []),
|
||||||
|
"correspondents": [raw["correspondents"]]
|
||||||
|
if isinstance(raw.get("correspondents"), str)
|
||||||
|
else raw.get("correspondents", []),
|
||||||
|
"document_types": [raw["document_types"]]
|
||||||
|
if isinstance(raw.get("document_types"), str)
|
||||||
|
else raw.get("document_types", []),
|
||||||
|
"storage_paths": raw.get("storage_paths", []),
|
||||||
|
"dates": [d for d in raw.get("dates", []) if d],
|
||||||
|
}
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
# fallback: try to extract JSON manually?
|
||||||
|
logger.exception(
|
||||||
|
"Failed to parse LLM classification response: %s",
|
||||||
|
text,
|
||||||
|
exc_info=True,
|
||||||
|
)
|
||||||
|
return {}
|
70
src/paperless/ai/client.py
Normal file
70
src/paperless/ai/client.py
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
import logging
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from paperless.config import AIConfig
|
||||||
|
|
||||||
|
logger = logging.getLogger("paperless.ai.client")
|
||||||
|
|
||||||
|
|
||||||
|
class AIClient:
|
||||||
|
"""
|
||||||
|
A client for interacting with an LLM backend.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.settings = AIConfig()
|
||||||
|
|
||||||
|
def run_llm_query(self, prompt: str) -> str:
|
||||||
|
logger.debug(
|
||||||
|
"Running LLM query against %s with model %s",
|
||||||
|
self.settings.llm_backend,
|
||||||
|
self.settings.llm_model,
|
||||||
|
)
|
||||||
|
match self.settings.llm_backend:
|
||||||
|
case "openai":
|
||||||
|
result = self._run_openai_query(prompt)
|
||||||
|
case "ollama":
|
||||||
|
result = self._run_ollama_query(prompt)
|
||||||
|
case _:
|
||||||
|
raise ValueError(
|
||||||
|
f"Unsupported LLM backend: {self.settings.llm_backend}",
|
||||||
|
)
|
||||||
|
logger.debug("LLM query result: %s", result)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def _run_ollama_query(self, prompt: str) -> str:
|
||||||
|
url = self.settings.llm_url or "http://localhost:11434"
|
||||||
|
with httpx.Client(timeout=30.0) as client:
|
||||||
|
response = client.post(
|
||||||
|
f"{url}/api/chat",
|
||||||
|
json={
|
||||||
|
"model": self.settings.llm_model,
|
||||||
|
"messages": [{"role": "user", "content": prompt}],
|
||||||
|
"stream": False,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()["message"]["content"]
|
||||||
|
|
||||||
|
def _run_openai_query(self, prompt: str) -> str:
|
||||||
|
if not self.settings.llm_api_key:
|
||||||
|
raise RuntimeError("PAPERLESS_LLM_API_KEY is not set")
|
||||||
|
|
||||||
|
url = self.settings.llm_url or "https://api.openai.com"
|
||||||
|
|
||||||
|
with httpx.Client(timeout=30.0) as client:
|
||||||
|
response = client.post(
|
||||||
|
f"{url}/v1/chat/completions",
|
||||||
|
headers={
|
||||||
|
"Authorization": f"Bearer {self.settings.llm_api_key}",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
},
|
||||||
|
json={
|
||||||
|
"model": self.settings.llm_model,
|
||||||
|
"messages": [{"role": "user", "content": prompt}],
|
||||||
|
"temperature": 0.3,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()["choices"][0]["message"]["content"]
|
100
src/paperless/ai/matching.py
Normal file
100
src/paperless/ai/matching.py
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
import difflib
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
|
||||||
|
from django.contrib.auth.models import User
|
||||||
|
|
||||||
|
from documents.models import Correspondent
|
||||||
|
from documents.models import DocumentType
|
||||||
|
from documents.models import StoragePath
|
||||||
|
from documents.models import Tag
|
||||||
|
from documents.permissions import get_objects_for_user_owner_aware
|
||||||
|
|
||||||
|
MATCH_THRESHOLD = 0.8
|
||||||
|
|
||||||
|
logger = logging.getLogger("paperless.ai.matching")
|
||||||
|
|
||||||
|
|
||||||
|
def match_tags_by_name(names: list[str], user: User) -> list[Tag]:
|
||||||
|
queryset = get_objects_for_user_owner_aware(
|
||||||
|
user,
|
||||||
|
["view_tag"],
|
||||||
|
Tag,
|
||||||
|
)
|
||||||
|
return _match_names_to_queryset(names, queryset, "name")
|
||||||
|
|
||||||
|
|
||||||
|
def match_correspondents_by_name(names: list[str], user: User) -> list[Correspondent]:
|
||||||
|
queryset = get_objects_for_user_owner_aware(
|
||||||
|
user,
|
||||||
|
["view_correspondent"],
|
||||||
|
Correspondent,
|
||||||
|
)
|
||||||
|
return _match_names_to_queryset(names, queryset, "name")
|
||||||
|
|
||||||
|
|
||||||
|
def match_document_types_by_name(names: list[str], user: User) -> list[DocumentType]:
|
||||||
|
queryset = get_objects_for_user_owner_aware(
|
||||||
|
user,
|
||||||
|
["view_documenttype"],
|
||||||
|
DocumentType,
|
||||||
|
)
|
||||||
|
return _match_names_to_queryset(names, queryset, "name")
|
||||||
|
|
||||||
|
|
||||||
|
def match_storage_paths_by_name(names: list[str], user: User) -> list[StoragePath]:
|
||||||
|
queryset = get_objects_for_user_owner_aware(
|
||||||
|
user,
|
||||||
|
["view_storagepath"],
|
||||||
|
StoragePath,
|
||||||
|
)
|
||||||
|
return _match_names_to_queryset(names, queryset, "name")
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize(s: str) -> str:
|
||||||
|
s = s.lower()
|
||||||
|
s = re.sub(r"[^\w\s]", "", s) # remove punctuation
|
||||||
|
s = s.strip()
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
def _match_names_to_queryset(names: list[str], queryset, attr: str):
|
||||||
|
results = []
|
||||||
|
objects = list(queryset)
|
||||||
|
object_names = [_normalize(getattr(obj, attr)) for obj in objects]
|
||||||
|
|
||||||
|
for name in names:
|
||||||
|
if not name:
|
||||||
|
continue
|
||||||
|
target = _normalize(name)
|
||||||
|
|
||||||
|
# First try exact match
|
||||||
|
if target in object_names:
|
||||||
|
index = object_names.index(target)
|
||||||
|
results.append(objects[index])
|
||||||
|
# Remove the matched name from the list to avoid fuzzy matching later
|
||||||
|
object_names.remove(target)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Fuzzy match fallback
|
||||||
|
matches = difflib.get_close_matches(
|
||||||
|
target,
|
||||||
|
object_names,
|
||||||
|
n=1,
|
||||||
|
cutoff=MATCH_THRESHOLD,
|
||||||
|
)
|
||||||
|
if matches:
|
||||||
|
index = object_names.index(matches[0])
|
||||||
|
results.append(objects[index])
|
||||||
|
else:
|
||||||
|
pass
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def extract_unmatched_names(
|
||||||
|
names: list[str],
|
||||||
|
matched_objects: list,
|
||||||
|
attr="name",
|
||||||
|
) -> list[str]:
|
||||||
|
matched_names = {getattr(obj, attr).lower() for obj in matched_objects}
|
||||||
|
return [name for name in names if name.lower() not in matched_names]
|
@@ -114,3 +114,25 @@ class GeneralConfig(BaseConfig):
|
|||||||
|
|
||||||
self.app_title = app_config.app_title or None
|
self.app_title = app_config.app_title or None
|
||||||
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
|
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclasses.dataclass
|
||||||
|
class AIConfig(BaseConfig):
|
||||||
|
"""
|
||||||
|
AI related settings that require global scope
|
||||||
|
"""
|
||||||
|
|
||||||
|
ai_enabled: bool = dataclasses.field(init=False)
|
||||||
|
llm_backend: str = dataclasses.field(init=False)
|
||||||
|
llm_model: str = dataclasses.field(init=False)
|
||||||
|
llm_api_key: str = dataclasses.field(init=False)
|
||||||
|
llm_url: str = dataclasses.field(init=False)
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
app_config = self._get_config_instance()
|
||||||
|
|
||||||
|
self.ai_enabled = app_config.ai_enabled or settings.AI_ENABLED
|
||||||
|
self.llm_backend = app_config.llm_backend or settings.LLM_BACKEND
|
||||||
|
self.llm_model = app_config.llm_model or settings.LLM_MODEL
|
||||||
|
self.llm_api_key = app_config.llm_api_key or settings.LLM_API_KEY
|
||||||
|
self.llm_url = app_config.llm_url or settings.LLM_URL
|
||||||
|
@@ -0,0 +1,63 @@
|
|||||||
|
# Generated by Django 5.1.7 on 2025-04-24 02:09
|
||||||
|
|
||||||
|
from django.db import migrations
|
||||||
|
from django.db import models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
dependencies = [
|
||||||
|
("paperless", "0003_alter_applicationconfiguration_max_image_pixels"),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="applicationconfiguration",
|
||||||
|
name="ai_enabled",
|
||||||
|
field=models.BooleanField(
|
||||||
|
default=False,
|
||||||
|
null=True,
|
||||||
|
verbose_name="Enables AI features",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="applicationconfiguration",
|
||||||
|
name="llm_api_key",
|
||||||
|
field=models.CharField(
|
||||||
|
blank=True,
|
||||||
|
max_length=128,
|
||||||
|
null=True,
|
||||||
|
verbose_name="Sets the LLM API key",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="applicationconfiguration",
|
||||||
|
name="llm_backend",
|
||||||
|
field=models.CharField(
|
||||||
|
blank=True,
|
||||||
|
choices=[("openai", "OpenAI"), ("ollama", "Ollama")],
|
||||||
|
max_length=32,
|
||||||
|
null=True,
|
||||||
|
verbose_name="Sets the LLM backend",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="applicationconfiguration",
|
||||||
|
name="llm_model",
|
||||||
|
field=models.CharField(
|
||||||
|
blank=True,
|
||||||
|
max_length=32,
|
||||||
|
null=True,
|
||||||
|
verbose_name="Sets the LLM model",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="applicationconfiguration",
|
||||||
|
name="llm_url",
|
||||||
|
field=models.CharField(
|
||||||
|
blank=True,
|
||||||
|
max_length=128,
|
||||||
|
null=True,
|
||||||
|
verbose_name="Sets the LLM URL, optional",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
]
|
@@ -74,6 +74,15 @@ class ColorConvertChoices(models.TextChoices):
|
|||||||
CMYK = ("CMYK", _("CMYK"))
|
CMYK = ("CMYK", _("CMYK"))
|
||||||
|
|
||||||
|
|
||||||
|
class LLMBackend(models.TextChoices):
|
||||||
|
"""
|
||||||
|
Matches to --llm-backend
|
||||||
|
"""
|
||||||
|
|
||||||
|
OPENAI = ("openai", _("OpenAI"))
|
||||||
|
OLLAMA = ("ollama", _("Ollama"))
|
||||||
|
|
||||||
|
|
||||||
class ApplicationConfiguration(AbstractSingletonModel):
|
class ApplicationConfiguration(AbstractSingletonModel):
|
||||||
"""
|
"""
|
||||||
Settings which are common across more than 1 parser
|
Settings which are common across more than 1 parser
|
||||||
@@ -184,6 +193,45 @@ class ApplicationConfiguration(AbstractSingletonModel):
|
|||||||
upload_to="logo/",
|
upload_to="logo/",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
"""
|
||||||
|
AI related settings
|
||||||
|
"""
|
||||||
|
|
||||||
|
ai_enabled = models.BooleanField(
|
||||||
|
verbose_name=_("Enables AI features"),
|
||||||
|
null=True,
|
||||||
|
default=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
llm_backend = models.CharField(
|
||||||
|
verbose_name=_("Sets the LLM backend"),
|
||||||
|
null=True,
|
||||||
|
blank=True,
|
||||||
|
max_length=32,
|
||||||
|
choices=LLMBackend.choices,
|
||||||
|
)
|
||||||
|
|
||||||
|
llm_model = models.CharField(
|
||||||
|
verbose_name=_("Sets the LLM model"),
|
||||||
|
null=True,
|
||||||
|
blank=True,
|
||||||
|
max_length=32,
|
||||||
|
)
|
||||||
|
|
||||||
|
llm_api_key = models.CharField(
|
||||||
|
verbose_name=_("Sets the LLM API key"),
|
||||||
|
null=True,
|
||||||
|
blank=True,
|
||||||
|
max_length=128,
|
||||||
|
)
|
||||||
|
|
||||||
|
llm_url = models.CharField(
|
||||||
|
verbose_name=_("Sets the LLM URL, optional"),
|
||||||
|
null=True,
|
||||||
|
blank=True,
|
||||||
|
max_length=128,
|
||||||
|
)
|
||||||
|
|
||||||
class Meta:
|
class Meta:
|
||||||
verbose_name = _("paperless application settings")
|
verbose_name = _("paperless application settings")
|
||||||
|
|
||||||
|
@@ -185,6 +185,10 @@ class ProfileSerializer(serializers.ModelSerializer):
|
|||||||
|
|
||||||
class ApplicationConfigurationSerializer(serializers.ModelSerializer):
|
class ApplicationConfigurationSerializer(serializers.ModelSerializer):
|
||||||
user_args = serializers.JSONField(binary=True, allow_null=True)
|
user_args = serializers.JSONField(binary=True, allow_null=True)
|
||||||
|
llm_api_key = ObfuscatedPasswordField(
|
||||||
|
required=False,
|
||||||
|
allow_null=True,
|
||||||
|
)
|
||||||
|
|
||||||
def run_validation(self, data):
|
def run_validation(self, data):
|
||||||
# Empty strings treated as None to avoid unexpected behavior
|
# Empty strings treated as None to avoid unexpected behavior
|
||||||
|
@@ -1267,3 +1267,12 @@ OUTLOOK_OAUTH_ENABLED = bool(
|
|||||||
and OUTLOOK_OAUTH_CLIENT_ID
|
and OUTLOOK_OAUTH_CLIENT_ID
|
||||||
and OUTLOOK_OAUTH_CLIENT_SECRET,
|
and OUTLOOK_OAUTH_CLIENT_SECRET,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
################################################################################
|
||||||
|
# AI Settings #
|
||||||
|
################################################################################
|
||||||
|
AI_ENABLED = __get_boolean("PAPERLESS_AI_ENABLED", "NO")
|
||||||
|
LLM_BACKEND = os.getenv("PAPERLESS_LLM_BACKEND", "openai") # or "ollama"
|
||||||
|
LLM_MODEL = os.getenv("PAPERLESS_LLM_MODEL")
|
||||||
|
LLM_API_KEY = os.getenv("PAPERLESS_LLM_API_KEY")
|
||||||
|
LLM_URL = os.getenv("PAPERLESS_LLM_URL")
|
||||||
|
100
src/paperless/tests/test_ai_classifier.py
Normal file
100
src/paperless/tests/test_ai_classifier.py
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
import json
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from documents.models import Document
|
||||||
|
from paperless.ai.ai_classifier import get_ai_document_classification
|
||||||
|
from paperless.ai.ai_classifier import parse_ai_classification_response
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def mock_document():
|
||||||
|
return Document(filename="test.pdf", content="This is a test document content.")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.django_db
|
||||||
|
@patch("paperless.ai.client.AIClient.run_llm_query")
|
||||||
|
def test_get_ai_document_classification_success(mock_run_llm_query, mock_document):
|
||||||
|
mock_response = json.dumps(
|
||||||
|
{
|
||||||
|
"title": "Test Title",
|
||||||
|
"tags": ["test", "document"],
|
||||||
|
"correspondents": ["John Doe"],
|
||||||
|
"document_types": ["report"],
|
||||||
|
"storage_paths": ["Reports"],
|
||||||
|
"dates": ["2023-01-01"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
mock_run_llm_query.return_value = mock_response
|
||||||
|
|
||||||
|
result = get_ai_document_classification(mock_document)
|
||||||
|
|
||||||
|
assert result["title"] == "Test Title"
|
||||||
|
assert result["tags"] == ["test", "document"]
|
||||||
|
assert result["correspondents"] == ["John Doe"]
|
||||||
|
assert result["document_types"] == ["report"]
|
||||||
|
assert result["storage_paths"] == ["Reports"]
|
||||||
|
assert result["dates"] == ["2023-01-01"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.django_db
|
||||||
|
@patch("paperless.ai.client.AIClient.run_llm_query")
|
||||||
|
def test_get_ai_document_classification_failure(mock_run_llm_query, mock_document):
|
||||||
|
mock_run_llm_query.side_effect = Exception("LLM query failed")
|
||||||
|
|
||||||
|
result = get_ai_document_classification(mock_document)
|
||||||
|
|
||||||
|
assert result == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_llm_classification_response_valid():
|
||||||
|
mock_response = json.dumps(
|
||||||
|
{
|
||||||
|
"title": "Test Title",
|
||||||
|
"tags": ["test", "document"],
|
||||||
|
"correspondents": ["John Doe"],
|
||||||
|
"document_types": ["report"],
|
||||||
|
"storage_paths": ["Reports"],
|
||||||
|
"dates": ["2023-01-01"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
result = parse_ai_classification_response(mock_response)
|
||||||
|
|
||||||
|
assert result["title"] == "Test Title"
|
||||||
|
assert result["tags"] == ["test", "document"]
|
||||||
|
assert result["correspondents"] == ["John Doe"]
|
||||||
|
assert result["document_types"] == ["report"]
|
||||||
|
assert result["storage_paths"] == ["Reports"]
|
||||||
|
assert result["dates"] == ["2023-01-01"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_llm_classification_response_invalid_json():
|
||||||
|
mock_response = "Invalid JSON"
|
||||||
|
|
||||||
|
result = parse_ai_classification_response(mock_response)
|
||||||
|
|
||||||
|
assert result == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_llm_classification_response_partial_data():
|
||||||
|
mock_response = json.dumps(
|
||||||
|
{
|
||||||
|
"title": "Partial Data",
|
||||||
|
"tags": ["partial"],
|
||||||
|
"correspondents": "Jane Doe",
|
||||||
|
"document_types": "note",
|
||||||
|
"storage_paths": [],
|
||||||
|
"dates": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
result = parse_ai_classification_response(mock_response)
|
||||||
|
|
||||||
|
assert result["title"] == "Partial Data"
|
||||||
|
assert result["tags"] == ["partial"]
|
||||||
|
assert result["correspondents"] == ["Jane Doe"]
|
||||||
|
assert result["document_types"] == ["note"]
|
||||||
|
assert result["storage_paths"] == []
|
||||||
|
assert result["dates"] == []
|
95
src/paperless/tests/test_ai_client.py
Normal file
95
src/paperless/tests/test_ai_client.py
Normal file
@@ -0,0 +1,95 @@
|
|||||||
|
import json
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from django.conf import settings
|
||||||
|
|
||||||
|
from paperless.ai.client import AIClient
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def mock_settings():
|
||||||
|
settings.LLM_BACKEND = "openai"
|
||||||
|
settings.LLM_MODEL = "gpt-3.5-turbo"
|
||||||
|
settings.LLM_API_KEY = "test-api-key"
|
||||||
|
yield settings
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.django_db
|
||||||
|
@patch("paperless.ai.client.AIClient._run_openai_query")
|
||||||
|
@patch("paperless.ai.client.AIClient._run_ollama_query")
|
||||||
|
def test_run_llm_query_openai(mock_ollama_query, mock_openai_query, mock_settings):
|
||||||
|
mock_settings.LLM_BACKEND = "openai"
|
||||||
|
mock_openai_query.return_value = "OpenAI response"
|
||||||
|
client = AIClient()
|
||||||
|
result = client.run_llm_query("Test prompt")
|
||||||
|
assert result == "OpenAI response"
|
||||||
|
mock_openai_query.assert_called_once_with("Test prompt")
|
||||||
|
mock_ollama_query.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.django_db
|
||||||
|
@patch("paperless.ai.client.AIClient._run_openai_query")
|
||||||
|
@patch("paperless.ai.client.AIClient._run_ollama_query")
|
||||||
|
def test_run_llm_query_ollama(mock_ollama_query, mock_openai_query, mock_settings):
|
||||||
|
mock_settings.LLM_BACKEND = "ollama"
|
||||||
|
mock_ollama_query.return_value = "Ollama response"
|
||||||
|
client = AIClient()
|
||||||
|
result = client.run_llm_query("Test prompt")
|
||||||
|
assert result == "Ollama response"
|
||||||
|
mock_ollama_query.assert_called_once_with("Test prompt")
|
||||||
|
mock_openai_query.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.django_db
|
||||||
|
def test_run_llm_query_unsupported_backend(mock_settings):
|
||||||
|
mock_settings.LLM_BACKEND = "unsupported"
|
||||||
|
client = AIClient()
|
||||||
|
with pytest.raises(ValueError, match="Unsupported LLM backend: unsupported"):
|
||||||
|
client.run_llm_query("Test prompt")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.django_db
|
||||||
|
def test_run_openai_query(httpx_mock, mock_settings):
|
||||||
|
mock_settings.LLM_BACKEND = "openai"
|
||||||
|
httpx_mock.add_response(
|
||||||
|
url="https://api.openai.com/v1/chat/completions",
|
||||||
|
json={
|
||||||
|
"choices": [{"message": {"content": "OpenAI response"}}],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
client = AIClient()
|
||||||
|
result = client.run_llm_query("Test prompt")
|
||||||
|
assert result == "OpenAI response"
|
||||||
|
|
||||||
|
request = httpx_mock.get_request()
|
||||||
|
assert request.method == "POST"
|
||||||
|
assert request.headers["Authorization"] == f"Bearer {mock_settings.LLM_API_KEY}"
|
||||||
|
assert request.headers["Content-Type"] == "application/json"
|
||||||
|
assert json.loads(request.content) == {
|
||||||
|
"model": mock_settings.LLM_MODEL,
|
||||||
|
"messages": [{"role": "user", "content": "Test prompt"}],
|
||||||
|
"temperature": 0.3,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.django_db
|
||||||
|
def test_run_ollama_query(httpx_mock, mock_settings):
|
||||||
|
mock_settings.LLM_BACKEND = "ollama"
|
||||||
|
httpx_mock.add_response(
|
||||||
|
url="http://localhost:11434/api/chat",
|
||||||
|
json={"message": {"content": "Ollama response"}},
|
||||||
|
)
|
||||||
|
|
||||||
|
client = AIClient()
|
||||||
|
result = client.run_llm_query("Test prompt")
|
||||||
|
assert result == "Ollama response"
|
||||||
|
|
||||||
|
request = httpx_mock.get_request()
|
||||||
|
assert request.method == "POST"
|
||||||
|
assert json.loads(request.content) == {
|
||||||
|
"model": mock_settings.LLM_MODEL,
|
||||||
|
"messages": [{"role": "user", "content": "Test prompt"}],
|
||||||
|
"stream": False,
|
||||||
|
}
|
70
src/paperless/tests/test_ai_matching.py
Normal file
70
src/paperless/tests/test_ai_matching.py
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from django.test import TestCase
|
||||||
|
|
||||||
|
from documents.models import Correspondent
|
||||||
|
from documents.models import DocumentType
|
||||||
|
from documents.models import StoragePath
|
||||||
|
from documents.models import Tag
|
||||||
|
from paperless.ai.matching import extract_unmatched_names
|
||||||
|
from paperless.ai.matching import match_correspondents_by_name
|
||||||
|
from paperless.ai.matching import match_document_types_by_name
|
||||||
|
from paperless.ai.matching import match_storage_paths_by_name
|
||||||
|
from paperless.ai.matching import match_tags_by_name
|
||||||
|
|
||||||
|
|
||||||
|
class TestAIMatching(TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
# Create test data for Tag
|
||||||
|
self.tag1 = Tag.objects.create(name="Test Tag 1")
|
||||||
|
self.tag2 = Tag.objects.create(name="Test Tag 2")
|
||||||
|
|
||||||
|
# Create test data for Correspondent
|
||||||
|
self.correspondent1 = Correspondent.objects.create(name="Test Correspondent 1")
|
||||||
|
self.correspondent2 = Correspondent.objects.create(name="Test Correspondent 2")
|
||||||
|
|
||||||
|
# Create test data for DocumentType
|
||||||
|
self.document_type1 = DocumentType.objects.create(name="Test Document Type 1")
|
||||||
|
self.document_type2 = DocumentType.objects.create(name="Test Document Type 2")
|
||||||
|
|
||||||
|
# Create test data for StoragePath
|
||||||
|
self.storage_path1 = StoragePath.objects.create(name="Test Storage Path 1")
|
||||||
|
self.storage_path2 = StoragePath.objects.create(name="Test Storage Path 2")
|
||||||
|
|
||||||
|
@patch("paperless.ai.matching.get_objects_for_user_owner_aware")
|
||||||
|
def test_match_tags_by_name(self, mock_get_objects):
|
||||||
|
mock_get_objects.return_value = Tag.objects.all()
|
||||||
|
names = ["Test Tag 1", "Nonexistent Tag"]
|
||||||
|
result = match_tags_by_name(names, user=None)
|
||||||
|
self.assertEqual(len(result), 1)
|
||||||
|
self.assertEqual(result[0].name, "Test Tag 1")
|
||||||
|
|
||||||
|
@patch("paperless.ai.matching.get_objects_for_user_owner_aware")
|
||||||
|
def test_match_correspondents_by_name(self, mock_get_objects):
|
||||||
|
mock_get_objects.return_value = Correspondent.objects.all()
|
||||||
|
names = ["Test Correspondent 1", "Nonexistent Correspondent"]
|
||||||
|
result = match_correspondents_by_name(names, user=None)
|
||||||
|
self.assertEqual(len(result), 1)
|
||||||
|
self.assertEqual(result[0].name, "Test Correspondent 1")
|
||||||
|
|
||||||
|
@patch("paperless.ai.matching.get_objects_for_user_owner_aware")
|
||||||
|
def test_match_document_types_by_name(self, mock_get_objects):
|
||||||
|
mock_get_objects.return_value = DocumentType.objects.all()
|
||||||
|
names = ["Test Document Type 1", "Nonexistent Document Type"]
|
||||||
|
result = match_document_types_by_name(names, user=None)
|
||||||
|
self.assertEqual(len(result), 1)
|
||||||
|
self.assertEqual(result[0].name, "Test Document Type 1")
|
||||||
|
|
||||||
|
@patch("paperless.ai.matching.get_objects_for_user_owner_aware")
|
||||||
|
def test_match_storage_paths_by_name(self, mock_get_objects):
|
||||||
|
mock_get_objects.return_value = StoragePath.objects.all()
|
||||||
|
names = ["Test Storage Path 1", "Nonexistent Storage Path"]
|
||||||
|
result = match_storage_paths_by_name(names, user=None)
|
||||||
|
self.assertEqual(len(result), 1)
|
||||||
|
self.assertEqual(result[0].name, "Test Storage Path 1")
|
||||||
|
|
||||||
|
def test_extract_unmatched_names(self):
|
||||||
|
llm_names = ["Test Tag 1", "Nonexistent Tag"]
|
||||||
|
matched_objects = [self.tag1]
|
||||||
|
unmatched_names = extract_unmatched_names(llm_names, matched_objects)
|
||||||
|
self.assertEqual(unmatched_names, ["Nonexistent Tag"])
|
Reference in New Issue
Block a user