diff --git a/.agents/skills/translate/glossary.md b/.agents/skills/translate/glossary.md index e242c26347..e754578086 100644 --- a/.agents/skills/translate/glossary.md +++ b/.agents/skills/translate/glossary.md @@ -41,6 +41,7 @@ SurfSense, Studio, Markdown, llama.cpp, GGUF, Hugging Face, API, MCP, MCP's tran | quiz | クイズ | 퀴즈 | 测验 | क्विज़ | Quiz † | cuestionario | Quiz | Quiz | Тест | | study guide | 学習ガイド | 학습 가이드 | 学习指南 | स्टडी गाइड | Lernzettel † | guía de estudio | Guide de révision | Guia de estudos | конспект | | podcast | ポッドキャスト | 팟캐스트 | 播客 | पॉडकास्ट | Podcast | podcast | Podcast | Podcast | подкаст | +| comic (the Studio format; its kinds keep their own names) | コミック | 만화 | 漫画 | कॉमिक | Comic | cómic | BD | quadrinhos | комикс | | speech (a model reading text aloud) | 音声合成 | 음성 | 语音 | वॉइस | Sprache | voz | voix | voz | речь | | speech recognition (turning recordings and the voice into text, on this computer) | 音声認識 | 음성 인식 | 语音识别 | स्पीच रिकग्निशन | Spracherkennung | reconocimiento de voz | reconnaissance vocale | reconhecimento de voz | распознавание речи | | voice input (dictating into the chat composer) | 音声入力 | 음성 입력 | 语音输入 | वॉइस इनपुट | Spracheingabe | entrada de voz | saisie vocale | entrada de voz | голосовой ввод | diff --git a/docs/adr/0077-an-image-edit-is-the-next-version.md b/docs/adr/0077-an-image-edit-is-the-next-version.md index 60423380cc..12d1700208 100644 --- a/docs/adr/0077-an-image-edit-is-the-next-version.md +++ b/docs/adr/0077-an-image-edit-is-the-next-version.md @@ -1,6 +1,6 @@ # ADR 0077: An image edit is the next version of the same document -- **Status:** Accepted +- **Status:** Accepted; its consequence on models that cannot edit superseded by [ADR 0089](0089-any-image-model-changes-a-picture.md) - **Date:** 2026-10-31 - **Supersedes:** the removal of Studio's earlier image Edit, which made a new artifact per edit - **Source:** maintainer decision, 31 Oct 2026 (no permalink) diff --git a/docs/adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md b/docs/adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md index 5c43055cdb..77e9d1b9ff 100644 --- a/docs/adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md +++ b/docs/adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md @@ -1,6 +1,6 @@ # ADR 0087: The agent makes pictures with its own SurfSense tool, not opencode's -- **Status:** Accepted +- **Status:** Accepted; its waiting for the user's next message superseded by [ADR 0090](0090-a-document-waits-on-the-server-for-its-pictures.md) - **Date:** 2026-11-07 - **Amends:** [ADR 0066](0066-web-pages-are-html-the-model-writes-without-network.md), in part: its rule that nothing is written into a page but the policy - **Source:** maintainer decision, 7 Nov 2026 (no permalink) diff --git a/docs/adr/0089-any-image-model-changes-a-picture.md b/docs/adr/0089-any-image-model-changes-a-picture.md new file mode 100644 index 0000000000..991e398c91 --- /dev/null +++ b/docs/adr/0089-any-image-model-changes-a-picture.md @@ -0,0 +1,23 @@ +# ADR 0089: Any image model changes a picture: one that takes pictures edits it, any other draws it again + +- **Status:** Accepted +- **Date:** 2026-11-10 +- **Supersedes:** in part, [ADR 0077](0077-an-image-edit-is-the-next-version.md): its consequence that a picture drawn by a model that cannot edit needs an edit model chosen before it can be edited +- **Source:** maintainer decision, 10 Nov 2026 (no permalink) + +## Context + +An edit ran only on the `image_edit` selection, which few local models can fill and most users never chose; an empty slot answered "Needs an image editing model" and sent them to Settings. Most remote image models take a picture in, but the image model the user had chosen was never asked. OpenRouter has no `/images/edits`, so an edit through it failed. A comic needs every page to keep page 1's characters, which needs a picture in, not a second model. + +## Decision + +- Whether a model takes a picture in is a fact about the model, read from its connection's model list or the catalog when it is chosen, not a slot the user fills. +- One rule picks the model for a picture in: the `image_edit` selection when chosen, as an override; else the image model when it takes pictures; else none. +- With none, an edit is drawn again by the image model from the prompt the picture was drawn with and the change, and says so in one plain line. No edit is refused, and nothing in Studio or chat points to Settings for it. +- A picture in goes to `/images/edits`, or to `/images` with `input_references` where a server has no edit route, so OpenRouter edits too. + +## Consequences + +- Most users choose one image model and edit with it; the edit slot is optional. +- A redraw does not keep the base's composition; the version's first line and the Edit box's placeholder say so. +- Every drawing keeps the prompt it sent, so a redraw has it, and changes add up from version to version. diff --git a/docs/adr/0090-a-document-waits-on-the-server-for-its-pictures.md b/docs/adr/0090-a-document-waits-on-the-server-for-its-pictures.md new file mode 100644 index 0000000000..f50784e060 --- /dev/null +++ b/docs/adr/0090-a-document-waits-on-the-server-for-its-pictures.md @@ -0,0 +1,22 @@ +# ADR 0090: A document waits on the server for the pictures it places + +- **Status:** Accepted +- **Date:** 2026-11-10 +- **Supersedes:** in part, [ADR 0087](0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md): "A document needing a picture still being drawn waits for the user's next message" +- **Source:** maintainer decision, 10 Nov 2026 (no permalink) + +## Context + +A local picture on a processor takes minutes, past a tool call's limit, so its call answers queued. A render naming it was refused, and the agent could only tell the user to ask again: a four-page comic drawn on this computer never became one PDF on its own. A tool that polls, or a turn kept open, would hold a call past its limit, which ADR 0087 rules out. + +## Decision + +- A render naming a picture still being drawn is made pending with the pictures it waits for, unqueued, and its call answers at once. +- The end of every Studio job queues the renders waiting on that picture once all of theirs are drawn, and fails one whose picture failed, was cancelled or deleted, with a sentence naming it. The worker does the same when it starts. +- Only a render waits. A web page still refuses a picture being drawn, since it is checked and screenshotted as it is saved. + +## Consequences + +- A comic drawn slowly ends as a PDF without the user asking again. +- No process, timer or model stays awake for it: the job that ends is the one event that can make the render ready. +- No one looks at a waiting render's pages, since it finishes after its turn; a comic's PDF is a fixed layout. diff --git a/docs/adr/README.md b/docs/adr/README.md index 7db13256de..aaa7aa90a5 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -117,3 +117,5 @@ The Date is the day the decision was first written down. Source links are permal | 0086 | [The contract redline ships with the Business switch, as an app runner on SurfSense's own Word engine](0086-the-contract-redline-ships-with-the-business-switch-on-the-word-engine.md) | Accepted | | 0087 | [The agent makes pictures with its own SurfSense tool, not opencode's](0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md) | Accepted | | 0088 | [The app draws Word and Excel files with one renderer](0088-the-app-draws-word-and-excel-files-with-one-renderer.md) | Accepted | +| 0089 | [Any image model changes a picture: one that takes pictures edits it, any other draws it again](0089-any-image-model-changes-a-picture.md) | Accepted | +| 0090 | [A document waits on the server for the pictures it places](0090-a-document-waits-on-the-server-for-its-pictures.md) | Accepted | diff --git a/docs/architecture/agent.md b/docs/architecture/agent.md index 9c36c66e42..422466aa24 100644 --- a/docs/architecture/agent.md +++ b/docs/architecture/agent.md @@ -164,7 +164,8 @@ Deleting a thread aborts and deletes its session, disposes its opencode instance | `surfsense_create_artifact` | `format`, one of Studio's formats except `docx`, `pdf`, `pptx`, `xlsx`, `html`, `image` and `infographic`; `source_ids`, the numbers at the end of the source files' names; `instructions`, at most 2,000 characters | Studio's own `create_artifact_job()`, which returns at once with the artifact pending. A source not ticked for the turn is refused. Asked for one of the four Office formats, it refuses and names `surfsense_render_document`, because Studio's draft keeps no script to edit; asked for `html`, it names `surfsense_save_page` and the HTML skill | | `surfsense_render_document` | `title`; `format`, `docx`, `pdf`, `pptx` or `xlsx`; `script`; `artifact_id`, to make the next version of, refused for any version of a document Studio made, which Refine rewrites in Studio, and for a revised copy or a converted PDF, which keep no script and have no Refine, naming `surfsense_revise_document` or the Word version or deck to change and convert again ([`unscripted_base.py`](../../surfsense_local/backend/modules/artifacts/script_documents/unscripted_base.py)); `images`, names from `surfsense_list_images`, of ticked sources only; `template_source_id`, a ticked source's own `.docx` for `docx` or `.pptx` for `pptx`, kept by later versions | runs the script as a new version in Studio and waits for it ([below](#documents-the-agent-makes)) | | `surfsense_save_page` | `path`, the page's file under the thread's `outputs/`; `title`; `artifact_id`, to make a page's next version | keeps the page as a version in Studio and waits for it to be indexed ([web pages the agent makes](#web-pages-the-agent-makes)): the `Rendered artifact , version : ` line, the page's size and opening text, the outside resources the viewer will block, up to four screenshots, and the working file to edit next | -| `surfsense_make_image` | `prompt`; `title`, needed for a new picture; `style`, `image` or `infographic`; `artifact_id`, to edit a picture this chat made | a picture kept in Studio and waited for ([pictures the agent makes](#pictures-the-agent-makes)): the `Rendered artifact <id>, version <n>: <title>` line, its name `image-<id>`, and the picture itself when the model reads images; or `Queued artifact …` | +| `surfsense_bind_comic` | `title`; `pages`, each `{image, lines}` in reading order; `layout`, `pages` or `scroll`; `direction`, `ltr` or `rtl`; `artifact_id`, for a comic's next version | a `comic` in Studio, its pages bound into one PDF ([comics the agent binds](studio.md#comics-the-agent-binds)): the `Rendered artifact <id>, version <n>: <title>` line; or `Queued artifact …` while a page is still being drawn | +| `surfsense_make_image` | `prompt`; `title`, needed for a new picture; `type`, `image` or `infographic`; `comic_page`, for a page of a comic, which Studio does not list; `artifact_id`, to edit a picture this chat made | a picture kept in Studio and waited for ([pictures the agent makes](#pictures-the-agent-makes)): the `Rendered artifact <id>, version <n>: <title>` line, its name `image-<id>`, and the picture itself when the model reads images; or `Queued artifact …` with its name, which a document or comic can place at once, waiting for it on the server, and a web page only once it is drawn | | `surfsense_read_document` | `artifact_id`, of any version, or `document_id`, a workbook source; `sheet`, a workbook's sheet; `offset`, an optional 1-based line, or a workbook's row, to start from | the agent's newest version's number (a lineage it continued from a Studio draft before such renders were refused skips Studio's later versions), artifact id, title, format, status and the images it places, and its script in pages of at most 1,000 lines and 40,000 bytes under "Script, lines a-b of n", ending with the offset of the next page, so a result stays under opencode's truncation limits ([`script_page.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/script_page.py)). A document Studio made is refused, as a render continuing one is: "This document was made in Studio: refine it there, or make a new document." Given the `artifact_id` of a revised copy, it reads the copy's newest ready version as it reads now ([`read_revised_copy.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/read_revised_copy.py)): a Word copy's body with every author's tracked changes marked inline as `<ins author="…">` and `<del author="…">`, `[comment N]` after the text each comment is on, then each comment with its author, whether it is internal, and that text ([`word_marked_text.py`](../../surfsense_local/backend/modules/artifacts/revised_copies/engines/word_marked_text.py)), under its counts of tracked changes and comments; a deck's text slide by slide; a workbook's cells as below. Word and deck text come in pages of lines as a script does. Excel and PowerPoint keep no tracked changes, so their opening counts the edits applied over the copy's versions instead. Every answer ends its opening with the `artifact_id` to revise. Given `document_id`, a ticked `.xlsx` or `.xlsm` source, or the `artifact_id` of a revised copy of one, it returns the workbook's cells instead, read from the file itself (the source's original, or the copy's newest ready version) with openpyxl in read-only mode and formulas as written: each sheet's name and used range, then each non-empty cell as `B7 = 1250.5`, `A1 = "text"` or `C3 = =SUM(C2:C2)`, under the same 40,000 bytes ([`read_workbook.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/read_workbook.py), [`cell_map.py`](../../surfsense_local/backend/modules/artifacts/revised_copies/cell_map.py)). `sheet` reads one sheet and `offset` is then the row to start from; a cut map ends with the row and the call that reads on, or, when one row holds more cells than a map shows, the first cell left out and the next row to read on from. The file is read once the tool's database reads have ended, so a big workbook's seconds of reading never hold SQLite's write lock. Sheet headings take at most a quarter of the map; the sheets past that are only named, up to 4,000 bytes, then counted. The indexed text of a workbook is Docling's tables, without sheet names, addresses or formulas, so the revise tool and the revisions skill send the agent here first. Any other source is refused, pointing at `sources/`. | | `surfsense_list_images` | `source_ids` | each source's figures, one line each: its name, size in pixels, page and caption when known, and its copy under `sources/figures/` to open with `read`; or that the source has none, is not in this workspace, or has its figures being extracted, which queues the figures-only pass ([documents](documents.md#figures)). A source not ticked for the turn is refused | | `surfsense_source_pages` | `document_id`; `pages`, at most four page or slide numbers from 1, the first ones when left out | a PDF, Word or PowerPoint source's own pages as images with the result, in ascending order, with the source's page or slide count; copies at most 1,000 pixels on their long side stay in `sources/pages/<id>-p<n>.png` for a closer look ([below](#source-pages)). Offered only when `opencode.json` declares image input. A source not ticked for the turn, an image, a note or any other kind of file is refused in a sentence saying what to use instead | @@ -217,13 +218,17 @@ The step links the version in the thread as a render's does (`surfsense_save_pag [`make_image.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/make_image.py) draws a picture from a prompt the agent wrote, with no sources and no chat model ([ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)), through Studio's job ([drawn pictures](studio.md#pictures-the-agent-asks-for)). Within one 190-second deadline it: -1. **Refuses** a missing argument, a prompt over 32,000 characters, only to catch a runaway model, no image model or, for an edit, no edit model, no embedding model, a title this chat already gave a picture of that style, and an `artifact_id` this chat did not make. None is a failed run. +1. **Refuses** a missing argument, a prompt over 32,000 characters, only to catch a runaway model, no image model, no embedding model, a title this chat already gave a picture of that style, an `artifact_id` this chat did not make, and a reference that is not a ready picture of this chat, more than four, or given with `artifact_id`. None is a failed run. 2. **Waits** up to 150 seconds. A failed picture is a failed run, under its own three-failure limit and never retried by the worker, since each try may be billed. A queued picture that fails after its call answered counts when the next one is asked for. Calls in one step are held apart while each checks its title and starts, so two with one title draw one picture. -3. **Answers** with the picture's name, and the picture as a tool result's image when the model reads images. It offers an edit only when an edit model is chosen; without one, it tells the agent to say what is wrong and point the user to Settings › Image edit, so the agent neither calls an edit that would be refused nor bills a redraw. A picture in a format documents cannot place, SVG or GIF, gets no name and says so. +3. **Answers** with the picture's name, where it was drawn ("Drawn on this computer" or "Drawn by <connection>"), and the picture as a tool result's image when the model reads images. It always offers a change: with no model that takes a picture in ([which model](studio.md#which-model-takes-a-picture)), it says the change is drawn again from the prompt; it never points to Settings. A reference drawn from words alone, or a change redrawn, says so in a line. A picture in a format documents cannot place, SVG or GIF, gets no name and says so. + +**Shape and references.** `shape` (`square`, `portrait`, `landscape`) reaches the image model ([shapes](studio.md#shapes-and-references)). `references`, up to four picture names of this chat, send those pictures with the prompt to the model that takes pictures, so a comic's later pages keep page 1's characters; with none, the picture is drawn from its prompt and the result says the characters may differ. The tool's listing stays the same whatever models are chosen, so a local model's prompt cache holds. + +**Comics and sets.** The `surfsense-pictures` skill ([`skills/surfsense-pictures/`](../../surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md)) is the craft beside the tool, as `surfsense-documents` is beside the render: one picture, or a set such as a comic, a storyboard or a picture book. A set is planned with a character sheet, confirmed once, drawn first page alone and the rest with it as reference, and bound by `surfsense_bind_comic` into one `comic` in Studio, with the binder Studio's comics use: one full-bleed picture a page, or, for a webtoon, the pictures stacked down one long page; marked right to left for a manga; placed as JPEG, with a page's exact lines set in a strip under it when its lettering came out wrong or it was drawn without words. `create_artifact` does not offer `comic`, so the agent never starts a comic whose pages it cannot look at. The prompt sends pictures and comics there, and the web page skill's description sends them away. **Prompt length.** As in Studio, nothing limits a prompt per model: the tool's description asks for one self-contained prompt, as short as it says what to draw, and a model's own refusal comes back as a failed picture. -**A slow picture.** A local picture on a processor takes minutes, past a tool call's hard limit, so the call answers `Queued artifact …` with the name it will have, and no tool polls. With a remote chat model the agent answers at once, and a render or page naming the picture is refused as still being drawn until it is ready. With a local chat model, the agent's next step waits in the model endpoint behind [the image's hold](local-models/admission.md#the-image-jobs-hold), so the picture is usually ready when the agent goes on. +**A slow picture.** A local picture on a processor takes minutes, past a tool call's hard limit, so the call answers `Queued artifact …` with the name it will have, and no tool polls. A render naming it is not refused: it is made pending with `waits_for`, unqueued, and answers at once that it renders once the picture is drawn ([`waiting_renders/`](../../surfsense_local/backend/modules/artifacts/waiting_renders/)). The end of every Studio job queues the renders waiting on that picture once all theirs are ready, or fails one whose picture failed, was cancelled or deleted, with a sentence naming it; one update takes `waits_for` off, so two pictures ending together queue it once. Cancelling a picture settles them too, since a cancelled job still queued is revoked and never reaches its end. A settle that fails at a job's end is logged, not raised, since a raise would have Huey run the job and bill its picture again. The Studio worker settles every waiting render when it starts, after failing interrupted jobs. A web page still refuses a picture being drawn, since it is checked and screenshotted as it is saved. With a local chat model, the agent's next step waits in the model endpoint behind [the image's hold](local-models/admission.md#the-image-jobs-hold), so the picture is usually ready when the agent goes on. **Placing one.** `image-<artifact id>` names one version, so it keeps meaning the same picture as edits are made. It works in a render's `images` and in a page ([`names.py`](../../surfsense_local/backend/modules/artifacts/drawn_images/names.py)), only in the chat that made the picture, as a chart's name does. Finding it reads the database; each use then writes it afresh as a PNG to `outputs/images/<name>.png`, outside any transaction, converted if the model answered in another format and refused past 40 million pixels, since the agent can write to its own outputs and what is placed must be what the model drew. A page's size is worked out before any image is written into it, since a small file can name one picture thousands of times. @@ -372,8 +377,9 @@ A document script is another matter: it runs in the Studio worker's runner, whic - The model endpoint neither simplifies tool schemas to what llama.cpp's grammar takes nor turns a tool call a model writes as text into a real one, so small local models stall where a remote one would not. - A file the agent writes to `outputs/` with its own tools does not become an artifact; only `surfsense_render_document`, `surfsense_revise_document`, `surfsense_convert_document`, `surfsense_make_image` and the PDF tools make one. An analysis's tables and charts stay in `outputs/analysis/` too, outside Studio, until a document places them. - An analysis waits on the Studio worker's four threads. Behind four long Studio jobs it is dropped after 60 seconds unstarted, and the call waits out its 190 seconds before saying so. -- A picture still being drawn when its call answers leaves the agent no way to wait or come back: a document that needs it is made on the user's next message. A local chat model's step waiting out a whole local picture through the agent's model endpoint is not measured. -- Lettering an image model draws is not measured, so one picture holding a comic's speech bubbles or a page of text may garble them. The tool's description sends exact words to the document around the picture. +- A local chat model's step waiting out a whole local picture through the agent's model endpoint is not measured. A render that waits on a picture finishes after its turn, so no one looks at its pages. +- A waiting render that fails does not count toward renders' three-failure stop. +- Nothing on the server limits pictures per turn; only the comics skill's twelve pages do. - A Studio job the agent starts with `surfsense_create_artifact` runs on the same local model as the agent, so the agent's next step waits behind it. - An artifact the agent starts with `surfsense_create_artifact` records neither the thread nor the step that made it, and one it renders records its thread but not the step: the tool call reaches SurfSense without the step. - A thread's folder narrows what its agent can reach to its own sources, but does not close it. Through a hole in opencode, below, an agent can still read what is outside its folder: opencode's tool-output folder is shared by all, and closing it needs a change in opencode. Through SurfSense's own tools it can too: a script `surfsense_render_document` or `surfsense_analyze_data` runs can read any file the user can ([Studio](studio.md#known-gaps)), and the text the render returns shows what it put in the document, another thread's sources included; and `surfsense_read_document` reads any document the agent rendered in the workspace, from any thread and filed in an unticked folder too, and such a script can hold what that thread's agent took from its own sources. diff --git a/docs/architecture/local-models/selection.md b/docs/architecture/local-models/selection.md index 523e91efaf..4ec7572646 100644 --- a/docs/architecture/local-models/selection.md +++ b/docs/architecture/local-models/selection.md @@ -22,7 +22,7 @@ type is the slot. |---|---|---|---| | `text_gen` | `llamacpp`, the bundled runtime | `openai_compatible`, with a `connection_id` | chat, titles, Studio's writing | | `image_gen` | `sdcpp`, the bundled sd-server | `openai_compatible`, with a `connection_id` | Studio's `image` and `infographic` | -| `image_edit` | `sdcpp`, a model whose entry names `edit` | `openai_compatible`, with a `connection_id` | Studio's image and infographic edits ([studio](../studio.md#editing-a-picture)) | +| `image_edit` | `sdcpp`, a model whose entry names `edit` | `openai_compatible`, with a `connection_id` | Optional: edits and references with a model other than the image model; empty, the image model takes them when it takes pictures in, and redraws when not ([studio](../studio.md#which-model-takes-a-picture)) | | `audio_gen` | `audiocpp`, the bundled audio.cpp server | `openai_compatible`, with a `connection_id` | Studio's `podcast` | | `video_gen` | `sdcpp`, a model whose entry has a `video` block | `openai_compatible`, with a `connection_id` | nothing yet | | `speech_to_text` | `audiocpp`, a recogniser, run on a second audio.cpp server | none: local only | the transcription routes ([admission](admission.md#speech-recognition)), which a recording's ingest calls ([documents](../documents.md#recordings)) | @@ -347,6 +347,7 @@ language, and `test_connections.py` that no connection fills it. ## Known gaps -- Only the quiz, flashcards and simple web page pass `json_schema`: image, infographic and the podcast's outline and draft still ask for JSON in the prompt alone, so their format compliance depends on it. The mind map is not JSON: it is a Markdown outline, written free. +- Settings shows the edit override as its own section, "Use a different model for edits", not folded under Image gen; and the **Image editing** badge is on server models only, not on local models to download. +- Only the quiz, flashcards, simple web page and comic plan pass `json_schema`: image, infographic and the podcast's outline and draft still ask for JSON in the prompt alone, so their format compliance depends on it. The mind map is not JSON: it is a Markdown outline, written free. - A schema whose node refers to itself, for a tree of any depth, crashed llama-server at b11050 twice in nine replies as Qwen3 0.6B nested deeper. No Studio schema refers to itself. - Nothing measures whether three tiers are still needed; once constrained decoding carries format compliance, a tier would carry reasoning depth only, which plausibly collapses three tiers to two. diff --git a/docs/architecture/studio.md b/docs/architecture/studio.md index 77420ce62d..a5a66b3bf8 100644 --- a/docs/architecture/studio.md +++ b/docs/architecture/studio.md @@ -21,7 +21,7 @@ Status lives on the document, not the sidecar, because ADR 0003 keeps indexing s ## Formats -The catalog is a tuple of twelve `Format` rows in [`formats.py`](../../surfsense_local/backend/modules/artifacts/formats.py), computed into availability on each request and never stored. +The catalog is a tuple of thirteen `Format` rows in [`formats.py`](../../surfsense_local/backend/modules/artifacts/formats.py), computed into availability on each request and never stored. | Key | Label | Needs | Worker path | Stored file | |---|---|---|---|---| @@ -37,6 +37,9 @@ The catalog is a tuple of twelve `Format` rows in [`formats.py`](../../surfsense | `podcast` | Podcast | text_gen, audio_gen | `media/audio/podcast/` | MP3 audio, or WAV without a working encoder | | `image` | Image | image_gen, text_gen | `media/visual/image/` | the image | | `infographic` | Infographic | image_gen, text_gen | `media/visual/infographic/` | the image | +| `comic` | Comic | image_gen, text_gen | `media/visual/comic/` | `.pdf`, the bound pages | + +- The panel shows `infographic` as a type of `image`: one Image card, whose dialog chooses Image or Infographic and sends that format's key ([`picture-types.ts`](../../surfsense_local/frontend/src/features/studio/picture-types.ts)). An infographic card shows only when the catalog has no `image`. A format's own options in the dialog, the podcast brief or the picture type, come from [`format-options/`](../../surfsense_local/frontend/src/features/studio/format-options/). - `requires_model_types` is a tuple in the order the pipeline's `render()` takes its models. A format is available when every required model type has a selection; otherwise the reason names every missing one in a fixed reading order: "Needs a chat model", "Needs an image model", "Needs an audio model", or two of them joined, as "Needs a chat model and an audio model". Naming only the first missing type made selecting it look like the gate moving to the other. - `podcast` runs with an `audio_gen` model on this computer or on a server. The format list and the brief read the model's voices without calling it, so neither needs the server's host allowed; the job does. Opening a podcast brief without an audio model answers `409` with "Needs an audio model" and the code `needs_audio`, in the same `{message, code}` detail as a refused create. The app ships Kokoro and chooses it at startup when no audio model is chosen, so that answer needs the choice cleared since the last start, or a build without audio.cpp ([`local-models/selection.md`](local-models/selection.md)). @@ -62,6 +65,7 @@ Eight formats follow it: - **podcast**: the model outlines the episode from the brief, then drafts it segment by segment. Every segment sends the same one-line system prompt and the sources, and its whole draft prompt after them, because llama-server reuses a prompt only up to its first changed token. Measured through the drafting code on Qwen3 1.7B with about 6,200 tokens of sources: with the draft prompt first, every segment re-read the sources; after them, segments 2 to 4 read about 230 tokens each. The draft prompt goes after them whole: split, with what the segments share left ahead of the sources, the same model wrote half the dialogue, in long lines. Each reply is capped at 12 tokens per word of the segment's target, and never under a planned 250-word segment's worth. Uncapped, Qwen3 1.7B looped on a 225-word segment until its 40,960-token window was full, and the JSON retry, which replays the failed reply, could not fit. The target is the outline's own guess: Qwen3 1.7B once gave a segment 20 words, wrote past them, and a 240-token cap ended both replies mid-JSON. The chosen audio model voices every line through audio.cpp's server, and the transcript is the body. - **image**: the model writes a title and an image prompt, and the image model paints the prompt. - **infographic**: the model writes a factual brief (a title, a summary and up to 8 sections of label, value and detail), and the image model paints a prompt built from it in a fixed sketchnote style. +- **comic**: options `{kind, pages}`, one of ten kinds ([`kinds.py`](../../surfsense_local/backend/modules/artifacts/comic/kinds.py)) and 1 to 12 pictures, checked when the job is created. The model plans a title, a style line, a character sheet and each page's panels and lines, asked for in [`schema.py`](../../surfsense_local/backend/worker/studio/media/visual/comic/schema.py)'s shape; the plan is cut to the count asked for, since each picture is billed or takes minutes, and one short of it is refused. A reply that is not a plan is asked for once more with the reply replayed ([`asking.py`](../../surfsense_local/backend/worker/studio/media/visual/comic/asking.py)), which costs no picture. A second reply with no JSON in it is the model declining, often over the sources themselves, so the job fails with the model's first line rather than a parse error. Page 1 is drawn alone, the rest with page 1 as their reference by [the model that takes a picture in](#which-model-takes-a-picture), side by side on a remote model and one at a time on this computer; with none, each is drawn from its words and the version keeps `drawn_from_words`. A remote model letters each page's lines in bubbles; when any page is drawn by a local model, the image model or the one drawing pages 2 on, no page carries words, its lettering being unmeasured, and every page's lines are set in a strip under it, as a picture book's and a storyboard's always are. The strip sets letters Helvetica lacks in the Word and PDF builder's fonts ([`pdf_fonts.py`](../../surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py)), and a word too wide for a line, as Chinese or Japanese is, is cut between letters. The pages are bound by [`comic_binding/`](../../surfsense_local/backend/worker/studio/media/visual/comic_binding/), which the agent's comics share; the body is every page's panels and lines. A failed comic is never retried by the worker, since every page would be drawn again. No page is kept on its own, so a Studio comic has no Edit: Regenerate draws it again. The small-model (compact) prompts ask for a count and the schema holds the model to it ([`study_count.py`](../../surfsense_local/backend/worker/studio/content/study_count.py)): one card per 600 characters of sources and one question per 1,200, so the 12,000 characters the smallest window holds ask for the full 20 and 10. Left to stop on its own, Qwen3 0.6B closed a deck after one card. The larger tiers' prompts leave the count to the material, so their schema asks for at least one. A flashcard, quiz or mind map with nothing left after the builder's checks fails the job with a reason, and the job's one retry asks again, rather than opening empty. @@ -242,7 +246,7 @@ worker/studio/ ├── web/html/ └── media/ ├── audio/podcast/ outline, draft, roster - └── visual/ image, infographic, drawn: a prompt the agent wrote, and edit: any picture changed to an instruction + └── visual/ image, infographic, comic, drawn: a prompt the agent wrote, edit: any picture changed to an instruction, and comic_binding: pages bound into a PDF ``` Every pipeline returns a `Built`: a `title`, the `markdown` that is always the indexed body, and an optional `primary` (and `preview`) with its MIME type and filename. Adding a format is a folder with a `render`, a `case` in `job_router.py` and a row in `formats.py`. Frontend entries below customize its translated label, icon, tooltip and viewer; an unknown key still appears with the server's label, a file icon, the generic tooltip and the document viewer. @@ -250,7 +254,7 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i ## Jobs - `studio_job` runs on the `studio` Huey queue in `huey.db` with `retries=1`. Every path that makes a version commits its row and then queues its job through one helper ([`queued_job.py`](../../surfsense_local/backend/modules/artifacts/queued_job.py)), since the worker is another process that must find the row. A job that cannot be queued fails its version with "This could not be queued. Retry it." rather than leaving it `pending` with no job to end it, which would refuse every Retry and edit. `worker-studio` drains it with four threads, because the work mostly waits on a model ([ADR 0008](../adr/0008-two-job-queues.md)). -- A job ([`job.py`](../../surfsense_local/backend/worker/studio/job.py)) marks the document `processing` unless it was cancelled, resolves one model per required model type, gathers the sources to the text model's window, and commits before rendering, so no write lock is held across a generation that can take minutes. It checks for a cancel before and after rendering. +- A job ([`job.py`](../../surfsense_local/backend/worker/studio/job.py)) marks the document `processing` unless it was cancelled, resolves one model per required model type, commits, asks the API for the text model's window, gathers the sources to it, and commits again before rendering, so no write lock is held across a generation that can take minutes. The first commit is because the API needs the write lock to answer: held, every window lookup waited out the busy timeout, failed as `database is locked`, and failed the API's other requests with it. It checks for a cancel before and after rendering. - [`persist.py`](../../surfsense_local/backend/worker/studio/shared/persist.py) chunks and embeds the body with the ingest code, with no transaction open, then sets the document's title and markdown and indexes that body, so the artifact is searchable and citable. If the format has a file, it clears the artifact's folder and file rows and writes the file named by its role, recording its size and SHA-256. No pipeline writes a `preview` yet. - The document is created without a `dedup_key`, so it is never deduplicated against another document. - A failure rolls back and writes `failed` with a reason cut to 500 characters: the error's first line, or for an HTTP error its whole message after "The model could not be reached: ". A model that refused a request past its window was reached, so that refusal (llama-server's `exceed_context_size_error`, or the `context_length_exceeded` OpenAI and servers copying it send) reads, for a draft, as the [window budget](#the-window-budget)'s "don't fit" sentence, and for a refine or a job run, "The request was longer than the model's context window. Choose a model with a larger context window." The job is then re-raised for Huey's one retry, except an image error, since the endpoint may already have generated, and billed, an image, a podcast refused for memory, since a retry would draft the episode again and refuse again, a request past the model's window or sources it cannot hold, since the same request is refused the same way, any failure of a script document ([above](#script-documents)) of a converted PDF ([above](#converted-pdfs)) or of a revised copy ([above](#revised-copies)), and any failure of a refine ([Refine](#refine)) or an image edit ([editing a picture](#editing-a-picture)), which Retry asks again. @@ -270,7 +274,7 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i | `GET` | `/workspaces/{workspace_id}/studio/podcast/brief` | the podcast brief to review before submitting, the model's `voices` (`null` when they are typed) and the `languages` it may use | | `GET` | `/workspaces/{workspace_id}/artifacts` | the workspace's artifacts, newest first | | `GET` | `/artifacts/{artifact_id}` | detail: the body, the files, quiz or flashcard progress, and for a revised copy `revision: {derived_from_document_id, source_name, counts, applied}` | -| `POST` | `/artifacts/{artifact_id}/regenerate` | run the job again; `202`, or `409` while a run is queued or running, with the code `already_generating` unless the artifact is a revised copy, whose refusal has no code yet. A draft re-resolves a recorded `source_scope` without the artifact itself, else replays the recorded ids that still exist, and gives `409` `sources_gone`, "None of this artifact's sources are left.", when none remain. A refine repeats its rewrite without its sources, and gives `409` `rewrite_too_long` when the rewrite no longer fits the selected model ([Refine](#refine)). An image edit repeats its edit on the same base without its sources, and gives `409` `needs_image_edit` when no edit model can be reached ([editing a picture](#editing-a-picture)). A revised copy applies its stored edits to the same input again and asks no model | +| `POST` | `/artifacts/{artifact_id}/regenerate` | run the job again; `202`, or `409` while a run is queued or running, with the code `already_generating` unless the artifact is a revised copy, whose refusal has no code yet. A draft re-resolves a recorded `source_scope` without the artifact itself, else replays the recorded ids that still exist, and gives `409` `sources_gone`, "None of this artifact's sources are left.", when none remain. A refine repeats its rewrite without its sources, and gives `409` `rewrite_too_long` when the rewrite no longer fits the selected model ([Refine](#refine)). An image edit repeats its edit on the same base without its sources, and a drawn picture paints its prompt again with none; either gives `409` `needs_image` when no image model is chosen or the chosen one is gone, since any image model edits or redraws ([editing a picture](#editing-a-picture)). A revised copy applies its stored edits to the same input again and asks no model | | `POST` | `/artifacts/{artifact_id}/refine` | `{instruction}`; the next version, `pending`; `202`; its `409`s are under [Refine](#refine), six of them with a code: `needs_chat`, as for an unavailable format, and five of a refine's own | | `POST` | `/artifacts/{artifact_id}/image-edit` | `{instruction}`; an image's or infographic's next version, `pending`; `202`; its four `409`s are under [editing a picture](#editing-a-picture) | | `POST` | `/artifacts/{artifact_id}/revisions/{accept-all\|reject-all}` | a Word revised copy's next version from its newest, every tracked change accepted or rejected; `202`, or `409` for a workbook or deck, a version with no changes, or while a version is being made ([revised copies](#revised-copies)) | @@ -325,36 +329,68 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i ## The image path -- A drawn `image` or `infographic` resolves the `image_gen` selection, and an edit the `image_edit` selection ([below](#editing-a-picture)). A remote selection loads its connection, which runs the egress check and decrypts the key. An `sdcpp` selection points the same client at the bundled sd-server's loopback URL, with no key and no egress decision, which is what makes an image possible on an offline machine. +- A drawn `image` or `infographic` resolves the `image_gen` selection, and an edit, or a drawing with references, the model that takes a picture in ([below](#which-model-takes-a-picture)). A remote selection loads its connection, which runs the egress check and decrypts the key. An `sdcpp` selection points the same client at the bundled sd-server's loopback URL, with no key and no egress decision, which is what makes an image possible on an offline machine. - sd-server runs only while Studio needs it, because it keeps a model's weights in memory until it exits. The API answers Electron's poll from the jobs ([`local_image_demand.py`](../../surfsense_local/backend/modules/artifacts/local_image_demand.py)): the chosen model of the image type the oldest processing job needs, read from its recipe, so an edit needs `image_edit` though its format is still `image` or `infographic`; else, for 5 minutes after the last such job ended, the same one, so a second image does not reload it; else nothing, and Electron stops it. A cancel ends the window at once, since sd-server cannot stop a generation, and so does any other Studio job running, since a podcast voices in that memory and a text job loads the chat model into it. The job first gives up the text runtime through the API, held for the whole image, because the two share the graphics card and its writing is done by then: the API lets running replies finish, unloads the text models and keeps local text waiting until the image is drawn, and unloads the speech recognition model, starting no recording's window meanwhile ([admission](local-models/admission.md#the-image-jobs-hold)). Then it waits up to 2 minutes for `/sdapi/v1/sd-models` to name its weights before it posts, and fails with a reason if sd-server never comes up. - sd-server is launched with each file of the build on its flag and the model's reviewed `image` defaults as its own (`-W`, `-H`, `--steps`, `--cfg-scale`, `--sampling-method`, `--flow-shift`), since a request that carries none of them starts from the launch settings ([`local-models/catalog.md`](local-models/catalog.md)). -- The client posts `{model, prompt}` to `/images/generations`, falls back once to `/images` only on `404` or `405`, and remembers the route that worked for the life of the process. It accepts `b64_json`, a data URL or a URL, caps the sizes, and checks the bytes against the claimed MIME type ([`connections.md`](connections.md)). +- The client posts `{model, prompt}`, and `size` when a shape is asked, to `/images/generations`, falls back once to `/images`, where the shape is `aspect_ratio`, only on `404` or `405`, and remembers the route that worked for the life of the process. A refusal is decided by its status alone, so one whose body is cut off still counts: OpenRouter has hung up partway through the body of a `404`, which failed a comic's pages before ([`image_response.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py)). Only a success must arrive whole. It accepts `b64_json`, a data URL or a URL, caps the sizes, and checks the bytes against the claimed MIME type ([`connections.md`](connections.md)). - The `text_gen` model writes first in both formats, except a picture the agent asked for ([below](#pictures-the-agent-asks-for)). The markdown body is the image prompt or the brief, so an image can be found by what it shows. - A drawn picture is v1 of its own document: the job records `version` and `recipe: {"kind": "draft"}` as a Word draft's, so its edits group under it. Regenerate on it still draws from the sources and replaces v1 in place. +- Every drawing, edit and redraw keeps the exact prompt it sent as `drawing_prompt`, so a later redraw starts from it ([`redraw/`](../../surfsense_local/backend/worker/studio/media/visual/redraw/)). + +### Which model takes a picture + +One rule picks the model for any job that sends a picture in, an edit or a drawing with references ([`pictures_in/`](../../surfsense_local/backend/modules/llm/pictures_in/)): + +1. the `image_edit` selection, when one is chosen: an override, for editing with another model; +2. else the `image_gen` selection, when it takes a picture in; +3. else none, and an edit is drawn again ([below](#editing-a-picture)). + +Whether a remote image model takes a picture in is read when it is chosen, from its connection's model list: a model that lists `image` among its `input_modalities` and its outputs, as OpenRouter's do, gets the `image_edit` type beside `image_gen`, as the catalog already gives a model with an image in and out; the choice keeps `takes_pictures` in the selection's `settings`, and another model chosen starts without it. A local model takes pictures when its entry names `edit`. A model chosen unlisted is taken as words only, since a picture it ignores is billed. `GET /llm/pictures-in` answers `{source}`, `edit_model`, `image_model` or `null`, for the app. + +### Shapes and references + +A drawing can ask for a `shape`, `square`, `portrait` or `landscape` ([`picture_shape.py`](../../surfsense_local/backend/modules/llm/providers/picture_shape.py)): + +- **Remote:** `size` `1024x1024`, `1024x1536` or `1536x1024` on `/images/generations` and `/images/edits`; `aspect_ratio` `1:1`, `2:3` or `3:2` on `/images`. A server that refuses one with a `400` is asked once more without it and remembered, separately from `response_format`, since GPT image models refuse that and take a size. +- **Local:** a drawing with a shape goes through sd-server's native `img_gen` with `width` and `height` at the launch's `-W` times `-H` pixels in the shape's proportions, each side a multiple of 64; a drawing with none keeps `/v1/images/generations`. + +References are pictures sent with the prompt, so a new picture keeps their characters and style, as a comic's later pages keep page 1's. They go to the model that takes a picture in, as an edit's one picture does: every reference as an `image[]` part of `/images/edits`, or, where a server answers `404` or `405` there, as OpenRouter does, `input_references` of `{type: "image_url", image_url: {url: <data URL>}}` on `/images`, remembered per server; locally, every one in `ref_images`. A refusal there comes before any drawing, so trying the other route bills nothing; a `5xx` or a timeout is never retried. With no model that takes a picture in, the picture is drawn from its prompt alone and keeps `drawn_from_words`. ### Pictures the agent asks for -`surfsense_make_image` ([agent](agent.md#pictures-the-agent-makes)) creates an `image` or `infographic` with no sources, the chat's `chat_thread_id` and the recipe `{"kind": "image_drawn", "prompt"}` ([`drawn_images/`](../../surfsense_local/backend/modules/artifacts/drawn_images/), [ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). The job paints the prompt as written, asking no chat model; an infographic adds Studio's sketchnote style around it ([`media/visual/drawn/`](../../surfsense_local/backend/worker/studio/media/visual/drawn/)). The body is the title and the prompt, so it can be found by what it shows. +`surfsense_make_image` ([agent](agent.md#pictures-the-agent-makes)) creates an `image` or `infographic` with no sources, the chat's `chat_thread_id` and the recipe `{"kind": "image_drawn", "prompt"}`, with `shape` and `references` (artifact ids) when asked ([`drawn_images/`](../../surfsense_local/backend/modules/artifacts/drawn_images/), [ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). The job paints the prompt as written, asking no chat model; an infographic adds Studio's sketchnote style around it ([`media/visual/drawn/`](../../surfsense_local/backend/worker/studio/media/visual/drawn/)). The body is the title and the prompt, so it can be found by what it shows. + +It needs only `image_gen`, its Regenerate paints the same prompt at the same shape from the same references, a deleted reference failing it, and Huey never retries it, since each try may be billed. It is v1 of its own document and takes Studio's Edit box like any picture. Every edit keeps its base's chat, from either box, so that chat can still place it. While a script of its chat names a picture, deleting that picture is refused with `placed_in_document`, since the script names it rather than copying it ([`placed_in.py`](../../surfsense_local/backend/modules/artifacts/drawn_images/placed_in.py)). + +### Comics the agent binds + +`surfsense_bind_comic` ([agent](agent.md#pictures-the-agent-makes)) creates a `comic` with the recipe `{"kind": "comic_bound", "pages": [{"picture", "lines"}], "layout", "direction"}`, naming the pictures it drew in its chat in reading order ([`comic/bound_comic.py`](../../surfsense_local/backend/modules/artifacts/comic/bound_comic.py)). A page still being drawn makes it wait with `waits_for`, as a document does ([`waiting_renders/`](../../surfsense_local/backend/modules/artifacts/waiting_renders/)). The job binds the pictures' own files with the shared binder and asks no model, so it asks for no local image model either. Called again with the comic's `artifact_id`, it makes the comic's next version; a Studio comic keeps no pages, so it cannot be continued that way. -It needs only `image_gen`, its Regenerate paints the same prompt, and Huey never retries it, since each try may be billed. It is v1 of its own document and takes Studio's Edit box like any picture. Every edit keeps its base's chat, from either box, so that chat can still place it. While a script of its chat names a picture, deleting that picture is refused with `placed_in_document`, since the script names it rather than copying it ([`placed_in.py`](../../surfsense_local/backend/modules/artifacts/drawn_images/placed_in.py)). +Its pages are never listed: a picture drawn with `comic_page`, every later version of it, and any picture a comic binds keep `comic_page: true`, and `GET /workspaces/{id}/artifacts` leaves them out ([`comic/pages.py`](../../surfsense_local/backend/modules/artifacts/comic/pages.py)), so Studio, mentions and the jobs dialog show the comic alone. A page is not indexed either, and binding takes a page drawn without the mark out of the index, so search finds the comic alone, as it finds a PDF. Its text is its title, each page's drawing prompt, a line every page repeats (its style, its cast) kept once, and the lines set under its pages ([`bound_pages.py`](../../surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py)): its bubbles are drawn, so the PDF holds no text to read them from. A page still opens by its id from the chat step that drew it. Deleting a comic deletes the pages no other comic binds and no document places, every version of each, newest first ([`unused_pages.py`](../../surfsense_local/backend/modules/artifacts/comic/unused_pages.py)); a page drawn and never bound stays, unlisted. ### Editing a picture `POST /artifacts/{id}/image-edit` with `{instruction}` (1 to 2,000 characters, as Refine's) makes an image's or infographic's next version from the open version's picture ([`image_edits/`](../../surfsense_local/backend/modules/artifacts/image_edits/), [ADR 0077](../adr/0077-an-image-edit-is-the-next-version.md)). -- It refuses, each a `409` whose `detail` is `{message, code}`: a version still being made (`already_generating`); one whose last run failed or was stopped (`image_version_unfinished`, its own code so the sentence says to edit another version, not refine one); anything but an `image` or `infographic` with a still `primary` picture in PNG, JPEG or WebP, since an edit model does not read the SVG a remote model may draw and `/images/edits` takes no GIF (`not_editable`); and, after those, no `image_edit` model it can use (`needs_image_edit`, "Needs an image editing model"), the way Refine answers `needs_chat`: none chosen, a local one not installed or whose entry no longer names `edit`, or a connection that is gone or no longer serves image editing. Egress to its host being off is refused as any call to a host is, with the egress refusal that asks to allow it ([egress](egress.md)). The check reads the selection and asks no server, which would add a round trip to every edit and could start sd-server only to answer; a model that cannot be reached fails the version with its reason, as a refine's does. `ArtifactRead.image_editable` is true for a ready `image` or `infographic` with a `primary` PNG, JPEG or WebP. +- It refuses, each a `409` whose `detail` is `{message, code}`: a version still being made (`already_generating`); one whose last run failed or was stopped (`image_version_unfinished`, its own code so the sentence says to edit another version, not refine one); anything but an `image` or `infographic` with a still `primary` picture in PNG, JPEG or WebP, since an edit model does not read the SVG a remote model may draw and `/images/edits` takes no GIF (`not_editable`); and, after those, no image model at all (`needs_image`, "Needs an image model"): with [no model that takes a picture in](#which-model-takes-a-picture), the image model draws it again, so no edit is refused for want of an edit model. Egress to its host being off is refused as any call to a host is, with the egress refusal that asks to allow it ([egress](egress.md)). The check reads the selection and asks no server, which would add a round trip to every edit and could start sd-server only to answer; a model that cannot be reached fails the version with its reason, as a refine's does. `ArtifactRead.image_editable` is true for a ready `image` or `infographic` with a `primary` PNG, JPEG or WebP. - It creates the next version, `pending`, with the base's title, prompt and `source_document_ids`, and the recipe `{"kind": "image_edit", "instruction", "base_artifact_id"}`, under the `version` keys a refine writes. An image drawn before versions were kept becomes v1 then. The recipe names the base rather than copying its bytes, so a base cannot go while a later version names it: `DELETE /artifacts/{id}`, and the documents route for its document, refuse it with `409` `edited_from`, "A later version is edited from this one: delete the whole document." ([`edited_from.py`](../../surfsense_local/backend/modules/artifacts/image_edits/edited_from.py)). The list deletes a document's versions newest first, each after the last, and an edit is always newer than its base, so it never meets the refusal. -- The job reads the base's own file, so edits chain: editing v3 starts from v3's picture. The instruction goes to the edit model as written, with no chat model call first ([`media/visual/edit/`](../../surfsense_local/backend/worker/studio/media/visual/edit/)). The body is the base's, then `Edited: <instruction>`, so the version stays findable by what it shows and says how it moved; the file keeps the base's name with the extension of what came back. Every failure ends the job without Huey's retry, as a refine's does. -- **Remote:** the client posts multipart `/images/edits` with `image`, `prompt`, `model` and `response_format: b64_json`, sent once more without the last if the model refuses it with a `400`, as a generation is ([connections](connections.md)), and no `input_fidelity`, which only some GPT image models take ([`image_edit.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py)). It reads the answer's `data[]` with the same bounded read, link fetch and MIME check as a generation ([`image_response.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py)). +- The job reads the base's own file, so edits chain: editing v3 starts from v3's picture. The instruction goes to the model that takes the picture as written, with no chat model call first ([`media/visual/edit/`](../../surfsense_local/backend/worker/studio/media/visual/edit/)). The body is the base's, then `Edited: <instruction>`, so the version stays findable by what it shows and says how it moved; the file keeps the base's name with the extension of what came back. Every failure ends the job without Huey's retry, as a refine's does. +- **A redraw.** With no model that takes a picture in, the image model draws `<drawing prompt>\n\nChange: <instruction>` at the base's proportions, with no chat model. The drawing prompt is the base's `drawing_prompt`, or, for a picture drawn before it was kept, its recipe's prompt, or its body under the title, so changes add up from version to version. The version keeps `redrawn: true` and its body starts "Drawn again with your change." ([`redraw/`](../../surfsense_local/backend/worker/studio/media/visual/redraw/)). Studio's Edit box says so in its placeholder, "Describe a change: the picture is drawn again", read from `GET /llm/pictures-in`. +- **Remote:** the client posts multipart `/images/edits` with `image`, `prompt`, `model` and `response_format: b64_json`, sent once more without the last if the model refuses it with a `400`, as a generation is ([connections](connections.md)); where the server has no `/images/edits`, the picture goes to `/images` as a reference ([above](#shapes-and-references)); and no `input_fidelity`, which only some GPT image models take ([`image_edit.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py)). It reads the answer's `data[]` with the same bounded read, link fetch and MIME check as a generation ([`image_response.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py)). - **Local:** inside the same text-runtime hold and model wait as a drawn image, the client posts sd-server's native `POST /sdcpp/v1/img_gen` with `{prompt, ref_images: [<data URL>], width, height, output_format: "png"}` and polls `GET /sdcpp/v1/jobs/{id}` each second, which also looks for a cancel ([`editor.py`](../../surfsense_local/backend/modules/llm/providers/sdcpp/editor.py)). A `failed` job, a `404`, `410` or `429`, or a server that stops answering fails the edit. Steps, guidance and sampler come from the launch, as for a drawn image; the size is the base's own, since an edit model launched at another size would crop or stretch it. sd.cpp picks the reference preset from the model's architecture, so the launch needs no `--ref-image-args`. - **Why `img_gen`.** At the pinned `master-869-07a85c7`, sd-server's `/v1/images/edits` makes the first upload the `init_image` as well as a reference, and `/sdapi/v1/img2img` takes `init_images`: both re-noise the picture at a strength of 0.75 rather than edit it. `ref_images` alone is how every edit model sd.cpp documents is run. - **Cancel:** a queued sd.cpp job is withdrawn with `POST /sdcpp/v1/jobs/{id}/cancel`, given 2 seconds, since the text runtime is still held while it is asked. A generating one cannot be (`cancel_generating: false`); the cancel ends the demand at once and Electron stops sd-server, as for a drawn image. -- **Which models edit:** a local model whose entry names `edit`, today FLUX.2 klein 4B, which fills both image slots from one download, so nothing reloads between drawing and editing; or a remote model that takes an image in and gives one out ([`classifier.py`](../../surfsense_local/backend/modules/llm/catalog/remote/classifier.py)). Models that can only re-noise a picture, SD 1.5, SDXL, Z-Image Turbo, ERNIE-Image Turbo and LongCat-Image, name no `edit` and cannot be chosen for it. +- **Which models edit:** a local model whose entry names `edit`, today FLUX.2 klein 4B, which fills both image slots from one download, so nothing reloads between drawing and editing; or a remote model that takes an image in and gives one out, by its connection's own list or the catalog ([`classifier.py`](../../surfsense_local/backend/modules/llm/catalog/remote/classifier.py)). Settings lists those with an **Image editing** badge beside **Image generation**. Models that can only re-noise a picture, SD 1.5, SDXL, Z-Image Turbo, ERNIE-Image Turbo and LongCat-Image, name no `edit`: they cannot be chosen as the edit override, and a picture they drew is redrawn. ## Known gaps +- A Studio comic keeps only its PDF, so one page cannot be changed: Regenerate draws every page again. +- A page the agent drew and never bound is kept unlisted, until its chat's comic binds it; nothing deletes it. - An image edit is measured only on klein Q4_0 through the pinned sd-server on a processor, with no graphics card: a photo's layout held, and an infographic's printed lettering held through a colour change and a changed title, 657 to 962 seconds an edit, about twice a draw. Lettering klein drew itself, a sketchnote's, is not measured, and neither is any time on a graphics card or a remote edit model. - An edit changes the whole picture by instruction; there is no mask or selected region, which sd.cpp and `/images/edits` both take, since drawing one needs its own interface. +- References are measured only on OpenRouter's `/images`, on six models drawing one four-page comic: each held its characters from page 1. OpenAI's `/images/edits` with several pictures, and local references and shapes through sd-server, are not measured; nor is a remote server that takes one picture on `/images/edits` and refuses several. +- The [images guide](../../www/content/docs/v2/guides/images.mdx)'s walkthrough and screenshots still choose a separate model for editing, from before the image model edits. +- A remote model's taking pictures is read when it is chosen; a provider that later starts or stops listing it is not seen until the model is chosen again. - The [window budget](#the-window-budget) prices a draft's first call. What only some calls add is not priced: a written web page's design guide (6,235 characters, about 2,100 tokens past the prompt it reserves), the failed reply a repair replays, and a podcast segment's recap. On a small window these can leave the reply short or be refused, which fails the job with the context sentence and no retry. Reasoning is not reserved either: Studio turns thinking off, which the local runtime honours and some remote endpoints do not, whose windows are large. - The budget prices text with admission's estimate, not the model's tokenizer, so scripts it prices at a token a character, such as Cyrillic or Devanagari, are cut further than they need. A long figure list leaves a Word or PDF draft less of its sources' text, since every figure they hold is listed. - Under an image's hold, a local model no request has seated since the app started has no last window, so its job keeps the 24,000-character ceiling and may be refused. diff --git a/surfsense_local/backend/api/main.py b/surfsense_local/backend/api/main.py index 7755375bc1..a13f63d5c4 100644 --- a/surfsense_local/backend/api/main.py +++ b/surfsense_local/backend/api/main.py @@ -61,6 +61,7 @@ from modules.llm.model_route.router import router as model_route_router from modules.llm.model_type import ModelType from modules.llm.models import SelectedModel +from modules.llm.pictures_in.router import router as pictures_in_router from modules.llm.residency import warm_selected from modules.llm.router import router as llm_router from modules.llm.subscriptions.claude.errors import ( @@ -260,6 +261,7 @@ def create_app() -> FastAPI: app.include_router(search_server_router) app.include_router(folders_router) app.include_router(llm_router) + app.include_router(pictures_in_router) app.include_router(model_route_router) app.include_router(transcription_windows_router) app.include_router(chat_router) diff --git a/surfsense_local/backend/modules/agent/agent_threads/steps.py b/surfsense_local/backend/modules/agent/agent_threads/steps.py index b390b9dc05..9e3725f794 100644 --- a/surfsense_local/backend/modules/agent/agent_threads/steps.py +++ b/surfsense_local/backend/modules/agent/agent_threads/steps.py @@ -3,6 +3,7 @@ from dataclasses import asdict from typing import Any +from modules.agent.tool_endpoint.bind_comic import TOOL_NAME as BIND_COMIC_TOOL from modules.agent.tool_endpoint.confirm_job import TOOL_NAME as CONFIRM_JOB_TOOL from modules.agent.tool_endpoint.convert_document import TOOL_NAME as CONVERT_TOOL from modules.agent.tool_endpoint.make_image import TOOL_NAME as MAKE_IMAGE_TOOL @@ -32,6 +33,8 @@ SAVE_PAGE_STEP = f"{SERVER}_{SAVE_PAGE_TOOL}" # So does a picture's. MAKE_IMAGE_STEP = f"{SERVER}_{MAKE_IMAGE_TOOL}" +# And a bound comic's. +BIND_COMIC_STEP = f"{SERVER}_{BIND_COMIC_TOOL}" # A job run opens the revised copy it filled in, and a confirmation the version it made. RUN_JOB_STEP = f"{SERVER}_{RUN_JOB_TOOL}" CONFIRM_JOB_STEP = f"{SERVER}_{CONFIRM_JOB_TOOL}" @@ -41,6 +44,7 @@ CONVERT_STEP, SAVE_PAGE_STEP, MAKE_IMAGE_STEP, + BIND_COMIC_STEP, RUN_JOB_STEP, CONFIRM_JOB_STEP, ) diff --git a/surfsense_local/backend/modules/agent/opencode_config.py b/surfsense_local/backend/modules/agent/opencode_config.py index ee91c97b7d..9deaf5d26f 100644 --- a/surfsense_local/backend/modules/agent/opencode_config.py +++ b/surfsense_local/backend/modules/agent/opencode_config.py @@ -61,7 +61,16 @@ PDF_SKILL = "surfsense-pdf" # How to write a web page and keep it with surfsense_save_page (ADR 0066). HTML_SKILL = "surfsense-html" -SKILLS = (DOCUMENTS_SKILL, REVISIONS_SKILL, DATA_SKILL, PDF_SKILL, HTML_SKILL) +# How to draw a picture, or a set such as a comic, and bind a set into a PDF. +PICTURES_SKILL = "surfsense-pictures" +SKILLS = ( + DOCUMENTS_SKILL, + REVISIONS_SKILL, + DATA_SKILL, + PDF_SKILL, + HTML_SKILL, + PICTURES_SKILL, +) # Denied as permissions, which also leave the tool out of what the model is # offered: document scripts reach the runner through a SurfSense tool (ADR 0039), diff --git a/surfsense_local/backend/modules/agent/prompts/agent.md b/surfsense_local/backend/modules/agent/prompts/agent.md index 6b35994ec2..6820139493 100644 --- a/surfsense_local/backend/modules/agent/prompts/agent.md +++ b/surfsense_local/backend/modules/agent/prompts/agent.md @@ -30,7 +30,7 @@ For a Word document, a PDF, a PowerPoint deck or an Excel workbook, load the `su For a quiz, flashcards, a podcast or a mind map, start it in Studio with `surfsense_create_artifact`, naming the sources by the number at the end of their file names, then tell the user it will appear in Studio. Studio reads those sources itself, so you need not read them first. -For a picture or infographic, when asked: `surfsense_make_image`. +Pictures/comics: `surfsense-pictures` skill, `surfsense_make_image`. For any other document, table or file, write it to `outputs/` with a clear file name, then say in one line what you wrote and where. diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md b/surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md index cb7a29e270..b4673153de 100644 --- a/surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md +++ b/surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md @@ -1,6 +1,6 @@ --- name: surfsense-html -description: Make or change a web page from the user's sources - a report, a briefing, a dashboard, a comparison, a timeline, an explainer, an interactive calculator or a figure that shows how something works - by writing an HTML file and keeping it with surfsense_save_page. Load it whenever the user asks for a web page, an HTML page, an interactive page, or a change to one you made. +description: Make or change a web page from the user's sources - a report, a briefing, a dashboard, a comparison, a timeline, an explainer, an interactive calculator or a figure that shows how something works - by writing an HTML file and keeping it with surfsense_save_page. Load it whenever the user asks for a web page, an HTML page, an interactive page, or a change to one you made. Not for a comic, a storyboard or a picture: those are drawn pictures, with the surfsense-pictures skill. --- # Web pages diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md new file mode 100644 index 0000000000..d3c5f21ee1 --- /dev/null +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md @@ -0,0 +1,48 @@ +--- +name: surfsense-pictures +description: Draw pictures with surfsense_make_image - one picture or infographic, or a set that belongs together, such as a comic, a manga, a manhwa or webtoon, a comic strip, a graphic novel, a storyboard or a picture book, bound into one comic with surfsense_bind_comic. Load it whenever the user asks for a picture, an illustration, or any set of pictures, or for a change to one you drew. Not for charts of numbers, and not for a web page. +--- + +# Pictures + +`surfsense_make_image` draws with the user's image model and keeps each picture in Studio, named `image-<id>`. This skill is how to use it well. + +## The files in this skill + +| File | Read it | +|---|---| +| `comics.md` | For any kind of comic (Western, manga, manhwa and webtoon, manhua, strip, yonkoma, graphic novel, bande dessinée), a picture book or a storyboard: how each reads, is laid out, is lettered and is bound. | + +## One picture + +1. Read any sources the picture draws on first. +2. Write one self-contained prompt: the subject, the setting, the style, the light, what is in front and what is behind. As short as it can be while still saying what to draw. +3. Choose `shape` when it matters: `portrait` for a page or a poster, `landscape` for a banner or a strip, `square` for an icon. Choose `type: infographic` when the picture sets out facts, in Studio's sketchnote style; leave `type` out for any other picture. +4. Look at what comes back. Image models get hands, counts and lettering wrong. Change it once with its `artifact_id` when it is plainly wrong, and say what you changed; otherwise tell the user what is off. + +No picture for numbers: a chart stays a chart, in a document. Keep any words in a picture short, a title or a label; text that must be exact goes in the document around it. + +## A set of pictures + +A comic, a storyboard, a picture book, or any pictures that must look like one set: + +1. **Plan** the set: what each picture shows, in order. Write a **character sheet**, each character's face, hair, build, clothes and colours in a few fixed words, and a **style line**: medium, palette, line weight. Both go word for word into every prompt. +2. **Confirm once.** Say how many pictures, the plan in short, and that each is billed by their provider, or takes minutes on their computer. On yes, draw every picture without asking again. A single picture needs no confirmation. +3. **Draw the first alone,** and look at it. When it is wrong, change it before going on: every later picture copies its characters. +4. **Draw the rest with the first as their reference**: `references: ["image-<first>"]`. Give each its own call, all in one step, so a remote model draws them side by side. A character picture the user gave, or a character sheet they asked for first, is the reference instead. +5. **Title each** "<title> · page <n>", or "· scene <n>": a chat cannot reuse a picture's title. For a comic, a storyboard or a picture book, give every page `comic_page: true`, from page 1 on: Studio lists the comic, never its pages. +6. **Bind them** with `surfsense_bind_comic`, every picture's name in reading order, with the `layout` and `direction` [`comics.md`](comics.md#which-kind) gives its kind. The comic is what the user keeps: one artifact in Studio, a PDF. +7. **Answer** with the comic and one line per picture, so the user can ask for "page 3 again". + +## When the model cannot take a picture in + +Some image models draw from words alone. Nothing is refused, and nothing needs setting up: + +- A reference is drawn from the words alone. The result says so: then every picture's character sheet carries the likeness, and in the plan you confirm, say once, plainly, that characters may differ a little between pictures. +- A change is drawn again from the picture's prompt and the change, and may not keep the rest. The result says so; tell the user in one line. + +Never send the user to Settings for this. + +## Changing a set later + +Change one picture with its `artifact_id`, then call `surfsense_bind_comic` with the comic's `artifact_id` and every page, the new picture's name in place of the old: that is the comic's next version. To add a picture, draw it with the first as reference and bind again. diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md new file mode 100644 index 0000000000..ab4c74cf12 --- /dev/null +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md @@ -0,0 +1,55 @@ +# Comics + +A comic of any kind: one picture per page, or per panel for a scrolling one, every picture from the same character sheet and style line, bound into one comic with `surfsense_bind_comic`. Follow the set's steps in [`SKILL.md`](SKILL.md#a-set-of-pictures); this file is what is particular to comics. + +## Which kind + +Take the kind the user names; when they name none, a Western comic. The kind decides how it reads, how its pictures are shaped and bound, and how it looks. A style the user asks for (noir, chibi, watercolour, superhero) goes in the style line on top of it. + +| Kind | Reads | Each picture | Binding (`surfsense_bind_comic`) | Looks | +|---|---|---|---|---| +| Western comic | left to right | a `portrait` page of 4 to 6 panels | `layout: "pages"` | full colour, bold inks, caption boxes | +| Graphic novel | left to right | a `portrait` page of 3 to 6 panels | `"pages"` | as a Western comic, with room for quiet, wordless panels | +| Bande dessinée | left to right | a `portrait` page of 6 to 9 smaller panels | `"pages"` | detailed album pages, often clean "ligne claire" lines and flat colour | +| Manga | **right to left**: panel 1 top right, the last bottom left | a `portrait` page of 3 to 6 panels | `"pages"`, `direction: "rtl"` | black and white with screentone, speed lines, expressive faces | +| Manhua | left to right | a `portrait` page, or one panel a picture to scroll | `"pages"` or `"scroll"` | full colour, often painterly | +| Manhwa, webtoon | top to bottom, by scrolling | one panel or beat a `portrait` picture, no page | `layout: "scroll"` | full colour, soft shading, wide gutters between beats | +| Comic strip | left to right | one `landscape` picture of 3 or 4 panels in a row | `"pages"`, one page a strip | simple shapes, a punchline in the last panel | +| Yonkoma (4-koma) | top to bottom | one `portrait` picture of 4 equal panels stacked | `"pages"`, one page a strip | a manga strip: setup, development, turn, punchline | +| Picture book | left to right | one illustration a page, no bubbles | `"pages"`, every page's text as its lines | soft colour, one moment a page | +| Storyboard | left to right | one `landscape` frame a shot | `"pages"`, each shot's number and note as its lines | rough, sketchy frames; camera and movement in the note | + +A manga's panels and bubbles still read right to left inside each page: say so in every prompt. Write its lines in the user's language, read left to right inside each bubble; ask for vertical lettering only when the user writes Japanese and asks for it. + +## The plan + +- **Length.** Four pages, or eight beats for a scrolling comic, when the user names none; at most twelve pictures, since each is billed or takes minutes. A comic strip or a yonkoma is one picture a strip. +- **Panels.** For each: where it sits (top, bottom left, wide), what it shows, and its lines. A scrolling comic has one beat a picture instead. +- **Lines.** One short line a bubble, at most about eight words, two or three bubbles a panel. A caption box for narration. Write every line in the plan, word for word: the binding may need them. A picture book's and a storyboard's text is never in the picture; it goes under it, as that page's `lines`. + +## A page's prompt + +The style line, the character sheet, then the page, laid out as the image model reads it best: + +```text +Full-colour comic book page, clean bold ink lines, flat colours, white gutters, +black-bordered panels, speech bubbles with clear hand-lettered capitals. +Characters: MILO, a small orange tabby cat with a white chest, green eyes and a red +scarf. NORA, a woman about seventy, short white hair, round glasses, yellow raincoat. +Page 2 of 4 of the comic "The Lighthouse Cat". A portrait page of 3 panels: +Panel 1 (wide, top): a fishing boat tosses on huge waves at night. +Panel 2 (bottom left): NORA taps the dead lamp. NORA says: "THE BULB IS BROKEN!" +Panel 3 (bottom right): MILO spots a box of spare bulbs on a high shelf. +``` + +For a manga: "Black-and-white manga page with screentone, read right to left. Panel 1 (top right): …". For a scrolling comic: "One tall webtoon panel, full colour, nothing else in the picture: …", one beat per picture. From the second picture on, add one line before the page: "Keep the characters looking exactly as in the reference picture: same faces, colours, clothes and drawing style. Draw a new page; do not copy its panels." + +## Lettering + +- **Bubbles drawn by the model.** Write each line in the prompt in quotes, in capitals for a Western comic. +- **A misspelt line is not redrawn.** Note the page, and give its lines as that page's `lines` when you bind: they are set under it, exactly, in a strip. A redraw would bill another picture for one letter. +- **On this computer** (the result says "Drawn on this computer"), **or when you cannot see the pictures**, draw every page without words: leave the bubbles out of the prompts, ask for "no text, no lettering, no speech bubbles", and give every page its strip. You cannot rely on the lettering, or check it. + +## Before you bind + +Look at every picture. Redraw or change one only when it is plainly wrong, such as a missing character, the wrong number of panels, panels in the wrong reading order, or a third arm, and say which and why. Then call `surfsense_bind_comic`: every picture's name in `pages`, in reading order, each with its `lines` when it needs a strip, and the kind's `layout` and `direction`. A page still being drawn may be named: the comic waits for it, and appears in Studio when its last page is drawn. Tell the user so, and stop. diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/bind_comic.py b/surfsense_local/backend/modules/agent/tool_endpoint/bind_comic.py new file mode 100644 index 0000000000..0848ea0985 --- /dev/null +++ b/surfsense_local/backend/modules/agent/tool_endpoint/bind_comic.py @@ -0,0 +1,214 @@ +"""Bind the pictures the agent drew into one comic, kept in Studio as a PDF.""" + +import time +from dataclasses import dataclass +from typing import Any, cast + +from sqlalchemy.orm import Session + +from modules.agent.tool_endpoint.failed_renders import FailedRunError +from modules.agent.tool_endpoint.job_outcome import wait_for_outcome +from modules.agent.tool_endpoint.registration import TOOL_CALL_SECONDS +from modules.agent.tool_endpoint.rendered_label import ( + RenderedArtifact, + first_line, + queued_line, +) +from modules.agent.tool_endpoint.same_title_documents import refuse_same_title +from modules.agent.tool_endpoint.tool import Tool, ToolCallError +from modules.agent.tool_endpoint.turn_scope import TurnScope +from modules.artifacts.comic.bound_comic import ( + FORMAT, + ComicRefusedError, + create_bound_comic, +) +from modules.artifacts.comic.kinds import MAX_PAGES +from modules.artifacts.comic.reading import DIRECTIONS, LAYOUTS, Direction, Layout +from modules.artifacts.script_documents.version import version_of +from modules.artifacts.waiting_renders.waits import waiting_sentence, waits_for +from modules.documents.models import DocumentStatus + +TOOL_NAME = "bind_comic" + +# Binding reads files and writes one PDF: seconds, unless Studio is busy. +WAIT_SECONDS = 60 +CALL_SECONDS = TOOL_CALL_SECONDS - 10 +LINE_CHARS = 2_000 + +LISTING: dict[str, Any] = { + "name": TOOL_NAME, + "description": ( + "Bind pictures you drew with surfsense_make_image in this chat into one " + "comic, kept in Studio as a PDF: a comic of any kind, a picture book or a " + "storyboard. Follow the surfsense-pictures skill's comics.md for the kind's " + "layout and direction. Give every page in reading order. A page still being " + "drawn may be named: the comic waits for it. To change a page, change its " + "picture, then call this again with the comic's artifact_id and every page." + ), + "inputSchema": { + "type": "object", + "properties": { + "title": {"type": "string", "description": "The comic's title."}, + "pages": { + "type": "array", + "maxItems": MAX_PAGES, + "items": { + "type": "object", + "properties": { + "image": { + "type": "string", + "description": "The picture's name, image-<id>.", + }, + "lines": { + "type": "array", + "items": {"type": "string"}, + "description": ( + "Text set under the picture exactly: for a page " + "drawn without words or misspelt, a picture book's " + "text, a storyboard shot's note. Left out, none." + ), + }, + }, + "required": ["image"], + }, + "description": "Every page, in reading order.", + }, + "layout": { + "type": "string", + "enum": list(LAYOUTS), + "description": ( + "pages, the default: one picture a page. scroll: stacked down " + "one long page, as a webtoon reads." + ), + }, + "direction": { + "type": "string", + "enum": list(DIRECTIONS), + "description": "rtl for a manga; ltr, the default, otherwise.", + }, + "artifact_id": { + "type": "integer", + "description": ( + "To change a comic you bound in this chat: its artifact id. The " + "result is its next version. Leave out for a new comic." + ), + }, + }, + "required": ["title", "pages"], + }, +} + + +@dataclass(frozen=True) +class _Request: + title: str + pages: list[tuple[str, tuple[str, ...]]] + layout: Layout + direction: Direction + base_artifact_id: int | None + + +def bind(session: Session, scope: TurnScope, arguments: dict[str, Any]) -> str: + """Create the comic's version in one short transaction, then wait for its job.""" + began = time.monotonic() + scope.require_known() + request = _request(arguments) + if request.base_artifact_id is None: + refuse_same_title(session, scope.thread_id, request.title, FORMAT) + try: + artifact = create_bound_comic( + session, + scope.workspace_id, + scope.thread_id, + title=request.title, + pages=request.pages, + layout=request.layout, + direction=request.direction, + base_artifact_id=request.base_artifact_id, + ) + except ComicRefusedError as refused: + session.rollback() + raise ToolCallError(str(refused)) from refused + version = version_of(artifact.artifact_metadata) + made = RenderedArtifact( + artifact.id, + " ".join(request.title.split()), + version.number if version is not None else 1, + ) + waiting = waits_for(artifact.artifact_metadata) + session.commit() + if waiting: + return f"{queued_line(made)}\n{waiting_sentence(waiting)}" + + wait = min(WAIT_SECONDS, began + CALL_SECONDS - time.monotonic()) + outcome = wait_for_outcome(session, made.id, wait) + if outcome is None: + raise ToolCallError(f"Artifact {made.id} was deleted before it was ready.") + if outcome.status is DocumentStatus.READY: + count = len(request.pages) + return ( + f"{first_line(made)}\n" + f"A comic of {count} {'page' if count == 1 else 'pages'}, bound as one " + "PDF in Studio. To change a page, change its picture with " + f"surfsense_make_image, then call this tool again with artifact_id " + f"{made.id} and every page." + ) + if outcome.status is DocumentStatus.FAILED: + raise FailedRunError( + f'The comic "{made.title}" (artifact {made.id}) failed: ' + f"{outcome.error_message or 'no reason given'}\n\nTell the user what failed." + ) + if outcome.status is DocumentStatus.CANCELLED: + raise ToolCallError( + f"Artifact {made.id} was cancelled in Studio. Ask the user before " + "binding it again." + ) + return ( + f"{queued_line(made)}\n" + f"The comic is still being bound after {time.monotonic() - began:.0f} s, " + "behind other Studio work. It appears in Studio when it is ready: tell the " + "user so, and do not bind it again." + ) + + +def _request(arguments: dict[str, Any]) -> _Request: + """The call's arguments, refused in a sentence naming the one to fix.""" + title = arguments.get("title") + if not isinstance(title, str) or not title.strip(): + raise ToolCallError("Give the comic a title.") + pages = arguments.get("pages") + if not isinstance(pages, list) or not pages: + raise ToolCallError("Give the pages: every picture's name, in reading order.") + if len(pages) > MAX_PAGES: + raise ToolCallError(f"Bind at most {MAX_PAGES} pages.") + layout = arguments.get("layout") or "pages" + if layout not in LAYOUTS: + raise ToolCallError(f"layout must be one of: {', '.join(LAYOUTS)}.") + direction = arguments.get("direction") or "ltr" + if direction not in DIRECTIONS: + raise ToolCallError(f"direction must be one of: {', '.join(DIRECTIONS)}.") + base = arguments.get("artifact_id") + if base is not None and (not isinstance(base, int) or isinstance(base, bool)): + raise ToolCallError( + "artifact_id must be the number of a comic you bound, or left out." + ) + return _Request( + title.strip(), + [_page(page) for page in pages], + cast(Layout, layout), + cast(Direction, direction), + base, + ) + + +def _page(page: object) -> tuple[str, tuple[str, ...]]: + if not isinstance(page, dict) or not isinstance(page.get("image"), str): + raise ToolCallError('Each page is {"image": "image-<id>"}, with lines if any.') + lines = page.get("lines") or [] + if not isinstance(lines, list) or not all(isinstance(line, str) for line in lines): + raise ToolCallError("A page's lines must be a list of strings.") + kept = tuple(line.strip()[:LINE_CHARS] for line in lines if line.strip()) + return page["image"].strip(), kept + + +BIND_COMIC = Tool(listing=LISTING, run=bind, waits=True) diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/create_artifact.py b/surfsense_local/backend/modules/agent/tool_endpoint/create_artifact.py index 3c7e312f57..adf902a203 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/create_artifact.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/create_artifact.py @@ -25,11 +25,12 @@ # Office files and PDFs are scripts the agent renders, and a web page it writes # itself (ADR 0066): Studio's draft keeps none to edit. A picture has its own tool, -# since a draft hands the agent nothing to look at or place. +# since a draft hands the agent nothing to look at or place; so does a comic, whose +# pages the agent draws and checks before it binds them. _KEYS = [ fmt.key for fmt in FORMATS - if fmt.key not in (*DOCUMENT_FORMATS, "html", *EDITABLE_FORMATS) + if fmt.key not in (*DOCUMENT_FORMATS, "html", *EDITABLE_FORMATS, "comic") ] # Written out flat, and the same on every turn, so a local model's prompt cache holds. diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/drawn_how.py b/surfsense_local/backend/modules/agent/tool_endpoint/drawn_how.py new file mode 100644 index 0000000000..50f5ddc45c --- /dev/null +++ b/surfsense_local/backend/modules/agent/tool_endpoint/drawn_how.py @@ -0,0 +1,56 @@ +"""Where a picture was drawn, and how when not as asked, for the agent to act on: +a comic draws wordless pages on this computer, and says when faces may differ.""" + +from sqlalchemy.orm import Session + +from modules.artifacts.drawn_images.recipe import image_drawing_of +from modules.artifacts.image_edits.recipe import image_edit_of +from modules.artifacts.models import Artifact +from modules.llm.model_type import ModelType +from modules.llm.models import ProviderConnection, SelectedModel +from modules.llm.pictures_in import model as pictures_in +from modules.llm.providers.sdcpp import provider as sdcpp + +REDRAWN = ( + "Drawn again from its prompt and the change: this image model takes no picture " + "in, so it may not keep the rest of the picture." +) +FROM_WORDS = ( + "Drawn from words alone: this image model takes no picture in, so the " + "characters may differ from the references." +) + + +def drawn_how(session: Session, artifact: Artifact) -> list[str]: + meta = artifact.artifact_metadata or {} + redrawn, from_words = bool(meta.get("redrawn")), bool(meta.get("drawn_from_words")) + drawing = image_drawing_of(meta) + took_pictures = not (redrawn or from_words) and ( + image_edit_of(meta) is not None + or (drawing is not None and bool(drawing.references)) + ) + lines = [_where(session, took_pictures)] + if redrawn: + lines.append(REDRAWN) + if from_words: + lines.append(FROM_WORDS) + return [line for line in lines if line] + + +def _where(session: Session, took_pictures: bool) -> str: + slot = ( + ModelType.IMAGE_EDIT + if took_pictures and pictures_in.picture_source(session) == "edit_model" + else ModelType.IMAGE_GEN + ) + selected = session.get(SelectedModel, slot) + if selected is None: + return "" + if selected.provider == sdcpp.PROVIDER: + return "Drawn on this computer." + connection = ( + session.get(ProviderConnection, selected.connection_id) + if selected.connection_id is not None + else None + ) + return f"Drawn by {connection.label}." if connection is not None else "" diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py index bf765bd01a..47810b8e7a 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py @@ -3,7 +3,7 @@ import threading import time from dataclasses import dataclass -from typing import Any +from typing import Any, cast from fastapi import HTTPException from PIL import Image @@ -11,6 +11,7 @@ from sqlalchemy.orm import Session from modules.agent.previews.inline_images import inline_image +from modules.agent.tool_endpoint.drawn_how import drawn_how from modules.agent.tool_endpoint.failed_renders import ( IMAGES, FailedRunError, @@ -36,7 +37,11 @@ ToolResult, ) from modules.agent.tool_endpoint.turn_scope import TurnScope -from modules.artifacts.drawn_images.names import image_name +from modules.artifacts.drawn_images.names import ( + ImageStillDrawingError, + image_name, + kept_picture, +) from modules.artifacts.drawn_images.service import create_drawn_image from modules.artifacts.image_edits.editable import EDITABLE_FORMATS, has_picture from modules.artifacts.image_edits.model import require_image_edit_model @@ -45,6 +50,8 @@ from modules.artifacts.script_documents.version import version_of from modules.documents.models import Document, DocumentStatus from modules.embedding.active import EmbeddingNotChosenError +from modules.llm.pictures_in import model as pictures_in +from modules.llm.providers.picture_shape import SHAPES, Shape from shared.config import get_storage_settings TOOL_NAME = "make_image" @@ -56,6 +63,8 @@ # minutes, past the tool call's hard limit, so the call answers without it. WAIT_SECONDS = 150 CALL_SECONDS = TOOL_CALL_SECONDS - 10 +# Bounds what one call uploads; page 1 and a character sheet are what matter. +MAX_REFERENCES = 4 _ENDED = (DocumentStatus.READY, DocumentStatus.FAILED, DocumentStatus.CANCELLED) _starting = threading.Lock() @@ -63,15 +72,17 @@ LISTING: dict[str, Any] = { "name": TOOL_NAME, "description": ( - "Draw a picture or an infographic with the user's image model, or edit one " + "Draw a picture or an infographic with the user's image model, or change one " "you made in this chat, and keep it in Studio. Only when the user asks for " - "one: each is billed, or takes minutes on their computer, so confirm before " - "making more than one. Read any sources first and write the whole prompt " - "yourself. Draw no data chart: numbers belong in a real chart. Words that " - "must be exact belong in the document around the picture, since image models " - "garble lettering. Write one self-contained prompt, as short as it can be " - "while still saying what to draw. Returns the picture's name, to place it in a document or " - "a web page, and the picture itself to check." + "one. When a request needs more than one picture, confirm how many once, " + "before the first: each is billed, or takes minutes on their computer. For a " + "comic, a storyboard or any set of pictures, load the surfsense-pictures " + "skill first. Read any sources first and write the whole prompt yourself. " + "Draw no data chart: numbers belong in a real chart. Lettering may come out " + "misspelt: keep words in a picture short, and put text that must be exact in " + "the document around it. Write one self-contained prompt, as short as it can " + "be while still saying what to draw. Returns the picture's name, to place it " + "in a document or a web page, and the picture itself to check." ), "inputSchema": { "type": "object", @@ -89,12 +100,13 @@ "The picture's title, as Studio lists it. Needed for a new picture." ), }, - "style": { + "type": { "type": "string", "enum": sorted(EDITABLE_FORMATS), "description": ( - "infographic to give the prompt's facts Studio's sketchnote " - "style; image, the default, to draw the prompt alone." + "The type of picture: image, the default, draws the prompt " + "alone; infographic gives the prompt's facts Studio's " + "sketchnote style." ), }, "artifact_id": { @@ -104,6 +116,32 @@ "The result is its next version. Leave out for a new picture." ), }, + "shape": { + "type": "string", + "enum": list(SHAPES), + "description": ( + "portrait for a page, landscape for a strip or a banner, square. " + "Left out, the model's own." + ), + }, + "comic_page": { + "type": "boolean", + "description": ( + "true for a page of a comic, storyboard or picture book you " + "will bind with surfsense_bind_comic: Studio lists the comic, " + "not its pages." + ), + }, + "references": { + "type": "array", + "items": {"type": "string"}, + "maxItems": MAX_REFERENCES, + "description": ( + "Names of pictures you made in this chat, each ready, for a new " + "picture to keep their characters and style, as a comic's later " + "pages keep page 1's. Not with artifact_id." + ), + }, }, "required": ["prompt"], }, @@ -114,8 +152,11 @@ class _Request: prompt: str title: str | None - style: str + picture_type: str base_artifact_id: int | None + shape: Shape | None = None + references: tuple[str, ...] = () + comic_page: bool = False def make( @@ -131,7 +172,9 @@ def make( with _starting: if request.base_artifact_id is None: assert request.title is not None # _request asked for it - refuse_same_title(session, scope.thread_id, request.title, request.style) + refuse_same_title( + session, scope.thread_id, request.title, request.picture_type + ) made = _start(session, scope, request) outcome = wait_for_outcome( @@ -159,8 +202,9 @@ def make( f"{queued_line(made)}\n" f"The picture is still being made after {time.monotonic() - began:.0f} s. Do " "not make it again: it appears in Studio when it is ready. Its name is " - f"{name}, and a document or page can place it once it is ready: if one says " - "it is still being drawn, tell the user so and stop." + f"{name}. A document or comic can place it now and is made once it is " + "drawn. A web page can place it only once it is ready: if one says it is " + "still being drawn, tell the user so and stop." ) @@ -168,7 +212,7 @@ def _request(arguments: dict[str, Any]) -> _Request: """The call's arguments, refused in a sentence naming the one to fix.""" prompt, title = arguments.get("prompt"), arguments.get("title") # Null is how many models leave an optional field out. - style = arguments.get("style") or "image" + picture_type = arguments.get("type") or "image" base = arguments.get("artifact_id") if not isinstance(prompt, str) or not prompt.strip(): raise ToolCallError("Give the prompt: what to draw, or how to change it.") @@ -177,9 +221,9 @@ def _request(arguments: dict[str, Any]) -> _Request: f"The prompt is {len(prompt.strip()):,} characters; the limit is " f"{PROMPT_CHARS:,}. Shorten it and call this tool again." ) - if style not in EDITABLE_FORMATS: + if picture_type not in EDITABLE_FORMATS: raise ToolCallError( - f"style must be one of: {', '.join(sorted(EDITABLE_FORMATS))}." + f"type must be one of: {', '.join(sorted(EDITABLE_FORMATS))}." ) if base is not None and (not isinstance(base, int) or isinstance(base, bool)): raise ToolCallError( @@ -187,8 +231,30 @@ def _request(arguments: dict[str, Any]) -> _Request: ) if base is None and (not isinstance(title, str) or not title.strip()): raise ToolCallError("Give the picture a title.") + shape = arguments.get("shape") or None + if shape is not None and shape not in SHAPES: + raise ToolCallError(f"shape must be one of: {', '.join(SHAPES)}.") + references = arguments.get("references") or [] + if not isinstance(references, list) or not all( + isinstance(name, str) and name.strip() for name in references + ): + raise ToolCallError("references must be a list of picture names.") + if references and base is not None: + raise ToolCallError( + "references are for a new picture: leave out artifact_id, or references." + ) + if len(references) > MAX_REFERENCES: + raise ToolCallError( + f"Give at most {MAX_REFERENCES} references: the pictures that matter most." + ) return _Request( - prompt.strip(), title.strip() if isinstance(title, str) else None, style, base + prompt.strip(), + title.strip() if isinstance(title, str) else None, + picture_type, + base, + cast(Shape, shape) if shape is not None else None, + tuple(dict.fromkeys(name.strip() for name in references)), + arguments.get("comic_page") is True, ) @@ -230,11 +296,35 @@ def _new_picture(session: Session, scope: TurnScope, request: _Request) -> Artif scope.workspace_id, title=request.title, prompt=request.prompt, - format=request.style, + format=request.picture_type, chat_thread_id=scope.thread_id, + shape=request.shape, + references=_reference_ids(session, scope, request.references), + comic_page=request.comic_page, ) +def _reference_ids( + session: Session, scope: TurnScope, names: tuple[str, ...] +) -> tuple[int, ...]: + """Checked as placing them is: this chat's, ready, a still picture.""" + ids: list[int] = [] + for name in names: + try: + kept_picture(session, scope.workspace_id, scope.thread_id, name) + except ImageStillDrawingError as error: + raise ToolCallError( + f"{name} is still being drawn: wait for it, then draw this one." + ) from error + except LookupError as error: + raise ToolCallError( + f"This chat has no picture {name} to keep: name one {TOOL_NAME} gave " + "in this chat." + ) from error + ids.append(int(name.removeprefix("image-"))) + return tuple(ids) + + def _edited_picture( session: Session, scope: TurnScope, base_id: int, instruction: str ) -> Artifact: @@ -266,16 +356,18 @@ def _shown( placeable = has_picture(artifact) primary = next(f for f in artifact.files if f.role is ArtifactFileRole.PRIMARY) storage_key = primary.storage_key - editable = _edit_model_chosen(session) + edits = pictures_in.picture_source(session) is not None + how = drawn_how(session, artifact) session.commit() lines = [ first_line(made), - _how_to_use(made.id, editable) if placeable else _NOT_PLACEABLE, + *how, + _how_to_use(made.id, edits) if placeable else _NOT_PLACEABLE, ] images: tuple[InlineImage, ...] = () if placeable: if scope.reads_images(): - images = _picture(storage_key, editable, lines) + images = _picture(storage_key, lines) else: lines.append(_UNSEEN) text = "\n".join(lines) @@ -286,41 +378,28 @@ def _shown( "The image model gave a format documents cannot place. It is in Studio for the " "user to open: tell them so." ) -# A redraw would bill another picture the user did not ask for. -_NO_EDIT_MODEL = ( - "No image editing model is chosen, so you cannot alter this picture: do not draw " - "it again unless the user asks. If it is wrong, tell the user what is wrong, and " - "that they can choose a model in Settings > Image edit." -) _UNSEEN = ( "You cannot see it: the selected model cannot read images. Do not describe it " "beyond what you asked for." ) -def _edit_model_chosen(session: Session) -> bool: - """Whether an edit could run: one offered without a model could only be refused.""" - try: - require_image_edit_model(session) - except HTTPException: - return False - return True - - -def _how_to_use(artifact_id: int, editable: bool) -> str: +def _how_to_use(artifact_id: int, edits: bool) -> str: name = image_name(artifact_id) - place = ( + change = f"To change it, call this tool again with artifact_id {artifact_id}" + if not edits: + # Neutral: every image model can change a picture, this one by redrawing. + change += ( + ": this image model takes no picture in, so the change is drawn again " + "from its prompt and may not keep the rest" + ) + return ( f"Its name is {name}. To place it, pass {name} in a render's images, or in " - f'a web page write src="surfsense-image:{name}".' + f'a web page write src="surfsense-image:{name}". {change}.' ) - if editable: - return f"{place} To change it, call this tool again with artifact_id {artifact_id}." - return f"{place} {_NO_EDIT_MODEL}" -def _picture( - storage_key: str, editable: bool, lines: list[str] -) -> tuple[InlineImage, ...]: +def _picture(storage_key: str, lines: list[str]) -> tuple[InlineImage, ...]: path = get_storage_settings().data_dir / storage_key try: shown = inline_image(path) @@ -329,12 +408,7 @@ def _picture( return () lines.append( "Look at it before you answer: image models get lettering, hands and counts " - "wrong. " - + ( - "If it is wrong, change it or tell the user." - if editable - else "If it is wrong, tell the user, as above." - ) + "wrong. If it is wrong, change it or tell the user." ) return (shown,) diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py b/surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py index 9f7e41aa45..7dc23b36e9 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py @@ -10,6 +10,7 @@ from api.dependencies import transact from modules.agent.tool_endpoint import replies from modules.agent.tool_endpoint.analyze_data import ANALYZE_DATA +from modules.agent.tool_endpoint.bind_comic import BIND_COMIC from modules.agent.tool_endpoint.confirm_job import CONFIRM_JOB, confirm_listing from modules.agent.tool_endpoint.convert_document import CONVERT_DOCUMENT from modules.agent.tool_endpoint.create_artifact import CREATE_ARTIFACT @@ -54,6 +55,7 @@ RENDER_DOCUMENT, SAVE_PAGE, MAKE_IMAGE, + BIND_COMIC, READ_DOCUMENT, LIST_IMAGES, SOURCE_PAGES, diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/render_document.py b/surfsense_local/backend/modules/agent/tool_endpoint/render_document.py index 2c2e573bd9..64fb49a113 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/render_document.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/render_document.py @@ -46,6 +46,7 @@ ) from modules.artifacts.script_documents.spec import DOCUMENT_FORMATS, FORMAT_NAMES from modules.artifacts.script_documents.version import version_of +from modules.artifacts.waiting_renders.waits import waiting_sentence, waits_for from modules.documents.models import DocumentStatus from modules.workspaces.models import Workspace @@ -148,6 +149,8 @@ class _Started: artifact_id: int version: int title: str + # The pictures still being drawn it waits for; it is queued once they are. + waits_for: tuple[int, ...] = () def render( @@ -166,6 +169,9 @@ def render( if request.base_artifact_id is None and scope.known: refuse_same_title(session, scope.thread_id, request.title, request.format) started = _start(session, scope, request) + if started.waits_for: + waiting = RenderedArtifact(started.artifact_id, started.title, started.version) + return f"{queued_line(waiting)}\n{waiting_sentence(started.waits_for)}" wait = min(WAIT_SECONDS, deadline - time.monotonic()) outcome = wait_for_outcome(session, started.artifact_id, wait) if outcome is None: @@ -253,7 +259,12 @@ def _start(session: Session, scope: TurnScope, request: _Request) -> _Started: raise version = version_of(artifact.artifact_metadata) assert version is not None # the service gave it one - return _Started(artifact.id, version.number, request.title.strip()) + return _Started( + artifact.id, + version.number, + request.title.strip(), + waits_for(artifact.artifact_metadata), + ) def _create(session: Session, scope: TurnScope, request: _Request) -> Artifact: diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py b/surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py index f8839c85cc..814697e272 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py @@ -22,7 +22,7 @@ def refuse_same_title( session: Session, thread_id: int, title: str, format: object ) -> None: """Refuse a new document whose title and format match one the thread rendered.""" - if format not in (*DOCUMENT_FORMATS, "html", *EDITABLE_FORMATS): + if format not in (*DOCUMENT_FORMATS, "html", *EDITABLE_FORMATS, "comic"): return # the service refuses it, naming the formats wanted = _key(title) rendered = session.execute( diff --git a/surfsense_local/backend/modules/artifacts/comic/__init__.py b/surfsense_local/backend/modules/artifacts/comic/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/modules/artifacts/comic/binding_recipe.py b/surfsense_local/backend/modules/artifacts/comic/binding_recipe.py new file mode 100644 index 0000000000..0098efbdbc --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/binding_recipe.py @@ -0,0 +1,54 @@ +"""How the agent's comic is made: the pictures it drew, in reading order, bound as it asked. + +Kept so Regenerate binds the same pages again, and a later version can change one. +""" + +from dataclasses import dataclass +from typing import Any, cast + +from modules.artifacts.comic.reading import DIRECTIONS, LAYOUTS, Direction, Layout +from modules.artifacts.studio_documents.recipe import RECIPE_KEY + +COMIC_BOUND = "comic_bound" + + +@dataclass(frozen=True) +class ComicPage: + # The picture's artifact id. + picture: int + lines: tuple[str, ...] = () + + +@dataclass(frozen=True) +class ComicBinding: + pages: tuple[ComicPage, ...] + layout: Layout + direction: Direction + + +def comic_bound(binding: ComicBinding) -> dict[str, Any]: + return { + "kind": COMIC_BOUND, + "pages": [ + {"picture": page.picture, "lines": list(page.lines)} + for page in binding.pages + ], + "layout": binding.layout, + "direction": binding.direction, + } + + +def comic_binding_of(metadata: dict[str, Any] | None) -> ComicBinding | None: + recipe = (metadata or {}).get(RECIPE_KEY) + if not isinstance(recipe, dict) or recipe.get("kind") != COMIC_BOUND: + return None + layout = recipe.get("layout") + direction = recipe.get("direction") + return ComicBinding( + pages=tuple( + ComicPage(int(page["picture"]), tuple(page.get("lines") or ())) + for page in recipe.get("pages") or () + ), + layout=cast(Layout, layout) if layout in LAYOUTS else "pages", + direction=cast(Direction, direction) if direction in DIRECTIONS else "ltr", + ) diff --git a/surfsense_local/backend/modules/artifacts/comic/bound_comic.py b/surfsense_local/backend/modules/artifacts/comic/bound_comic.py new file mode 100644 index 0000000000..0d30400239 --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/bound_comic.py @@ -0,0 +1,174 @@ +"""Create the agent's comic: a pending `comic` artifact that binds pictures drawn in its chat. + +Commits before enqueueing: the worker is another process and must find the row. +""" + +import logging + +from sqlalchemy.orm import Session + +from modules.artifacts.comic.binding_recipe import ( + ComicBinding, + ComicPage, + comic_bound, +) +from modules.artifacts.comic.kinds import MAX_PAGES +from modules.artifacts.comic.pages import mark_comic_pages +from modules.artifacts.comic.reading import Direction, Layout +from modules.artifacts.drawn_images.names import ( + ImageStillDrawingError, + is_image_name, + kept_picture, +) +from modules.artifacts.models import Artifact +from modules.artifacts.queued_job import queue_job +from modules.artifacts.script_documents.version import ( + ArtifactVersion, + next_version_number, + version_of, +) +from modules.artifacts.studio_documents.recipe import RECIPE_KEY +from modules.artifacts.waiting_renders.settle import settle_waiting_renders +from modules.artifacts.waiting_renders.waits import KEY as WAITS_FOR +from modules.documents.models import Document, DocumentStatus, DocumentType +from modules.embedding.active import EmbeddingNotChosenError, require_active_index + +logger = logging.getLogger(__name__) + +TITLE_CHARS = 200 +FORMAT = "comic" + + +class ComicRefusedError(ValueError): + """Why the comic cannot be bound, in a sentence the agent can act on.""" + + +def create_bound_comic( + session: Session, + workspace_id: int, + chat_thread_id: int, + *, + title: str, + pages: list[tuple[str, tuple[str, ...]]], + layout: Layout, + direction: Direction, + base_artifact_id: int | None, +) -> Artifact: + """`pages` are picture names in reading order, each with the lines set under it.""" + title = title.strip() + if not 1 <= len(title) <= TITLE_CHARS: + raise ComicRefusedError( + f"Give the comic a title of 1 to {TITLE_CHARS} characters." + ) + if not 1 <= len(pages) <= MAX_PAGES: + raise ComicRefusedError( + f"Bind 1 to {MAX_PAGES} pages: at most {MAX_PAGES} pages." + ) + try: + require_active_index(session) + except EmbeddingNotChosenError as error: + raise ComicRefusedError( + "Studio is not ready on this computer: no embedding model is chosen yet." + ) from error + + waiting: list[int] = [] + for name, _ in pages: + try: + kept_picture(session, workspace_id, chat_thread_id, name) + except ImageStillDrawingError: + waiting.append(int(name.removeprefix("image-"))) + except LookupError as error: + raise ComicRefusedError( + f'This chat has no picture named "{name}" to bind: use a name ' + "surfsense_make_image gave in this chat." + ) from error + base = ( + None + if base_artifact_id is None + else _base_version(session, workspace_id, chat_thread_id, base_artifact_id) + ) + + document = Document( + workspace_id=workspace_id, + title=title, + document_type=DocumentType.ARTIFACT, + status=DocumentStatus.PENDING, + ) + session.add(document) + session.flush() + artifact = Artifact( + document_id=document.id, + workspace_id=workspace_id, + format=FORMAT, + chat_thread_id=chat_thread_id, + ) + session.add(artifact) + session.flush() + version = ( + ArtifactVersion(root=artifact.id, number=1, parent=None) + if base is None + else ArtifactVersion( + root=base.root, + number=next_version_number(session, workspace_id, base.root), + parent=base_artifact_id, + ) + ) + binding = ComicBinding( + pages=tuple(ComicPage(picture_id(name), lines) for name, lines in pages), + layout=layout, + direction=direction, + ) + mark_comic_pages(session, [page.picture for page in binding.pages]) + artifact.artifact_metadata = { + RECIPE_KEY: comic_bound(binding), + "version": version.as_metadata(), + "source_document_ids": [], + "prompt": None, + **({WAITS_FOR: list(dict.fromkeys(waiting))} if waiting else {}), + } + if waiting: + session.commit() + # A picture may have ended between the check and the commit. + settle_waiting_renders(session, render_id=artifact.id) + else: + queue_job(session, artifact) + logger.info( + "studio: enqueued bound comic %s of %s pages, root=%s v%s", + artifact.id, + len(pages), + version.root, + version.number, + ) + return artifact + + +def picture_id(name: str) -> int: + """The artifact id in a picture's name; ComicRefusedError for any other name.""" + if not is_image_name(name): + raise ComicRefusedError( + f'"{name}" is not a picture\'s name: use a name surfsense_make_image gave.' + ) + return int(name.removeprefix("image-")) + + +def _base_version( + session: Session, workspace_id: int, chat_thread_id: int, base_id: int +) -> ArtifactVersion: + base = session.get(Artifact, base_id) + if ( + base is None + or base.workspace_id != workspace_id + or base.chat_thread_id != chat_thread_id + or base.format != FORMAT + ): + raise ComicRefusedError( + f"This chat bound no comic with artifact id {base_id}: name one " + "surfsense_bind_comic gave in this chat." + ) + version = version_of(base.artifact_metadata) + if version is None: + raise ComicRefusedError( + f"Comic {base_id} was made in Studio and keeps no pages to bind again: " + "bind a new comic without artifact_id." + ) + return version diff --git a/surfsense_local/backend/modules/artifacts/comic/kinds.py b/surfsense_local/backend/modules/artifacts/comic/kinds.py new file mode 100644 index 0000000000..0752b856d1 --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/kinds.py @@ -0,0 +1,155 @@ +"""Each kind of comic: how it reads, how its pictures are shaped and bound, and how it looks. + +The agent's skill (`surfsense-pictures/comics.md`) states the same table for the agent. +""" + +from dataclasses import dataclass +from typing import Literal + +Shape = Literal["portrait", "landscape"] + + +@dataclass(frozen=True) +class Kind: + key: str + name: str + # What one picture is, as a page prompt says it. + picture: str + unit: str + shape: Shape + layout: Literal["pages", "scroll"] + direction: Literal["ltr", "rtl"] + looks: str + default_pages: int + # False for a picture book or a storyboard: its text is never in the picture. + bubbles: bool = True + + +KINDS: dict[str, Kind] = { + kind.key: kind + for kind in ( + Kind( + "western", + "Western comic", + "A portrait comic page of 4 to 6 black-bordered panels", + "Page", + "portrait", + "pages", + "ltr", + "Full-colour comic book art, clean bold ink lines, flat colours, white " + "gutters, caption boxes, speech bubbles with clear hand-lettered capitals.", + 4, + ), + Kind( + "graphic_novel", + "Graphic novel", + "A portrait graphic novel page of 3 to 6 panels, with room for quiet, " + "wordless panels", + "Page", + "portrait", + "pages", + "ltr", + "Full-colour graphic novel art, confident ink lines, cinematic framing.", + 4, + ), + Kind( + "bande_dessinee", + "Bande dessinée", + "A detailed portrait album page of 6 to 9 smaller panels", + "Page", + "portrait", + "pages", + "ltr", + "Franco-Belgian bande dessinée, clean 'ligne claire' lines, flat colour, " + "detailed backgrounds.", + 4, + ), + Kind( + "manga", + "Manga", + "A portrait manga page of 3 to 6 panels, read right to left: panel 1 top " + "right, the last bottom left; its bubbles read right to left too", + "Page", + "portrait", + "pages", + "rtl", + "Black-and-white manga with screentone, speed lines and expressive faces.", + 4, + ), + Kind( + "manhua", + "Manhua", + "A portrait manhua page of 3 to 6 panels", + "Page", + "portrait", + "pages", + "ltr", + "Full-colour, painterly manhua art.", + 4, + ), + Kind( + "webtoon", + "Webtoon", + "One tall webtoon panel, one beat of the story, nothing else in the picture", + "Beat", + "portrait", + "scroll", + "ltr", + "Full-colour webtoon art, soft shading, clean digital lines.", + 8, + ), + Kind( + "comic_strip", + "Comic strip", + "One landscape comic strip of 3 or 4 panels in a row, the punchline in the " + "last", + "Strip", + "landscape", + "pages", + "ltr", + "Newspaper comic strip, simple shapes, bold outlines, flat colour.", + 1, + ), + Kind( + "yonkoma", + "Yonkoma (4-koma)", + "One portrait yonkoma of 4 equal panels stacked top to bottom: setup, " + "development, turn, punchline", + "Strip", + "portrait", + "pages", + "ltr", + "Manga-style 4-koma strip, black and white, simple expressive faces.", + 1, + ), + Kind( + "picture_book", + "Picture book", + "One full-page illustration of one moment, no speech bubbles", + "Page", + "portrait", + "pages", + "ltr", + "Children's picture book illustration, soft colour, gentle shapes.", + 4, + bubbles=False, + ), + Kind( + "storyboard", + "Storyboard", + "One landscape storyboard frame of one shot", + "Shot", + "landscape", + "pages", + "ltr", + "Rough, sketchy storyboard frame, grey pencil tones, arrows for camera " + "and movement.", + 6, + bubbles=False, + ), + ) +} + +DEFAULT_KIND = "western" +# Each picture is billed or takes minutes, so a comic stops here. +MAX_PAGES = 12 diff --git a/surfsense_local/backend/modules/artifacts/comic/options.py b/surfsense_local/backend/modules/artifacts/comic/options.py new file mode 100644 index 0000000000..27500ed513 --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/options.py @@ -0,0 +1,19 @@ +"""A comic job's options, the kind and how many pictures, checked when it is created.""" + +from sqlalchemy.orm import Session + +from modules.artifacts.comic.kinds import DEFAULT_KIND, KINDS, MAX_PAGES + + +def validate_options(_session: Session, raw: dict | None) -> dict: + """The Format hook: raises ValueError naming the option to fix.""" + raw = raw or {} + kind = raw.get("kind") or DEFAULT_KIND + if kind not in KINDS: + raise ValueError(f"kind: choose one of {', '.join(KINDS)}") + pages = raw.get("pages", KINDS[kind].default_pages) + if not isinstance(pages, int) or isinstance(pages, bool): + raise ValueError("pages: give a whole number") + if not 1 <= pages <= MAX_PAGES: + raise ValueError(f"pages: from 1 to {MAX_PAGES}") + return {"kind": kind, "pages": pages} diff --git a/surfsense_local/backend/modules/artifacts/comic/pages.py b/surfsense_local/backend/modules/artifacts/comic/pages.py new file mode 100644 index 0000000000..d18c881ac6 --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/pages.py @@ -0,0 +1,31 @@ +"""A picture drawn as a comic's page: kept for the agent, never listed apart from its comic.""" + +from typing import Any + +from sqlalchemy import delete +from sqlalchemy.orm import Session + +from modules.artifacts.models import Artifact +from modules.chunks.models import Chunk + +COMIC_PAGE = "comic_page" + + +def is_comic_page(metadata: dict[str, Any] | None) -> bool: + return bool((metadata or {}).get(COMIC_PAGE)) + + +def mark_comic_pages(session: Session, picture_ids: list[int]) -> None: + """Marked again when bound, for a page the agent drew without the mark, and + taken out of search: a comic is found by its own text, as a PDF is.""" + for picture_id in picture_ids: + picture = session.get(Artifact, picture_id) + if picture is not None and not is_comic_page(picture.artifact_metadata): + picture.artifact_metadata = { + **(picture.artifact_metadata or {}), + COMIC_PAGE: True, + } + # The delete trigger clears both search indexes for these rows. + session.execute( + delete(Chunk).where(Chunk.document_id == picture.document_id) + ) diff --git a/surfsense_local/backend/modules/artifacts/comic/reading.py b/surfsense_local/backend/modules/artifacts/comic/reading.py new file mode 100644 index 0000000000..60f25ccecd --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/reading.py @@ -0,0 +1,8 @@ +"""How a bound comic reads: page by page or as one scroll, left to right or right to left.""" + +from typing import Literal + +Layout = Literal["pages", "scroll"] +Direction = Literal["ltr", "rtl"] +LAYOUTS: tuple[Layout, ...] = ("pages", "scroll") +DIRECTIONS: tuple[Direction, ...] = ("ltr", "rtl") diff --git a/surfsense_local/backend/modules/artifacts/comic/unused_pages.py b/surfsense_local/backend/modules/artifacts/comic/unused_pages.py new file mode 100644 index 0000000000..483960cf8d --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/unused_pages.py @@ -0,0 +1,76 @@ +"""The pages that go with a deleted comic: those no other comic binds and no document places. + +A page is never listed, so nothing else would remove it. Every version of a page +goes with it, newest first, since an edit names the version it was made from. +""" + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from modules.artifacts.comic.binding_recipe import comic_binding_of +from modules.artifacts.comic.pages import is_comic_page +from modules.artifacts.models import Artifact +from modules.artifacts.script_documents.spec import DOCUMENT_FORMATS, document_script +from modules.artifacts.script_documents.version import version_of + + +def unused_pages(session: Session, comic: Artifact) -> list[Artifact]: + binding = comic_binding_of(comic.artifact_metadata) + if binding is None: + return [] + still_used = _used_elsewhere(session, comic) + gone: dict[int, Artifact] = {} + for page in binding.pages: + picture = session.get(Artifact, page.picture) + if picture is None or not is_comic_page(picture.artifact_metadata): + continue + versions = _versions(session, picture) + if any(version.id in still_used for version in versions): + continue + gone.update({version.id: version for version in versions}) + return sorted(gone.values(), key=lambda artifact: artifact.id, reverse=True) + + +def _used_elsewhere(session: Session, comic: Artifact) -> set[int]: + """Pictures another comic binds, or a document of the workspace places.""" + others = session.scalars( + select(Artifact).where( + Artifact.workspace_id == comic.workspace_id, + Artifact.id != comic.id, + Artifact.format.in_(("comic", *DOCUMENT_FORMATS)), + ) + ) + used: set[int] = set() + for other in others: + binding = comic_binding_of(other.artifact_metadata) + if binding is not None: + used.update(page.picture for page in binding.pages) + script = document_script(other.artifact_metadata) + if script is not None: + used.update( + int(name.removeprefix("image-")) + for name in script.images + if name.removeprefix("image-").isdigit() + ) + return used + + +def _versions(session: Session, picture: Artifact) -> list[Artifact]: + """Every comic page in the picture's line of versions, or the picture alone.""" + version = version_of(picture.artifact_metadata) + if version is None: + return [picture] + siblings = session.scalars( + select(Artifact).where( + Artifact.workspace_id == picture.workspace_id, + Artifact.chat_thread_id == picture.chat_thread_id, + Artifact.format == picture.format, + ) + ) + return [ + sibling + for sibling in siblings + if is_comic_page(sibling.artifact_metadata) + and (found := version_of(sibling.artifact_metadata)) is not None + and found.root == version.root + ] or [picture] diff --git a/surfsense_local/backend/modules/artifacts/drawn_images/recipe.py b/surfsense_local/backend/modules/artifacts/drawn_images/recipe.py index a7d119e981..2a3078bcdd 100644 --- a/surfsense_local/backend/modules/artifacts/drawn_images/recipe.py +++ b/surfsense_local/backend/modules/artifacts/drawn_images/recipe.py @@ -1,9 +1,11 @@ -"""How a picture the agent asked for is made, so Regenerate paints the same prompt.""" +"""How a picture the agent asked for is made, so Regenerate paints the same prompt, +at the same shape, from the same reference pictures.""" from dataclasses import dataclass -from typing import Any +from typing import Any, cast from modules.artifacts.studio_documents.recipe import RECIPE_KEY +from modules.llm.providers.picture_shape import SHAPES, Shape IMAGE_DRAWN = "image_drawn" @@ -11,14 +13,29 @@ @dataclass(frozen=True) class ImageDrawing: prompt: str + shape: Shape | None = None + # Artifact ids of the pictures it keeps the characters and style of. + references: tuple[int, ...] = () -def image_drawn(prompt: str) -> dict[str, Any]: - return {"kind": IMAGE_DRAWN, "prompt": prompt} +def image_drawn( + prompt: str, shape: Shape | None = None, references: tuple[int, ...] = () +) -> dict[str, Any]: + recipe: dict[str, Any] = {"kind": IMAGE_DRAWN, "prompt": prompt} + if shape is not None: + recipe["shape"] = shape + if references: + recipe["references"] = list(references) + return recipe def image_drawing_of(metadata: dict[str, Any] | None) -> ImageDrawing | None: recipe = (metadata or {}).get(RECIPE_KEY) if not isinstance(recipe, dict) or recipe.get("kind") != IMAGE_DRAWN: return None - return ImageDrawing(recipe["prompt"]) + shape = recipe.get("shape") + return ImageDrawing( + recipe["prompt"], + cast(Shape, shape) if shape in SHAPES else None, + tuple(int(i) for i in recipe.get("references") or ()), + ) diff --git a/surfsense_local/backend/modules/artifacts/drawn_images/service.py b/surfsense_local/backend/modules/artifacts/drawn_images/service.py index a88618876b..f8f83b3dda 100644 --- a/surfsense_local/backend/modules/artifacts/drawn_images/service.py +++ b/surfsense_local/backend/modules/artifacts/drawn_images/service.py @@ -4,6 +4,7 @@ from sqlalchemy.orm import Session +from modules.artifacts.comic.pages import COMIC_PAGE from modules.artifacts.drawn_images.model import require_image_model from modules.artifacts.drawn_images.recipe import image_drawn from modules.artifacts.models import Artifact @@ -11,6 +12,7 @@ from modules.artifacts.studio_documents.recipe import RECIPE_KEY from modules.documents.models import Document, DocumentStatus, DocumentType from modules.embedding.active import require_active_index +from modules.llm.providers.picture_shape import Shape logger = logging.getLogger(__name__) @@ -23,6 +25,9 @@ def create_drawn_image( prompt: str, format: str, chat_thread_id: int | None, + shape: Shape | None = None, + references: tuple[int, ...] = (), + comic_page: bool = False, ) -> Artifact: """Pending until Studio's job paints it. Commits before enqueueing: the worker is another process and must find the row.""" @@ -43,9 +48,10 @@ def create_drawn_image( format=format, chat_thread_id=chat_thread_id, artifact_metadata={ - RECIPE_KEY: image_drawn(prompt), + RECIPE_KEY: image_drawn(prompt, shape, references), "source_document_ids": [], "prompt": prompt, + **({COMIC_PAGE: True} if comic_page else {}), }, ) session.add(artifact) diff --git a/surfsense_local/backend/modules/artifacts/formats.py b/surfsense_local/backend/modules/artifacts/formats.py index 8623470e9b..28f39ae601 100644 --- a/surfsense_local/backend/modules/artifacts/formats.py +++ b/surfsense_local/backend/modules/artifacts/formats.py @@ -4,6 +4,7 @@ from sqlalchemy.orm import Session +from modules.artifacts.comic import options as comic_options from modules.artifacts.podcast import brief from modules.llm.model_type import ModelType @@ -31,6 +32,9 @@ class Format: grounding: Grounding = Grounding.PASSAGES # What to search a selection too big to share for, when no prompt says. default_focus: str = "the main points, findings, figures and conclusions" + # Its pipeline also takes the model that draws from a reference picture, as + # `keeper`: None when no chosen model takes pictures in. + draws_with_references: bool = False # worker/studio/job_router.py must name every key here and nothing else @@ -65,6 +69,14 @@ class Format: "Infographic", requires_model_types=(ModelType.IMAGE_GEN, ModelType.TEXT_GEN), ), + Format( + "comic", + "Comic", + requires_model_types=(ModelType.IMAGE_GEN, ModelType.TEXT_GEN), + validate_options=comic_options.validate_options, + draws_with_references=True, + default_focus="the story: who, what happens, and how it ends", + ), ) FORMATS_BY_KEY: dict[str, Format] = {fmt.key: fmt for fmt in FORMATS} diff --git a/surfsense_local/backend/modules/artifacts/image_edits/model.py b/surfsense_local/backend/modules/artifacts/image_edits/model.py index b72cff6ad0..b1336895c5 100644 --- a/surfsense_local/backend/modules/artifacts/image_edits/model.py +++ b/surfsense_local/backend/modules/artifacts/image_edits/model.py @@ -1,17 +1,12 @@ -"""Refuse an edit, or its Retry, that no chosen model can make.""" +"""Refuse an edit, or its Retry, only when no image model is chosen at all: a +model that takes pictures edits, and any other image model draws it again.""" -from fastapi import HTTPException, status from sqlalchemy.orm import Session -from modules.llm.resolution import ModelResolutionError, resolve_image_edit +from modules.artifacts.drawn_images.model import require_image_model +from modules.llm.pictures_in import model as pictures_in def require_image_edit_model(session: Session) -> None: - """Nothing fills the `image_edit` slot, or what fills it is gone; mirrors Refine's `needs_chat`.""" - try: - resolve_image_edit(session) - except ModelResolutionError as error: - raise HTTPException( - status.HTTP_409_CONFLICT, - {"message": "Needs an image editing model", "code": "needs_image_edit"}, - ) from error + if pictures_in.picture_source(session) is None: + require_image_model(session) diff --git a/surfsense_local/backend/modules/artifacts/image_edits/service.py b/surfsense_local/backend/modules/artifacts/image_edits/service.py index 1563b83853..95a8e047fd 100644 --- a/surfsense_local/backend/modules/artifacts/image_edits/service.py +++ b/surfsense_local/backend/modules/artifacts/image_edits/service.py @@ -5,6 +5,7 @@ from fastapi import HTTPException, status from sqlalchemy.orm import Session +from modules.artifacts.comic.pages import COMIC_PAGE, is_comic_page from modules.artifacts.image_edits.editable import EDITABLE_FORMATS, has_picture from modules.artifacts.image_edits.recipe import image_edited from modules.artifacts.models import Artifact @@ -99,6 +100,8 @@ def create_image_edit_version( "version": version.as_metadata(), "source_document_ids": list(meta.get("source_document_ids") or []), "prompt": meta.get("prompt"), + # A comic's page changed is its page still. + **({COMIC_PAGE: True} if is_comic_page(meta) else {}), } queue_job(session, artifact) logger.info( diff --git a/surfsense_local/backend/modules/artifacts/local_image_demand.py b/surfsense_local/backend/modules/artifacts/local_image_demand.py index d269dd80cb..cfc5e6b50f 100644 --- a/surfsense_local/backend/modules/artifacts/local_image_demand.py +++ b/surfsense_local/backend/modules/artifacts/local_image_demand.py @@ -6,6 +6,7 @@ from sqlalchemy import Row, select from sqlalchemy.orm import Session +from modules.artifacts.comic.binding_recipe import comic_binding_of from modules.artifacts.formats import FORMATS, Format from modules.artifacts.image_edits.recipe import image_edit_of from modules.artifacts.models import Artifact @@ -35,12 +36,14 @@ def local_image_demand(session: Session, now: datetime) -> ModelType | None: ).join(Document, Artifact.document_id == Document.id) jobs = studio.where(Artifact.format.in_(needs)) running = session.execute( - jobs.where(Document.status == DocumentStatus.PROCESSING) - .order_by(Document.updated_at) - .limit(1) - ).first() - if running is not None: - return _job_type(running, needs) + jobs.where(Document.status == DocumentStatus.PROCESSING).order_by( + Document.updated_at + ) + ).all() + # A bound comic draws nothing, so a picture running beside it still decides. + drawing = (slot for job in running if (slot := _job_type(job, needs))) + if (slot := next(drawing, None)) is not None: + return slot # A podcast voices on the processor and a text job loads the chat model; # either needs the memory more than a second image would. if session.execute( @@ -57,10 +60,13 @@ def local_image_demand(session: Session, now: datetime) -> ModelType | None: return _job_type(last, needs) if now - last.updated_at < IDLE else None -def _job_type(job: Row, needs: dict[str, ModelType]) -> ModelType: - """By the job, not its format: an edit is still an `image` or an `infographic`.""" +def _job_type(job: Row, needs: dict[str, ModelType]) -> ModelType | None: + """By the job, not its format: an edit is still an `image` or an `infographic`, + and a comic the agent bound draws nothing.""" if image_edit_of(job.artifact_metadata) is not None: return ModelType.IMAGE_EDIT + if comic_binding_of(job.artifact_metadata) is not None: + return None return needs[job.format] diff --git a/surfsense_local/backend/modules/artifacts/router.py b/surfsense_local/backend/modules/artifacts/router.py index c4490692e8..6a047d7a0c 100644 --- a/surfsense_local/backend/modules/artifacts/router.py +++ b/surfsense_local/backend/modules/artifacts/router.py @@ -8,6 +8,8 @@ from sqlalchemy.orm import Session from api.dependencies import SessionDep, transact +from modules.artifacts.comic.pages import is_comic_page +from modules.artifacts.comic.unused_pages import unused_pages from modules.artifacts.dependencies import ArtifactDep, get_artifact from modules.artifacts.drawn_images.placed_in import refuse_deleting_a_placed_picture from modules.artifacts.flashcard_progress import ( @@ -106,7 +108,12 @@ def list_artifacts( ) .order_by(Artifact.created_at.desc()) ).all() - return [ArtifactRead.of(artifact) for artifact in artifacts] + # A comic's pages are its own: the comic is listed, they are not. + return [ + ArtifactRead.of(artifact) + for artifact in artifacts + if not is_comic_page(artifact.artifact_metadata) + ] @router.get( @@ -398,11 +405,16 @@ def delete_artifact(artifact: ArtifactDep, session: SessionDep) -> Response: refuse_deleting_a_placed_picture(session, artifact) # The ARTIFACT document is the root: deleting it cascades the sidecar, its # files and its chunks. Only the blobs live beyond the database. - directory = get_storage_settings().artifact_dir(artifact.workspace_id, artifact.id) - document = artifact.document - - session.delete(document) + storage = get_storage_settings() + # A comic's pages go with it, once nothing else uses them: nothing lists them. + gone = [artifact, *unused_pages(session, artifact)] + directories = [storage.artifact_dir(a.workspace_id, a.id) for a in gone] + + for each in gone: + session.delete(each.document) + session.flush() session.commit() - shutil.rmtree(directory, ignore_errors=True) + for directory in directories: + shutil.rmtree(directory, ignore_errors=True) return Response(status_code=status.HTTP_204_NO_CONTENT) diff --git a/surfsense_local/backend/modules/artifacts/script_documents/service.py b/surfsense_local/backend/modules/artifacts/script_documents/service.py index a7f3ff8eb9..a0d7daa2f1 100644 --- a/surfsense_local/backend/modules/artifacts/script_documents/service.py +++ b/surfsense_local/backend/modules/artifacts/script_documents/service.py @@ -30,6 +30,8 @@ version_of, ) from modules.artifacts.studio_documents.recipe import studio_made +from modules.artifacts.waiting_renders.settle import settle_waiting_renders +from modules.artifacts.waiting_renders.waits import KEY as WAITS_FOR from modules.documents.models import Document, DocumentStatus, DocumentType from modules.documents.source_figures import parse_figure_name from modules.embedding.active import EmbeddingNotChosenError, require_active_index @@ -87,7 +89,7 @@ def create_script_document( ) from error images = tuple(dict.fromkeys(image_names)) - source_ids = _image_sources(session, workspace.id, chat_thread_id, images) + source_ids, waiting = _image_sources(session, workspace.id, chat_thread_id, images) base = ( None if base_artifact_id is None @@ -132,8 +134,14 @@ def create_script_document( "version": version.as_metadata(), "source_document_ids": source_ids, "prompt": None, + **({WAITS_FOR: list(waiting)} if waiting else {}), } - queue_job(session, artifact) + if waiting: + session.commit() + # A picture may have ended between the check and the commit. + settle_waiting_renders(session, render_id=artifact.id) + else: + queue_job(session, artifact) logger.info( "studio: enqueued script document %s format=%s root=%s v%s", artifact.id, @@ -149,22 +157,22 @@ def _image_sources( workspace_id: int, chat_thread_id: int | None, names: tuple[str, ...], -) -> list[int]: - """The sources the named figures come from, refusing a name none of them, nor - the chat's analyses, has. A chart is the chat's own output, made from no one source.""" +) -> tuple[list[int], tuple[int, ...]]: + """The sources the named figures come from, and the pictures still being drawn + that the document waits for; refusing a name none of them, nor the chat's + analyses, has. A chart is the chat's own output, made from no one source.""" + waiting: list[int] = [] for name in names: try: script_image(session, workspace_id, chat_thread_id, name) - except ImageStillDrawingError as error: - raise ScriptDocumentRefusedError( - f'The picture "{name}" is still being drawn. Tell the user it will be ' - "placed once it appears in Studio, and stop." - ) from error + except ImageStillDrawingError: + waiting.append(int(name.removeprefix("image-"))) except LookupError as error: raise ScriptDocumentRefusedError(_no_image(name)) from error figures = [parse_figure_name(name) for name in names if not is_chart_name(name)] # script_image accepted each name, so each figure's parses. - return list(dict.fromkeys(parsed[0] for parsed in figures if parsed is not None)) + sources = list(dict.fromkeys(p[0] for p in figures if p is not None)) + return sources, tuple(waiting) def _no_image(name: str) -> str: diff --git a/surfsense_local/backend/modules/artifacts/service.py b/surfsense_local/backend/modules/artifacts/service.py index 1a9a8ecfa3..d96bcc72b8 100644 --- a/surfsense_local/backend/modules/artifacts/service.py +++ b/surfsense_local/backend/modules/artifacts/service.py @@ -14,6 +14,7 @@ from modules.artifacts.refused import refused from modules.artifacts.schemas import FormatRead, StudioJobCreate from modules.artifacts.studio_documents.recipe import refinement, renders_as_stored +from modules.artifacts.waiting_renders.settle import settle_waiting_renders from modules.documents.models import Document, DocumentStatus, DocumentType from modules.documents.sources import load_selected_sources from modules.llm.model_type import ModelType @@ -145,6 +146,8 @@ def cancel_artifact(session: Session, artifact: Artifact) -> Artifact: """Stop a queued or running generation. The worker exits without writing ready.""" if not cancel_studio_job(session, artifact.document, artifact.id): raise refused(status.HTTP_409_CONFLICT, "nothing is running", "nothing_running") + # A queued job is revoked and never runs its end, so what waits on it settles here. + settle_waiting_renders(session, picture_id=artifact.id) return artifact diff --git a/surfsense_local/backend/modules/artifacts/waiting_renders/__init__.py b/surfsense_local/backend/modules/artifacts/waiting_renders/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/modules/artifacts/waiting_renders/settle.py b/surfsense_local/backend/modules/artifacts/waiting_renders/settle.py new file mode 100644 index 0000000000..90270dc3df --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/waiting_renders/settle.py @@ -0,0 +1,100 @@ +"""Queue a waiting document once its pictures are drawn, or fail it when one is not. + +Run where a picture's job ends, when a waiting document is made, and when the +worker starts, since a picture can end while no worker is running. +""" + +from sqlalchemy import func, select, update +from sqlalchemy.orm import Session + +from modules.artifacts.drawn_images.names import image_name +from modules.artifacts.models import Artifact +from modules.artifacts.queued_job import queue_job +from modules.artifacts.waiting_renders.waits import KEY, waits_for +from modules.documents.models import Document, DocumentStatus, DocumentType + +_ENDED_BADLY = (DocumentStatus.FAILED, DocumentStatus.CANCELLED) + + +def settle_waiting_renders( + session: Session, *, picture_id: int | None = None, render_id: int | None = None +) -> list[Artifact]: + """The waiting documents this settled, queued or failed, for the caller to announce.""" + settled: list[Artifact] = [] + for render in _waiting(session, picture_id, render_id): + pictures = waits_for(render.artifact_metadata) + statuses = _statuses(session, pictures) + gone = [ + i + for i in pictures + if statuses.get(i) is None or statuses[i] in _ENDED_BADLY + ] + if gone: + _fail(session, render, gone[0], statuses.get(gone[0])) + settled.append(render) + elif all(statuses[i] is DocumentStatus.READY for i in pictures) and _claim( + session, render + ): + queue_job(session, render) + settled.append(render) + session.commit() + return settled + + +def _waiting( + session: Session, picture_id: int | None, render_id: int | None +) -> list[Artifact]: + query = ( + select(Artifact) + .join(Document, Artifact.document_id == Document.id) + .where( + Document.status == DocumentStatus.PENDING, + Document.document_type == DocumentType.ARTIFACT, + func.json_extract(Artifact.artifact_metadata, f"$.{KEY}").is_not(None), + ) + ) + if render_id is not None: + query = query.where(Artifact.id == render_id) + found = session.scalars(query).all() + if picture_id is None: + return list(found) + return [r for r in found if picture_id in waits_for(r.artifact_metadata)] + + +def _statuses( + session: Session, artifact_ids: tuple[int, ...] +) -> dict[int, DocumentStatus]: + rows = session.execute( + select(Artifact.id, Document.status) + .join(Document, Artifact.document_id == Document.id) + .where(Artifact.id.in_(artifact_ids)) + ) + return dict(rows.tuples().all()) + + +def _claim(session: Session, render: Artifact) -> bool: + """Take the wait off in one update, so two pictures ending at once queue it once.""" + meta = {k: v for k, v in (render.artifact_metadata or {}).items() if k != KEY} + claimed = session.execute( + update(Artifact) + .where( + Artifact.id == render.id, + func.json_extract(Artifact.artifact_metadata, f"$.{KEY}").is_not(None), + ) + .values(artifact_metadata=meta) + .execution_options(synchronize_session=False) + ) + session.expire(render) + return claimed.rowcount == 1 + + +def _fail( + session: Session, render: Artifact, picture_id: int, status: DocumentStatus | None +) -> None: + name = image_name(picture_id) + what = "was deleted" if status is None else "failed" + render.document.status = DocumentStatus.FAILED + render.document.error_message = ( + f"{name} {what}, so this document was not made. Draw that picture again, " + "or render without it." + ) diff --git a/surfsense_local/backend/modules/artifacts/waiting_renders/waits.py b/surfsense_local/backend/modules/artifacts/waiting_renders/waits.py new file mode 100644 index 0000000000..e66fbe761b --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/waiting_renders/waits.py @@ -0,0 +1,25 @@ +"""The pictures a pending document waits for, kept in its metadata until it is queued.""" + +from typing import Any + +from modules.artifacts.drawn_images.names import image_name + +KEY = "waits_for" + + +def waits_for(metadata: dict[str, Any] | None) -> tuple[int, ...]: + return tuple(int(i) for i in (metadata or {}).get(KEY) or ()) + + +def waiting_sentence(artifact_ids: tuple[int, ...]) -> str: + """How the render's call answers: what it waits for, and not to render again.""" + names = [image_name(i) for i in artifact_ids] + drawn = ( + f"{names[0]} is drawn" + if len(names) == 1 + else f"{', '.join(names[:-1])} and {names[-1]} are drawn" + ) + return ( + f"It renders once {drawn}, and appears in Studio then: tell the user so, " + "and do not render it again." + ) diff --git a/surfsense_local/backend/modules/llm/connections/service.py b/surfsense_local/backend/modules/llm/connections/service.py index 2a77b5965b..3426857736 100644 --- a/surfsense_local/backend/modules/llm/connections/service.py +++ b/surfsense_local/backend/modules/llm/connections/service.py @@ -76,10 +76,10 @@ def normalize_base_url(value: str) -> str: return urlunsplit((parsed.scheme, netloc, path, "", "")) -def _modalities(entry: dict) -> set[str]: - raw = entry.get("output_modalities") +def _modalities(entry: dict, key: str = "output_modalities") -> set[str]: + raw = entry.get(key) if raw is None and isinstance(entry.get("architecture"), dict): - raw = entry["architecture"].get("output_modalities") + raw = entry["architecture"].get(key) if not isinstance(raw, list): return set() return { @@ -92,6 +92,7 @@ def _modalities(entry: dict) -> set[str]: @dataclass class _Entry: modalities: set[str] = field(default_factory=set) + inputs: set[str] = field(default_factory=set) voices: tuple[str, ...] = () free: bool | None = None @@ -116,6 +117,7 @@ def _entries(payload: object) -> dict[str, _Entry]: continue listed = entries.setdefault(name, _Entry()) listed.modalities.update(_modalities(entry)) + listed.inputs.update(_modalities(entry, "input_modalities")) listed.voices = listed.voices or tuple(voices_of(entry)) if listed.free is None: listed.free = listed_as_free(entry) @@ -123,7 +125,10 @@ def _entries(payload: object) -> dict[str, _Entry]: def _classify( - name: str, modalities: set[str], catalog_provider: str | None = None + name: str, + modalities: set[str], + catalog_provider: str | None = None, + inputs: frozenset[str] | set[str] = frozenset(), ) -> DiscoveredModel: """Resolve one model against every source, in order, first answer winning. @@ -144,6 +149,9 @@ def _classify( if modality in modalities ) ) + if ModelType.IMAGE_GEN in declared and "image" in inputs: + # A picture in as well as out: it can edit, and keep a reference. + declared = (*declared, ModelType.IMAGE_EDIT) return DiscoveredModel(name, declared, "declared", unusable) catalogued = lookup.classify(name, provider=catalog_provider) if catalogued.known: @@ -155,7 +163,8 @@ def _classify( def parse_models(payload: object) -> list[DiscoveredModel]: return [ - _classify(name, listed.modalities) for name, listed in _entries(payload).items() + _classify(name, listed.modalities, inputs=listed.inputs) + for name, listed in _entries(payload).items() ] @@ -177,7 +186,10 @@ async def _request_entries( async def probe_connection(base_url: str, api_key: str | None) -> list[DiscoveredModel]: entries = await _request_entries(base_url, api_key) - return [_classify(name, listed.modalities) for name, listed in entries.items()] + return [ + _classify(name, listed.modalities, inputs=listed.inputs) + for name, listed in entries.items() + ] async def discover_models(connection: ProviderConnection) -> list[DiscoveredModel]: @@ -189,6 +201,7 @@ async def discover_models(connection: ProviderConnection) -> list[DiscoveredMode for name, other in optional.items(): listed = merged.setdefault(name, _Entry()) listed.modalities.update(other.modalities) + listed.inputs.update(other.inputs) listed.voices = listed.voices or other.voices if listed.free is None: listed.free = other.free @@ -199,7 +212,9 @@ async def discover_models(connection: ProviderConnection) -> list[DiscoveredMode return sorted( ( replace( - classified := _classify(name, listed.modalities, provider), + classified := _classify( + name, listed.modalities, provider, listed.inputs + ), voices=tuple(Voice(v, v, None, ()) for v in listed.voices), unusable_reason=unusable_reason( provider, listed.free, classified.unusable_reason diff --git a/surfsense_local/backend/modules/llm/pictures_in/__init__.py b/surfsense_local/backend/modules/llm/pictures_in/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/modules/llm/pictures_in/kept.py b/surfsense_local/backend/modules/llm/pictures_in/kept.py new file mode 100644 index 0000000000..f46bf03510 --- /dev/null +++ b/surfsense_local/backend/modules/llm/pictures_in/kept.py @@ -0,0 +1,20 @@ +"""Whether the chosen image model takes a picture in, kept with the choice. + +Read from the connection's model list when the model is chosen, so nothing asks +the network again when a picture is edited or a page needs a reference. +""" + +from modules.llm.models import SelectedModel + +KEY = "takes_pictures" + + +def kept_takes_pictures(selected: SelectedModel) -> bool: + return bool((selected.settings or {}).get(KEY)) + + +def keep_takes_pictures(selected: SelectedModel, takes_pictures: bool) -> None: + settings = {k: v for k, v in (selected.settings or {}).items() if k != KEY} + if takes_pictures: + settings[KEY] = True + selected.settings = settings or None diff --git a/surfsense_local/backend/modules/llm/pictures_in/model.py b/surfsense_local/backend/modules/llm/pictures_in/model.py new file mode 100644 index 0000000000..f3f5fdc426 --- /dev/null +++ b/surfsense_local/backend/modules/llm/pictures_in/model.py @@ -0,0 +1,54 @@ +"""The one model a picture goes to: the edit model when one is chosen, else the +image model when it takes pictures, else none and the picture is drawn from words.""" + +from dataclasses import dataclass +from typing import Literal + +from sqlalchemy.orm import Session + +from modules.llm.catalog.local.dependencies import get_local_catalog +from modules.llm.model_type import ModelType +from modules.llm.models import SelectedModel +from modules.llm.pictures_in.kept import kept_takes_pictures +from modules.llm.providers.protocols import ImageEditor +from modules.llm.providers.sdcpp import provider as sdcpp +from modules.llm.resolution import ModelResolutionError, image_editor_for + +PictureSource = Literal["edit_model", "image_model"] + + +@dataclass(frozen=True) +class PictureModel: + source: PictureSource + selection: SelectedModel + editor: ImageEditor + + +def picture_model(session: Session) -> PictureModel | None: + """None when no chosen model takes a picture in. Egress and a missing install + raise ModelResolutionError, as resolving any model does.""" + chosen = session.get(SelectedModel, ModelType.IMAGE_EDIT) + if chosen is not None: + return PictureModel( + "edit_model", chosen, image_editor_for(session, chosen).editor + ) + image = session.get(SelectedModel, ModelType.IMAGE_GEN) + if image is None or not _takes_pictures(image): + return None + return PictureModel("image_model", image, image_editor_for(session, image).editor) + + +def picture_source(session: Session) -> PictureSource | None: + """The same answer without building a client: what the app shows.""" + try: + found = picture_model(session) + except ModelResolutionError: + return None + return None if found is None else found.source + + +def _takes_pictures(selected: SelectedModel) -> bool: + if selected.provider == sdcpp.PROVIDER: + installed = get_local_catalog().sdcpp.installed_image(selected.name) + return installed is not None and ModelType.IMAGE_EDIT in installed.types + return kept_takes_pictures(selected) diff --git a/surfsense_local/backend/modules/llm/pictures_in/router.py b/surfsense_local/backend/modules/llm/pictures_in/router.py new file mode 100644 index 0000000000..df98eea266 --- /dev/null +++ b/surfsense_local/backend/modules/llm/pictures_in/router.py @@ -0,0 +1,23 @@ +"""Which model a picture goes to, for the app: an edit, or a redraw when none.""" + +from fastapi import APIRouter +from pydantic import BaseModel + +from api.dependencies import SessionDep +from modules.llm.pictures_in.model import PictureSource, picture_source + +router = APIRouter(prefix="/llm", tags=["llm"]) + + +class PicturesInRead(BaseModel): + # None: every change to a picture is drawn again from its prompt. + source: PictureSource | None + + +@router.get( + "/pictures-in", + response_model=PicturesInRead, + summary="Which chosen model takes a picture in, if any", +) +def read_pictures_in(session: SessionDep) -> PicturesInRead: + return PicturesInRead(source=picture_source(session)) diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py index c054450a47..082e748086 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py @@ -4,12 +4,12 @@ from modules.llm.connections.key_headers import key_headers from modules.llm.providers.openai_compatible.image_response import ( - MAX_RESPONSE_BYTES, PICTURE_ITSELF, NonRetryableImageError, - bounded_body, read_image, + reply_payload, ) +from modules.llm.providers.picture_shape import ASPECT_RATIO, SHAPES, SIZE, Shape from modules.llm.providers.protocols import GeneratedImage __all__ = ["NonRetryableImageError", "OpenAICompatibleImageProvider"] @@ -21,6 +21,10 @@ _route_cache: dict[tuple[int, str], str] = {} # Whether an endpoint takes PICTURE_ITSELF; GPT image models refuse it. _asks_for_picture: dict[tuple[int, str], bool] = {} +# Endpoints that refuse a size: they draw their own. +_refuses_size: set[tuple[int, str]] = set() +# OpenRouter's /images names a shape by its ratio, as it does for references. +_RATIO_OF_SIZE = {SIZE[shape]: ASPECT_RATIO[shape] for shape in SHAPES} class OpenAICompatibleImageProvider: @@ -37,20 +41,23 @@ def __init__( def _headers(self) -> dict[str, str]: return key_headers(self._base_url, self._api_key) - async def generate(self, model: str, prompt: str) -> GeneratedImage: + async def generate( + self, model: str, prompt: str, *, shape: Shape | None = None + ) -> GeneratedImage: key = (self._connection_id, self._base_url) + size = None if shape is None or key in _refuses_size else SIZE[shape] preferred = _route_cache.get(key, STANDARD_ROUTE) alternate = EXTENSION_ROUTE if preferred == STANDARD_ROUTE else STANDARD_ROUTE picture = _asks_for_picture.get(key, True) try: - status, payload, picture = await self._ask( - preferred, model, prompt, picture + status, payload, picture, size = await self._ask( + preferred, model, prompt, picture, size ) if status in {404, 405}: - status, payload, picture = await self._ask( - alternate, model, prompt, picture + status, payload, picture, size = await self._ask( + alternate, model, prompt, picture, size ) route = alternate else: @@ -65,21 +72,38 @@ async def generate(self, model: str, prompt: str) -> GeneratedImage: _route_cache[key] = route _asks_for_picture[key] = picture + if shape is not None and size is None: + _refuses_size.add(key) return image async def _ask( - self, route: str, model: str, prompt: str, picture: bool - ) -> tuple[int, object, bool]: - """Once more without PICTURE_ITSELF if it is refused: a 400 comes before - any work, so nothing is billed twice.""" - status, payload = await self._post(route, model, prompt, picture) + self, route: str, model: str, prompt: str, picture: bool, size: str | None + ) -> tuple[int, object, bool, str | None]: + """Once more without PICTURE_ITSELF, then without the size, each if it is + refused: a 400 comes before any work, so nothing is billed twice.""" + status, payload = await self._sent(route, model, prompt, picture, size) if status == 400 and picture: - status, payload = await self._post(route, model, prompt, False) picture = False - return status, payload, picture + status, payload = await self._sent(route, model, prompt, picture, size) + if status == 400 and size is not None: + size = None + status, payload = await self._sent(route, model, prompt, picture, size) + return status, payload, picture, size + + async def _sent( + self, route: str, model: str, prompt: str, picture: bool, size: str | None + ) -> tuple[int, object]: + if size is None: + return await self._post(route, model, prompt, picture) + return await self._post(route, model, prompt, picture, size=size) async def _post( - self, route: str, model: str, prompt: str, picture: bool + self, + route: str, + model: str, + prompt: str, + picture: bool, + size: str | None = None, ) -> tuple[int, object]: async with ( httpx.AsyncClient(timeout=TIMEOUT, headers=self._headers()) as client, @@ -90,15 +114,14 @@ async def _post( "model": model, "prompt": prompt, **(PICTURE_ITSELF if picture else {}), + **(_shape_field(route, size) if size else {}), }, ) as reply, ): - content = await bounded_body(reply, MAX_RESPONSE_BYTES) - try: - payload = json.loads(content) - except json.JSONDecodeError: - if reply.status_code >= 400: - payload = {} - else: - raise - return reply.status_code, payload + return reply.status_code, await reply_payload(reply) + + +def _shape_field(route: str, size: str) -> dict[str, str]: + if route == EXTENSION_ROUTE: + return {"aspect_ratio": _RATIO_OF_SIZE[size]} + return {"size": size} diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py index 170c33a41b..4c6c25521f 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py @@ -1,25 +1,38 @@ -"""A remote edit model, reached on `/images/edits` with one picture and its instruction.""" +"""A remote model given pictures and an instruction: an edit, or a page that keeps +its references. `/images/edits` as OpenAI takes it; `/images` with +`input_references` where a server has no edit route, as OpenRouter.""" +import base64 import json import httpx from modules.llm.connections.key_headers import key_headers from modules.llm.providers.openai_compatible.image_response import ( - MAX_RESPONSE_BYTES, PICTURE_ITSELF, NonRetryableImageError, - bounded_body, read_image, + reply_payload, ) +from modules.llm.providers.picture_shape import ASPECT_RATIO, SIZE, Shape from modules.llm.providers.protocols import GeneratedImage # An edit reads the picture as well as drawing one: slower than a generation. TIMEOUT = httpx.Timeout(300.0, connect=5.0) -ROUTE = "/images/edits" +EDITS = "/images/edits" +REFERENCES = "/images" +# A missing route answers before any work, so trying the other bills nothing. +_NO_ROUTE = {404, 405} -# Whether an endpoint takes PICTURE_ITSELF; GPT image models refuse it. -_asks_for_picture: dict[str, bool] = {} +# Optional fields, dropped one at a time when a server refuses them: GPT image +# models refuse `response_format` and take a size, so each is its own. +PICTURE = "response_format" +SHAPE = "shape" +_OPTIONAL = (PICTURE, SHAPE) + +# Per server: which route takes pictures, and which optional fields it refuses. +_routes: dict[str, str] = {} +_refused: dict[str, frozenset[str]] = {} class OpenAICompatibleImageEditor: @@ -35,15 +48,32 @@ def __init__( self._transport = transport async def edit( - self, model: str, prompt: str, image: GeneratedImage + self, + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: Shape | None = None, ) -> GeneratedImage: - picture = _asks_for_picture.get(self._base_url, True) + route = _routes.get(self._base_url, EDITS) + refused = _refused.get(self._base_url, frozenset()) try: - status, payload = await self._post(model, prompt, image, picture) - # Once more without it if refused: a 400 comes before any work. - if status == 400 and picture: - picture = False - status, payload = await self._post(model, prompt, image, picture) + status, payload = await self._post( + route, model, prompt, images, shape, refused + ) + if status in _NO_ROUTE and route == EDITS: + route = REFERENCES + status, payload = await self._post( + route, model, prompt, images, shape, refused + ) + # Without each optional field in turn if refused: a 400 comes before any work. + for field in _OPTIONAL: + if status != 400 or field not in _sent(route, shape, refused): + continue + refused = refused | {field} + status, payload = await self._post( + route, model, prompt, images, shape, refused + ) if status >= 400: raise NonRetryableImageError(f"image endpoint returned HTTP {status}") edited = await read_image(payload) @@ -51,39 +81,95 @@ async def edit( raise except (httpx.HTTPError, ValueError, KeyError, json.JSONDecodeError) as error: raise NonRetryableImageError(str(error)) from error - _asks_for_picture[self._base_url] = picture + _routes[self._base_url] = route + _refused[self._base_url] = refused return edited async def _post( - self, model: str, prompt: str, image: GeneratedImage, picture: bool + self, + route: str, + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + shape: Shape | None, + refused: frozenset[str], ) -> tuple[int, object]: - extension = image.media_type.removeprefix("image/").replace("jpeg", "jpg") - # No `input_fidelity`: only some GPT image models take it. - async with ( - httpx.AsyncClient( - timeout=TIMEOUT, - headers=key_headers(self._base_url, self._api_key), - transport=self._transport, - ) as client, - client.stream( - "POST", - f"{self._base_url}{ROUTE}", - data={ - "model": model, - "prompt": prompt, - **(PICTURE_ITSELF if picture else {}), - }, - files={ - "image": (f"image.{extension}", image.content, image.media_type) - }, - ) as reply, - ): - content = await bounded_body(reply, MAX_RESPONSE_BYTES) - try: - payload = json.loads(content) - except json.JSONDecodeError: - if reply.status_code >= 400: - payload = {} - else: - raise - return reply.status_code, payload + async with httpx.AsyncClient( + timeout=TIMEOUT, + headers=key_headers(self._base_url, self._api_key), + transport=self._transport, + ) as client: + if route == REFERENCES: + request = client.stream( + "POST", + f"{self._base_url}{route}", + json=_references_body(model, prompt, images, shape, refused), + ) + else: + request = client.stream( + "POST", + f"{self._base_url}{route}", + data=_edits_fields(model, prompt, shape, refused), + files=_edits_files(images), + ) + async with request as reply: + return reply.status_code, await reply_payload(reply) + + +def _sent(route: str, shape: Shape | None, refused: frozenset[str]) -> set[str]: + """The optional fields a request carries: dropping one it lacks resends it unchanged.""" + carried = {PICTURE} if route == EDITS else set() + if shape is not None: + carried.add(SHAPE) + return carried - refused + + +def _edits_fields( + model: str, prompt: str, shape: Shape | None, refused: frozenset[str] +) -> dict[str, str]: + # No `input_fidelity`: only some GPT image models take it. + fields = {"model": model, "prompt": prompt} + if PICTURE not in refused: + fields.update(PICTURE_ITSELF) + if shape is not None and SHAPE not in refused: + fields["size"] = SIZE[shape] + return fields + + +def _edits_files(images: tuple[GeneratedImage, ...]) -> list[tuple[str, tuple]]: + # One picture keeps the field every edit server takes; several are OpenAI's `image[]`. + field = "image" if len(images) == 1 else "image[]" + return [ + (field, (f"image-{n}.{_extension(image)}", image.content, image.media_type)) + if len(images) > 1 + else (field, (f"image.{_extension(image)}", image.content, image.media_type)) + for n, image in enumerate(images, 1) + ] + + +def _references_body( + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + shape: Shape | None, + refused: frozenset[str], +) -> dict: + body: dict = { + "model": model, + "prompt": prompt, + "input_references": [ + {"type": "image_url", "image_url": {"url": _data_url(image)}} + for image in images + ], + } + if shape is not None and SHAPE not in refused: + body["aspect_ratio"] = ASPECT_RATIO[shape] + return body + + +def _data_url(image: GeneratedImage) -> str: + return f"data:{image.media_type};base64,{base64.b64encode(image.content).decode()}" + + +def _extension(image: GeneratedImage) -> str: + return image.media_type.removeprefix("image/").replace("jpeg", "jpg") diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py index c001b53d28..99a39d28b0 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py @@ -2,6 +2,7 @@ import base64 import binascii +import json from urllib.parse import urlsplit import httpx @@ -42,6 +43,18 @@ async def read_image(payload: object) -> GeneratedImage: return await download_image(url) +async def reply_payload(reply: httpx.Response) -> object: + """The reply's JSON. A refusal is decided by its status, so one whose body is + cut off, or is not JSON, still answers with its status: OpenRouter has hung up + partway through a 404's body. Only a success must arrive whole.""" + if reply.status_code < 400: + return json.loads(await bounded_body(reply, MAX_RESPONSE_BYTES)) + try: + return json.loads(await bounded_body(reply, MAX_RESPONSE_BYTES)) + except (httpx.TransportError, ValueError): + return {} + + async def bounded_body(reply: httpx.Response, limit: int) -> bytes: length = reply.headers.get("content-length") if length is not None and length.isdigit() and int(length) > limit: diff --git a/surfsense_local/backend/modules/llm/providers/picture_shape.py b/surfsense_local/backend/modules/llm/providers/picture_shape.py new file mode 100644 index 0000000000..e0843f8fe6 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/picture_shape.py @@ -0,0 +1,44 @@ +"""A picture's proportions, as each kind of image endpoint asks for them.""" + +from typing import Literal + +Shape = Literal["square", "portrait", "landscape"] +SHAPES: tuple[Shape, ...] = ("square", "portrait", "landscape") + +# The sizes OpenAI's image models take; other servers round to their own. +SIZE: dict[Shape, str] = { + "square": "1024x1024", + "portrait": "1024x1536", + "landscape": "1536x1024", +} +ASPECT_RATIO: dict[Shape, str] = { + "square": "1:1", + "portrait": "2:3", + "landscape": "3:2", +} +_RATIO: dict[Shape, tuple[int, int]] = { + "square": (1, 1), + "portrait": (2, 3), + "landscape": (3, 2), +} + + +def local_dimensions(shape: Shape, pixels: int) -> tuple[int, int]: + """About `pixels` in all, at the shape's ratio, each side a multiple of 64 as sd.cpp needs.""" + across, down = _RATIO[shape] + unit = (pixels / (across * down)) ** 0.5 + return _to_64(across * unit), _to_64(down * unit) + + +def _to_64(side: float) -> int: + return max(64, round(side / 64) * 64) + + +def shape_of(width: int, height: int) -> Shape: + """The nearest shape to a picture's own proportions, so a redraw keeps them.""" + ratio = width / height + if ratio > 1.2: + return "landscape" + if ratio < 1 / 1.2: + return "portrait" + return "square" diff --git a/surfsense_local/backend/modules/llm/providers/protocols.py b/surfsense_local/backend/modules/llm/providers/protocols.py index 3aed8c942f..54cb690d02 100644 --- a/surfsense_local/backend/modules/llm/providers/protocols.py +++ b/surfsense_local/backend/modules/llm/providers/protocols.py @@ -2,6 +2,7 @@ from dataclasses import dataclass from typing import Protocol +from modules.llm.providers.picture_shape import Shape from modules.llm.providers.types import Delta, Message, Model @@ -81,14 +82,24 @@ class GeneratedImage: class ImageGenerator(Protocol): - async def generate(self, model: str, prompt: str) -> GeneratedImage: ... + """A model that draws from words alone.""" + + async def generate( + self, model: str, prompt: str, *, shape: Shape | None = None + ) -> GeneratedImage: ... class ImageEditor(Protocol): - """A model that changes a picture as an instruction says, given it as a reference.""" + """A model given pictures as references: an edit of one, or a new picture + that keeps the characters and style of the others.""" async def edit( - self, model: str, prompt: str, image: GeneratedImage + self, + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: Shape | None = None, ) -> GeneratedImage: ... diff --git a/surfsense_local/backend/modules/llm/providers/sdcpp/editor.py b/surfsense_local/backend/modules/llm/providers/sdcpp/editor.py index 16ac29b066..125564c7eb 100644 --- a/surfsense_local/backend/modules/llm/providers/sdcpp/editor.py +++ b/surfsense_local/backend/modules/llm/providers/sdcpp/editor.py @@ -1,32 +1,23 @@ """Studio's edit client for the bundled sd-server: its native job route, with the -picture as a reference image, as every edit model sd.cpp documents is run. +pictures as reference images, as every edit model sd.cpp documents is run. Its `/v1/images/edits` and `/sdapi/v1/img2img` also make the picture the starting noise, which re-draws it at strength 0.75 instead of following the instruction. """ -import asyncio import base64 -import contextlib import io -from collections.abc import Awaitable, Callable +from collections.abc import Callable from contextlib import AbstractAsyncContextManager import httpx from PIL import Image -from modules.llm.providers.openai_compatible.image_response import ( - NonRetryableImageError, - decode_image, -) +from modules.llm.providers.picture_shape import Shape, local_dimensions from modules.llm.providers.protocols import GeneratedImage +from modules.llm.providers.sdcpp.native_job import run_native_job from modules.llm.providers.sdcpp.serving import wait_until_serving -from shared import cancellation - -_TIMEOUT = httpx.Timeout(30.0, connect=5.0) -# A finished job's result is kept 600 seconds, so a poll each second meets it. -_ENDED = ("completed", "failed", "cancelled") class LocalImageEditor: @@ -46,7 +37,12 @@ def __init__( self._transport = transport async def edit( - self, model: str, prompt: str, image: GeneratedImage + self, + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: Shape | None = None, ) -> GeneratedImage: # Held for the whole edit, as for a drawn image (generator.py). async with self._give_up_text_runtime(): @@ -56,68 +52,27 @@ async def edit( interval=self._interval, transport=self._transport, ) - async with httpx.AsyncClient( - base_url=self._root_url, timeout=_TIMEOUT, transport=self._transport - ) as client: - job = await self._submit(client, prompt, image) - try: - return await self._result(client, job) - except BaseException: - # Only a queued job can be withdrawn; a generating one runs on - # until Electron stops sd-server, once the cancel ends its demand. - await _withdraw(client, job) - raise - - async def _submit( - self, client: httpx.AsyncClient, prompt: str, image: GeneratedImage - ) -> str: - width, height = Image.open(io.BytesIO(image.content)).size - encoded = base64.b64encode(image.content).decode() - # A native request starts from the launch's settings, so steps and - # guidance are the model's own; the size must be the picture's. - body = { - "prompt": prompt, - "ref_images": [f"data:{image.media_type};base64,{encoded}"], - "width": width, - "height": height, - "output_format": "png", - } - reply = await _call(client.post("/sdcpp/v1/img_gen", json=body)) - return reply.json()["id"] - - async def _result(self, client: httpx.AsyncClient, job: str) -> GeneratedImage: - while True: - cancellation.raise_if_cancelled() - state = (await _call(client.get(f"/sdcpp/v1/jobs/{job}"))).json() - if state["status"] == "completed": - return decode_image(state["result"]["images"][0]["b64_json"], None) - if state["status"] in _ENDED: - error = state.get("error") or {} - raise NonRetryableImageError( - f"The image model could not edit this picture: " - f"{error.get('message') or state['status']}" - ) - await asyncio.sleep(self._interval) - - -async def _call(request: Awaitable[httpx.Response]) -> httpx.Response: - """The reply, or the edit's failure: a full queue, a job gone, a server down.""" - try: - reply = await request - except httpx.HTTPError as error: - raise NonRetryableImageError( - f"The image model could not be reached: {error}" - ) from error - if reply.status_code >= 400: - raise NonRetryableImageError(f"sd-server returned HTTP {reply.status_code}") - return reply - - -# The text runtime is still held while it is asked; a cancel ends the demand -# anyway, and Electron then stops sd-server. -_WITHDRAW_TIMEOUT = httpx.Timeout(2.0) + return await run_native_job( + self._root_url, + _body(prompt, images, shape), + interval=self._interval, + transport=self._transport, + ) -async def _withdraw(client: httpx.AsyncClient, job: str) -> None: - with contextlib.suppress(httpx.HTTPError): - await client.post(f"/sdcpp/v1/jobs/{job}/cancel", timeout=_WITHDRAW_TIMEOUT) +def _body(prompt: str, images: tuple[GeneratedImage, ...], shape: Shape | None) -> dict: + # A native request starts from the launch's settings, so steps and guidance + # are the model's own; the size must be the picture's, or the shape's. + width, height = Image.open(io.BytesIO(images[0].content)).size + if shape is not None: + width, height = local_dimensions(shape, width * height) + return { + "prompt": prompt, + "ref_images": [ + f"data:{image.media_type};base64,{base64.b64encode(image.content).decode()}" + for image in images + ], + "width": width, + "height": height, + "output_format": "png", + } diff --git a/surfsense_local/backend/modules/llm/providers/sdcpp/generator.py b/surfsense_local/backend/modules/llm/providers/sdcpp/generator.py index 1c65e28259..6102cd4688 100644 --- a/surfsense_local/backend/modules/llm/providers/sdcpp/generator.py +++ b/surfsense_local/backend/modules/llm/providers/sdcpp/generator.py @@ -1,15 +1,21 @@ """Studio's image client for the bundled sd-server: the OpenAI-compatible one, once sd-server serves the model the job needs and the text runtime has given up -the graphics card.""" +the graphics card. A shape goes through the native job route, the one that +takes a width and height per request.""" from collections.abc import Callable from contextlib import AbstractAsyncContextManager import httpx +from modules.llm.providers.picture_shape import Shape, local_dimensions from modules.llm.providers.protocols import GeneratedImage, ImageGenerator +from modules.llm.providers.sdcpp.native_job import run_native_job from modules.llm.providers.sdcpp.serving import wait_until_serving +# sd-server's own default when a model's launch names no size. +DEFAULT_PIXELS = 1024 * 1024 + class LocalImageGenerator: def __init__( @@ -19,6 +25,7 @@ def __init__( served_file: str, give_up_text_runtime: Callable[[], AbstractAsyncContextManager[None]], *, + pixels: int = DEFAULT_PIXELS, interval: float = 1.0, transport: httpx.AsyncBaseTransport | None = None, ) -> None: @@ -26,10 +33,13 @@ def __init__( self._root_url = root_url self._served_file = served_file self._give_up_text_runtime = give_up_text_runtime + self._pixels = pixels self._interval = interval self._transport = transport - async def generate(self, model: str, prompt: str) -> GeneratedImage: + async def generate( + self, model: str, prompt: str, *, shape: Shape | None = None + ) -> GeneratedImage: # sd-server loads its weights on this post, and Windows reports the card # as free while llama-server holds it. Held for the whole image, so a # chat sent meanwhile cannot load a text model back under it. @@ -40,4 +50,17 @@ async def generate(self, model: str, prompt: str) -> GeneratedImage: interval=self._interval, transport=self._transport, ) - return await self._inner.generate(model, prompt) + if shape is None: + return await self._inner.generate(model, prompt) + width, height = local_dimensions(shape, self._pixels) + return await run_native_job( + self._root_url, + { + "prompt": prompt, + "width": width, + "height": height, + "output_format": "png", + }, + interval=self._interval, + transport=self._transport, + ) diff --git a/surfsense_local/backend/modules/llm/providers/sdcpp/launch_size.py b/surfsense_local/backend/modules/llm/providers/sdcpp/launch_size.py new file mode 100644 index 0000000000..213563b659 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/sdcpp/launch_size.py @@ -0,0 +1,21 @@ +"""The pixel count a local model is launched to draw, from its `-W` and `-H`.""" + +from collections.abc import Sequence +from itertools import pairwise + +from modules.llm.providers.sdcpp.generator import DEFAULT_PIXELS + + +def launch_pixels(args: Sequence[str]) -> int: + """Its reviewed size, kept when a shape changes the proportions.""" + width, height = _flag(args, "-W"), _flag(args, "-H") + if width is None or height is None: + return DEFAULT_PIXELS + return width * height + + +def _flag(args: Sequence[str], name: str) -> int | None: + for flag, value in pairwise(args): + if flag == name and value.isdigit(): + return int(value) + return None diff --git a/surfsense_local/backend/modules/llm/providers/sdcpp/native_job.py b/surfsense_local/backend/modules/llm/providers/sdcpp/native_job.py new file mode 100644 index 0000000000..8334808ecd --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/sdcpp/native_job.py @@ -0,0 +1,81 @@ +"""One picture through sd-server's native job route: submit, poll each second, +and withdraw it if the wait ends early. The only route that takes reference +images and a size per request, so both drawing and editing use it.""" + +import asyncio +import contextlib +from collections.abc import Awaitable + +import httpx + +from modules.llm.providers.openai_compatible.image_response import ( + NonRetryableImageError, + decode_image, +) +from modules.llm.providers.protocols import GeneratedImage +from shared import cancellation + +_TIMEOUT = httpx.Timeout(30.0, connect=5.0) +# A finished job's result is kept 600 seconds, so a poll each second meets it. +_ENDED = ("completed", "failed", "cancelled") + + +async def run_native_job( + root_url: str, + body: dict, + *, + interval: float, + transport: httpx.AsyncBaseTransport | None, +) -> GeneratedImage: + async with httpx.AsyncClient( + base_url=root_url, timeout=_TIMEOUT, transport=transport + ) as client: + reply = await _call(client.post("/sdcpp/v1/img_gen", json=body)) + job = reply.json()["id"] + try: + return await _result(client, job, interval) + except BaseException: + # Only a queued job can be withdrawn; a generating one runs on + # until Electron stops sd-server, once the cancel ends its demand. + await _withdraw(client, job) + raise + + +async def _result( + client: httpx.AsyncClient, job: str, interval: float +) -> GeneratedImage: + while True: + cancellation.raise_if_cancelled() + state = (await _call(client.get(f"/sdcpp/v1/jobs/{job}"))).json() + if state["status"] == "completed": + return decode_image(state["result"]["images"][0]["b64_json"], None) + if state["status"] in _ENDED: + error = state.get("error") or {} + raise NonRetryableImageError( + f"The image model could not make this picture: " + f"{error.get('message') or state['status']}" + ) + await asyncio.sleep(interval) + + +async def _call(request: Awaitable[httpx.Response]) -> httpx.Response: + """The reply, or the edit's failure: a full queue, a job gone, a server down.""" + try: + reply = await request + except httpx.HTTPError as error: + raise NonRetryableImageError( + f"The image model could not be reached: {error}" + ) from error + if reply.status_code >= 400: + raise NonRetryableImageError(f"sd-server returned HTTP {reply.status_code}") + return reply + + +# The text runtime is still held while it is asked; a cancel ends the demand +# anyway, and Electron then stops sd-server. +_WITHDRAW_TIMEOUT = httpx.Timeout(2.0) + + +async def _withdraw(client: httpx.AsyncClient, job: str) -> None: + with contextlib.suppress(httpx.HTTPError): + await client.post(f"/sdcpp/v1/jobs/{job}/cancel", timeout=_WITHDRAW_TIMEOUT) diff --git a/surfsense_local/backend/modules/llm/resolution.py b/surfsense_local/backend/modules/llm/resolution.py index 14bc2670ee..4bd5a455b0 100644 --- a/surfsense_local/backend/modules/llm/resolution.py +++ b/surfsense_local/backend/modules/llm/resolution.py @@ -32,6 +32,7 @@ from modules.llm.providers.sdcpp import provider as sdcpp from modules.llm.providers.sdcpp.editor import LocalImageEditor from modules.llm.providers.sdcpp.generator import LocalImageGenerator +from modules.llm.providers.sdcpp.launch_size import launch_pixels from modules.llm.subscriptions.claude.account import CLAUDE_PLAN from modules.llm.subscriptions.claude.hosts import require_claude_hosts from modules.llm.subscriptions.claude.usable_claude import usable_claude @@ -119,6 +120,7 @@ def resolve_image_generation(session: Session) -> ResolvedImageGeneration: sdcpp.root_url(), image.served_file, text_runtime_given_up, + pixels=launch_pixels(image.args), ), ) connection = _connection(session, selected) @@ -135,6 +137,11 @@ def resolve_image_edit(session: Session) -> ResolvedImageEdit: selected = session.get(SelectedModel, ModelType.IMAGE_EDIT) if selected is None: raise ModelResolutionError("no image editing model selected") + return image_editor_for(session, selected) + + +def image_editor_for(session: Session, selected: SelectedModel) -> ResolvedImageEdit: + """The client that sends this chosen model a picture, whichever slot chose it.""" if selected.provider == sdcpp.PROVIDER: image = get_local_catalog().sdcpp.installed_image(selected.name) if image is None: diff --git a/surfsense_local/backend/modules/llm/selection.py b/surfsense_local/backend/modules/llm/selection.py index f930feda8e..c1121feec6 100644 --- a/surfsense_local/backend/modules/llm/selection.py +++ b/surfsense_local/backend/modules/llm/selection.py @@ -13,8 +13,10 @@ from modules.llm.connections.listing import connection_models from modules.llm.connections.router import allowed_connection from modules.llm.connections.serves import connection_serves +from modules.llm.connections.service import DiscoveredModel from modules.llm.model_type import ModelType from modules.llm.models import OnboardingCompletion, SelectedModel +from modules.llm.pictures_in.kept import keep_takes_pictures from modules.llm.profile import Fingerprint, from_name from modules.llm.providers import audiocpp, get_provider, llamacpp from modules.llm.providers.openai_compatible import OpenAICompatibleChatProvider @@ -43,6 +45,7 @@ async def choose_model( "model name must not be empty", ) + listed: DiscoveredModel | None = None if provider_name == llamacpp.PROVIDER: await _validate_local(model_type, model_name, connection_id) elif provider_name == sdcpp.PROVIDER: @@ -50,7 +53,7 @@ async def choose_model( elif provider_name == audiocpp.PROVIDER: _validate_local_audio(model_type, model_name, connection_id) elif provider_name == "openai_compatible": - await _validate_remote( + listed = await _validate_remote( session, model_type, model_name, connection_id, allow_unlisted ) else: @@ -60,6 +63,8 @@ async def choose_model( ) fingerprint = await _collect(session, provider_name, model_name, connection_id) + # An unlisted model is taken as words only: a picture it ignores is billed. + takes_pictures = listed is not None and ModelType.IMAGE_EDIT in listed.types selected = await transact( session, _store, @@ -68,6 +73,7 @@ async def choose_model( connection_id, model_name, fingerprint, + takes_pictures, ) if model_type is ModelType.SPEECH_TO_TEXT: # Recordings uploaded before any speech model waited for this one. @@ -116,6 +122,7 @@ def _store( connection_id: int | None, model_name: str, fingerprint: Fingerprint, + takes_pictures: bool = False, ) -> SelectedModel: selected = session.get(SelectedModel, model_type) if selected is None: @@ -135,6 +142,8 @@ def _store( selected.params_b = fingerprint.params_b selected.vendor = fingerprint.vendor selected.line = fingerprint.line + if model_type is ModelType.IMAGE_GEN: + keep_takes_pictures(selected, takes_pictures) session.flush() # updated_at is set by the database; load it here rather than lazily on the loop. session.refresh(selected) @@ -255,7 +264,8 @@ async def _validate_remote( model_name: str, connection_id: int | None, allow_unlisted: bool, -) -> None: +) -> DiscoveredModel | None: + """The model as its connection lists it; None when chosen unlisted.""" if connection_id is None: raise HTTPException( status.HTTP_422_UNPROCESSABLE_CONTENT, @@ -276,12 +286,12 @@ async def _validate_remote( status.HTTP_422_UNPROCESSABLE_CONTENT, "connection models could not be verified; confirm manual selection", ) from error - return + return None model = next((entry for entry in models if entry.name == model_name), None) if model is None: if allow_unlisted: - return + return None raise HTTPException( status.HTTP_422_UNPROCESSABLE_CONTENT, f"model is not listed by this connection: {model_name}", @@ -299,3 +309,4 @@ async def _validate_remote( status.HTTP_422_UNPROCESSABLE_CONTENT, f"model does not support {model_type.value}: {model_name}", ) + return model diff --git a/surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py b/surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py index 8807020899..2c44419b49 100644 --- a/surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py +++ b/surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py @@ -40,7 +40,9 @@ def _png() -> bytes: class Painter: - async def generate(self, _model: str, prompt: str) -> GeneratedImage: + async def generate( + self, _model: str, prompt: str, *, shape: str | None = None + ) -> GeneratedImage: self.prompt = prompt return GeneratedImage(_png(), "image/png") diff --git a/surfsense_local/backend/tests/integration/agent/test_bind_comic.py b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py new file mode 100644 index 0000000000..be16b596eb --- /dev/null +++ b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py @@ -0,0 +1,448 @@ +"""The agent binds the pages it drew into a comic: one Studio artifact, a PDF in reading order.""" + +import asyncio +import io +import threading +import time +from datetime import UTC, datetime, timedelta + +import pytest +from pypdf import PdfReader +from sqlalchemy import Engine, select + +from modules.artifacts.local_image_demand import local_image_demand +from modules.artifacts.models import Artifact, ArtifactFileRole +from modules.documents.models import DocumentStatus +from modules.llm.model_type import ModelType +from shared.config import get_storage_settings +from shared.db import create_session_factory +from shared.search import retrieve +from tests.integration.agent.test_make_image import ( # noqa: F401 - fixtures + Painter, + _artifact_id, + painter, +) +from tests.integration.agent.tool_endpoint_client import ToolEndpoint +from tests.integration.worker.conftest import stub_model # noqa: F401 + +pytestmark = [ + pytest.mark.integration, + pytest.mark.usefixtures("stub_model", "model_reads_images", "studio_worker"), +] + + +async def _picture(tools: ToolEndpoint, workspace_id: int, title: str) -> str: + text, is_error = await tools.call( + workspace_id, "make_image", {"prompt": f"A page: {title}", "title": title} + ) + assert is_error is False, text + return f"image-{_artifact_id(text)}" + + +def _comic(engine: Engine, artifact_id: int) -> tuple[str, dict, PdfReader]: + """Its format, its metadata and its PDF.""" + with create_session_factory(engine)() as session: + artifact = session.get(Artifact, artifact_id) + assert artifact is not None + primary = next(f for f in artifact.files if f.role is ArtifactFileRole.PRIMARY) + pdf = (get_storage_settings().data_dir / primary.storage_key).read_bytes() + return ( + artifact.format, + dict(artifact.artifact_metadata or {}), + PdfReader(io.BytesIO(pdf)), + ) + + +async def _ended(engine: Engine, artifact_id: int) -> DocumentStatus: + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + with create_session_factory(engine)() as session: + status = session.get(Artifact, artifact_id).document.status + if status not in (DocumentStatus.PENDING, DocumentStatus.PROCESSING): + return status + await asyncio.sleep(0.1) + raise AssertionError(f"artifact {artifact_id} never ended") + + +async def test_the_pages_become_one_comic_in_studio( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """One artifact in Studio, its pages in the order given, read as asked.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + second = await _picture(tools, workspace_id, "Cat · page 2") + + text, is_error = await tools.call( + workspace_id, + "bind_comic", + { + "title": "The Lighthouse Cat", + "pages": [ + {"image": first}, + {"image": second, "lines": ["NORA: The bulb is broken!"]}, + ], + "direction": "rtl", + }, + ) + + assert is_error is False, text + comic_id = _artifact_id(text) + assert text.splitlines()[0] == ( + f"Rendered artifact {comic_id}, version 1: The Lighthouse Cat" + ) + assert "2 pages" in text + format, _, pdf = _comic(engine, comic_id) + assert format == "comic" + assert len(pdf.pages) == 2 + assert "NORA: The bulb is broken!" in pdf.pages[1].extract_text() + assert pdf.trailer["/Root"]["/ViewerPreferences"]["/Direction"] == "/R2L" + listed = (await tools.client.get(f"/workspaces/{workspace_id}/artifacts")).json() + row = next(a for a in listed if a["id"] == comic_id) + assert (row["format"], row["status"]) == ("comic", "ready") + + +async def test_binding_again_with_its_id_makes_the_comics_next_version( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """A page is changed with make_image, then the comic is bound again around it.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + made, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": first}]} + ) + second = await _picture(tools, workspace_id, "Cat · page 2") + + text, is_error = await tools.call( + workspace_id, + "bind_comic", + { + "title": "Cat", + "pages": [{"image": first}, {"image": second}], + "artifact_id": _artifact_id(made), + }, + ) + + assert is_error is False, text + assert ", version 2: Cat" in text.splitlines()[0] + _, metadata, pdf = _comic(engine, _artifact_id(text)) + assert metadata["version"]["parent"] == _artifact_id(made) + assert len(pdf.pages) == 2 + + +async def test_a_page_still_being_drawn_is_waited_for( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A comic drawn slowly on this computer still ends as one PDF, without asking again.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + monkeypatch.setattr("modules.agent.tool_endpoint.make_image.WAIT_SECONDS", 0.2) + painter.hold = threading.Event() + second = await _picture(tools, workspace_id, "Cat · page 2") + + text, is_error = await tools.call( + workspace_id, + "bind_comic", + {"title": "Cat", "pages": [{"image": first}, {"image": second}]}, + ) + + assert is_error is False, text + assert text.startswith("Queued artifact ") + assert f"renders once {second} is drawn" in text + painter.hold.set() + assert await _ended(engine, _artifact_id(text)) is DocumentStatus.READY + + +@pytest.mark.parametrize( + "arguments, words", + [ + ({"pages": [{"image": "image-1"}]}, "Give the comic a title"), + ({"title": "Cat", "pages": []}, "Give the pages"), + ({"title": "Cat", "pages": [{"image": "image-999"}]}, "no picture named"), + ( + {"title": "Cat", "pages": [{"image": "image-1"}], "layout": "spiral"}, + "layout must be one of: pages, scroll", + ), + ( + {"title": "Cat", "pages": [{"image": "image-1"}] * 13}, + "at most 12 pages", + ), + ], +) +async def test_a_call_missing_something_says_what_to_fix( + tools: ToolEndpoint, arguments: dict, words: str +) -> None: + """A small model may leave arguments out; the refusal names the one to fix.""" + workspace_id = await tools.workspace() + + text, is_error = await tools.call(workspace_id, "bind_comic", arguments) + + assert is_error is True + assert words in text + + +async def test_binding_a_comic_does_not_start_the_local_image_model( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """Binding draws nothing: the image model's memory stays the chat model's.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + made, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": first}]} + ) + with create_session_factory(engine)() as session: + comic = session.get(Artifact, _artifact_id(made)) + comic.document.status = DocumentStatus.PROCESSING + session.commit() + + assert local_image_demand(session, datetime.now(UTC)) is None + + +async def test_a_picture_drawn_beside_a_binding_keeps_the_local_image_model( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """A binding that started first must not stop sd-server under the picture.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + second = await _picture(tools, workspace_id, "Cat · page 2") + made, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": first}]} + ) + with create_session_factory(engine)() as session: + comic = session.get(Artifact, _artifact_id(made)) + picture = session.get(Artifact, _id(second)) + started = datetime.now(UTC) + comic.document.status = DocumentStatus.PROCESSING + comic.document.updated_at = started - timedelta(seconds=30) + picture.document.status = DocumentStatus.PROCESSING + picture.document.updated_at = started + session.commit() + + assert local_image_demand(session, started) is ModelType.IMAGE_GEN + + +async def _listed_ids(tools: ToolEndpoint, workspace_id: int) -> set[int]: + listed = await tools.client.get(f"/workspaces/{workspace_id}/artifacts") + return {row["id"] for row in listed.json()} + + +async def _page(tools: ToolEndpoint, workspace_id: int, title: str) -> str: + text, is_error = await tools.call( + workspace_id, + "make_image", + {"prompt": f"A page: {title}", "title": title, "comic_page": True}, + ) + assert is_error is False, text + return f"image-{_artifact_id(text)}" + + +def _id(name: str) -> int: + return int(name.removeprefix("image-")) + + +async def test_only_the_comic_is_listed_never_its_pages( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 +) -> None: + """A comic is one artifact in Studio, from its first page drawn to its binding.""" + workspace_id = await tools.workspace() + first = await _page(tools, workspace_id, "Cat · page 1") + assert _id(first) not in await _listed_ids(tools, workspace_id) + second = await _page(tools, workspace_id, "Cat · page 2") + + made, _ = await tools.call( + workspace_id, + "bind_comic", + {"title": "Cat", "pages": [{"image": first}, {"image": second}]}, + ) + + assert await _listed_ids(tools, workspace_id) == {_artifact_id(made)} + + +async def test_a_page_the_agent_changes_stays_out_of_the_list( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 +) -> None: + """Page 3 again is the page's next version, and a page all the same.""" + workspace_id = await tools.workspace() + page = await _page(tools, workspace_id, "Cat · page 1") + + changed, is_error = await tools.call( + workspace_id, + "make_image", + {"prompt": "Make it night", "artifact_id": _id(page)}, + ) + + assert is_error is False, changed + assert await _listed_ids(tools, workspace_id) == set() + + +async def test_binding_hides_a_page_drawn_without_the_mark( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 +) -> None: + """A page the agent forgot to mark leaves the list once its comic binds it.""" + workspace_id = await tools.workspace() + page = await _picture(tools, workspace_id, "Cat · page 1") + + made, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": page}]} + ) + + assert await _listed_ids(tools, workspace_id) == {_artifact_id(made)} + + +async def test_deleting_a_comic_deletes_the_pages_no_other_version_binds( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """Pages cannot be deleted from the list, so they go with the comic, last of all.""" + workspace_id = await tools.workspace() + first = await _page(tools, workspace_id, "Cat · page 1") + v1, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": first}]} + ) + second = await _page(tools, workspace_id, "Cat · page 2") + v2, _ = await tools.call( + workspace_id, + "bind_comic", + { + "title": "Cat", + "pages": [{"image": first}, {"image": second}], + "artifact_id": _artifact_id(v1), + }, + ) + + deleted = await tools.client.delete(f"/artifacts/{_artifact_id(v2)}") + assert deleted.status_code == 204, deleted.text + with create_session_factory(engine)() as session: + assert session.get(Artifact, _id(second)) is None + assert session.get(Artifact, _id(first)) is not None + + deleted = await tools.client.delete(f"/artifacts/{_artifact_id(v1)}") + assert deleted.status_code == 204, deleted.text + with create_session_factory(engine)() as session: + assert session.get(Artifact, _id(first)) is None + + +async def test_deleting_a_comic_takes_a_pages_earlier_versions_too( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """A changed page's first drawing is still kept: nothing else would remove it.""" + workspace_id = await tools.workspace() + page = await _page(tools, workspace_id, "Cat · page 1") + changed, _ = await tools.call( + workspace_id, + "make_image", + {"prompt": "Make it night", "artifact_id": _id(page)}, + ) + made, _ = await tools.call( + workspace_id, + "bind_comic", + {"title": "Cat", "pages": [{"image": f"image-{_artifact_id(changed)}"}]}, + ) + + deleted = await tools.client.delete(f"/artifacts/{_artifact_id(made)}") + + assert deleted.status_code == 204, deleted.text + with create_session_factory(engine)() as session: + assert session.get(Artifact, _id(page)) is None + assert session.get(Artifact, _artifact_id(changed)) is None + + +def _found(engine: Engine, workspace_id: int, query: str) -> set[int]: + """The artifacts a workspace search finds, as a chat's sources would.""" + with create_session_factory(engine)() as session: + hits = retrieve(session, workspace_id, query, top_k=20) + return { + artifact.id + for hit in hits + if ( + artifact := session.scalar( + select(Artifact).where(Artifact.document_id == hit.document_id) + ) + ) + is not None + } + + +async def test_search_finds_the_comic_never_its_pages( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """A comic is found as a PDF is, by what it says, not page by page by its prompts.""" + workspace_id = await tools.workspace() + marked = await _page(tools, workspace_id, "Lighthouse storm page") + unmarked = await _picture(tools, workspace_id, "Lighthouse keeper page") + + made, _ = await tools.call( + workspace_id, + "bind_comic", + { + "title": "Lighthouse", + "pages": [ + {"image": marked}, + {"image": unmarked, "lines": ["NORA: The lighthouse is dark!"]}, + ], + }, + ) + + assert _found(engine, workspace_id, "lighthouse") == {_artifact_id(made)} + + +async def test_a_comic_is_found_by_what_its_pages_were_drawn_from( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """Its story and bubbles are in its pages' prompts, not in any text the PDF holds.""" + workspace_id = await tools.workspace() + style = "Bold ink lines, flat colours.\nCharacters: NORA, seventy, yellow raincoat." + names = [] + for number, panel in enumerate( + ( + 'NORA taps the dead lamp. NORA says: "THE BULB IS BROKEN!"', + "MILO finds spare bulbs on a high shelf.", + ), + start=1, + ): + text, _ = await tools.call( + workspace_id, + "make_image", + { + "prompt": f"{style}\nPage {number} of 2.\nPanel 1: {panel}", + "title": f"Cat · page {number}", + "comic_page": True, + }, + ) + names.append(f"image-{_artifact_id(text)}") + + made, _ = await tools.call( + workspace_id, + "bind_comic", + {"title": "The Lighthouse Cat", "pages": [{"image": n} for n in names]}, + ) + + comic = _artifact_id(made) + assert _found(engine, workspace_id, "broken bulb") == {comic} + assert _found(engine, workspace_id, "shelf") == {comic} + with create_session_factory(engine)() as session: + body = session.get(Artifact, comic).document.content or "" + assert 'NORA says: "THE BULB IS BROKEN!"' in body + assert "MILO finds spare bulbs on a high shelf." in body + # Said once, though every page's prompt repeats it. + assert body.count("yellow raincoat") == 1 diff --git a/surfsense_local/backend/tests/integration/agent/test_make_image.py b/surfsense_local/backend/tests/integration/agent/test_make_image.py index d9ef05d7ab..07124e3e04 100644 --- a/surfsense_local/backend/tests/integration/agent/test_make_image.py +++ b/surfsense_local/backend/tests/integration/agent/test_make_image.py @@ -20,13 +20,10 @@ from modules.artifacts.models import Artifact, ArtifactFileRole from modules.llm.model_type import ModelType from modules.llm.models import SelectedModel +from modules.llm.pictures_in.model import PictureModel from modules.llm.providers.openai_compatible import NonRetryableImageError from modules.llm.providers.protocols import GeneratedImage -from modules.llm.resolution import ( - ModelResolutionError, - ResolvedImageEdit, - ResolvedImageGeneration, -) +from modules.llm.resolution import ResolvedImageGeneration from modules.source_scope.schemas import SourceScope from modules.workspaces.models import Workspace from shared.config import get_storage_settings @@ -54,13 +51,18 @@ class Painter: def __init__(self) -> None: self.prompts: list[str] = [] + self.shapes: list[str | None] = [] self.edits: list[tuple[str, bytes]] = [] + self.references: list[tuple[bytes, ...]] = [] self.fail_with: Exception | None = None self.hold: threading.Event | None = None self.gives = GeneratedImage(_png(), "image/png") - async def generate(self, _model: str, prompt: str) -> GeneratedImage: + async def generate( + self, _model: str, prompt: str, *, shape: str | None = None + ) -> GeneratedImage: self.prompts.append(prompt) + self.shapes.append(shape) if self.hold is not None: await asyncio.to_thread(self.hold.wait) if self.fail_with is not None: @@ -68,31 +70,39 @@ async def generate(self, _model: str, prompt: str) -> GeneratedImage: return self.gives async def edit( - self, _model: str, prompt: str, image: GeneratedImage + self, + _model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: str | None = None, ) -> GeneratedImage: - self.edits.append((prompt, image.content)) + if len(images) == 1: + self.edits.append((prompt, images[0].content)) + self.references.append(tuple(image.content for image in images)) + self.shapes.append(shape) return GeneratedImage(_png("navy"), "image/png") @pytest.fixture def painter(engine: Engine, monkeypatch: pytest.MonkeyPatch) -> Iterator[Painter]: - """An image and an edit model chosen, and no chat model: a drawing never needs one.""" + """An image model that takes pictures, and no chat model: a drawing never needs one.""" with create_session_factory(engine)() as session: - for model_type in (ModelType.IMAGE_GEN, ModelType.IMAGE_EDIT): - session.add( - SelectedModel(model_type=model_type, provider="sdcpp", name="m") - ) + session.add( + SelectedModel(model_type=ModelType.IMAGE_GEN, provider="sdcpp", name="m") + ) session.commit() fake = Painter() selection = type("Selection", (), {"provider": "fake", "name": "m"})() draws = lambda _session: ResolvedImageGeneration(selection, fake) # noqa: E731 - edits = lambda _session: ResolvedImageEdit(selection, fake) # noqa: E731 monkeypatch.setattr("worker.studio.job.resolve_image_generation", draws) monkeypatch.setattr( "modules.artifacts.drawn_images.model.resolve_image_generation", draws ) - monkeypatch.setattr("worker.studio.job.resolve_image_edit", edits) - monkeypatch.setattr("modules.artifacts.image_edits.model.resolve_image_edit", edits) + monkeypatch.setattr( + "modules.llm.pictures_in.model.picture_model", + lambda _session: PictureModel("image_model", selection, fake), + ) yield fake if fake.hold is not None: fake.hold.set() @@ -118,7 +128,7 @@ async def test_a_picture_is_drawn_from_the_prompt_as_written_with_no_sources( text, images, is_error = await tools.call_content( workspace_id, "make_image", - {"prompt": "A lighthouse at dusk", "title": "Lighthouse", "style": "image"}, + {"prompt": "A lighthouse at dusk", "title": "Lighthouse", "type": "image"}, ) assert is_error is False, text @@ -142,13 +152,13 @@ async def test_a_picture_is_drawn_from_the_prompt_as_written_with_no_sources( async def test_an_infographic_paints_the_agents_facts_in_studios_style( tools: ToolEndpoint, studio_worker: None, painter: Painter ) -> None: - """The style is Studio's own, so both paths draw alike.""" + """An infographic is a type of picture, in Studio's own style, so both paths draw alike.""" workspace_id = await tools.workspace() text, is_error = await tools.call( workspace_id, "make_image", - {"prompt": "Revenue rose 12%", "title": "Q3", "style": "infographic"}, + {"prompt": "Revenue rose 12%", "title": "Q3", "type": "infographic"}, ) assert is_error is False, text @@ -266,7 +276,7 @@ async def test_the_same_title_again_is_refused_so_a_retry_does_not_bill_twice( {"prompt": "x" * 32_001, "title": "Boat"}, "The prompt is 32,001 characters; the limit is 32,000. Shorten it", ), - ({"prompt": "A boat", "title": "Boat", "style": "mural"}, "style must be"), + ({"prompt": "A boat", "title": "Boat", "type": "mural"}, "type must be"), ], ) async def test_a_call_missing_something_says_what_to_fix( @@ -340,6 +350,9 @@ async def test_a_slow_picture_answers_queued_with_the_name_it_will_have( assert text.splitlines()[0] == f"Queued artifact {artifact_id}, version 1: Boat" assert f"image-{artifact_id}" in text assert "Do not make it again" in text + # A document or comic waits for it on the server; a web page cannot. + assert "A document or comic can place it now" in text + assert "A web page can place it only once it is ready" in text async def test_regenerating_paints_the_same_prompt_with_no_chat_model( @@ -363,7 +376,8 @@ async def test_regenerating_paints_the_same_prompt_with_no_chat_model( async def test_the_studio_tool_does_not_offer_a_picture(tools: ToolEndpoint) -> None: - """One way to make a picture: Studio's draft takes sources, and hands nothing back.""" + """One way to make a picture: Studio's draft takes sources, and hands nothing back. + A comic too: the agent draws its pages and looks at each, then binds them.""" workspace_id = await tools.workspace() listed = await tools.request(workspace_id, "tools/list") @@ -375,7 +389,7 @@ async def test_the_studio_tool_does_not_offer_a_picture(tools: ToolEndpoint) -> ) offered = studio["inputSchema"]["properties"]["format"]["enum"] - assert not {"image", "infographic"} & set(offered) + assert not {"image", "infographic", "comic"} & set(offered) assert is_error is True assert text.startswith("Name a format, one of:") @@ -409,31 +423,145 @@ async def test_with_an_edit_model_the_picture_can_be_changed( assert "change it or tell the user" in text -async def test_with_no_edit_model_the_agent_is_not_sent_to_change_the_picture( - tools: ToolEndpoint, - studio_worker: None, - painter: Painter, - monkeypatch: pytest.MonkeyPatch, +@pytest.fixture +def words_only(monkeypatch: pytest.MonkeyPatch) -> None: + """The image model takes no picture in, and no edit model is chosen.""" + monkeypatch.setattr( + "modules.llm.pictures_in.model.picture_model", lambda _session: None + ) + + +async def test_with_a_model_that_takes_no_pictures_a_change_is_drawn_again( + tools: ToolEndpoint, studio_worker: None, painter: Painter, words_only: None ) -> None: - """A call to edit could only be refused, and a redraw would bill another picture.""" + """Every image model can change a picture, and the agent is told how, plainly: + never sent to Settings for a model the user need not have.""" + workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + root = _artifact_id(first) + assert f"call this tool again with artifact_id {root}" in first + assert "drawn again" in first + assert "Settings" not in first - def none_chosen(_session: object) -> None: - raise ModelResolutionError("no image editing model selected") + text, is_error = await tools.call( + workspace_id, "make_image", {"prompt": "Make it night", "artifact_id": root} + ) - monkeypatch.setattr( - "modules.artifacts.image_edits.model.resolve_image_edit", none_chosen + assert is_error is False, text + assert painter.edits == [] + assert painter.prompts[-1] == "A boat\n\nChange: Make it night" + assert "Drawn again from its prompt and the change" in text + assert "Settings" not in text + + +async def test_a_shape_reaches_the_image_model( + tools: ToolEndpoint, studio_worker: None, painter: Painter +) -> None: + """A comic page is portrait.""" + workspace_id = await tools.workspace() + + text, is_error = await tools.call( + workspace_id, + "make_image", + {"prompt": "Page one", "title": "Comic · page 1", "shape": "portrait"}, ) + + assert is_error is False, text + assert painter.shapes == ["portrait"] + assert "Drawn on this computer." in text + + +async def test_references_reach_the_model_that_takes_pictures_in_order( + tools: ToolEndpoint, studio_worker: None, painter: Painter, engine: Engine +) -> None: + """A comic's later pages keep page one's characters.""" workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page one", "title": "Comic · page 1"} + ) + sheet, _ = await tools.call( + workspace_id, "make_image", {"prompt": "The cat", "title": "Character sheet"} + ) + page_one, cat = _artifact_id(first), _artifact_id(sheet) text, is_error = await tools.call( - workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + workspace_id, + "make_image", + { + "prompt": "Page two", + "title": "Comic · page 2", + "shape": "portrait", + "references": [f"image-{page_one}", f"image-{cat}"], + }, ) assert is_error is False, text - assert "image-" in text - assert "artifact_id" not in text - assert "change it" not in text - assert "Settings > Image edit" in text + assert painter.references[-1] == ( + _stored(engine, page_one), + _stored(engine, cat), + ) + assert painter.shapes[-1] == "portrait" + assert painter.prompts == ["Page one", "The cat"] + + +async def test_with_a_model_that_takes_no_pictures_references_are_drawn_from_words( + tools: ToolEndpoint, studio_worker: None, painter: Painter, words_only: None +) -> None: + """Never refused: the page is drawn, and the agent told the faces may differ.""" + workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page one", "title": "Comic · page 1"} + ) + + text, is_error = await tools.call( + workspace_id, + "make_image", + { + "prompt": "Page two", + "title": "Comic · page 2", + "references": [f"image-{_artifact_id(first)}"], + }, + ) + + assert is_error is False, text + assert painter.references == [] + assert painter.prompts == ["Page one", "Page two"] + assert "Drawn from words alone" in text + + +async def test_references_are_refused_unbilled_when_they_cannot_be_kept( + tools: ToolEndpoint, studio_worker: None, painter: Painter +) -> None: + """Another chat's picture, an unknown one, five of them, or with an edit.""" + workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page one", "title": "Comic · page 1"} + ) + name = f"image-{_artifact_id(first)}" + asks = [ + ({"references": [name]}, tools.thread(workspace_id), "no picture"), + ({"references": ["image-999"]}, None, "no picture"), + ({"references": [name, "a", "b", "c", "d"]}, None, "at most 4"), + ( + {"references": [name], "artifact_id": _artifact_id(first)}, + None, + "references are for a new picture", + ), + ] + + for n, (extra, thread, said) in enumerate(asks): + text, is_error = await tools.call( + workspace_id, + "make_image", + {"prompt": "Page two", "title": f"Page {n}", **extra}, + **({"thread": thread} if thread is not None else {}), + ) + assert is_error is True, text + assert said in text + assert painter.prompts == ["Page one"] + assert painter.references == [] # Fails unless what it is handed is really a PNG, as IMAGES_DIR promises. @@ -529,28 +657,6 @@ async def test_a_copy_the_agent_overwrote_is_not_what_gets_placed( assert _pictures_in_word(engine, _artifact_id(text)) == [_png()] -async def test_a_picture_still_being_drawn_is_refused_saying_so( - tools: ToolEndpoint, - studio_worker: None, - painter: Painter, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """The agent tells the user, rather than reading it as a wrong name.""" - monkeypatch.setattr("modules.agent.tool_endpoint.make_image.WAIT_SECONDS", 0.2) - painter.hold = threading.Event() - workspace_id = await tools.workspace() - queued, _ = await tools.call( - workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} - ) - - text, is_error = await tools.call( - workspace_id, "render_document", _placing(f"image-{_artifact_id(queued)}") - ) - - assert is_error is True - assert "still being drawn" in text - - async def test_another_chats_picture_cannot_be_placed( tools: ToolEndpoint, studio_worker: None, painter: Painter ) -> None: diff --git a/surfsense_local/backend/tests/integration/agent/test_tool_endpoint.py b/surfsense_local/backend/tests/integration/agent/test_tool_endpoint.py index d9c2f84da8..fc184a5ec4 100644 --- a/surfsense_local/backend/tests/integration/agent/test_tool_endpoint.py +++ b/surfsense_local/backend/tests/integration/agent/test_tool_endpoint.py @@ -88,6 +88,7 @@ async def test_it_offers_its_tools_with_flat_schemas(tools: ToolEndpoint) -> Non "render_document", "save_page", "make_image", + "bind_comic", "read_document", "list_images", "convert_document", @@ -174,6 +175,7 @@ async def test_a_model_that_reads_images_is_also_offered_source_pages( "render_document", "save_page", "make_image", + "bind_comic", "read_document", "list_images", "source_pages", diff --git a/surfsense_local/backend/tests/integration/agent/test_waiting_render.py b/surfsense_local/backend/tests/integration/agent/test_waiting_render.py new file mode 100644 index 0000000000..08e07ff2c1 --- /dev/null +++ b/surfsense_local/backend/tests/integration/agent/test_waiting_render.py @@ -0,0 +1,180 @@ +"""A document that places a picture still being drawn waits for it, on the +server, and renders once its pictures are drawn: a comic drawn slowly on this +computer still ends as one PDF without the user asking again.""" + +import asyncio +import threading +import time + +import pytest +from sqlalchemy import Engine + +from modules.artifacts.models import Artifact +from modules.documents.models import DocumentStatus +from shared.db import create_session_factory +from tests.integration.agent.test_make_image import ( # noqa: F401 - fixtures + Painter, + _artifact_id, + _pictures_in_word, + _placing, + _png, + painter, +) +from tests.integration.agent.tool_endpoint_client import ToolEndpoint +from tests.integration.worker.conftest import stub_model # noqa: F401 +from worker.waiting_renders import settle_every_waiting_render + +pytestmark = [ + pytest.mark.integration, + pytest.mark.usefixtures("stub_model", "model_reads_images"), +] + + +def _status(engine: Engine, artifact_id: int) -> tuple[DocumentStatus, str | None]: + with create_session_factory(engine)() as session: + document = session.get(Artifact, artifact_id).document + return document.status, document.error_message + + +async def _ended(engine: Engine, artifact_id: int) -> tuple[DocumentStatus, str | None]: + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + status, error = _status(engine, artifact_id) + if status not in (DocumentStatus.PENDING, DocumentStatus.PROCESSING): + return status, error + await asyncio.sleep(0.1) + raise AssertionError(f"artifact {artifact_id} never ended") + + +@pytest.fixture +def slow(monkeypatch: pytest.MonkeyPatch, painter: Painter) -> Painter: # noqa: F811 + """The picture call answers queued, and the drawing waits to be let go.""" + monkeypatch.setattr("modules.agent.tool_endpoint.make_image.WAIT_SECONDS", 0.2) + painter.hold = threading.Event() + return painter + + +async def test_a_render_naming_a_picture_still_drawn_waits_and_then_renders( + tools: ToolEndpoint, studio_worker: None, slow: Painter, engine: Engine +) -> None: + """Answered at once, not refused; made once the picture is drawn.""" + workspace_id = await tools.workspace() + queued, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + name = f"image-{_artifact_id(queued)}" + + began = time.monotonic() + text, is_error = await tools.call(workspace_id, "render_document", _placing(name)) + + assert is_error is False, text + assert time.monotonic() - began < 10 + assert text.startswith("Queued artifact ") + assert f"renders once {name} is drawn" in text + render = _artifact_id(text) + assert _status(engine, render)[0] is DocumentStatus.PENDING + + assert slow.hold is not None + slow.hold.set() + + assert (await _ended(engine, render))[0] is DocumentStatus.READY + assert _pictures_in_word(engine, render) == [_png()] + + +async def test_it_waits_only_for_the_pictures_not_yet_drawn( + tools: ToolEndpoint, + studio_worker: None, + painter: Painter, # noqa: F811 - the fixture + engine: Engine, +) -> None: + """Page one is ready, page two is still drawing: it names only page two.""" + workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page one", "title": "Page 1"} + ) + painter.hold = threading.Event() + with pytest.MonkeyPatch.context() as patch: + patch.setattr("modules.agent.tool_endpoint.make_image.WAIT_SECONDS", 0.2) + second, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page two", "title": "Page 2"} + ) + one, two = f"image-{_artifact_id(first)}", f"image-{_artifact_id(second)}" + placing = {**_placing(one), "images": [one, two]} + + text, is_error = await tools.call(workspace_id, "render_document", placing) + + assert is_error is False, text + assert f"renders once {two} is drawn" in text + assert one not in text.split("renders once", 1)[1] + painter.hold.set() + assert (await _ended(engine, _artifact_id(text)))[0] is DocumentStatus.READY + + +async def test_a_picture_that_fails_fails_the_render_saying_which( + tools: ToolEndpoint, studio_worker: None, slow: Painter, engine: Engine +) -> None: + """The agent can draw that page again, or render without it.""" + workspace_id = await tools.workspace() + queued, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + name = f"image-{_artifact_id(queued)}" + text, _ = await tools.call(workspace_id, "render_document", _placing(name)) + + slow.fail_with = RuntimeError("the provider refused it") + assert slow.hold is not None + slow.hold.set() + + status, error = await _ended(engine, _artifact_id(text)) + assert status is DocumentStatus.FAILED + assert error is not None and f"{name} failed" in error + + +async def test_a_picture_that_ended_while_the_worker_was_down_is_settled_at_start( + tools: ToolEndpoint, studio_worker: None, slow: Painter, engine: Engine +) -> None: + """The job that would have queued it never ran its end.""" + workspace_id = await tools.workspace() + queued, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + picture = _artifact_id(queued) + text, _ = await tools.call( + workspace_id, "render_document", _placing(f"image-{picture}") + ) + render = _artifact_id(text) + # As a quit leaves it: the picture's run failed as interrupted, unannounced. + with create_session_factory(engine)() as session: + document = session.get(Artifact, picture).document + document.status = DocumentStatus.FAILED + document.error_message = "interrupted when the app closed" + session.commit() + + settle_every_waiting_render() + + status, error = _status(engine, render) + assert status is DocumentStatus.FAILED + assert error is not None and f"image-{picture} failed" in error + assert slow.hold is not None + slow.hold.set() + + +async def test_a_picture_cancelled_while_queued_fails_the_render_at_once( + tools: ToolEndpoint, slow: Painter, engine: Engine +) -> None: + """A revoked job never runs, so nothing at its end would settle the render.""" + workspace_id = await tools.workspace() + queued, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + picture = _artifact_id(queued) + text, _ = await tools.call( + workspace_id, "render_document", _placing(f"image-{picture}") + ) + + cancelled = await tools.client.post(f"/artifacts/{picture}/cancel") + + assert cancelled.status_code == 200, cancelled.text + status, error = _status(engine, _artifact_id(text)) + assert status is DocumentStatus.FAILED + assert error is not None and f"image-{picture} failed" in error diff --git a/surfsense_local/backend/tests/integration/artifacts/test_image_edit.py b/surfsense_local/backend/tests/integration/artifacts/test_image_edit.py index afaa3864c9..d600d1e0e8 100644 --- a/surfsense_local/backend/tests/integration/artifacts/test_image_edit.py +++ b/surfsense_local/backend/tests/integration/artifacts/test_image_edit.py @@ -1,5 +1,6 @@ """Editing a Studio image: the instruction and the open version's picture go to -the edit model, and what comes back is the document's next version.""" +the model that takes pictures, or, with none, the picture is drawn again from its +prompt and the change; either way it is the document's next version.""" import asyncio @@ -14,13 +15,10 @@ from modules.documents.models import Document, DocumentStatus from modules.llm.model_type import ModelType from modules.llm.models import SelectedModel +from modules.llm.pictures_in.model import PictureModel from modules.llm.providers.openai_compatible import NonRetryableImageError from modules.llm.providers.protocols import GeneratedImage -from modules.llm.resolution import ( - ResolvedImageEdit, - ResolvedImageGeneration, - resolve_image_edit, -) +from modules.llm.resolution import ResolvedImageGeneration from modules.workspaces.models import Workspace from shared.config import get_storage_settings from shared.queue import studio_queue @@ -37,16 +35,26 @@ class Painter: def __init__(self) -> None: self.edits: list[tuple[str, bytes]] = [] + self.drawn: list[str] = [] self.fail_with: Exception | None = None - async def generate(self, _model: str, _prompt: str) -> GeneratedImage: + async def generate( + self, _model: str, prompt: str, *, shape: str | None = None + ) -> GeneratedImage: + self.drawn.append(prompt) return GeneratedImage(DRAWN, "image/png") async def edit( - self, _model: str, prompt: str, image: GeneratedImage + self, + _model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: str | None = None, ) -> GeneratedImage: if self.fail_with is not None: raise self.fail_with + (image,) = images self.edits.append((prompt, image.content)) number = len(self.edits) return GeneratedImage(b"\x89PNG\r\n\x1a\nedit%d" % number, "image/png") @@ -54,30 +62,39 @@ async def edit( @pytest.fixture def painter(session: Session, monkeypatch: pytest.MonkeyPatch) -> Painter: - """A chat model, an image model and an edit model, all chosen.""" + """A chat model, and an image model that takes pictures.""" session.add( SelectedModel(model_type=ModelType.TEXT_GEN, provider="llamacpp", name="m") ) - for model_type in (ModelType.IMAGE_GEN, ModelType.IMAGE_EDIT): - session.add(SelectedModel(model_type=model_type, provider="sdcpp", name="m")) + session.add( + SelectedModel(model_type=ModelType.IMAGE_GEN, provider="sdcpp", name="m") + ) session.commit() fake = Painter() selection = type("Selection", (), {"provider": "fake", "name": "m"})() monkeypatch.setattr( "worker.studio.shared.generate.run_model", lambda *_a, **_k: WRITER ) + draws = lambda _session: ResolvedImageGeneration(selection, fake) # noqa: E731 + monkeypatch.setattr("worker.studio.job.resolve_image_generation", draws) monkeypatch.setattr( - "worker.studio.job.resolve_image_generation", - lambda _session: ResolvedImageGeneration(selection, fake), + "modules.artifacts.drawn_images.model.resolve_image_generation", draws ) - edit_model = lambda _session: ResolvedImageEdit(selection, fake) # noqa: E731 - monkeypatch.setattr("worker.studio.job.resolve_image_edit", edit_model) monkeypatch.setattr( - "modules.artifacts.image_edits.model.resolve_image_edit", edit_model + "modules.llm.pictures_in.model.picture_model", + lambda _session: PictureModel("image_model", selection, fake), ) return fake +@pytest.fixture +def words_only(monkeypatch: pytest.MonkeyPatch) -> None: + """The image model takes no pictures in, and no edit model is chosen.""" + monkeypatch.setattr( + "modules.llm.pictures_in.model.picture_model", lambda _session: None + ) + + async def _work_off(queue: Huey) -> None: """On a thread, as Studio's worker runs: the image clients start their own loop.""" @@ -272,7 +289,47 @@ async def test_a_failed_edit_is_not_retried_and_leaves_earlier_versions_alone( assert _primary(first) == DRAWN -async def test_editing_with_no_edit_model_says_so_as_a_code( +async def test_with_no_model_that_takes_pictures_an_edit_is_drawn_again( + client: AsyncClient, + session: Session, + workspace: Workspace, + logo_source: Document, + painter: Painter, + words_only: None, +) -> None: + """Edit works for every image model: one that cannot take the picture draws + it again from the prompt it was drawn with and the change, and says so.""" + first = await _drawn(session, workspace, logo_source) + + second = await _edited(client, session, first.id, "Make it night") + + assert painter.edits == [] + assert painter.drawn[-1] == "a lighthouse at dusk\n\nChange: Make it night" + assert second.document.content.startswith("Drawn again with your change.") + assert second.artifact_metadata["redrawn"] is True + assert second.artifact_metadata["version"]["number"] == 2 + + +async def test_changes_add_up_when_a_redrawn_picture_is_changed_again( + client: AsyncClient, + session: Session, + workspace: Workspace, + logo_source: Document, + painter: Painter, + words_only: None, +) -> None: + """Each redraw starts from the prompt its base was drawn with.""" + first = await _drawn(session, workspace, logo_source) + second = await _edited(client, session, first.id, "Make it night") + + await _edited(client, session, second.id, "Add stars") + + assert painter.drawn[-1] == ( + "a lighthouse at dusk\n\nChange: Make it night\n\nChange: Add stars" + ) + + +async def test_an_edit_keeps_the_prompt_a_later_redraw_starts_from( client: AsyncClient, session: Session, workspace: Workspace, @@ -280,20 +337,19 @@ async def test_editing_with_no_edit_model_says_so_as_a_code( painter: Painter, monkeypatch: pytest.MonkeyPatch, ) -> None: - """The box shows the interface's own sentence, as Refine does for a chat model.""" + """Edited by a model that takes pictures, then redrawn after a switch to one + that does not: the redraw still knows what was changed.""" first = await _drawn(session, workspace, logo_source) + second = await _edited(client, session, first.id, "Make it night") monkeypatch.setattr( - "modules.artifacts.image_edits.model.resolve_image_edit", resolve_image_edit + "modules.llm.pictures_in.model.picture_model", lambda _session: None ) - session.delete(session.get(SelectedModel, ModelType.IMAGE_EDIT)) - response = await _edit(client, session, first.id, "Make it night") + await _edited(client, session, second.id, "Add stars") - assert response.status_code == 409 - assert response.json()["detail"] == { - "message": "Needs an image editing model", - "code": "needs_image_edit", - } + assert painter.drawn[-1] == ( + "a lighthouse at dusk\n\nChange: Make it night\n\nChange: Add stars" + ) async def test_only_a_ready_image_or_infographic_can_be_edited( diff --git a/surfsense_local/backend/tests/integration/artifacts/test_routes.py b/surfsense_local/backend/tests/integration/artifacts/test_routes.py index 612e9e614b..3cf7db3fb0 100644 --- a/surfsense_local/backend/tests/integration/artifacts/test_routes.py +++ b/surfsense_local/backend/tests/integration/artifacts/test_routes.py @@ -32,7 +32,11 @@ def choose_model(engine: Engine) -> None: """A generation model must be selected before a job can run.""" with create_session_factory(engine)() as session: session.add( - SelectedModel(model_type=ModelType.TEXT_GEN, provider="llamacpp", name="Qwen3-4B-Q4_K_M") + SelectedModel( + model_type=ModelType.TEXT_GEN, + provider="llamacpp", + name="Qwen3-4B-Q4_K_M", + ) ) session.commit() @@ -218,7 +222,9 @@ async def test_a_server_that_lists_no_voices_takes_the_ones_added_for_it( assert "Settings" in refused.json()["detail"] for voice in ("alloy", "echo"): - added = await client.post("/llm/selection/audio_gen/voices", json={"voice": voice}) + added = await client.post( + "/llm/selection/audio_gen/voices", json={"voice": voice} + ) assert added.status_code == 201, added.text opened = (await client.get(brief_url)).json() @@ -671,3 +677,51 @@ async def test_a_ready_artifact_cannot_be_cancelled( response = await client.post(f"/artifacts/{artifact_id}/cancel") assert response.status_code == 409 + + +async def test_a_comic_job_checks_and_stores_its_kind_and_page_count( + client: AsyncClient, engine: Engine, workspace_id: int, choose_model: None +) -> None: + """A kind Studio has no table for, or a thirteenth page, is refused at the door.""" + with create_session_factory(engine)() as session: + connection = ProviderConnection( + label="Images", provider="openai_compatible", base_url="http://127.0.0.1:9" + ) + session.add(connection) + session.flush() + session.add( + SelectedModel( + model_type=ModelType.IMAGE_GEN, + provider="openai_compatible", + connection_id=connection.id, + name="flux", + ) + ) + session.commit() + source_id = make_ready_source(engine, workspace_id) + url = f"/workspaces/{workspace_id}/studio/jobs" + + for options, words in ( + ({"kind": "opera", "pages": 4}, "kind"), + ({"kind": "manga", "pages": 13}, "pages"), + ): + refused = await client.post( + url, + json={"format": "comic", "document_ids": [source_id], "options": options}, + ) + assert refused.status_code == 422, refused.text + assert words in refused.json()["detail"] + + created = await client.post( + url, + json={ + "format": "comic", + "document_ids": [source_id], + "options": {"kind": "manga", "pages": 6}, + }, + ) + assert created.status_code == 201, created.text + with create_session_factory(engine)() as session: + stored = session.get(Artifact, created.json()["id"]) + assert stored.format == "comic" + assert stored.artifact_metadata["options"] == {"kind": "manga", "pages": 6} diff --git a/surfsense_local/backend/tests/integration/llm/test_picture_model.py b/surfsense_local/backend/tests/integration/llm/test_picture_model.py new file mode 100644 index 0000000000..1b815ffe1b --- /dev/null +++ b/surfsense_local/backend/tests/integration/llm/test_picture_model.py @@ -0,0 +1,170 @@ +"""Which model takes a picture in: the edit model if one is chosen, else the image +model when it can, else none, and then an edit is drawn again from words.""" + +from pathlib import Path + +import pytest +from httpx import AsyncClient +from sqlalchemy import Engine + +from modules.llm.pictures_in.model import picture_model +from modules.llm.providers import sdcpp +from shared.db import create_session_factory +from tests.integration.llm import conftest +from tests.integration.llm.test_sdcpp_slots import KLEIN, SD15, put_on_disk + +pytestmark = [pytest.mark.integration, pytest.mark.asyncio] + +TAKES_PICTURES = "openai/gpt-image-2.6-flare" +WORDS_ONLY = "recraft/recraft-v4.1-flash" + + +@pytest.fixture +def image_models(monkeypatch: pytest.MonkeyPatch) -> None: + """As OpenRouter lists them: what each model takes, and what it returns.""" + monkeypatch.setattr( + conftest, + "REMOTE_MODELS", + [ + *conftest.REMOTE_MODELS, + { + "id": TAKES_PICTURES, + "architecture": { + "input_modalities": ["text", "image"], + "output_modalities": ["image"], + }, + }, + { + "id": WORDS_ONLY, + "architecture": { + "input_modalities": ["text"], + "output_modalities": ["image"], + }, + }, + ], + ) + + +async def _connect(client: AsyncClient, base_url: str) -> int: + reply = await client.post( + "/llm/connections", + json={ + "label": "Gateway", + "provider": "openai_compatible", + "base_url": base_url, + "api_key": "secret", + }, + ) + assert reply.status_code == 201, reply.text + return reply.json()["id"] + + +async def _choose( + client: AsyncClient, slot: str, name: str, connection_id: int | None +) -> None: + provider = "openai_compatible" if connection_id is not None else sdcpp.PROVIDER + reply = await client.put( + f"/llm/selection/{slot}", + json={"provider": provider, "connection_id": connection_id, "name": name}, + ) + assert reply.status_code == 200, reply.text + + +def _picture_model(engine: Engine): + with create_session_factory(engine)() as session: + return picture_model(session) + + +async def test_an_image_model_that_takes_pictures_edits_with_no_edit_model( + client: AsyncClient, openai_server: str, image_models: None, engine: Engine +) -> None: + """Most remote image models take a picture in; none needs a second choice.""" + connection = await _connect(client, openai_server) + await _choose(client, "image_gen", TAKES_PICTURES, connection) + + chosen = _picture_model(engine) + + assert chosen is not None + assert chosen.source == "image_model" + assert chosen.selection.name == TAKES_PICTURES + + +async def test_an_image_model_that_takes_only_words_has_no_picture_model( + client: AsyncClient, openai_server: str, image_models: None, engine: Engine +) -> None: + """Its edits are drawn again from words instead.""" + connection = await _connect(client, openai_server) + await _choose(client, "image_gen", WORDS_ONLY, connection) + + assert _picture_model(engine) is None + + +async def test_a_chosen_edit_model_is_used_before_the_image_model( + client: AsyncClient, openai_server: str, image_models: None, engine: Engine +) -> None: + """The override, for the user who wants another model for edits.""" + connection = await _connect(client, openai_server) + await _choose(client, "image_gen", TAKES_PICTURES, connection) + await _choose(client, "image_edit", TAKES_PICTURES, connection) + await _choose(client, "image_gen", WORDS_ONLY, connection) + + chosen = _picture_model(engine) + + assert chosen is not None + assert chosen.source == "edit_model" + assert chosen.selection.name == TAKES_PICTURES + + +async def test_another_image_model_does_not_inherit_taking_pictures( + client: AsyncClient, openai_server: str, image_models: None, engine: Engine +) -> None: + """What a model takes is its own, not the slot's.""" + connection = await _connect(client, openai_server) + await _choose(client, "image_gen", TAKES_PICTURES, connection) + await _choose(client, "image_gen", WORDS_ONLY, connection) + + assert _picture_model(engine) is None + + +async def test_a_local_image_model_that_edits_takes_pictures( + client: AsyncClient, images_dir: Path, engine: Engine +) -> None: + """FLUX.2 klein's entry names editing; the same files draw and edit.""" + put_on_disk(images_dir, "flux2-klein-4b") + await _choose(client, "image_gen", KLEIN, None) + + chosen = _picture_model(engine) + + assert chosen is not None + assert chosen.source == "image_model" + assert chosen.selection.name == KLEIN + + +async def test_a_local_image_model_that_only_draws_has_no_picture_model( + client: AsyncClient, images_dir: Path, engine: Engine +) -> None: + """Its entry names drawing only.""" + put_on_disk(images_dir, "stable-diffusion-1.5") + await _choose(client, "image_gen", SD15, None) + + assert _picture_model(engine) is None + + +async def test_the_app_is_told_which_model_takes_a_picture( + client: AsyncClient, openai_server: str, image_models: None +) -> None: + """Studio's Edit says whether a change is an edit or a redraw.""" + connection = await _connect(client, openai_server) + + nothing = (await client.get("/llm/pictures-in")).json() + await _choose(client, "image_gen", WORDS_ONLY, connection) + words = (await client.get("/llm/pictures-in")).json() + await _choose(client, "image_gen", TAKES_PICTURES, connection) + pictures = (await client.get("/llm/pictures-in")).json() + await _choose(client, "image_edit", TAKES_PICTURES, connection) + override = (await client.get("/llm/pictures-in")).json() + + assert nothing == {"source": None} + assert words == {"source": None} + assert pictures == {"source": "image_model"} + assert override == {"source": "edit_model"} diff --git a/surfsense_local/backend/tests/integration/worker/test_studio.py b/surfsense_local/backend/tests/integration/worker/test_studio.py index bd06cdf2ba..53c01f2d66 100644 --- a/surfsense_local/backend/tests/integration/worker/test_studio.py +++ b/surfsense_local/backend/tests/integration/worker/test_studio.py @@ -1,16 +1,19 @@ import contextlib +import io import json import time from collections.abc import Iterator import httpx import pytest +from PIL import Image from sqlalchemy import Engine, text from sqlalchemy.orm import Session from modules.artifacts.models import Artifact, ArtifactFileRole from modules.documents.models import Document, DocumentStatus, DocumentType from modules.llm.catalog.local.manifest import load_local_manifest +from modules.llm.pictures_in.model import PictureModel from modules.llm.profile import Fingerprint, Tier from modules.llm.providers.audiocpp.memory import NotEnoughMemoryError from modules.llm.providers.audiocpp.speech import AudioCppSpeech, VoicedModel @@ -119,9 +122,11 @@ def _capture_image(monkeypatch: pytest.MonkeyPatch) -> list[str]: seen: list[str] = [] class FakeImageGenerator: - async def generate(self, _model: str, content: str) -> GeneratedImage: + async def generate( + self, _model: str, content: str, *, shape: str | None = None + ) -> GeneratedImage: seen.append(content) - return GeneratedImage(b"\x89PNG\r\n\x1a\n" + b"\x00" * 8, "image/png") + return GeneratedImage(_tiny_png(), "image/png") selection = type("Selection", (), {"provider": "fake", "name": "flux"})() monkeypatch.setattr( @@ -131,6 +136,13 @@ async def generate(self, _model: str, content: str) -> GeneratedImage: return seen +def _tiny_png() -> bytes: + """A real picture: a comic's binder opens every page it places.""" + out = io.BytesIO() + Image.new("RGB", (8, 12), "teal").save(out, "PNG") + return out.getvalue() + + def _primary_bytes(artifact: Artifact) -> bytes: return ( get_storage_settings().data_dir / artifact.files[0].storage_key @@ -686,6 +698,65 @@ def api_keeps_writing() -> None: assert all(a.document.status is DocumentStatus.READY for a in artifacts) +def test_the_api_can_write_while_the_job_reads_the_models_window( + session: Session, engine: Engine, stub_model: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """The window is read over HTTP from the API, which writes in the same file.""" + import sqlite3 + + api_could_write: list[bool] = [] + + class Generator: + async def context_tokens(self, _model: str) -> int: + api = sqlite3.connect(str(engine.url.database), timeout=0.1) + try: + api.execute("BEGIN IMMEDIATE") + api.rollback() + api_could_write.append(True) + except sqlite3.OperationalError: + api_could_write.append(False) + finally: + api.close() + return 32_000 + + selection = type( + "Selection", + (), + {"provider": "fake", "name": "fake", "connection": None, "tier": Tier.COMPACT}, + )() + + def resolve(job_session: Session) -> ResolvedGeneration: + # The real lookup reads the selection, which opens the job's transaction. + job_session.execute(text("SELECT 1")) + return ResolvedGeneration(selection, Generator()) + + monkeypatch.setattr("worker.studio.job.resolve_routed_generation", resolve) + _capture_model(monkeypatch, SUMMARY) + artifact = make_artifact(session) + + run(artifact.id) + + assert api_could_write == [True] + + +def test_a_settle_that_fails_after_the_job_does_not_ask_for_a_retry( + session: Session, stub_model: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """A raise here would make Huey run the job, and bill its picture, again.""" + + def locked(*_args: object, **_kwargs: object) -> list: + raise RuntimeError("database is locked") + + monkeypatch.setattr("worker.waiting_renders.settle_waiting_renders", locked) + _capture_model(monkeypatch, SUMMARY) + artifact = make_artifact(session) + + run(artifact.id) + + session.expire_all() + assert artifact.document.status is DocumentStatus.READY + + def test_a_generation_failure_leaves_a_reason( session: Session, stub_model: None, monkeypatch: pytest.MonkeyPatch ) -> None: @@ -725,6 +796,95 @@ def fail(*_args: object, **_kwargs: object) -> Built: assert "HTTP 500" in (artifact.document.error_message or "") +def test_a_comic_job_binds_its_drawn_pages_into_one_pdf( + session: Session, stub_model: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """The chat model plans it, the image model draws page 1, the model that takes + pictures draws the rest from it, and the PDF is the comic's one file.""" + plan = { + "title": "Cassini's Last Dive", + "style": "Bold ink.", + "characters": "CASSINI, a gold-foil probe with a white dish.", + "pages": [ + { + "panels": [ + { + "where": "top", + "shows": "Cassini reaches Saturn.", + "lines": ["Caption: 2004."], + } + ] + }, + { + "panels": [ + { + "where": "top", + "shows": "Cassini dives.", + "lines": ["Caption: 2017."], + } + ] + }, + ], + } + asked = _capture_model(monkeypatch, json.dumps(plan)) + drawn = _capture_image(monkeypatch) + from_page_one: list[bytes] = [] + + class Keeper: + async def edit( + self, _model: str, _prompt: str, images, *, shape=None + ) -> GeneratedImage: + from_page_one.extend(image.content for image in images) + return GeneratedImage(_tiny_png(), "image/png") + + keeper_selection = type("Selection", (), {"provider": "fake", "name": "flux"})() + monkeypatch.setattr( + "worker.studio.job.pictures_in.picture_model", + lambda _session: PictureModel("image_model", keeper_selection, Keeper()), + ) + artifact = make_artifact( + session, + fmt="comic", + prompt="the finale", + options={"kind": "western", "pages": 2}, + ) + + run(artifact.id) + + session.expire_all() + assert artifact.document.status is DocumentStatus.READY, ( + artifact.document.error_message + ) + assert "the finale" in asked[0] + assert len(drawn) == 1 + assert len(from_page_one) == 1 + assert artifact.document.title == "Cassini's Last Dive" + assert "Cassini dives." in (artifact.document.content or "") + _one_file(artifact, "application/pdf", b"%PDF") + assert artifact.artifact_metadata["comic"]["pages"] == 2 + + +def test_a_failed_comic_is_never_retried_since_every_page_is_billed_again( + session: Session, stub_model: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """A comic's failure ends the task: a retry would draw every page again.""" + + def fail(*_args: object, **_kwargs: object) -> Built: + raise RuntimeError("the image model hung up on page 3") + + monkeypatch.setattr("worker.studio.media.visual.comic.pipeline.render", fail) + monkeypatch.setattr("worker.studio.job.pictures_in.picture_model", lambda _s: None) + artifact = make_artifact( + session, fmt="comic", options={"kind": "western", "pages": 4} + ) + + run(artifact.id) # returns: a raise here would ask Huey for a retry + + session.expire_all() + assert artifact.document.status is DocumentStatus.FAILED + assert "page 3" in (artifact.document.error_message or "") + + def test_persist_stores_a_primary_blob(session: Session, stub_model: None) -> None: """A file format's bytes land on disk with a row that resolves back to them.""" artifact = make_artifact(session) diff --git a/surfsense_local/backend/tests/unit/agent/test_opencode_config.py b/surfsense_local/backend/tests/unit/agent/test_opencode_config.py index dffedbfc11..9697bb9d28 100644 --- a/surfsense_local/backend/tests/unit/agent/test_opencode_config.py +++ b/surfsense_local/backend/tests/unit/agent/test_opencode_config.py @@ -210,6 +210,7 @@ def test_only_surfsenses_own_skills_load_from_the_shipped_folder( "surfsense-data", "surfsense-pdf", "surfsense-html", + "surfsense-pictures", ) for name in names: assert (Path(skills) / name / "SKILL.md").is_file() diff --git a/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py new file mode 100644 index 0000000000..ca3ccc0982 --- /dev/null +++ b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py @@ -0,0 +1,86 @@ +"""The pictures skill as the agent reads it.""" + +import re +from pathlib import Path + +import pytest + +from modules.agent.engines.claude_code.skills_plugin import build_skills_plugin +from modules.agent.opencode_config import PICTURES_SKILL, skills_folder +from modules.agent.tool_endpoint.make_image import LISTING + +pytestmark = pytest.mark.unit + + +def _file(name: str) -> str: + return (skills_folder() / PICTURES_SKILL / name).read_text(encoding="utf-8") + + +def test_the_skill_names_itself_as_opencode_finds_it() -> None: + """opencode lists a skill by its frontmatter name, and the config allows only that one.""" + frontmatter = _file("SKILL.md").split("---")[1] + + assert f"name: {PICTURES_SKILL}\n" in frontmatter + assert "comic" in frontmatter + + +def test_the_tool_sends_a_set_of_pictures_to_the_skill() -> None: + """The prompt is full, so the tool's description names the skill too.""" + assert f"load the {PICTURES_SKILL} skill first" in LISTING["description"] + + +def test_the_skill_never_sends_the_user_to_settings() -> None: + """A model that takes no picture in is said plainly, never set up mid-comic.""" + for name in ("SKILL.md", "comics.md"): + # The app writes the path with a chevron; the agent's prompts with ">". + assert "Settings \u203a" not in _file(name), name + assert "Settings >" not in _file(name), name + + +def test_a_claude_plan_thread_gets_the_skill_with_its_own_tool_names( + tmp_path: Path, +) -> None: + """Claude Code reads the same skill, its SurfSense tools named as it lists them.""" + plugin = build_skills_plugin(skills_folder(), tmp_path / "plugin") + + assert f"surfsense:{PICTURES_SKILL}" in plugin.names + for name in ("SKILL.md", "comics.md"): + text = (plugin.skills / PICTURES_SKILL / name).read_text(encoding="utf-8") + assert not re.search( + r"surfsense_(make_image|bind_comic|render_document|read_document)", text + ) + skill = (plugin.skills / PICTURES_SKILL / "SKILL.md").read_text(encoding="utf-8") + assert "mcp__surfsense__make_image" in skill + + +def test_comics_cover_every_kind_the_user_may_ask_for() -> None: + """Each reads and is laid out its own way; a manga is not a Western comic.""" + comics = _file("comics.md").lower() + + for kind in ( + "manga", + "manhwa", + "webtoon", + "manhua", + "comic strip", + "yonkoma", + "graphic novel", + "bande dessinée", + "picture book", + "storyboard", + ): + assert kind in comics, kind + + +def test_a_comic_is_bound_by_its_own_tool_not_a_script() -> None: + """The binder is the app's; the agent no longer copies a script to fill.""" + assert not (skills_folder() / PICTURES_SKILL / "pdf.md").exists() + for name in ("SKILL.md", "comics.md"): + text = _file(name) + assert "surfsense_bind_comic" in text, name + assert "surfsense_render_document" not in text, name + + +def test_a_comics_pages_are_drawn_as_its_pages() -> None: + """Marked from the first, a page is never listed in Studio apart from its comic.""" + assert "comic_page: true" in _file("SKILL.md") diff --git a/surfsense_local/backend/tests/unit/llm/prompting/test_prompts_ship.py b/surfsense_local/backend/tests/unit/llm/prompting/test_prompts_ship.py index a40f635543..019762347e 100644 --- a/surfsense_local/backend/tests/unit/llm/prompting/test_prompts_ship.py +++ b/surfsense_local/backend/tests/unit/llm/prompting/test_prompts_ship.py @@ -4,6 +4,7 @@ import pytest +from modules.artifacts.comic.kinds import KINDS from modules.artifacts.podcast.brief import Duration, PodcastBrief, Speaker, Style from modules.llm.profile import Tier from modules.llm.prompting import load @@ -13,6 +14,7 @@ from worker.studio.content.summary import pipeline as summary from worker.studio.media.audio.podcast import draft, outline from worker.studio.media.audio.podcast.outline import Segment +from worker.studio.media.visual.comic import plan as comic from worker.studio.media.visual.image import pipeline as image from worker.studio.media.visual.infographic import pipeline as infographic from worker.studio.office import prompt as office @@ -69,6 +71,8 @@ def _every_prompt(tier: Tier) -> dict[str, str]: "html": html.prompt(tier, FOCUS), "image": image.prompt(tier, FOCUS), "infographic": infographic.prompt(tier, FOCUS), + "comic": comic.prompt(tier, FOCUS, KINDS["western"], 4), + "picture book": comic.prompt(tier, FOCUS, KINDS["picture_book"], 4), "outline": outline.prompt(tier, BRIEF, FOCUS), "draft": draft.prompt(tier, BRIEF, SEGMENT, 1, 2, "Sam: Welcome."), } | {spec.key: office.build(tier, spec, FOCUS) for spec in (docx, pptx, xlsx, pdf)} diff --git a/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_editor.py b/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_editor.py index 6afcddc6dd..8a1317d097 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_editor.py +++ b/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_editor.py @@ -95,7 +95,7 @@ async def test_it_sends_the_picture_as_a_reference_at_its_own_size() -> None: """The launch's size would crop or stretch a picture of another shape.""" server = SdServer(_job("queued"), _job("generating"), _done()) - image = await _editor(server).edit("klein", "make it night", BASE) + image = await _editor(server).edit("klein", "make it night", (BASE,)) assert image == GeneratedImage(EDITED, "image/png") [body] = server.posted @@ -131,7 +131,7 @@ def watch(request: httpx.Request) -> httpx.Response: transport=httpx.MockTransport(watch), ) - await edit.edit("klein", "make it night", BASE) + await edit.edit("klein", "make it night", (BASE,)) assert seen and all(seen) assert runtime.held is False @@ -144,7 +144,7 @@ async def test_a_failed_job_fails_the_edit_with_its_reason() -> None: ) with pytest.raises(NonRetryableImageError, match="out of memory"): - await _editor(server).edit("klein", "make it night", BASE) + await _editor(server).edit("klein", "make it night", (BASE,)) @pytest.mark.parametrize("status", [404, 410]) @@ -153,7 +153,7 @@ async def test_a_job_the_server_no_longer_has_fails_the_edit(status: int) -> Non server = SdServer(httpx.Response(status, json={"error": "gone"})) with pytest.raises(NonRetryableImageError, match=str(status)): - await _editor(server).edit("klein", "make it night", BASE) + await _editor(server).edit("klein", "make it night", (BASE,)) async def test_a_full_queue_fails_the_edit() -> None: @@ -161,7 +161,7 @@ async def test_a_full_queue_fails_the_edit() -> None: server = SdServer(_done(), submit=httpx.Response(429, json={"error": "full"})) with pytest.raises(NonRetryableImageError, match="429"): - await _editor(server).edit("klein", "make it night", BASE) + await _editor(server).edit("klein", "make it night", (BASE,)) async def test_a_cancel_withdraws_the_queued_job() -> None: @@ -175,7 +175,7 @@ def cancelled() -> None: raise CancelledError with cancellation.watching(cancelled), pytest.raises(CancelledError): - await _editor(server).edit("klein", "make it night", BASE) + await _editor(server).edit("klein", "make it night", (BASE,)) assert server.asked[-1] == "POST /sdcpp/v1/jobs/job_1/cancel" @@ -203,8 +203,24 @@ def cancelled() -> None: transport=httpx.MockTransport(watch), ) with cancellation.watching(cancelled), pytest.raises(CancelledError): - await edit.edit("klein", "make it night", BASE) + await edit.edit("klein", "make it night", (BASE,)) withdraw = seen[-1] assert withdraw.url.path == "/sdcpp/v1/jobs/job_1/cancel" assert withdraw.extensions["timeout"]["read"] <= 2 + + +async def test_every_reference_goes_in_at_the_shape_asked_for() -> None: + """A comic page keeps its characters from page one, drawn portrait at about + the reference's pixel count, each side a multiple of 64.""" + server = SdServer(_done()) + page_one = GeneratedImage(_png(1024, 1024), "image/png") + + await _editor(server).edit("klein", "page two", (page_one, BASE), shape="portrait") + + [body] = server.posted + assert body["ref_images"] == [ + "data:image/png;base64," + base64.b64encode(page_one.content).decode(), + "data:image/png;base64," + base64.b64encode(BASE.content).decode(), + ] + assert (body["width"], body["height"]) == (832, 1280) diff --git a/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_generator.py b/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_generator.py new file mode 100644 index 0000000000..8f945d8876 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_generator.py @@ -0,0 +1,70 @@ +"""A local drawing: today's route as it was, and the native job route when a +shape is asked, since only that one takes a width and height per request.""" + +import httpx +import pytest + +from modules.llm.providers.protocols import GeneratedImage +from modules.llm.providers.sdcpp.generator import LocalImageGenerator +from tests.unit.llm.providers.sdcpp.test_editor import ( + EDITED, + SERVED, + Runtime, + SdServer, + _done, +) + +pytestmark = [pytest.mark.unit, pytest.mark.asyncio] + + +class Inner: + """Today's client, reaching sd-server's OpenAI route.""" + + def __init__(self) -> None: + self.asked: list[str] = [] + + async def generate(self, model: str, prompt: str) -> GeneratedImage: + self.asked.append(prompt) + return GeneratedImage(b"square", "image/png") + + +def _generator(server: SdServer, inner: Inner) -> LocalImageGenerator: + return LocalImageGenerator( + inner, + "http://127.0.0.1:1", + SERVED, + Runtime().given_up, + pixels=1024 * 1024, + interval=0, + transport=httpx.MockTransport(server), + ) + + +async def test_a_drawing_with_no_shape_keeps_its_route() -> None: + """Nothing changes for a picture that asks for no shape.""" + server, inner = SdServer(_done()), Inner() + + image = await _generator(server, inner).generate("klein", "a lighthouse") + + assert image.content == b"square" + assert inner.asked == ["a lighthouse"] + assert server.posted == [] + + +async def test_a_shape_is_drawn_through_the_native_route_at_its_size() -> None: + """At the launch's pixel count, in the shape's proportions.""" + server, inner = SdServer(_done()), Inner() + + image = await _generator(server, inner).generate( + "klein", "a comic page", shape="portrait" + ) + + assert image == GeneratedImage(EDITED, "image/png") + assert inner.asked == [] + [body] = server.posted + assert body == { + "prompt": "a comic page", + "width": 832, + "height": 1280, + "output_format": "png", + } diff --git a/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py b/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py index 52210bd08f..4b7130b501 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py +++ b/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py @@ -1,13 +1,15 @@ """A remote edit model: the picture and the instruction go to `/images/edits`.""" import base64 +import json import httpx import pytest from modules.llm.providers.openai_compatible.image_edit import ( OpenAICompatibleImageEditor, - _asks_for_picture, + _refused, + _routes, ) from modules.llm.providers.openai_compatible.image_response import ( NonRetryableImageError, @@ -41,7 +43,7 @@ async def test_it_posts_the_picture_and_the_instruction_as_a_form() -> None: 200, json={"data": [{"b64_json": base64.b64encode(EDITED).decode()}]} ) - image = await _editor(reply, asked).edit("gpt-image-1", "make it night", BASE) + image = await _editor(reply, asked).edit("gpt-image-1", "make it night", (BASE,)) assert image == GeneratedImage(EDITED, "image/png") [request] = asked @@ -65,20 +67,22 @@ async def test_a_refused_edit_is_not_retried() -> None: asked: list[httpx.Request] = [] with pytest.raises(NonRetryableImageError, match="500"): - await _editor(httpx.Response(500, json={}), asked).edit("m", "x", BASE) + await _editor(httpx.Response(500, json={}), asked).edit("m", "x", (BASE,)) assert len(asked) == 1 async def test_an_answer_with_no_picture_fails_the_edit() -> None: """Read as a generation's answer is.""" with pytest.raises(NonRetryableImageError, match="no data list"): - await _editor(httpx.Response(200, json={"error": "?"}), []).edit("m", "x", BASE) + await _editor(httpx.Response(200, json={"error": "?"}), []).edit( + "m", "x", (BASE,) + ) async def test_a_model_that_refuses_the_setting_is_asked_again_without_it() -> None: """GPT image models answer with the picture and refuse the setting; a 400 comes before any work, so asking again bills nothing twice.""" - _asks_for_picture.clear() + _refused.clear() asked: list[bytes] = [] done = httpx.Response( 200, json={"data": [{"b64_json": base64.b64encode(EDITED).decode()}]} @@ -92,7 +96,151 @@ def answer(request: httpx.Request) -> httpx.Response: editor = OpenAICompatibleImageEditor( "https://example.test/v1", transport=httpx.MockTransport(answer) ) - image = await editor.edit("gpt-image-1", "make it night", BASE) + image = await editor.edit("gpt-image-1", "make it night", (BASE,)) assert image.content == EDITED assert [b"response_format" in form for form in asked] == [True, False] + + +PAGE_ONE = GeneratedImage(b"\x89PNG\r\n\x1a\npage-one", "image/png") +DONE = {"data": [{"b64_json": base64.b64encode(EDITED).decode()}]} + + +async def test_several_pictures_go_as_one_part_each() -> None: + """A comic page keeps its characters from the pages before it.""" + asked: list[httpx.Request] = [] + editor = _editor(httpx.Response(200, json=DONE), asked) + + await editor.edit("gpt-image-1", "page two", (PAGE_ONE, BASE)) + + form = asked[0].read() + assert form.count(b'name="image[]"') == 2 + assert form.index(PAGE_ONE.content) < form.index(BASE.content) + + +async def test_a_shape_is_asked_for_as_a_size() -> None: + """As OpenAI's edit route takes it.""" + asked: list[httpx.Request] = [] + editor = _editor(httpx.Response(200, json=DONE), asked) + + await editor.edit("gpt-image-1", "page two", (PAGE_ONE,), shape="portrait") + + assert b'name="size"\r\n\r\n1024x1536' in asked[0].read() + + +async def test_a_server_with_no_edit_route_is_sent_references_instead() -> None: + """OpenRouter answers 404 on /images/edits, before any work, and takes the + pictures as input_references on /images; the route is kept per server.""" + _routes.clear() + asked: list[httpx.Request] = [] + + def answer(request: httpx.Request) -> httpx.Response: + asked.append(request) + if request.url.path.endswith("/images/edits"): + return httpx.Response(404, json={}) + return httpx.Response(200, json=DONE) + + editor = OpenAICompatibleImageEditor( + "https://gateway.test/v1", "sk-test", transport=httpx.MockTransport(answer) + ) + image = await editor.edit( + "openai/gpt-image-2.6", "page two", (PAGE_ONE,), shape="portrait" + ) + await editor.edit("openai/gpt-image-2.6", "page three", (PAGE_ONE,)) + + assert image.content == EDITED + assert [r.url.path for r in asked] == [ + "/v1/images/edits", + "/v1/images", + "/v1/images", + ] + sent = json.loads(asked[1].read()) + assert sent["model"] == "openai/gpt-image-2.6" + assert sent["prompt"] == "page two" + assert sent["aspect_ratio"] == "2:3" + assert sent["input_references"] == [ + { + "type": "image_url", + "image_url": { + "url": "data:image/png;base64," + + base64.b64encode(PAGE_ONE.content).decode() + }, + } + ] + assert "aspect_ratio" not in json.loads(asked[2].read()) + + +async def test_a_refusal_on_references_is_not_resent_unchanged() -> None: + """/images is sent no response_format, so dropping it changes nothing to resend.""" + _routes.clear() + _refused.clear() + asked: list[httpx.Request] = [] + + def answer(request: httpx.Request) -> httpx.Response: + asked.append(request) + if request.url.path.endswith("/images/edits"): + return httpx.Response(404, json={}) + return httpx.Response(400, json={}) + + editor = OpenAICompatibleImageEditor( + "https://gateway.test/v1", "sk-test", transport=httpx.MockTransport(answer) + ) + with pytest.raises(NonRetryableImageError, match="400"): + await editor.edit("m", "page two", (PAGE_ONE,), shape="portrait") + + bodies = [r.read() for r in asked[1:]] + assert len(bodies) == len(set(bodies)) == 2 + + +async def test_a_server_error_on_the_edit_route_is_not_tried_elsewhere() -> None: + """A 5xx may follow billed work: no second route, no second picture.""" + _routes.clear() + asked: list[httpx.Request] = [] + + with pytest.raises(NonRetryableImageError, match="502"): + await _editor(httpx.Response(502, json={}), asked).edit("m", "x", (BASE,)) + assert len(asked) == 1 + + +class _CutOff(httpx.AsyncByteStream): + """A body the server stops sending partway, as OpenRouter did after a 404.""" + + async def __aiter__(self): + yield b'{"error": {"message": "Not Fou' + raise httpx.RemoteProtocolError( + "peer closed connection without sending complete message body " + "(incomplete chunked read)" + ) + + +async def test_a_missing_edit_route_cut_off_mid_reply_still_sends_references() -> None: + """Its 404 says there is no such route before any work; the body adds nothing, + so a server that hangs up while sending it is no reason to fail the page.""" + _routes.clear() + asked: list[httpx.Request] = [] + + def answer(request: httpx.Request) -> httpx.Response: + asked.append(request) + if request.url.path.endswith("/images/edits"): + return httpx.Response(404, stream=_CutOff()) + return httpx.Response(200, json=DONE) + + editor = OpenAICompatibleImageEditor( + "https://cutoff.test/v1", "sk-test", transport=httpx.MockTransport(answer) + ) + image = await editor.edit("google/gemini-2.5-flash-image", "page two", (PAGE_ONE,)) + + assert image.content == EDITED + assert [r.url.path for r in asked] == ["/v1/images/edits", "/v1/images"] + + +async def test_a_picture_cut_off_mid_reply_still_fails() -> None: + """A success must arrive whole: half a picture is no picture.""" + _routes.clear() + asked: list[httpx.Request] = [] + + with pytest.raises(NonRetryableImageError, match="incomplete chunked read"): + await _editor(httpx.Response(200, stream=_CutOff()), asked).edit( + "m", "x", (BASE,) + ) + assert len(asked) == 1 diff --git a/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py b/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py index 070eba4744..bafc17418c 100644 --- a/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py +++ b/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py @@ -1,5 +1,7 @@ import base64 +import json +import httpx import pytest from modules.llm.connections.service import normalize_base_url, parse_models @@ -131,6 +133,49 @@ def test_an_endpoints_own_word_for_speech_is_the_audio_slot() -> None: assert models["openai/whisper-1"].capability_known is True +def test_an_image_model_that_declares_pictures_in_can_edit() -> None: + """OpenRouter lists what a model takes as well as what it returns; a picture in + and a picture out is what an edit, or a comic page's reference, needs.""" + models = { + model.name: model + for model in parse_models( + { + "data": [ + { + "id": "openai/gpt-image-2.6-flare", + "architecture": { + "input_modalities": ["text", "image"], + "output_modalities": ["image"], + }, + }, + { + "id": "recraft/recraft-v4.1-flash", + "architecture": { + "input_modalities": ["text"], + "output_modalities": ["image"], + }, + }, + { + "id": "openai/gpt-4o", + "architecture": { + "input_modalities": ["text", "image"], + "output_modalities": ["text"], + }, + }, + ] + } + ) + } + + assert models["openai/gpt-image-2.6-flare"].types == ( + ModelType.IMAGE_GEN, + ModelType.IMAGE_EDIT, + ) + assert models["recraft/recraft-v4.1-flash"].types == (ModelType.IMAGE_GEN,) + # Reading pictures is not editing them: a chat model stays a chat model. + assert models["openai/gpt-4o"].types == (ModelType.TEXT_GEN,) + + def test_delta_reads_openai_sse_and_ignores_done() -> None: """Chat streaming retains the existing OpenAI delta behavior.""" assert _delta('data: {"choices":[{"delta":{"content":"hi"}}]}') == Delta("hi") @@ -247,3 +292,113 @@ async def download(url: str) -> GeneratedImage: assert fetched == [ "https://oaidalleapiprodscus.blob.core.windows.net/private/x.png" ] + + +async def test_a_shape_is_asked_for_as_a_size_and_dropped_where_refused( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A comic page is portrait. A server that refuses the size draws its own, + and a 400 is a refusal before any work, so asking again bills nothing.""" + _route_cache.clear() + _asks_for_picture.clear() + provider = _provider(6) + sizes: list[str | None] = [] + + async def post(_route, _model, _prompt, _picture, size=None): + sizes.append(size) + return (400, {}) if size else (200, PAYLOAD) + + monkeypatch.setattr(provider, "_post", post) + await provider.generate("flux", "page one", shape="portrait") + await provider.generate("flux", "page two", shape="portrait") + + # Without the size, after the picture setting was tried without first. + assert sizes == ["1024x1536", "1024x1536", None, None] + + +async def test_a_shape_reaches_a_server_that_takes_it( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Only asked for when a picture names a shape.""" + _route_cache.clear() + _asks_for_picture.clear() + provider = _provider(7) + sizes: list[str | None] = [] + + async def post(_route, _model, _prompt, _picture, size=None): + sizes.append(size) + return 200, PAYLOAD + + monkeypatch.setattr(provider, "_post", post) + await provider.generate("gpt-image-1", "a strip", shape="landscape") + await provider.generate("gpt-image-1", "a picture") + + assert sizes == ["1536x1024", None] + + +class _CutOff(httpx.AsyncByteStream): + """A body the server stops sending partway, as OpenRouter did after a 404.""" + + async def __aiter__(self): + yield b'{"error": {"message": "Not Fou' + raise httpx.RemoteProtocolError( + "peer closed connection without sending complete message body " + "(incomplete chunked read)" + ) + + +async def test_a_missing_route_cut_off_mid_reply_still_falls_back( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The 404 is the answer; a server that hangs up while sending its body has + still said the route is missing, so the picture goes to the other route.""" + _route_cache.clear() + asked: list[str] = [] + + def answer(request: httpx.Request) -> httpx.Response: + asked.append(request.url.path) + if request.url.path.endswith("/images/generations"): + return httpx.Response(404, stream=_CutOff()) + return httpx.Response(200, json=PAYLOAD) + + real_client = httpx.AsyncClient + + def client(**kwargs: object) -> httpx.AsyncClient: + return real_client(transport=httpx.MockTransport(answer), **kwargs) + + monkeypatch.setattr( + "modules.llm.providers.openai_compatible.image.httpx.AsyncClient", client + ) + image = await _provider(5).generate("image", "draw") + + assert image.content == PNG + assert asked == ["/v1/images/generations", "/v1/images"] + + +async def test_a_shape_on_the_images_route_is_an_aspect_ratio( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """OpenRouter's /images takes `aspect_ratio`, not OpenAI's `size`.""" + _route_cache.clear() + _asks_for_picture.clear() + sent: dict[str, dict] = {} + + def answer(request: httpx.Request) -> httpx.Response: + sent[request.url.path] = json.loads(request.read()) + if request.url.path.endswith("/images/generations"): + return httpx.Response(404, json={}) + return httpx.Response(200, json=PAYLOAD) + + real_client = httpx.AsyncClient + + def client(**kwargs: object) -> httpx.AsyncClient: + return real_client(transport=httpx.MockTransport(answer), **kwargs) + + monkeypatch.setattr( + "modules.llm.providers.openai_compatible.image.httpx.AsyncClient", client + ) + await _provider(8).generate("flux", "page one", shape="portrait") + + assert sent["/v1/images/generations"]["size"] == "1024x1536" + assert sent["/v1/images"]["aspect_ratio"] == "2:3" + assert "size" not in sent["/v1/images"] diff --git a/surfsense_local/backend/tests/unit/worker/test_comic_binding.py b/surfsense_local/backend/tests/unit/worker/test_comic_binding.py new file mode 100644 index 0000000000..30447ccb34 --- /dev/null +++ b/surfsense_local/backend/tests/unit/worker/test_comic_binding.py @@ -0,0 +1,150 @@ +"""Binding a comic's pictures into one PDF: by page or as one scroll, in its reading direction.""" + +import io + +import pytest +from PIL import Image +from pypdf import PdfReader + +from worker.studio.media.visual.comic_binding.bind import ( + MARGIN, + WIDTH, + BoundPage, + _wrapped, + bind, +) + +pytestmark = pytest.mark.unit + +POINTS_WIDE = 6.625 * 72 + + +def _png(width: int, height: int, colour: str = "teal") -> bytes: + out = io.BytesIO() + Image.new("RGB", (width, height), colour).save(out, "PNG") + return out.getvalue() + + +def _sizes(pdf: bytes) -> list[tuple[float, float]]: + return [ + (float(page.mediabox.width), float(page.mediabox.height)) + for page in PdfReader(io.BytesIO(pdf)).pages + ] + + +def test_each_picture_is_a_page_at_comic_book_width_as_tall_as_its_picture() -> None: + """No picture is cropped or letterboxed to fit a paper size.""" + pdf = bind( + "The Lighthouse Cat", + [BoundPage(_png(400, 600)), BoundPage(_png(400, 600, "navy"))], + layout="pages", + direction="ltr", + ) + + assert _sizes(pdf) == [(POINTS_WIDE, POINTS_WIDE * 1.5)] * 2 + assert PdfReader(io.BytesIO(pdf)).metadata.title == "The Lighthouse Cat" + + +def test_a_pages_lines_are_set_under_it_exactly() -> None: + """For a misspelt page, or one drawn without words, the strip carries its text.""" + pdf = bind( + "Cat", + [BoundPage(_png(400, 400), ("NORA: The bulb is broken!",))], + layout="pages", + direction="ltr", + ) + + ((width, height),) = _sizes(pdf) + assert height > width + assert ( + "NORA: The bulb is broken!" + in PdfReader(io.BytesIO(pdf)).pages[0].extract_text() + ) + + +def test_a_webtoon_scrolls_down_one_long_page() -> None: + """Its beats stack in one column, read by scrolling, not page by page.""" + pdf = bind( + "Beats", + [BoundPage(_png(400, 600)) for _ in range(3)], + layout="scroll", + direction="ltr", + ) + + ((_, height),) = _sizes(pdf) + assert height > 3 * POINTS_WIDE * 1.5 + + +def test_a_long_scroll_starts_a_new_page_every_twelve() -> None: + """No viewer meets one page tall enough to fail.""" + pdf = bind( + "Beats", + [BoundPage(_png(40, 60)) for _ in range(13)], + layout="scroll", + direction="ltr", + ) + + assert len(_sizes(pdf)) == 2 + + +def test_a_manga_tells_its_reader_it_reads_right_to_left() -> None: + """A reader then lays its spreads out as a manga's are.""" + pdf = bind("Manga", [BoundPage(_png(400, 600))], layout="pages", direction="rtl") + + preferences = PdfReader(io.BytesIO(pdf)).trailer["/Root"]["/ViewerPreferences"] + assert preferences["/Direction"] == "/R2L" + + +def test_pages_are_placed_as_jpeg_so_a_book_stays_small() -> None: + """Four PNG pages made 10 to 28 MB in the spike.""" + noisy = Image.effect_noise((800, 1200), 64).convert("RGB") + out = io.BytesIO() + noisy.save(out, "PNG") + + pdf = bind("Noise", [BoundPage(out.getvalue())], layout="pages", direction="ltr") + + assert len(pdf) < len(out.getvalue()) + + +def _fonts(pdf: bytes) -> set[str]: + page = PdfReader(io.BytesIO(pdf)).pages[0] + return {str(font["/BaseFont"]) for font in page["/Resources"]["/Font"].values()} + + +def test_a_mangas_japanese_lines_are_set_in_a_font_that_has_them() -> None: + """Helvetica draws a box for every kana and kanji.""" + pdf = bind( + "猫", + [BoundPage(_png(400, 400), ("ノラ: 電球が壊れた!",))], + layout="pages", + direction="rtl", + ) + + assert "/HeiseiKakuGo-W5" in _fonts(pdf) + + +def test_a_line_with_no_spaces_wraps_inside_the_page() -> None: + """Japanese and Chinese have no spaces to break at; the strip grows instead.""" + short = bind( + "猫", [BoundPage(_png(400, 400), ("猫",))], layout="pages", direction="rtl" + ) + long = bind( + "猫", + [BoundPage(_png(400, 400), ("猫" * 200,))], + layout="pages", + direction="rtl", + ) + + assert _sizes(long)[0][1] > _sizes(short)[0][1] + + +def test_a_word_too_wide_is_cut_with_no_spaces_added() -> None: + """Letters of a cut word join as written; a real space between words stays.""" + line = "電球" * 60 + " 猫" + + wrapped = _wrapped(line, WIDTH - 2 * MARGIN) + + assert len(wrapped) > 1 + # No space is added inside the cut word; the break takes the real one's place. + assert " " not in "".join(wrapped[:-1]) + assert "".join(wrapped).replace(" ", "") == line.replace(" ", "") diff --git a/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py b/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py new file mode 100644 index 0000000000..869d233a3e --- /dev/null +++ b/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py @@ -0,0 +1,377 @@ +"""A Studio comic: the chat model plans it, the image model draws its pages, the binder makes the PDF.""" + +import io +import json + +import pytest +from PIL import Image +from pypdf import PdfReader + +from modules.artifacts.comic.options import validate_options +from modules.llm.pictures_in.model import PictureModel +from modules.llm.profile import Tier +from modules.llm.providers.protocols import GeneratedImage +from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration +from worker.studio.media.visual.comic import pipeline as comic +from worker.studio.media.visual.drawn.pipeline import DRAWN_FROM_WORDS +from worker.studio.shared.artifact import Source + +pytestmark = pytest.mark.unit + + +def _png(colour: str) -> bytes: + out = io.BytesIO() + Image.new("RGB", (40, 60), colour).save(out, "PNG") + return out.getvalue() + + +PLAN = { + "title": "The Lighthouse Cat", + "style": "Bold ink lines, flat colours.", + "characters": "MILO, a small orange tabby cat with a red scarf.", + "pages": [ + { + "panels": [ + {"where": "top", "shows": "A storm hits the lighthouse.", "lines": []}, + { + "where": "bottom", + "shows": "MILO looks at the dead lamp.", + "lines": ["MILO: The light is out!"], + }, + ] + }, + { + "panels": [ + { + "where": "whole page", + "shows": "MILO finds a bulb.", + "lines": ["MILO: Got it!"], + } + ] + }, + {"panels": [{"where": "whole page", "shows": "The lamp shines.", "lines": []}]}, + ], +} + + +class Artist: + """The image model, and the model that takes page 1 as reference.""" + + def __init__(self) -> None: + self.drawn: list[str] = [] + self.with_reference: list[tuple[str, bytes]] = [] + self.shapes: list[str | None] = [] + + async def generate( + self, _model: str, prompt: str, *, shape: str | None = None + ) -> GeneratedImage: + self.drawn.append(prompt) + self.shapes.append(shape) + return GeneratedImage(_png("teal"), "image/png") + + async def edit( + self, + _model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: str | None = None, + ) -> GeneratedImage: + (reference,) = images + self.with_reference.append((prompt, reference.content)) + self.shapes.append(shape) + return GeneratedImage(_png("navy"), "image/png") + + +def _writer() -> ResolvedGeneration: + selection = type("Selection", (), {"name": "qwen3:8b", "tier": Tier.CAPABLE})() + return ResolvedGeneration(selection, None) + + +def _painter( + artist: Artist, provider: str = "openai_compatible" +) -> ResolvedImageGeneration: + selection = type("Selection", (), {"name": "flux", "provider": provider})() + return ResolvedImageGeneration(selection, artist) + + +def _keeper(artist: Artist, provider: str = "openai_compatible") -> PictureModel: + selection = type("Selection", (), {"name": "flux", "provider": provider})() + return PictureModel("image_model", selection, artist) + + +@pytest.fixture +def asked(monkeypatch: pytest.MonkeyPatch) -> list[str]: + """The chat model answers with PLAN; what it was asked is recorded.""" + seen: list[str] = [] + + def writer(_model: object, system: str, _sources: object, **_asked: object) -> str: + seen.append(system) + return json.dumps(PLAN) + + monkeypatch.setattr("worker.studio.shared.generate.run_model", writer) + return seen + + +def _text(pdf: bytes) -> str: + return "\n".join(page.extract_text() for page in PdfReader(io.BytesIO(pdf)).pages) + + +def test_page_one_is_drawn_alone_and_every_later_page_keeps_its_characters( + asked: list[str], +) -> None: + """Page 1 sets the cast; every later page is drawn from it, lettered in its bubbles.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + "make it funny", + {"kind": "western", "pages": 3}, + keeper=_keeper(artist), + ) + + assert "make it funny" in asked[0] + assert "Western comic" in asked[0] + (first,) = artist.drawn + assert "Page 1 of 3" in first + assert "MILO, a small orange tabby cat" in first + assert 'MILO says: "The light is out!"' in first + assert [prompt.split("Page ")[1][:6] for prompt, _ in artist.with_reference] == [ + "2 of 3", + "3 of 3", + ] + assert all(reference == _png("teal") for _, reference in artist.with_reference) + assert all( + "Keep the characters looking exactly as in the reference" in p + for p, _ in artist.with_reference + ) + assert set(artist.shapes) == {"portrait"} + assert built.title == "The Lighthouse Cat" + assert built.primary_mime == "application/pdf" + assert built.primary_filename == "the-lighthouse-cat.pdf" + assert len(PdfReader(io.BytesIO(built.primary)).pages) == 3 + # Drawn in the bubbles, so not set again under the page. + assert "The light is out!" not in _text(built.primary) + assert "The light is out!" in built.markdown + + +def test_a_local_model_draws_no_words_and_every_line_goes_in_the_strip( + asked: list[str], +) -> None: + """Its lettering is not measured, and nobody looks at the page to check it.""" + artist = Artist() + + built = comic.render( + _painter(artist, provider="sdcpp"), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 3}, + keeper=_keeper(artist), + ) + + prompts = artist.drawn + [prompt for prompt, _ in artist.with_reference] + assert all("no text, no lettering, no speech bubbles" in p for p in prompts) + assert not any("The light is out!" in p for p in prompts) + assert "MILO: The light is out!" in _text(built.primary) + + +def test_later_pages_drawn_on_this_computer_leave_every_line_to_the_strip( + asked: list[str], +) -> None: + """A local edit model draws pages 2 on; one comic is lettered one way throughout.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 3}, + keeper=_keeper(artist, provider="sdcpp"), + ) + + prompts = artist.drawn + [prompt for prompt, _ in artist.with_reference] + assert all("no text, no lettering, no speech bubbles" in p for p in prompts) + assert "MILO: The light is out!" in _text(built.primary) + + +def test_a_manga_reads_right_to_left_in_its_pages_and_its_book( + asked: list[str], +) -> None: + """Each page's prompt and the bound book both read right to left.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "manga", "pages": 3}, + keeper=_keeper(artist), + ) + + assert "read right to left" in artist.drawn[0] + root = PdfReader(io.BytesIO(built.primary)).trailer["/Root"] + assert root["/ViewerPreferences"]["/Direction"] == "/R2L" + + +def test_with_no_model_that_takes_pictures_every_page_is_drawn_from_the_words( + asked: list[str], +) -> None: + """Nothing is refused; the comic records that its cast may drift.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 3}, + keeper=None, + ) + + assert len(artist.drawn) == 3 + assert artist.with_reference == [] + assert built.metadata is not None and built.metadata[DRAWN_FROM_WORDS] is True + + +def test_a_plan_longer_than_asked_is_cut_to_the_page_count(asked: list[str]) -> None: + """Each page is billed or takes minutes: the count the user chose is the most drawn.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 2}, + keeper=_keeper(artist), + ) + + assert len(artist.drawn) + len(artist.with_reference) == 2 + assert len(PdfReader(io.BytesIO(built.primary)).pages) == 2 + + +REFUSAL = ( + "I can't turn this document into a comic.\n\n" + "It glorifies real violence, and a dramatised version would spread it." +) + + +def _replying(monkeypatch: pytest.MonkeyPatch, *replies: str) -> list[dict]: + """The chat model answers `replies` in turn; each call's keywords are recorded.""" + calls: list[dict] = [] + + def writer(_model: object, _system: str, _sources: object, **asked: object) -> str: + calls.append(asked) + return replies[len(calls) - 1] + + monkeypatch.setattr("worker.studio.shared.generate.run_model", writer) + return calls + + +def _render_three(artist: Artist) -> object: + return comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 3}, + keeper=_keeper(artist), + ) + + +def test_the_plan_is_asked_for_in_its_json_shape( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """An endpoint that takes a schema cannot answer outside it.""" + calls = _replying(monkeypatch, json.dumps(PLAN)) + + _render_three(Artist()) + + schema = calls[0]["json_schema"] + assert schema["required"] == ["title", "style", "characters", "pages"] + + +def test_a_plan_that_is_not_json_is_asked_for_once_more( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Nothing is drawn yet, so asking again costs one text reply, not pages.""" + calls = _replying( + monkeypatch, "Here is the plan: title The Lighthouse Cat", json.dumps(PLAN) + ) + artist = Artist() + + built = _render_three(artist) + + assert calls[1]["repair"].reply == "Here is the plan: title The Lighthouse Cat" + assert built.title == "The Lighthouse Cat" + + +def test_a_plan_short_of_the_pages_asked_for_is_asked_for_again( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Four pages asked for are four drawn, not a one-page comic marked ready.""" + four = {**PLAN, "pages": [*PLAN["pages"], PLAN["pages"][0]]} + calls = _replying(monkeypatch, json.dumps(PLAN), json.dumps(four)) + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 4}, + keeper=_keeper(artist), + ) + + assert "3 of the 4 pages" in calls[1]["repair"].instruction + assert len(PdfReader(io.BytesIO(built.primary)).pages) == 4 + + +def test_a_model_that_declines_twice_fails_with_what_it_said( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A refusal reads as one, not as a JSON parse error, and no page is drawn.""" + _replying(monkeypatch, REFUSAL, REFUSAL) + artist = Artist() + + with pytest.raises(ValueError) as failed: + _render_three(artist) + + assert str(failed.value).splitlines()[0] == ( + "The model did not plan the comic. It said: " + "I can't turn this document into a comic." + ) + assert artist.drawn == [] + + +def test_a_webtoon_is_one_scroll_of_beats() -> None: + """A webtoon's count is its beats, up to the same twelve.""" + picked = validate_options(None, {"kind": "webtoon", "pages": 8}) # type: ignore[arg-type] + assert picked == {"kind": "webtoon", "pages": 8} + + +def test_options_default_to_a_four_page_western_comic() -> None: + """What the dialog sends when the user changes nothing.""" + assert validate_options(None, None) == {"kind": "western", "pages": 4} # type: ignore[arg-type] + + +@pytest.mark.parametrize( + "raw, words", + [ + ({"kind": "opera", "pages": 4}, "kind"), + ({"kind": "western", "pages": 0}, "pages"), + ({"kind": "western", "pages": 13}, "pages"), + ({"kind": "western", "pages": "four"}, "pages"), + ], +) +def test_options_outside_the_kinds_or_twelve_pages_are_refused( + raw: dict, words: str +) -> None: + """Refused at the door, naming the option to fix, before any picture is billed.""" + with pytest.raises(ValueError, match=words): + validate_options(None, raw) # type: ignore[arg-type] diff --git a/surfsense_local/backend/tests/unit/worker/test_consumer.py b/surfsense_local/backend/tests/unit/worker/test_consumer.py index 1dca29a804..778082986d 100644 --- a/surfsense_local/backend/tests/unit/worker/test_consumer.py +++ b/surfsense_local/backend/tests/unit/worker/test_consumer.py @@ -35,6 +35,7 @@ def run(self) -> None: # Startup steps that read the database, which a unit test has none of. monkeypatch.setattr(consumer, "wait_for_schema", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_documents", lambda _kinds: None) + monkeypatch.setattr(consumer, "settle_every_waiting_render", lambda: None) consumer.consume("ingest") consumer.consume("studio") @@ -64,6 +65,7 @@ def run(self) -> None: monkeypatch.setattr(consumer, "Consumer", FakeConsumer) monkeypatch.setattr(consumer, "wait_for_schema", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_documents", lambda _kinds: None) + monkeypatch.setattr(consumer, "settle_every_waiting_render", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_runs", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_pulls", lambda: None) monkeypatch.setattr(consumer, "remove_leftover_run_folders", lambda: None) @@ -102,6 +104,7 @@ def run(self) -> None: monkeypatch.setattr(consumer, "Consumer", FakeConsumer) monkeypatch.setattr(consumer, "wait_for_schema", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_documents", lambda _kinds: None) + monkeypatch.setattr(consumer, "settle_every_waiting_render", lambda: None) left = data_dir / "tmp" / "document-scripts" / "0f1e2d3c" (left / "images").mkdir(parents=True) (left / "images" / "7-1.png").write_bytes(b"a figure") diff --git a/surfsense_local/backend/tests/unit/worker/test_drawing_prompt.py b/surfsense_local/backend/tests/unit/worker/test_drawing_prompt.py new file mode 100644 index 0000000000..5ccf218462 --- /dev/null +++ b/surfsense_local/backend/tests/unit/worker/test_drawing_prompt.py @@ -0,0 +1,28 @@ +"""What a redraw starts from, for pictures made before their prompt was kept.""" + +import pytest + +from modules.artifacts.models import Artifact +from modules.documents.models import Document +from worker.studio.media.visual.infographic.style import in_infographic_style +from worker.studio.media.visual.redraw.drawing_prompt import drawing_prompt_of + +pytestmark = pytest.mark.unit + + +def _older(fmt: str) -> Artifact: + """No `drawing_prompt` and no recipe: only the body Studio wrote under its title.""" + document = Document(title="Saturn", content="# Saturn\n\nRings: 7 main rings") + return Artifact(format=fmt, artifact_metadata={}, document=document) + + +def test_an_older_infographic_is_redrawn_in_studios_style() -> None: + """Its body is the brief alone; the style it was drawn in is not in it.""" + assert drawing_prompt_of(_older("infographic")) == in_infographic_style( + "Rings: 7 main rings" + ) + + +def test_an_older_image_is_redrawn_from_its_body_as_written() -> None: + """An image's body is the prompt it was drawn from.""" + assert drawing_prompt_of(_older("image")) == "Rings: 7 main rings" diff --git a/surfsense_local/backend/tests/unit/worker/test_studio_job_router.py b/surfsense_local/backend/tests/unit/worker/test_studio_job_router.py index 7a36e6e4e9..23d4eb7d8a 100644 --- a/surfsense_local/backend/tests/unit/worker/test_studio_job_router.py +++ b/surfsense_local/backend/tests/unit/worker/test_studio_job_router.py @@ -23,8 +23,14 @@ def test_every_kind_has_a_pipeline() -> None: def test_every_pipeline_takes_what_its_format_declares() -> None: """The harness passes requires_model_types positionally, then sources and prompt, - then the checked options for the formats that take them.""" + then the checked options for the formats that take them, then `keeper` for a + format that draws from a reference picture.""" for fmt in FORMATS: render = job_router.pipeline_for(job_router.Kind(fmt.key)) - expected = len(fmt.requires_model_types) + 2 + (fmt.validate_options is not None) + expected = ( + len(fmt.requires_model_types) + + 2 + + (fmt.validate_options is not None) + + fmt.draws_with_references + ) assert len(inspect.signature(render).parameters) == expected, fmt.key diff --git a/surfsense_local/backend/worker/consumer.py b/surfsense_local/backend/worker/consumer.py index 850168748f..361708056f 100644 --- a/surfsense_local/backend/worker/consumer.py +++ b/surfsense_local/backend/worker/consumer.py @@ -12,6 +12,7 @@ from worker.interrupted_documents import fail_interrupted_documents from worker.interrupted_pulls import fail_interrupted_pulls from worker.wait_for_schema import wait_for_schema +from worker.waiting_renders import settle_every_waiting_render # Studio jobs wait on a model: overlap them. Ingest jobs saturate the CPU: one. # ponytail: laptop defaults; becomes a setting for power users. @@ -46,6 +47,8 @@ def consume(name: str) -> None: # No script runs yet, so every run folder is one a killed worker left. remove_leftover_run_folders() fail_interrupted_documents({DocumentType.ARTIFACT}) + # After those: a document waiting on an interrupted picture fails too. + settle_every_waiting_render() logging.getLogger(__name__).info( "%s: worker consuming with %s threads", name, workers ) diff --git a/surfsense_local/backend/worker/studio/job.py b/surfsense_local/backend/worker/studio/job.py index 149b25523f..22ae2ac4fe 100644 --- a/surfsense_local/backend/worker/studio/job.py +++ b/surfsense_local/backend/worker/studio/job.py @@ -7,6 +7,7 @@ from sqlalchemy.exc import OperationalError from sqlalchemy.orm import Session +from modules.artifacts.comic.binding_recipe import ComicBinding, comic_binding_of from modules.artifacts.converted_documents.conversion import ( Conversion, conversion_of, @@ -17,7 +18,7 @@ from modules.artifacts.image_edits.editable import EDITABLE_FORMATS from modules.artifacts.image_edits.recipe import ImageEdit, image_edit_of from modules.artifacts.made_file import made_by -from modules.artifacts.models import Artifact +from modules.artifacts.models import Artifact, ArtifactFileRole from modules.artifacts.revised_copies.revision import revision_of from modules.artifacts.script_documents.script_error import PREFIX, script_error from modules.artifacts.script_documents.spec import ( @@ -38,6 +39,7 @@ from modules.jobs.runs.record import job_run_of from modules.llm.model_route.client import resolve_routed_generation from modules.llm.model_type import ModelType +from modules.llm.pictures_in import model as pictures_in from modules.llm.providers.audiocpp.memory import NotEnoughMemoryError from modules.llm.providers.context_overflow import ContextOverflowError from modules.llm.providers.openai_compatible import NonRetryableImageError @@ -47,7 +49,6 @@ ModelResolutionError, ResolvedGeneration, ResolvedImageGeneration, - resolve_image_edit, resolve_image_generation, resolve_text_to_speech, ) @@ -60,9 +61,13 @@ from worker.studio.converted_document import pipeline as converted_document from worker.studio.job_run import pipeline as job_run from worker.studio.made_file import pipeline as made_file +from worker.studio.media.visual.comic_binding import bound_pages from worker.studio.media.visual.drawn import pipeline as drawn_image +from worker.studio.media.visual.drawn.references import reference_pictures from worker.studio.media.visual.edit import pipeline as image_edit from worker.studio.media.visual.edit.base import base_image +from worker.studio.media.visual.redraw import pipeline as image_redraw +from worker.studio.media.visual.redraw.drawing_prompt import drawing_prompt_of from worker.studio.office.document import figure_shelf from worker.studio.office.document.refine import refine from worker.studio.office.docx import docx @@ -74,6 +79,7 @@ from worker.studio.shared import gather, persist, window_budget from worker.studio.shared.artifact import Built, Source from worker.studio.web.html.written import page as written_page +from worker.waiting_renders import settle_after_job logger = logging.getLogger(__name__) @@ -105,6 +111,8 @@ def run(artifact_id: int) -> None: _generate(session, artifact) finally: engine.dispose() + # However it ended: a document waiting on this picture may now go. + settle_after_job(artifact_id) def _generate(session: Session, artifact: Artifact) -> None: @@ -120,6 +128,7 @@ def _generate(session: Session, artifact: Artifact) -> None: refining = refinement(meta) editing = image_edit_of(meta) drawing = image_drawing_of(meta) + binding = comic_binding_of(meta) script = document_script(meta) if renders_as_stored(meta) else None converting = conversion_of(meta) revised: revised_copy.Made | None = None @@ -134,6 +143,8 @@ def _generate(session: Session, artifact: Artifact) -> None: built = _edit_image(session, artifact, document, editing) elif drawing is not None: built = _draw_image(session, artifact, document, drawing) + elif binding is not None: + built = _bind_comic(session, document, binding) elif made_by(meta) is not None: built = made_file.built(artifact, document.title) elif script is not None: @@ -225,6 +236,9 @@ def _generate(session: Session, artifact: Artifact) -> None: # The agent's tool reads this FAILED; a retry could bill the picture twice. if image_drawing_of(artifact.artifact_metadata) is not None: return + # Every page of a comic would be drawn, and billed, again. + if artifact.format == job_router.Kind.COMIC: + return raise # Huey retries; a later success clears the message. @@ -240,6 +254,8 @@ def _draft(session: Session, artifact: Artifact, document: Document) -> Built: writer = cast( ResolvedGeneration, models[fmt.requires_model_types.index(ModelType.TEXT_GEN)] ) + # The window is asked of the API, which needs the write lock to answer. + session.commit() window = _window(writer) # The prompt is what to search for, where the format reads passages. query = prompt if fmt.grounding is Grounding.PASSAGES else None @@ -273,6 +289,9 @@ def gathered(budget_chars: int | None) -> list[Source]: ) # Options were checked at job creation; only formats that take them get them. extras = [meta.get("options")] if fmt.validate_options else [] + references = ( + {"keeper": _picture_model(session)} if fmt.draws_with_references else {} + ) # Generation runs for minutes; the write lock must not be held across it. session.commit() raise_if_cancelled(session, document) @@ -280,7 +299,9 @@ def gathered(budget_chars: int | None) -> list[Source]: # A cancel hangs up on a model mid-reply; other stages still finish first. with cancellation.watching(lambda: _check_cancelled(session, document)): try: - return job_router.pipeline_for(kind)(*models, sources, prompt, *extras) + return job_router.pipeline_for(kind)( + *models, sources, prompt, *extras, **references + ) except ContextOverflowError as refused: raise window_budget.SourcesDoNotFitError( window_budget.DOES_NOT_FIT @@ -363,32 +384,80 @@ def _refine( def _edit_image( session: Session, artifact: Artifact, document: Document, editing: ImageEdit ) -> Built: - """One call to the edit model, from the picture of the version it names.""" + """One call: the picture to the model that takes pictures, or, with none, + the image model drawing it again from the words it was drawn with.""" try: - editor = resolve_image_edit(session) + editor = pictures_in.picture_model(session) except ModelResolutionError as error: raise NoModelSelectedError(str(error)) from error + painter = _choose_model(session, ModelType.IMAGE_GEN) if editor is None else None base = base_image(session, editing) session.commit() raise_if_cancelled(session, document) - # The local editor polls; each poll looks for a cancel. + # The local clients poll; each poll looks for a cancel. with cancellation.watching(lambda: _check_cancelled(session, document)): - return image_edit.render(editor, base, editing.instruction) + if editor is not None: + return image_edit.render(editor, base, editing.instruction) + assert painter is not None + return image_redraw.render(painter, base, editing.instruction) def _draw_image( session: Session, artifact: Artifact, document: Document, drawing: ImageDrawing ) -> Built: - """One call to the image model, with the prompt the agent wrote.""" + """One call to the image model, with the prompt the agent wrote; with + reference pictures, to the model that takes them, when one does.""" painter = _choose_model(session, ModelType.IMAGE_GEN) + keeper = None + if drawing.references: + try: + keeper = pictures_in.picture_model(session) + except ModelResolutionError as error: + raise NoModelSelectedError(str(error)) from error + references = reference_pictures(session, drawing.references) title, format = document.title, artifact.format session.commit() raise_if_cancelled(session, document) - # The local generator polls; each poll looks for a cancel. + # The local clients poll; each poll looks for a cancel. with cancellation.watching(lambda: _check_cancelled(session, document)): - return drawn_image.render(painter, drawing.prompt, title, format) + return drawn_image.render( + painter, + drawing.prompt, + title, + format, + shape=drawing.shape, + references=references, + keeper=keeper, + ) + + +def _bind_comic(session: Session, document: Document, binding: ComicBinding) -> Built: + """The agent's pages, as drawn when it bound them; no model is asked.""" + pages = [_kept_page(session, page.picture) for page in binding.pages] + title = document.title + session.commit() + raise_if_cancelled(session, document) + return bound_pages.render(title, binding, pages) + + +def _kept_page(session: Session, picture_id: int) -> bound_pages.KeptPage: + """Its file to bind, and the words it was drawn from, for the comic's search.""" + picture = session.get(Artifact, picture_id) + primary = ( + None + if picture is None + else next( + (f for f in picture.files if f.role is ArtifactFileRole.PRIMARY), None + ) + ) + if picture is None or primary is None: + raise ValueError(f"image-{picture_id} is gone, so its page cannot be bound") + return bound_pages.KeptPage( + get_storage_settings().data_dir / primary.storage_key, + drawing_prompt_of(picture), + ) def _keep_spec(artifact: Artifact, spec: dict) -> None: @@ -465,6 +534,13 @@ def _window(model: ResolvedGeneration) -> int | None: return None +def _picture_model(session: Session) -> pictures_in.PictureModel | None: + try: + return pictures_in.picture_model(session) + except ModelResolutionError as error: + raise NoModelSelectedError(str(error)) from error + + def _choose_model( session: Session, model_type: ModelType ) -> ResolvedGeneration | ResolvedImageGeneration | TextToSpeech: diff --git a/surfsense_local/backend/worker/studio/job_router.py b/surfsense_local/backend/worker/studio/job_router.py index 8a29be0b84..293791c11e 100644 --- a/surfsense_local/backend/worker/studio/job_router.py +++ b/surfsense_local/backend/worker/studio/job_router.py @@ -14,6 +14,7 @@ from worker.studio.content.quiz import pipeline as quiz from worker.studio.content.summary import pipeline as summary from worker.studio.media.audio.podcast import pipeline as podcast +from worker.studio.media.visual.comic import pipeline as comic from worker.studio.media.visual.image import pipeline as image from worker.studio.media.visual.infographic import pipeline as infographic from worker.studio.office import pipeline as office @@ -40,6 +41,7 @@ class Kind(enum.StrEnum): PDF = "pdf" IMAGE = "image" INFOGRAPHIC = "infographic" + COMIC = "comic" PODCAST = "podcast" @@ -71,5 +73,7 @@ def pipeline_for(kind: Kind) -> Render: return image.render case Kind.INFOGRAPHIC: return infographic.render + case Kind.COMIC: + return comic.render case Kind.PODCAST: return podcast.render diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/__init__.py b/surfsense_local/backend/worker/studio/media/visual/comic/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/asking.py b/surfsense_local/backend/worker/studio/media/visual/comic/asking.py new file mode 100644 index 0000000000..bf8f65ec76 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/asking.py @@ -0,0 +1,61 @@ +"""Asking the chat model for the comic's plan, once more if its reply is not one.""" + +import logging + +from modules.artifacts.comic.kinds import Kind +from modules.llm.resolution import ResolvedGeneration +from worker.studio.media.visual.comic import plan as planning +from worker.studio.media.visual.comic.schema import REPLY +from worker.studio.shared import generate +from worker.studio.shared.artifact import Source + +logger = logging.getLogger(__name__) + +# Enough of a reply to tell a refusal from broken JSON in the log. +_LOGGED_CHARS = 500 + + +def ask_for_plan( + writer: ResolvedGeneration, + sources: list[Source], + user_prompt: str | None, + kind: Kind, + count: int, + untitled: str, +) -> planning.Plan: + """Nothing is drawn yet, so a second ask costs one text reply, not pages.""" + system = planning.prompt(writer.tier, user_prompt, kind, count) + reply = generate.run_model(writer, system, sources, json_schema=REPLY) + try: + return planning.parse(reply, untitled, count) + except ValueError as first: + first_error = first + logger.warning( + "studio: comic plan unreadable (%s); asking again. It began: %r", + first, + reply[:_LOGGED_CHARS], + ) + retry = generate.run_model( + writer, + system, + sources, + json_schema=REPLY, + repair=generate.Repair( + reply, + f"Your previous reply could not be used: {first_error}. Return only " + f"the plan as one JSON object, with exactly {count} pages.", + ), + ) + try: + return planning.parse(retry, untitled, count) + except ValueError as error: + logger.warning("studio: comic plan unreadable again: %r", retry[:_LOGGED_CHARS]) + raise _unplanned(retry, error) from error + + +def _unplanned(reply: str, error: ValueError) -> ValueError: + """A reply with no JSON in it is the model saying no; the user reads its words.""" + said = next((line.strip() for line in reply.splitlines() if line.strip()), "") + if said and "{" not in reply: + return ValueError(f"The model did not plan the comic. It said: {said}") + return ValueError(f"The model's comic plan could not be read: {error}") diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/draw.py b/surfsense_local/backend/worker/studio/media/visual/comic/draw.py new file mode 100644 index 0000000000..7c36715293 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/draw.py @@ -0,0 +1,43 @@ +"""Page 1 alone, then every later page with page 1 as its reference, so the cast stays the same.""" + +import asyncio +from collections.abc import Awaitable + +from modules.llm.pictures_in.model import PictureModel +from modules.llm.providers.protocols import GeneratedImage +from modules.llm.providers.sdcpp import provider as sdcpp +from modules.llm.resolution import ResolvedImageGeneration + + +async def draw_pages( + painter: ResolvedImageGeneration, + keeper: PictureModel | None, + prompts: list[str], + shape: str, +) -> list[GeneratedImage]: + first = await painter.generator.generate( + painter.selection.name, prompts[0], shape=shape + ) + + def page(prompt: str) -> Awaitable[GeneratedImage]: + if keeper is None: + return painter.generator.generate( + painter.selection.name, prompt, shape=shape + ) + return keeper.editor.edit(keeper.selection.name, prompt, (first,), shape=shape) + + rest = prompts[1:] + # A remote model draws the later pages side by side; this computer's, one at a time. + if _on_this_computer(painter, keeper): + drawn = [await page(prompt) for prompt in rest] + else: + drawn = list(await asyncio.gather(*(page(prompt) for prompt in rest))) + return [first, *drawn] + + +def _on_this_computer( + painter: ResolvedImageGeneration, keeper: PictureModel | None +) -> bool: + return painter.selection.provider == sdcpp.PROVIDER or ( + keeper is not None and keeper.selection.provider == sdcpp.PROVIDER + ) diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/page_prompt.py b/surfsense_local/backend/worker/studio/media/visual/comic/page_prompt.py new file mode 100644 index 0000000000..4679df80c7 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/page_prompt.py @@ -0,0 +1,56 @@ +"""One page's prompt, laid out as image models read a comic page best.""" + +from modules.artifacts.comic.kinds import Kind +from worker.studio.media.visual.comic.plan import Panel, Plan + +KEEP_CHARACTERS = ( + "Keep the characters looking exactly as in the reference picture: same faces, " + "colours, clothes and drawing style. Draw a new page; do not copy its panels." +) +NO_WORDS = "Draw no text, no lettering, no speech bubbles." + + +def page_prompt( + kind: Kind, + plan: Plan, + number: int, + *, + lettered: bool, + referenced: bool, +) -> str: + panels = plan.pages[number - 1] + lines = [" ".join(part for part in (kind.looks, plan.style) if part)] + if plan.characters: + lines.append(f"Characters: {plan.characters}") + if referenced: + lines.append(KEEP_CHARACTERS) + lines.append( + f'{kind.unit} {number} of {len(plan.pages)} of the comic "{plan.title}". ' + f"{kind.picture}:" + ) + lines += [ + _panel(index, panel, lettered) for index, panel in enumerate(panels, start=1) + ] + if not lettered: + lines.append(NO_WORDS) + return "\n".join(lines) + + +def strip_lines(panels: tuple[Panel, ...]) -> tuple[str, ...]: + """A page's lines in reading order, for the strip under a page drawn without words.""" + return tuple(line for panel in panels for line in panel.lines) + + +def _panel(index: int, panel: Panel, lettered: bool) -> str: + where = f" ({panel.where})" if panel.where else "" + said = [_said(line) for line in panel.lines] if lettered else [] + return " ".join([f"Panel {index}{where}: {panel.shows}", *said]) + + +def _said(line: str) -> str: + speaker, colon, words = line.partition(":") + if not colon or not words.strip(): + return f'Caption box: "{line.strip()}"' + if speaker.strip().lower() == "caption": + return f'Caption box: "{words.strip()}"' + return f'{speaker.strip()} says: "{words.strip()}"' diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py new file mode 100644 index 0000000000..8ebde04868 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py @@ -0,0 +1,88 @@ +"""Comic: the chat model plans it from the sources, the image model draws each +page, and the pages are bound into one PDF in the kind's reading order.""" + +import asyncio + +from modules.artifacts.comic.kinds import KINDS +from modules.llm.pictures_in.model import PictureModel +from modules.llm.providers.sdcpp import provider as sdcpp +from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration +from worker.studio.media.visual.comic.asking import ask_for_plan +from worker.studio.media.visual.comic.draw import draw_pages +from worker.studio.media.visual.comic.page_prompt import page_prompt, strip_lines +from worker.studio.media.visual.comic.plan import Plan +from worker.studio.media.visual.comic_binding.bind import BoundPage, bind +from worker.studio.media.visual.drawn.pipeline import DRAWN_FROM_WORDS +from worker.studio.shared.artifact import Built, Source, fallback_title +from worker.studio.shared.text import slug + + +def render( + painter: ResolvedImageGeneration, + writer: ResolvedGeneration, + sources: list[Source], + user_prompt: str | None, + options: dict, + *, + keeper: PictureModel | None, +) -> Built: + kind = KINDS[options["kind"]] + plan = ask_for_plan( + writer, + sources, + user_prompt, + kind, + options["pages"], + fallback_title(user_prompt, sources, "Comic"), + ) + # Studio cannot look at a page to check its lettering; a local model's is not + # measured at all, so its pages are drawn without words and lettered below. + # Any page drawn locally letters them all so, keeping one comic one way. + drawers = [painter.selection.provider] + if keeper is not None and len(plan.pages) > 1: + drawers.append(keeper.selection.provider) + lettered = kind.bubbles and sdcpp.PROVIDER not in drawers + prompts = [ + page_prompt( + kind, + plan, + number, + lettered=lettered, + referenced=keeper is not None and number > 1, + ) + for number in range(1, len(plan.pages) + 1) + ] + pictures = asyncio.run(draw_pages(painter, keeper, prompts, kind.shape)) + pdf = bind( + plan.title, + [ + BoundPage(picture.content, () if lettered else strip_lines(panels)) + for picture, panels in zip(pictures, plan.pages, strict=True) + ], + layout=kind.layout, + direction=kind.direction, + ) + metadata: dict = { + "comic": {"kind": kind.key, "pages": len(plan.pages), "lettered": lettered} + } + if keeper is None and len(plan.pages) > 1: + metadata[DRAWN_FROM_WORDS] = True + return Built( + title=plan.title, + markdown=_markdown(plan, kind.unit), + primary=pdf, + primary_mime="application/pdf", + primary_filename=f"{slug(plan.title, 'comic')}.pdf", + metadata=metadata, + ) + + +def _markdown(plan: Plan, unit: str) -> str: + """Every page's panels and lines, so the comic is found by what it says.""" + parts = [f"# {plan.title}"] + for number, panels in enumerate(plan.pages, start=1): + parts.append(f"\n## {unit} {number}\n") + for index, panel in enumerate(panels, start=1): + parts.append(f"- Panel {index}: {panel.shows}") + parts += [f" - {line}" for line in panel.lines] + return "\n".join(parts) diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/plan.py b/surfsense_local/backend/worker/studio/media/visual/comic/plan.py new file mode 100644 index 0000000000..35f6f0e7e6 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/plan.py @@ -0,0 +1,81 @@ +"""The comic the chat model plans from the sources: its look, its cast, and every page's panels.""" + +from dataclasses import dataclass + +from modules.artifacts.comic.kinds import Kind +from modules.llm import prompting +from modules.llm.profile import Tier +from worker.studio.shared.text import as_list, as_text, parse_json + +_BUBBLES = ( + "Give each panel its lines: one short line a bubble, at most about eight words, " + 'two or three a panel, each as "NAME: words", or "Caption: words" for narration. ' + "Write every line word for word." +) +_UNDER = ( + "Give each picture one panel. Its lines are the text set under the picture, never " + "in it: a picture book's words, or a storyboard shot's number, camera and movement." +) + + +@dataclass(frozen=True) +class Panel: + where: str + shows: str + lines: tuple[str, ...] + + +@dataclass(frozen=True) +class Plan: + title: str + style: str + characters: str + pages: tuple[tuple[Panel, ...], ...] + + +def prompt(tier: Tier, user_prompt: str | None, kind: Kind, count: int) -> str: + return prompting.load( + __package__, + tier, + focus=prompting.focus(user_prompt), + kind=kind.name, + picture=kind.picture, + count=count, + unit=kind.unit.lower(), + lettering=_BUBBLES if kind.bubbles else _UNDER, + ) + + +def parse(raw: str, untitled: str, count: int) -> Plan: + """The plan, cut to the pages asked for: each one drawn is billed or takes minutes. + + One short of them is refused, so it is asked for again before any is drawn. + """ + spec = parse_json(raw) + pages = tuple( + panels + for page in as_list(spec.get("pages")) + if isinstance(page, dict) and (panels := _panels(page)) + )[:count] + if len(pages) < count: + raise ValueError(f"the plan has {len(pages)} of the {count} pages asked for") + return Plan( + title=as_text(spec.get("title")) or untitled, + style=as_text(spec.get("style")), + characters=as_text(spec.get("characters")), + pages=pages, + ) + + +def _panels(page: dict) -> tuple[Panel, ...]: + return tuple( + Panel( + where=as_text(panel.get("where")), + shows=as_text(panel.get("shows")), + lines=tuple( + line for item in as_list(panel.get("lines")) if (line := as_text(item)) + ), + ) + for panel in as_list(page.get("panels")) + if isinstance(panel, dict) and as_text(panel.get("shows")) + ) diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/prompts/capable.md b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/capable.md new file mode 100644 index 0000000000..0318e32547 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/capable.md @@ -0,0 +1,26 @@ +You plan a comic from source documents: a $kind of $count $unit(s), told from their facts only. + +Task: plan the comic. Each $unit is one picture: $picture. +$focus +Work in this order: + +1. Read every source and pick the story it tells best in $count $unit(s): who, what happens, and how it ends. +2. Write a style line: medium, palette and line weight, in one sentence. +3. Write a character sheet: each character's face, hair, build, clothes and colours in a few fixed words. Every page repeats it word for word, so keep it short and exact. +4. Plan each $unit in reading order: for every panel, where it sits (top, bottom left, wide), what it shows, and its lines. + +$lettering + +Grounding rules: + +- Use only what the sources state. Simplify, but never invent facts, names or figures. +- Plan exactly $count $unit(s), no more. +- A panel shows one moment. Describe what is seen, not what is meant. + +Return only JSON, no prose: {"title": str, "style": str, "characters": str, "pages": [{"panels": [{"where": str, "shows": str, "lines": [str]}]}]} + +Worked example — a different story in the same shape. Copy the structure, never the content: + +{"title": "The Lighthouse Cat", "style": "Bold ink lines, flat warm colours, night-blue sky.", "characters": "MILO, a small orange tabby cat with a white chest, green eyes and a red scarf. NORA, a woman about seventy, short white hair, round glasses, yellow raincoat.", "pages": [{"panels": [{"where": "wide, top", "shows": "A fishing boat tosses on huge waves at night.", "lines": ["Caption: The storm came at midnight."]}, {"where": "bottom left", "shows": "NORA taps the dead lamp.", "lines": ["NORA: The bulb is broken!"]}]}]} + +Return only the JSON. Nothing before it, nothing after it. diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/prompts/compact.md b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/compact.md new file mode 100644 index 0000000000..9ddf1ce0fc --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/compact.md @@ -0,0 +1,25 @@ +You plan a short comic from the sources below. Use their facts only. + +The comic is a $kind. It has exactly $count $unit(s). Each $unit is one picture: $picture. +$focus + +Write: + +- "title": the comic's title. +- "style": one sentence: medium, colours, line weight. +- "characters": each character in a few fixed words: face, hair, clothes, colours. +- "pages": exactly $count items. Each has "panels". Each panel has "where" (top, bottom left, wide), "shows" (what is seen in it), and "lines". + +$lettering + +Rules: + +- Only facts from the sources. Do not invent names or numbers. +- Exactly $count $unit(s). +- One moment per panel. + +Example of the shape only, not the content: + +{"title": "The Lighthouse Cat", "style": "Bold ink lines, flat warm colours.", "characters": "MILO, a small orange tabby cat with a red scarf.", "pages": [{"panels": [{"where": "top", "shows": "A boat in a storm at night.", "lines": ["Caption: The storm came at midnight."]}, {"where": "bottom", "shows": "MILO looks at the dark lamp.", "lines": ["MILO: The light is out!"]}]}]} + +Return only the JSON. Nothing before it, nothing after it. diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/prompts/frontier.md b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/frontier.md new file mode 100644 index 0000000000..242a6cac1e --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/frontier.md @@ -0,0 +1,10 @@ +Plan a $kind of exactly $count $unit(s) that tells the sources' story faithfully. Each $unit is one picture: $picture. +$focus + +Choose the story the sources tell best at this length. Write a one-sentence style line (medium, palette, line weight) and a short, exact character sheet (each character's face, hair, build, clothes and colours): both are repeated word for word in every picture's prompt, so the cast looks the same throughout. Then plan each $unit in reading order: for every panel, where it sits, what is seen in it, and its lines. + +$lettering + +Use only what the sources state; simplify, never invent facts, names or figures. + +Return only JSON: {"title": str, "style": str, "characters": str, "pages": [{"panels": [{"where": str, "shows": str, "lines": [str]}]}]} diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/schema.py b/surfsense_local/backend/worker/studio/media/visual/comic/schema.py new file mode 100644 index 0000000000..58d9fd06e8 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/schema.py @@ -0,0 +1,33 @@ +"""The plan reply's shape, enforced as a grammar where the endpoint takes one. + +Only types and `required`, like the deck's: the page count stays the prompt's +to ask for and the plan's to cut. +""" + +_PANEL = { + "type": "object", + "properties": { + "where": {"type": "string"}, + "shows": {"type": "string"}, + "lines": {"type": "array", "items": {"type": "string"}}, + }, + "required": ["where", "shows", "lines"], +} + +REPLY = { + "type": "object", + "properties": { + "title": {"type": "string"}, + "style": {"type": "string"}, + "characters": {"type": "string"}, + "pages": { + "type": "array", + "items": { + "type": "object", + "properties": {"panels": {"type": "array", "items": _PANEL}}, + "required": ["panels"], + }, + }, + }, + "required": ["title", "style", "characters", "pages"], +} diff --git a/surfsense_local/backend/worker/studio/media/visual/comic_binding/__init__.py b/surfsense_local/backend/worker/studio/media/visual/comic_binding/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py new file mode 100644 index 0000000000..c229d264d5 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py @@ -0,0 +1,119 @@ +"""A comic's pictures bound into one PDF, as Studio and the agent both bind them.""" + +from dataclasses import dataclass +from io import BytesIO + +from PIL import Image +from reportlab.lib.units import inch +from reportlab.lib.utils import ImageReader +from reportlab.pdfbase.pdfmetrics import stringWidth +from reportlab.pdfgen import canvas + +from modules.artifacts.comic.reading import Direction, Layout +from worker.studio.office.markdown.pdf_fonts import font_runs + +# Comic-book trim width; each page is as tall as its picture. +WIDTH = 6.625 * inch +MARGIN = 0.25 * inch +GUTTER = 0.2 * inch +FONT, SIZE, LEADING = "Helvetica", 11, 15 +# A scroll starts a new page every twelve pictures, so no viewer meets one huge page. +PER_SCROLL = 12 +# JPEG, not PNG: four PNG pages made 10 to 28 MB in the spike. +JPEG_QUALITY = 90 + + +@dataclass(frozen=True) +class BoundPage: + picture: bytes + # Set under the picture exactly: a misspelt page's, or one drawn without words. + lines: tuple[str, ...] = () + + +@dataclass(frozen=True) +class _Block: + image: ImageReader + height: float + lines: list[str] + strip: float + + +def bind( + title: str, pages: list[BoundPage], *, layout: Layout, direction: Direction +) -> bytes: + out = BytesIO() + pdf = canvas.Canvas(out) + pdf.setTitle(title) + if direction == "rtl": + pdf.setViewerPreference("Direction", "R2L") + blocks = [_block(page) for page in pages] + if layout == "scroll": + for start in range(0, len(blocks), PER_SCROLL): + _draw(pdf, blocks[start : start + PER_SCROLL], GUTTER) + else: + for one in blocks: + _draw(pdf, [one], 0) + pdf.save() + return out.getvalue() + + +def _block(page: BoundPage) -> _Block: + with Image.open(BytesIO(page.picture)) as opened: + picture = opened.convert("RGB") + jpeg = BytesIO() + picture.save(jpeg, "JPEG", quality=JPEG_QUALITY) + jpeg.seek(0) + lines = [line for text in page.lines for line in _wrapped(text, WIDTH - 2 * MARGIN)] + strip = len(lines) * LEADING + 2 * MARGIN if lines else 0 + return _Block( + ImageReader(jpeg), WIDTH * picture.height / picture.width, lines, strip + ) + + +def _wrapped(text: str, width: float) -> list[str]: + lines: list[str] = [] + line = "" + for piece, glued in _breakable(text, width): + trial = f"{line}{'' if glued else ' '}{piece}" if line else piece + if line and _width(trial) > width: + lines.append(line) + line = piece + else: + line = trial + return [*lines, line] if line else lines + + +def _breakable(text: str, width: float) -> list[tuple[str, bool]]: + """Words, with one too wide for a line, as Chinese or Japanese is, cut into + letters; each piece says whether it joins the one before with no space.""" + pieces: list[tuple[str, bool]] = [] + for word in text.split(): + cut = list(word) if _width(word) > width else [word] + pieces += [(piece, n > 0) for n, piece in enumerate(cut)] + return pieces + + +def _width(text: str) -> float: + return sum(stringWidth(run, font or FONT, SIZE) for font, run in font_runs(text)) + + +def _set_line(pdf: canvas.Canvas, x: float, y: float, line: str) -> None: + for font, run in font_runs(line): + pdf.setFont(font or FONT, SIZE) + pdf.drawString(x, y, run) + x += stringWidth(run, font or FONT, SIZE) + + +def _draw(pdf: canvas.Canvas, blocks: list[_Block], gutter: float) -> None: + """Top to bottom on one page, each picture with its strip under it.""" + height = sum(b.height + b.strip for b in blocks) + gutter * (len(blocks) - 1) + pdf.setPageSize((WIDTH, height)) + top = height + for block in blocks: + pdf.drawImage(block.image, 0, top - block.height, WIDTH, block.height) + y = top - block.height - MARGIN - SIZE + for line in block.lines: + _set_line(pdf, MARGIN, y, line) + y -= LEADING + top -= block.height + block.strip + gutter + pdf.showPage() diff --git a/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py new file mode 100644 index 0000000000..bc456708c3 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py @@ -0,0 +1,52 @@ +"""The agent's comic: the pictures it drew, read from Studio's store and bound as it asked.""" + +from dataclasses import dataclass +from pathlib import Path + +from modules.artifacts.comic.binding_recipe import ComicBinding +from worker.studio.media.visual.comic_binding.bind import BoundPage, bind +from worker.studio.shared.artifact import Built +from worker.studio.shared.text import slug + + +@dataclass(frozen=True) +class KeptPage: + picture: Path + # The words the page was drawn from: its panels and bubbles, as Studio's plan holds them. + drawn_from: str + + +def render(title: str, binding: ComicBinding, pages: list[KeptPage]) -> Built: + """`pages` are the binding's pages' files and prompts, in its order.""" + pdf = bind( + title, + [ + BoundPage(kept.picture.read_bytes(), page.lines) + for kept, page in zip(pages, binding.pages, strict=True) + ], + layout=binding.layout, + direction=binding.direction, + ) + return Built( + title=title, + markdown=_searchable(title, binding, pages), + primary=pdf, + primary_mime="application/pdf", + primary_filename=f"{slug(title, 'comic')}.pdf", + ) + + +def _searchable(title: str, binding: ComicBinding, pages: list[KeptPage]) -> str: + """Each page's prompt and lines, a line every page repeats (its style, its + cast) kept the first time only, so it does not outweigh what the pages say.""" + seen: set[str] = set() + body = [f"# {title}"] + for number, (kept, page) in enumerate(zip(pages, binding.pages, strict=True), 1): + body.append(f"\n## Page {number}\n") + for line in kept.drawn_from.splitlines(): + line = line.strip() + if line and line not in seen: + seen.add(line) + body.append(line) + body += [f"- {line}" for line in page.lines] + return "\n".join(body) diff --git a/surfsense_local/backend/worker/studio/media/visual/drawn/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/drawn/pipeline.py index 45b2beee07..e4a4c0152e 100644 --- a/surfsense_local/backend/worker/studio/media/visual/drawn/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/drawn/pipeline.py @@ -1,26 +1,53 @@ """Drawn: the prompt the agent wrote goes to the image model as written. No chat model rewrites it: the agent has already read what the picture draws on. +With reference pictures it goes to the model that takes pictures, so a comic's +pages keep their characters; with none, it is drawn from the words alone. """ import asyncio +from modules.llm.pictures_in.model import PictureModel +from modules.llm.providers.picture_shape import Shape +from modules.llm.providers.protocols import GeneratedImage from modules.llm.resolution import ResolvedImageGeneration from worker.studio.media.visual import EXTENSIONS from worker.studio.media.visual.infographic.style import in_infographic_style +from worker.studio.media.visual.redraw.drawing_prompt import kept_drawing_prompt from worker.studio.shared.artifact import Built from worker.studio.shared.text import slug +# Kept on a version whose references no chosen model could take. +DRAWN_FROM_WORDS = "drawn_from_words" + def render( - painter: ResolvedImageGeneration, prompt: str, title: str, format: str + painter: ResolvedImageGeneration, + prompt: str, + title: str, + format: str, + *, + shape: Shape | None = None, + references: tuple[GeneratedImage, ...] = (), + keeper: PictureModel | None = None, ) -> Built: painted = in_infographic_style(prompt) if format == "infographic" else prompt - image = asyncio.run(painter.generator.generate(painter.selection.name, painted)) + metadata = kept_drawing_prompt(painted) + if references and keeper is not None: + image = asyncio.run( + keeper.editor.edit(keeper.selection.name, painted, references, shape=shape) + ) + else: + image = asyncio.run( + painter.generator.generate(painter.selection.name, painted, shape=shape) + ) + if references: + metadata[DRAWN_FROM_WORDS] = True return Built( title=title, markdown=f"# {title}\n\n{prompt}", primary=image.content, primary_mime=image.media_type, primary_filename=f"{slug(title, format)}.{EXTENSIONS[image.media_type]}", + metadata=metadata, ) diff --git a/surfsense_local/backend/worker/studio/media/visual/drawn/references.py b/surfsense_local/backend/worker/studio/media/visual/drawn/references.py new file mode 100644 index 0000000000..fe4bd4d9e9 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/drawn/references.py @@ -0,0 +1,32 @@ +"""The pictures a drawing keeps the characters and style of, read as stored.""" + +from sqlalchemy.orm import Session + +from modules.artifacts.models import Artifact, ArtifactFileRole +from modules.llm.providers.protocols import GeneratedImage +from shared.config import get_storage_settings + + +class ReferenceGoneError(RuntimeError): + """A picture this drawing keeps was deleted before it was drawn.""" + + +def reference_pictures( + session: Session, artifact_ids: tuple[int, ...] +) -> tuple[GeneratedImage, ...]: + return tuple(_picture(session, artifact_id) for artifact_id in artifact_ids) + + +def _picture(session: Session, artifact_id: int) -> GeneratedImage: + artifact = session.get(Artifact, artifact_id) + file = ( + next((f for f in artifact.files if f.role is ArtifactFileRole.PRIMARY), None) + if artifact is not None + else None + ) + path = get_storage_settings().data_dir / file.storage_key if file else None + if file is None or path is None or not path.is_file(): + raise ReferenceGoneError( + f"The picture image-{artifact_id} this one keeps is gone." + ) + return GeneratedImage(path.read_bytes(), file.mime_type) diff --git a/surfsense_local/backend/worker/studio/media/visual/edit/base.py b/surfsense_local/backend/worker/studio/media/visual/edit/base.py index b4179d5bd3..51f2f9d9af 100644 --- a/surfsense_local/backend/worker/studio/media/visual/edit/base.py +++ b/surfsense_local/backend/worker/studio/media/visual/edit/base.py @@ -9,6 +9,7 @@ from modules.artifacts.models import Artifact, ArtifactFileRole from modules.llm.providers.protocols import GeneratedImage from shared.config import get_storage_settings +from worker.studio.media.visual.redraw.drawing_prompt import drawing_prompt_of class BaseImageGoneError(RuntimeError): @@ -22,6 +23,8 @@ class BaseImage: picture: GeneratedImage # The base's file name without its extension, which the edit's format decides. stem: str + # The words it was drawn from, which a redraw starts from. + drawing_prompt: str def base_image(session: Session, edit: ImageEdit) -> BaseImage: @@ -39,4 +42,5 @@ def base_image(session: Session, edit: ImageEdit) -> BaseImage: body=base.document.content or "", picture=GeneratedImage(path.read_bytes(), file.mime_type), stem=PurePath(file.original_filename).stem, + drawing_prompt=drawing_prompt_of(base), ) diff --git a/surfsense_local/backend/worker/studio/media/visual/edit/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/edit/pipeline.py index 2cbc85c735..a086aa4259 100644 --- a/surfsense_local/backend/worker/studio/media/visual/edit/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/edit/pipeline.py @@ -6,15 +6,19 @@ import asyncio -from modules.llm.resolution import ResolvedImageEdit +from modules.llm.pictures_in.model import PictureModel from worker.studio.media.visual import EXTENSIONS from worker.studio.media.visual.edit.base import BaseImage +from worker.studio.media.visual.redraw.drawing_prompt import ( + changed, + kept_drawing_prompt, +) from worker.studio.shared.artifact import Built -def render(editor: ResolvedImageEdit, base: BaseImage, instruction: str) -> Built: +def render(editor: PictureModel, base: BaseImage, instruction: str) -> Built: image = asyncio.run( - editor.editor.edit(editor.selection.name, instruction, base.picture) + editor.editor.edit(editor.selection.name, instruction, (base.picture,)) ) # The base's body still says what the picture shows; the line says how it moved. body = ( @@ -28,4 +32,6 @@ def render(editor: ResolvedImageEdit, base: BaseImage, instruction: str) -> Buil primary=image.content, primary_mime=image.media_type, primary_filename=f"{base.stem}.{EXTENSIONS[image.media_type]}", + # So a later redraw, on a model that takes no pictures, keeps this change. + metadata=kept_drawing_prompt(changed(base.drawing_prompt, instruction)), ) diff --git a/surfsense_local/backend/worker/studio/media/visual/image/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/image/pipeline.py index 12edef3a7c..ec7b07e961 100644 --- a/surfsense_local/backend/worker/studio/media/visual/image/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/image/pipeline.py @@ -7,6 +7,7 @@ from modules.llm.profile import Tier from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration from worker.studio.media.visual import EXTENSIONS +from worker.studio.media.visual.redraw.drawing_prompt import kept_drawing_prompt from worker.studio.shared import generate from worker.studio.shared.artifact import Built, Source, fallback_title from worker.studio.shared.text import as_text, parse_json, slug @@ -35,6 +36,7 @@ def render( primary=image.content, primary_mime=image.media_type, primary_filename=f"{slug(title, 'image')}.{EXTENSIONS[image.media_type]}", + metadata=kept_drawing_prompt(image_prompt), ) diff --git a/surfsense_local/backend/worker/studio/media/visual/infographic/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/infographic/pipeline.py index 64a965dcac..fc878e8021 100644 --- a/surfsense_local/backend/worker/studio/media/visual/infographic/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/infographic/pipeline.py @@ -11,6 +11,7 @@ from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration from worker.studio.media.visual import EXTENSIONS from worker.studio.media.visual.infographic.style import in_infographic_style +from worker.studio.media.visual.redraw.drawing_prompt import kept_drawing_prompt from worker.studio.shared import generate from worker.studio.shared.artifact import Built, Source, fallback_title from worker.studio.shared.text import as_list, as_text, parse_json, slug @@ -30,9 +31,8 @@ def render( generate.run_model(writer, prompt(writer.tier, user_prompt), sources), fallback_title(user_prompt, sources, "Infographic"), ) - image = asyncio.run( - painter.generator.generate(painter.selection.name, _image_prompt(brief)) - ) + painted = _image_prompt(brief) + image = asyncio.run(painter.generator.generate(painter.selection.name, painted)) title = brief[0] return Built( title=title, @@ -40,6 +40,7 @@ def render( primary=image.content, primary_mime=image.media_type, primary_filename=f"{slug(title, 'infographic')}.{EXTENSIONS[image.media_type]}", + metadata=kept_drawing_prompt(painted), ) diff --git a/surfsense_local/backend/worker/studio/media/visual/redraw/__init__.py b/surfsense_local/backend/worker/studio/media/visual/redraw/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py b/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py new file mode 100644 index 0000000000..9056f23b72 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py @@ -0,0 +1,46 @@ +"""The words a picture was drawn from, which a redraw starts from. + +Kept as `drawing_prompt` from the first version that keeps it; a picture drawn +before has only its recipe's prompt, or the body Studio wrote under its title. +""" + +from typing import Any + +from modules.artifacts.drawn_images.recipe import image_drawing_of +from modules.artifacts.models import Artifact +from worker.studio.media.visual.infographic.style import in_infographic_style + +KEY = "drawing_prompt" + + +def kept_drawing_prompt(prompt: str) -> dict[str, Any]: + return {KEY: prompt} + + +def changed(prompt: str, instruction: str) -> str: + """The prompt with one more change, so changes add up from version to version.""" + return f"{prompt}\n\nChange: {instruction}" + + +def drawing_prompt_of(artifact: Artifact) -> str: + meta = artifact.artifact_metadata or {} + kept = meta.get(KEY) + if isinstance(kept, str) and kept.strip(): + return kept + drawing = image_drawing_of(meta) + written = ( + drawing.prompt + if drawing is not None + else _below_title(artifact.document.content or "", artifact.document.title) + ) + # Neither the agent's prompt nor Studio's body carries the infographic's style. + if artifact.format == "infographic": + return in_infographic_style(written) + return written + + +def _below_title(body: str, title: str) -> str: + """Studio's image body is `# <title>` and then the prompt it drew.""" + heading = f"# {title}" + text = body[len(heading) :] if body.startswith(heading) else body + return text.strip() or title diff --git a/surfsense_local/backend/worker/studio/media/visual/redraw/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/redraw/pipeline.py new file mode 100644 index 0000000000..44874365d4 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/redraw/pipeline.py @@ -0,0 +1,51 @@ +"""Redraw: an edit on an image model that takes no picture in. It draws again +from the words the base was drawn with and the change, so Edit works on every +image model; the version says it was drawn again, since it will not keep the +base's composition.""" + +import asyncio +import io + +from PIL import Image, UnidentifiedImageError + +from modules.llm.providers.picture_shape import Shape, shape_of +from modules.llm.resolution import ResolvedImageGeneration +from worker.studio.media.visual import EXTENSIONS +from worker.studio.media.visual.edit.base import BaseImage +from worker.studio.media.visual.redraw.drawing_prompt import ( + changed, + kept_drawing_prompt, +) +from worker.studio.shared.artifact import Built + +SAID = "Drawn again with your change." + + +def render( + painter: ResolvedImageGeneration, base: BaseImage, instruction: str +) -> Built: + prompt = changed(base.drawing_prompt, instruction) + image = asyncio.run( + painter.generator.generate(painter.selection.name, prompt, shape=_shape(base)) + ) + edited = ( + f"{base.body}\n\nEdited: {instruction}" + if base.body + else f"Edited: {instruction}" + ) + return Built( + title=base.title, + markdown=f"{SAID}\n\n{edited}", + primary=image.content, + primary_mime=image.media_type, + primary_filename=f"{base.stem}.{EXTENSIONS[image.media_type]}", + metadata={**kept_drawing_prompt(prompt), "redrawn": True}, + ) + + +def _shape(base: BaseImage) -> Shape | None: + try: + width, height = Image.open(io.BytesIO(base.picture.content)).size + except (UnidentifiedImageError, OSError): + return None + return shape_of(width, height) diff --git a/surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py b/surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py index 07c4246978..67b96c0019 100644 --- a/surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py +++ b/surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py @@ -44,21 +44,26 @@ def register_fonts() -> None: def font_markup(text: str) -> str: """The text escaped for a Paragraph, each run Helvetica cannot set in a font that can.""" + return "".join(_wrapped(run, font) for font, run in font_runs(text)) + + +def font_runs(text: str) -> list[tuple[str | None, str]]: + """The text cut where its font changes; None is the default font's run.""" register_fonts() han = _han_font(text) - parts: list[str] = [] + runs: list[tuple[str | None, str]] = [] run: list[str] = [] current: str | None = None for character in text: font = _font_for(character, han) if font != current and run: - parts.append(_wrapped("".join(run), current)) + runs.append((current, "".join(run))) run = [] current = font run.append(character) if run: - parts.append(_wrapped("".join(run), current)) - return "".join(parts) + runs.append((current, "".join(run))) + return runs def _wrapped(text: str, font: str | None) -> str: diff --git a/surfsense_local/backend/worker/studio/shared/persist.py b/surfsense_local/backend/worker/studio/shared/persist.py index 5be120ef8f..1e755f1b1f 100644 --- a/surfsense_local/backend/worker/studio/shared/persist.py +++ b/surfsense_local/backend/worker/studio/shared/persist.py @@ -4,6 +4,7 @@ from sqlalchemy.orm import Session +from modules.artifacts.comic.pages import is_comic_page from modules.artifacts.models import Artifact, ArtifactFile, ArtifactFileRole from modules.documents.models import Document from modules.embedding.active import ActiveIndex, require_active_index @@ -38,7 +39,9 @@ def persist( source; the sidecar's files hold only the deliverable bytes. """ logger.info("studio: persist artifact %s indexing", artifact.id) - index, passages, vectors = _embed(session, built.markdown) + # A comic's page is found through its comic, as a PDF's pages are. + searchable = "" if is_comic_page(artifact.artifact_metadata) else built.markdown + index, passages, vectors = _embed(session, searchable) document.title = built.title document.content = built.markdown indexing.replace_chunks(session, document, index, passages, vectors) diff --git a/surfsense_local/backend/worker/waiting_renders.py b/surfsense_local/backend/worker/waiting_renders.py new file mode 100644 index 0000000000..2be44cd503 --- /dev/null +++ b/surfsense_local/backend/worker/waiting_renders.py @@ -0,0 +1,38 @@ +"""Settle the documents waiting on pictures from the worker, and tell open windows.""" + +import logging + +from modules.artifacts.waiting_renders.settle import settle_waiting_renders +from shared.config import get_storage_settings +from shared.db import create_db_engine, create_session_factory +from worker.notify import notify_artifact_updates + +logger = logging.getLogger(__name__) + + +def settle_after_job(artifact_id: int) -> None: + """A job ended, or found its artifact gone: what waited on that picture may go. + + Never raises: the job is done, and a raise would have Huey draw it again. + What is left waiting is settled by the sweep at the worker's next start. + """ + try: + _settle(picture_id=artifact_id) + except Exception: + logger.exception("could not settle what waited on picture %s", artifact_id) + + +def settle_every_waiting_render() -> None: + """At start: a picture may have ended, or been failed as interrupted, unannounced.""" + _settle() + + +def _settle(picture_id: int | None = None) -> None: + engine = create_db_engine(get_storage_settings().database_path) + try: + with create_session_factory(engine)() as session: + for render in settle_waiting_renders(session, picture_id=picture_id): + session.refresh(render) + notify_artifact_updates(render) + finally: + engine.dispose() diff --git a/surfsense_local/frontend/src/components/ui/comic-kind-icons.tsx b/surfsense_local/frontend/src/components/ui/comic-kind-icons.tsx new file mode 100644 index 0000000000..fa35706ceb --- /dev/null +++ b/surfsense_local/frontend/src/components/ui/comic-kind-icons.tsx @@ -0,0 +1,497 @@ +import { type SVGProps, useId } from "react" + +import { + Bubble, + Ground, + INK, + Panel, + PictureArt, + Sheet, + burst, +} from "./picture-art" + +// Each kind of comic Studio draws, as a picture for its card. +// SVG's own from and to animate; here they name a picture's backdrop colours. +type ArtProps = Omit<SVGProps<SVGSVGElement>, "from" | "to"> + +/** A page of colour panels, a speech bubble and a POW burst. */ +export function WesternComicIcon(props: ArtProps) { + return ( + <PictureArt from="#dbeafe" to="#e0e7ff" {...props}> + <Ground /> + <Sheet x="34" y="8" width="52" height="62" /> + <Panel x="38" y="12" width="44" height="20" fill="#93c5fd" /> + <path + d="M38 32 V26 H43 V22 H48 V27 H52 V20 H57 V25 H62 V32 Z" + fill="#3b82f6" + /> + <Bubble cx={70} cy={19} /> + <Panel x="38" y="35" width="20" height="31" fill="#fca5a5" /> + <circle + cx="48" + cy="47" + r="5" + fill="#fde68a" + stroke={INK} + strokeWidth="0.7" + /> + <path d="M43 66 C43 58 53 58 53 66 Z" fill="#ef4444" /> + <Panel x="61" y="35" width="21" height="31" fill="#fde68a" /> + <path d="M64 63 L71 41 L78 63 Z" fill="#fb923c" opacity="0.8" /> + <polygon + points={burst(92, 27, 13, 7.5)} + fill="#facc15" + stroke={INK} + strokeWidth="0.8" + strokeLinejoin="round" + /> + <polygon points={burst(92, 27, 7, 4)} fill="#ef4444" /> + </PictureArt> + ) +} + +/** A hardcover: a moonlit cover on a thick block of pages. */ +export function GraphicNovelIcon(props: ArtProps) { + const cover = `${useId()}-cover` + return ( + <PictureArt from="#ede9fe" to="#fae8ff" {...props}> + <defs> + <linearGradient id={cover} x1="0" y1="0" x2="0" y2="1"> + <stop offset="0" stopColor="#7c3aed" /> + <stop offset="1" stopColor="#4c1d95" /> + </linearGradient> + </defs> + <Ground /> + <rect + x="40" + y="10" + width="44" + height="60" + rx="2" + fill="#f5f5f4" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1" + /> + <path d="M81 12 V68 M83 13 V67" stroke="#d6d3d1" strokeWidth="0.6" /> + <rect + x="36" + y="8" + width="44" + height="60" + rx="2.5" + fill={`url(#${cover})`} + stroke={INK} + strokeOpacity="0.6" + strokeWidth="1" + /> + <rect x="36" y="8" width="5" height="60" rx="1.5" fill="#3b0764" /> + <circle cx="66" cy="22" r="6" fill="#fde68a" /> + <circle cx="68.5" cy="20.5" r="5" fill="#6d28d9" /> + <path d="M41 52 L52 40 L60 47 L69 36 L80 46 V68 H41 Z" fill="#2e1065" /> + <rect + x="46" + y="56" + width="26" + height="3" + rx="1.5" + fill="#ffffff" + opacity="0.9" + /> + <rect + x="50" + y="61" + width="18" + height="2" + rx="1" + fill="#ffffff" + opacity="0.55" + /> + <circle cx="49" cy="17" r="0.9" fill="#ffffff" /> + <circle cx="56" cy="26" r="0.7" fill="#ffffff" /> + <circle cx="74" cy="32" r="0.7" fill="#ffffff" /> + </PictureArt> + ) +} + +/** An album page in clean lines and flat colour, nine small panels. */ +export function BandeDessineeIcon(props: ArtProps) { + const fills = [ + "#93c5fd", + "#fde68a", + "#fca5a5", + "#86efac", + "#ffffff", + "#93c5fd", + "#fdba74", + "#c4b5fd", + "#fde68a", + ] + return ( + <PictureArt from="#ffedd5" to="#fef3c7" {...props}> + <Ground /> + <Sheet x="31" y="7" width="58" height="64" /> + {fills.map((fill, i) => ( + <Panel + key={fill + i} + x={35 + (i % 3) * 17.3} + y={11 + Math.floor(i / 3) * 19.6} + width="15.3" + height="17.6" + fill={fill} + /> + ))} + <Bubble cx={60.5} cy={35.5} rx={5} ry={3} /> + <circle cx="43" cy="21" r="3" fill="#fbbf24" /> + <path + d="M35 28.6 L41 23 L45 26 L50.3 21.5 V28.6 Z" + fill="#2563eb" + opacity="0.7" + /> + <path d="M78 66 l2.5 -5 l2.5 5 Z" fill="#ef4444" /> + </PictureArt> + ) +} + +/** A black-and-white page: screentone, speed lines and a wide eye, read right to left. */ +export function MangaIcon(props: ArtProps) { + const tone = `${useId()}-tone` + return ( + <PictureArt from="#fce7f3" to="#f5f3ff" {...props}> + <defs> + <pattern + id={tone} + width="2.6" + height="2.6" + patternUnits="userSpaceOnUse" + > + <circle cx="1.3" cy="1.3" r="0.55" fill={INK} opacity="0.55" /> + </pattern> + </defs> + <Ground /> + <Sheet x="36" y="8" width="48" height="62" /> + <Panel x="40" y="12" width="40" height="24" fill="#ffffff" /> + <path + d="M41 13 L60 24 M41 22 L60 24 M41 34 L60 24 M79 13 L60 24 M79 30 L60 24 M70 35 L60 24 M50 13 L60 24" + stroke={INK} + strokeOpacity="0.35" + strokeWidth="0.6" + /> + <path + d="M49 24 C53 18 67 18 71 24 C67 30 53 30 49 24 Z" + fill="#ffffff" + stroke={INK} + strokeWidth="1" + /> + <circle cx="60" cy="24" r="4.2" fill={INK} /> + <circle cx="61.4" cy="22.6" r="1.3" fill="#ffffff" /> + <Panel x="61" y="39" width="19" height="27" fill={`url(#${tone})`} /> + <Panel x="40" y="39" width="18" height="27" fill="#ffffff" /> + <ellipse + cx="49" + cy="50" + rx="5.5" + ry="7.5" + fill="#ffffff" + stroke={INK} + strokeWidth="0.7" + /> + <path + d="M47 46 V54 M50.5 45 V55" + stroke={INK} + strokeWidth="0.9" + strokeLinecap="round" + /> + <path + d="M100 15 H90 M93 12 L90 15 L93 18" + stroke="#ec4899" + strokeWidth="1.5" + strokeLinecap="round" + strokeLinejoin="round" + /> + </PictureArt> + ) +} + +/** A painterly page: a sunset over mountains, and a red lantern. */ +export function ManhuaIcon(props: ArtProps) { + const sunset = `${useId()}-sunset` + return ( + <PictureArt from="#fee2e2" to="#ffedd5" {...props}> + <defs> + <linearGradient id={sunset} x1="0" y1="0" x2="0" y2="1"> + <stop offset="0" stopColor="#fb923c" /> + <stop offset="1" stopColor="#f43f5e" /> + </linearGradient> + </defs> + <Ground /> + <Sheet x="36" y="8" width="48" height="62" /> + <Panel x="40" y="12" width="40" height="28" fill={`url(#${sunset})`} /> + <circle cx="66" cy="22" r="5" fill="#fde68a" opacity="0.9" /> + <path d="M40.4 40 L48 27 L55 34 L63 24 L79.6 39.6 V40 Z" fill="#9f1239" /> + <path d="M40.4 40 L46 34 L53 38 L58 33 L68 40 Z" fill="#4c0519" /> + <Panel x="40" y="43" width="40" height="23" fill="#fef3c7" /> + <path d="M51 43 V47" stroke="#b45309" strokeWidth="0.8" /> + <rect x="47.5" y="47" width="7" height="2" rx="0.6" fill="#f59e0b" /> + <ellipse cx="51" cy="54" rx="6" ry="5.5" fill="#dc2626" /> + <path + d="M48 50 Q51 54 48 58 M54 50 Q51 54 54 58" + stroke="#991b1b" + strokeWidth="0.6" + /> + <rect x="47.5" y="58.5" width="7" height="2" rx="0.6" fill="#f59e0b" /> + <path d="M51 60.5 V64" stroke="#f59e0b" strokeWidth="1" /> + <path + d="M62 49 H75 M62 53 H72 M62 57 H74" + stroke="#d6b88a" + strokeWidth="1.2" + strokeLinecap="round" + /> + </PictureArt> + ) +} + +/** A phone of tall panels stacked to scroll down. */ +export function WebtoonIcon(props: ArtProps) { + const screen = `${useId()}-screen` + return ( + <PictureArt from="#d1fae5" to="#cffafe" {...props}> + <defs> + <clipPath id={screen}> + <rect x="46.5" y="10" width="27" height="58" rx="2" /> + </clipPath> + </defs> + <Ground rx={22} /> + <rect x="44" y="6" width="32" height="66" rx="5" fill={INK} /> + <rect x="46.5" y="10" width="27" height="58" rx="2" fill="#ffffff" /> + <g clipPath={`url(#${screen})`}> + <Panel x="48.5" y="12" width="23" height="17" fill="#a7f3d0" /> + <circle cx="64" cy="18" r="3.5" fill="#fbbf24" /> + <path d="M48.5 29 L55 22 L61 29 Z" fill="#10b981" /> + <Panel x="48.5" y="32" width="23" height="17" fill="#bfdbfe" /> + <Bubble cx={57} cy={38} rx={5} ry={3} /> + <Panel x="48.5" y="52" width="23" height="17" fill="#fbcfe8" /> + <circle cx="60" cy="61" r="4" fill="#f472b6" /> + </g> + <rect x="55" y="7.5" width="10" height="1.5" rx="0.75" fill="#475569" /> + <path + d="M88 22 V54 M84 50 L88 55 L92 50" + stroke="#10b981" + strokeWidth="1.8" + strokeLinecap="round" + strokeLinejoin="round" + /> + <path + d="M88 22 V54" + stroke="#10b981" + strokeWidth="1.8" + strokeDasharray="3 3" + strokeLinecap="round" + /> + </PictureArt> + ) +} + +/** Three panels in a row, the punchline last. */ +export function ComicStripIcon(props: ArtProps) { + return ( + <PictureArt from="#fef9c3" to="#fde68a" {...props}> + <Ground rx={46} /> + <Sheet x="12" y="22" width="96" height="38" /> + <Panel x="15.5" y="25.5" width="28" height="31" fill="#bae6fd" /> + <Panel x="46" y="25.5" width="28" height="31" fill="#c7d2fe" /> + <Panel x="76.5" y="25.5" width="28" height="31" fill="#fecaca" /> + <Bubble cx={26} cy={32} rx={6} ry={3.2} /> + <circle + cx="31" + cy="48" + r="4.5" + fill="#fde68a" + stroke={INK} + strokeWidth="0.7" + /> + <Bubble cx={57} cy={32} rx={6} ry={3.2} /> + <circle + cx="62" + cy="48" + r="4.5" + fill="#fde68a" + stroke={INK} + strokeWidth="0.7" + /> + <polygon + points={burst(90, 36, 7, 3.5, 8)} + fill="#facc15" + stroke={INK} + strokeWidth="0.7" + strokeLinejoin="round" + /> + <circle + cx="92" + cy="49" + r="4.5" + fill="#fde68a" + stroke={INK} + strokeWidth="0.7" + /> + <path + d="M90 50 Q92 52 94 50" + stroke={INK} + strokeWidth="0.7" + strokeLinecap="round" + /> + </PictureArt> + ) +} + +/** Four equal panels stacked top to bottom, its title down the side. */ +export function YonkomaIcon(props: ArtProps) { + const fills = ["#fde68a", "#bbf7d0", "#bfdbfe", "#fecaca"] + return ( + <PictureArt from="#ede9fe" to="#e0e7ff" {...props}> + <Ground rx={20} /> + <Sheet x="45" y="5" width="30" height="67" /> + {fills.map((fill, i) => ( + <Panel + key={fill} + x="48.5" + y={8.5 + i * 15.6} + width="23" + height="13.6" + fill={fill} + /> + ))} + <circle cx="56" cy="15" r="2.6" fill="#fb923c" /> + <circle cx="62" cy="31" r="2.6" fill="#22c55e" /> + <circle cx="58" cy="46" r="2.6" fill="#3b82f6" /> + <polygon points={burst(60, 62, 4, 2, 6)} fill="#ef4444" /> + <rect + x="78" + y="9" + width="3.5" + height="26" + rx="1.75" + fill={INK} + opacity="0.75" + /> + <rect x="78" y="38" width="3.5" height="10" rx="1.75" fill="#a78bfa" /> + </PictureArt> + ) +} + +/** An open book: a sunny picture on one page, its words on the other. */ +export function PictureBookIcon(props: ArtProps) { + return ( + <PictureArt from="#dcfce7" to="#ecfccb" {...props}> + <Ground rx={42} /> + <path + d="M60 66 C50 61 33 61 18 64 V16 C33 13 50 13 60 18 C70 13 87 13 102 16 V64 C87 61 70 61 60 66 Z" + fill="#f97316" + /> + <path + d="M60 62 C50 57 34 57 21 60 V14 C34 11 50 11 60 16 Z" + fill="#ffffff" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1" + strokeLinejoin="round" + /> + <path + d="M60 62 C70 57 86 57 99 60 V14 C86 11 70 11 60 16 Z" + fill="#ffffff" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1" + strokeLinejoin="round" + /> + <path + d="M24 50 C33 44 46 46 57 51 V57 C46 54 34 54 24 56 Z" + fill="#86efac" + /> + <circle cx="47" cy="24" r="4.5" fill="#fbbf24" /> + <rect x="31" y="38" width="2.6" height="12" rx="1" fill="#a16207" /> + <circle cx="32.3" cy="35" r="6" fill="#22c55e" /> + <circle cx="29" cy="33" r="1" fill="#f472b6" /> + <path + d="M66 22 H92 M66 28 H90 M66 34 H93 M66 40 H86 M66 46 H91" + stroke="#cbd5e1" + strokeWidth="2" + strokeLinecap="round" + /> + <path + d="M85 53 C85 51 88 51 88 53 C88 51 91 51 91 53 C91 55.5 88 57 88 57 C88 57 85 55.5 85 53 Z" + fill="#f472b6" + /> + </PictureArt> + ) +} + +/** A sheet of rough frames, each a shot, with arrows and a camera. */ +export function StoryboardIcon(props: ArtProps) { + const frames: [number, number][] = [ + [29, 13], + [62, 13], + [29, 41], + [62, 41], + ] + return ( + <PictureArt from="#e2e8f0" to="#dbeafe" {...props}> + <Ground rx={38} /> + <Sheet x="24" y="8" width="72" height="62" /> + {frames.map(([x, y]) => ( + <g key={`${x}-${y}`}> + <rect + x={x} + y={y} + width="29" + height="18" + rx="1" + fill="#f1f5f9" + stroke={INK} + strokeOpacity="0.7" + strokeWidth="0.8" + /> + <path + d={`M${x + 3} ${y + 15} C${x + 9} ${y + 9} ${x + 15} ${y + 13} ${x + 26} ${y + 7}`} + stroke="#64748b" + strokeWidth="0.8" + strokeLinecap="round" + /> + <circle + cx={x + 10} + cy={y + 7} + r="2.6" + stroke="#64748b" + strokeWidth="0.8" + /> + <rect x={x} y={y + 20} width="18" height="2" rx="1" fill="#cbd5e1" /> + </g> + ))} + <path + d="M58.5 22 H61.5 M59.8 20.5 L61.5 22 L59.8 23.5" + stroke="#3b82f6" + strokeWidth="1.1" + strokeLinecap="round" + strokeLinejoin="round" + /> + <path + d="M76 33.5 V39 M74.5 37.5 L76 39 L77.5 37.5" + stroke="#3b82f6" + strokeWidth="1.1" + strokeLinecap="round" + strokeLinejoin="round" + /> + <rect x="88" y="4" width="20" height="13" rx="3" fill={INK} /> + <rect x="92" y="2.5" width="7" height="3" rx="1" fill={INK} /> + <circle + cx="98" + cy="10.5" + r="4.2" + fill="#60a5fa" + stroke="#ffffff" + strokeWidth="1" + /> + <circle cx="98" cy="10.5" r="1.6" fill="#1e3a8a" /> + </PictureArt> + ) +} diff --git a/surfsense_local/frontend/src/components/ui/icons.tsx b/surfsense_local/frontend/src/components/ui/icons.tsx index e953111ad0..2cf931de01 100644 --- a/surfsense_local/frontend/src/components/ui/icons.tsx +++ b/surfsense_local/frontend/src/components/ui/icons.tsx @@ -28,6 +28,8 @@ import { CheckmarkBadge02Icon as CheckmarkBadge02IconData, CheckmarkCircle02Icon, ChevronDownIcon as ChevronDownIconData, + CanvasIcon as CanvasIconData, + ChevronLeftIcon as ChevronLeftIconData, ChevronRightIcon as ChevronRightIconData, Clock01Icon as Clock01IconData, CircleArrowUp02Icon as CircleArrowUp02IconData, @@ -166,6 +168,8 @@ export const CheckCircle2Icon = createIcon(CheckmarkCircle02Icon) export const CheckIcon = createIcon(CheckIconData) export const CheckmarkBadgeIcon = createIcon(CheckmarkBadge02IconData) export const ChevronDownIcon = createIcon(ChevronDownIconData) +export const CanvasIcon = createIcon(CanvasIconData) +export const ChevronLeftIcon = createIcon(ChevronLeftIconData) export const ChevronRightIcon = createIcon(ChevronRightIconData) export const ClockIcon = createIcon(Clock01IconData) export const CircleArrowUp02Icon = createIcon(CircleArrowUp02IconData) @@ -256,3 +260,19 @@ export const Xls01Icon = createIcon(Xls01IconData) export const YoutubeIcon = createIcon(YoutubeIconData) export const ZoomInIcon = createIcon(ZoomInIconData) export const ZoomOutIcon = createIcon(ZoomOutIconData) + +// Studio's picture types, drawn for their cards rather than taken from the icon set. +export { ImageTypeIcon, InfographicTypeIcon } from "./picture-type-icons" +// The kinds of comic Studio draws, drawn for their cards in the same way. +export { + BandeDessineeIcon, + ComicStripIcon, + GraphicNovelIcon, + MangaIcon, + ManhuaIcon, + PictureBookIcon, + StoryboardIcon, + WebtoonIcon, + WesternComicIcon, + YonkomaIcon, +} from "./comic-kind-icons" diff --git a/surfsense_local/frontend/src/components/ui/picture-art.tsx b/surfsense_local/frontend/src/components/ui/picture-art.tsx new file mode 100644 index 0000000000..1e031a41b2 --- /dev/null +++ b/surfsense_local/frontend/src/components/ui/picture-art.tsx @@ -0,0 +1,123 @@ +import { type ReactNode, type SVGProps, useId } from "react" + +import { cn } from "@/lib/utils" + +// The frame every Studio card picture shares: a 3:2 canvas on its own +// gradient backdrop, so it reads the same in light and dark. +export const INK = "#1f2a44" + +export function PictureArt({ + from, + to, + className, + children, + ...props +}: Omit<SVGProps<SVGSVGElement>, "from" | "to"> & { + from: string + to: string + children: ReactNode +}) { + const backdrop = `${useId()}-backdrop` + return ( + <svg + viewBox="0 0 120 80" + fill="none" + aria-hidden="true" + data-slot="picture-type-art" + className={cn("size-full", className)} + {...props} + > + <defs> + <linearGradient id={backdrop} x1="0" y1="0" x2="1" y2="1"> + <stop offset="0" stopColor={from} /> + <stop offset="1" stopColor={to} /> + </linearGradient> + </defs> + <rect width="120" height="80" fill={`url(#${backdrop})`} /> + <circle cx="12" cy="10" r="16" fill="#ffffff" opacity="0.45" /> + <circle cx="112" cy="74" r="20" fill="#ffffff" opacity="0.35" /> + {children} + </svg> + ) +} + +/** The soft shadow a picture's object stands on. */ +export function Ground({ cx = 60, rx = 32 }: { cx?: number; rx?: number }) { + return <ellipse cx={cx} cy="74" rx={rx} ry="3" fill="#000" opacity="0.12" /> +} + +/** A white sheet with the ink edge every printed object has. */ +export function Sheet(props: SVGProps<SVGRectElement>) { + return ( + <rect + rx="3" + fill="#ffffff" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1.1" + {...props} + /> + ) +} + +/** A comic panel: a flat colour inside a thin ink border. */ +export function Panel(props: SVGProps<SVGRectElement>) { + return ( + <rect + rx="1" + stroke={INK} + strokeOpacity="0.7" + strokeWidth="0.8" + {...props} + /> + ) +} + +/** A speech bubble with its tail pointing down and left. */ +export function Bubble({ + cx, + cy, + rx = 6, + ry = 3.5, +}: { + cx: number + cy: number + rx?: number + ry?: number +}) { + return ( + <g> + <path + d={`M${cx - rx * 0.4} ${cy + ry * 0.7} L${cx - rx * 0.75} ${cy + ry + 2.5} L${cx - rx * 0.05} ${cy + ry * 0.85} Z`} + fill="#ffffff" + stroke={INK} + strokeWidth="0.7" + strokeLinejoin="round" + /> + <ellipse + cx={cx} + cy={cy} + rx={rx} + ry={ry} + fill="#ffffff" + stroke={INK} + strokeWidth="0.7" + /> + </g> + ) +} + +/** A star burst's points, as an SVG polygon takes them. */ +export function burst( + cx: number, + cy: number, + outer: number, + inner: number, + points = 10 +): string { + return Array.from({ length: points * 2 }, (_, i) => { + const radius = i % 2 === 0 ? outer : inner + const angle = (Math.PI * i) / points - Math.PI / 2 + return `${(cx + radius * Math.cos(angle)).toFixed(1)},${(cy + radius * Math.sin(angle)).toFixed(1)}` + }).join(" ") +} diff --git a/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx b/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx new file mode 100644 index 0000000000..92fd91798b --- /dev/null +++ b/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx @@ -0,0 +1,215 @@ +import { type SVGProps, useId } from "react" + +import { cn } from "@/lib/utils" + +// Studio's picture types as small pictures for their cards. They carry their +// own colours and backdrop, as a thumbnail does, so they read the same in light and dark. +// 3:2, the card's thumbnail. +const VIEW_BOX = "0 0 120 80" +const INK = "#1f2a44" + +/** A photo print over another: a landscape under a sun, and sparks for AI. */ +export function ImageTypeIcon({ + className, + ...props +}: SVGProps<SVGSVGElement>) { + const id = useId() + const photo = `${id}-photo` + const sky = `${id}-sky` + const backdrop = `${id}-backdrop` + return ( + <svg + viewBox={VIEW_BOX} + fill="none" + aria-hidden="true" + data-slot="picture-type-art" + className={cn("size-full", className)} + {...props} + > + <defs> + <clipPath id={photo}> + <rect x="33" y="19" width="56" height="38" rx="2.5" /> + </clipPath> + <linearGradient id={backdrop} x1="0" y1="0" x2="1" y2="1"> + <stop offset="0" stopColor="#ffe3ee" /> + <stop offset="1" stopColor="#e2e6ff" /> + </linearGradient> + <linearGradient id={sky} x1="0" y1="0" x2="0" y2="1"> + <stop offset="0" stopColor="#9fd3ff" /> + <stop offset="1" stopColor="#e3f3ff" /> + </linearGradient> + </defs> + <rect width="120" height="80" fill={`url(#${backdrop})`} /> + <circle cx="12" cy="10" r="16" fill="#ffffff" opacity="0.45" /> + <circle cx="112" cy="74" r="20" fill="#ffffff" opacity="0.35" /> + <ellipse cx="61" cy="73" rx="34" ry="3" fill="#000" opacity="0.12" /> + <rect + x="30" + y="11" + width="62" + height="52" + rx="4" + transform="rotate(-9 61 37)" + fill="#ffd3e2" + stroke={INK} + strokeOpacity="0.35" + strokeWidth="1" + /> + <g transform="rotate(4 61 41)"> + <rect + x="28" + y="14" + width="66" + height="56" + rx="4" + fill="#ffffff" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1.1" + /> + <g clipPath={`url(#${photo})`}> + <rect x="33" y="19" width="56" height="38" fill={`url(#${sky})`} /> + <circle cx="75" cy="29" r="5.5" fill="#ffc83d" /> + <path + d="M54 27.5 a3 3 0 0 1 5.6 -1.2 a2.4 2.4 0 0 1 3.4 2.2 h-9 Z" + fill="#ffffff" + /> + <path + d="M33 50 L48 32 L58 43 L67 35 L89 53 V57 H33 Z" + fill="#7c8cf8" + /> + <path + d="M44.6 36.9 L48 32 L51.6 36.3 L49.6 35.3 L47.4 37.6 Z" + fill="#ffffff" + /> + <path d="M60 41 L67 35 L89 53 V57 H60 Z" fill="#5664e0" /> + <path + d="M33 57 V50 C45 44 56 47 64 50 C74 54 82 49 89 47 V57 Z" + fill="#4cc38a" + /> + <path + d="M33 57 V53 C44 50 52 52 60 55 C66 57 72 55 76 57 Z" + fill="#2fa56e" + /> + </g> + <rect x="33" y="62" width="22" height="2.6" rx="1.3" fill="#e4e7ee" /> + </g> + <path + d="M98 12 C98.6 15.8 99.8 17 103.5 17.5 C99.8 18 98.6 19.2 98 23 C97.4 19.2 96.2 18 92.5 17.5 C96.2 17 97.4 15.8 98 12 Z" + fill="#ff6fa8" + /> + <path + d="M22 50 C22.3 52 23 52.7 25 53 C23 53.3 22.3 54 22 56 C21.7 54 21 53.3 19 53 C21 52.7 21.7 52 22 50 Z" + fill="#ffc83d" + /> + <circle cx="104" cy="28" r="1.4" fill="#7c8cf8" /> + </svg> + ) +} + +/** A poster of figures: a title, a donut with its legend, and a bar chart. */ +export function InfographicTypeIcon({ + className, + ...props +}: SVGProps<SVGSVGElement>) { + const id = useId() + const sheet = `${id}-sheet` + const header = `${id}-header` + const backdrop = `${id}-backdrop` + return ( + <svg + viewBox={VIEW_BOX} + fill="none" + aria-hidden="true" + data-slot="picture-type-art" + className={cn("size-full", className)} + {...props} + > + <defs> + <clipPath id={sheet}> + <rect x="22" y="8" width="76" height="62" rx="5" /> + </clipPath> + <linearGradient id={backdrop} x1="0" y1="0" x2="1" y2="1"> + <stop offset="0" stopColor="#ddf7ea" /> + <stop offset="1" stopColor="#dcebff" /> + </linearGradient> + <linearGradient id={header} x1="0" y1="0" x2="1" y2="0"> + <stop offset="0" stopColor="#8b5cf6" /> + <stop offset="1" stopColor="#3b82f6" /> + </linearGradient> + </defs> + <rect width="120" height="80" fill={`url(#${backdrop})`} /> + <circle cx="108" cy="12" r="18" fill="#ffffff" opacity="0.45" /> + <circle cx="10" cy="70" r="16" fill="#ffffff" opacity="0.35" /> + <ellipse cx="60" cy="74" rx="36" ry="3" fill="#000" opacity="0.12" /> + <rect x="22" y="8" width="76" height="62" rx="5" fill="#ffffff" /> + <rect + x="22" + y="8" + width="76" + height="7" + clipPath={`url(#${sheet})`} + fill={`url(#${header})`} + /> + <rect + x="22" + y="8" + width="76" + height="62" + rx="5" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1.1" + /> + <rect x="30" y="20" width="32" height="4" rx="2" fill={INK} /> + <rect x="30" y="27" width="22" height="2.5" rx="1.25" fill="#a3acc2" /> + <circle cx="42" cy="49" r="9" strokeWidth="5" stroke="#e8ebf2" /> + <circle + cx="42" + cy="49" + r="9" + strokeWidth="5" + strokeDasharray="24 56.5" + transform="rotate(-90 42 49)" + stroke="#ff7a59" + /> + <circle + cx="42" + cy="49" + r="9" + strokeWidth="5" + strokeDasharray="17 56.5" + strokeDashoffset="-24" + transform="rotate(-90 42 49)" + stroke="#ffc83d" + /> + <circle + cx="42" + cy="49" + r="9" + strokeWidth="5" + strokeDasharray="15.5 56.5" + strokeDashoffset="-41" + transform="rotate(-90 42 49)" + stroke="#3b82f6" + /> + <circle cx="72" cy="21" r="2" fill="#ff7a59" /> + <rect x="76" y="20" width="14" height="2" rx="1" fill="#e4e7ee" /> + <circle cx="72" cy="27" r="2" fill="#3b82f6" /> + <rect x="76" y="26" width="10" height="2" rx="1" fill="#e4e7ee" /> + <rect x="60" y="49" width="5" height="12" rx="1" fill="#3b82f6" /> + <rect x="67.5" y="43" width="5" height="18" rx="1" fill="#22c55e" /> + <rect x="75" y="46" width="5" height="15" rx="1" fill="#ffc83d" /> + <rect x="82.5" y="37" width="5" height="24" rx="1" fill="#ff7a59" /> + <path d="M58 61.5 H90" stroke={INK} strokeOpacity="0.4" strokeWidth="1" /> + <path + d="M60 40 L67 35 L74 38 L84 31" + stroke="#8b5cf6" + strokeWidth="1.2" + strokeLinecap="round" + strokeLinejoin="round" + strokeDasharray="2 2" + /> + </svg> + ) +} diff --git a/surfsense_local/frontend/src/components/ui/scroll-fade.tsx b/surfsense_local/frontend/src/components/ui/scroll-fade.tsx index 7e8abca926..504c923670 100644 --- a/surfsense_local/frontend/src/components/ui/scroll-fade.tsx +++ b/surfsense_local/frontend/src/components/ui/scroll-fade.tsx @@ -9,6 +9,7 @@ export function ScrollFade({ className, viewportClassName, scroll = true, + orientation = "vertical", ...viewportProps }: { children: ReactNode @@ -18,6 +19,8 @@ export function ScrollFade({ // dialog section that scrolls its heading and body together). Renders // children at their natural height instead of a clipped, scrolling box. scroll?: boolean + // Horizontal fades the start and end edges, for a row that scrolls sideways. + orientation?: "vertical" | "horizontal" } & Omit<ComponentProps<"div">, "children" | "className">) { // The rest, `ref` included, land on the viewport: it is the element that // scrolls, so it is the one a caller measures, labels or listens to. @@ -35,7 +38,9 @@ export function ScrollFade({ {...viewportProps} data-slot="scroll-fade-viewport" className={cn( - "h-full min-h-0 scroll-fade overflow-y-auto overscroll-contain [--scroll-fade-reveal:24px] scroll-fade-6", + orientation === "horizontal" + ? "scroll-fade-x overflow-x-auto overscroll-x-contain [--scroll-fade-reveal:24px] scroll-fade-6" + : "h-full min-h-0 scroll-fade overflow-y-auto overscroll-contain [--scroll-fade-reveal:24px] scroll-fade-6", viewportClassName )} > diff --git a/surfsense_local/frontend/src/features/agent/agent-steps.test.tsx b/surfsense_local/frontend/src/features/agent/agent-steps.test.tsx index bd8100dfc7..3c5eae02d3 100644 --- a/surfsense_local/frontend/src/features/agent/agent-steps.test.tsx +++ b/surfsense_local/frontend/src/features/agent/agent-steps.test.tsx @@ -232,6 +232,50 @@ describe("agent steps", () => { expect(screen.getByText("Asked for a picture")).toBeTruthy() }) + it("opens the comic a bind_comic step made", async () => { + const openArtifact = vi.fn() + const user = userEvent.setup() + render( + <OpenArtifactContext.Provider value={openArtifact}> + <AgentSteps + steps={[ + rendered({ + tool: "surfsense_bind_comic", + input: { title: "The Lighthouse Cat", pages: [] }, + artifact: { id: 70, title: "The Lighthouse Cat", version: 2 }, + }), + ]} + /> + </OpenArtifactContext.Provider> + ) + + await user.click( + screen.getByRole("button", { + name: "Updated The Lighthouse Cat to v2", + }) + ) + expect(openArtifact).toHaveBeenCalledExactlyOnceWith(70) + }) + + it("names a comic still waiting for its pages by its title", () => { + render( + <AgentSteps + steps={[ + rendered({ + tool: "surfsense_bind_comic", + input: { title: "The Lighthouse Cat", pages: [] }, + artifact: undefined, + output: "Queued artifact 70, version 1: The Lighthouse Cat", + }), + ]} + /> + ) + + expect(screen.getByText(/Binding the comic/).textContent).toBe( + "Binding the comic The Lighthouse Cat" + ) + }) + it("names reading a document’s script and listing source images", () => { render( <AgentSteps diff --git a/surfsense_local/frontend/src/features/agent/step-label.tsx b/surfsense_local/frontend/src/features/agent/step-label.tsx index 90674b4025..33d8f405ff 100644 --- a/surfsense_local/frontend/src/features/agent/step-label.tsx +++ b/surfsense_local/frontend/src/features/agent/step-label.tsx @@ -224,6 +224,41 @@ export function stepLabel( defaultMessage: "Asked for a picture", }) } + case "surfsense_bind_comic": { + const made = step.artifact + if (made) { + return (made.created ?? made.version === 1) + ? intl.formatMessage( + { + id: "agent_steps_bind_comic_created_label", + defaultMessage: "Created <doc>{title}</doc> v{version, number}", + }, + { title: made.title, version: made.version, doc: documentTitle } + ) + : intl.formatMessage( + { + id: "agent_steps_bind_comic_updated_label", + defaultMessage: + "Updated <doc>{title}</doc> to v{version, number}", + }, + { title: made.title, version: made.version, doc: documentTitle } + ) + } + // Running, waiting for its pages, refused or failed: the title it was given. + const title = text(input.title) + return title + ? intl.formatMessage( + { + id: "agent_steps_bind_comic_label", + defaultMessage: "Binding the comic <doc>{title}</doc>", + }, + { title, doc: documentTitle } + ) + : intl.formatMessage({ + id: "agent_steps_bind_comic_unnamed_label", + defaultMessage: "Binding a comic", + }) + } case "surfsense_convert_document": { const made = step.artifact return made diff --git a/surfsense_local/frontend/src/features/models/pictures-in/use-pictures-in.ts b/surfsense_local/frontend/src/features/models/pictures-in/use-pictures-in.ts new file mode 100644 index 0000000000..3d18876c76 --- /dev/null +++ b/surfsense_local/frontend/src/features/models/pictures-in/use-pictures-in.ts @@ -0,0 +1,19 @@ +import { useQuery } from "@tanstack/react-query" + +import { requestJson } from "@/lib/api" + +import { MODELS_QUERY_KEY } from "../models-query" + +/** Which chosen model takes a picture in. Null: a change to a picture is drawn + * again from its prompt, by the image model. */ +export type PicturesIn = { source: "edit_model" | "image_model" | null } + +/** Under the models key, so choosing a model reads it again. */ +export function usePicturesIn({ enabled = true }: { enabled?: boolean } = {}) { + return useQuery({ + enabled, + queryKey: [...MODELS_QUERY_KEY, "pictures-in"], + queryFn: ({ signal }) => + requestJson<PicturesIn>("/llm/pictures-in", { signal }), + }) +} diff --git a/surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx b/surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx index 5ef1a9fbc6..a8343ef82e 100644 --- a/surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx +++ b/surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx @@ -47,7 +47,7 @@ export function InUseSummary({ { id: "models_in_use_summary_empty", defaultMessage: - "{slot, select, chat {No chat model chosen} image {No image model chosen} image_edit {No image editing model chosen} video {No video model chosen} audio {No audio model chosen} speech {No speech recognition model chosen} other {No model chosen}}", + "{slot, select, chat {No chat model chosen} image {No image model chosen} image_edit {Same as the image model} video {No video model chosen} audio {No audio model chosen} speech {No speech recognition model chosen} other {No model chosen}}", }, { slot } )} diff --git a/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.test.tsx b/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.test.tsx index 73ef417b50..fe3964f1fc 100644 --- a/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.test.tsx +++ b/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.test.tsx @@ -122,7 +122,9 @@ describe("image editing model settings", () => { render(<ImageEditModelsSettings onModelUnavailable={() => undefined} />) expect( - await screen.findByRole("heading", { name: "Image editing models" }) + await screen.findByRole("heading", { + name: "Use a different model for edits", + }) ).toBeTruthy() await user.click( await screen.findByRole("button", { name: "Use FLUX.2 klein 4B" }) diff --git a/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx b/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx index 40217d2efe..b2bef70eb4 100644 --- a/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx +++ b/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx @@ -25,12 +25,12 @@ export function ImageEditModelsSettings({ <ModelSlotSettings title={intl.formatMessage({ id: "settings_image_edit_models_title", - defaultMessage: "Image editing models", + defaultMessage: "Use a different model for edits", })} description={intl.formatMessage({ id: "settings_image_edit_models_body", defaultMessage: - "The model that edits images. Run one on this computer, or use one from a server.", + "Optional. Your image model edits pictures when it can, and draws them again when it cannot. Choose a model here only to edit with a different one.", })} slot="image_edit" modelType="image_edit" diff --git a/surfsense_local/frontend/src/features/studio/api.ts b/surfsense_local/frontend/src/features/studio/api.ts index 713a1cd012..96de9ce28d 100644 --- a/surfsense_local/frontend/src/features/studio/api.ts +++ b/surfsense_local/frontend/src/features/studio/api.ts @@ -111,7 +111,13 @@ export type StudioJobCreate = { // What the server resolves into the job's sources, folders included. source_scope?: SourceScope prompt?: string - options?: PodcastBrief + options?: PodcastBrief | ComicOptions +} + +/** A comic job's options, checked by the server against its kinds (modules/artifacts/comic/). */ +export type ComicOptions = { + kind: string + pages: number } export const PODCAST_STYLES = [ diff --git a/surfsense_local/frontend/src/features/studio/artifact-panel.test.tsx b/surfsense_local/frontend/src/features/studio/artifact-panel.test.tsx index 62b11384a3..668748b3d6 100644 --- a/surfsense_local/frontend/src/features/studio/artifact-panel.test.tsx +++ b/surfsense_local/frontend/src/features/studio/artifact-panel.test.tsx @@ -909,12 +909,12 @@ describe("artifact panel image edit", () => { expect(onRefine).not.toHaveBeenCalled() }) - it("says an image editing model is needed, and links to its settings", async () => { + it("says an image model is needed when none is chosen, and links to it", async () => { const v1 = picture() vi.stubGlobal( "fetch", vi.fn(async (url: string) => - String(url).includes("/llm/selection/image_edit") + String(url).includes("/llm/selection/image_gen") ? Response.json({ detail: "no selection" }, { status: 404 }) : Response.json({ ...v1, @@ -945,9 +945,7 @@ describe("artifact panel image edit", () => { await user.click( await screen.findByRole("button", { name: "Edit this image" }) ) - expect( - await screen.findByText("Needs an image editing model.") - ).toBeTruthy() + expect(await screen.findByText("Needs an image model.")).toBeTruthy() expect( screen .getByRole("textbox", { name: "How to change this image" }) @@ -955,24 +953,23 @@ describe("artifact panel image edit", () => { ).toBe(true) await user.click(screen.getByRole("button", { name: "Choose a model" })) - expect(onOpenModelSettings).toHaveBeenCalledExactlyOnceWith( - "image-edit-models" - ) + // Never the editing models: every image model can change a picture. + expect(onOpenModelSettings).toHaveBeenCalledExactlyOnceWith("image-models") expect(onImageEdit).not.toHaveBeenCalled() }) - it("holds the instruction until it is known whether the edit model is chosen", async () => { + it("holds the instruction until it is known whether an image model is chosen", async () => { const v1 = picture() let answer = () => {} vi.stubGlobal( "fetch", vi.fn((url: string) => - String(url).includes("/llm/selection/image_edit") + String(url).includes("/llm/selection/image_gen") ? new Promise<Response>((resolve) => { answer = () => resolve( Response.json({ - model_type: "image_edit", + model_type: "image_gen", provider: "sdcpp", connection_id: null, name: "flux-2-klein-4b-Q4_0", @@ -1002,13 +999,52 @@ describe("artifact panel image edit", () => { name: "How to change this image", }) expect(field.hasAttribute("disabled")).toBe(true) - expect(screen.queryByText("Needs an image editing model.")).toBeNull() + expect(screen.queryByText("Needs an image model.")).toBeNull() answer() await vi.waitFor(() => expect(field.hasAttribute("disabled")).toBe(false)) await vi.waitFor(() => expect(document.activeElement).toBe(field)) }) + it("says a change is drawn again when the image model takes no picture in", async () => { + const v1 = picture() + vi.stubGlobal( + "fetch", + vi.fn(async (url: string) => + String(url).includes("/llm/pictures-in") + ? Response.json({ source: null }) + : Response.json({ + ...v1, + format: "summary", + content: "A lighthouse at dusk", + files: [], + quiz_state: null, + flashcard_state: null, + }) + ) + ) + const onImageEdit = vi.fn(async () => {}) + const user = userEvent.setup() + + render(panel([v1], vi.fn(), onImageEdit)) + await user.click( + await screen.findByRole("button", { name: "Edit this image" }) + ) + const field = screen.getByRole("textbox", { + name: "How to change this image", + }) + + // Plain, not a warning: the edit still works, as a redraw. + await vi.waitFor(() => + expect(field.getAttribute("placeholder")).toBe( + "Describe a change: the picture is drawn again" + ) + ) + expect(screen.queryByRole("button", { name: "Choose a model" })).toBeNull() + await user.type(field, "Make it night{Enter}") + expect(onImageEdit).toHaveBeenCalledExactlyOnceWith(40, "Make it night") + }) + it("offers no edit on a picture that is not ready", async () => { const failed = picture({ status: "failed", image_editable: false }) serveDetail(failed) diff --git a/surfsense_local/frontend/src/features/studio/edit-model.ts b/surfsense_local/frontend/src/features/studio/edit-model.ts index ea4b793a7c..cf36d5335d 100644 --- a/surfsense_local/frontend/src/features/studio/edit-model.ts +++ b/surfsense_local/frontend/src/features/studio/edit-model.ts @@ -3,20 +3,21 @@ import { useSelection } from "@/features/models/selection/use-selection" /** What the Refine box changes: a document's spec, or a picture. */ export type RefineSubject = "document" | "image" -/** Where Settings chooses the model each subject's edit needs. */ -export type EditModelSection = "chat-models" | "image-edit-models" +/** Where Settings chooses the model each subject's edit needs. Any image model + * changes a picture, by editing it or drawing it again, so a picture needs one. */ +export type EditModelSection = "chat-models" | "image-models" -const SLOT = { document: "text_gen", image: "image_edit" } as const +const SLOT = { document: "text_gen", image: "image_gen" } as const export const EDIT_MODEL_SECTION = { document: "chat-models", - image: "image-edit-models", + image: "image-models", } as const satisfies Record<RefineSubject, EditModelSection> // The codes the API refuses with when that slot is empty. export const EDIT_MODEL_CODE = { document: "needs_chat", - image: "needs_image_edit", + image: "needs_image", } as const satisfies Record<RefineSubject, string> /** Whether the model this subject's edit needs is chosen; null while that is diff --git a/surfsense_local/frontend/src/features/studio/format-options/comic-kind-picker.tsx b/surfsense_local/frontend/src/features/studio/format-options/comic-kind-picker.tsx new file mode 100644 index 0000000000..ecacf58d76 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/comic-kind-picker.tsx @@ -0,0 +1,170 @@ +import { + type ComponentType, + useCallback, + useEffect, + useRef, + useState, +} from "react" + +import { Button } from "@/components/ui/button" +import { + BandeDessineeIcon, + ChevronLeftIcon, + ChevronRightIcon, + ComicStripIcon, + GraphicNovelIcon, + MangaIcon, + ManhuaIcon, + PictureBookIcon, + StoryboardIcon, + WebtoonIcon, + WesternComicIcon, + YonkomaIcon, +} from "@/components/ui/icons" +import { ScrollFade } from "@/components/ui/scroll-fade" +import { intl } from "@/i18n/intl" +import { cn } from "@/lib/utils" + +import { COMIC_KINDS, type ComicKind } from "./comic-kinds" +import { PictureCard } from "./picture-card" + +const KIND_ART: Record<ComicKind, ComponentType<{ className?: string }>> = { + western: WesternComicIcon, + graphic_novel: GraphicNovelIcon, + bande_dessinee: BandeDessineeIcon, + manga: MangaIcon, + manhua: ManhuaIcon, + webtoon: WebtoonIcon, + comic_strip: ComicStripIcon, + yonkoma: YonkomaIcon, + picture_book: PictureBookIcon, + storyboard: StoryboardIcon, +} + +/** Every kind of comic as a card in one row that scrolls sideways; an arrow + * shows on a side only while more kinds lie that way. */ +export function ComicKindPicker({ + value, + onChange, +}: { + value: ComicKind + onChange: (kind: ComicKind) => void +}) { + const row = useRef<HTMLDivElement>(null) + const [more, setMore] = useState({ back: false, ahead: false }) + + const measure = useCallback(() => { + const element = row.current + if (!element) return + setMore({ + // A few pixels of slack: the snap settles within the row's padding. + back: element.scrollLeft > 4, + ahead: element.scrollLeft + element.clientWidth < element.scrollWidth - 4, + }) + }, []) + + useEffect(() => { + measure() + const element = row.current + if (!element || typeof ResizeObserver === "undefined") return + const observer = new ResizeObserver(measure) + observer.observe(element) + return () => observer.disconnect() + }, [measure]) + + const page = (direction: 1 | -1) => { + const element = row.current + if (!element) return + const still = window.matchMedia?.( + "(prefers-reduced-motion: reduce)" + ).matches + element.scrollBy({ + left: direction * element.clientWidth * 0.8, + behavior: still ? "auto" : "smooth", + }) + } + + const label = intl.formatMessage({ + id: "studio_comic_kind_label", + defaultMessage: "Kind", + }) + return ( + <div className="space-y-2"> + <p className="text-xs font-medium text-muted-foreground">{label}</p> + {/* Sized by the dialog, not by its ten cards: a grid or flex parent would + otherwise grow to fit the whole row instead of letting it scroll. */} + <div className="relative [contain:inline-size]"> + <ScrollFade + orientation="horizontal" + ref={row} + role="radiogroup" + aria-label={label} + onScroll={measure} + viewportClassName="flex snap-x scroll-px-0.5 [scrollbar-width:none] gap-2 p-0.5 [&::-webkit-scrollbar]:hidden" + > + {COMIC_KINDS.map((kind) => ( + <PictureCard + key={kind.key} + name="studio-comic-kind" + value={kind.key} + label={kind.label()} + Art={KIND_ART[kind.key]} + checked={value === kind.key} + className="w-32 shrink-0 snap-start" + onChange={(event) => { + onChange(kind.key) + // A kind reached by the arrow keys comes into view. + event.currentTarget + .closest("label") + ?.scrollIntoView?.({ block: "nearest", inline: "nearest" }) + }} + /> + ))} + </ScrollFade> + <ScrollArrow side="back" shown={more.back} onClick={() => page(-1)} /> + <ScrollArrow side="ahead" shown={more.ahead} onClick={() => page(1)} /> + </div> + </div> + ) +} + +function ScrollArrow({ + side, + shown, + onClick, +}: { + side: "back" | "ahead" + shown: boolean + onClick: () => void +}) { + if (!shown) return null + const Icon = side === "back" ? ChevronLeftIcon : ChevronRightIcon + return ( + <Button + type="button" + variant="default" + size="icon-sm" + // Pointer only: the arrow keys already move through the kinds. + tabIndex={-1} + aria-label={ + side === "back" + ? intl.formatMessage({ + id: "studio_comic_kind_scroll_back_aria", + defaultMessage: "Show earlier kinds", + }) + : intl.formatMessage({ + id: "studio_comic_kind_scroll_ahead_aria", + defaultMessage: "Show more kinds", + }) + } + onClick={onClick} + className={cn( + // Centred on the cards' pictures, which are 2/3 of a 128 px card wide. + "absolute top-[calc(128px/3-12px)] rounded-full shadow-md", + side === "back" ? "-left-2" : "-right-2" + )} + > + <Icon /> + </Button> + ) +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/comic-kinds.ts b/surfsense_local/frontend/src/features/studio/format-options/comic-kinds.ts new file mode 100644 index 0000000000..98da70b7e4 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/comic-kinds.ts @@ -0,0 +1,105 @@ +import { intl } from "@/i18n/intl" + +// The server's kinds (backend/modules/artifacts/comic/kinds.py), key for key, +// each with the length it starts at. +export const COMIC_KINDS = [ + { + key: "western", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_western_label", + defaultMessage: "Western comic", + }), + }, + { + key: "graphic_novel", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_graphic_novel_label", + defaultMessage: "Graphic novel", + }), + }, + { + key: "bande_dessinee", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_bande_dessinee_label", + defaultMessage: "Bande dessinée", + }), + }, + { + key: "manga", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_manga_label", + defaultMessage: "Manga", + }), + }, + { + key: "manhua", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_manhua_label", + defaultMessage: "Manhua", + }), + }, + { + key: "webtoon", + pages: 8, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_webtoon_label", + defaultMessage: "Webtoon", + }), + }, + { + key: "comic_strip", + pages: 1, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_comic_strip_label", + defaultMessage: "Comic strip", + }), + }, + { + key: "yonkoma", + pages: 1, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_yonkoma_label", + defaultMessage: "Yonkoma (4-koma)", + }), + }, + { + key: "picture_book", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_picture_book_label", + defaultMessage: "Picture book", + }), + }, + { + key: "storyboard", + pages: 6, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_storyboard_label", + defaultMessage: "Storyboard", + }), + }, +] as const + +export type ComicKind = (typeof COMIC_KINDS)[number]["key"] + +// Each picture is billed or takes minutes, so a comic stops here. +export const MAX_COMIC_PAGES = 12 + +export function startingPages(kind: ComicKind): number { + return COMIC_KINDS.find((entry) => entry.key === kind)?.pages ?? 4 +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx b/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx new file mode 100644 index 0000000000..8b759f6b8e --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx @@ -0,0 +1,65 @@ +import { useId } from "react" + +import { Select } from "@/components/ui/select" +import { intl } from "@/i18n/intl" + +import { ComicKindPicker } from "./comic-kind-picker" +import { type ComicKind, MAX_COMIC_PAGES } from "./comic-kinds" + +const PAGE_COUNTS = Array.from({ length: MAX_COMIC_PAGES }, (_, i) => i + 1) + +/** The kind of comic and how many pictures it draws, with what that costs. */ +export function ComicOptionsForm({ + kind, + pages, + onKindChange, + onPagesChange, +}: { + kind: ComicKind + pages: number + onKindChange: (kind: ComicKind) => void + onPagesChange: (pages: number) => void +}) { + const id = useId() + return ( + <div className="space-y-3"> + <ComicKindPicker value={kind} onChange={onKindChange} /> + <div className="space-y-1"> + <label + htmlFor={`${id}-pages`} + className="text-xs font-medium text-muted-foreground" + > + {intl.formatMessage({ + id: "studio_comic_pages_label", + defaultMessage: "Pages", + })} + </label> + <div className="flex items-center gap-3"> + <div className="w-20 shrink-0"> + <Select + id={`${id}-pages`} + value={String(pages)} + onChange={(event) => onPagesChange(Number(event.target.value))} + > + {PAGE_COUNTS.map((count) => ( + <option key={count} value={count}> + {intl.formatNumber(count)} + </option> + ))} + </Select> + </div> + <p className="text-xs text-muted-foreground"> + {intl.formatMessage( + { + id: "studio_comic_cost_body", + defaultMessage: + "Draws {count, plural, one {# picture} other {# pictures}} with your image model. Each is billed by its provider, or takes minutes on this computer.", + }, + { count: pages } + )} + </p> + </div> + </div> + </div> + ) +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/comic-options.tsx b/surfsense_local/frontend/src/features/studio/format-options/comic-options.tsx new file mode 100644 index 0000000000..7246cca6ac --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/comic-options.tsx @@ -0,0 +1,28 @@ +import { useState } from "react" + +import { type ComicKind, startingPages } from "./comic-kinds" +import { ComicOptionsForm } from "./comic-options-form" +import type { FormatOptions } from "./format-options" + +/** A comic starts as a four-page Western comic; a new kind starts at its own length. */ +export function useComicOptions(format: string): FormatOptions | null { + const [kind, setKind] = useState<ComicKind>("western") + const [pages, setPages] = useState(startingPages("western")) + if (format !== "comic") return null + + return { + panel: ( + <ComicOptionsForm + kind={kind} + pages={pages} + onKindChange={(next) => { + setKind(next) + setPages(startingPages(next)) + }} + onPagesChange={setPages} + /> + ), + job: { format, options: { kind, pages } }, + ready: true, + } +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/format-options.ts b/surfsense_local/frontend/src/features/studio/format-options/format-options.ts new file mode 100644 index 0000000000..ea3c445ed8 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/format-options.ts @@ -0,0 +1,11 @@ +import type { ReactNode } from "react" + +import type { StudioJobCreate } from "../api" + +/** What a format adds to the composer and to the job it sends. */ +export type FormatOptions = { + panel: ReactNode + job: Pick<StudioJobCreate, "format" | "options"> + // False while the format still needs something before it can generate. + ready: boolean +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/picture-card.tsx b/surfsense_local/frontend/src/features/studio/format-options/picture-card.tsx new file mode 100644 index 0000000000..882fa804d3 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/picture-card.tsx @@ -0,0 +1,62 @@ +import type { ChangeEvent, ComponentType } from "react" + +import { CheckIcon } from "@/components/ui/icons" +import { cn } from "@/lib/utils" + +/** One choice shown by its picture, its label under it, the chosen one ticked. + * A radio underneath, so a group of them keeps the keyboard and screen readers. */ +export function PictureCard({ + name, + value, + label, + Art, + checked, + disabled = false, + className, + onChange, +}: { + name: string + value: string + label: string + Art: ComponentType<{ className?: string }> | undefined + checked: boolean + disabled?: boolean + className?: string + onChange: (event: ChangeEvent<HTMLInputElement>) => void +}) { + return ( + <label + className={cn( + "group relative flex min-w-0 flex-col gap-1.5", + disabled ? "cursor-not-allowed opacity-50" : "cursor-pointer", + className + )} + > + <input + type="radio" + name={name} + value={value} + checked={checked} + disabled={disabled} + className="peer sr-only" + onChange={onChange} + /> + <span + className={cn( + "relative flex aspect-3/2 items-center justify-center overflow-hidden rounded-lg border transition-colors duration-150 ease-out peer-focus-visible:ring-2 peer-focus-visible:ring-ring motion-reduce:transition-none", + checked + ? "border-primary ring-1 ring-primary" + : "border-border group-hover:border-foreground/30" + )} + > + {Art ? <Art /> : null} + {checked ? ( + <span className="absolute top-1.5 right-1.5 flex size-5 items-center justify-center rounded-full bg-primary text-primary-foreground"> + <CheckIcon className="size-3" strokeWidth={2.5} /> + </span> + ) : null} + </span> + <span className="truncate text-sm">{label}</span> + </label> + ) +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/picture-type-options.tsx b/surfsense_local/frontend/src/features/studio/format-options/picture-type-options.tsx new file mode 100644 index 0000000000..5b76706c34 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/picture-type-options.tsx @@ -0,0 +1,23 @@ +import { useState } from "react" + +import type { StudioFormat } from "../api" +import { pictureTypes } from "../picture-types" +import type { FormatOptions } from "./format-options" +import { PictureTypePicker } from "./picture-type-picker" + +/** The Image card makes the picture type chosen in its dialog, Image first. */ +export function usePictureTypeOptions( + format: string, + catalog: StudioFormat[] +): FormatOptions | null { + const [type, setType] = useState(format) + if (format !== "image") return null + + const types = pictureTypes(catalog) + if (types.length < 2) return null + return { + panel: <PictureTypePicker types={types} value={type} onChange={setType} />, + job: { format: type }, + ready: types.some((entry) => entry.key === type && entry.available), + } +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx new file mode 100644 index 0000000000..37607582f9 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx @@ -0,0 +1,53 @@ +import type { ComponentType } from "react" + +import { ImageTypeIcon, InfographicTypeIcon } from "@/components/ui/icons" +import { intl } from "@/i18n/intl" + +import type { StudioFormat } from "../api" +import { formatLabel } from "../studio-formats" +import { PictureCard } from "./picture-card" + +const TYPE_ART: Record<string, ComponentType<{ className?: string }>> = { + image: ImageTypeIcon, + infographic: InfographicTypeIcon, +} + +/** Image or infographic as cards, each shown by its picture, the chosen one ticked. + * Both need the same models, so a type is closed only if the server ever says so. */ +export function PictureTypePicker({ + types, + value, + onChange, +}: { + types: StudioFormat[] + value: string + onChange: (key: string) => void +}) { + const label = intl.formatMessage({ + id: "studio_picture_type_label", + defaultMessage: "Type", + }) + return ( + <div className="space-y-2"> + <p className="text-xs font-medium text-muted-foreground">{label}</p> + <div + role="radiogroup" + aria-label={label} + className="grid grid-cols-2 gap-2" + > + {types.map((entry) => ( + <PictureCard + key={entry.key} + name="studio-picture-type" + value={entry.key} + label={formatLabel(entry)} + Art={TYPE_ART[entry.key]} + checked={value === entry.key} + disabled={!entry.available} + onChange={() => onChange(entry.key)} + /> + ))} + </div> + </div> + ) +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/podcast-options.tsx b/surfsense_local/frontend/src/features/studio/format-options/podcast-options.tsx new file mode 100644 index 0000000000..5f2a5163b1 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/podcast-options.tsx @@ -0,0 +1,46 @@ +import { intl } from "@/i18n/intl" + +import { PodcastBriefForm } from "../podcast-brief-form" +import { usePodcastBrief } from "../use-podcast-brief" +import type { FormatOptions } from "./format-options" + +/** A podcast is generated from its reviewed brief, so it waits for the brief. A + * server model with no voices yet can voice nothing: set them up first. */ +export function usePodcastOptions( + format: string, + workspaceId: number, + onSetUpVoices: () => void +): FormatOptions | null { + const podcast = usePodcastBrief(format === "podcast" ? workspaceId : null) + if (format !== "podcast") return null + + const needsVoices = + podcast.opened?.voices_source === "saved" && + podcast.opened.voices.length === 0 + const panel = + podcast.brief && podcast.opened ? ( + <PodcastBriefForm + brief={podcast.brief} + voices={podcast.opened.voices} + languages={podcast.opened.languages} + voicesSource={podcast.opened.voices_source} + voicesProblem={podcast.opened.problem} + voicedBy={podcast.opened.voiced_by} + onSetUpVoices={onSetUpVoices} + onChange={podcast.setBrief} + /> + ) : ( + <p className="text-sm text-muted-foreground"> + {podcast.error ?? + intl.formatMessage({ + id: "studio_composer_brief_status", + defaultMessage: "Preparing the brief…", + })} + </p> + ) + return { + panel, + job: { format, options: podcast.brief ?? undefined }, + ready: podcast.brief != null && !needsVoices, + } +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts b/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts new file mode 100644 index 0000000000..445a7d246c --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts @@ -0,0 +1,21 @@ +import type { StudioFormat } from "../api" +import { useComicOptions } from "./comic-options" +import type { FormatOptions } from "./format-options" +import { usePictureTypeOptions } from "./picture-type-options" +import { usePodcastOptions } from "./podcast-options" + +/** The chosen format's own options, or none: a format with options adds its hook here. */ +export function useFormatOptions( + format: string, + workspaceId: number, + catalog: StudioFormat[], + onSetUpVoices: () => void +): FormatOptions { + // Every hook runs on every render; each answers only for its own format. + const podcast = usePodcastOptions(format, workspaceId, onSetUpVoices) + const picture = usePictureTypeOptions(format, catalog) + const comic = useComicOptions(format) + return ( + podcast ?? picture ?? comic ?? { panel: null, job: { format }, ready: true } + ) +} diff --git a/surfsense_local/frontend/src/features/studio/picture-types.ts b/surfsense_local/frontend/src/features/studio/picture-types.ts new file mode 100644 index 0000000000..b9cb393fef --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/picture-types.ts @@ -0,0 +1,20 @@ +import type { StudioFormat } from "./api" + +// An infographic is offered as a type of image, not a card of its own. Each type +// still makes its own format, so its edits and the agent's tool see no change. +export const PICTURE_TYPES = ["image", "infographic"] as const + +export type PictureType = (typeof PICTURE_TYPES)[number] + +/** The catalog's entry for each picture type it offers, in the picker's order. */ +export function pictureTypes(catalog: StudioFormat[]): StudioFormat[] { + return PICTURE_TYPES.flatMap( + (key) => catalog.find((entry) => entry.key === key) ?? [] + ) +} + +/** The cards Studio shows: an infographic only when there is no image to hold it. */ +export function formatCards(catalog: StudioFormat[]): StudioFormat[] { + if (!catalog.some((entry) => entry.key === "image")) return catalog + return catalog.filter((entry) => entry.key !== "infographic") +} diff --git a/surfsense_local/frontend/src/features/studio/refine-box.tsx b/surfsense_local/frontend/src/features/studio/refine-box.tsx index acf1153438..35640d4794 100644 --- a/surfsense_local/frontend/src/features/studio/refine-box.tsx +++ b/surfsense_local/frontend/src/features/studio/refine-box.tsx @@ -24,6 +24,8 @@ import { } from "@/components/ui/tooltip" import { intl } from "@/i18n/intl" +import { usePicturesIn } from "@/features/models/pictures-in/use-pictures-in" + import { useEditModelChosen, type RefineSubject } from "./edit-model" import { EditModelNotice } from "./edit-model-notice" import { messageFrom } from "./use-studio" @@ -161,7 +163,10 @@ export function RefineBox({ if (width) box.current?.style.setProperty("--pill-width", `${width + 2}px`) }, [shownVersion]) - const labels = subject === "image" ? imageLabels() : documentLabels() + const picturesIn = usePicturesIn({ enabled: subject === "image" }) + // Said only once known: an answer that is not one leaves the usual words. + const redraws = picturesIn.data?.source === null + const labels = subject === "image" ? imageLabels(redraws) : documentLabels() const openLabel = labels.open const makingLabel = (version: number) => @@ -334,7 +339,7 @@ function documentLabels() { } } -function imageLabels() { +function imageLabels(redraws: boolean) { return { open: intl.formatMessage({ id: "studio_image_edit_open_aria", @@ -344,10 +349,15 @@ function imageLabels() { id: "studio_image_edit_instruction_aria", defaultMessage: "How to change this image", }), - placeholder: intl.formatMessage({ - id: "studio_image_edit_instruction_placeholder", - defaultMessage: "Describe a change, such as a darker background", - }), + placeholder: redraws + ? intl.formatMessage({ + id: "studio_image_edit_redraw_placeholder", + defaultMessage: "Describe a change: the picture is drawn again", + }) + : intl.formatMessage({ + id: "studio_image_edit_instruction_placeholder", + defaultMessage: "Describe a change, such as a darker background", + }), submit: intl.formatMessage({ id: "studio_image_edit_submit_button", defaultMessage: "Edit", diff --git a/surfsense_local/frontend/src/features/studio/studio-error-text.test.ts b/surfsense_local/frontend/src/features/studio/studio-error-text.test.ts index e82152769c..b5614eea56 100644 --- a/surfsense_local/frontend/src/features/studio/studio-error-text.test.ts +++ b/surfsense_local/frontend/src/features/studio/studio-error-text.test.ts @@ -34,7 +34,6 @@ describe("a refused Studio job", () => { "image_version_unfinished", "This version did not finish: retry it, or edit another version.", ], - ["needs_image_edit", "Needs an image editing model."], [ "edited_from", "A later version is edited from this one: delete the whole document.", diff --git a/surfsense_local/frontend/src/features/studio/studio-error-text.ts b/surfsense_local/frontend/src/features/studio/studio-error-text.ts index b33ba69199..85e93e5a18 100644 --- a/surfsense_local/frontend/src/features/studio/studio-error-text.ts +++ b/surfsense_local/frontend/src/features/studio/studio-error-text.ts @@ -110,11 +110,6 @@ const studioErrorText: Record<string, (detail: Detail) => string | null> = { id: "studio_error_needs_image", defaultMessage: "Needs an image model.", }), - needs_image_edit: () => - intl.formatMessage({ - id: "studio_error_needs_image_edit", - defaultMessage: "Needs an image editing model.", - }), needs_audio: () => intl.formatMessage({ id: "studio_error_needs_audio", diff --git a/surfsense_local/frontend/src/features/studio/studio-formats.ts b/surfsense_local/frontend/src/features/studio/studio-formats.ts index 6798cb61dc..8680444bf5 100644 --- a/surfsense_local/frontend/src/features/studio/studio-formats.ts +++ b/surfsense_local/frontend/src/features/studio/studio-formats.ts @@ -2,6 +2,7 @@ import type { ComponentType } from "react" import { AiSearchLinesIcon, + CanvasIcon, Cards01Icon, ChartHistogramIcon, File02Icon, @@ -37,6 +38,7 @@ export const FORMAT_ICONS: Record< podcast: PodcastIcon, image: Image01Icon, infographic: ChartHistogramIcon, + comic: CanvasIcon, } // The backend's catalog sends English labels; a known key shows its own. @@ -101,22 +103,28 @@ const FORMAT_LABELS: Record<string, () => string> = { id: "studio_format_infographic_label", defaultMessage: "Infographic", }), + comic: () => + intl.formatMessage({ + id: "studio_format_comic_label", + defaultMessage: "Comic", + }), } export function formatLabel(entry: { key: string; label: string }): string { return FORMAT_LABELS[entry.key]?.() ?? entry.label } +const DRAWN_FORMATS = new Set(["image", "infographic", "comic"]) + // The formats offered before the API answers, and if it never does. The // backend's catalog replaces this as soon as it loads. export function studioCatalog(): StudioFormat[] { return Object.keys(FORMAT_LABELS).map((key) => ({ key, label: FORMAT_LABELS[key](), - requires_model_types: - key === "image" || key === "infographic" - ? ["image_gen", "text_gen"] - : ["text_gen"], + requires_model_types: DRAWN_FORMATS.has(key) + ? ["image_gen", "text_gen"] + : ["text_gen"], available: true, unavailable_reason: null, })) diff --git a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx index 01e1314fb0..f07f8aeaf0 100644 --- a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx +++ b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx @@ -181,7 +181,8 @@ describe("studio panel", () => { ) expect(screen.getByRole("button", { name: "Markdown" })).toBeTruthy() - expect(screen.getByRole("button", { name: "Infographic" })).toBeTruthy() + expect(screen.getByRole("button", { name: "Image" })).toBeTruthy() + expect(screen.queryByRole("button", { name: "Infographic" })).toBeNull() expect(document.querySelector("[data-slot=skeleton]")).toBeNull() }) @@ -290,6 +291,135 @@ describe("studio panel", () => { ).toBeTruthy() }) + it("offers an infographic as a type of image, and makes the type chosen", async () => { + const picture = (key: string, label: string) => ({ + key, + label, + requires_model_types: ["image_gen", "text_gen"], + available: true, + unavailable_reason: null, + }) + const fetchMock = vi.fn( + async (input: RequestInfo | URL, init?: RequestInit) => { + const path = String(input) + if (path === "/workspaces/1/studio/formats") { + return Response.json([ + picture("image", "Image"), + picture("infographic", "Infographic"), + ]) + } + if (path === "/workspaces/1/studio/jobs" && init?.method === "POST") { + return Response.json( + { ...pendingArtifact, format: "infographic", title: "Infographic" }, + { status: 201 } + ) + } + if (path === "/workspaces/1/artifacts") return Response.json([]) + return Response.json({ detail: "not found" }, { status: 404 }) + } + ) + vi.stubGlobal("fetch", fetchMock) + const user = userEvent.setup() + + renderStudio() + + const formats = screen.getByRole("region", { name: "Studio formats" }) + await within(formats).findByRole("button", { name: "Image" }) + expect( + within(formats).queryByRole("button", { name: "Infographic" }) + ).toBeNull() + + await user.click(within(formats).getByRole("button", { name: "Image" })) + const dialog = screen.getByRole("dialog", { name: "Image" }) + const type = within(dialog).getByRole("radiogroup", { name: "Type" }) + // A card for each type, shown by its own picture. + expect(type.querySelectorAll("[data-slot=picture-type-art]")).toHaveLength( + 2 + ) + expect( + (within(type).getByRole("radio", { name: "Image" }) as HTMLInputElement) + .checked + ).toBe(true) + await user.click(within(type).getByRole("radio", { name: "Infographic" })) + await user.click(within(dialog).getByRole("button", { name: /Generate/ })) + + const jobCall = await vi.waitFor(() => + fetchMock.mock.calls.find( + ([path, init]) => + path === "/workspaces/1/studio/jobs" && init?.method === "POST" + ) + ) + expect(JSON.parse(String(jobCall?.[1]?.body))).toEqual({ + format: "infographic", + document_ids: [4], + }) + }) + + it("makes a comic of the kind and length chosen, and says what it costs", async () => { + const fetchMock = vi.fn( + async (input: RequestInfo | URL, init?: RequestInit) => { + const path = String(input) + if (path === "/workspaces/1/studio/formats") { + return Response.json([ + { + key: "comic", + label: "Comic", + requires_model_types: ["image_gen", "text_gen"], + available: true, + unavailable_reason: null, + }, + ]) + } + if (path === "/workspaces/1/studio/jobs" && init?.method === "POST") { + return Response.json( + { ...pendingArtifact, format: "comic", title: "Comic" }, + { status: 201 } + ) + } + if (path === "/workspaces/1/artifacts") return Response.json([]) + return Response.json({ detail: "not found" }, { status: 404 }) + } + ) + vi.stubGlobal("fetch", fetchMock) + const user = userEvent.setup() + + renderStudio() + + await user.click(await screen.findByRole("button", { name: "Comic" })) + const dialog = screen.getByRole("dialog", { name: "Comic" }) + // Every kind is a card shown by its own picture, a Western comic first. + const kind = within(dialog).getByRole("radiogroup", { name: "Kind" }) + expect(within(kind).getAllByRole("radio")).toHaveLength(10) + expect(kind.querySelectorAll("[data-slot=picture-type-art]")).toHaveLength( + 10 + ) + const western = within(kind).getByRole("radio", { name: "Western comic" }) + expect((western as HTMLInputElement).checked).toBe(true) + const pages = within(dialog).getByLabelText("Pages") as HTMLSelectElement + expect(pages.value).toBe("4") + expect(within(dialog).getByText(/Draws 4 pictures/)).toBeTruthy() + + // A webtoon counts beats, and starts at its own length. + await user.click(within(kind).getByRole("radio", { name: "Webtoon" })) + expect(pages.value).toBe("8") + await user.click(within(kind).getByRole("radio", { name: "Manga" })) + await user.selectOptions(pages, "6") + expect(within(dialog).getByText(/Draws 6 pictures/)).toBeTruthy() + await user.click(within(dialog).getByRole("button", { name: /Generate/ })) + + const jobCall = await vi.waitFor(() => + fetchMock.mock.calls.find( + ([path, init]) => + path === "/workspaces/1/studio/jobs" && init?.method === "POST" + ) + ) + expect(JSON.parse(String(jobCall?.[1]?.body))).toEqual({ + format: "comic", + document_ids: [4], + options: { kind: "manga", pages: 6 }, + }) + }) + it("opens the podcast brief for review and sends it with the job", async () => { const brief = { language: "en-US", diff --git a/surfsense_local/frontend/src/features/studio/studio-panel.tsx b/surfsense_local/frontend/src/features/studio/studio-panel.tsx index 7a0e553f3a..01d35e372d 100644 --- a/surfsense_local/frontend/src/features/studio/studio-panel.tsx +++ b/surfsense_local/frontend/src/features/studio/studio-panel.tsx @@ -30,10 +30,10 @@ import { intl } from "@/i18n/intl" import { cn } from "@/lib/utils" import type { StudioFormat, StudioJobCreate } from "./api" -import { PodcastBriefForm } from "./podcast-brief-form" +import { useFormatOptions } from "./format-options/use-format-options" +import { formatCards } from "./picture-types" import { FORMAT_ICONS, formatLabel, studioCatalog } from "./studio-formats" import { translatedUnavailable } from "./studio-unavailable-text" -import { usePodcastBrief } from "./use-podcast-brief" const FORMAT_HINTS: Record<string, () => string> = { summary: () => @@ -92,10 +92,10 @@ const FORMAT_HINTS: Record<string, () => string> = { id: "studio_format_image_tooltip", defaultMessage: "Generate an AI image based on your sources", }), - infographic: () => + comic: () => intl.formatMessage({ - id: "studio_format_infographic_tooltip", - defaultMessage: "Generate an AI infographic based on your sources", + id: "studio_format_comic_tooltip", + defaultMessage: "Generate an AI comic based on your sources", }), } @@ -136,6 +136,7 @@ function formatHint(entry: StudioFormat) { function Composer({ workspaceId, format, + catalog, documents, selectedDocumentIds, sourceScope, @@ -147,6 +148,7 @@ function Composer({ }: { workspaceId: number format: string + catalog: StudioFormat[] documents: WorkspaceDocument[] selectedDocumentIds: number[] sourceScope?: SourceScope @@ -160,17 +162,9 @@ function Composer({ const selected = new Set(selectedDocumentIds) const [prompt, setPrompt] = useState("") const [view, setView] = useState<"main" | "sources">("main") - const podcast = usePodcastBrief(format === "podcast" ? workspaceId : null) + const options = useFormatOptions(format, workspaceId, catalog, onSetUpVoices) const allSelected = ready.length > 0 && selected.size === ready.length - - // A podcast is generated from its reviewed brief, so it waits for the brief. - // A server model with no voices yet can voice nothing: set them up first. - const needsVoices = - podcast.opened?.voices_source === "saved" && - podcast.opened.voices.length === 0 - const briefReady = - format !== "podcast" || (podcast.brief != null && !needsVoices) - const canGenerate = selected.size > 0 && !isCreating && briefReady + const canGenerate = selected.size > 0 && !isCreating && options.ready return ( // A taller dialog for every format: Generate stays at the bottom, and the @@ -186,28 +180,7 @@ function Composer({ aria-hidden={view !== "main"} > <div className="space-y-3"> - {format === "podcast" ? ( - podcast.brief && podcast.opened ? ( - <PodcastBriefForm - brief={podcast.brief} - voices={podcast.opened.voices} - languages={podcast.opened.languages} - voicesSource={podcast.opened.voices_source} - voicesProblem={podcast.opened.problem} - voicedBy={podcast.opened.voiced_by} - onSetUpVoices={onSetUpVoices} - onChange={podcast.setBrief} - /> - ) : ( - <p className="text-sm text-muted-foreground"> - {podcast.error ?? - intl.formatMessage({ - id: "studio_composer_brief_status", - defaultMessage: "Preparing the brief…", - })} - </p> - ) - ) : null} + {options.panel} {ready.length === 0 ? ( <div className="space-y-2"> @@ -275,11 +248,10 @@ function Composer({ disabled={!canGenerate} onClick={() => { onGenerate({ - format, + ...options.job, document_ids: [...selected], ...(sourceScope ? { source_scope: sourceScope } : {}), prompt: prompt.trim() || undefined, - options: podcast.brief ?? undefined, }) }} > @@ -430,6 +402,7 @@ export const StudioPanel = memo(function StudioPanel({ const [format, setFormat] = useState<string | null>(null) const [formatOpen, setFormatOpen] = useState(false) const catalog = useMemo(() => catalogFormats(formats), [formats]) + const cards = useMemo(() => formatCards(catalog), [catalog]) const selectedFormat = catalog.find((entry) => entry.key === format) const selectFormat = useCallback((key: string) => { setFormat(key) @@ -458,7 +431,7 @@ export const StudioPanel = memo(function StudioPanel({ })} > <div className="grid grid-cols-3 gap-1.5"> - {catalog.map((entry) => ( + {cards.map((entry) => ( <FormatCard key={entry.key} entry={entry} @@ -489,6 +462,7 @@ export const StudioPanel = memo(function StudioPanel({ key={selectedFormat.key} workspaceId={workspaceId} format={selectedFormat.key} + catalog={catalog} documents={documents} selectedDocumentIds={selectedDocumentIds} sourceScope={sourceScope} diff --git a/surfsense_local/frontend/src/features/studio/viewers/registry.tsx b/surfsense_local/frontend/src/features/studio/viewers/registry.tsx index 13464503ae..d954e0491b 100644 --- a/surfsense_local/frontend/src/features/studio/viewers/registry.tsx +++ b/surfsense_local/frontend/src/features/studio/viewers/registry.tsx @@ -59,6 +59,8 @@ const ARTIFACT_VIEWERS: Record<string, ComponentType<ArtifactViewerProps>> = { infographic: MediaViewer, xlsx: XlsxViewer, pdf: PdfViewer, + // A comic is its bound pages, one PDF. + comic: PdfViewer, pptx: PptxViewer, html: HtmlViewer, docx: DocxViewer, diff --git a/surfsense_local/frontend/translations/de.json b/surfsense_local/frontend/translations/de.json index ae08f9ee0e..69e4d64761 100644 --- a/surfsense_local/frontend/translations/de.json +++ b/surfsense_local/frontend/translations/de.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "{hosts} erlauben", "agent_steps_analyze_data_label": "Analyse <doc>{title}</doc> ausgeführt", "agent_steps_analyze_data_untitled_label": "Daten analysiert", + "agent_steps_bind_comic_created_label": "<doc>{title}</doc> v{version, number} erstellt", + "agent_steps_bind_comic_label": "Comic <doc>{title}</doc> wird gebunden", + "agent_steps_bind_comic_unnamed_label": "Comic wird gebunden", + "agent_steps_bind_comic_updated_label": "<doc>{title}</doc> auf v{version, number} aktualisiert", "agent_steps_confirm_job_label": "Antwortvorschläge in <doc>{title}</doc> bestätigt", "agent_steps_confirm_job_pending_label": "Bestätigung der Antwortvorschläge gestartet", "agent_steps_convert_label": "<doc>{title}</doc> in PDF umgewandelt", @@ -942,7 +946,7 @@ "models_in_use_local_label": "Dieser Computer", "models_in_use_missing_label": "Nicht auf diesem Computer gefunden", "models_in_use_summary_aria": "{slot, select, chat {Aktives Chat-Modell} image {Aktives Bildmodell} image_edit {Aktives Bildbearbeitungsmodell} video {Aktives Videomodell} audio {Aktives Audiomodell} speech {Aktives Spracherkennungsmodell} other {Aktives Modell}}", - "models_in_use_summary_empty": "{slot, select, chat {Kein Chat-Modell gewählt} image {Kein Bildmodell gewählt} image_edit {Kein Bildbearbeitungsmodell gewählt} video {Kein Videomodell gewählt} audio {Kein Audiomodell gewählt} speech {Kein Spracherkennungsmodell gewählt} other {Kein Modell gewählt}}", + "models_in_use_summary_empty": "{slot, select, chat {Kein Chat-Modell gewählt} image {Kein Bildmodell gewählt} image_edit {Wie das Bildmodell} video {Kein Videomodell gewählt} audio {Kein Audiomodell gewählt} speech {Kein Spracherkennungsmodell gewählt} other {Kein Modell gewählt}}", "models_in_use_summary_label": "Aktiv:", "models_in_use_unknown_server_label": "Ein Server", "models_install_cancelled_short_status": "Abgebrochen", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "Wie System", "settings_general_reduce_motion_title": "Bewegung reduzieren", "settings_general_title": "Allgemein", - "settings_image_edit_models_body": "Das Modell, das Bilder bearbeitet. Führe eines auf diesem Computer aus oder nutze eines von einem Server.", - "settings_image_edit_models_title": "Modelle für Bildbearbeitung", + "settings_image_edit_models_body": "Optional. Dein Bildmodell bearbeitet Bilder, wenn es kann, und zeichnet sie sonst neu. Wähle hier nur ein Modell, wenn du mit einem anderen bearbeiten willst.", + "settings_image_edit_models_title": "Anderes Modell für Bearbeitungen", "settings_image_models_body": "Das Modell, das Bilder in Studio erstellt. Führe eines auf diesem Computer aus oder nutze eines von einem Server.", "settings_image_models_title": "Modelle für Bilderzeugung", "settings_models_add_button": "Modell hinzufügen", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "Erstellung von {name} fehlgeschlagen", "studio_artifact_row_script_failed_tooltip": "Das Dokumentskript ist fehlgeschlagen. Bitte den Agenten, es zu beheben.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Zeichnet {count, plural, one {# Bild} other {# Bilder}} mit deinem Bildmodell. Jedes rechnet dein Anbieter ab, oder es dauert auf diesem Computer einige Minuten.", + "studio_comic_kind_bande_dessinee_label": "Frankobelgischer Comic", + "studio_comic_kind_comic_strip_label": "Comicstrip", + "studio_comic_kind_graphic_novel_label": "Graphic Novel", + "studio_comic_kind_label": "Art", + "studio_comic_kind_manga_label": "Manga", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Bilderbuch", + "studio_comic_kind_scroll_ahead_aria": "Weitere Arten zeigen", + "studio_comic_kind_scroll_back_aria": "Frühere Arten zeigen", + "studio_comic_kind_storyboard_label": "Storyboard", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "Westlicher Comic", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-Koma)", + "studio_comic_pages_label": "Seiten", "studio_composer_brief_status": "Briefing wird vorbereitet…", "studio_composer_generate_button": "Erstellen", "studio_composer_prompt_label": "Prompt (optional)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "Ein Chatmodell und ein Audiomodell sind erforderlich.", "studio_error_needs_chat_image": "Ein Chatmodell und ein Bildmodell sind erforderlich.", "studio_error_needs_image": "Ein Bildmodell ist erforderlich.", - "studio_error_needs_image_edit": "Ein Modell für Bildbearbeitung ist erforderlich.", "studio_error_no_sources": "Wähle mindestens eine Quelle aus.", "studio_error_not_editable": "Nur ein in Studio erstelltes Bild oder eine Infografik kann bearbeitet werden.", "studio_error_not_refinable": "Nur ein in Studio erstelltes Word-Dokument, PDF oder eine Webseite kann überarbeitet werden.", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "Karten mischen", "studio_flashcards_viewer_shuffle_button": "Mischen", "studio_flashcards_viewer_shuffle_error": "Mischen konnte nicht gespeichert werden", + "studio_format_comic_label": "Comic", + "studio_format_comic_tooltip": "Comic mit KI aus deinen Quellen erstellen", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Word-Dokument mit KI aus deinen Quellen erstellen", "studio_format_flashcards_label": "Karteikarten", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "Bild", "studio_format_image_tooltip": "Bild mit KI aus deinen Quellen erstellen", "studio_format_infographic_label": "Infografik", - "studio_format_infographic_tooltip": "Infografik mit KI aus deinen Quellen erstellen", "studio_format_mindmap_label": "Mindmap", "studio_format_mindmap_tooltip": "Mindmap mit KI aus deinen Quellen erstellen", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "Wie dieses Bild geändert werden soll", "studio_image_edit_instruction_placeholder": "Beschreibe eine Änderung, etwa einen dunkleren Hintergrund", "studio_image_edit_open_aria": "Dieses Bild bearbeiten", + "studio_image_edit_redraw_placeholder": "Beschreibe eine Änderung: Das Bild wird neu gezeichnet", "studio_image_edit_submit_button": "Bearbeiten", "studio_mindmap_viewer_fit_aria": "Mindmap einpassen", "studio_mindmap_viewer_tree_aria": "Mindmap", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "PDF konnte nicht geladen werden", "studio_pdf_viewer_zoom_in_aria": "Vergrößern", "studio_pdf_viewer_zoom_out_aria": "Verkleinern", + "studio_picture_type_label": "Typ", "studio_podcast_brief_add_speaker_button": "Sprecher hinzufügen", "studio_podcast_brief_language_label": "Sprache", "studio_podcast_brief_length_label": "Länge", diff --git a/surfsense_local/frontend/translations/en.json b/surfsense_local/frontend/translations/en.json index bf10ebac99..d40e961f00 100644 --- a/surfsense_local/frontend/translations/en.json +++ b/surfsense_local/frontend/translations/en.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Allow {hosts}", "agent_steps_analyze_data_label": "Ran the analysis <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "Analysed data", + "agent_steps_bind_comic_created_label": "Created <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Binding the comic <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Binding a comic", + "agent_steps_bind_comic_updated_label": "Updated <doc>{title}</doc> to v{version, number}", "agent_steps_confirm_job_label": "Confirmed suggested answers in <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "Started confirming suggested answers", "agent_steps_convert_label": "Converted <doc>{title}</doc> to PDF", @@ -942,7 +946,7 @@ "models_in_use_local_label": "This computer", "models_in_use_missing_label": "Not found on this computer", "models_in_use_summary_aria": "{slot, select, chat {chat model in use} image {image model in use} image_edit {image editing model in use} video {video model in use} audio {audio model in use} speech {speech recognition model in use} other {model in use}}", - "models_in_use_summary_empty": "{slot, select, chat {No chat model chosen} image {No image model chosen} image_edit {No image editing model chosen} video {No video model chosen} audio {No audio model chosen} speech {No speech recognition model chosen} other {No model chosen}}", + "models_in_use_summary_empty": "{slot, select, chat {No chat model chosen} image {No image model chosen} image_edit {Same as the image model} video {No video model chosen} audio {No audio model chosen} speech {No speech recognition model chosen} other {No model chosen}}", "models_in_use_summary_label": "In use:", "models_in_use_unknown_server_label": "A server", "models_install_cancelled_short_status": "Cancelled", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "Match system", "settings_general_reduce_motion_title": "Reduce motion", "settings_general_title": "General", - "settings_image_edit_models_body": "The model that edits images. Run one on this computer, or use one from a server.", - "settings_image_edit_models_title": "Image editing models", + "settings_image_edit_models_body": "Optional. Your image model edits pictures when it can, and draws them again when it cannot. Choose a model here only to edit with a different one.", + "settings_image_edit_models_title": "Use a different model for edits", "settings_image_models_body": "The model that makes images in Studio. Run one on this computer, or use one from a server.", "settings_image_models_title": "Image generation models", "settings_models_add_button": "Add model", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "Generation failed for {name}", "studio_artifact_row_script_failed_tooltip": "The document script failed. Ask the agent to fix it.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Draws {count, plural, one {# picture} other {# pictures}} with your image model. Each is billed by its provider, or takes minutes on this computer.", + "studio_comic_kind_bande_dessinee_label": "Bande dessinée", + "studio_comic_kind_comic_strip_label": "Comic strip", + "studio_comic_kind_graphic_novel_label": "Graphic novel", + "studio_comic_kind_label": "Kind", + "studio_comic_kind_manga_label": "Manga", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Picture book", + "studio_comic_kind_scroll_ahead_aria": "Show more kinds", + "studio_comic_kind_scroll_back_aria": "Show earlier kinds", + "studio_comic_kind_storyboard_label": "Storyboard", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "Western comic", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-koma)", + "studio_comic_pages_label": "Pages", "studio_composer_brief_status": "Preparing the brief…", "studio_composer_generate_button": "Generate", "studio_composer_prompt_label": "Prompt (optional)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "Needs a chat model and an audio model.", "studio_error_needs_chat_image": "Needs a chat model and an image model.", "studio_error_needs_image": "Needs an image model.", - "studio_error_needs_image_edit": "Needs an image editing model.", "studio_error_no_sources": "Pick at least one source.", "studio_error_not_editable": "Only an image or infographic made in Studio can be edited.", "studio_error_not_refinable": "Only a Word document, PDF or web page made in Studio can be refined.", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "Shuffle cards", "studio_flashcards_viewer_shuffle_button": "Shuffle", "studio_flashcards_viewer_shuffle_error": "Shuffle could not be saved", + "studio_format_comic_label": "Comic", + "studio_format_comic_tooltip": "Generate an AI comic based on your sources", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Generate an AI Word document based on your sources", "studio_format_flashcards_label": "Flashcards", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "Image", "studio_format_image_tooltip": "Generate an AI image based on your sources", "studio_format_infographic_label": "Infographic", - "studio_format_infographic_tooltip": "Generate an AI infographic based on your sources", "studio_format_mindmap_label": "Mind map", "studio_format_mindmap_tooltip": "Generate an AI mind map based on your sources", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "How to change this image", "studio_image_edit_instruction_placeholder": "Describe a change, such as a darker background", "studio_image_edit_open_aria": "Edit this image", + "studio_image_edit_redraw_placeholder": "Describe a change: the picture is drawn again", "studio_image_edit_submit_button": "Edit", "studio_mindmap_viewer_fit_aria": "Fit mind map", "studio_mindmap_viewer_tree_aria": "Mind map", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "Failed to load PDF", "studio_pdf_viewer_zoom_in_aria": "Zoom in", "studio_pdf_viewer_zoom_out_aria": "Zoom out", + "studio_picture_type_label": "Type", "studio_podcast_brief_add_speaker_button": "Add speaker", "studio_podcast_brief_language_label": "Language", "studio_podcast_brief_length_label": "Length", diff --git a/surfsense_local/frontend/translations/es.json b/surfsense_local/frontend/translations/es.json index 4c207d2bd7..2d3b98863d 100644 --- a/surfsense_local/frontend/translations/es.json +++ b/surfsense_local/frontend/translations/es.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Permitir {hosts}", "agent_steps_analyze_data_label": "Ejecutó el análisis <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "Analizó datos", + "agent_steps_bind_comic_created_label": "Creó <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Encuadernando el cómic <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Encuadernando un cómic", + "agent_steps_bind_comic_updated_label": "Actualizó <doc>{title}</doc> a la v{version, number}", "agent_steps_confirm_job_label": "Confirmó respuestas sugeridas en <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "Empezó a confirmar respuestas sugeridas", "agent_steps_convert_label": "Convirtió <doc>{title}</doc> a PDF", @@ -942,7 +946,7 @@ "models_in_use_local_label": "Este equipo", "models_in_use_missing_label": "No se encuentra en este equipo", "models_in_use_summary_aria": "{slot, select, chat {modelo de chat en uso} image {modelo de imagen en uso} image_edit {modelo de edición de imágenes en uso} video {modelo de video en uso} audio {modelo de audio en uso} speech {modelo de reconocimiento de voz en uso} other {modelo en uso}}", - "models_in_use_summary_empty": "{slot, select, chat {Sin modelo de chat elegido} image {Sin modelo de imagen elegido} image_edit {Sin modelo de edición de imágenes elegido} video {Sin modelo de video elegido} audio {Sin modelo de audio elegido} speech {Sin modelo de reconocimiento de voz elegido} other {Sin modelo elegido}}", + "models_in_use_summary_empty": "{slot, select, chat {Sin modelo de chat elegido} image {Sin modelo de imagen elegido} image_edit {Igual que el modelo de imagen} video {Sin modelo de video elegido} audio {Sin modelo de audio elegido} speech {Sin modelo de reconocimiento de voz elegido} other {Sin modelo elegido}}", "models_in_use_summary_label": "En uso:", "models_in_use_unknown_server_label": "Un servidor", "models_install_cancelled_short_status": "Cancelado", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "Igual que el sistema", "settings_general_reduce_motion_title": "Reducir movimiento", "settings_general_title": "General", - "settings_image_edit_models_body": "El modelo que edita imágenes. Ejecuta uno en este equipo o usa uno de un servidor.", - "settings_image_edit_models_title": "Modelos de edición de imágenes", + "settings_image_edit_models_body": "Opcional. Tu modelo de imagen edita las imágenes cuando puede y, si no, las vuelve a dibujar. Elige aquí un modelo solo para editar con otro distinto.", + "settings_image_edit_models_title": "Usar otro modelo para ediciones", "settings_image_models_body": "El modelo que crea imágenes en Studio. Ejecuta uno en este equipo o usa uno de un servidor.", "settings_image_models_title": "Modelos de generación de imágenes", "settings_models_add_button": "Añadir modelo", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "Falló la generación de {name}", "studio_artifact_row_script_failed_tooltip": "Falló el script del documento. Pide al agente que lo corrija.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Dibuja {count, plural, one {# imagen} many {# imágenes} other {# imágenes}} con tu modelo de imagen. Cada una la cobra su proveedor, o tarda unos minutos en este equipo.", + "studio_comic_kind_bande_dessinee_label": "Cómic franco-belga", + "studio_comic_kind_comic_strip_label": "Tira cómica", + "studio_comic_kind_graphic_novel_label": "Novela gráfica", + "studio_comic_kind_label": "Tipo", + "studio_comic_kind_manga_label": "Manga", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Libro ilustrado", + "studio_comic_kind_scroll_ahead_aria": "Mostrar más tipos", + "studio_comic_kind_scroll_back_aria": "Mostrar tipos anteriores", + "studio_comic_kind_storyboard_label": "Guion gráfico", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "Cómic occidental", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-koma)", + "studio_comic_pages_label": "Páginas", "studio_composer_brief_status": "Preparando las indicaciones…", "studio_composer_generate_button": "Generar", "studio_composer_prompt_label": "Prompt (opcional)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "Se necesitan un modelo de chat y uno de audio.", "studio_error_needs_chat_image": "Se necesitan un modelo de chat y uno de imagen.", "studio_error_needs_image": "Se necesita un modelo de imagen.", - "studio_error_needs_image_edit": "Se necesita un modelo de edición de imágenes.", "studio_error_no_sources": "Elige al menos una fuente.", "studio_error_not_editable": "Solo se puede editar una imagen o una infografía creadas en Studio.", "studio_error_not_refinable": "Solo se puede refinar un documento de Word, un PDF o una página web creados en Studio.", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "Mezclar las tarjetas", "studio_flashcards_viewer_shuffle_button": "Mezclar", "studio_flashcards_viewer_shuffle_error": "No se pudo guardar la mezcla", + "studio_format_comic_label": "Cómic", + "studio_format_comic_tooltip": "Genera un cómic con IA a partir de tus fuentes", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Genera un documento de Word con IA a partir de tus fuentes", "studio_format_flashcards_label": "Tarjetas didácticas", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "Imagen", "studio_format_image_tooltip": "Genera una imagen con IA a partir de tus fuentes", "studio_format_infographic_label": "Infografía", - "studio_format_infographic_tooltip": "Genera una infografía con IA a partir de tus fuentes", "studio_format_mindmap_label": "Mapa mental", "studio_format_mindmap_tooltip": "Genera un mapa mental con IA a partir de tus fuentes", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "Cómo cambiar esta imagen", "studio_image_edit_instruction_placeholder": "Describe un cambio, como un fondo más oscuro", "studio_image_edit_open_aria": "Editar esta imagen", + "studio_image_edit_redraw_placeholder": "Describe un cambio: la imagen se vuelve a dibujar", "studio_image_edit_submit_button": "Editar", "studio_mindmap_viewer_fit_aria": "Ajustar el mapa mental", "studio_mindmap_viewer_tree_aria": "Mapa mental", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "No se pudo cargar el PDF", "studio_pdf_viewer_zoom_in_aria": "Acercar", "studio_pdf_viewer_zoom_out_aria": "Alejar", + "studio_picture_type_label": "Tipo", "studio_podcast_brief_add_speaker_button": "Añadir participante", "studio_podcast_brief_language_label": "Idioma", "studio_podcast_brief_length_label": "Duración", diff --git a/surfsense_local/frontend/translations/fr.json b/surfsense_local/frontend/translations/fr.json index dd780e5cf8..6d8cbcfda7 100644 --- a/surfsense_local/frontend/translations/fr.json +++ b/surfsense_local/frontend/translations/fr.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Autoriser {hosts}", "agent_steps_analyze_data_label": "A exécuté l’analyse <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "A analysé des données", + "agent_steps_bind_comic_created_label": "A créé <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Assemblage de la BD <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Assemblage d’une BD", + "agent_steps_bind_comic_updated_label": "A mis à jour <doc>{title}</doc> en v{version, number}", "agent_steps_confirm_job_label": "A confirmé des réponses suggérées dans <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "A commencé à confirmer des réponses suggérées", "agent_steps_convert_label": "<doc>{title}</doc> converti en PDF", @@ -942,7 +946,7 @@ "models_in_use_local_label": "Cet ordinateur", "models_in_use_missing_label": "Introuvable sur cet ordinateur", "models_in_use_summary_aria": "{slot, select, chat {modèle de chat actif} image {modèle d’image actif} image_edit {modèle de modification d’images actif} video {modèle vidéo actif} audio {modèle audio actif} speech {modèle de reconnaissance vocale actif} other {modèle actif}}", - "models_in_use_summary_empty": "{slot, select, chat {Aucun modèle de chat choisi} image {Aucun modèle d’image choisi} image_edit {Aucun modèle de modification d’images choisi} video {Aucun modèle vidéo choisi} audio {Aucun modèle audio choisi} speech {Aucun modèle de reconnaissance vocale choisi} other {Aucun modèle choisi}}", + "models_in_use_summary_empty": "{slot, select, chat {Aucun modèle de chat choisi} image {Aucun modèle d’image choisi} image_edit {Identique au modèle d’image} video {Aucun modèle vidéo choisi} audio {Aucun modèle audio choisi} speech {Aucun modèle de reconnaissance vocale choisi} other {Aucun modèle choisi}}", "models_in_use_summary_label": "Actif :", "models_in_use_unknown_server_label": "Un serveur", "models_install_cancelled_short_status": "Annulé", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "Comme le système", "settings_general_reduce_motion_title": "Réduire les animations", "settings_general_title": "Général", - "settings_image_edit_models_body": "Le modèle qui modifie les images. Exécutez-en un sur cet ordinateur ou utilisez-en un depuis un serveur.", - "settings_image_edit_models_title": "Modèles de modification d’images", + "settings_image_edit_models_body": "Facultatif. Votre modèle d’image modifie les images quand il le peut, et les redessine sinon. Choisissez un modèle ici uniquement pour modifier avec un autre.", + "settings_image_edit_models_title": "Utiliser un autre modèle pour les modifications", "settings_image_models_body": "Le modèle qui crée les images dans Studio. Exécutez-en un sur cet ordinateur ou utilisez-en un depuis un serveur.", "settings_image_models_title": "Modèles de génération d’images", "settings_models_add_button": "Ajouter un modèle", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "Échec de la génération de {name}", "studio_artifact_row_script_failed_tooltip": "Le script du document a échoué. Demandez à l’agent de le corriger.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Dessine {count, plural, one {# image} many {# images} other {# images}} avec votre modèle d’image. Chacune est facturée par son fournisseur, ou prend quelques minutes sur cet ordinateur.", + "studio_comic_kind_bande_dessinee_label": "BD franco-belge", + "studio_comic_kind_comic_strip_label": "Strip", + "studio_comic_kind_graphic_novel_label": "Roman graphique", + "studio_comic_kind_label": "Genre", + "studio_comic_kind_manga_label": "Manga", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Album illustré", + "studio_comic_kind_scroll_ahead_aria": "Afficher plus de genres", + "studio_comic_kind_scroll_back_aria": "Afficher les genres précédents", + "studio_comic_kind_storyboard_label": "Storyboard", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "Comics", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-koma)", + "studio_comic_pages_label": "Pages", "studio_composer_brief_status": "Préparation du brief…", "studio_composer_generate_button": "Générer", "studio_composer_prompt_label": "Prompt (facultatif)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "Un modèle de chat et un modèle audio sont requis.", "studio_error_needs_chat_image": "Un modèle de chat et un modèle d’image sont requis.", "studio_error_needs_image": "Un modèle d’image est requis.", - "studio_error_needs_image_edit": "Un modèle de modification d’images est requis.", "studio_error_no_sources": "Choisissez au moins une source.", "studio_error_not_editable": "Seule une image ou une infographie créée dans Studio peut être modifiée.", "studio_error_not_refinable": "Seul un document Word, un PDF ou une page web créé dans Studio peut être affiné.", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "Mélanger les cartes", "studio_flashcards_viewer_shuffle_button": "Mélanger", "studio_flashcards_viewer_shuffle_error": "Le mélange n’a pas pu être enregistré", + "studio_format_comic_label": "BD", + "studio_format_comic_tooltip": "Générer une BD par IA à partir de vos sources", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Générer un document Word par IA à partir de vos sources", "studio_format_flashcards_label": "Cartes mémoire", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "Image", "studio_format_image_tooltip": "Générer une image par IA à partir de vos sources", "studio_format_infographic_label": "Infographie", - "studio_format_infographic_tooltip": "Générer une infographie par IA à partir de vos sources", "studio_format_mindmap_label": "Carte mentale", "studio_format_mindmap_tooltip": "Générer une carte mentale par IA à partir de vos sources", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "Comment modifier cette image", "studio_image_edit_instruction_placeholder": "Décrivez une modification, comme un arrière-plan plus sombre", "studio_image_edit_open_aria": "Modifier cette image", + "studio_image_edit_redraw_placeholder": "Décrivez une modification : l’image est redessinée", "studio_image_edit_submit_button": "Modifier", "studio_mindmap_viewer_fit_aria": "Ajuster la carte mentale", "studio_mindmap_viewer_tree_aria": "Carte mentale", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "Impossible de charger le PDF", "studio_pdf_viewer_zoom_in_aria": "Zoom avant", "studio_pdf_viewer_zoom_out_aria": "Zoom arrière", + "studio_picture_type_label": "Type", "studio_podcast_brief_add_speaker_button": "Ajouter un intervenant", "studio_podcast_brief_language_label": "Langue", "studio_podcast_brief_length_label": "Durée", diff --git a/surfsense_local/frontend/translations/hi.json b/surfsense_local/frontend/translations/hi.json index e2eb04ec80..791e44ac52 100644 --- a/surfsense_local/frontend/translations/hi.json +++ b/surfsense_local/frontend/translations/hi.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "{hosts} को अनुमति दें", "agent_steps_analyze_data_label": "<doc>{title}</doc> विश्लेषण चलाया", "agent_steps_analyze_data_untitled_label": "डेटा का विश्लेषण किया", + "agent_steps_bind_comic_created_label": "<doc>{title}</doc> v{version, number} बनाया", + "agent_steps_bind_comic_label": "कॉमिक <doc>{title}</doc> जोड़ा जा रहा है", + "agent_steps_bind_comic_unnamed_label": "कॉमिक जोड़ा जा रहा है", + "agent_steps_bind_comic_updated_label": "<doc>{title}</doc> को v{version, number} में अपडेट किया", "agent_steps_confirm_job_label": "<doc>{title}</doc> में सुझाए गए जवाबों की पुष्टि की", "agent_steps_confirm_job_pending_label": "सुझाए गए जवाबों की पुष्टि शुरू की", "agent_steps_convert_label": "<doc>{title}</doc> को PDF में बदला", @@ -942,7 +946,7 @@ "models_in_use_local_label": "यह कंप्यूटर", "models_in_use_missing_label": "इस कंप्यूटर पर नहीं मिला", "models_in_use_summary_aria": "{slot, select, chat {उपयोग में चैट मॉडल} image {उपयोग में इमेज मॉडल} image_edit {उपयोग में इमेज एडिटिंग मॉडल} video {उपयोग में वीडियो मॉडल} audio {उपयोग में ऑडियो मॉडल} speech {उपयोग में स्पीच रिकग्निशन मॉडल} other {उपयोग में मॉडल}}", - "models_in_use_summary_empty": "{slot, select, chat {कोई चैट मॉडल नहीं चुना गया} image {कोई इमेज मॉडल नहीं चुना गया} image_edit {कोई इमेज एडिटिंग मॉडल नहीं चुना गया} video {कोई वीडियो मॉडल नहीं चुना गया} audio {कोई ऑडियो मॉडल नहीं चुना गया} speech {कोई स्पीच रिकग्निशन मॉडल नहीं चुना गया} other {कोई मॉडल नहीं चुना गया}}", + "models_in_use_summary_empty": "{slot, select, chat {कोई चैट मॉडल नहीं चुना गया} image {कोई इमेज मॉडल नहीं चुना गया} image_edit {इमेज मॉडल जैसा ही} video {कोई वीडियो मॉडल नहीं चुना गया} audio {कोई ऑडियो मॉडल नहीं चुना गया} speech {कोई स्पीच रिकग्निशन मॉडल नहीं चुना गया} other {कोई मॉडल नहीं चुना गया}}", "models_in_use_summary_label": "उपयोग में:", "models_in_use_unknown_server_label": "एक सर्वर", "models_install_cancelled_short_status": "रद्द किया गया", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "सिस्टम के अनुसार", "settings_general_reduce_motion_title": "मोशन कम करें", "settings_general_title": "सामान्य", - "settings_image_edit_models_body": "वह मॉडल जो इमेज एडिट करता है। इसे इस कंप्यूटर पर चलाएं, या किसी सर्वर से इस्तेमाल करें।", - "settings_image_edit_models_title": "इमेज एडिटिंग मॉडल", + "settings_image_edit_models_body": "वैकल्पिक। आपका इमेज मॉडल जब कर सकता है तब इमेज एडिट करता है, और नहीं कर सकता तो उन्हें फिर से बनाता है। यहाँ मॉडल तभी चुनें जब आप किसी दूसरे मॉडल से एडिट करना चाहें।", + "settings_image_edit_models_title": "एडिट के लिए दूसरा मॉडल इस्तेमाल करें", "settings_image_models_body": "वह मॉडल जो Studio में इमेज बनाता है। इसे इस कंप्यूटर पर चलाएं, या किसी सर्वर से इस्तेमाल करें।", "settings_image_models_title": "इमेज जनरेशन मॉडल", "settings_models_add_button": "मॉडल जोड़ें", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "{name} का जनरेशन विफल रहा", "studio_artifact_row_script_failed_tooltip": "दस्तावेज़ स्क्रिप्ट विफल रही। एजेंट से इसे ठीक करने को कहें।", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "आपके इमेज मॉडल से {count, plural, one {# इमेज} other {# इमेज}} बनाता है। हर इमेज का शुल्क उसका प्रोवाइडर लेता है, या इस कंप्यूटर पर उसमें कुछ मिनट लगते हैं।", + "studio_comic_kind_bande_dessinee_label": "फ़्रेंको-बेल्जियन कॉमिक", + "studio_comic_kind_comic_strip_label": "कॉमिक स्ट्रिप", + "studio_comic_kind_graphic_novel_label": "ग्राफ़िक नॉवेल", + "studio_comic_kind_label": "प्रकार", + "studio_comic_kind_manga_label": "मांगा", + "studio_comic_kind_manhua_label": "मनहुआ", + "studio_comic_kind_picture_book_label": "चित्र पुस्तक", + "studio_comic_kind_scroll_ahead_aria": "और प्रकार दिखाएं", + "studio_comic_kind_scroll_back_aria": "पिछले प्रकार दिखाएं", + "studio_comic_kind_storyboard_label": "स्टोरीबोर्ड", + "studio_comic_kind_webtoon_label": "वेबटून", + "studio_comic_kind_western_label": "पश्चिमी कॉमिक", + "studio_comic_kind_yonkoma_label": "योनकोमा (4-कोमा)", + "studio_comic_pages_label": "पेज", "studio_composer_brief_status": "ब्रीफ़ तैयार हो रहा है…", "studio_composer_generate_button": "जनरेट करें", "studio_composer_prompt_label": "प्रॉम्प्ट (वैकल्पिक)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "एक चैट मॉडल और एक ऑडियो मॉडल चाहिए।", "studio_error_needs_chat_image": "एक चैट मॉडल और एक इमेज मॉडल चाहिए।", "studio_error_needs_image": "एक इमेज मॉडल चाहिए।", - "studio_error_needs_image_edit": "एक इमेज एडिटिंग मॉडल चाहिए।", "studio_error_no_sources": "कम से कम एक स्रोत चुनें।", "studio_error_not_editable": "केवल Studio में बनी इमेज या इन्फ़ोग्राफ़िक ही एडिट की जा सकती है।", "studio_error_not_refinable": "केवल Studio में बना Word दस्तावेज़, PDF या वेब पेज ही सुधारा जा सकता है।", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "कार्ड शफ़ल करें", "studio_flashcards_viewer_shuffle_button": "शफ़ल करें", "studio_flashcards_viewer_shuffle_error": "शफ़ल सहेजा नहीं जा सका", + "studio_format_comic_label": "कॉमिक", + "studio_format_comic_tooltip": "अपने स्रोतों से AI कॉमिक जनरेट करें", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "अपने स्रोतों से AI Word दस्तावेज़ जनरेट करें", "studio_format_flashcards_label": "फ़्लैशकार्ड", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "इमेज", "studio_format_image_tooltip": "अपने स्रोतों से AI इमेज जनरेट करें", "studio_format_infographic_label": "इन्फ़ोग्राफ़िक", - "studio_format_infographic_tooltip": "अपने स्रोतों से AI इन्फ़ोग्राफ़िक जनरेट करें", "studio_format_mindmap_label": "माइंड मैप", "studio_format_mindmap_tooltip": "अपने स्रोतों से AI माइंड मैप जनरेट करें", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "इस इमेज को कैसे बदलें", "studio_image_edit_instruction_placeholder": "कोई बदलाव बताएं, जैसे गहरा बैकग्राउंड", "studio_image_edit_open_aria": "इस इमेज को एडिट करें", + "studio_image_edit_redraw_placeholder": "बदलाव बताएं: इमेज फिर से बनाई जाएगी", "studio_image_edit_submit_button": "एडिट करें", "studio_mindmap_viewer_fit_aria": "माइंड मैप फ़िट करें", "studio_mindmap_viewer_tree_aria": "माइंड मैप", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "PDF लोड नहीं हो सका", "studio_pdf_viewer_zoom_in_aria": "ज़ूम इन", "studio_pdf_viewer_zoom_out_aria": "ज़ूम आउट", + "studio_picture_type_label": "प्रकार", "studio_podcast_brief_add_speaker_button": "स्पीकर जोड़ें", "studio_podcast_brief_language_label": "भाषा", "studio_podcast_brief_length_label": "लंबाई", diff --git a/surfsense_local/frontend/translations/ja.json b/surfsense_local/frontend/translations/ja.json index 5649f0c6dc..6fff2c9f4c 100644 --- a/surfsense_local/frontend/translations/ja.json +++ b/surfsense_local/frontend/translations/ja.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "{hosts}を許可", "agent_steps_analyze_data_label": "分析<doc>{title}</doc>を実行", "agent_steps_analyze_data_untitled_label": "データを分析", + "agent_steps_bind_comic_created_label": "<doc>{title}</doc> v{version, number}を作成", + "agent_steps_bind_comic_label": "コミック<doc>{title}</doc>を製本中", + "agent_steps_bind_comic_unnamed_label": "コミックを製本中", + "agent_steps_bind_comic_updated_label": "<doc>{title}</doc>をv{version, number}に更新", "agent_steps_confirm_job_label": "<doc>{title}</doc>の回答案を確定", "agent_steps_confirm_job_pending_label": "回答案の確定を開始", "agent_steps_convert_label": "<doc>{title}</doc>をPDFに変換", @@ -942,7 +946,7 @@ "models_in_use_local_label": "このコンピューター", "models_in_use_missing_label": "このコンピューターに見つかりません", "models_in_use_summary_aria": "{slot, select, chat {使用中のチャットモデル} image {使用中の画像モデル} image_edit {使用中の画像編集モデル} video {使用中の動画モデル} audio {使用中の音声モデル} speech {使用中の音声認識モデル} other {使用中のモデル}}", - "models_in_use_summary_empty": "{slot, select, chat {チャットモデルが選択されていません} image {画像モデルが選択されていません} image_edit {画像編集モデルが選択されていません} video {動画モデルが選択されていません} audio {音声モデルが選択されていません} speech {音声認識モデルが選択されていません} other {モデルが選択されていません}}", + "models_in_use_summary_empty": "{slot, select, chat {チャットモデルが選択されていません} image {画像モデルが選択されていません} image_edit {画像モデルと同じ} video {動画モデルが選択されていません} audio {音声モデルが選択されていません} speech {音声認識モデルが選択されていません} other {モデルが選択されていません}}", "models_in_use_summary_label": "使用中:", "models_in_use_unknown_server_label": "サーバー", "models_install_cancelled_short_status": "キャンセル済み", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "システムに合わせる", "settings_general_reduce_motion_title": "動きを減らす", "settings_general_title": "一般", - "settings_image_edit_models_body": "画像を編集するモデルです。このコンピューターで実行するか、サーバー上のモデルを使います。", - "settings_image_edit_models_title": "画像編集モデル", + "settings_image_edit_models_body": "任意です。画像モデルは、可能な場合は画像を編集し、できない場合は描き直します。別のモデルで編集したい場合にのみ、ここでモデルを選択してください。", + "settings_image_edit_models_title": "編集に別のモデルを使う", "settings_image_models_body": "Studioで画像を生成するモデルです。このコンピューターで実行するか、サーバー上のモデルを使います。", "settings_image_models_title": "画像生成モデル", "settings_models_add_button": "モデルを追加", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "{name}の生成に失敗しました", "studio_artifact_row_script_failed_tooltip": "ドキュメントのスクリプトが失敗しました。エージェントに修正を依頼してください。", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "画像モデルで{count, plural, other {#枚の画像}}を描きます。1枚ごとにプロバイダーから課金されるか、このコンピューターで数分かかります。", + "studio_comic_kind_bande_dessinee_label": "バンド・デシネ", + "studio_comic_kind_comic_strip_label": "コミックストリップ", + "studio_comic_kind_graphic_novel_label": "グラフィックノベル", + "studio_comic_kind_label": "種類", + "studio_comic_kind_manga_label": "マンガ", + "studio_comic_kind_manhua_label": "マンファ(中国の漫画)", + "studio_comic_kind_picture_book_label": "絵本", + "studio_comic_kind_scroll_ahead_aria": "他の種類を表示", + "studio_comic_kind_scroll_back_aria": "前の種類を表示", + "studio_comic_kind_storyboard_label": "絵コンテ", + "studio_comic_kind_webtoon_label": "ウェブトゥーン", + "studio_comic_kind_western_label": "西洋コミック", + "studio_comic_kind_yonkoma_label": "4コマ漫画", + "studio_comic_pages_label": "ページ数", "studio_composer_brief_status": "概要を準備しています…", "studio_composer_generate_button": "生成", "studio_composer_prompt_label": "プロンプト(任意)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "チャットモデルと音声モデルが必要です。", "studio_error_needs_chat_image": "チャットモデルと画像モデルが必要です。", "studio_error_needs_image": "画像モデルが必要です。", - "studio_error_needs_image_edit": "画像編集モデルが必要です。", "studio_error_no_sources": "ソースを1つ以上選んでください。", "studio_error_not_editable": "編集できるのは、Studioで作成した画像とインフォグラフィックだけです。", "studio_error_not_refinable": "改訂できるのは、Studioで作成したWordドキュメント、PDF、Webページだけです。", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "カードをシャッフル", "studio_flashcards_viewer_shuffle_button": "シャッフル", "studio_flashcards_viewer_shuffle_error": "シャッフルを保存できませんでした", + "studio_format_comic_label": "コミック", + "studio_format_comic_tooltip": "ソースをもとにAIでコミックを生成", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "ソースをもとにAIでWord文書を生成", "studio_format_flashcards_label": "フラッシュカード", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "画像", "studio_format_image_tooltip": "ソースをもとにAIで画像を生成", "studio_format_infographic_label": "インフォグラフィック", - "studio_format_infographic_tooltip": "ソースをもとにAIでインフォグラフィックを生成", "studio_format_mindmap_label": "マインドマップ", "studio_format_mindmap_tooltip": "ソースをもとにAIでマインドマップを生成", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "この画像の変更内容", "studio_image_edit_instruction_placeholder": "変更内容を入力してください(例: 背景を暗くする)", "studio_image_edit_open_aria": "この画像を編集", + "studio_image_edit_redraw_placeholder": "変更内容を入力してください(画像は描き直されます)", "studio_image_edit_submit_button": "編集", "studio_mindmap_viewer_fit_aria": "マインドマップ全体を表示", "studio_mindmap_viewer_tree_aria": "マインドマップ", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "PDFを読み込めませんでした", "studio_pdf_viewer_zoom_in_aria": "拡大", "studio_pdf_viewer_zoom_out_aria": "縮小", + "studio_picture_type_label": "種類", "studio_podcast_brief_add_speaker_button": "話者を追加", "studio_podcast_brief_language_label": "言語", "studio_podcast_brief_length_label": "長さ", diff --git a/surfsense_local/frontend/translations/ko.json b/surfsense_local/frontend/translations/ko.json index 3adb0897b6..1bbf147437 100644 --- a/surfsense_local/frontend/translations/ko.json +++ b/surfsense_local/frontend/translations/ko.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "{hosts} 허용", "agent_steps_analyze_data_label": "<doc>{title}</doc> 분석 실행함", "agent_steps_analyze_data_untitled_label": "데이터를 분석함", + "agent_steps_bind_comic_created_label": "<doc>{title}</doc> v{version, number} 생성함", + "agent_steps_bind_comic_label": "만화 <doc>{title}</doc> 제본 중", + "agent_steps_bind_comic_unnamed_label": "만화 제본 중", + "agent_steps_bind_comic_updated_label": "<doc>{title}</doc>을(를) v{version, number}(으)로 업데이트함", "agent_steps_confirm_job_label": "<doc>{title}</doc>의 제안된 답변을 확정함", "agent_steps_confirm_job_pending_label": "제안된 답변 확정을 시작함", "agent_steps_convert_label": "<doc>{title}</doc>을(를) PDF로 변환", @@ -942,7 +946,7 @@ "models_in_use_local_label": "이 컴퓨터", "models_in_use_missing_label": "이 컴퓨터에서 찾을 수 없음", "models_in_use_summary_aria": "{slot, select, chat {사용 중인 채팅 모델} image {사용 중인 이미지 모델} image_edit {사용 중인 이미지 편집 모델} video {사용 중인 동영상 모델} audio {사용 중인 음성 모델} speech {사용 중인 음성 인식 모델} other {사용 중인 모델}}", - "models_in_use_summary_empty": "{slot, select, chat {선택된 채팅 모델이 없습니다} image {선택된 이미지 모델이 없습니다} image_edit {선택된 이미지 편집 모델이 없습니다} video {선택된 동영상 모델이 없습니다} audio {선택된 음성 모델이 없습니다} speech {선택된 음성 인식 모델이 없습니다} other {선택된 모델이 없습니다}}", + "models_in_use_summary_empty": "{slot, select, chat {선택된 채팅 모델이 없습니다} image {선택된 이미지 모델이 없습니다} image_edit {이미지 모델과 동일} video {선택된 동영상 모델이 없습니다} audio {선택된 음성 모델이 없습니다} speech {선택된 음성 인식 모델이 없습니다} other {선택된 모델이 없습니다}}", "models_in_use_summary_label": "사용 중:", "models_in_use_unknown_server_label": "서버", "models_install_cancelled_short_status": "취소됨", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "시스템에 맞춤", "settings_general_reduce_motion_title": "동작 줄이기", "settings_general_title": "일반", - "settings_image_edit_models_body": "이미지를 편집하는 모델입니다. 이 컴퓨터에서 실행하거나, 서버의 모델을 사용하세요.", - "settings_image_edit_models_title": "이미지 편집 모델", + "settings_image_edit_models_body": "선택 사항입니다. 이미지 모델은 가능하면 이미지를 편집하고, 그렇지 않으면 다시 그립니다. 다른 모델로 편집하려는 경우에만 여기에서 모델을 선택하세요.", + "settings_image_edit_models_title": "편집에 다른 모델 사용", "settings_image_models_body": "Studio에서 이미지를 만드는 모델입니다. 이 컴퓨터에서 실행하거나, 서버의 모델을 사용하세요.", "settings_image_models_title": "이미지 생성 모델", "settings_models_add_button": "모델 추가", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "{name} 생성에 실패했습니다", "studio_artifact_row_script_failed_tooltip": "문서 스크립트가 실패했습니다. 에이전트에게 수정을 요청하세요.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "이미지 모델로 {count, plural, other {이미지 #장}}을 그립니다. 한 장마다 제공업체가 요금을 청구하거나, 이 컴퓨터에서 몇 분이 걸립니다.", + "studio_comic_kind_bande_dessinee_label": "방드 데시네", + "studio_comic_kind_comic_strip_label": "신문 만화", + "studio_comic_kind_graphic_novel_label": "그래픽 노블", + "studio_comic_kind_label": "종류", + "studio_comic_kind_manga_label": "일본 만화", + "studio_comic_kind_manhua_label": "중국 만화", + "studio_comic_kind_picture_book_label": "그림책", + "studio_comic_kind_scroll_ahead_aria": "더 많은 종류 보기", + "studio_comic_kind_scroll_back_aria": "이전 종류 보기", + "studio_comic_kind_storyboard_label": "스토리보드", + "studio_comic_kind_webtoon_label": "웹툰", + "studio_comic_kind_western_label": "서양 만화", + "studio_comic_kind_yonkoma_label": "4컷 만화", + "studio_comic_pages_label": "페이지 수", "studio_composer_brief_status": "개요를 준비하는 중…", "studio_composer_generate_button": "생성", "studio_composer_prompt_label": "프롬프트 (선택 사항)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "채팅 모델과 음성 모델이 필요합니다.", "studio_error_needs_chat_image": "채팅 모델과 이미지 모델이 필요합니다.", "studio_error_needs_image": "이미지 모델이 필요합니다.", - "studio_error_needs_image_edit": "이미지 편집 모델이 필요합니다.", "studio_error_no_sources": "소스를 하나 이상 선택하세요.", "studio_error_not_editable": "Studio에서 만든 이미지와 인포그래픽만 편집할 수 있습니다.", "studio_error_not_refinable": "Studio에서 만든 Word 문서, PDF, 웹 페이지만 다듬을 수 있습니다.", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "카드 섞기", "studio_flashcards_viewer_shuffle_button": "섞기", "studio_flashcards_viewer_shuffle_error": "섞기를 저장하지 못했습니다", + "studio_format_comic_label": "만화", + "studio_format_comic_tooltip": "소스를 바탕으로 AI 만화 생성", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "소스를 바탕으로 AI Word 문서 생성", "studio_format_flashcards_label": "플래시카드", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "이미지", "studio_format_image_tooltip": "소스를 바탕으로 AI 이미지 생성", "studio_format_infographic_label": "인포그래픽", - "studio_format_infographic_tooltip": "소스를 바탕으로 AI 인포그래픽 생성", "studio_format_mindmap_label": "마인드맵", "studio_format_mindmap_tooltip": "소스를 바탕으로 AI 마인드맵 생성", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "이 이미지를 바꿀 방법", "studio_image_edit_instruction_placeholder": "배경을 어둡게 하는 등 바꿀 내용을 설명하세요", "studio_image_edit_open_aria": "이 이미지 편집", + "studio_image_edit_redraw_placeholder": "변경 사항을 입력하세요. 이미지가 다시 그려집니다", "studio_image_edit_submit_button": "편집", "studio_mindmap_viewer_fit_aria": "마인드맵 맞춤", "studio_mindmap_viewer_tree_aria": "마인드맵", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "PDF를 불러오지 못했습니다", "studio_pdf_viewer_zoom_in_aria": "확대", "studio_pdf_viewer_zoom_out_aria": "축소", + "studio_picture_type_label": "유형", "studio_podcast_brief_add_speaker_button": "화자 추가", "studio_podcast_brief_language_label": "언어", "studio_podcast_brief_length_label": "길이", diff --git a/surfsense_local/frontend/translations/pt-BR.json b/surfsense_local/frontend/translations/pt-BR.json index 5b1c12dc89..51227e7f40 100644 --- a/surfsense_local/frontend/translations/pt-BR.json +++ b/surfsense_local/frontend/translations/pt-BR.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Permitir {hosts}", "agent_steps_analyze_data_label": "Executou a análise <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "Analisou dados", + "agent_steps_bind_comic_created_label": "Criou <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Montando os quadrinhos <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Montando quadrinhos", + "agent_steps_bind_comic_updated_label": "Atualizou <doc>{title}</doc> para a v{version, number}", "agent_steps_confirm_job_label": "Confirmou respostas sugeridas em <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "Começou a confirmar respostas sugeridas", "agent_steps_convert_label": "Converteu <doc>{title}</doc> em PDF", @@ -942,7 +946,7 @@ "models_in_use_local_label": "Este computador", "models_in_use_missing_label": "Não encontrado neste computador", "models_in_use_summary_aria": "{slot, select, chat {modelo de chat em uso} image {modelo de imagem em uso} image_edit {modelo de edição de imagem em uso} video {modelo de vídeo em uso} audio {modelo de áudio em uso} speech {modelo de reconhecimento de voz em uso} other {modelo em uso}}", - "models_in_use_summary_empty": "{slot, select, chat {Nenhum modelo de chat escolhido} image {Nenhum modelo de imagem escolhido} image_edit {Nenhum modelo de edição de imagem escolhido} video {Nenhum modelo de vídeo escolhido} audio {Nenhum modelo de áudio escolhido} speech {Nenhum modelo de reconhecimento de voz escolhido} other {Nenhum modelo escolhido}}", + "models_in_use_summary_empty": "{slot, select, chat {Nenhum modelo de chat escolhido} image {Nenhum modelo de imagem escolhido} image_edit {Igual ao modelo de imagem} video {Nenhum modelo de vídeo escolhido} audio {Nenhum modelo de áudio escolhido} speech {Nenhum modelo de reconhecimento de voz escolhido} other {Nenhum modelo escolhido}}", "models_in_use_summary_label": "Em uso:", "models_in_use_unknown_server_label": "Um servidor", "models_install_cancelled_short_status": "Cancelado", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "Igual ao sistema", "settings_general_reduce_motion_title": "Reduzir movimento", "settings_general_title": "Geral", - "settings_image_edit_models_body": "O modelo que edita imagens. Execute um neste computador ou use um de um servidor.", - "settings_image_edit_models_title": "Modelos de edição de imagem", + "settings_image_edit_models_body": "Opcional. Seu modelo de imagem edita as imagens quando consegue e, quando não consegue, desenha de novo. Escolha um modelo aqui só para editar com outro.", + "settings_image_edit_models_title": "Usar outro modelo para edições", "settings_image_models_body": "O modelo que cria imagens no Studio. Execute um neste computador ou use um de um servidor.", "settings_image_models_title": "Modelos de geração de imagem", "settings_models_add_button": "Adicionar modelo", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "Falha na geração de {name}", "studio_artifact_row_script_failed_tooltip": "O script do documento falhou. Peça ao agente para corrigi-lo.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Desenha {count, plural, one {# imagem} many {# imagens} other {# imagens}} com seu modelo de imagem. Cada uma é cobrada pelo provedor, ou leva alguns minutos neste computador.", + "studio_comic_kind_bande_dessinee_label": "Quadrinho franco-belga", + "studio_comic_kind_comic_strip_label": "Tirinha", + "studio_comic_kind_graphic_novel_label": "Romance gráfico", + "studio_comic_kind_label": "Tipo", + "studio_comic_kind_manga_label": "Mangá", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Livro ilustrado", + "studio_comic_kind_scroll_ahead_aria": "Mostrar mais tipos", + "studio_comic_kind_scroll_back_aria": "Mostrar tipos anteriores", + "studio_comic_kind_storyboard_label": "Storyboard", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "HQ ocidental", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-koma)", + "studio_comic_pages_label": "Páginas", "studio_composer_brief_status": "Preparando o briefing…", "studio_composer_generate_button": "Gerar", "studio_composer_prompt_label": "Prompt (opcional)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "São necessários um modelo de chat e um modelo de áudio.", "studio_error_needs_chat_image": "São necessários um modelo de chat e um modelo de imagem.", "studio_error_needs_image": "É necessário um modelo de imagem.", - "studio_error_needs_image_edit": "É necessário um modelo de edição de imagem.", "studio_error_no_sources": "Escolha pelo menos uma fonte.", "studio_error_not_editable": "Só é possível editar uma imagem ou um infográfico criados no Studio.", "studio_error_not_refinable": "Só é possível refinar um documento do Word, um PDF ou uma página web criados no Studio.", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "Embaralhar os cartões", "studio_flashcards_viewer_shuffle_button": "Embaralhar", "studio_flashcards_viewer_shuffle_error": "Não foi possível salvar o embaralhamento", + "studio_format_comic_label": "Quadrinhos", + "studio_format_comic_tooltip": "Gerar quadrinhos com IA a partir das suas fontes", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Gerar um documento do Word com IA a partir das suas fontes", "studio_format_flashcards_label": "Flashcards", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "Imagem", "studio_format_image_tooltip": "Gerar uma imagem com IA a partir das suas fontes", "studio_format_infographic_label": "Infográfico", - "studio_format_infographic_tooltip": "Gerar um infográfico com IA a partir das suas fontes", "studio_format_mindmap_label": "Mapa mental", "studio_format_mindmap_tooltip": "Gerar um mapa mental com IA a partir das suas fontes", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "Como alterar esta imagem", "studio_image_edit_instruction_placeholder": "Descreva uma alteração, como um fundo mais escuro", "studio_image_edit_open_aria": "Editar esta imagem", + "studio_image_edit_redraw_placeholder": "Descreva uma mudança: a imagem será desenhada de novo", "studio_image_edit_submit_button": "Editar", "studio_mindmap_viewer_fit_aria": "Ajustar o mapa mental", "studio_mindmap_viewer_tree_aria": "Mapa mental", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "Não foi possível carregar o PDF", "studio_pdf_viewer_zoom_in_aria": "Ampliar", "studio_pdf_viewer_zoom_out_aria": "Reduzir", + "studio_picture_type_label": "Tipo", "studio_podcast_brief_add_speaker_button": "Adicionar locutor", "studio_podcast_brief_language_label": "Idioma", "studio_podcast_brief_length_label": "Duração", diff --git a/surfsense_local/frontend/translations/ru.json b/surfsense_local/frontend/translations/ru.json index eaefde430c..2fb16f6976 100644 --- a/surfsense_local/frontend/translations/ru.json +++ b/surfsense_local/frontend/translations/ru.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Разрешить {hosts}", "agent_steps_analyze_data_label": "Выполнил анализ <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "Проанализировал данные", + "agent_steps_bind_comic_created_label": "Создал <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Сборка комикса <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Сборка комикса", + "agent_steps_bind_comic_updated_label": "Обновил <doc>{title}</doc> до v{version, number}", "agent_steps_confirm_job_label": "Подтвердил предложенные ответы в <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "Начал подтверждать предложенные ответы", "agent_steps_convert_label": "Преобразовал <doc>{title}</doc> в PDF", @@ -942,7 +946,7 @@ "models_in_use_local_label": "Этот компьютер", "models_in_use_missing_label": "Не найдена на этом компьютере", "models_in_use_summary_aria": "{slot, select, chat {используемая модель для чата} image {используемая модель для изображений} image_edit {используемая модель для редактирования изображений} video {используемая модель для видео} audio {используемая модель для аудио} speech {используемая модель распознавания речи} other {используемая модель}}", - "models_in_use_summary_empty": "{slot, select, chat {Модель для чата не выбрана} image {Модель для изображений не выбрана} image_edit {Модель для редактирования изображений не выбрана} video {Модель для видео не выбрана} audio {Модель для аудио не выбрана} speech {Модель распознавания речи не выбрана} other {Модель не выбрана}}", + "models_in_use_summary_empty": "{slot, select, chat {Модель для чата не выбрана} image {Модель для изображений не выбрана} image_edit {Как у модели изображений} video {Модель для видео не выбрана} audio {Модель для аудио не выбрана} speech {Модель распознавания речи не выбрана} other {Модель не выбрана}}", "models_in_use_summary_label": "Используется:", "models_in_use_unknown_server_label": "Сервер", "models_install_cancelled_short_status": "Отменено", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "Как в системе", "settings_general_reduce_motion_title": "Уменьшить движение", "settings_general_title": "Общие", - "settings_image_edit_models_body": "Модель, которая редактирует изображения. Запустите её на этом компьютере или используйте модель с сервера.", - "settings_image_edit_models_title": "Модели редактирования изображений", + "settings_image_edit_models_body": "Необязательно. Ваша модель изображений редактирует изображения, когда может, а иначе рисует их заново. Выбирайте модель здесь, только чтобы редактировать другой моделью.", + "settings_image_edit_models_title": "Другая модель для редактирования", "settings_image_models_body": "Модель, которая создаёт изображения в Studio. Запустите её на этом компьютере или используйте модель с сервера.", "settings_image_models_title": "Модели генерации изображений", "settings_models_add_button": "Добавить модель", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "Не удалось создать {name}", "studio_artifact_row_script_failed_tooltip": "Скрипт документа не сработал. Попросите агента исправить его.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Рисует {count, plural, one {# изображение} few {# изображения} many {# изображений} other {# изображения}} вашей моделью изображений. Каждое оплачивается у провайдера или занимает несколько минут на этом компьютере.", + "studio_comic_kind_bande_dessinee_label": "Франко-бельгийский комикс", + "studio_comic_kind_comic_strip_label": "Комикс-стрип", + "studio_comic_kind_graphic_novel_label": "Графический роман", + "studio_comic_kind_label": "Вид", + "studio_comic_kind_manga_label": "Манга", + "studio_comic_kind_manhua_label": "Маньхуа", + "studio_comic_kind_picture_book_label": "Книжка с картинками", + "studio_comic_kind_scroll_ahead_aria": "Показать больше видов", + "studio_comic_kind_scroll_back_aria": "Показать предыдущие виды", + "studio_comic_kind_storyboard_label": "Раскадровка", + "studio_comic_kind_webtoon_label": "Вебтун", + "studio_comic_kind_western_label": "Западный комикс", + "studio_comic_kind_yonkoma_label": "Ёнкома (4-кома)", + "studio_comic_pages_label": "Страницы", "studio_composer_brief_status": "Подготовка брифа…", "studio_composer_generate_button": "Создать", "studio_composer_prompt_label": "Промпт (необязательно)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "Требуются модель для чата и модель для аудио.", "studio_error_needs_chat_image": "Требуются модель для чата и модель для изображений.", "studio_error_needs_image": "Требуется модель для изображений.", - "studio_error_needs_image_edit": "Требуется модель редактирования изображений.", "studio_error_no_sources": "Выберите хотя бы один источник.", "studio_error_not_editable": "Изменить можно только изображение или инфографику, созданные в Studio.", "studio_error_not_refinable": "Доработать можно только документ Word, PDF или веб-страницу, созданные в Studio.", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "Перемешать карточки", "studio_flashcards_viewer_shuffle_button": "Перемешать", "studio_flashcards_viewer_shuffle_error": "Не удалось сохранить перемешивание", + "studio_format_comic_label": "Комикс", + "studio_format_comic_tooltip": "Создать с помощью ИИ комикс на основе ваших источников", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Создать с помощью ИИ документ Word на основе ваших источников", "studio_format_flashcards_label": "Карточки", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "Изображение", "studio_format_image_tooltip": "Создать с помощью ИИ изображение на основе ваших источников", "studio_format_infographic_label": "Инфографика", - "studio_format_infographic_tooltip": "Создать с помощью ИИ инфографику на основе ваших источников", "studio_format_mindmap_label": "Интеллект-карта", "studio_format_mindmap_tooltip": "Создать с помощью ИИ интеллект-карту на основе ваших источников", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "Как изменить это изображение", "studio_image_edit_instruction_placeholder": "Опишите изменение, например более тёмный фон", "studio_image_edit_open_aria": "Изменить изображение", + "studio_image_edit_redraw_placeholder": "Опишите изменение: изображение будет нарисовано заново", "studio_image_edit_submit_button": "Изменить", "studio_mindmap_viewer_fit_aria": "Вписать интеллект-карту", "studio_mindmap_viewer_tree_aria": "Интеллект-карта", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "Не удалось загрузить PDF", "studio_pdf_viewer_zoom_in_aria": "Увеличить", "studio_pdf_viewer_zoom_out_aria": "Уменьшить", + "studio_picture_type_label": "Тип", "studio_podcast_brief_add_speaker_button": "Добавить спикера", "studio_podcast_brief_language_label": "Язык", "studio_podcast_brief_length_label": "Длительность", diff --git a/surfsense_local/frontend/translations/zh-CN.json b/surfsense_local/frontend/translations/zh-CN.json index 59e16c69fe..376fbfa97a 100644 --- a/surfsense_local/frontend/translations/zh-CN.json +++ b/surfsense_local/frontend/translations/zh-CN.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "允许 {hosts}", "agent_steps_analyze_data_label": "运行了分析 <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "分析了数据", + "agent_steps_bind_comic_created_label": "创建了 <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "正在装订漫画 <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "正在装订漫画", + "agent_steps_bind_comic_updated_label": "将 <doc>{title}</doc> 更新到 v{version, number}", "agent_steps_confirm_job_label": "确认了 <doc>{title}</doc> 中的建议答案", "agent_steps_confirm_job_pending_label": "开始确认建议答案", "agent_steps_convert_label": "已将 <doc>{title}</doc> 转换为 PDF", @@ -942,7 +946,7 @@ "models_in_use_local_label": "这台电脑", "models_in_use_missing_label": "在这台电脑上找不到", "models_in_use_summary_aria": "{slot, select, chat {使用中的对话模型} image {使用中的图像模型} image_edit {使用中的图像编辑模型} video {使用中的视频模型} audio {使用中的语音模型} speech {使用中的语音识别模型} other {使用中的模型}}", - "models_in_use_summary_empty": "{slot, select, chat {未选择对话模型} image {未选择图像模型} image_edit {未选择图像编辑模型} video {未选择视频模型} audio {未选择语音模型} speech {未选择语音识别模型} other {未选择模型}}", + "models_in_use_summary_empty": "{slot, select, chat {未选择对话模型} image {未选择图像模型} image_edit {与图像模型相同} video {未选择视频模型} audio {未选择语音模型} speech {未选择语音识别模型} other {未选择模型}}", "models_in_use_summary_label": "使用中:", "models_in_use_unknown_server_label": "某个服务器", "models_install_cancelled_short_status": "已取消", @@ -1586,8 +1590,8 @@ "settings_general_reduce_motion_system_label": "跟随系统", "settings_general_reduce_motion_title": "减弱动态效果", "settings_general_title": "通用", - "settings_image_edit_models_body": "编辑图像的模型。可以在这台电脑上运行一个,也可以使用服务器上的。", - "settings_image_edit_models_title": "图像编辑模型", + "settings_image_edit_models_body": "可选。你的图像模型在可以时编辑图像,不能时重新绘制。只有想用其他模型编辑时,才需要在这里选择模型。", + "settings_image_edit_models_title": "使用其他模型进行编辑", "settings_image_models_body": "在 Studio 中生成图像的模型。可以在这台电脑上运行一个,也可以使用服务器上的。", "settings_image_models_title": "图像生成模型", "settings_models_add_button": "添加模型", @@ -1891,6 +1895,21 @@ "studio_artifact_row_script_failed_aria": "{name} 生成失败", "studio_artifact_row_script_failed_tooltip": "文档脚本失败。请让智能体修复它。", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "用你的图像模型绘制 {count, plural, other {# 张图片}}。每张由提供商计费,或在这台电脑上需要几分钟。", + "studio_comic_kind_bande_dessinee_label": "法比漫画", + "studio_comic_kind_comic_strip_label": "漫画条", + "studio_comic_kind_graphic_novel_label": "图像小说", + "studio_comic_kind_label": "类型", + "studio_comic_kind_manga_label": "日本漫画", + "studio_comic_kind_manhua_label": "国漫", + "studio_comic_kind_picture_book_label": "绘本", + "studio_comic_kind_scroll_ahead_aria": "显示更多类型", + "studio_comic_kind_scroll_back_aria": "显示前面的类型", + "studio_comic_kind_storyboard_label": "分镜", + "studio_comic_kind_webtoon_label": "条漫", + "studio_comic_kind_western_label": "欧美漫画", + "studio_comic_kind_yonkoma_label": "四格漫画", + "studio_comic_pages_label": "页数", "studio_composer_brief_status": "正在准备播客设定…", "studio_composer_generate_button": "生成", "studio_composer_prompt_label": "提示词(可选)", @@ -1922,7 +1941,6 @@ "studio_error_needs_chat_audio": "需要对话模型和语音模型。", "studio_error_needs_chat_image": "需要对话模型和图像模型。", "studio_error_needs_image": "需要图像模型。", - "studio_error_needs_image_edit": "需要图像编辑模型。", "studio_error_no_sources": "请至少选择一个来源。", "studio_error_not_editable": "只能编辑在 Studio 中创建的图像或信息图。", "studio_error_not_refinable": "只能完善在 Studio 中创建的 Word 文档、PDF 或网页。", @@ -1958,6 +1976,8 @@ "studio_flashcards_viewer_shuffle_aria": "打乱卡片", "studio_flashcards_viewer_shuffle_button": "打乱", "studio_flashcards_viewer_shuffle_error": "无法保存打乱结果", + "studio_format_comic_label": "漫画", + "studio_format_comic_tooltip": "根据你的来源用 AI 生成漫画", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "根据你的来源用 AI 生成 Word 文档", "studio_format_flashcards_label": "记忆卡片", @@ -1968,7 +1988,6 @@ "studio_format_image_label": "图像", "studio_format_image_tooltip": "根据你的来源用 AI 生成图像", "studio_format_infographic_label": "信息图", - "studio_format_infographic_tooltip": "根据你的来源用 AI 生成信息图", "studio_format_mindmap_label": "思维导图", "studio_format_mindmap_tooltip": "根据你的来源用 AI 生成思维导图", "studio_format_pdf_label": "PDF", @@ -1994,6 +2013,7 @@ "studio_image_edit_instruction_aria": "如何修改此图像", "studio_image_edit_instruction_placeholder": "描述一处修改,例如让背景更暗", "studio_image_edit_open_aria": "编辑此图像", + "studio_image_edit_redraw_placeholder": "描述更改:图像将重新绘制", "studio_image_edit_submit_button": "编辑", "studio_mindmap_viewer_fit_aria": "缩放以适应思维导图", "studio_mindmap_viewer_tree_aria": "思维导图", @@ -2005,6 +2025,7 @@ "studio_pdf_viewer_unknown_error": "无法加载 PDF", "studio_pdf_viewer_zoom_in_aria": "放大", "studio_pdf_viewer_zoom_out_aria": "缩小", + "studio_picture_type_label": "类型", "studio_podcast_brief_add_speaker_button": "添加说话人", "studio_podcast_brief_language_label": "语言", "studio_podcast_brief_length_label": "时长", diff --git a/www/content/docs/v2/agent.mdx b/www/content/docs/v2/agent.mdx index 06071820ac..7ca2d9c944 100644 --- a/www/content/docs/v2/agent.mdx +++ b/www/content/docs/v2/agent.mdx @@ -31,12 +31,14 @@ Documents the agent made are changed in its chat, not with Studio's **Refine**. ### Pictures -Ask for a picture, an infographic or a document with a picture in it. The agent reads what it needs from your sources, writes the prompt itself and draws with the model in **Settings › Image gen**. To change a picture, ask in the same chat, which uses **Settings › Image edit** and makes its next version. Each picture appears in [Studio](/docs/v2/studio). +Ask for a picture, an infographic or a document with a picture in it. The agent reads what it needs from your sources, writes the prompt itself and draws with the model in **Settings › Image gen**. To change a picture, ask in the same chat, and it makes its next version. Each picture appears in [Studio](/docs/v2/studio). + +Ask for a comic, a manga, a webtoon, a comic strip, a storyboard or a picture book and the agent plans the pages, tells you how many it will draw, and asks once before it starts. It draws one picture per page, keeps the characters from page to page, and binds the pages into one comic in Studio, a PDF. Studio lists the comic, not its pages; deleting the comic deletes them too. To change a page, ask, and it redraws that page and binds the comic again. With an image model that can't take a picture in, each page is drawn from a description of the characters, so they may look a little different from page to page. - **It draws only when you ask.** A remote model bills each picture. A model on a computer without a graphics card takes several minutes each. -- **A slow picture may finish after the agent replies.** It appears in Studio when it's ready. To place it in a document, ask again in the same chat. +- **A slow picture may finish after the agent replies.** It appears in Studio when it's ready. A document that uses it, such as a comic, waits for it and appears in Studio once its pictures are drawn. - **Pictures used in a document stay until the document is gone.** If the agent placed a picture in a Word document, PDF, deck or workbook, delete that document before deleting the picture, because SurfSense needs the picture to make the document again. -- **Exact words go in the document.** Image models can garble lettering, so check any words in a picture. Data charts stay real charts. +- **Exact words go in the document.** Image models can garble lettering, so check any words in a picture. When a comic page's speech bubbles come out wrong, the comic prints that page's lines under it. Data charts stay real charts. ### Start from your own template diff --git a/www/content/docs/v2/guides/images.mdx b/www/content/docs/v2/guides/images.mdx index 229537cbdd..dd1e6a74cc 100644 --- a/www/content/docs/v2/guides/images.mdx +++ b/www/content/docs/v2/guides/images.mdx @@ -7,7 +7,7 @@ import { Step, Steps } from 'fumadocs-ui/components/steps'; <GuideVideo slug="images" poster="/guides/images/poster.webp" title="Downloading an image model, drawing a picture and editing it" /> -**Image** and **Infographic** in Studio need an image model. A model on your computer works offline and runs only while Studio needs it; with a graphics card a picture takes seconds, without one several minutes. See [Image models](/docs/v2/studio#image-models). +**Image** in Studio needs an image model. A model on your computer works offline and runs only while Studio needs it; with a graphics card a picture takes seconds, without one several minutes. See [Image models](/docs/v2/studio#image-models). <Steps> <Step> @@ -32,7 +32,7 @@ When the download is done, click **Use**. FLUX.2 klein 4B both draws and edits f ### Draw a picture -In Studio, click **Image** or **Infographic**, describe what you want under **Prompt (optional)**, and click **Generate**. Open it when it's ready. +In Studio, click **Image**. Under **Type**, keep **Image**, or choose **Infographic** for a picture that lays out your sources' key points. Describe what you want under **Prompt (optional)**, and click **Generate**. Open it when it's ready. ![The picture Studio drew, open in the right panel](/guides/images/03-picture.webp) diff --git a/www/content/docs/v2/studio.mdx b/www/content/docs/v2/studio.mdx index dc5d40dee7..6c29348b00 100644 --- a/www/content/docs/v2/studio.mdx +++ b/www/content/docs/v2/studio.mdx @@ -32,8 +32,8 @@ The result appears in the **Artifacts** list below, and a notice says when it's | **Flashcards** | A deck you go through one card at a time, marking **Got it** or **Needs review**, with **Shuffle** | none | | **Quiz** | Questions with explanations, a score, and **Retake quiz** for all or only the missed questions | none | | **Podcast** | An audio episode with its transcript | `.mp3` | -| **Image** | A picture drawn from your sources | the image | -| **Infographic** | A picture that lays out your sources' key points | the image | +| **Image** | A picture drawn from your sources; choose **Infographic** under **Type** for one that lays out their key points | the image | +| **Comic** | A comic, manga, webtoon, strip, picture book or storyboard drawn from your sources, its pages in one PDF | `.pdf` | Word, Slides and Spreadsheet files are real Office files: open and edit them in Word, PowerPoint, Excel or LibreOffice. Flashcard marks and quiz answers are saved as you go. @@ -42,7 +42,7 @@ Word, Slides and Spreadsheet files are real Office files: open and edit them in Every format needs a chat model. A format that can't run says what it needs, such as "Needs an image model." - **Podcast** also needs an audio model. SurfSense ships one and picks it for you. -- **Image** and **Infographic** also need an image model. See [Image models](#image-models). +- **Image**, of either type, and **Comic** also need an image model. See [Image models](#image-models). Studio uses the chat model selected in Settings at the time you click **Generate**. Strong models make Word documents and PDFs with richer layout; small local models make simpler ones from Markdown. @@ -59,6 +59,15 @@ Click **Podcast** to open its brief before you generate: See [Voice](/docs/v2/voice#podcast-voices) for voices and audio models. +### Comics + +Click **Comic**, then choose: + +- **Kind**: Western comic, graphic novel, bande dessinée, manga, manhua, webtoon, comic strip, yonkoma, picture book or storyboard. A manga reads right to left, and a webtoon scrolls down one long page. +- **Pages**: 1 to 12. Each kind starts at its own length, such as 4 pages, or 8 for a webtoon. + +Studio draws one picture per page, keeping the characters from the first page. Each picture is billed by your image model's provider, or takes minutes on this computer. With a model on this computer, the pages are drawn without words and every line is printed under its page. To change a comic, regenerate it, which draws every page again, or ask the agent, which can change one page. + ## Open, download and delete Click an artifact to open it in the right panel. @@ -92,11 +101,11 @@ Documents the agent made are changed in its chat instead: "The agent made this d ## Edit an image -Over a ready image or infographic, the same round button opens an edit box. Describe a change, such as "a darker background" or a new title, and click **Edit**. Studio sends the picture and your instruction to the image editing model, and what comes back is the next version. +Over a ready image or infographic, the same round button opens an edit box. Describe a change, such as "a darker background" or a new title, and click **Edit**. Studio sends the picture and your instruction to your image model, and what comes back is the next version. Each edit starts from the version you have open, so edits chain, and every earlier version stays. There's no way to select one region of the picture yet: an edit applies to the whole image. -Without an image editing model, the box says "Needs an image editing model." with **Choose a model**. +Some image models can't take a picture in. With one of those, the box says "Describe a change: the picture is drawn again": Studio draws the picture again from what it was drawn from and your change, so the rest of the picture may change too. Image models that edit are marked **Image editing** in Settings. <Callout type="info"> Editing an image on a computer without a graphics card is slow. In testing, @@ -108,7 +117,7 @@ Without an image editing model, the box says "Needs an image editing model." wit Choose them in Settings: - **Settings › Image gen** for drawing. Run a model on this computer, such as FLUX.2 klein 4B, Z-Image Turbo or Stable Diffusion XL, or use one from an API connection. -- **Settings › Image edit** for editing. FLUX.2 klein 4B both draws and edits from one download. A remote model that takes a picture in and gives one back works too. +- **Settings › Image edit** is optional: your image model edits pictures when it can, and draws them again when it can't. Choose a model there only to edit with a different one. FLUX.2 klein 4B both draws and edits from one download, and most remote image models take a picture in. A local image model runs only while Studio needs it, and works offline. It frees its memory a few minutes after the last image. See [Models](/docs/v2/models).