From a7022d11ae3596c7433049faa81814315b3f2e42 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Tue, 10 Nov 2026 22:14:47 +0530 Subject: [PATCH 01/33] docs(proposals): comics the agent draws as pages bound into a PDF --- docs/proposals/comics.md | 157 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 docs/proposals/comics.md diff --git a/docs/proposals/comics.md b/docs/proposals/comics.md new file mode 100644 index 0000000000..720a6d96a6 --- /dev/null +++ b/docs/proposals/comics.md @@ -0,0 +1,157 @@ +--- +status: proposed +code: + - surfsense_local/backend/modules/agent/prompts/agent.md + - surfsense_local/backend/modules/agent/skills/surfsense-comics/ + - surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md + - surfsense_local/backend/modules/agent/tool_endpoint/make_image.py + - surfsense_local/backend/modules/agent/tool_endpoint/render_document.py + - surfsense_local/backend/modules/artifacts/drawn_images/ + - surfsense_local/backend/modules/artifacts/script_documents/ + - surfsense_local/backend/modules/llm/providers/ + - surfsense_local/backend/worker/studio/ +--- + +# Comics the agent draws + +> Asked for a comic, the agent plans its pages, confirms the count once, draws one picture per page with the user's image model, keeping its characters the same from page to page, and binds the pages into one PDF. A local model's slow pages no longer leave the comic for the user's next message: the PDF renders itself once its last page is drawn. + +## Today + +Asked for a comic, the agent wrote a web page and drew the panels as SVG. When it did call `surfsense_make_image`, it drew one picture and stopped. Nothing in the agent's text makes a comic a picture job, and two lines push it away: + +- The prompt sends only "a picture or infographic" to `surfsense_make_image` ([`agent.md`](../../surfsense_local/backend/modules/agent/prompts/agent.md)). A comic is neither, and the web page skill's description, "an explainer … a figure that shows how something works", drawn "as inline SVG" ([`SKILL.md`](../../surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md)), is the nearest match. +- The tool's description says "confirm before making more than one" and "words that must be exact belong in the document around the picture" ([`make_image.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/make_image.py)). A comic is many pictures, mostly dialogue, so both read as "do not draw this". + +Three limits stay even with the wording fixed: + +- **Characters drift.** Every picture is drawn from its prompt alone. The same character sheet in six prompts gives six different faces. +- **Every picture is the model's default shape**, usually square. Neither the remote request ([`image.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image.py)) nor the local one sends a size, and a comic page is portrait. +- **A local page is past the call's limit.** A drawing on a processor takes several minutes and an edit 657 to 962 seconds ([studio](../architecture/studio.md#editing-a-picture)). The call answers `Queued` after 150 seconds, and a render naming a picture still being drawn is refused, so the PDF waits for the user's next message ([agent, Known gaps](../architecture/agent.md#known-gaps)). Six pages is six of those. + +## What changes + +| | Today | With this | +|---|---|---| +| "Make a comic about …" | a web page with SVG panels, or one picture | a plan, one confirmation, one picture per page, one PDF | +| Characters across pages | redrawn from words each time | page 1, or a character sheet, given to every later page as a reference picture | +| A page's shape | the model's default, usually square | `portrait` for a page, `landscape` for a strip, `square` | +| Speech bubbles | discouraged | drawn by the model; any page whose lettering comes back garbled gets its exact lines typeset under it in the PDF | +| A page still drawing when the PDF is rendered | refused; made on the user's next message | the PDF waits as a pending version and renders when its last page is ready | +| Changing page 3 | possible, but nothing says how to put it back | edit page 3, then render the PDF's next version with the new picture's name | + +The tool stays `surfsense_make_image` ([ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). opencode's `experimental_image_generate` is not used, now or as a fallback. + +## Design + +### 1. Routing to a comics skill + +- One line in the prompt, under *Producing files*: "For a comic, manga, comic strip, storyboard or picture book, load the `surfsense-comics` skill." It sits beside the picture line, which stays for a single picture. +- `surfsense-comics` is a new skill folder beside the others ([`skills/`](../../surfsense_local/backend/modules/agent/skills/)): `SKILL.md`, the steps below, and `pdf.md`, the ReportLab script that binds the pages, to copy and fill. +- The web page skill's description gains "not a comic or a storyboard: those are drawn pictures". A page that *contains* a finished comic's pictures stays possible when the user asks for a web page by name. +- Both engines load skills the same way, so opencode threads and Claude plan threads get it alike. + +### 2. What the skill tells the agent to do + +1. **Plan.** From the user's request and any sources: a title, the page count, and for each page its panels, what each shows and its lines. A **character sheet**: each character's face, hair, build, clothes and colours, in a few fixed words. A **style line**: medium, palette, line weight, panel borders. Four pages when the user names no number; at most twelve, since each is billed or takes minutes. +2. **Confirm once.** One message with the page count, the plan in short, and the cost: billed once per page by the user's provider, or about a few minutes a page on this computer, a longer time with references. On yes, the agent draws every page without asking again. A one-page comic or strip needs no confirmation, as a single picture needs none today. +3. **Draw page 1 alone**, `shape: portrait`. Its prompt is the style line, the character sheet and page 1's panels, laid out as "a comic page of N panels: panel 1, top left: …". Look at it. When it is wrong, edit it before going on, since every later page copies its characters. +4. **Draw the rest with page 1 as their reference.** Each is its own call with `references: ["image-"]`, the same style line and character sheet, and its panels. They go in one step, so a remote model draws them side by side. A user who gave a character picture of their own, or asked for a character sheet first, has that as the reference instead. +5. **Lettering.** Each bubble holds one short line, at most about eight words, two or three bubbles a panel, written in the prompt in quotes. When a page comes back with a line misspelt, the agent does not redraw it: the PDF typesets that page's exact lines in a strip under it. When the picture was drawn on this computer, or the agent cannot see pictures, every page is drawn without words and every page gets its strip, since lettering cannot be relied on or checked. +6. **Bind the PDF** with `surfsense_render_document`, `format: pdf`, every page's name in `images`: one picture per page, full bleed, in order, a title page only when the user asks for one. A page still being drawn is named all the same ([section 5](#5-a-render-that-waits-for-its-pictures)). +7. **Answer** with the PDF and one line naming each page, so the user can ask for "page 3 again". +8. **A later change.** To change a page, edit it with its `artifact_id`, then read the PDF's script with `surfsense_read_document` and render its next version with the new name in place of the old. To add a page, draw it with page 1 as reference and render again. + +### 3. `surfsense_make_image` changes + +Two new arguments and a rewritten description. Everything else, the waiting, the three-failure limit, the naming and the placing, stays as it is. + +| Argument | Type | Meaning | +|---|---|---| +| `references` | up to 4 picture names | Pictures this chat made, each ready, for the new picture to keep: the same characters, the same style. Only for a new picture, not with `artifact_id`. | +| `shape` | `square`, `portrait` or `landscape` | The picture's proportions. Left out, the model's default, as today. | + +- **A picture with references is drawn by the edit model**, the one that already reads a picture with its instruction ([image edit](../architecture/studio.md#editing-a-picture)). It is a new artifact, v1 of its own document, with the recipe `{"kind": "image_drawn", "prompt", "references": []}`, so Studio lists it, versions it and lets it be edited like any other picture. With no edit model chosen, the call is refused before anything is billed: "No image editing model is chosen, so a page cannot copy another: draw it from the character sheet alone, or tell the user they can choose one in Settings › Image edit." The skill then draws every page from words alone and says so to the user. +- **References are checked as placing is**: a name of this chat, ready (a picture still being drawn is refused, with a sentence to draw page 1 first), a still picture, and at most four, to bound what one call uploads; the spike may lower it. A refused reference is not a failed run. +- **The description**, rewritten: + - "Only when the user asks for one" stays. "Confirm before making more than one" becomes "When a request needs more than one picture, confirm how many once, before the first: each is billed, or takes minutes on their computer." + - "Words that must be exact belong in the document around the picture" becomes "Lettering may come out misspelt: keep any words in a picture short, and put text that must be exact in the document around it." + - "Draw no data chart" stays. +- **The result** adds where it was drawn, "Drawn on this computer" or "Drawn by ", so the skill can choose wordless pages for a local model. + +### 4. Many references and a shape, per provider + +The image protocols ([`protocols.py`](../../surfsense_local/backend/modules/llm/providers/protocols.py)) widen: + +```python +class ImageGenerator(Protocol): + async def generate(self, model: str, prompt: str, shape: Shape | None = None) -> GeneratedImage: ... + +class ImageEditor(Protocol): + async def edit(self, model: str, prompt: str, images: Sequence[GeneratedImage], shape: Shape | None = None) -> GeneratedImage: ... +``` + +An edit of one picture passes one image and no shape, so the edit job keeps its output size. + +- **Remote drawing** sends `size`: `1024x1024`, `1024x1536` or `1536x1024`. A `400` sends it once more without `size`, as a refused `response_format` already is ([`image.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image.py)), remembered per endpoint, and the result says the model kept its own shape. +- **Remote with references** posts every picture as an `image[]` part of the multipart `/images/edits` ([`image_edit.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py)), with `size` as above. An endpoint that answers `400` to several pictures gets the first alone, once, remembered per endpoint, and the result says only one reference reached the model. The spike below decides whether that fallback is kept or the call is refused instead. +- **Local drawing with a shape** moves to sd-server's native `POST /sdcpp/v1/img_gen`, which the edit client already uses ([`editor.py`](../../surfsense_local/backend/modules/llm/providers/sdcpp/editor.py)), with `width` and `height` set to the shape at the launch default's pixel count, each a multiple of 64. A drawing with no shape keeps its route. +- **Local with references** puts every picture in `ref_images`, the field the edit client already sends one picture in, and sets `width` and `height` from the shape, or from the first reference when none is given. + +### 5. A render that waits for its pictures + +Amends ADR 0087's "A document needing a picture still being drawn waits for the user's next message". + +- `surfsense_render_document` with a picture name still being drawn is no longer refused. It creates the pending version as today, with the metadata `waits_for: []`, and does not queue its job. The call answers at once: "Queued artifact , version : it renders once image-12 and image-14 are drawn. Do not render it again." The agent tells the user the PDF will appear in Studio. +- **A picture's job ending queues what waits on it.** At the end of every Studio job, in the same place it notifies the app ([`job.py`](../../surfsense_local/backend/worker/studio/job.py)), the worker finds the pending renders in that picture's thread whose `waits_for` names it. Each one whose pictures have all ended is queued then, through `queue_job` as every version is. +- **A picture that failed or was cancelled** fails the waiting render with a sentence naming it: "image-14 failed, so this PDF was not made. Draw that page again, or render without it." The render's failure counts under renders' three-failure limit when the next render is asked for, as a queued render's late failure does today ([`failed_renders.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/failed_renders.py)). +- **Only drawn pictures wait.** A source's figure or a chat's chart is either there or not, so a missing one is refused as today. A web page saved with `surfsense_save_page` still refuses a picture being drawn, since the page is checked and screenshotted as it is saved. +- **A restart** settles waiting renders at worker start, beside the interrupted documents ([`interrupted_documents.py`](../../surfsense_local/backend/worker/interrupted_documents.py)): a render whose pictures have all ended is queued, one still waiting stays pending, and one waiting on a picture that was itself interrupted fails with the same sentence. +- **Cancel** in Studio works as on any pending version: it is cancelled and never queued. +- **The agent cannot check its previews** when it renders this way, since the render finishes after its turn. The skill's PDF script is a fixed layout, one full-bleed picture per page, so there is little to go wrong. The open thread already reads its messages again when an artifact becomes ready, so the step links the PDF then ([agent](../architecture/agent.md#pictures-the-agent-makes)). + +### 6. In Studio + +- Each page is its own picture, titled " · page ", so the same-title refusal ([`same_title_documents.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py)) never meets two pages of one comic, and the user can open, edit or download any page. The PDF is its own artifact. Nothing in Studio's lists, viewer or buttons changes. + +## Decisions + +- **One picture per page, not per panel.** A page is one call, so six pages bill six pictures, not twenty-four, and a local comic takes minutes a page rather than an hour a page. A page also composes its panels as a comic artist would. The cost is that one bad panel redraws its page; an edit fixes most of those without a redraw. +- **Consistency by reference picture, not by words.** A character sheet in words is kept in every prompt too, but words alone redraw a face from scratch each time. Passing page 1 to the edit model is the one way both remote and local models already take to hold a face. A remote API's own "character" or "seed" features are not used: few providers have them and none agree. +- **The edit model draws a page with references.** It is the slot that reads pictures; `/images/generations` takes none. A user with only a drawing model still gets a comic, from words alone, and is told why the faces may differ. +- **The fallback for lettering costs nothing.** A garbled bubble is covered by a typeset strip rather than a redraw, so a bad letter never bills a second picture. Local pages are wordless because lettering the local model draws itself is not measured ([studio, Known gaps](../architecture/studio.md#known-gaps)); a measurement can lift that. +- **The PDF waits on the server, not in the agent.** A tool that polls, or a turn that waits, holds a tool call past its limit or a chat open for an hour; ADR 0087 rules out both. A pending version that the last picture's job queues needs no new process, no polling and no model awake. +- **No Studio comic format.** A Studio button would need a chat model to write the plan from sources, a second path beside the agent's, and Studio's drafts cannot place these pictures. It can be added later on top of the same tool and recipe. +- **Not a web page.** A comic is read as pages and printed or shared as a file; the PDF is what the user keeps. SVG panels drawn by a chat model are not a comic. +- **opencode's image tool is never used.** It finishes after the turn ends and is listed only on opencode 2.2 ([ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). + +## Not decided here + +- Grouping a comic's pages under its PDF in Studio, so twelve pages are not twelve rows. +- A comic button in Studio. +- Panels placed and lettered by SurfSense from wordless panel pictures, with real speech bubbles drawn by the PDF script where the model left room. It needs the agent to know where each bubble may go. +- A limit on pictures per turn enforced by the server rather than by the skill's twelve. + +## Spikes before code + +Each runs in the scratchpad against the real endpoint or binary, and its result is posted on the implementation PR. + +- **Remote, several references.** gpt-image-1 on OpenAI's `/images/edits` with two and four `image[]` parts and `size=1024x1536`; the same on one OpenAI-compatible gateway in the catalog that serves an image model from another vendor. Decides the one-reference fallback. +- **Remote shape.** Which providers in the catalog take `size` on `/images/generations`, and how each refuses it. +- **Local references.** The pinned sd-server with the catalog's edit model, `ref_images` of one and of two pictures and a portrait `width` and `height`: whether a face holds over four pages, and the seconds a page on a processor, to put in the confirmation. +- **Local shape.** The native route's drawing at a portrait size against the OpenAI-compatible route's square one, for time and quality. + +## Checks + +- **Routing:** an agent test, with the stand-in model, that "make a comic" loads `surfsense-comics`, draws page 1, then the rest with `references`, then renders one PDF naming them all, as [`test_agent_makes_image.py`](../../surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py) does for one picture. +- **Tool:** [`test_make_image.py`](../../surfsense_local/backend/tests/integration/agent/test_make_image.py) gains `references` and `shape` against the stand-in painter: references reach the edit model in order, a reference from another chat, a picture still drawing, a fifth reference and references with `artifact_id` are refused unbilled, no edit model refuses references, and the result names where it was drawn. +- **Providers:** unit tests on the multipart parts and the `400` fallbacks for the remote clients, and on the `img_gen` body for the local ones. +- **Waiting render:** a render naming a picture still drawing returns `Queued`; it renders once the picture is ready; a failed or cancelled picture fails it with the sentence; one of two pictures ending leaves it waiting; a worker restart queues, keeps or fails it as section 5 says. +- **Manual:** a four-page comic on a remote model and on a local one, each read as a PDF, with page 3 changed afterwards. + +## When it ships + +- [agent](../architecture/agent.md): the comics skill, the tool's new arguments and the waiting render, and the two Known gaps on slow pictures and lettering deleted. +- [studio](../architecture/studio.md): the `image_drawn` recipe's references, the shape, and the providers' new requests. +- An ADR for the render that waits on its pictures, amending ADR 0087. +- The user docs: [`guides/images.mdx`](../../www/content/docs/v2/guides/images.mdx) and [`agent.mdx`](../../www/content/docs/v2/agent.mdx) gain making a comic. From 39036b860e90d8db56e66ea635666dc492ccada1 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Tue, 10 Nov 2026 23:15:18 +0530 Subject: [PATCH 02/33] docs(proposals): one image model that edits or redraws, and comics bound into a PDF --- docs/proposals/comics.md | 157 ------------------ docs/proposals/pictures/01-image-models.md | 79 +++++++++ docs/proposals/pictures/02-make-image-tool.md | 46 +++++ docs/proposals/pictures/03-comics.md | 59 +++++++ docs/proposals/pictures/04-waiting-render.md | 32 ++++ docs/proposals/pictures/05-in-the-app.md | 41 +++++ docs/proposals/pictures/README.md | 114 +++++++++++++ 7 files changed, 371 insertions(+), 157 deletions(-) delete mode 100644 docs/proposals/comics.md create mode 100644 docs/proposals/pictures/01-image-models.md create mode 100644 docs/proposals/pictures/02-make-image-tool.md create mode 100644 docs/proposals/pictures/03-comics.md create mode 100644 docs/proposals/pictures/04-waiting-render.md create mode 100644 docs/proposals/pictures/05-in-the-app.md create mode 100644 docs/proposals/pictures/README.md diff --git a/docs/proposals/comics.md b/docs/proposals/comics.md deleted file mode 100644 index 720a6d96a6..0000000000 --- a/docs/proposals/comics.md +++ /dev/null @@ -1,157 +0,0 @@ ---- -status: proposed -code: - - surfsense_local/backend/modules/agent/prompts/agent.md - - surfsense_local/backend/modules/agent/skills/surfsense-comics/ - - surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md - - surfsense_local/backend/modules/agent/tool_endpoint/make_image.py - - surfsense_local/backend/modules/agent/tool_endpoint/render_document.py - - surfsense_local/backend/modules/artifacts/drawn_images/ - - surfsense_local/backend/modules/artifacts/script_documents/ - - surfsense_local/backend/modules/llm/providers/ - - surfsense_local/backend/worker/studio/ ---- - -# Comics the agent draws - -> Asked for a comic, the agent plans its pages, confirms the count once, draws one picture per page with the user's image model, keeping its characters the same from page to page, and binds the pages into one PDF. A local model's slow pages no longer leave the comic for the user's next message: the PDF renders itself once its last page is drawn. - -## Today - -Asked for a comic, the agent wrote a web page and drew the panels as SVG. When it did call `surfsense_make_image`, it drew one picture and stopped. Nothing in the agent's text makes a comic a picture job, and two lines push it away: - -- The prompt sends only "a picture or infographic" to `surfsense_make_image` ([`agent.md`](../../surfsense_local/backend/modules/agent/prompts/agent.md)). A comic is neither, and the web page skill's description, "an explainer … a figure that shows how something works", drawn "as inline SVG" ([`SKILL.md`](../../surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md)), is the nearest match. -- The tool's description says "confirm before making more than one" and "words that must be exact belong in the document around the picture" ([`make_image.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/make_image.py)). A comic is many pictures, mostly dialogue, so both read as "do not draw this". - -Three limits stay even with the wording fixed: - -- **Characters drift.** Every picture is drawn from its prompt alone. The same character sheet in six prompts gives six different faces. -- **Every picture is the model's default shape**, usually square. Neither the remote request ([`image.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image.py)) nor the local one sends a size, and a comic page is portrait. -- **A local page is past the call's limit.** A drawing on a processor takes several minutes and an edit 657 to 962 seconds ([studio](../architecture/studio.md#editing-a-picture)). The call answers `Queued` after 150 seconds, and a render naming a picture still being drawn is refused, so the PDF waits for the user's next message ([agent, Known gaps](../architecture/agent.md#known-gaps)). Six pages is six of those. - -## What changes - -| | Today | With this | -|---|---|---| -| "Make a comic about …" | a web page with SVG panels, or one picture | a plan, one confirmation, one picture per page, one PDF | -| Characters across pages | redrawn from words each time | page 1, or a character sheet, given to every later page as a reference picture | -| A page's shape | the model's default, usually square | `portrait` for a page, `landscape` for a strip, `square` | -| Speech bubbles | discouraged | drawn by the model; any page whose lettering comes back garbled gets its exact lines typeset under it in the PDF | -| A page still drawing when the PDF is rendered | refused; made on the user's next message | the PDF waits as a pending version and renders when its last page is ready | -| Changing page 3 | possible, but nothing says how to put it back | edit page 3, then render the PDF's next version with the new picture's name | - -The tool stays `surfsense_make_image` ([ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). opencode's `experimental_image_generate` is not used, now or as a fallback. - -## Design - -### 1. Routing to a comics skill - -- One line in the prompt, under *Producing files*: "For a comic, manga, comic strip, storyboard or picture book, load the `surfsense-comics` skill." It sits beside the picture line, which stays for a single picture. -- `surfsense-comics` is a new skill folder beside the others ([`skills/`](../../surfsense_local/backend/modules/agent/skills/)): `SKILL.md`, the steps below, and `pdf.md`, the ReportLab script that binds the pages, to copy and fill. -- The web page skill's description gains "not a comic or a storyboard: those are drawn pictures". A page that *contains* a finished comic's pictures stays possible when the user asks for a web page by name. -- Both engines load skills the same way, so opencode threads and Claude plan threads get it alike. - -### 2. What the skill tells the agent to do - -1. **Plan.** From the user's request and any sources: a title, the page count, and for each page its panels, what each shows and its lines. A **character sheet**: each character's face, hair, build, clothes and colours, in a few fixed words. A **style line**: medium, palette, line weight, panel borders. Four pages when the user names no number; at most twelve, since each is billed or takes minutes. -2. **Confirm once.** One message with the page count, the plan in short, and the cost: billed once per page by the user's provider, or about a few minutes a page on this computer, a longer time with references. On yes, the agent draws every page without asking again. A one-page comic or strip needs no confirmation, as a single picture needs none today. -3. **Draw page 1 alone**, `shape: portrait`. Its prompt is the style line, the character sheet and page 1's panels, laid out as "a comic page of N panels: panel 1, top left: …". Look at it. When it is wrong, edit it before going on, since every later page copies its characters. -4. **Draw the rest with page 1 as their reference.** Each is its own call with `references: ["image-"]`, the same style line and character sheet, and its panels. They go in one step, so a remote model draws them side by side. A user who gave a character picture of their own, or asked for a character sheet first, has that as the reference instead. -5. **Lettering.** Each bubble holds one short line, at most about eight words, two or three bubbles a panel, written in the prompt in quotes. When a page comes back with a line misspelt, the agent does not redraw it: the PDF typesets that page's exact lines in a strip under it. When the picture was drawn on this computer, or the agent cannot see pictures, every page is drawn without words and every page gets its strip, since lettering cannot be relied on or checked. -6. **Bind the PDF** with `surfsense_render_document`, `format: pdf`, every page's name in `images`: one picture per page, full bleed, in order, a title page only when the user asks for one. A page still being drawn is named all the same ([section 5](#5-a-render-that-waits-for-its-pictures)). -7. **Answer** with the PDF and one line naming each page, so the user can ask for "page 3 again". -8. **A later change.** To change a page, edit it with its `artifact_id`, then read the PDF's script with `surfsense_read_document` and render its next version with the new name in place of the old. To add a page, draw it with page 1 as reference and render again. - -### 3. `surfsense_make_image` changes - -Two new arguments and a rewritten description. Everything else, the waiting, the three-failure limit, the naming and the placing, stays as it is. - -| Argument | Type | Meaning | -|---|---|---| -| `references` | up to 4 picture names | Pictures this chat made, each ready, for the new picture to keep: the same characters, the same style. Only for a new picture, not with `artifact_id`. | -| `shape` | `square`, `portrait` or `landscape` | The picture's proportions. Left out, the model's default, as today. | - -- **A picture with references is drawn by the edit model**, the one that already reads a picture with its instruction ([image edit](../architecture/studio.md#editing-a-picture)). It is a new artifact, v1 of its own document, with the recipe `{"kind": "image_drawn", "prompt", "references": []}`, so Studio lists it, versions it and lets it be edited like any other picture. With no edit model chosen, the call is refused before anything is billed: "No image editing model is chosen, so a page cannot copy another: draw it from the character sheet alone, or tell the user they can choose one in Settings › Image edit." The skill then draws every page from words alone and says so to the user. -- **References are checked as placing is**: a name of this chat, ready (a picture still being drawn is refused, with a sentence to draw page 1 first), a still picture, and at most four, to bound what one call uploads; the spike may lower it. A refused reference is not a failed run. -- **The description**, rewritten: - - "Only when the user asks for one" stays. "Confirm before making more than one" becomes "When a request needs more than one picture, confirm how many once, before the first: each is billed, or takes minutes on their computer." - - "Words that must be exact belong in the document around the picture" becomes "Lettering may come out misspelt: keep any words in a picture short, and put text that must be exact in the document around it." - - "Draw no data chart" stays. -- **The result** adds where it was drawn, "Drawn on this computer" or "Drawn by ", so the skill can choose wordless pages for a local model. - -### 4. Many references and a shape, per provider - -The image protocols ([`protocols.py`](../../surfsense_local/backend/modules/llm/providers/protocols.py)) widen: - -```python -class ImageGenerator(Protocol): - async def generate(self, model: str, prompt: str, shape: Shape | None = None) -> GeneratedImage: ... - -class ImageEditor(Protocol): - async def edit(self, model: str, prompt: str, images: Sequence[GeneratedImage], shape: Shape | None = None) -> GeneratedImage: ... -``` - -An edit of one picture passes one image and no shape, so the edit job keeps its output size. - -- **Remote drawing** sends `size`: `1024x1024`, `1024x1536` or `1536x1024`. A `400` sends it once more without `size`, as a refused `response_format` already is ([`image.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image.py)), remembered per endpoint, and the result says the model kept its own shape. -- **Remote with references** posts every picture as an `image[]` part of the multipart `/images/edits` ([`image_edit.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py)), with `size` as above. An endpoint that answers `400` to several pictures gets the first alone, once, remembered per endpoint, and the result says only one reference reached the model. The spike below decides whether that fallback is kept or the call is refused instead. -- **Local drawing with a shape** moves to sd-server's native `POST /sdcpp/v1/img_gen`, which the edit client already uses ([`editor.py`](../../surfsense_local/backend/modules/llm/providers/sdcpp/editor.py)), with `width` and `height` set to the shape at the launch default's pixel count, each a multiple of 64. A drawing with no shape keeps its route. -- **Local with references** puts every picture in `ref_images`, the field the edit client already sends one picture in, and sets `width` and `height` from the shape, or from the first reference when none is given. - -### 5. A render that waits for its pictures - -Amends ADR 0087's "A document needing a picture still being drawn waits for the user's next message". - -- `surfsense_render_document` with a picture name still being drawn is no longer refused. It creates the pending version as today, with the metadata `waits_for: []`, and does not queue its job. The call answers at once: "Queued artifact , version : it renders once image-12 and image-14 are drawn. Do not render it again." The agent tells the user the PDF will appear in Studio. -- **A picture's job ending queues what waits on it.** At the end of every Studio job, in the same place it notifies the app ([`job.py`](../../surfsense_local/backend/worker/studio/job.py)), the worker finds the pending renders in that picture's thread whose `waits_for` names it. Each one whose pictures have all ended is queued then, through `queue_job` as every version is. -- **A picture that failed or was cancelled** fails the waiting render with a sentence naming it: "image-14 failed, so this PDF was not made. Draw that page again, or render without it." The render's failure counts under renders' three-failure limit when the next render is asked for, as a queued render's late failure does today ([`failed_renders.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/failed_renders.py)). -- **Only drawn pictures wait.** A source's figure or a chat's chart is either there or not, so a missing one is refused as today. A web page saved with `surfsense_save_page` still refuses a picture being drawn, since the page is checked and screenshotted as it is saved. -- **A restart** settles waiting renders at worker start, beside the interrupted documents ([`interrupted_documents.py`](../../surfsense_local/backend/worker/interrupted_documents.py)): a render whose pictures have all ended is queued, one still waiting stays pending, and one waiting on a picture that was itself interrupted fails with the same sentence. -- **Cancel** in Studio works as on any pending version: it is cancelled and never queued. -- **The agent cannot check its previews** when it renders this way, since the render finishes after its turn. The skill's PDF script is a fixed layout, one full-bleed picture per page, so there is little to go wrong. The open thread already reads its messages again when an artifact becomes ready, so the step links the PDF then ([agent](../architecture/agent.md#pictures-the-agent-makes)). - -### 6. In Studio - -- Each page is its own picture, titled " · page ", so the same-title refusal ([`same_title_documents.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py)) never meets two pages of one comic, and the user can open, edit or download any page. The PDF is its own artifact. Nothing in Studio's lists, viewer or buttons changes. - -## Decisions - -- **One picture per page, not per panel.** A page is one call, so six pages bill six pictures, not twenty-four, and a local comic takes minutes a page rather than an hour a page. A page also composes its panels as a comic artist would. The cost is that one bad panel redraws its page; an edit fixes most of those without a redraw. -- **Consistency by reference picture, not by words.** A character sheet in words is kept in every prompt too, but words alone redraw a face from scratch each time. Passing page 1 to the edit model is the one way both remote and local models already take to hold a face. A remote API's own "character" or "seed" features are not used: few providers have them and none agree. -- **The edit model draws a page with references.** It is the slot that reads pictures; `/images/generations` takes none. A user with only a drawing model still gets a comic, from words alone, and is told why the faces may differ. -- **The fallback for lettering costs nothing.** A garbled bubble is covered by a typeset strip rather than a redraw, so a bad letter never bills a second picture. Local pages are wordless because lettering the local model draws itself is not measured ([studio, Known gaps](../architecture/studio.md#known-gaps)); a measurement can lift that. -- **The PDF waits on the server, not in the agent.** A tool that polls, or a turn that waits, holds a tool call past its limit or a chat open for an hour; ADR 0087 rules out both. A pending version that the last picture's job queues needs no new process, no polling and no model awake. -- **No Studio comic format.** A Studio button would need a chat model to write the plan from sources, a second path beside the agent's, and Studio's drafts cannot place these pictures. It can be added later on top of the same tool and recipe. -- **Not a web page.** A comic is read as pages and printed or shared as a file; the PDF is what the user keeps. SVG panels drawn by a chat model are not a comic. -- **opencode's image tool is never used.** It finishes after the turn ends and is listed only on opencode 2.2 ([ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). - -## Not decided here - -- Grouping a comic's pages under its PDF in Studio, so twelve pages are not twelve rows. -- A comic button in Studio. -- Panels placed and lettered by SurfSense from wordless panel pictures, with real speech bubbles drawn by the PDF script where the model left room. It needs the agent to know where each bubble may go. -- A limit on pictures per turn enforced by the server rather than by the skill's twelve. - -## Spikes before code - -Each runs in the scratchpad against the real endpoint or binary, and its result is posted on the implementation PR. - -- **Remote, several references.** gpt-image-1 on OpenAI's `/images/edits` with two and four `image[]` parts and `size=1024x1536`; the same on one OpenAI-compatible gateway in the catalog that serves an image model from another vendor. Decides the one-reference fallback. -- **Remote shape.** Which providers in the catalog take `size` on `/images/generations`, and how each refuses it. -- **Local references.** The pinned sd-server with the catalog's edit model, `ref_images` of one and of two pictures and a portrait `width` and `height`: whether a face holds over four pages, and the seconds a page on a processor, to put in the confirmation. -- **Local shape.** The native route's drawing at a portrait size against the OpenAI-compatible route's square one, for time and quality. - -## Checks - -- **Routing:** an agent test, with the stand-in model, that "make a comic" loads `surfsense-comics`, draws page 1, then the rest with `references`, then renders one PDF naming them all, as [`test_agent_makes_image.py`](../../surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py) does for one picture. -- **Tool:** [`test_make_image.py`](../../surfsense_local/backend/tests/integration/agent/test_make_image.py) gains `references` and `shape` against the stand-in painter: references reach the edit model in order, a reference from another chat, a picture still drawing, a fifth reference and references with `artifact_id` are refused unbilled, no edit model refuses references, and the result names where it was drawn. -- **Providers:** unit tests on the multipart parts and the `400` fallbacks for the remote clients, and on the `img_gen` body for the local ones. -- **Waiting render:** a render naming a picture still drawing returns `Queued`; it renders once the picture is ready; a failed or cancelled picture fails it with the sentence; one of two pictures ending leaves it waiting; a worker restart queues, keeps or fails it as section 5 says. -- **Manual:** a four-page comic on a remote model and on a local one, each read as a PDF, with page 3 changed afterwards. - -## When it ships - -- [agent](../architecture/agent.md): the comics skill, the tool's new arguments and the waiting render, and the two Known gaps on slow pictures and lettering deleted. -- [studio](../architecture/studio.md): the `image_drawn` recipe's references, the shape, and the providers' new requests. -- An ADR for the render that waits on its pictures, amending ADR 0087. -- The user docs: [`guides/images.mdx`](../../www/content/docs/v2/guides/images.mdx) and [`agent.mdx`](../../www/content/docs/v2/agent.mdx) gain making a comic. diff --git a/docs/proposals/pictures/01-image-models.md b/docs/proposals/pictures/01-image-models.md new file mode 100644 index 0000000000..304405b4cc --- /dev/null +++ b/docs/proposals/pictures/01-image-models.md @@ -0,0 +1,79 @@ +# Image models + +> Each image model says whether it can take a picture in. One rule picks the model for any job that sends a picture: the user's edit model if they chose one, else the image model if it can, else none, and then the job redraws. One interface draws, edits and keeps references, and each provider's client knows its own routes. + +## What a model can do + +Each image model carries `takes_pictures`: whether it draws from a picture as well as from words. + +- **Remote:** from what the connection's `/models` publishes, read live. Discovery already reads each entry's `output_modalities` to find image models ([`service.py`](../../../surfsense_local/backend/modules/llm/connections/service.py)); it reads `input_modalities` beside it, from the entry or its `architecture`, and `DiscoveredModel` gains `takes_pictures`. OpenRouter lists `text,image` for every model the spike drew with. The bundled manifest's `modalities.input` answers when the endpoint publishes nothing; it lacks most OpenRouter image models, so it is the fallback, not the source. +- **Local:** the sd.cpp catalog's `edit` type, which FLUX.2 klein has today ([selection](../../architecture/local-models/selection.md)). +- **Kept with the choice.** When the user chooses an image model, its `takes_pictures` is kept in the selection's existing `settings` JSON, and refreshed whenever that connection's model list is read again. No migration: the column is there ([`models.py`](../../../surfsense_local/backend/modules/llm/models.py)). +- **Unknown is no.** A model whose endpoint says nothing and that the manifest does not carry gets `takes_pictures: false`, so its edits redraw. Sending a picture to a model that ignores it would bill a picture that keeps nothing. + +## Which model handles a picture + +One function, `picture_model(session)`, beside the others in [`resolution.py`](../../../surfsense_local/backend/modules/llm/resolution.py), answers every job that sends a picture in, an edit or a comic page with references: + +1. the `image_edit` selection, when the user chose one: the override; +2. else the `image_gen` selection, when it `takes_pictures`; +3. else none. + +The API exposes the answer once, as `pictures_in: "edit_model" | "image_model" | null`, beside the selections it is worked out from. Studio, the agent's tool list and the skill all read that, so no two places can disagree. + +## One interface + +The two protocols, `ImageGenerator` and `ImageEditor` ([`protocols.py`](../../../surfsense_local/backend/modules/llm/providers/protocols.py)), become one: + +```python +class ImageModel(Protocol): + async def draw( + self, + model: str, + prompt: str, + *, + shape: Shape | None = None, + references: Sequence[GeneratedImage] = (), + ) -> GeneratedImage: ... +``` + +An edit is a drawing with the picture as its one reference and the instruction as its prompt. `Shape` is `square`, `portrait` or `landscape`; left out, the model's default, as today. + +## One client per provider + +| Provider | No references | With references | +|---|---|---| +| **OpenRouter** (`catalog_provider` `openrouter`) | `POST /images/generations` with `size` | `POST /images` with `input_references` and `aspect_ratio` | +| **Any other OpenAI-compatible server** | `POST /images/generations` with `size`, then `/images` as today's route cache does | `POST /images/edits`, multipart, one `image[]` part per reference, and `size` | +| **Local sd-server** | native `POST /sdcpp/v1/img_gen` with `width` and `height` | the same, with `ref_images` holding every reference | + +- **Sizes.** `square` is `1024x1024` or `1:1`, `portrait` `1024x1536` or `2:3`, `landscape` `1536x1024` or `3:2`. Locally, the launch's default pixel count at that proportion, each side a multiple of 64. +- **OpenRouter gets its own client** because it answers 404 on `/images/edits` and its pure image models refuse `/chat/completions` ([the spike](README.md#the-spike)). The client is chosen by the connection's `catalog_provider`, which every connection already records. +- **Local drawing moves to the native route** only when a shape is asked; a drawing with no shape keeps today's `/v1/images/generations`. The native route is the one the edit client already uses, because sd-server's OpenAI and A1111 edit routes re-noise the picture ([ADR 0077](../../adr/0077-an-image-edit-is-the-next-version.md)). +- **Retries, the same for every client.** A `400` to an optional field (`size`, `aspect_ratio`, `response_format`) is sent once more without it, remembered per connection, as `response_format` already is: a `400` comes before any drawing. A timeout or a `5xx` is never retried, since the provider may still be drawing and a second request bills a second picture. +- **Several references refused.** A server that answers `400` to several `image[]` parts gets the first alone, once, and the result says only one reached the model. The spike below decides whether that is kept. + +## Edits and redraws + +An edit of a picture, from Studio's Edit box or from the agent, stays the next version of the same document, with the recipe ADR 0077 gives it. What runs depends on the rule: + +- **A model takes pictures:** the edit draws with the base version's picture as its one reference and the instruction as its prompt, at the base's proportions. The same as today, through the rule instead of the slot. +- **None does: a redraw.** The image model draws again from the prompt that drew the base and the instruction: `"\n\nChange: "`, at the base's proportions, with no chat model in between. The recipe records `"redrawn": true`, and the version's body starts with one line, "Drawn again with your change." It is billed as one picture, as an edit is. +- **The prompt a redraw starts from** is the one the base was drawn with. A picture the agent drew keeps it as `prompt` already. A Studio draft keeps it only in its body under the title ([`image/pipeline.py`](../../../surfsense_local/backend/worker/studio/media/visual/image/pipeline.py)); from now on every drawing also keeps it as `drawing_prompt` in its metadata, and a redraw of an older picture reads it from the body. A redraw of a redraw starts from the redrawn version's own prompt, so changes add up. +- **The `needs_image_edit` refusal goes.** No edit is refused for want of a model; only for want of an image model at all, which is `needs_image` as today. + +## Spikes before code + +Each runs in the scratchpad against the real endpoint or binary, and its result is posted on the implementation PR. + +- **OpenAI's `/images/edits` with two and four `image[]` parts and `size=1024x1536`,** on gpt-image. Decides the one-reference fallback. +- **Local references:** the pinned sd-server with FLUX.2 klein, `ref_images` of one and two pictures at a portrait `width` and `height`: whether a face holds over four pages, and the seconds a page on a processor. +- **Local shape:** the native route's drawing at a portrait size against today's route's square one, for time and quality. +- **`input_modalities` across the providers the catalog connects to,** to see which publish it and how often the manifest must answer instead. + +## Checks + +- Discovery: an OpenRouter-shaped `/models` entry with `input_modalities` `text,image` gives `takes_pictures`; one with `text` alone does not; one with neither falls to the manifest, then to no. +- The rule: each of its three answers, and `pictures_in` in the API. +- Clients: the request each sends with and without a shape and references, the `400` retries, no retry on a timeout or a `5xx`, against a stand-in server. +- Edits: a real edit through the image model; a redraw through a model that takes no pictures, its recipe and its first line; a redraw of a Studio draft made before `drawing_prompt`; no `needs_image_edit` anywhere. diff --git a/docs/proposals/pictures/02-make-image-tool.md b/docs/proposals/pictures/02-make-image-tool.md new file mode 100644 index 0000000000..6398c28f58 --- /dev/null +++ b/docs/proposals/pictures/02-make-image-tool.md @@ -0,0 +1,46 @@ +# The agent's picture tool + +> `surfsense_make_image` gains `shape` and `references`, its description stops steering away from pictures with words or from several pictures, and an edit or a reference that no model can take is drawn from words instead of refused. + +Everything else in [`make_image.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/make_image.py) stays: the wait, the three-failure limit, the `image-` names, placing, and the one-title-per-chat refusal. + +## New arguments + +| Argument | Type | Meaning | +|---|---|---| +| `shape` | `square`, `portrait` or `landscape` | The picture's proportions. Left out, the model's default. | +| `references` | up to 4 picture names | Pictures this chat made, each ready, for the new picture to keep: the same characters and style. Only for a new picture, not with `artifact_id`. | + +- **References are checked as placing is** ([`names.py`](../../../surfsense_local/backend/modules/artifacts/drawn_images/names.py)): a name of this chat, ready, a still picture. A picture still being drawn is refused with "Draw it first and wait for it", and none of these refusals is a failed run. Four bounds what one call uploads; the local spike may lower it. +- **A picture with references runs on the model [the rule](01-image-models.md#which-model-handles-a-picture) picks.** When it picks none, the picture is drawn from its prompt alone and the result says so: "Drawn from words alone: the image model cannot take a picture in, so the characters may differ from the reference." Nothing is refused, so the agent never loops on a setting it cannot change. +- **An edit with `artifact_id`** runs as Studio's does ([edits and redraws](01-image-models.md#edits-and-redraws)): a real edit, or a redraw that the result names as one. The tool's refusal for want of an edit model, and its text sending the user to Settings › Image edit, go. +- **The recipe** of a new picture records `shape` and `references` as artifact ids: `{"kind": "image_drawn", "prompt", "shape", "references"}`. Regenerate repeats it with the same references. + +## The listing stays fixed + +Tool listings are fixed and in one order, so a local model's prompt cache holds from turn to turn ([`offered_tools.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py)). So `references` is always listed, and what happens without a model is in the result, not in the listing. + +## The description + +Rewritten, as short as it can be: + +- "Only when the user asks for one" stays, and so does "Draw no data chart". +- "Confirm before making more than one" becomes "When a request needs more than one picture, confirm how many once, before the first: each is billed, or takes minutes on their computer." +- "Words that must be exact belong in the document around the picture" becomes "Lettering may come out misspelt: keep words in a picture short, and put text that must be exact in the document around it." +- One line: "For a comic, a storyboard or any set of pictures, load the `surfsense-pictures` skill first." + +## The result + +Adds, to what it says today: + +- **Where it was drawn:** "Drawn on this computer" or "Drawn by ". The skill draws wordless pages on this computer ([comics](03-comics.md#lettering)). +- **How it was drawn,** when it was not as asked: from words alone, a redraw, or a shape or reference the model did not take. + +## Checks + +[`test_make_image.py`](../../../surfsense_local/backend/tests/integration/agent/test_make_image.py), against the stand-in painter: + +- `shape` reaches the model; references reach it in order; a reference from another chat, a picture still drawing, a fifth reference, and references with `artifact_id` are refused, unbilled. +- With no model that takes pictures, references draw from words and the result says so; an edit redraws and the result says so. +- The result names where the picture was drawn. +- The listing is the same, byte for byte, whatever models are chosen. diff --git a/docs/proposals/pictures/03-comics.md b/docs/proposals/pictures/03-comics.md new file mode 100644 index 0000000000..534ffdaf87 --- /dev/null +++ b/docs/proposals/pictures/03-comics.md @@ -0,0 +1,59 @@ +# Comics + +> A comic, a storyboard or a picture book loads one skill, `surfsense-pictures`. The agent plans the pages, confirms the count once, draws page 1, draws the rest with page 1 as their reference, and binds every page into one PDF. The PDF is the comic. + +## Routing + +- **The prompt.** The picture line in [`agent.md`](../../../surfsense_local/backend/modules/agent/prompts/agent.md) becomes: "For a picture, an infographic, a comic, a storyboard, a picture book or any set of pictures, load the `surfsense-pictures` skill, then use `surfsense_make_image`." +- **The web page skill** ([`SKILL.md`](../../../surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md)) adds to its description: "Not for a comic, a storyboard or a picture: those are drawn pictures." A page that holds a finished comic's pictures stays possible when the user asks for a web page by name. +- **Both engines** load skills the same way, so opencode threads and Claude plan threads get it alike. + +## The skill + +`surfsense-pictures` is the craft beside the tool's contract, as `surfsense-documents` is beside `surfsense_render_document`. It sits with the other skills ([`skills/`](../../../surfsense_local/backend/modules/agent/skills/)): + +| File | Holds | +|---|---| +| `SKILL.md` | One picture or a set; writing a prompt; lettering; the series steps below; what to say when a model cannot take pictures | +| `comics.md` | Pages and panels, bubbles and captions, the comic's PDF | +| `pdf.md` | The binding script, to copy and fill | + +A storyboard or a picture book later is one more file beside `comics.md`, with no new skill and no prompt change. + +## The steps + +1. **Plan.** A title, the page count, and for each page its panels, what each shows and its lines. A **character sheet**: each character's face, hair, build, clothes and colours, in a few fixed words. A **style line**: medium, palette, line weight, panel borders. Four pages when the user names no number; at most twelve, since each is billed or takes minutes. +2. **Confirm once.** One message: the page count, the plan in short, and what it costs: billed once a page by their provider, or minutes a page on this computer. When the image model cannot take pictures in, one more plain line: "Each page is drawn from the description, so the characters may differ a little between pages." On yes, every page is drawn without asking again. A one-page comic or strip needs no confirmation, as a single picture needs none. +3. **Page 1 alone,** `shape: portrait`. Its prompt is the style line, the character sheet and page 1's panels, laid out as "A comic page of N panels: panel 1, top left: …". Look at it; when it is wrong, edit it before going on, since every later page copies its characters. +4. **The rest, with page 1 as their reference.** Each is its own call, with `references: ["image-"]`, the same style line and character sheet, and its panels, all in one step so a remote model draws them side by side. A character picture the user gave, or a character sheet they asked for first, is the reference instead. +5. **Look at every page.** Redraw or edit one only when it is plainly wrong, and say so; a misspelt word is not a reason ([below](#lettering)). +6. **Bind the PDF** ([below](#the-pdf)). +7. **Answer** with the PDF and one line per page, so the user can ask for "page 3 again". +8. **A later change.** Edit the page with its `artifact_id`, read the PDF's script with `surfsense_read_document`, and render its next version with the new name in place of the old. To add a page, draw it with page 1 as reference and render again. + +Each page is its own picture, titled " · page ", so the one-title-per-chat refusal never meets two pages of one comic. + +## Lettering + +- **Bubbles are drawn by the model.** Each holds one short line, at most about eight words, two or three to a panel, written in the prompt in quotes. The best three models in [the spike](README.md#the-spike) spelled every line right. +- **A misspelt page is not redrawn.** The PDF typesets that page's exact lines in a strip under it, so a bad letter never bills a second picture. +- **On this computer, or when the agent cannot see pictures,** every page is drawn without words and every page gets its strip. Lettering a local model draws itself is not measured ([studio, Known gaps](../../architecture/studio.md#known-gaps)), and a page the agent cannot see it cannot check. + +## The PDF + +Made with `surfsense_render_document`, `format: pdf`, every page's name in `images`, from the script in `pdf.md`: + +- One picture per page, full bleed, in order, at comic-book width (6.625 in) and the page's own height. A title page only when the user asks for one. +- **Pages placed as JPEG at quality 90**, not PNG: four PNG pages made 10 to 28 MB in the spike. Pillow converts each before ReportLab places it; a page is drawn once, so the loss is once. +- A page with a lettering strip is a little taller, the strip below the picture, the lines in the PDF's own font. +- A page still being drawn is named all the same ([the render that waits](04-waiting-render.md)). + +## Which models to recommend + +From [the spike](README.md#the-spike): `gpt-image-2.6`, Gemini's image models and Seedream letter well and hold characters, and the user docs name them for comics. FLUX.2 klein, remote or local, draws fast and holds characters but garbles lettering, so its pages get strips. The app recommends nothing in the moment: the user's chosen model is used as it is. + +## Checks + +- An agent test, with the stand-in model, as [`test_agent_makes_image.py`](../../../surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py) does for one picture: "make a comic" loads `surfsense-pictures`, draws page 1, then the rest with `references`, then renders one PDF naming every page. +- `pdf.md`'s script, run as the render runs it, on four stand-in pages: four PDF pages, JPEG images, under 5 MB, and a strip page when asked. +- Manual: a four-page comic on a remote model that takes pictures, on one that does not, and on a local model, each read as a PDF, with page 3 changed after. diff --git a/docs/proposals/pictures/04-waiting-render.md b/docs/proposals/pictures/04-waiting-render.md new file mode 100644 index 0000000000..30734195cf --- /dev/null +++ b/docs/proposals/pictures/04-waiting-render.md @@ -0,0 +1,32 @@ +# The render that waits + +> A document that names a picture still being drawn is not refused. It becomes a pending version that the last of its pictures' jobs queues, so a comic drawn on a slow local model still ends as a PDF without the user asking again. + +Amends [ADR 0087](../../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)'s "A document needing a picture still being drawn waits for the user's next message". + +## Today + +A local picture takes minutes, and an edit 657 to 962 seconds on a processor ([studio](../../architecture/studio.md#editing-a-picture)). `surfsense_make_image` answers `Queued` after 150 seconds, and a render naming that picture is refused while it is drawing ([`script_documents/service.py`](../../../surfsense_local/backend/modules/artifacts/script_documents/service.py)), so the agent can only tell the user to ask again. + +## Design + +- **The render is accepted.** `surfsense_render_document` with a picture name still being drawn creates the pending version as today, with `waits_for: []` in its metadata, and does not queue its job. The call answers at once: "Queued artifact , version : it renders once image-12 and image-14 are drawn. Do not render it again." The agent tells the user the PDF will appear in Studio. +- **A picture's job ending queues what waits on it.** At the end of every Studio job, where the worker already notifies the app ([`job.py`](../../../surfsense_local/backend/worker/studio/job.py)), it finds the pending renders in that picture's thread whose `waits_for` names it. Each one whose pictures have all ended is queued then, through `queue_job` as every version is ([`queued_job.py`](../../../surfsense_local/backend/modules/artifacts/queued_job.py)). +- **A picture that failed or was cancelled** fails the waiting render with a sentence naming it: "image-14 failed, so this PDF was not made. Draw that page again, or render without it." That failure counts under renders' three-failure limit when the next render is asked for, as a queued render's late failure does today ([`failed_renders.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/failed_renders.py)). +- **Only drawn pictures wait.** A source's figure or a chat's chart is there or not, so a missing one is refused as today. A web page saved with `surfsense_save_page` still refuses a picture being drawn, since a page is checked and screenshotted as it is saved. +- **A restart** settles waiting renders when the worker starts, beside the interrupted documents ([`interrupted_documents.py`](../../../surfsense_local/backend/worker/interrupted_documents.py)): one whose pictures have all ended is queued, one still waiting stays pending, and one waiting on a picture that was itself interrupted fails with the same sentence. +- **Cancel** in Studio works as on any pending version. +- **No one checks the previews** of a render that finishes after its turn. The comic's PDF is a fixed layout, one full-bleed picture a page, so little can go wrong, and the open thread links the PDF once it is ready, as it does for any queued version ([agent](../../architecture/agent.md#pictures-the-agent-makes)). + +## Decisions + +- **On the server, not in the agent.** A tool that polls, or a turn kept open, holds a call past its limit or a chat open for an hour; ADR 0087 rules out both. A pending version that the last picture's job queues needs no new process, no polling and no model awake. +- **In the job's end, not a timer.** The job that ends is the one event that can make a render ready, so nothing runs in between. + +## Checks + +- A render naming a picture still drawing answers `Queued` and is not queued; it renders once the picture is ready. +- One of two pictures ending leaves it waiting; the second queues it. +- A failed or a cancelled picture fails it, with the sentence, and the failure counts toward the next render. +- A worker restart queues it, keeps it, or fails it, as above. +- A web page naming a picture still drawing is still refused. diff --git a/docs/proposals/pictures/05-in-the-app.md b/docs/proposals/pictures/05-in-the-app.md new file mode 100644 index 0000000000..4cbad846d9 --- /dev/null +++ b/docs/proposals/pictures/05-in-the-app.md @@ -0,0 +1,41 @@ +# In the app + +> Edit works on every picture. Settings marks the image models that can edit, with a small label and a three-word tooltip. Nothing in Studio or chat sends the user to Settings for pictures, and a model that cannot do something is described in a neutral line, never a warning. + +## Studio's Edit + +- **The Edit box shows over every ready picture**, as it does over a document's Refine ([`artifact-panel.tsx`](../../../surfsense_local/frontend/src/features/studio/artifact-panel.tsx)). It no longer checks the `image_edit` slot ([`edit-model.ts`](../../../surfsense_local/frontend/src/features/studio/edit-model.ts)): every picture can be edited or redrawn ([edits and redraws](../pictures/01-image-models.md#edits-and-redraws)). +- **Its placeholder** says what will happen: "Describe a change" when the model edits, "Describe a change: the picture is drawn again" when it redraws. The box reads `pictures_in` from the API to choose. +- **A redrawn version** shows its body's first line, "Drawn again with your change.", in the viewer's normal text. +- **`needs_image_edit` goes**, with its message "Needs an image editing model." and its link to Settings › Image editing models. Only an image picture with no image model at all says "Needs an image model.", as today. + +## Settings + +- **The label.** In Settings › Image models, each model that takes pictures in carries a small **Edits** label beside its name, in the list of a server's models and in the list of local models to download ([`server-models.tsx`](../../../surfsense_local/frontend/src/features/models/remote/models/server-models.tsx), [`download-image-models.tsx`](../../../surfsense_local/frontend/src/features/models/local/image/download-image-models.tsx)). It is information only: choosing a model without it works. +- **The tooltip** on the label: "Can edit pictures". The label is focusable, so the tooltip opens on keyboard focus as on hover, and a screen reader reads the same words. It uses the app's own tooltip, as the rest of Settings does. +- **Image editing models** ([`image-edit-models-settings.tsx`](../../../surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx)) becomes an optional override, collapsed under the image models, titled "Use a different model for edits". Its summary reads "Same as the image model" when nothing is chosen there, in place of "No image editing model chosen" ([`in-use-summary.tsx`](../../../surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx)). + +## Words + +Every new or changed string goes through the `translate` skill, into every language the app ships, short enough not to overflow: + +| Where | Text | +|---|---| +| Label | Edits | +| Tooltip | Can edit pictures | +| Edit placeholder, a model that edits | Describe a change | +| Edit placeholder, a redraw | Describe a change: the picture is drawn again | +| A redrawn version | Drawn again with your change. | +| The override's title | Use a different model for edits | +| The override's summary, unset | Same as the image model | + +## The user docs + +[`guides/images.mdx`](../../../www/content/docs/v2/guides/images.mdx) gains making a comic and what Edit does on each kind of model; [`studio.mdx`](../../../www/content/docs/v2/studio.mdx) the redraw; the image model pages under `models/` the label and the optional edit model; and the models that letter well for comics are named there. + +## Checks + +- The Edit box over a picture with each of `pictures_in`'s three answers: its placeholder, and that it sends. +- No `needs_image_edit` and no link to the edit settings anywhere in Studio. +- The label shows only on models that take pictures in; its tooltip opens on hover and on focus. +- The override collapsed and summarised as "Same as the image model" when unset. diff --git a/docs/proposals/pictures/README.md b/docs/proposals/pictures/README.md new file mode 100644 index 0000000000..7289abc25d --- /dev/null +++ b/docs/proposals/pictures/README.md @@ -0,0 +1,114 @@ +--- +status: proposed +code: + - surfsense_local/backend/modules/llm/providers/ + - surfsense_local/backend/modules/llm/connections/ + - surfsense_local/backend/modules/llm/resolution.py + - surfsense_local/backend/modules/artifacts/drawn_images/ + - surfsense_local/backend/modules/artifacts/image_edits/ + - surfsense_local/backend/modules/artifacts/script_documents/ + - surfsense_local/backend/modules/agent/prompts/agent.md + - surfsense_local/backend/modules/agent/skills/ + - surfsense_local/backend/modules/agent/tool_endpoint/ + - surfsense_local/backend/worker/studio/ + - surfsense_local/frontend/src/features/studio/ + - surfsense_local/frontend/src/features/models/ + - surfsense_local/frontend/src/features/settings/models/ +--- + +# Pictures: one image model, edits for every model, and comics + +> The user chooses one image model. It draws, and when it can take a picture in it also edits and keeps characters the same from page to page. A model that cannot gets its edits redrawn instead, so Edit works for every model and nothing asks for a second model. On that, the agent makes comics: it plans the pages, confirms the count once, draws one picture per page and binds them into one PDF. + +## Today + +Asked for a comic, the agent wrote a web page and drew the panels as SVG, or drew one picture and stopped. Under that sit four limits in how images work: + +- **Nothing makes a comic a picture job.** The prompt sends only "a picture or infographic" to `surfsense_make_image` ([`agent.md`](../../../surfsense_local/backend/modules/agent/prompts/agent.md)); the web page skill's description, "an explainer … a figure that shows how something works", drawn "as inline SVG", is the nearest match; and the tool's own description says "confirm before making more than one" and sends exact words "to the document around the picture" ([`make_image.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/make_image.py)). +- **Editing needs a second model most users never choose.** Studio's Edit and the agent's edit run only on the `image_edit` slot ([ADR 0077](../../adr/0077-an-image-edit-is-the-next-version.md)). Few local models can fill it, and an empty slot answers "Needs an image editing model" with a link to Settings. The image model the user did choose is never asked, though most remote ones take a picture in. +- **Editing through OpenRouter fails.** A remote edit always posts `/images/edits` ([`image_edit.py`](../../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py)), and OpenRouter has no such route: it answers 404. +- **Every picture is the model's default shape, and drawn from words alone.** No request sends a size or a reference picture, so a comic page comes out square and its characters change from page to page. A local page is also past a tool call's limit, and a document naming a picture still being drawn is refused until the user's next message ([agent, Known gaps](../../architecture/agent.md#known-gaps)). + +## What changes + +| | Today | With this | +|---|---|---| +| Models the user chooses for pictures | an image model, and an edit model for edits | one image model; a different edit model only if they want one | +| Edit, with a model that takes pictures in | needs the edit slot filled | a real edit by the image model | +| Edit, with a model that does not | "Needs an image editing model", and a link to Settings | the picture drawn again from its prompt and the change, said in one neutral line | +| Editing through OpenRouter | 404 | works, through `/images` with `input_references` | +| Settings › Image models | no sign of which models edit | an **Edits** label on those that do, tooltip "Can edit pictures" | +| "Make a comic about …" | a web page with SVG panels, or one picture | a plan, one confirmation, one picture per page, one PDF | +| Characters across pages | redrawn from words each time | page 1 given to every later page as a reference | +| A page's shape | the model's default | `portrait`, `landscape` or `square` | +| A document waiting on a picture still being drawn | refused until the user's next message | a pending version that renders when its pictures are drawn | + +## Parts + +| Part | File | Owns | +|---|---|---| +| **Image models** | [`01-image-models.md`](01-image-models.md) | what a model can do, which model handles a picture, one client per provider, edits and redraws | +| **The agent's picture tool** | [`02-make-image-tool.md`](02-make-image-tool.md) | `shape` and `references` on `surfsense_make_image`, and its description | +| **Comics** | [`03-comics.md`](03-comics.md) | the `surfsense-pictures` skill, the routing, the PDF | +| **The render that waits** | [`04-waiting-render.md`](04-waiting-render.md) | a document that renders once its pictures are drawn | +| **In the app** | [`05-in-the-app.md`](05-in-the-app.md) | Settings' label, Studio's Edit, the words users see | + +## The spike + +Run on 10 Nov 2026 against OpenRouter with one four-page story, "The Lighthouse Cat": page 1 drawn from words with a portrait size, pages 2 to 4 drawn with page 1 as their reference, the four bound into a PDF. Six models, 24 pictures. The script and its outputs stayed in the scratchpad. + +| Model | Lettering | Characters held | Follows the script | Seconds a page | +|---|---|---|---|---| +| `openai/gpt-image-2.6-flare` | exact | yes | every line and panel | 45–75 | +| `google/gemini-nano-banana-2.1` | exact | yes | adds lines of its own | fast | +| `bytedance-seed/seedream-5-0-pro` | exact | yes | close; adds a page number on some pages | fast | +| `qwen/qwen-image-3-pro` | good | yes | repeats a caption | 130–195, one upstream 504 | +| `black-forest-labs/flux-3-image` | good | mostly | draws two cats in one panel | 65–80 | +| `black-forest-labs/flux-3-image-klein` | garbled ("WENT CUT", labels unreadable) | yes | repeats captions | 30–50 | + +What it settled: + +- **OpenRouter has no `/images/edits`.** It has two kinds of image model. Pure image models (GPT image, FLUX, Qwen) refuse `/chat/completions` with a 404 that names `/api/v1/images`; chat image models (Gemini, Seedream) take it. `POST /images` with `input_references`, a list of `{"type": "image_url", "image_url": {"url": }}`, and `aspect_ratio` worked for every pure image model tried, so it is the one route for a picture in. +- **`size` works.** `1024x1536` on `/images/generations` came back portrait from all six; FLUX gave its own 880 x 1184. +- **A reference holds a character on every model tried**, with no edit model involved. +- **Lettering is good enough on the best three** to let the model draw speech bubbles; klein shows why the PDF also needs a typed fallback. +- **A page PDF of PNGs is 10 to 28 MB** for four pages. JPEG pages are needed. +- **A read timeout must not be retried**: the provider may still be drawing, and a second request bills a second picture. + +Not spiked: a local model with references (no image model is installed on the machine the spike ran on), and providers other than OpenRouter. Both are spikes before their part's code ([image models](01-image-models.md#spikes-before-code)). + +## Build order + +On one branch, `feat/comics`, a commit per step, each test first: + +1. Image models: the capability, the rule, the clients ([01](01-image-models.md)). +2. Edits for every model: the edit job on the rule, and redraws ([01](01-image-models.md#edits-and-redraws)). +3. `shape` and `references` on the tool ([02](02-make-image-tool.md)). +4. The `surfsense-pictures` skill and its routing ([03](03-comics.md)). +5. The comic PDF ([03](03-comics.md#the-pdf)). +6. The render that waits ([04](04-waiting-render.md)). +7. Settings' label and Studio's Edit ([05](05-in-the-app.md)). +8. Docs: architecture, ADRs, user docs, and this proposal deleted ([below](#when-it-ships)). + +## Decisions + +- **One image model, not two slots.** Whether a model can take a picture in is a fact about the model, so it is read from the model, not asked of the user. The edit slot stays as an override for the few who want another model for edits. +- **Every model can be edited.** A model that cannot take a picture in redraws from the picture's prompt and the change. It is not an edit, and the version says so, but the user never meets a dead end or a setting. +- **The agent's own tool, never opencode's.** opencode's `experimental_image_generate` finishes after the turn ends and needs opencode 2.2 ([ADR 0087](../../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). It is not used, now or as a fallback. +- **Comics are pictures bound into a PDF.** Not a web page, not a single picture, not a new Studio format. +- **Nothing in Studio or chat points to Settings** for pictures. A model that cannot do something is described in a neutral line, never as a warning. + +## Not decided here + +- Grouping a comic's pages under its PDF in Studio, so twelve pages are not twelve rows. +- A comic button in Studio, which would need a chat model to write the plan from sources. +- Panels lettered by SurfSense from wordless pictures, with speech bubbles the PDF script draws. +- A server limit on pictures per turn, beyond the skill's twelve pages. + +## When it ships + +- [studio](../../architecture/studio.md): image models and the rule, the clients, edits and redraws, `shape` and `references` in the recipe. +- [agent](../../architecture/agent.md): the skill, the tool's arguments, the render that waits, and the Known gaps on slow pictures and lettering deleted. +- [selection](../../architecture/local-models/selection.md): the edit slot as an override. +- ADRs: one amending [ADR 0077](../../adr/0077-an-image-edit-is-the-next-version.md) (the image model edits; a model that cannot redraws), one amending [ADR 0087](../../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md) (a document waits on its pictures on the server). +- The user docs: [`guides/images.mdx`](../../../www/content/docs/v2/guides/images.mdx), [`studio.mdx`](../../../www/content/docs/v2/studio.mdx), [`agent.mdx`](../../../www/content/docs/v2/agent.mdx) and the image model pages under `models/`. From bb5504535cff518750ae215aa3fd65807950e8cd Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:50:47 +0530 Subject: [PATCH 03/33] feat(local): edit pictures with any image model, and draw them with a shape and references The image model edits when it takes a picture in, read from its connection's input_modalities or the catalog; the image_edit slot becomes an optional override, and with no model that takes pictures an edit is drawn again from the prompt the picture was drawn with. Pictures go to /images/edits, or to /images with input_references where a server has no edit route, as OpenRouter. surfsense_make_image takes shape and references, so a comic's pages keep page 1's characters, and never points the agent to Settings. --- surfsense_local/backend/api/main.py | 2 + .../modules/agent/tool_endpoint/drawn_how.py | 56 +++++ .../modules/agent/tool_endpoint/make_image.py | 147 +++++++++---- .../modules/artifacts/drawn_images/recipe.py | 27 ++- .../modules/artifacts/drawn_images/service.py | 5 +- .../modules/artifacts/image_edits/model.py | 17 +- .../modules/llm/connections/service.py | 29 ++- .../modules/llm/pictures_in/__init__.py | 0 .../backend/modules/llm/pictures_in/kept.py | 20 ++ .../backend/modules/llm/pictures_in/model.py | 54 +++++ .../backend/modules/llm/pictures_in/router.py | 23 ++ .../llm/providers/openai_compatible/image.py | 50 +++-- .../providers/openai_compatible/image_edit.py | 169 ++++++++++---- .../modules/llm/providers/picture_shape.py | 44 ++++ .../modules/llm/providers/protocols.py | 17 +- .../modules/llm/providers/sdcpp/editor.py | 109 +++------ .../modules/llm/providers/sdcpp/generator.py | 29 ++- .../llm/providers/sdcpp/launch_size.py | 21 ++ .../modules/llm/providers/sdcpp/native_job.py | 81 +++++++ .../backend/modules/llm/resolution.py | 7 + .../backend/modules/llm/selection.py | 19 +- .../agent/test_agent_makes_image.py | 4 +- .../integration/agent/test_make_image.py | 208 +++++++++++++----- .../integration/artifacts/test_image_edit.py | 108 ++++++--- .../integration/llm/test_picture_model.py | 170 ++++++++++++++ .../unit/llm/providers/sdcpp/test_editor.py | 30 ++- .../llm/providers/sdcpp/test_generator.py | 70 ++++++ .../llm/providers/test_remote_image_edit.py | 94 +++++++- .../tests/unit/llm/test_openai_compatible.py | 85 +++++++ surfsense_local/backend/worker/studio/job.py | 39 +++- .../studio/media/visual/drawn/pipeline.py | 31 ++- .../studio/media/visual/drawn/references.py | 32 +++ .../worker/studio/media/visual/edit/base.py | 4 + .../studio/media/visual/edit/pipeline.py | 12 +- .../studio/media/visual/image/pipeline.py | 2 + .../media/visual/infographic/pipeline.py | 7 +- .../studio/media/visual/redraw/__init__.py | 0 .../media/visual/redraw/drawing_prompt.py | 43 ++++ .../studio/media/visual/redraw/pipeline.py | 51 +++++ 39 files changed, 1597 insertions(+), 319 deletions(-) create mode 100644 surfsense_local/backend/modules/agent/tool_endpoint/drawn_how.py create mode 100644 surfsense_local/backend/modules/llm/pictures_in/__init__.py create mode 100644 surfsense_local/backend/modules/llm/pictures_in/kept.py create mode 100644 surfsense_local/backend/modules/llm/pictures_in/model.py create mode 100644 surfsense_local/backend/modules/llm/pictures_in/router.py create mode 100644 surfsense_local/backend/modules/llm/providers/picture_shape.py create mode 100644 surfsense_local/backend/modules/llm/providers/sdcpp/launch_size.py create mode 100644 surfsense_local/backend/modules/llm/providers/sdcpp/native_job.py create mode 100644 surfsense_local/backend/tests/integration/llm/test_picture_model.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_generator.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/drawn/references.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/redraw/__init__.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/redraw/pipeline.py diff --git a/surfsense_local/backend/api/main.py b/surfsense_local/backend/api/main.py index 7755375bc1..a13f63d5c4 100644 --- a/surfsense_local/backend/api/main.py +++ b/surfsense_local/backend/api/main.py @@ -61,6 +61,7 @@ from modules.llm.model_route.router import router as model_route_router from modules.llm.model_type import ModelType from modules.llm.models import SelectedModel +from modules.llm.pictures_in.router import router as pictures_in_router from modules.llm.residency import warm_selected from modules.llm.router import router as llm_router from modules.llm.subscriptions.claude.errors import ( @@ -260,6 +261,7 @@ def create_app() -> FastAPI: app.include_router(search_server_router) app.include_router(folders_router) app.include_router(llm_router) + app.include_router(pictures_in_router) app.include_router(model_route_router) app.include_router(transcription_windows_router) app.include_router(chat_router) diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/drawn_how.py b/surfsense_local/backend/modules/agent/tool_endpoint/drawn_how.py new file mode 100644 index 0000000000..50f5ddc45c --- /dev/null +++ b/surfsense_local/backend/modules/agent/tool_endpoint/drawn_how.py @@ -0,0 +1,56 @@ +"""Where a picture was drawn, and how when not as asked, for the agent to act on: +a comic draws wordless pages on this computer, and says when faces may differ.""" + +from sqlalchemy.orm import Session + +from modules.artifacts.drawn_images.recipe import image_drawing_of +from modules.artifacts.image_edits.recipe import image_edit_of +from modules.artifacts.models import Artifact +from modules.llm.model_type import ModelType +from modules.llm.models import ProviderConnection, SelectedModel +from modules.llm.pictures_in import model as pictures_in +from modules.llm.providers.sdcpp import provider as sdcpp + +REDRAWN = ( + "Drawn again from its prompt and the change: this image model takes no picture " + "in, so it may not keep the rest of the picture." +) +FROM_WORDS = ( + "Drawn from words alone: this image model takes no picture in, so the " + "characters may differ from the references." +) + + +def drawn_how(session: Session, artifact: Artifact) -> list[str]: + meta = artifact.artifact_metadata or {} + redrawn, from_words = bool(meta.get("redrawn")), bool(meta.get("drawn_from_words")) + drawing = image_drawing_of(meta) + took_pictures = not (redrawn or from_words) and ( + image_edit_of(meta) is not None + or (drawing is not None and bool(drawing.references)) + ) + lines = [_where(session, took_pictures)] + if redrawn: + lines.append(REDRAWN) + if from_words: + lines.append(FROM_WORDS) + return [line for line in lines if line] + + +def _where(session: Session, took_pictures: bool) -> str: + slot = ( + ModelType.IMAGE_EDIT + if took_pictures and pictures_in.picture_source(session) == "edit_model" + else ModelType.IMAGE_GEN + ) + selected = session.get(SelectedModel, slot) + if selected is None: + return "" + if selected.provider == sdcpp.PROVIDER: + return "Drawn on this computer." + connection = ( + session.get(ProviderConnection, selected.connection_id) + if selected.connection_id is not None + else None + ) + return f"Drawn by {connection.label}." if connection is not None else "" diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py index bf765bd01a..ca839506fd 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py @@ -3,7 +3,7 @@ import threading import time from dataclasses import dataclass -from typing import Any +from typing import Any, cast from fastapi import HTTPException from PIL import Image @@ -11,6 +11,7 @@ from sqlalchemy.orm import Session from modules.agent.previews.inline_images import inline_image +from modules.agent.tool_endpoint.drawn_how import drawn_how from modules.agent.tool_endpoint.failed_renders import ( IMAGES, FailedRunError, @@ -36,7 +37,11 @@ ToolResult, ) from modules.agent.tool_endpoint.turn_scope import TurnScope -from modules.artifacts.drawn_images.names import image_name +from modules.artifacts.drawn_images.names import ( + ImageStillDrawingError, + image_name, + kept_picture, +) from modules.artifacts.drawn_images.service import create_drawn_image from modules.artifacts.image_edits.editable import EDITABLE_FORMATS, has_picture from modules.artifacts.image_edits.model import require_image_edit_model @@ -45,6 +50,8 @@ from modules.artifacts.script_documents.version import version_of from modules.documents.models import Document, DocumentStatus from modules.embedding.active import EmbeddingNotChosenError +from modules.llm.pictures_in import model as pictures_in +from modules.llm.providers.picture_shape import SHAPES, Shape from shared.config import get_storage_settings TOOL_NAME = "make_image" @@ -56,6 +63,8 @@ # minutes, past the tool call's hard limit, so the call answers without it. WAIT_SECONDS = 150 CALL_SECONDS = TOOL_CALL_SECONDS - 10 +# Bounds what one call uploads; page 1 and a character sheet are what matter. +MAX_REFERENCES = 4 _ENDED = (DocumentStatus.READY, DocumentStatus.FAILED, DocumentStatus.CANCELLED) _starting = threading.Lock() @@ -63,15 +72,17 @@ LISTING: dict[str, Any] = { "name": TOOL_NAME, "description": ( - "Draw a picture or an infographic with the user's image model, or edit one " + "Draw a picture or an infographic with the user's image model, or change one " "you made in this chat, and keep it in Studio. Only when the user asks for " - "one: each is billed, or takes minutes on their computer, so confirm before " - "making more than one. Read any sources first and write the whole prompt " - "yourself. Draw no data chart: numbers belong in a real chart. Words that " - "must be exact belong in the document around the picture, since image models " - "garble lettering. Write one self-contained prompt, as short as it can be " - "while still saying what to draw. Returns the picture's name, to place it in a document or " - "a web page, and the picture itself to check." + "one. When a request needs more than one picture, confirm how many once, " + "before the first: each is billed, or takes minutes on their computer. For a " + "comic, a storyboard or any set of pictures, load the surfsense-pictures " + "skill first. Read any sources first and write the whole prompt yourself. " + "Draw no data chart: numbers belong in a real chart. Lettering may come out " + "misspelt: keep words in a picture short, and put text that must be exact in " + "the document around it. Write one self-contained prompt, as short as it can " + "be while still saying what to draw. Returns the picture's name, to place it " + "in a document or a web page, and the picture itself to check." ), "inputSchema": { "type": "object", @@ -104,6 +115,24 @@ "The result is its next version. Leave out for a new picture." ), }, + "shape": { + "type": "string", + "enum": list(SHAPES), + "description": ( + "portrait for a page, landscape for a strip or a banner, square. " + "Left out, the model's own." + ), + }, + "references": { + "type": "array", + "items": {"type": "string"}, + "maxItems": MAX_REFERENCES, + "description": ( + "Names of pictures you made in this chat, each ready, for a new " + "picture to keep their characters and style, as a comic's later " + "pages keep page 1's. Not with artifact_id." + ), + }, }, "required": ["prompt"], }, @@ -116,6 +145,8 @@ class _Request: title: str | None style: str base_artifact_id: int | None + shape: Shape | None = None + references: tuple[str, ...] = () def make( @@ -187,8 +218,29 @@ def _request(arguments: dict[str, Any]) -> _Request: ) if base is None and (not isinstance(title, str) or not title.strip()): raise ToolCallError("Give the picture a title.") + shape = arguments.get("shape") or None + if shape is not None and shape not in SHAPES: + raise ToolCallError(f"shape must be one of: {', '.join(SHAPES)}.") + references = arguments.get("references") or [] + if not isinstance(references, list) or not all( + isinstance(name, str) and name.strip() for name in references + ): + raise ToolCallError("references must be a list of picture names.") + if references and base is not None: + raise ToolCallError( + "references are for a new picture: leave out artifact_id, or references." + ) + if len(references) > MAX_REFERENCES: + raise ToolCallError( + f"Give at most {MAX_REFERENCES} references: the pictures that matter most." + ) return _Request( - prompt.strip(), title.strip() if isinstance(title, str) else None, style, base + prompt.strip(), + title.strip() if isinstance(title, str) else None, + style, + base, + cast(Shape, shape) if shape is not None else None, + tuple(dict.fromkeys(name.strip() for name in references)), ) @@ -232,9 +284,32 @@ def _new_picture(session: Session, scope: TurnScope, request: _Request) -> Artif prompt=request.prompt, format=request.style, chat_thread_id=scope.thread_id, + shape=request.shape, + references=_reference_ids(session, scope, request.references), ) +def _reference_ids( + session: Session, scope: TurnScope, names: tuple[str, ...] +) -> tuple[int, ...]: + """Checked as placing them is: this chat's, ready, a still picture.""" + ids: list[int] = [] + for name in names: + try: + kept_picture(session, scope.workspace_id, scope.thread_id, name) + except ImageStillDrawingError as error: + raise ToolCallError( + f"{name} is still being drawn: wait for it, then draw this one." + ) from error + except LookupError as error: + raise ToolCallError( + f"This chat has no picture {name} to keep: name one {TOOL_NAME} gave " + "in this chat." + ) from error + ids.append(int(name.removeprefix("image-"))) + return tuple(ids) + + def _edited_picture( session: Session, scope: TurnScope, base_id: int, instruction: str ) -> Artifact: @@ -266,16 +341,18 @@ def _shown( placeable = has_picture(artifact) primary = next(f for f in artifact.files if f.role is ArtifactFileRole.PRIMARY) storage_key = primary.storage_key - editable = _edit_model_chosen(session) + edits = pictures_in.picture_source(session) is not None + how = drawn_how(session, artifact) session.commit() lines = [ first_line(made), - _how_to_use(made.id, editable) if placeable else _NOT_PLACEABLE, + *how, + _how_to_use(made.id, edits) if placeable else _NOT_PLACEABLE, ] images: tuple[InlineImage, ...] = () if placeable: if scope.reads_images(): - images = _picture(storage_key, editable, lines) + images = _picture(storage_key, lines) else: lines.append(_UNSEEN) text = "\n".join(lines) @@ -286,41 +363,28 @@ def _shown( "The image model gave a format documents cannot place. It is in Studio for the " "user to open: tell them so." ) -# A redraw would bill another picture the user did not ask for. -_NO_EDIT_MODEL = ( - "No image editing model is chosen, so you cannot alter this picture: do not draw " - "it again unless the user asks. If it is wrong, tell the user what is wrong, and " - "that they can choose a model in Settings > Image edit." -) _UNSEEN = ( "You cannot see it: the selected model cannot read images. Do not describe it " "beyond what you asked for." ) -def _edit_model_chosen(session: Session) -> bool: - """Whether an edit could run: one offered without a model could only be refused.""" - try: - require_image_edit_model(session) - except HTTPException: - return False - return True - - -def _how_to_use(artifact_id: int, editable: bool) -> str: +def _how_to_use(artifact_id: int, edits: bool) -> str: name = image_name(artifact_id) - place = ( + change = f"To change it, call this tool again with artifact_id {artifact_id}" + if not edits: + # Neutral: every image model can change a picture, this one by redrawing. + change += ( + ": this image model takes no picture in, so the change is drawn again " + "from its prompt and may not keep the rest" + ) + return ( f"Its name is {name}. To place it, pass {name} in a render's images, or in " - f'a web page write src="surfsense-image:{name}".' + f'a web page write src="surfsense-image:{name}". {change}.' ) - if editable: - return f"{place} To change it, call this tool again with artifact_id {artifact_id}." - return f"{place} {_NO_EDIT_MODEL}" -def _picture( - storage_key: str, editable: bool, lines: list[str] -) -> tuple[InlineImage, ...]: +def _picture(storage_key: str, lines: list[str]) -> tuple[InlineImage, ...]: path = get_storage_settings().data_dir / storage_key try: shown = inline_image(path) @@ -329,12 +393,7 @@ def _picture( return () lines.append( "Look at it before you answer: image models get lettering, hands and counts " - "wrong. " - + ( - "If it is wrong, change it or tell the user." - if editable - else "If it is wrong, tell the user, as above." - ) + "wrong. If it is wrong, change it or tell the user." ) return (shown,) diff --git a/surfsense_local/backend/modules/artifacts/drawn_images/recipe.py b/surfsense_local/backend/modules/artifacts/drawn_images/recipe.py index a7d119e981..2a3078bcdd 100644 --- a/surfsense_local/backend/modules/artifacts/drawn_images/recipe.py +++ b/surfsense_local/backend/modules/artifacts/drawn_images/recipe.py @@ -1,9 +1,11 @@ -"""How a picture the agent asked for is made, so Regenerate paints the same prompt.""" +"""How a picture the agent asked for is made, so Regenerate paints the same prompt, +at the same shape, from the same reference pictures.""" from dataclasses import dataclass -from typing import Any +from typing import Any, cast from modules.artifacts.studio_documents.recipe import RECIPE_KEY +from modules.llm.providers.picture_shape import SHAPES, Shape IMAGE_DRAWN = "image_drawn" @@ -11,14 +13,29 @@ @dataclass(frozen=True) class ImageDrawing: prompt: str + shape: Shape | None = None + # Artifact ids of the pictures it keeps the characters and style of. + references: tuple[int, ...] = () -def image_drawn(prompt: str) -> dict[str, Any]: - return {"kind": IMAGE_DRAWN, "prompt": prompt} +def image_drawn( + prompt: str, shape: Shape | None = None, references: tuple[int, ...] = () +) -> dict[str, Any]: + recipe: dict[str, Any] = {"kind": IMAGE_DRAWN, "prompt": prompt} + if shape is not None: + recipe["shape"] = shape + if references: + recipe["references"] = list(references) + return recipe def image_drawing_of(metadata: dict[str, Any] | None) -> ImageDrawing | None: recipe = (metadata or {}).get(RECIPE_KEY) if not isinstance(recipe, dict) or recipe.get("kind") != IMAGE_DRAWN: return None - return ImageDrawing(recipe["prompt"]) + shape = recipe.get("shape") + return ImageDrawing( + recipe["prompt"], + cast(Shape, shape) if shape in SHAPES else None, + tuple(int(i) for i in recipe.get("references") or ()), + ) diff --git a/surfsense_local/backend/modules/artifacts/drawn_images/service.py b/surfsense_local/backend/modules/artifacts/drawn_images/service.py index a88618876b..1f6db1a469 100644 --- a/surfsense_local/backend/modules/artifacts/drawn_images/service.py +++ b/surfsense_local/backend/modules/artifacts/drawn_images/service.py @@ -11,6 +11,7 @@ from modules.artifacts.studio_documents.recipe import RECIPE_KEY from modules.documents.models import Document, DocumentStatus, DocumentType from modules.embedding.active import require_active_index +from modules.llm.providers.picture_shape import Shape logger = logging.getLogger(__name__) @@ -23,6 +24,8 @@ def create_drawn_image( prompt: str, format: str, chat_thread_id: int | None, + shape: Shape | None = None, + references: tuple[int, ...] = (), ) -> Artifact: """Pending until Studio's job paints it. Commits before enqueueing: the worker is another process and must find the row.""" @@ -43,7 +46,7 @@ def create_drawn_image( format=format, chat_thread_id=chat_thread_id, artifact_metadata={ - RECIPE_KEY: image_drawn(prompt), + RECIPE_KEY: image_drawn(prompt, shape, references), "source_document_ids": [], "prompt": prompt, }, diff --git a/surfsense_local/backend/modules/artifacts/image_edits/model.py b/surfsense_local/backend/modules/artifacts/image_edits/model.py index b72cff6ad0..b1336895c5 100644 --- a/surfsense_local/backend/modules/artifacts/image_edits/model.py +++ b/surfsense_local/backend/modules/artifacts/image_edits/model.py @@ -1,17 +1,12 @@ -"""Refuse an edit, or its Retry, that no chosen model can make.""" +"""Refuse an edit, or its Retry, only when no image model is chosen at all: a +model that takes pictures edits, and any other image model draws it again.""" -from fastapi import HTTPException, status from sqlalchemy.orm import Session -from modules.llm.resolution import ModelResolutionError, resolve_image_edit +from modules.artifacts.drawn_images.model import require_image_model +from modules.llm.pictures_in import model as pictures_in def require_image_edit_model(session: Session) -> None: - """Nothing fills the `image_edit` slot, or what fills it is gone; mirrors Refine's `needs_chat`.""" - try: - resolve_image_edit(session) - except ModelResolutionError as error: - raise HTTPException( - status.HTTP_409_CONFLICT, - {"message": "Needs an image editing model", "code": "needs_image_edit"}, - ) from error + if pictures_in.picture_source(session) is None: + require_image_model(session) diff --git a/surfsense_local/backend/modules/llm/connections/service.py b/surfsense_local/backend/modules/llm/connections/service.py index 2a77b5965b..3426857736 100644 --- a/surfsense_local/backend/modules/llm/connections/service.py +++ b/surfsense_local/backend/modules/llm/connections/service.py @@ -76,10 +76,10 @@ def normalize_base_url(value: str) -> str: return urlunsplit((parsed.scheme, netloc, path, "", "")) -def _modalities(entry: dict) -> set[str]: - raw = entry.get("output_modalities") +def _modalities(entry: dict, key: str = "output_modalities") -> set[str]: + raw = entry.get(key) if raw is None and isinstance(entry.get("architecture"), dict): - raw = entry["architecture"].get("output_modalities") + raw = entry["architecture"].get(key) if not isinstance(raw, list): return set() return { @@ -92,6 +92,7 @@ def _modalities(entry: dict) -> set[str]: @dataclass class _Entry: modalities: set[str] = field(default_factory=set) + inputs: set[str] = field(default_factory=set) voices: tuple[str, ...] = () free: bool | None = None @@ -116,6 +117,7 @@ def _entries(payload: object) -> dict[str, _Entry]: continue listed = entries.setdefault(name, _Entry()) listed.modalities.update(_modalities(entry)) + listed.inputs.update(_modalities(entry, "input_modalities")) listed.voices = listed.voices or tuple(voices_of(entry)) if listed.free is None: listed.free = listed_as_free(entry) @@ -123,7 +125,10 @@ def _entries(payload: object) -> dict[str, _Entry]: def _classify( - name: str, modalities: set[str], catalog_provider: str | None = None + name: str, + modalities: set[str], + catalog_provider: str | None = None, + inputs: frozenset[str] | set[str] = frozenset(), ) -> DiscoveredModel: """Resolve one model against every source, in order, first answer winning. @@ -144,6 +149,9 @@ def _classify( if modality in modalities ) ) + if ModelType.IMAGE_GEN in declared and "image" in inputs: + # A picture in as well as out: it can edit, and keep a reference. + declared = (*declared, ModelType.IMAGE_EDIT) return DiscoveredModel(name, declared, "declared", unusable) catalogued = lookup.classify(name, provider=catalog_provider) if catalogued.known: @@ -155,7 +163,8 @@ def _classify( def parse_models(payload: object) -> list[DiscoveredModel]: return [ - _classify(name, listed.modalities) for name, listed in _entries(payload).items() + _classify(name, listed.modalities, inputs=listed.inputs) + for name, listed in _entries(payload).items() ] @@ -177,7 +186,10 @@ async def _request_entries( async def probe_connection(base_url: str, api_key: str | None) -> list[DiscoveredModel]: entries = await _request_entries(base_url, api_key) - return [_classify(name, listed.modalities) for name, listed in entries.items()] + return [ + _classify(name, listed.modalities, inputs=listed.inputs) + for name, listed in entries.items() + ] async def discover_models(connection: ProviderConnection) -> list[DiscoveredModel]: @@ -189,6 +201,7 @@ async def discover_models(connection: ProviderConnection) -> list[DiscoveredMode for name, other in optional.items(): listed = merged.setdefault(name, _Entry()) listed.modalities.update(other.modalities) + listed.inputs.update(other.inputs) listed.voices = listed.voices or other.voices if listed.free is None: listed.free = other.free @@ -199,7 +212,9 @@ async def discover_models(connection: ProviderConnection) -> list[DiscoveredMode return sorted( ( replace( - classified := _classify(name, listed.modalities, provider), + classified := _classify( + name, listed.modalities, provider, listed.inputs + ), voices=tuple(Voice(v, v, None, ()) for v in listed.voices), unusable_reason=unusable_reason( provider, listed.free, classified.unusable_reason diff --git a/surfsense_local/backend/modules/llm/pictures_in/__init__.py b/surfsense_local/backend/modules/llm/pictures_in/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/modules/llm/pictures_in/kept.py b/surfsense_local/backend/modules/llm/pictures_in/kept.py new file mode 100644 index 0000000000..f46bf03510 --- /dev/null +++ b/surfsense_local/backend/modules/llm/pictures_in/kept.py @@ -0,0 +1,20 @@ +"""Whether the chosen image model takes a picture in, kept with the choice. + +Read from the connection's model list when the model is chosen, so nothing asks +the network again when a picture is edited or a page needs a reference. +""" + +from modules.llm.models import SelectedModel + +KEY = "takes_pictures" + + +def kept_takes_pictures(selected: SelectedModel) -> bool: + return bool((selected.settings or {}).get(KEY)) + + +def keep_takes_pictures(selected: SelectedModel, takes_pictures: bool) -> None: + settings = {k: v for k, v in (selected.settings or {}).items() if k != KEY} + if takes_pictures: + settings[KEY] = True + selected.settings = settings or None diff --git a/surfsense_local/backend/modules/llm/pictures_in/model.py b/surfsense_local/backend/modules/llm/pictures_in/model.py new file mode 100644 index 0000000000..f3f5fdc426 --- /dev/null +++ b/surfsense_local/backend/modules/llm/pictures_in/model.py @@ -0,0 +1,54 @@ +"""The one model a picture goes to: the edit model when one is chosen, else the +image model when it takes pictures, else none and the picture is drawn from words.""" + +from dataclasses import dataclass +from typing import Literal + +from sqlalchemy.orm import Session + +from modules.llm.catalog.local.dependencies import get_local_catalog +from modules.llm.model_type import ModelType +from modules.llm.models import SelectedModel +from modules.llm.pictures_in.kept import kept_takes_pictures +from modules.llm.providers.protocols import ImageEditor +from modules.llm.providers.sdcpp import provider as sdcpp +from modules.llm.resolution import ModelResolutionError, image_editor_for + +PictureSource = Literal["edit_model", "image_model"] + + +@dataclass(frozen=True) +class PictureModel: + source: PictureSource + selection: SelectedModel + editor: ImageEditor + + +def picture_model(session: Session) -> PictureModel | None: + """None when no chosen model takes a picture in. Egress and a missing install + raise ModelResolutionError, as resolving any model does.""" + chosen = session.get(SelectedModel, ModelType.IMAGE_EDIT) + if chosen is not None: + return PictureModel( + "edit_model", chosen, image_editor_for(session, chosen).editor + ) + image = session.get(SelectedModel, ModelType.IMAGE_GEN) + if image is None or not _takes_pictures(image): + return None + return PictureModel("image_model", image, image_editor_for(session, image).editor) + + +def picture_source(session: Session) -> PictureSource | None: + """The same answer without building a client: what the app shows.""" + try: + found = picture_model(session) + except ModelResolutionError: + return None + return None if found is None else found.source + + +def _takes_pictures(selected: SelectedModel) -> bool: + if selected.provider == sdcpp.PROVIDER: + installed = get_local_catalog().sdcpp.installed_image(selected.name) + return installed is not None and ModelType.IMAGE_EDIT in installed.types + return kept_takes_pictures(selected) diff --git a/surfsense_local/backend/modules/llm/pictures_in/router.py b/surfsense_local/backend/modules/llm/pictures_in/router.py new file mode 100644 index 0000000000..df98eea266 --- /dev/null +++ b/surfsense_local/backend/modules/llm/pictures_in/router.py @@ -0,0 +1,23 @@ +"""Which model a picture goes to, for the app: an edit, or a redraw when none.""" + +from fastapi import APIRouter +from pydantic import BaseModel + +from api.dependencies import SessionDep +from modules.llm.pictures_in.model import PictureSource, picture_source + +router = APIRouter(prefix="/llm", tags=["llm"]) + + +class PicturesInRead(BaseModel): + # None: every change to a picture is drawn again from its prompt. + source: PictureSource | None + + +@router.get( + "/pictures-in", + response_model=PicturesInRead, + summary="Which chosen model takes a picture in, if any", +) +def read_pictures_in(session: SessionDep) -> PicturesInRead: + return PicturesInRead(source=picture_source(session)) diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py index c054450a47..e353fa95eb 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py @@ -10,6 +10,7 @@ bounded_body, read_image, ) +from modules.llm.providers.picture_shape import SIZE, Shape from modules.llm.providers.protocols import GeneratedImage __all__ = ["NonRetryableImageError", "OpenAICompatibleImageProvider"] @@ -21,6 +22,8 @@ _route_cache: dict[tuple[int, str], str] = {} # Whether an endpoint takes PICTURE_ITSELF; GPT image models refuse it. _asks_for_picture: dict[tuple[int, str], bool] = {} +# Endpoints that refuse a size: they draw their own. +_refuses_size: set[tuple[int, str]] = set() class OpenAICompatibleImageProvider: @@ -37,20 +40,23 @@ def __init__( def _headers(self) -> dict[str, str]: return key_headers(self._base_url, self._api_key) - async def generate(self, model: str, prompt: str) -> GeneratedImage: + async def generate( + self, model: str, prompt: str, *, shape: Shape | None = None + ) -> GeneratedImage: key = (self._connection_id, self._base_url) + size = None if shape is None or key in _refuses_size else SIZE[shape] preferred = _route_cache.get(key, STANDARD_ROUTE) alternate = EXTENSION_ROUTE if preferred == STANDARD_ROUTE else STANDARD_ROUTE picture = _asks_for_picture.get(key, True) try: - status, payload, picture = await self._ask( - preferred, model, prompt, picture + status, payload, picture, size = await self._ask( + preferred, model, prompt, picture, size ) if status in {404, 405}: - status, payload, picture = await self._ask( - alternate, model, prompt, picture + status, payload, picture, size = await self._ask( + alternate, model, prompt, picture, size ) route = alternate else: @@ -65,21 +71,38 @@ async def generate(self, model: str, prompt: str) -> GeneratedImage: _route_cache[key] = route _asks_for_picture[key] = picture + if shape is not None and size is None: + _refuses_size.add(key) return image async def _ask( - self, route: str, model: str, prompt: str, picture: bool - ) -> tuple[int, object, bool]: - """Once more without PICTURE_ITSELF if it is refused: a 400 comes before - any work, so nothing is billed twice.""" - status, payload = await self._post(route, model, prompt, picture) + self, route: str, model: str, prompt: str, picture: bool, size: str | None + ) -> tuple[int, object, bool, str | None]: + """Once more without PICTURE_ITSELF, then without the size, each if it is + refused: a 400 comes before any work, so nothing is billed twice.""" + status, payload = await self._sent(route, model, prompt, picture, size) if status == 400 and picture: - status, payload = await self._post(route, model, prompt, False) picture = False - return status, payload, picture + status, payload = await self._sent(route, model, prompt, picture, size) + if status == 400 and size is not None: + size = None + status, payload = await self._sent(route, model, prompt, picture, size) + return status, payload, picture, size + + async def _sent( + self, route: str, model: str, prompt: str, picture: bool, size: str | None + ) -> tuple[int, object]: + if size is None: + return await self._post(route, model, prompt, picture) + return await self._post(route, model, prompt, picture, size=size) async def _post( - self, route: str, model: str, prompt: str, picture: bool + self, + route: str, + model: str, + prompt: str, + picture: bool, + size: str | None = None, ) -> tuple[int, object]: async with ( httpx.AsyncClient(timeout=TIMEOUT, headers=self._headers()) as client, @@ -90,6 +113,7 @@ async def _post( "model": model, "prompt": prompt, **(PICTURE_ITSELF if picture else {}), + **({"size": size} if size else {}), }, ) as reply, ): diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py index 170c33a41b..8cb9559fa3 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py @@ -1,5 +1,8 @@ -"""A remote edit model, reached on `/images/edits` with one picture and its instruction.""" +"""A remote model given pictures and an instruction: an edit, or a page that keeps +its references. `/images/edits` as OpenAI takes it; `/images` with +`input_references` where a server has no edit route, as OpenRouter.""" +import base64 import json import httpx @@ -12,14 +15,25 @@ bounded_body, read_image, ) +from modules.llm.providers.picture_shape import ASPECT_RATIO, SIZE, Shape from modules.llm.providers.protocols import GeneratedImage # An edit reads the picture as well as drawing one: slower than a generation. TIMEOUT = httpx.Timeout(300.0, connect=5.0) -ROUTE = "/images/edits" +EDITS = "/images/edits" +REFERENCES = "/images" +# A missing route answers before any work, so trying the other bills nothing. +_NO_ROUTE = {404, 405} -# Whether an endpoint takes PICTURE_ITSELF; GPT image models refuse it. -_asks_for_picture: dict[str, bool] = {} +# Optional fields, dropped one at a time when a server refuses them: GPT image +# models refuse `response_format` and take a size, so each is its own. +PICTURE = "response_format" +SHAPE = "shape" +_OPTIONAL = (PICTURE, SHAPE) + +# Per server: which route takes pictures, and which optional fields it refuses. +_routes: dict[str, str] = {} +_refused: dict[str, frozenset[str]] = {} class OpenAICompatibleImageEditor: @@ -35,15 +49,32 @@ def __init__( self._transport = transport async def edit( - self, model: str, prompt: str, image: GeneratedImage + self, + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: Shape | None = None, ) -> GeneratedImage: - picture = _asks_for_picture.get(self._base_url, True) + route = _routes.get(self._base_url, EDITS) + refused = _refused.get(self._base_url, frozenset()) try: - status, payload = await self._post(model, prompt, image, picture) - # Once more without it if refused: a 400 comes before any work. - if status == 400 and picture: - picture = False - status, payload = await self._post(model, prompt, image, picture) + status, payload = await self._post( + route, model, prompt, images, shape, refused + ) + if status in _NO_ROUTE and route == EDITS: + route = REFERENCES + status, payload = await self._post( + route, model, prompt, images, shape, refused + ) + # Without each optional field in turn if refused: a 400 comes before any work. + for field in _OPTIONAL: + if status != 400 or field in refused: + continue + refused = refused | {field} + status, payload = await self._post( + route, model, prompt, images, shape, refused + ) if status >= 400: raise NonRetryableImageError(f"image endpoint returned HTTP {status}") edited = await read_image(payload) @@ -51,39 +82,93 @@ async def edit( raise except (httpx.HTTPError, ValueError, KeyError, json.JSONDecodeError) as error: raise NonRetryableImageError(str(error)) from error - _asks_for_picture[self._base_url] = picture + _routes[self._base_url] = route + _refused[self._base_url] = refused return edited async def _post( - self, model: str, prompt: str, image: GeneratedImage, picture: bool + self, + route: str, + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + shape: Shape | None, + refused: frozenset[str], ) -> tuple[int, object]: - extension = image.media_type.removeprefix("image/").replace("jpeg", "jpg") - # No `input_fidelity`: only some GPT image models take it. - async with ( - httpx.AsyncClient( - timeout=TIMEOUT, - headers=key_headers(self._base_url, self._api_key), - transport=self._transport, - ) as client, - client.stream( - "POST", - f"{self._base_url}{ROUTE}", - data={ - "model": model, - "prompt": prompt, - **(PICTURE_ITSELF if picture else {}), - }, - files={ - "image": (f"image.{extension}", image.content, image.media_type) - }, - ) as reply, - ): - content = await bounded_body(reply, MAX_RESPONSE_BYTES) - try: - payload = json.loads(content) - except json.JSONDecodeError: - if reply.status_code >= 400: - payload = {} - else: + async with httpx.AsyncClient( + timeout=TIMEOUT, + headers=key_headers(self._base_url, self._api_key), + transport=self._transport, + ) as client: + if route == REFERENCES: + request = client.stream( + "POST", + f"{self._base_url}{route}", + json=_references_body(model, prompt, images, shape, refused), + ) + else: + request = client.stream( + "POST", + f"{self._base_url}{route}", + data=_edits_fields(model, prompt, shape, refused), + files=_edits_files(images), + ) + async with request as reply: + content = await bounded_body(reply, MAX_RESPONSE_BYTES) + try: + return reply.status_code, json.loads(content) + except json.JSONDecodeError: + if reply.status_code >= 400: + return reply.status_code, {} raise - return reply.status_code, payload + + +def _edits_fields( + model: str, prompt: str, shape: Shape | None, refused: frozenset[str] +) -> dict[str, str]: + # No `input_fidelity`: only some GPT image models take it. + fields = {"model": model, "prompt": prompt} + if PICTURE not in refused: + fields.update(PICTURE_ITSELF) + if shape is not None and SHAPE not in refused: + fields["size"] = SIZE[shape] + return fields + + +def _edits_files(images: tuple[GeneratedImage, ...]) -> list[tuple[str, tuple]]: + # One picture keeps the field every edit server takes; several are OpenAI's `image[]`. + field = "image" if len(images) == 1 else "image[]" + return [ + (field, (f"image-{n}.{_extension(image)}", image.content, image.media_type)) + if len(images) > 1 + else (field, (f"image.{_extension(image)}", image.content, image.media_type)) + for n, image in enumerate(images, 1) + ] + + +def _references_body( + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + shape: Shape | None, + refused: frozenset[str], +) -> dict: + body: dict = { + "model": model, + "prompt": prompt, + "input_references": [ + {"type": "image_url", "image_url": {"url": _data_url(image)}} + for image in images + ], + } + if shape is not None and SHAPE not in refused: + body["aspect_ratio"] = ASPECT_RATIO[shape] + return body + + +def _data_url(image: GeneratedImage) -> str: + return f"data:{image.media_type};base64,{base64.b64encode(image.content).decode()}" + + +def _extension(image: GeneratedImage) -> str: + return image.media_type.removeprefix("image/").replace("jpeg", "jpg") diff --git a/surfsense_local/backend/modules/llm/providers/picture_shape.py b/surfsense_local/backend/modules/llm/providers/picture_shape.py new file mode 100644 index 0000000000..e0843f8fe6 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/picture_shape.py @@ -0,0 +1,44 @@ +"""A picture's proportions, as each kind of image endpoint asks for them.""" + +from typing import Literal + +Shape = Literal["square", "portrait", "landscape"] +SHAPES: tuple[Shape, ...] = ("square", "portrait", "landscape") + +# The sizes OpenAI's image models take; other servers round to their own. +SIZE: dict[Shape, str] = { + "square": "1024x1024", + "portrait": "1024x1536", + "landscape": "1536x1024", +} +ASPECT_RATIO: dict[Shape, str] = { + "square": "1:1", + "portrait": "2:3", + "landscape": "3:2", +} +_RATIO: dict[Shape, tuple[int, int]] = { + "square": (1, 1), + "portrait": (2, 3), + "landscape": (3, 2), +} + + +def local_dimensions(shape: Shape, pixels: int) -> tuple[int, int]: + """About `pixels` in all, at the shape's ratio, each side a multiple of 64 as sd.cpp needs.""" + across, down = _RATIO[shape] + unit = (pixels / (across * down)) ** 0.5 + return _to_64(across * unit), _to_64(down * unit) + + +def _to_64(side: float) -> int: + return max(64, round(side / 64) * 64) + + +def shape_of(width: int, height: int) -> Shape: + """The nearest shape to a picture's own proportions, so a redraw keeps them.""" + ratio = width / height + if ratio > 1.2: + return "landscape" + if ratio < 1 / 1.2: + return "portrait" + return "square" diff --git a/surfsense_local/backend/modules/llm/providers/protocols.py b/surfsense_local/backend/modules/llm/providers/protocols.py index 3aed8c942f..54cb690d02 100644 --- a/surfsense_local/backend/modules/llm/providers/protocols.py +++ b/surfsense_local/backend/modules/llm/providers/protocols.py @@ -2,6 +2,7 @@ from dataclasses import dataclass from typing import Protocol +from modules.llm.providers.picture_shape import Shape from modules.llm.providers.types import Delta, Message, Model @@ -81,14 +82,24 @@ class GeneratedImage: class ImageGenerator(Protocol): - async def generate(self, model: str, prompt: str) -> GeneratedImage: ... + """A model that draws from words alone.""" + + async def generate( + self, model: str, prompt: str, *, shape: Shape | None = None + ) -> GeneratedImage: ... class ImageEditor(Protocol): - """A model that changes a picture as an instruction says, given it as a reference.""" + """A model given pictures as references: an edit of one, or a new picture + that keeps the characters and style of the others.""" async def edit( - self, model: str, prompt: str, image: GeneratedImage + self, + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: Shape | None = None, ) -> GeneratedImage: ... diff --git a/surfsense_local/backend/modules/llm/providers/sdcpp/editor.py b/surfsense_local/backend/modules/llm/providers/sdcpp/editor.py index 16ac29b066..125564c7eb 100644 --- a/surfsense_local/backend/modules/llm/providers/sdcpp/editor.py +++ b/surfsense_local/backend/modules/llm/providers/sdcpp/editor.py @@ -1,32 +1,23 @@ """Studio's edit client for the bundled sd-server: its native job route, with the -picture as a reference image, as every edit model sd.cpp documents is run. +pictures as reference images, as every edit model sd.cpp documents is run. Its `/v1/images/edits` and `/sdapi/v1/img2img` also make the picture the starting noise, which re-draws it at strength 0.75 instead of following the instruction. """ -import asyncio import base64 -import contextlib import io -from collections.abc import Awaitable, Callable +from collections.abc import Callable from contextlib import AbstractAsyncContextManager import httpx from PIL import Image -from modules.llm.providers.openai_compatible.image_response import ( - NonRetryableImageError, - decode_image, -) +from modules.llm.providers.picture_shape import Shape, local_dimensions from modules.llm.providers.protocols import GeneratedImage +from modules.llm.providers.sdcpp.native_job import run_native_job from modules.llm.providers.sdcpp.serving import wait_until_serving -from shared import cancellation - -_TIMEOUT = httpx.Timeout(30.0, connect=5.0) -# A finished job's result is kept 600 seconds, so a poll each second meets it. -_ENDED = ("completed", "failed", "cancelled") class LocalImageEditor: @@ -46,7 +37,12 @@ def __init__( self._transport = transport async def edit( - self, model: str, prompt: str, image: GeneratedImage + self, + model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: Shape | None = None, ) -> GeneratedImage: # Held for the whole edit, as for a drawn image (generator.py). async with self._give_up_text_runtime(): @@ -56,68 +52,27 @@ async def edit( interval=self._interval, transport=self._transport, ) - async with httpx.AsyncClient( - base_url=self._root_url, timeout=_TIMEOUT, transport=self._transport - ) as client: - job = await self._submit(client, prompt, image) - try: - return await self._result(client, job) - except BaseException: - # Only a queued job can be withdrawn; a generating one runs on - # until Electron stops sd-server, once the cancel ends its demand. - await _withdraw(client, job) - raise - - async def _submit( - self, client: httpx.AsyncClient, prompt: str, image: GeneratedImage - ) -> str: - width, height = Image.open(io.BytesIO(image.content)).size - encoded = base64.b64encode(image.content).decode() - # A native request starts from the launch's settings, so steps and - # guidance are the model's own; the size must be the picture's. - body = { - "prompt": prompt, - "ref_images": [f"data:{image.media_type};base64,{encoded}"], - "width": width, - "height": height, - "output_format": "png", - } - reply = await _call(client.post("/sdcpp/v1/img_gen", json=body)) - return reply.json()["id"] - - async def _result(self, client: httpx.AsyncClient, job: str) -> GeneratedImage: - while True: - cancellation.raise_if_cancelled() - state = (await _call(client.get(f"/sdcpp/v1/jobs/{job}"))).json() - if state["status"] == "completed": - return decode_image(state["result"]["images"][0]["b64_json"], None) - if state["status"] in _ENDED: - error = state.get("error") or {} - raise NonRetryableImageError( - f"The image model could not edit this picture: " - f"{error.get('message') or state['status']}" - ) - await asyncio.sleep(self._interval) - - -async def _call(request: Awaitable[httpx.Response]) -> httpx.Response: - """The reply, or the edit's failure: a full queue, a job gone, a server down.""" - try: - reply = await request - except httpx.HTTPError as error: - raise NonRetryableImageError( - f"The image model could not be reached: {error}" - ) from error - if reply.status_code >= 400: - raise NonRetryableImageError(f"sd-server returned HTTP {reply.status_code}") - return reply - - -# The text runtime is still held while it is asked; a cancel ends the demand -# anyway, and Electron then stops sd-server. -_WITHDRAW_TIMEOUT = httpx.Timeout(2.0) + return await run_native_job( + self._root_url, + _body(prompt, images, shape), + interval=self._interval, + transport=self._transport, + ) -async def _withdraw(client: httpx.AsyncClient, job: str) -> None: - with contextlib.suppress(httpx.HTTPError): - await client.post(f"/sdcpp/v1/jobs/{job}/cancel", timeout=_WITHDRAW_TIMEOUT) +def _body(prompt: str, images: tuple[GeneratedImage, ...], shape: Shape | None) -> dict: + # A native request starts from the launch's settings, so steps and guidance + # are the model's own; the size must be the picture's, or the shape's. + width, height = Image.open(io.BytesIO(images[0].content)).size + if shape is not None: + width, height = local_dimensions(shape, width * height) + return { + "prompt": prompt, + "ref_images": [ + f"data:{image.media_type};base64,{base64.b64encode(image.content).decode()}" + for image in images + ], + "width": width, + "height": height, + "output_format": "png", + } diff --git a/surfsense_local/backend/modules/llm/providers/sdcpp/generator.py b/surfsense_local/backend/modules/llm/providers/sdcpp/generator.py index 1c65e28259..6102cd4688 100644 --- a/surfsense_local/backend/modules/llm/providers/sdcpp/generator.py +++ b/surfsense_local/backend/modules/llm/providers/sdcpp/generator.py @@ -1,15 +1,21 @@ """Studio's image client for the bundled sd-server: the OpenAI-compatible one, once sd-server serves the model the job needs and the text runtime has given up -the graphics card.""" +the graphics card. A shape goes through the native job route, the one that +takes a width and height per request.""" from collections.abc import Callable from contextlib import AbstractAsyncContextManager import httpx +from modules.llm.providers.picture_shape import Shape, local_dimensions from modules.llm.providers.protocols import GeneratedImage, ImageGenerator +from modules.llm.providers.sdcpp.native_job import run_native_job from modules.llm.providers.sdcpp.serving import wait_until_serving +# sd-server's own default when a model's launch names no size. +DEFAULT_PIXELS = 1024 * 1024 + class LocalImageGenerator: def __init__( @@ -19,6 +25,7 @@ def __init__( served_file: str, give_up_text_runtime: Callable[[], AbstractAsyncContextManager[None]], *, + pixels: int = DEFAULT_PIXELS, interval: float = 1.0, transport: httpx.AsyncBaseTransport | None = None, ) -> None: @@ -26,10 +33,13 @@ def __init__( self._root_url = root_url self._served_file = served_file self._give_up_text_runtime = give_up_text_runtime + self._pixels = pixels self._interval = interval self._transport = transport - async def generate(self, model: str, prompt: str) -> GeneratedImage: + async def generate( + self, model: str, prompt: str, *, shape: Shape | None = None + ) -> GeneratedImage: # sd-server loads its weights on this post, and Windows reports the card # as free while llama-server holds it. Held for the whole image, so a # chat sent meanwhile cannot load a text model back under it. @@ -40,4 +50,17 @@ async def generate(self, model: str, prompt: str) -> GeneratedImage: interval=self._interval, transport=self._transport, ) - return await self._inner.generate(model, prompt) + if shape is None: + return await self._inner.generate(model, prompt) + width, height = local_dimensions(shape, self._pixels) + return await run_native_job( + self._root_url, + { + "prompt": prompt, + "width": width, + "height": height, + "output_format": "png", + }, + interval=self._interval, + transport=self._transport, + ) diff --git a/surfsense_local/backend/modules/llm/providers/sdcpp/launch_size.py b/surfsense_local/backend/modules/llm/providers/sdcpp/launch_size.py new file mode 100644 index 0000000000..213563b659 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/sdcpp/launch_size.py @@ -0,0 +1,21 @@ +"""The pixel count a local model is launched to draw, from its `-W` and `-H`.""" + +from collections.abc import Sequence +from itertools import pairwise + +from modules.llm.providers.sdcpp.generator import DEFAULT_PIXELS + + +def launch_pixels(args: Sequence[str]) -> int: + """Its reviewed size, kept when a shape changes the proportions.""" + width, height = _flag(args, "-W"), _flag(args, "-H") + if width is None or height is None: + return DEFAULT_PIXELS + return width * height + + +def _flag(args: Sequence[str], name: str) -> int | None: + for flag, value in pairwise(args): + if flag == name and value.isdigit(): + return int(value) + return None diff --git a/surfsense_local/backend/modules/llm/providers/sdcpp/native_job.py b/surfsense_local/backend/modules/llm/providers/sdcpp/native_job.py new file mode 100644 index 0000000000..8334808ecd --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/sdcpp/native_job.py @@ -0,0 +1,81 @@ +"""One picture through sd-server's native job route: submit, poll each second, +and withdraw it if the wait ends early. The only route that takes reference +images and a size per request, so both drawing and editing use it.""" + +import asyncio +import contextlib +from collections.abc import Awaitable + +import httpx + +from modules.llm.providers.openai_compatible.image_response import ( + NonRetryableImageError, + decode_image, +) +from modules.llm.providers.protocols import GeneratedImage +from shared import cancellation + +_TIMEOUT = httpx.Timeout(30.0, connect=5.0) +# A finished job's result is kept 600 seconds, so a poll each second meets it. +_ENDED = ("completed", "failed", "cancelled") + + +async def run_native_job( + root_url: str, + body: dict, + *, + interval: float, + transport: httpx.AsyncBaseTransport | None, +) -> GeneratedImage: + async with httpx.AsyncClient( + base_url=root_url, timeout=_TIMEOUT, transport=transport + ) as client: + reply = await _call(client.post("/sdcpp/v1/img_gen", json=body)) + job = reply.json()["id"] + try: + return await _result(client, job, interval) + except BaseException: + # Only a queued job can be withdrawn; a generating one runs on + # until Electron stops sd-server, once the cancel ends its demand. + await _withdraw(client, job) + raise + + +async def _result( + client: httpx.AsyncClient, job: str, interval: float +) -> GeneratedImage: + while True: + cancellation.raise_if_cancelled() + state = (await _call(client.get(f"/sdcpp/v1/jobs/{job}"))).json() + if state["status"] == "completed": + return decode_image(state["result"]["images"][0]["b64_json"], None) + if state["status"] in _ENDED: + error = state.get("error") or {} + raise NonRetryableImageError( + f"The image model could not make this picture: " + f"{error.get('message') or state['status']}" + ) + await asyncio.sleep(interval) + + +async def _call(request: Awaitable[httpx.Response]) -> httpx.Response: + """The reply, or the edit's failure: a full queue, a job gone, a server down.""" + try: + reply = await request + except httpx.HTTPError as error: + raise NonRetryableImageError( + f"The image model could not be reached: {error}" + ) from error + if reply.status_code >= 400: + raise NonRetryableImageError(f"sd-server returned HTTP {reply.status_code}") + return reply + + +# The text runtime is still held while it is asked; a cancel ends the demand +# anyway, and Electron then stops sd-server. +_WITHDRAW_TIMEOUT = httpx.Timeout(2.0) + + +async def _withdraw(client: httpx.AsyncClient, job: str) -> None: + with contextlib.suppress(httpx.HTTPError): + await client.post(f"/sdcpp/v1/jobs/{job}/cancel", timeout=_WITHDRAW_TIMEOUT) diff --git a/surfsense_local/backend/modules/llm/resolution.py b/surfsense_local/backend/modules/llm/resolution.py index 14bc2670ee..4bd5a455b0 100644 --- a/surfsense_local/backend/modules/llm/resolution.py +++ b/surfsense_local/backend/modules/llm/resolution.py @@ -32,6 +32,7 @@ from modules.llm.providers.sdcpp import provider as sdcpp from modules.llm.providers.sdcpp.editor import LocalImageEditor from modules.llm.providers.sdcpp.generator import LocalImageGenerator +from modules.llm.providers.sdcpp.launch_size import launch_pixels from modules.llm.subscriptions.claude.account import CLAUDE_PLAN from modules.llm.subscriptions.claude.hosts import require_claude_hosts from modules.llm.subscriptions.claude.usable_claude import usable_claude @@ -119,6 +120,7 @@ def resolve_image_generation(session: Session) -> ResolvedImageGeneration: sdcpp.root_url(), image.served_file, text_runtime_given_up, + pixels=launch_pixels(image.args), ), ) connection = _connection(session, selected) @@ -135,6 +137,11 @@ def resolve_image_edit(session: Session) -> ResolvedImageEdit: selected = session.get(SelectedModel, ModelType.IMAGE_EDIT) if selected is None: raise ModelResolutionError("no image editing model selected") + return image_editor_for(session, selected) + + +def image_editor_for(session: Session, selected: SelectedModel) -> ResolvedImageEdit: + """The client that sends this chosen model a picture, whichever slot chose it.""" if selected.provider == sdcpp.PROVIDER: image = get_local_catalog().sdcpp.installed_image(selected.name) if image is None: diff --git a/surfsense_local/backend/modules/llm/selection.py b/surfsense_local/backend/modules/llm/selection.py index f930feda8e..c1121feec6 100644 --- a/surfsense_local/backend/modules/llm/selection.py +++ b/surfsense_local/backend/modules/llm/selection.py @@ -13,8 +13,10 @@ from modules.llm.connections.listing import connection_models from modules.llm.connections.router import allowed_connection from modules.llm.connections.serves import connection_serves +from modules.llm.connections.service import DiscoveredModel from modules.llm.model_type import ModelType from modules.llm.models import OnboardingCompletion, SelectedModel +from modules.llm.pictures_in.kept import keep_takes_pictures from modules.llm.profile import Fingerprint, from_name from modules.llm.providers import audiocpp, get_provider, llamacpp from modules.llm.providers.openai_compatible import OpenAICompatibleChatProvider @@ -43,6 +45,7 @@ async def choose_model( "model name must not be empty", ) + listed: DiscoveredModel | None = None if provider_name == llamacpp.PROVIDER: await _validate_local(model_type, model_name, connection_id) elif provider_name == sdcpp.PROVIDER: @@ -50,7 +53,7 @@ async def choose_model( elif provider_name == audiocpp.PROVIDER: _validate_local_audio(model_type, model_name, connection_id) elif provider_name == "openai_compatible": - await _validate_remote( + listed = await _validate_remote( session, model_type, model_name, connection_id, allow_unlisted ) else: @@ -60,6 +63,8 @@ async def choose_model( ) fingerprint = await _collect(session, provider_name, model_name, connection_id) + # An unlisted model is taken as words only: a picture it ignores is billed. + takes_pictures = listed is not None and ModelType.IMAGE_EDIT in listed.types selected = await transact( session, _store, @@ -68,6 +73,7 @@ async def choose_model( connection_id, model_name, fingerprint, + takes_pictures, ) if model_type is ModelType.SPEECH_TO_TEXT: # Recordings uploaded before any speech model waited for this one. @@ -116,6 +122,7 @@ def _store( connection_id: int | None, model_name: str, fingerprint: Fingerprint, + takes_pictures: bool = False, ) -> SelectedModel: selected = session.get(SelectedModel, model_type) if selected is None: @@ -135,6 +142,8 @@ def _store( selected.params_b = fingerprint.params_b selected.vendor = fingerprint.vendor selected.line = fingerprint.line + if model_type is ModelType.IMAGE_GEN: + keep_takes_pictures(selected, takes_pictures) session.flush() # updated_at is set by the database; load it here rather than lazily on the loop. session.refresh(selected) @@ -255,7 +264,8 @@ async def _validate_remote( model_name: str, connection_id: int | None, allow_unlisted: bool, -) -> None: +) -> DiscoveredModel | None: + """The model as its connection lists it; None when chosen unlisted.""" if connection_id is None: raise HTTPException( status.HTTP_422_UNPROCESSABLE_CONTENT, @@ -276,12 +286,12 @@ async def _validate_remote( status.HTTP_422_UNPROCESSABLE_CONTENT, "connection models could not be verified; confirm manual selection", ) from error - return + return None model = next((entry for entry in models if entry.name == model_name), None) if model is None: if allow_unlisted: - return + return None raise HTTPException( status.HTTP_422_UNPROCESSABLE_CONTENT, f"model is not listed by this connection: {model_name}", @@ -299,3 +309,4 @@ async def _validate_remote( status.HTTP_422_UNPROCESSABLE_CONTENT, f"model does not support {model_type.value}: {model_name}", ) + return model diff --git a/surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py b/surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py index 8807020899..2c44419b49 100644 --- a/surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py +++ b/surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py @@ -40,7 +40,9 @@ def _png() -> bytes: class Painter: - async def generate(self, _model: str, prompt: str) -> GeneratedImage: + async def generate( + self, _model: str, prompt: str, *, shape: str | None = None + ) -> GeneratedImage: self.prompt = prompt return GeneratedImage(_png(), "image/png") diff --git a/surfsense_local/backend/tests/integration/agent/test_make_image.py b/surfsense_local/backend/tests/integration/agent/test_make_image.py index d9ef05d7ab..92dd5b37a2 100644 --- a/surfsense_local/backend/tests/integration/agent/test_make_image.py +++ b/surfsense_local/backend/tests/integration/agent/test_make_image.py @@ -20,13 +20,10 @@ from modules.artifacts.models import Artifact, ArtifactFileRole from modules.llm.model_type import ModelType from modules.llm.models import SelectedModel +from modules.llm.pictures_in.model import PictureModel from modules.llm.providers.openai_compatible import NonRetryableImageError from modules.llm.providers.protocols import GeneratedImage -from modules.llm.resolution import ( - ModelResolutionError, - ResolvedImageEdit, - ResolvedImageGeneration, -) +from modules.llm.resolution import ResolvedImageGeneration from modules.source_scope.schemas import SourceScope from modules.workspaces.models import Workspace from shared.config import get_storage_settings @@ -54,13 +51,18 @@ class Painter: def __init__(self) -> None: self.prompts: list[str] = [] + self.shapes: list[str | None] = [] self.edits: list[tuple[str, bytes]] = [] + self.references: list[tuple[bytes, ...]] = [] self.fail_with: Exception | None = None self.hold: threading.Event | None = None self.gives = GeneratedImage(_png(), "image/png") - async def generate(self, _model: str, prompt: str) -> GeneratedImage: + async def generate( + self, _model: str, prompt: str, *, shape: str | None = None + ) -> GeneratedImage: self.prompts.append(prompt) + self.shapes.append(shape) if self.hold is not None: await asyncio.to_thread(self.hold.wait) if self.fail_with is not None: @@ -68,31 +70,39 @@ async def generate(self, _model: str, prompt: str) -> GeneratedImage: return self.gives async def edit( - self, _model: str, prompt: str, image: GeneratedImage + self, + _model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: str | None = None, ) -> GeneratedImage: - self.edits.append((prompt, image.content)) + if len(images) == 1: + self.edits.append((prompt, images[0].content)) + self.references.append(tuple(image.content for image in images)) + self.shapes.append(shape) return GeneratedImage(_png("navy"), "image/png") @pytest.fixture def painter(engine: Engine, monkeypatch: pytest.MonkeyPatch) -> Iterator[Painter]: - """An image and an edit model chosen, and no chat model: a drawing never needs one.""" + """An image model that takes pictures, and no chat model: a drawing never needs one.""" with create_session_factory(engine)() as session: - for model_type in (ModelType.IMAGE_GEN, ModelType.IMAGE_EDIT): - session.add( - SelectedModel(model_type=model_type, provider="sdcpp", name="m") - ) + session.add( + SelectedModel(model_type=ModelType.IMAGE_GEN, provider="sdcpp", name="m") + ) session.commit() fake = Painter() selection = type("Selection", (), {"provider": "fake", "name": "m"})() draws = lambda _session: ResolvedImageGeneration(selection, fake) # noqa: E731 - edits = lambda _session: ResolvedImageEdit(selection, fake) # noqa: E731 monkeypatch.setattr("worker.studio.job.resolve_image_generation", draws) monkeypatch.setattr( "modules.artifacts.drawn_images.model.resolve_image_generation", draws ) - monkeypatch.setattr("worker.studio.job.resolve_image_edit", edits) - monkeypatch.setattr("modules.artifacts.image_edits.model.resolve_image_edit", edits) + monkeypatch.setattr( + "modules.llm.pictures_in.model.picture_model", + lambda _session: PictureModel("image_model", selection, fake), + ) yield fake if fake.hold is not None: fake.hold.set() @@ -409,31 +419,145 @@ async def test_with_an_edit_model_the_picture_can_be_changed( assert "change it or tell the user" in text -async def test_with_no_edit_model_the_agent_is_not_sent_to_change_the_picture( - tools: ToolEndpoint, - studio_worker: None, - painter: Painter, - monkeypatch: pytest.MonkeyPatch, +@pytest.fixture +def words_only(monkeypatch: pytest.MonkeyPatch) -> None: + """The image model takes no picture in, and no edit model is chosen.""" + monkeypatch.setattr( + "modules.llm.pictures_in.model.picture_model", lambda _session: None + ) + + +async def test_with_a_model_that_takes_no_pictures_a_change_is_drawn_again( + tools: ToolEndpoint, studio_worker: None, painter: Painter, words_only: None ) -> None: - """A call to edit could only be refused, and a redraw would bill another picture.""" + """Every image model can change a picture, and the agent is told how, plainly: + never sent to Settings for a model the user need not have.""" + workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + root = _artifact_id(first) + assert f"call this tool again with artifact_id {root}" in first + assert "drawn again" in first + assert "Settings" not in first - def none_chosen(_session: object) -> None: - raise ModelResolutionError("no image editing model selected") + text, is_error = await tools.call( + workspace_id, "make_image", {"prompt": "Make it night", "artifact_id": root} + ) - monkeypatch.setattr( - "modules.artifacts.image_edits.model.resolve_image_edit", none_chosen + assert is_error is False, text + assert painter.edits == [] + assert painter.prompts[-1] == "A boat\n\nChange: Make it night" + assert "Drawn again from its prompt and the change" in text + assert "Settings" not in text + + +async def test_a_shape_reaches_the_image_model( + tools: ToolEndpoint, studio_worker: None, painter: Painter +) -> None: + """A comic page is portrait.""" + workspace_id = await tools.workspace() + + text, is_error = await tools.call( + workspace_id, + "make_image", + {"prompt": "Page one", "title": "Comic · page 1", "shape": "portrait"}, ) + + assert is_error is False, text + assert painter.shapes == ["portrait"] + assert "Drawn on this computer." in text + + +async def test_references_reach_the_model_that_takes_pictures_in_order( + tools: ToolEndpoint, studio_worker: None, painter: Painter, engine: Engine +) -> None: + """A comic's later pages keep page one's characters.""" workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page one", "title": "Comic · page 1"} + ) + sheet, _ = await tools.call( + workspace_id, "make_image", {"prompt": "The cat", "title": "Character sheet"} + ) + page_one, cat = _artifact_id(first), _artifact_id(sheet) text, is_error = await tools.call( - workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + workspace_id, + "make_image", + { + "prompt": "Page two", + "title": "Comic · page 2", + "shape": "portrait", + "references": [f"image-{page_one}", f"image-{cat}"], + }, ) assert is_error is False, text - assert "image-" in text - assert "artifact_id" not in text - assert "change it" not in text - assert "Settings > Image edit" in text + assert painter.references[-1] == ( + _stored(engine, page_one), + _stored(engine, cat), + ) + assert painter.shapes[-1] == "portrait" + assert painter.prompts == ["Page one", "The cat"] + + +async def test_with_a_model_that_takes_no_pictures_references_are_drawn_from_words( + tools: ToolEndpoint, studio_worker: None, painter: Painter, words_only: None +) -> None: + """Never refused: the page is drawn, and the agent told the faces may differ.""" + workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page one", "title": "Comic · page 1"} + ) + + text, is_error = await tools.call( + workspace_id, + "make_image", + { + "prompt": "Page two", + "title": "Comic · page 2", + "references": [f"image-{_artifact_id(first)}"], + }, + ) + + assert is_error is False, text + assert painter.references == [] + assert painter.prompts == ["Page one", "Page two"] + assert "Drawn from words alone" in text + + +async def test_references_are_refused_unbilled_when_they_cannot_be_kept( + tools: ToolEndpoint, studio_worker: None, painter: Painter +) -> None: + """Another chat's picture, an unknown one, five of them, or with an edit.""" + workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page one", "title": "Comic · page 1"} + ) + name = f"image-{_artifact_id(first)}" + asks = [ + ({"references": [name]}, tools.thread(workspace_id), "no picture"), + ({"references": ["image-999"]}, None, "no picture"), + ({"references": [name, "a", "b", "c", "d"]}, None, "at most 4"), + ( + {"references": [name], "artifact_id": _artifact_id(first)}, + None, + "references are for a new picture", + ), + ] + + for n, (extra, thread, said) in enumerate(asks): + text, is_error = await tools.call( + workspace_id, + "make_image", + {"prompt": "Page two", "title": f"Page {n}", **extra}, + **({"thread": thread} if thread is not None else {}), + ) + assert is_error is True, text + assert said in text + assert painter.prompts == ["Page one"] + assert painter.references == [] # Fails unless what it is handed is really a PNG, as IMAGES_DIR promises. @@ -529,28 +653,6 @@ async def test_a_copy_the_agent_overwrote_is_not_what_gets_placed( assert _pictures_in_word(engine, _artifact_id(text)) == [_png()] -async def test_a_picture_still_being_drawn_is_refused_saying_so( - tools: ToolEndpoint, - studio_worker: None, - painter: Painter, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """The agent tells the user, rather than reading it as a wrong name.""" - monkeypatch.setattr("modules.agent.tool_endpoint.make_image.WAIT_SECONDS", 0.2) - painter.hold = threading.Event() - workspace_id = await tools.workspace() - queued, _ = await tools.call( - workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} - ) - - text, is_error = await tools.call( - workspace_id, "render_document", _placing(f"image-{_artifact_id(queued)}") - ) - - assert is_error is True - assert "still being drawn" in text - - async def test_another_chats_picture_cannot_be_placed( tools: ToolEndpoint, studio_worker: None, painter: Painter ) -> None: diff --git a/surfsense_local/backend/tests/integration/artifacts/test_image_edit.py b/surfsense_local/backend/tests/integration/artifacts/test_image_edit.py index afaa3864c9..d600d1e0e8 100644 --- a/surfsense_local/backend/tests/integration/artifacts/test_image_edit.py +++ b/surfsense_local/backend/tests/integration/artifacts/test_image_edit.py @@ -1,5 +1,6 @@ """Editing a Studio image: the instruction and the open version's picture go to -the edit model, and what comes back is the document's next version.""" +the model that takes pictures, or, with none, the picture is drawn again from its +prompt and the change; either way it is the document's next version.""" import asyncio @@ -14,13 +15,10 @@ from modules.documents.models import Document, DocumentStatus from modules.llm.model_type import ModelType from modules.llm.models import SelectedModel +from modules.llm.pictures_in.model import PictureModel from modules.llm.providers.openai_compatible import NonRetryableImageError from modules.llm.providers.protocols import GeneratedImage -from modules.llm.resolution import ( - ResolvedImageEdit, - ResolvedImageGeneration, - resolve_image_edit, -) +from modules.llm.resolution import ResolvedImageGeneration from modules.workspaces.models import Workspace from shared.config import get_storage_settings from shared.queue import studio_queue @@ -37,16 +35,26 @@ class Painter: def __init__(self) -> None: self.edits: list[tuple[str, bytes]] = [] + self.drawn: list[str] = [] self.fail_with: Exception | None = None - async def generate(self, _model: str, _prompt: str) -> GeneratedImage: + async def generate( + self, _model: str, prompt: str, *, shape: str | None = None + ) -> GeneratedImage: + self.drawn.append(prompt) return GeneratedImage(DRAWN, "image/png") async def edit( - self, _model: str, prompt: str, image: GeneratedImage + self, + _model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: str | None = None, ) -> GeneratedImage: if self.fail_with is not None: raise self.fail_with + (image,) = images self.edits.append((prompt, image.content)) number = len(self.edits) return GeneratedImage(b"\x89PNG\r\n\x1a\nedit%d" % number, "image/png") @@ -54,30 +62,39 @@ async def edit( @pytest.fixture def painter(session: Session, monkeypatch: pytest.MonkeyPatch) -> Painter: - """A chat model, an image model and an edit model, all chosen.""" + """A chat model, and an image model that takes pictures.""" session.add( SelectedModel(model_type=ModelType.TEXT_GEN, provider="llamacpp", name="m") ) - for model_type in (ModelType.IMAGE_GEN, ModelType.IMAGE_EDIT): - session.add(SelectedModel(model_type=model_type, provider="sdcpp", name="m")) + session.add( + SelectedModel(model_type=ModelType.IMAGE_GEN, provider="sdcpp", name="m") + ) session.commit() fake = Painter() selection = type("Selection", (), {"provider": "fake", "name": "m"})() monkeypatch.setattr( "worker.studio.shared.generate.run_model", lambda *_a, **_k: WRITER ) + draws = lambda _session: ResolvedImageGeneration(selection, fake) # noqa: E731 + monkeypatch.setattr("worker.studio.job.resolve_image_generation", draws) monkeypatch.setattr( - "worker.studio.job.resolve_image_generation", - lambda _session: ResolvedImageGeneration(selection, fake), + "modules.artifacts.drawn_images.model.resolve_image_generation", draws ) - edit_model = lambda _session: ResolvedImageEdit(selection, fake) # noqa: E731 - monkeypatch.setattr("worker.studio.job.resolve_image_edit", edit_model) monkeypatch.setattr( - "modules.artifacts.image_edits.model.resolve_image_edit", edit_model + "modules.llm.pictures_in.model.picture_model", + lambda _session: PictureModel("image_model", selection, fake), ) return fake +@pytest.fixture +def words_only(monkeypatch: pytest.MonkeyPatch) -> None: + """The image model takes no pictures in, and no edit model is chosen.""" + monkeypatch.setattr( + "modules.llm.pictures_in.model.picture_model", lambda _session: None + ) + + async def _work_off(queue: Huey) -> None: """On a thread, as Studio's worker runs: the image clients start their own loop.""" @@ -272,7 +289,47 @@ async def test_a_failed_edit_is_not_retried_and_leaves_earlier_versions_alone( assert _primary(first) == DRAWN -async def test_editing_with_no_edit_model_says_so_as_a_code( +async def test_with_no_model_that_takes_pictures_an_edit_is_drawn_again( + client: AsyncClient, + session: Session, + workspace: Workspace, + logo_source: Document, + painter: Painter, + words_only: None, +) -> None: + """Edit works for every image model: one that cannot take the picture draws + it again from the prompt it was drawn with and the change, and says so.""" + first = await _drawn(session, workspace, logo_source) + + second = await _edited(client, session, first.id, "Make it night") + + assert painter.edits == [] + assert painter.drawn[-1] == "a lighthouse at dusk\n\nChange: Make it night" + assert second.document.content.startswith("Drawn again with your change.") + assert second.artifact_metadata["redrawn"] is True + assert second.artifact_metadata["version"]["number"] == 2 + + +async def test_changes_add_up_when_a_redrawn_picture_is_changed_again( + client: AsyncClient, + session: Session, + workspace: Workspace, + logo_source: Document, + painter: Painter, + words_only: None, +) -> None: + """Each redraw starts from the prompt its base was drawn with.""" + first = await _drawn(session, workspace, logo_source) + second = await _edited(client, session, first.id, "Make it night") + + await _edited(client, session, second.id, "Add stars") + + assert painter.drawn[-1] == ( + "a lighthouse at dusk\n\nChange: Make it night\n\nChange: Add stars" + ) + + +async def test_an_edit_keeps_the_prompt_a_later_redraw_starts_from( client: AsyncClient, session: Session, workspace: Workspace, @@ -280,20 +337,19 @@ async def test_editing_with_no_edit_model_says_so_as_a_code( painter: Painter, monkeypatch: pytest.MonkeyPatch, ) -> None: - """The box shows the interface's own sentence, as Refine does for a chat model.""" + """Edited by a model that takes pictures, then redrawn after a switch to one + that does not: the redraw still knows what was changed.""" first = await _drawn(session, workspace, logo_source) + second = await _edited(client, session, first.id, "Make it night") monkeypatch.setattr( - "modules.artifacts.image_edits.model.resolve_image_edit", resolve_image_edit + "modules.llm.pictures_in.model.picture_model", lambda _session: None ) - session.delete(session.get(SelectedModel, ModelType.IMAGE_EDIT)) - response = await _edit(client, session, first.id, "Make it night") + await _edited(client, session, second.id, "Add stars") - assert response.status_code == 409 - assert response.json()["detail"] == { - "message": "Needs an image editing model", - "code": "needs_image_edit", - } + assert painter.drawn[-1] == ( + "a lighthouse at dusk\n\nChange: Make it night\n\nChange: Add stars" + ) async def test_only_a_ready_image_or_infographic_can_be_edited( diff --git a/surfsense_local/backend/tests/integration/llm/test_picture_model.py b/surfsense_local/backend/tests/integration/llm/test_picture_model.py new file mode 100644 index 0000000000..1b815ffe1b --- /dev/null +++ b/surfsense_local/backend/tests/integration/llm/test_picture_model.py @@ -0,0 +1,170 @@ +"""Which model takes a picture in: the edit model if one is chosen, else the image +model when it can, else none, and then an edit is drawn again from words.""" + +from pathlib import Path + +import pytest +from httpx import AsyncClient +from sqlalchemy import Engine + +from modules.llm.pictures_in.model import picture_model +from modules.llm.providers import sdcpp +from shared.db import create_session_factory +from tests.integration.llm import conftest +from tests.integration.llm.test_sdcpp_slots import KLEIN, SD15, put_on_disk + +pytestmark = [pytest.mark.integration, pytest.mark.asyncio] + +TAKES_PICTURES = "openai/gpt-image-2.6-flare" +WORDS_ONLY = "recraft/recraft-v4.1-flash" + + +@pytest.fixture +def image_models(monkeypatch: pytest.MonkeyPatch) -> None: + """As OpenRouter lists them: what each model takes, and what it returns.""" + monkeypatch.setattr( + conftest, + "REMOTE_MODELS", + [ + *conftest.REMOTE_MODELS, + { + "id": TAKES_PICTURES, + "architecture": { + "input_modalities": ["text", "image"], + "output_modalities": ["image"], + }, + }, + { + "id": WORDS_ONLY, + "architecture": { + "input_modalities": ["text"], + "output_modalities": ["image"], + }, + }, + ], + ) + + +async def _connect(client: AsyncClient, base_url: str) -> int: + reply = await client.post( + "/llm/connections", + json={ + "label": "Gateway", + "provider": "openai_compatible", + "base_url": base_url, + "api_key": "secret", + }, + ) + assert reply.status_code == 201, reply.text + return reply.json()["id"] + + +async def _choose( + client: AsyncClient, slot: str, name: str, connection_id: int | None +) -> None: + provider = "openai_compatible" if connection_id is not None else sdcpp.PROVIDER + reply = await client.put( + f"/llm/selection/{slot}", + json={"provider": provider, "connection_id": connection_id, "name": name}, + ) + assert reply.status_code == 200, reply.text + + +def _picture_model(engine: Engine): + with create_session_factory(engine)() as session: + return picture_model(session) + + +async def test_an_image_model_that_takes_pictures_edits_with_no_edit_model( + client: AsyncClient, openai_server: str, image_models: None, engine: Engine +) -> None: + """Most remote image models take a picture in; none needs a second choice.""" + connection = await _connect(client, openai_server) + await _choose(client, "image_gen", TAKES_PICTURES, connection) + + chosen = _picture_model(engine) + + assert chosen is not None + assert chosen.source == "image_model" + assert chosen.selection.name == TAKES_PICTURES + + +async def test_an_image_model_that_takes_only_words_has_no_picture_model( + client: AsyncClient, openai_server: str, image_models: None, engine: Engine +) -> None: + """Its edits are drawn again from words instead.""" + connection = await _connect(client, openai_server) + await _choose(client, "image_gen", WORDS_ONLY, connection) + + assert _picture_model(engine) is None + + +async def test_a_chosen_edit_model_is_used_before_the_image_model( + client: AsyncClient, openai_server: str, image_models: None, engine: Engine +) -> None: + """The override, for the user who wants another model for edits.""" + connection = await _connect(client, openai_server) + await _choose(client, "image_gen", TAKES_PICTURES, connection) + await _choose(client, "image_edit", TAKES_PICTURES, connection) + await _choose(client, "image_gen", WORDS_ONLY, connection) + + chosen = _picture_model(engine) + + assert chosen is not None + assert chosen.source == "edit_model" + assert chosen.selection.name == TAKES_PICTURES + + +async def test_another_image_model_does_not_inherit_taking_pictures( + client: AsyncClient, openai_server: str, image_models: None, engine: Engine +) -> None: + """What a model takes is its own, not the slot's.""" + connection = await _connect(client, openai_server) + await _choose(client, "image_gen", TAKES_PICTURES, connection) + await _choose(client, "image_gen", WORDS_ONLY, connection) + + assert _picture_model(engine) is None + + +async def test_a_local_image_model_that_edits_takes_pictures( + client: AsyncClient, images_dir: Path, engine: Engine +) -> None: + """FLUX.2 klein's entry names editing; the same files draw and edit.""" + put_on_disk(images_dir, "flux2-klein-4b") + await _choose(client, "image_gen", KLEIN, None) + + chosen = _picture_model(engine) + + assert chosen is not None + assert chosen.source == "image_model" + assert chosen.selection.name == KLEIN + + +async def test_a_local_image_model_that_only_draws_has_no_picture_model( + client: AsyncClient, images_dir: Path, engine: Engine +) -> None: + """Its entry names drawing only.""" + put_on_disk(images_dir, "stable-diffusion-1.5") + await _choose(client, "image_gen", SD15, None) + + assert _picture_model(engine) is None + + +async def test_the_app_is_told_which_model_takes_a_picture( + client: AsyncClient, openai_server: str, image_models: None +) -> None: + """Studio's Edit says whether a change is an edit or a redraw.""" + connection = await _connect(client, openai_server) + + nothing = (await client.get("/llm/pictures-in")).json() + await _choose(client, "image_gen", WORDS_ONLY, connection) + words = (await client.get("/llm/pictures-in")).json() + await _choose(client, "image_gen", TAKES_PICTURES, connection) + pictures = (await client.get("/llm/pictures-in")).json() + await _choose(client, "image_edit", TAKES_PICTURES, connection) + override = (await client.get("/llm/pictures-in")).json() + + assert nothing == {"source": None} + assert words == {"source": None} + assert pictures == {"source": "image_model"} + assert override == {"source": "edit_model"} diff --git a/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_editor.py b/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_editor.py index 6afcddc6dd..8a1317d097 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_editor.py +++ b/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_editor.py @@ -95,7 +95,7 @@ async def test_it_sends_the_picture_as_a_reference_at_its_own_size() -> None: """The launch's size would crop or stretch a picture of another shape.""" server = SdServer(_job("queued"), _job("generating"), _done()) - image = await _editor(server).edit("klein", "make it night", BASE) + image = await _editor(server).edit("klein", "make it night", (BASE,)) assert image == GeneratedImage(EDITED, "image/png") [body] = server.posted @@ -131,7 +131,7 @@ def watch(request: httpx.Request) -> httpx.Response: transport=httpx.MockTransport(watch), ) - await edit.edit("klein", "make it night", BASE) + await edit.edit("klein", "make it night", (BASE,)) assert seen and all(seen) assert runtime.held is False @@ -144,7 +144,7 @@ async def test_a_failed_job_fails_the_edit_with_its_reason() -> None: ) with pytest.raises(NonRetryableImageError, match="out of memory"): - await _editor(server).edit("klein", "make it night", BASE) + await _editor(server).edit("klein", "make it night", (BASE,)) @pytest.mark.parametrize("status", [404, 410]) @@ -153,7 +153,7 @@ async def test_a_job_the_server_no_longer_has_fails_the_edit(status: int) -> Non server = SdServer(httpx.Response(status, json={"error": "gone"})) with pytest.raises(NonRetryableImageError, match=str(status)): - await _editor(server).edit("klein", "make it night", BASE) + await _editor(server).edit("klein", "make it night", (BASE,)) async def test_a_full_queue_fails_the_edit() -> None: @@ -161,7 +161,7 @@ async def test_a_full_queue_fails_the_edit() -> None: server = SdServer(_done(), submit=httpx.Response(429, json={"error": "full"})) with pytest.raises(NonRetryableImageError, match="429"): - await _editor(server).edit("klein", "make it night", BASE) + await _editor(server).edit("klein", "make it night", (BASE,)) async def test_a_cancel_withdraws_the_queued_job() -> None: @@ -175,7 +175,7 @@ def cancelled() -> None: raise CancelledError with cancellation.watching(cancelled), pytest.raises(CancelledError): - await _editor(server).edit("klein", "make it night", BASE) + await _editor(server).edit("klein", "make it night", (BASE,)) assert server.asked[-1] == "POST /sdcpp/v1/jobs/job_1/cancel" @@ -203,8 +203,24 @@ def cancelled() -> None: transport=httpx.MockTransport(watch), ) with cancellation.watching(cancelled), pytest.raises(CancelledError): - await edit.edit("klein", "make it night", BASE) + await edit.edit("klein", "make it night", (BASE,)) withdraw = seen[-1] assert withdraw.url.path == "/sdcpp/v1/jobs/job_1/cancel" assert withdraw.extensions["timeout"]["read"] <= 2 + + +async def test_every_reference_goes_in_at_the_shape_asked_for() -> None: + """A comic page keeps its characters from page one, drawn portrait at about + the reference's pixel count, each side a multiple of 64.""" + server = SdServer(_done()) + page_one = GeneratedImage(_png(1024, 1024), "image/png") + + await _editor(server).edit("klein", "page two", (page_one, BASE), shape="portrait") + + [body] = server.posted + assert body["ref_images"] == [ + "data:image/png;base64," + base64.b64encode(page_one.content).decode(), + "data:image/png;base64," + base64.b64encode(BASE.content).decode(), + ] + assert (body["width"], body["height"]) == (832, 1280) diff --git a/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_generator.py b/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_generator.py new file mode 100644 index 0000000000..8f945d8876 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/sdcpp/test_generator.py @@ -0,0 +1,70 @@ +"""A local drawing: today's route as it was, and the native job route when a +shape is asked, since only that one takes a width and height per request.""" + +import httpx +import pytest + +from modules.llm.providers.protocols import GeneratedImage +from modules.llm.providers.sdcpp.generator import LocalImageGenerator +from tests.unit.llm.providers.sdcpp.test_editor import ( + EDITED, + SERVED, + Runtime, + SdServer, + _done, +) + +pytestmark = [pytest.mark.unit, pytest.mark.asyncio] + + +class Inner: + """Today's client, reaching sd-server's OpenAI route.""" + + def __init__(self) -> None: + self.asked: list[str] = [] + + async def generate(self, model: str, prompt: str) -> GeneratedImage: + self.asked.append(prompt) + return GeneratedImage(b"square", "image/png") + + +def _generator(server: SdServer, inner: Inner) -> LocalImageGenerator: + return LocalImageGenerator( + inner, + "http://127.0.0.1:1", + SERVED, + Runtime().given_up, + pixels=1024 * 1024, + interval=0, + transport=httpx.MockTransport(server), + ) + + +async def test_a_drawing_with_no_shape_keeps_its_route() -> None: + """Nothing changes for a picture that asks for no shape.""" + server, inner = SdServer(_done()), Inner() + + image = await _generator(server, inner).generate("klein", "a lighthouse") + + assert image.content == b"square" + assert inner.asked == ["a lighthouse"] + assert server.posted == [] + + +async def test_a_shape_is_drawn_through_the_native_route_at_its_size() -> None: + """At the launch's pixel count, in the shape's proportions.""" + server, inner = SdServer(_done()), Inner() + + image = await _generator(server, inner).generate( + "klein", "a comic page", shape="portrait" + ) + + assert image == GeneratedImage(EDITED, "image/png") + assert inner.asked == [] + [body] = server.posted + assert body == { + "prompt": "a comic page", + "width": 832, + "height": 1280, + "output_format": "png", + } diff --git a/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py b/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py index 52210bd08f..df1fe1d6bc 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py +++ b/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py @@ -1,13 +1,15 @@ """A remote edit model: the picture and the instruction go to `/images/edits`.""" import base64 +import json import httpx import pytest from modules.llm.providers.openai_compatible.image_edit import ( OpenAICompatibleImageEditor, - _asks_for_picture, + _refused, + _routes, ) from modules.llm.providers.openai_compatible.image_response import ( NonRetryableImageError, @@ -41,7 +43,7 @@ async def test_it_posts_the_picture_and_the_instruction_as_a_form() -> None: 200, json={"data": [{"b64_json": base64.b64encode(EDITED).decode()}]} ) - image = await _editor(reply, asked).edit("gpt-image-1", "make it night", BASE) + image = await _editor(reply, asked).edit("gpt-image-1", "make it night", (BASE,)) assert image == GeneratedImage(EDITED, "image/png") [request] = asked @@ -65,20 +67,22 @@ async def test_a_refused_edit_is_not_retried() -> None: asked: list[httpx.Request] = [] with pytest.raises(NonRetryableImageError, match="500"): - await _editor(httpx.Response(500, json={}), asked).edit("m", "x", BASE) + await _editor(httpx.Response(500, json={}), asked).edit("m", "x", (BASE,)) assert len(asked) == 1 async def test_an_answer_with_no_picture_fails_the_edit() -> None: """Read as a generation's answer is.""" with pytest.raises(NonRetryableImageError, match="no data list"): - await _editor(httpx.Response(200, json={"error": "?"}), []).edit("m", "x", BASE) + await _editor(httpx.Response(200, json={"error": "?"}), []).edit( + "m", "x", (BASE,) + ) async def test_a_model_that_refuses_the_setting_is_asked_again_without_it() -> None: """GPT image models answer with the picture and refuse the setting; a 400 comes before any work, so asking again bills nothing twice.""" - _asks_for_picture.clear() + _refused.clear() asked: list[bytes] = [] done = httpx.Response( 200, json={"data": [{"b64_json": base64.b64encode(EDITED).decode()}]} @@ -92,7 +96,85 @@ def answer(request: httpx.Request) -> httpx.Response: editor = OpenAICompatibleImageEditor( "https://example.test/v1", transport=httpx.MockTransport(answer) ) - image = await editor.edit("gpt-image-1", "make it night", BASE) + image = await editor.edit("gpt-image-1", "make it night", (BASE,)) assert image.content == EDITED assert [b"response_format" in form for form in asked] == [True, False] + + +PAGE_ONE = GeneratedImage(b"\x89PNG\r\n\x1a\npage-one", "image/png") +DONE = {"data": [{"b64_json": base64.b64encode(EDITED).decode()}]} + + +async def test_several_pictures_go_as_one_part_each() -> None: + """A comic page keeps its characters from the pages before it.""" + asked: list[httpx.Request] = [] + editor = _editor(httpx.Response(200, json=DONE), asked) + + await editor.edit("gpt-image-1", "page two", (PAGE_ONE, BASE)) + + form = asked[0].read() + assert form.count(b'name="image[]"') == 2 + assert form.index(PAGE_ONE.content) < form.index(BASE.content) + + +async def test_a_shape_is_asked_for_as_a_size() -> None: + """As OpenAI's edit route takes it.""" + asked: list[httpx.Request] = [] + editor = _editor(httpx.Response(200, json=DONE), asked) + + await editor.edit("gpt-image-1", "page two", (PAGE_ONE,), shape="portrait") + + assert b'name="size"\r\n\r\n1024x1536' in asked[0].read() + + +async def test_a_server_with_no_edit_route_is_sent_references_instead() -> None: + """OpenRouter answers 404 on /images/edits, before any work, and takes the + pictures as input_references on /images; the route is kept per server.""" + _routes.clear() + asked: list[httpx.Request] = [] + + def answer(request: httpx.Request) -> httpx.Response: + asked.append(request) + if request.url.path.endswith("/images/edits"): + return httpx.Response(404, json={}) + return httpx.Response(200, json=DONE) + + editor = OpenAICompatibleImageEditor( + "https://gateway.test/v1", "sk-test", transport=httpx.MockTransport(answer) + ) + image = await editor.edit( + "openai/gpt-image-2.6", "page two", (PAGE_ONE,), shape="portrait" + ) + await editor.edit("openai/gpt-image-2.6", "page three", (PAGE_ONE,)) + + assert image.content == EDITED + assert [r.url.path for r in asked] == [ + "/v1/images/edits", + "/v1/images", + "/v1/images", + ] + sent = json.loads(asked[1].read()) + assert sent["model"] == "openai/gpt-image-2.6" + assert sent["prompt"] == "page two" + assert sent["aspect_ratio"] == "2:3" + assert sent["input_references"] == [ + { + "type": "image_url", + "image_url": { + "url": "data:image/png;base64," + + base64.b64encode(PAGE_ONE.content).decode() + }, + } + ] + assert "aspect_ratio" not in json.loads(asked[2].read()) + + +async def test_a_server_error_on_the_edit_route_is_not_tried_elsewhere() -> None: + """A 5xx may follow billed work: no second route, no second picture.""" + _routes.clear() + asked: list[httpx.Request] = [] + + with pytest.raises(NonRetryableImageError, match="502"): + await _editor(httpx.Response(502, json={}), asked).edit("m", "x", (BASE,)) + assert len(asked) == 1 diff --git a/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py b/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py index 070eba4744..87d7f71343 100644 --- a/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py +++ b/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py @@ -131,6 +131,49 @@ def test_an_endpoints_own_word_for_speech_is_the_audio_slot() -> None: assert models["openai/whisper-1"].capability_known is True +def test_an_image_model_that_declares_pictures_in_can_edit() -> None: + """OpenRouter lists what a model takes as well as what it returns; a picture in + and a picture out is what an edit, or a comic page's reference, needs.""" + models = { + model.name: model + for model in parse_models( + { + "data": [ + { + "id": "openai/gpt-image-2.6-flare", + "architecture": { + "input_modalities": ["text", "image"], + "output_modalities": ["image"], + }, + }, + { + "id": "recraft/recraft-v4.1-flash", + "architecture": { + "input_modalities": ["text"], + "output_modalities": ["image"], + }, + }, + { + "id": "openai/gpt-4o", + "architecture": { + "input_modalities": ["text", "image"], + "output_modalities": ["text"], + }, + }, + ] + } + ) + } + + assert models["openai/gpt-image-2.6-flare"].types == ( + ModelType.IMAGE_GEN, + ModelType.IMAGE_EDIT, + ) + assert models["recraft/recraft-v4.1-flash"].types == (ModelType.IMAGE_GEN,) + # Reading pictures is not editing them: a chat model stays a chat model. + assert models["openai/gpt-4o"].types == (ModelType.TEXT_GEN,) + + def test_delta_reads_openai_sse_and_ignores_done() -> None: """Chat streaming retains the existing OpenAI delta behavior.""" assert _delta('data: {"choices":[{"delta":{"content":"hi"}}]}') == Delta("hi") @@ -247,3 +290,45 @@ async def download(url: str) -> GeneratedImage: assert fetched == [ "https://oaidalleapiprodscus.blob.core.windows.net/private/x.png" ] + + +async def test_a_shape_is_asked_for_as_a_size_and_dropped_where_refused( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A comic page is portrait. A server that refuses the size draws its own, + and a 400 is a refusal before any work, so asking again bills nothing.""" + _route_cache.clear() + _asks_for_picture.clear() + provider = _provider(6) + sizes: list[str | None] = [] + + async def post(_route, _model, _prompt, _picture, size=None): + sizes.append(size) + return (400, {}) if size else (200, PAYLOAD) + + monkeypatch.setattr(provider, "_post", post) + await provider.generate("flux", "page one", shape="portrait") + await provider.generate("flux", "page two", shape="portrait") + + # Without the size, after the picture setting was tried without first. + assert sizes == ["1024x1536", "1024x1536", None, None] + + +async def test_a_shape_reaches_a_server_that_takes_it( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Only asked for when a picture names a shape.""" + _route_cache.clear() + _asks_for_picture.clear() + provider = _provider(7) + sizes: list[str | None] = [] + + async def post(_route, _model, _prompt, _picture, size=None): + sizes.append(size) + return 200, PAYLOAD + + monkeypatch.setattr(provider, "_post", post) + await provider.generate("gpt-image-1", "a strip", shape="landscape") + await provider.generate("gpt-image-1", "a picture") + + assert sizes == ["1536x1024", None] diff --git a/surfsense_local/backend/worker/studio/job.py b/surfsense_local/backend/worker/studio/job.py index 149b25523f..43bf78b4cb 100644 --- a/surfsense_local/backend/worker/studio/job.py +++ b/surfsense_local/backend/worker/studio/job.py @@ -38,6 +38,7 @@ from modules.jobs.runs.record import job_run_of from modules.llm.model_route.client import resolve_routed_generation from modules.llm.model_type import ModelType +from modules.llm.pictures_in import model as pictures_in from modules.llm.providers.audiocpp.memory import NotEnoughMemoryError from modules.llm.providers.context_overflow import ContextOverflowError from modules.llm.providers.openai_compatible import NonRetryableImageError @@ -47,7 +48,6 @@ ModelResolutionError, ResolvedGeneration, ResolvedImageGeneration, - resolve_image_edit, resolve_image_generation, resolve_text_to_speech, ) @@ -61,8 +61,10 @@ from worker.studio.job_run import pipeline as job_run from worker.studio.made_file import pipeline as made_file from worker.studio.media.visual.drawn import pipeline as drawn_image +from worker.studio.media.visual.drawn.references import reference_pictures from worker.studio.media.visual.edit import pipeline as image_edit from worker.studio.media.visual.edit.base import base_image +from worker.studio.media.visual.redraw import pipeline as image_redraw from worker.studio.office.document import figure_shelf from worker.studio.office.document.refine import refine from worker.studio.office.docx import docx @@ -363,32 +365,53 @@ def _refine( def _edit_image( session: Session, artifact: Artifact, document: Document, editing: ImageEdit ) -> Built: - """One call to the edit model, from the picture of the version it names.""" + """One call: the picture to the model that takes pictures, or, with none, + the image model drawing it again from the words it was drawn with.""" try: - editor = resolve_image_edit(session) + editor = pictures_in.picture_model(session) except ModelResolutionError as error: raise NoModelSelectedError(str(error)) from error + painter = _choose_model(session, ModelType.IMAGE_GEN) if editor is None else None base = base_image(session, editing) session.commit() raise_if_cancelled(session, document) - # The local editor polls; each poll looks for a cancel. + # The local clients poll; each poll looks for a cancel. with cancellation.watching(lambda: _check_cancelled(session, document)): - return image_edit.render(editor, base, editing.instruction) + if editor is not None: + return image_edit.render(editor, base, editing.instruction) + assert painter is not None + return image_redraw.render(painter, base, editing.instruction) def _draw_image( session: Session, artifact: Artifact, document: Document, drawing: ImageDrawing ) -> Built: - """One call to the image model, with the prompt the agent wrote.""" + """One call to the image model, with the prompt the agent wrote; with + reference pictures, to the model that takes them, when one does.""" painter = _choose_model(session, ModelType.IMAGE_GEN) + keeper = None + if drawing.references: + try: + keeper = pictures_in.picture_model(session) + except ModelResolutionError as error: + raise NoModelSelectedError(str(error)) from error + references = reference_pictures(session, drawing.references) title, format = document.title, artifact.format session.commit() raise_if_cancelled(session, document) - # The local generator polls; each poll looks for a cancel. + # The local clients poll; each poll looks for a cancel. with cancellation.watching(lambda: _check_cancelled(session, document)): - return drawn_image.render(painter, drawing.prompt, title, format) + return drawn_image.render( + painter, + drawing.prompt, + title, + format, + shape=drawing.shape, + references=references, + keeper=keeper, + ) def _keep_spec(artifact: Artifact, spec: dict) -> None: diff --git a/surfsense_local/backend/worker/studio/media/visual/drawn/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/drawn/pipeline.py index 45b2beee07..e4a4c0152e 100644 --- a/surfsense_local/backend/worker/studio/media/visual/drawn/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/drawn/pipeline.py @@ -1,26 +1,53 @@ """Drawn: the prompt the agent wrote goes to the image model as written. No chat model rewrites it: the agent has already read what the picture draws on. +With reference pictures it goes to the model that takes pictures, so a comic's +pages keep their characters; with none, it is drawn from the words alone. """ import asyncio +from modules.llm.pictures_in.model import PictureModel +from modules.llm.providers.picture_shape import Shape +from modules.llm.providers.protocols import GeneratedImage from modules.llm.resolution import ResolvedImageGeneration from worker.studio.media.visual import EXTENSIONS from worker.studio.media.visual.infographic.style import in_infographic_style +from worker.studio.media.visual.redraw.drawing_prompt import kept_drawing_prompt from worker.studio.shared.artifact import Built from worker.studio.shared.text import slug +# Kept on a version whose references no chosen model could take. +DRAWN_FROM_WORDS = "drawn_from_words" + def render( - painter: ResolvedImageGeneration, prompt: str, title: str, format: str + painter: ResolvedImageGeneration, + prompt: str, + title: str, + format: str, + *, + shape: Shape | None = None, + references: tuple[GeneratedImage, ...] = (), + keeper: PictureModel | None = None, ) -> Built: painted = in_infographic_style(prompt) if format == "infographic" else prompt - image = asyncio.run(painter.generator.generate(painter.selection.name, painted)) + metadata = kept_drawing_prompt(painted) + if references and keeper is not None: + image = asyncio.run( + keeper.editor.edit(keeper.selection.name, painted, references, shape=shape) + ) + else: + image = asyncio.run( + painter.generator.generate(painter.selection.name, painted, shape=shape) + ) + if references: + metadata[DRAWN_FROM_WORDS] = True return Built( title=title, markdown=f"# {title}\n\n{prompt}", primary=image.content, primary_mime=image.media_type, primary_filename=f"{slug(title, format)}.{EXTENSIONS[image.media_type]}", + metadata=metadata, ) diff --git a/surfsense_local/backend/worker/studio/media/visual/drawn/references.py b/surfsense_local/backend/worker/studio/media/visual/drawn/references.py new file mode 100644 index 0000000000..fe4bd4d9e9 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/drawn/references.py @@ -0,0 +1,32 @@ +"""The pictures a drawing keeps the characters and style of, read as stored.""" + +from sqlalchemy.orm import Session + +from modules.artifacts.models import Artifact, ArtifactFileRole +from modules.llm.providers.protocols import GeneratedImage +from shared.config import get_storage_settings + + +class ReferenceGoneError(RuntimeError): + """A picture this drawing keeps was deleted before it was drawn.""" + + +def reference_pictures( + session: Session, artifact_ids: tuple[int, ...] +) -> tuple[GeneratedImage, ...]: + return tuple(_picture(session, artifact_id) for artifact_id in artifact_ids) + + +def _picture(session: Session, artifact_id: int) -> GeneratedImage: + artifact = session.get(Artifact, artifact_id) + file = ( + next((f for f in artifact.files if f.role is ArtifactFileRole.PRIMARY), None) + if artifact is not None + else None + ) + path = get_storage_settings().data_dir / file.storage_key if file else None + if file is None or path is None or not path.is_file(): + raise ReferenceGoneError( + f"The picture image-{artifact_id} this one keeps is gone." + ) + return GeneratedImage(path.read_bytes(), file.mime_type) diff --git a/surfsense_local/backend/worker/studio/media/visual/edit/base.py b/surfsense_local/backend/worker/studio/media/visual/edit/base.py index b4179d5bd3..51f2f9d9af 100644 --- a/surfsense_local/backend/worker/studio/media/visual/edit/base.py +++ b/surfsense_local/backend/worker/studio/media/visual/edit/base.py @@ -9,6 +9,7 @@ from modules.artifacts.models import Artifact, ArtifactFileRole from modules.llm.providers.protocols import GeneratedImage from shared.config import get_storage_settings +from worker.studio.media.visual.redraw.drawing_prompt import drawing_prompt_of class BaseImageGoneError(RuntimeError): @@ -22,6 +23,8 @@ class BaseImage: picture: GeneratedImage # The base's file name without its extension, which the edit's format decides. stem: str + # The words it was drawn from, which a redraw starts from. + drawing_prompt: str def base_image(session: Session, edit: ImageEdit) -> BaseImage: @@ -39,4 +42,5 @@ def base_image(session: Session, edit: ImageEdit) -> BaseImage: body=base.document.content or "", picture=GeneratedImage(path.read_bytes(), file.mime_type), stem=PurePath(file.original_filename).stem, + drawing_prompt=drawing_prompt_of(base), ) diff --git a/surfsense_local/backend/worker/studio/media/visual/edit/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/edit/pipeline.py index 2cbc85c735..a086aa4259 100644 --- a/surfsense_local/backend/worker/studio/media/visual/edit/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/edit/pipeline.py @@ -6,15 +6,19 @@ import asyncio -from modules.llm.resolution import ResolvedImageEdit +from modules.llm.pictures_in.model import PictureModel from worker.studio.media.visual import EXTENSIONS from worker.studio.media.visual.edit.base import BaseImage +from worker.studio.media.visual.redraw.drawing_prompt import ( + changed, + kept_drawing_prompt, +) from worker.studio.shared.artifact import Built -def render(editor: ResolvedImageEdit, base: BaseImage, instruction: str) -> Built: +def render(editor: PictureModel, base: BaseImage, instruction: str) -> Built: image = asyncio.run( - editor.editor.edit(editor.selection.name, instruction, base.picture) + editor.editor.edit(editor.selection.name, instruction, (base.picture,)) ) # The base's body still says what the picture shows; the line says how it moved. body = ( @@ -28,4 +32,6 @@ def render(editor: ResolvedImageEdit, base: BaseImage, instruction: str) -> Buil primary=image.content, primary_mime=image.media_type, primary_filename=f"{base.stem}.{EXTENSIONS[image.media_type]}", + # So a later redraw, on a model that takes no pictures, keeps this change. + metadata=kept_drawing_prompt(changed(base.drawing_prompt, instruction)), ) diff --git a/surfsense_local/backend/worker/studio/media/visual/image/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/image/pipeline.py index 12edef3a7c..ec7b07e961 100644 --- a/surfsense_local/backend/worker/studio/media/visual/image/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/image/pipeline.py @@ -7,6 +7,7 @@ from modules.llm.profile import Tier from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration from worker.studio.media.visual import EXTENSIONS +from worker.studio.media.visual.redraw.drawing_prompt import kept_drawing_prompt from worker.studio.shared import generate from worker.studio.shared.artifact import Built, Source, fallback_title from worker.studio.shared.text import as_text, parse_json, slug @@ -35,6 +36,7 @@ def render( primary=image.content, primary_mime=image.media_type, primary_filename=f"{slug(title, 'image')}.{EXTENSIONS[image.media_type]}", + metadata=kept_drawing_prompt(image_prompt), ) diff --git a/surfsense_local/backend/worker/studio/media/visual/infographic/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/infographic/pipeline.py index 64a965dcac..fc878e8021 100644 --- a/surfsense_local/backend/worker/studio/media/visual/infographic/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/infographic/pipeline.py @@ -11,6 +11,7 @@ from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration from worker.studio.media.visual import EXTENSIONS from worker.studio.media.visual.infographic.style import in_infographic_style +from worker.studio.media.visual.redraw.drawing_prompt import kept_drawing_prompt from worker.studio.shared import generate from worker.studio.shared.artifact import Built, Source, fallback_title from worker.studio.shared.text import as_list, as_text, parse_json, slug @@ -30,9 +31,8 @@ def render( generate.run_model(writer, prompt(writer.tier, user_prompt), sources), fallback_title(user_prompt, sources, "Infographic"), ) - image = asyncio.run( - painter.generator.generate(painter.selection.name, _image_prompt(brief)) - ) + painted = _image_prompt(brief) + image = asyncio.run(painter.generator.generate(painter.selection.name, painted)) title = brief[0] return Built( title=title, @@ -40,6 +40,7 @@ def render( primary=image.content, primary_mime=image.media_type, primary_filename=f"{slug(title, 'infographic')}.{EXTENSIONS[image.media_type]}", + metadata=kept_drawing_prompt(painted), ) diff --git a/surfsense_local/backend/worker/studio/media/visual/redraw/__init__.py b/surfsense_local/backend/worker/studio/media/visual/redraw/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py b/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py new file mode 100644 index 0000000000..124c7bd674 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py @@ -0,0 +1,43 @@ +"""The words a picture was drawn from, which a redraw starts from. + +Kept as `drawing_prompt` from the first version that keeps it; a picture drawn +before has only its recipe's prompt, or the body Studio wrote under its title. +""" + +from typing import Any + +from modules.artifacts.drawn_images.recipe import image_drawing_of +from modules.artifacts.models import Artifact +from worker.studio.media.visual.infographic.style import in_infographic_style + +KEY = "drawing_prompt" + + +def kept_drawing_prompt(prompt: str) -> dict[str, Any]: + return {KEY: prompt} + + +def changed(prompt: str, instruction: str) -> str: + """The prompt with one more change, so changes add up from version to version.""" + return f"{prompt}\n\nChange: {instruction}" + + +def drawing_prompt_of(artifact: Artifact) -> str: + meta = artifact.artifact_metadata or {} + kept = meta.get(KEY) + if isinstance(kept, str) and kept.strip(): + return kept + drawing = image_drawing_of(meta) + if drawing is not None: + # Drawn as the agent wrote it, in Studio's style for an infographic. + if artifact.format == "infographic": + return in_infographic_style(drawing.prompt) + return drawing.prompt + return _below_title(artifact.document.content or "", artifact.document.title) + + +def _below_title(body: str, title: str) -> str: + """Studio's image body is `# ` and then the prompt it drew.""" + heading = f"# {title}" + text = body[len(heading) :] if body.startswith(heading) else body + return text.strip() or title diff --git a/surfsense_local/backend/worker/studio/media/visual/redraw/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/redraw/pipeline.py new file mode 100644 index 0000000000..44874365d4 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/redraw/pipeline.py @@ -0,0 +1,51 @@ +"""Redraw: an edit on an image model that takes no picture in. It draws again +from the words the base was drawn with and the change, so Edit works on every +image model; the version says it was drawn again, since it will not keep the +base's composition.""" + +import asyncio +import io + +from PIL import Image, UnidentifiedImageError + +from modules.llm.providers.picture_shape import Shape, shape_of +from modules.llm.resolution import ResolvedImageGeneration +from worker.studio.media.visual import EXTENSIONS +from worker.studio.media.visual.edit.base import BaseImage +from worker.studio.media.visual.redraw.drawing_prompt import ( + changed, + kept_drawing_prompt, +) +from worker.studio.shared.artifact import Built + +SAID = "Drawn again with your change." + + +def render( + painter: ResolvedImageGeneration, base: BaseImage, instruction: str +) -> Built: + prompt = changed(base.drawing_prompt, instruction) + image = asyncio.run( + painter.generator.generate(painter.selection.name, prompt, shape=_shape(base)) + ) + edited = ( + f"{base.body}\n\nEdited: {instruction}" + if base.body + else f"Edited: {instruction}" + ) + return Built( + title=base.title, + markdown=f"{SAID}\n\n{edited}", + primary=image.content, + primary_mime=image.media_type, + primary_filename=f"{base.stem}.{EXTENSIONS[image.media_type]}", + metadata={**kept_drawing_prompt(prompt), "redrawn": True}, + ) + + +def _shape(base: BaseImage) -> Shape | None: + try: + width, height = Image.open(io.BytesIO(base.picture.content)).size + except (UnidentifiedImageError, OSError): + return None + return shape_of(width, height) From 561147e123e040ca01d5a7f424ba0b44674d897c Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:50:57 +0530 Subject: [PATCH 04/33] feat(local): comics as drawn pages bound into one PDF, with a skill that plans them The surfsense-pictures skill plans a set of pictures with a character sheet, confirms the count once, draws the first page alone and the rest with it as their reference, and binds them with a script that places each page as JPEG, setting a page's exact lines under it when its lettering came out wrong. The prompt sends pictures and comics there, and the web page skill sends them away. --- .../backend/modules/agent/opencode_config.py | 11 +- .../backend/modules/agent/prompts/agent.md | 2 +- .../agent/skills/surfsense-html/SKILL.md | 2 +- .../agent/skills/surfsense-pictures/SKILL.md | 49 ++++++++ .../agent/skills/surfsense-pictures/comics.md | 36 ++++++ .../agent/skills/surfsense-pictures/pdf.md | 64 ++++++++++ .../tests/unit/agent/test_opencode_config.py | 1 + .../tests/unit/agent/test_pictures_skill.py | 110 ++++++++++++++++++ 8 files changed, 272 insertions(+), 3 deletions(-) create mode 100644 surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md create mode 100644 surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md create mode 100644 surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md create mode 100644 surfsense_local/backend/tests/unit/agent/test_pictures_skill.py diff --git a/surfsense_local/backend/modules/agent/opencode_config.py b/surfsense_local/backend/modules/agent/opencode_config.py index ee91c97b7d..9deaf5d26f 100644 --- a/surfsense_local/backend/modules/agent/opencode_config.py +++ b/surfsense_local/backend/modules/agent/opencode_config.py @@ -61,7 +61,16 @@ PDF_SKILL = "surfsense-pdf" # How to write a web page and keep it with surfsense_save_page (ADR 0066). HTML_SKILL = "surfsense-html" -SKILLS = (DOCUMENTS_SKILL, REVISIONS_SKILL, DATA_SKILL, PDF_SKILL, HTML_SKILL) +# How to draw a picture, or a set such as a comic, and bind a set into a PDF. +PICTURES_SKILL = "surfsense-pictures" +SKILLS = ( + DOCUMENTS_SKILL, + REVISIONS_SKILL, + DATA_SKILL, + PDF_SKILL, + HTML_SKILL, + PICTURES_SKILL, +) # Denied as permissions, which also leave the tool out of what the model is # offered: document scripts reach the runner through a SurfSense tool (ADR 0039), diff --git a/surfsense_local/backend/modules/agent/prompts/agent.md b/surfsense_local/backend/modules/agent/prompts/agent.md index 6b35994ec2..6820139493 100644 --- a/surfsense_local/backend/modules/agent/prompts/agent.md +++ b/surfsense_local/backend/modules/agent/prompts/agent.md @@ -30,7 +30,7 @@ For a Word document, a PDF, a PowerPoint deck or an Excel workbook, load the `su For a quiz, flashcards, a podcast or a mind map, start it in Studio with `surfsense_create_artifact`, naming the sources by the number at the end of their file names, then tell the user it will appear in Studio. Studio reads those sources itself, so you need not read them first. -For a picture or infographic, when asked: `surfsense_make_image`. +Pictures/comics: `surfsense-pictures` skill, `surfsense_make_image`. For any other document, table or file, write it to `outputs/` with a clear file name, then say in one line what you wrote and where. diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md b/surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md index cb7a29e270..b4673153de 100644 --- a/surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md +++ b/surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md @@ -1,6 +1,6 @@ --- name: surfsense-html -description: Make or change a web page from the user's sources - a report, a briefing, a dashboard, a comparison, a timeline, an explainer, an interactive calculator or a figure that shows how something works - by writing an HTML file and keeping it with surfsense_save_page. Load it whenever the user asks for a web page, an HTML page, an interactive page, or a change to one you made. +description: Make or change a web page from the user's sources - a report, a briefing, a dashboard, a comparison, a timeline, an explainer, an interactive calculator or a figure that shows how something works - by writing an HTML file and keeping it with surfsense_save_page. Load it whenever the user asks for a web page, an HTML page, an interactive page, or a change to one you made. Not for a comic, a storyboard or a picture: those are drawn pictures, with the surfsense-pictures skill. --- # Web pages diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md new file mode 100644 index 0000000000..bec0287367 --- /dev/null +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md @@ -0,0 +1,49 @@ +--- +name: surfsense-pictures +description: Draw pictures with surfsense_make_image - one picture or infographic, or a set that belongs together, such as a comic, a manga, a comic strip, a storyboard or a picture book, bound into one PDF. Load it whenever the user asks for a picture, an illustration, or any set of pictures, or for a change to one you drew. Not for charts of numbers, and not for a web page. +--- + +# Pictures + +`surfsense_make_image` draws with the user's image model and keeps each picture in Studio, named `image-<id>`. This skill is how to use it well. + +## The files in this skill + +| File | Read it | +|---|---| +| `comics.md` | For a comic, a manga, a strip, a storyboard or a picture book: pages, panels and lettering. | +| `pdf.md` | When you bind pictures into a PDF: the script to copy and fill. | + +## One picture + +1. Read any sources the picture draws on first. +2. Write one self-contained prompt: the subject, the setting, the style, the light, what is in front and what is behind. As short as it can be while still saying what to draw. +3. Choose `shape` when it matters: `portrait` for a page or a poster, `landscape` for a banner or a strip, `square` for an icon. +4. Look at what comes back. Image models get hands, counts and lettering wrong. Change it once with its `artifact_id` when it is plainly wrong, and say what you changed; otherwise tell the user what is off. + +No picture for numbers: a chart stays a chart, in a document. Keep any words in a picture short, a title or a label; text that must be exact goes in the document around it. + +## A set of pictures + +A comic, a storyboard, a picture book, or any pictures that must look like one set: + +1. **Plan** the set: what each picture shows, in order. Write a **character sheet**, each character's face, hair, build, clothes and colours in a few fixed words, and a **style line**: medium, palette, line weight. Both go word for word into every prompt. +2. **Confirm once.** Say how many pictures, the plan in short, and that each is billed by their provider, or takes minutes on their computer. On yes, draw every picture without asking again. A single picture needs no confirmation. +3. **Draw the first alone,** and look at it. When it is wrong, change it before going on: every later picture copies its characters. +4. **Draw the rest with the first as their reference**: `references: ["image-<first>"]`. Give each its own call, all in one step, so a remote model draws them side by side. A character picture the user gave, or a character sheet they asked for first, is the reference instead. +5. **Title each** "<title> · page <n>", or "· scene <n>": a chat cannot reuse a picture's title. +6. **Bind them** into one PDF with `surfsense_render_document` ([`pdf.md`](pdf.md)). The PDF is what the user keeps. +7. **Answer** with the PDF and one line per picture, so the user can ask for "page 3 again". + +## When the model cannot take a picture in + +Some image models draw from words alone. Nothing is refused, and nothing needs setting up: + +- A reference is drawn from the words alone. The result says so: then every picture's character sheet carries the likeness, and in the plan you confirm, say once, plainly, that characters may differ a little between pictures. +- A change is drawn again from the picture's prompt and the change, and may not keep the rest. The result says so; tell the user in one line. + +Never send the user to Settings for this. + +## Changing a set later + +Change one picture with its `artifact_id`, then read the PDF's script with `surfsense_read_document`, put the new name in place of the old, and render the PDF's next version. To add a picture, draw it with the first as reference and render again. diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md new file mode 100644 index 0000000000..5f6b37a0a4 --- /dev/null +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md @@ -0,0 +1,36 @@ +# Comics + +A comic, a manga, a comic strip, a storyboard or a picture book: one picture per page, every page from the same character sheet and style line, bound into one PDF. Follow the set's steps in [`SKILL.md`](SKILL.md#a-set-of-pictures); this file is what is particular to comics. + +## The plan + +- **Pages.** Four when the user names no number; at most twelve, since each is billed or takes minutes. A comic strip is one `landscape` picture of three or four panels. +- **Panels.** Two to five a page. For each: where it sits (top, bottom left, wide), what it shows, and its lines. +- **Lines.** One short line a bubble, at most about eight words, two or three bubbles a panel. A caption box for narration. Write every line in the plan, word for word: the PDF may need them. + +## A page's prompt + +The style line, the character sheet, then the page, laid out as the image model reads it best: + +```text +Full-colour comic book page, clean bold ink lines, flat colours, white gutters, +black-bordered panels, speech bubbles with clear hand-lettered capitals. +Characters: MILO, a small orange tabby cat with a white chest, green eyes and a red +scarf. NORA, a woman about seventy, short white hair, round glasses, yellow raincoat. +Page 2 of 4 of the comic "The Lighthouse Cat". A portrait page of 3 panels: +Panel 1 (wide, top): a fishing boat tosses on huge waves at night. +Panel 2 (bottom left): NORA taps the dead lamp. NORA says: "THE BULB IS BROKEN!" +Panel 3 (bottom right): MILO spots a box of spare bulbs on a high shelf. +``` + +From page 2 on, add one line before the page: "Keep the characters looking exactly as in the reference picture: same faces, colours, clothes and drawing style. Draw a new page; do not copy its panels." Every page is `shape: portrait`. + +## Lettering + +- **Bubbles drawn by the model.** Write each line in the prompt in quotes, in capitals. +- **A misspelt line is not redrawn.** Note the page; the PDF sets that page's lines under it, exactly, in a strip ([`pdf.md`](pdf.md)). A redraw would bill another picture for one letter. +- **On this computer** (the result says "Drawn on this computer"), **or when you cannot see the pictures**, draw every page without words: leave the bubbles out of the prompts, ask for "no text, no lettering, no speech bubbles", and give every page its strip. You cannot rely on the lettering, or check it. + +## Before you bind + +Look at every page. Redraw or change one only when it is plainly wrong, such as a missing character, the wrong number of panels, or a third arm, and say which and why. Then bind the PDF, every page's name in `images`, in order. diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md new file mode 100644 index 0000000000..75c114b308 --- /dev/null +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md @@ -0,0 +1,64 @@ +# Binding pictures into a PDF + +Render this with `surfsense_render_document`, `format: pdf`, every page's name in `images`, in order. Fill in `TITLE` and `PAGES`, and change nothing else unless the user asks: it is the layout every comic and picture book uses. + +- One picture per page, full bleed, at comic-book width, each page as tall as its picture. +- A page's lines, when given, are set under its picture in a strip, exactly as written. Give them for a page whose lettering came out wrong, or for every page drawn without words. +- Pictures are placed as JPEG, so a book of pages stays a few megabytes. +- A page still being drawn may be named all the same: the PDF waits for it, and appears in Studio when its last page is drawn. Tell the user so, and stop. + +```python +import os +from io import BytesIO + +from PIL import Image +from reportlab.lib.units import inch +from reportlab.lib.utils import ImageReader +from reportlab.pdfbase.pdfmetrics import stringWidth +from reportlab.pdfgen import canvas + +TITLE = "The Lighthouse Cat" +# Each page in order: its picture's name, and the lines to set under it, or none. +PAGES = [ + ("image-12", []), + ("image-13", ["NORA: The bulb is broken!"]), +] + +WIDTH = 6.625 * inch +MARGIN = 0.25 * inch +FONT, SIZE, LEADING = "Helvetica", 11, 15 + + +def wrapped(text, width): + lines, line = [], "" + for word in text.split(): + trial = f"{line} {word}".strip() + if line and stringWidth(trial, FONT, SIZE) > width: + lines.append(line) + line = word + else: + line = trial + return lines + [line] if line else lines + + +pdf = canvas.Canvas(os.environ["OUTPUT_PATH"]) +pdf.setTitle(TITLE) +for name, said in PAGES: + path = os.path.join(os.environ["IMAGES_DIR"], name + ".png") + picture = Image.open(path).convert("RGB") + jpeg = BytesIO() + picture.save(jpeg, "JPEG", quality=90) + jpeg.seek(0) + height = WIDTH * picture.height / picture.width + lines = [line for text in said for line in wrapped(text, WIDTH - 2 * MARGIN)] + strip = len(lines) * LEADING + 2 * MARGIN if lines else 0 + pdf.setPageSize((WIDTH, height + strip)) + pdf.drawImage(ImageReader(jpeg), 0, strip, WIDTH, height) + pdf.setFont(FONT, SIZE) + y = strip - MARGIN - SIZE + for line in lines: + pdf.drawString(MARGIN, y, line) + y -= LEADING + pdf.showPage() +pdf.save() +``` diff --git a/surfsense_local/backend/tests/unit/agent/test_opencode_config.py b/surfsense_local/backend/tests/unit/agent/test_opencode_config.py index dffedbfc11..9697bb9d28 100644 --- a/surfsense_local/backend/tests/unit/agent/test_opencode_config.py +++ b/surfsense_local/backend/tests/unit/agent/test_opencode_config.py @@ -210,6 +210,7 @@ def test_only_surfsenses_own_skills_load_from_the_shipped_folder( "surfsense-data", "surfsense-pdf", "surfsense-html", + "surfsense-pictures", ) for name in names: assert (Path(skills) / name / "SKILL.md").is_file() diff --git a/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py new file mode 100644 index 0000000000..bd074514c1 --- /dev/null +++ b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py @@ -0,0 +1,110 @@ +"""The pictures skill as the agent reads it, and its binding script as a render +runs it: alone, from an empty folder, with the pictures named in IMAGES_DIR.""" + +import io +import os +import re +import subprocess +import sys +from pathlib import Path + +import pytest +from PIL import Image +from pypdf import PdfReader + +from modules.agent.opencode_config import PICTURES_SKILL, skills_folder +from modules.agent.tool_endpoint.make_image import LISTING + +pytestmark = pytest.mark.unit + + +def _file(name: str) -> str: + return (skills_folder() / PICTURES_SKILL / name).read_text(encoding="utf-8") + + +def test_the_skill_names_itself_as_opencode_finds_it() -> None: + """opencode lists a skill by its frontmatter name, and the config allows only that one.""" + frontmatter = _file("SKILL.md").split("---")[1] + + assert f"name: {PICTURES_SKILL}\n" in frontmatter + assert "comic" in frontmatter + + +def test_the_tool_sends_a_set_of_pictures_to_the_skill() -> None: + """The prompt is full, so the tool's description names the skill too.""" + assert f"load the {PICTURES_SKILL} skill first" in LISTING["description"] + + +def test_the_skill_never_sends_the_user_to_settings() -> None: + """A model that takes no picture in is said plainly, never set up mid-comic.""" + for name in ("SKILL.md", "comics.md", "pdf.md"): + # The app writes the path with a chevron; the agent's prompts with ">". + assert "Settings \u203a" not in _file(name), name + assert "Settings >" not in _file(name), name + + +def _script(pages: list[tuple[str, list[str]]]) -> str: + """The script the skill gives, filled as the agent fills it.""" + (script,) = re.findall(r"```python\n(.*?)```", _file("pdf.md"), re.S) + filled = re.sub( + r"PAGES = \[\n.*?\n\]", f"PAGES = {pages!r}", script, count=1, flags=re.S + ) + assert filled != script + return filled + + +def _page(colour: str, size: tuple[int, int]) -> bytes: + out = io.BytesIO() + Image.new("RGB", size, colour).save(out, "PNG") + return out.getvalue() + + +def test_the_pages_are_bound_one_a_page_as_jpeg(tmp_path: Path) -> None: + """Four comic pages in a PDF of a few megabytes, not the tens PNG made.""" + images, work = tmp_path / "images", tmp_path / "work" + images.mkdir() + work.mkdir() + noisy = Image.effect_noise((1024, 1536), 64).convert("RGB") + for n in range(1, 5): + out = io.BytesIO() + noisy.save(out, "PNG") + (images / f"image-{n}.png").write_bytes(out.getvalue()) + output = tmp_path / "comic.pdf" + + subprocess.run( + [sys.executable, "-c", _script([(f"image-{n}", []) for n in range(1, 5)])], + cwd=work, + env={**os.environ, "OUTPUT_PATH": str(output), "IMAGES_DIR": str(images)}, + check=True, + ) + + pdf = PdfReader(output) + assert len(pdf.pages) == 4 + for page in pdf.pages: + (picture,) = page.images + assert picture.name.endswith(".jpg"), picture.name + assert output.stat().st_size < 5 * 1024 * 1024 + + +def test_a_page_with_lines_gets_them_exactly_in_a_strip_below(tmp_path: Path) -> None: + """A page whose lettering came out wrong keeps the exact words, unredrawn.""" + images, work = tmp_path / "images", tmp_path / "work" + images.mkdir() + work.mkdir() + (images / "image-1.png").write_bytes(_page("white", (1024, 1536))) + (images / "image-2.png").write_bytes(_page("white", (1024, 1536))) + output = tmp_path / "comic.pdf" + said = "NORA: The bulb is broken!" + + subprocess.run( + [sys.executable, "-c", _script([("image-1", []), ("image-2", [said])])], + cwd=work, + env={**os.environ, "OUTPUT_PATH": str(output), "IMAGES_DIR": str(images)}, + check=True, + ) + + plain, lettered = PdfReader(output).pages + assert said in lettered.extract_text() + assert said not in plain.extract_text() + assert float(lettered.mediabox.height) > float(plain.mediabox.height) + assert float(lettered.mediabox.width) == float(plain.mediabox.width) From a716ffc5b1077eedaca04b5e92e9694fec4dcb5b Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:51:06 +0530 Subject: [PATCH 05/33] feat(local): a document waits on the server for the pictures it places A render naming a picture still being drawn is made pending with the pictures it waits for and answers at once. The end of every Studio job queues the renders waiting on that picture once all of theirs are drawn, or fails one whose picture failed, saying which; one update takes the wait off, so two pictures ending together queue it once. The worker does the same at start. --- .../agent/tool_endpoint/render_document.py | 13 +- .../artifacts/script_documents/service.py | 30 ++-- .../artifacts/waiting_renders/__init__.py | 0 .../artifacts/waiting_renders/settle.py | 100 +++++++++++ .../artifacts/waiting_renders/waits.py | 25 +++ .../integration/agent/test_waiting_render.py | 159 ++++++++++++++++++ .../tests/unit/worker/test_consumer.py | 3 + surfsense_local/backend/worker/consumer.py | 3 + surfsense_local/backend/worker/studio/job.py | 3 + .../backend/worker/waiting_renders.py | 27 +++ 10 files changed, 351 insertions(+), 12 deletions(-) create mode 100644 surfsense_local/backend/modules/artifacts/waiting_renders/__init__.py create mode 100644 surfsense_local/backend/modules/artifacts/waiting_renders/settle.py create mode 100644 surfsense_local/backend/modules/artifacts/waiting_renders/waits.py create mode 100644 surfsense_local/backend/tests/integration/agent/test_waiting_render.py create mode 100644 surfsense_local/backend/worker/waiting_renders.py diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/render_document.py b/surfsense_local/backend/modules/agent/tool_endpoint/render_document.py index 2c2e573bd9..64fb49a113 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/render_document.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/render_document.py @@ -46,6 +46,7 @@ ) from modules.artifacts.script_documents.spec import DOCUMENT_FORMATS, FORMAT_NAMES from modules.artifacts.script_documents.version import version_of +from modules.artifacts.waiting_renders.waits import waiting_sentence, waits_for from modules.documents.models import DocumentStatus from modules.workspaces.models import Workspace @@ -148,6 +149,8 @@ class _Started: artifact_id: int version: int title: str + # The pictures still being drawn it waits for; it is queued once they are. + waits_for: tuple[int, ...] = () def render( @@ -166,6 +169,9 @@ def render( if request.base_artifact_id is None and scope.known: refuse_same_title(session, scope.thread_id, request.title, request.format) started = _start(session, scope, request) + if started.waits_for: + waiting = RenderedArtifact(started.artifact_id, started.title, started.version) + return f"{queued_line(waiting)}\n{waiting_sentence(started.waits_for)}" wait = min(WAIT_SECONDS, deadline - time.monotonic()) outcome = wait_for_outcome(session, started.artifact_id, wait) if outcome is None: @@ -253,7 +259,12 @@ def _start(session: Session, scope: TurnScope, request: _Request) -> _Started: raise version = version_of(artifact.artifact_metadata) assert version is not None # the service gave it one - return _Started(artifact.id, version.number, request.title.strip()) + return _Started( + artifact.id, + version.number, + request.title.strip(), + waits_for(artifact.artifact_metadata), + ) def _create(session: Session, scope: TurnScope, request: _Request) -> Artifact: diff --git a/surfsense_local/backend/modules/artifacts/script_documents/service.py b/surfsense_local/backend/modules/artifacts/script_documents/service.py index a7f3ff8eb9..a0d7daa2f1 100644 --- a/surfsense_local/backend/modules/artifacts/script_documents/service.py +++ b/surfsense_local/backend/modules/artifacts/script_documents/service.py @@ -30,6 +30,8 @@ version_of, ) from modules.artifacts.studio_documents.recipe import studio_made +from modules.artifacts.waiting_renders.settle import settle_waiting_renders +from modules.artifacts.waiting_renders.waits import KEY as WAITS_FOR from modules.documents.models import Document, DocumentStatus, DocumentType from modules.documents.source_figures import parse_figure_name from modules.embedding.active import EmbeddingNotChosenError, require_active_index @@ -87,7 +89,7 @@ def create_script_document( ) from error images = tuple(dict.fromkeys(image_names)) - source_ids = _image_sources(session, workspace.id, chat_thread_id, images) + source_ids, waiting = _image_sources(session, workspace.id, chat_thread_id, images) base = ( None if base_artifact_id is None @@ -132,8 +134,14 @@ def create_script_document( "version": version.as_metadata(), "source_document_ids": source_ids, "prompt": None, + **({WAITS_FOR: list(waiting)} if waiting else {}), } - queue_job(session, artifact) + if waiting: + session.commit() + # A picture may have ended between the check and the commit. + settle_waiting_renders(session, render_id=artifact.id) + else: + queue_job(session, artifact) logger.info( "studio: enqueued script document %s format=%s root=%s v%s", artifact.id, @@ -149,22 +157,22 @@ def _image_sources( workspace_id: int, chat_thread_id: int | None, names: tuple[str, ...], -) -> list[int]: - """The sources the named figures come from, refusing a name none of them, nor - the chat's analyses, has. A chart is the chat's own output, made from no one source.""" +) -> tuple[list[int], tuple[int, ...]]: + """The sources the named figures come from, and the pictures still being drawn + that the document waits for; refusing a name none of them, nor the chat's + analyses, has. A chart is the chat's own output, made from no one source.""" + waiting: list[int] = [] for name in names: try: script_image(session, workspace_id, chat_thread_id, name) - except ImageStillDrawingError as error: - raise ScriptDocumentRefusedError( - f'The picture "{name}" is still being drawn. Tell the user it will be ' - "placed once it appears in Studio, and stop." - ) from error + except ImageStillDrawingError: + waiting.append(int(name.removeprefix("image-"))) except LookupError as error: raise ScriptDocumentRefusedError(_no_image(name)) from error figures = [parse_figure_name(name) for name in names if not is_chart_name(name)] # script_image accepted each name, so each figure's parses. - return list(dict.fromkeys(parsed[0] for parsed in figures if parsed is not None)) + sources = list(dict.fromkeys(p[0] for p in figures if p is not None)) + return sources, tuple(waiting) def _no_image(name: str) -> str: diff --git a/surfsense_local/backend/modules/artifacts/waiting_renders/__init__.py b/surfsense_local/backend/modules/artifacts/waiting_renders/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/modules/artifacts/waiting_renders/settle.py b/surfsense_local/backend/modules/artifacts/waiting_renders/settle.py new file mode 100644 index 0000000000..90270dc3df --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/waiting_renders/settle.py @@ -0,0 +1,100 @@ +"""Queue a waiting document once its pictures are drawn, or fail it when one is not. + +Run where a picture's job ends, when a waiting document is made, and when the +worker starts, since a picture can end while no worker is running. +""" + +from sqlalchemy import func, select, update +from sqlalchemy.orm import Session + +from modules.artifacts.drawn_images.names import image_name +from modules.artifacts.models import Artifact +from modules.artifacts.queued_job import queue_job +from modules.artifacts.waiting_renders.waits import KEY, waits_for +from modules.documents.models import Document, DocumentStatus, DocumentType + +_ENDED_BADLY = (DocumentStatus.FAILED, DocumentStatus.CANCELLED) + + +def settle_waiting_renders( + session: Session, *, picture_id: int | None = None, render_id: int | None = None +) -> list[Artifact]: + """The waiting documents this settled, queued or failed, for the caller to announce.""" + settled: list[Artifact] = [] + for render in _waiting(session, picture_id, render_id): + pictures = waits_for(render.artifact_metadata) + statuses = _statuses(session, pictures) + gone = [ + i + for i in pictures + if statuses.get(i) is None or statuses[i] in _ENDED_BADLY + ] + if gone: + _fail(session, render, gone[0], statuses.get(gone[0])) + settled.append(render) + elif all(statuses[i] is DocumentStatus.READY for i in pictures) and _claim( + session, render + ): + queue_job(session, render) + settled.append(render) + session.commit() + return settled + + +def _waiting( + session: Session, picture_id: int | None, render_id: int | None +) -> list[Artifact]: + query = ( + select(Artifact) + .join(Document, Artifact.document_id == Document.id) + .where( + Document.status == DocumentStatus.PENDING, + Document.document_type == DocumentType.ARTIFACT, + func.json_extract(Artifact.artifact_metadata, f"$.{KEY}").is_not(None), + ) + ) + if render_id is not None: + query = query.where(Artifact.id == render_id) + found = session.scalars(query).all() + if picture_id is None: + return list(found) + return [r for r in found if picture_id in waits_for(r.artifact_metadata)] + + +def _statuses( + session: Session, artifact_ids: tuple[int, ...] +) -> dict[int, DocumentStatus]: + rows = session.execute( + select(Artifact.id, Document.status) + .join(Document, Artifact.document_id == Document.id) + .where(Artifact.id.in_(artifact_ids)) + ) + return dict(rows.tuples().all()) + + +def _claim(session: Session, render: Artifact) -> bool: + """Take the wait off in one update, so two pictures ending at once queue it once.""" + meta = {k: v for k, v in (render.artifact_metadata or {}).items() if k != KEY} + claimed = session.execute( + update(Artifact) + .where( + Artifact.id == render.id, + func.json_extract(Artifact.artifact_metadata, f"$.{KEY}").is_not(None), + ) + .values(artifact_metadata=meta) + .execution_options(synchronize_session=False) + ) + session.expire(render) + return claimed.rowcount == 1 + + +def _fail( + session: Session, render: Artifact, picture_id: int, status: DocumentStatus | None +) -> None: + name = image_name(picture_id) + what = "was deleted" if status is None else "failed" + render.document.status = DocumentStatus.FAILED + render.document.error_message = ( + f"{name} {what}, so this document was not made. Draw that picture again, " + "or render without it." + ) diff --git a/surfsense_local/backend/modules/artifacts/waiting_renders/waits.py b/surfsense_local/backend/modules/artifacts/waiting_renders/waits.py new file mode 100644 index 0000000000..e66fbe761b --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/waiting_renders/waits.py @@ -0,0 +1,25 @@ +"""The pictures a pending document waits for, kept in its metadata until it is queued.""" + +from typing import Any + +from modules.artifacts.drawn_images.names import image_name + +KEY = "waits_for" + + +def waits_for(metadata: dict[str, Any] | None) -> tuple[int, ...]: + return tuple(int(i) for i in (metadata or {}).get(KEY) or ()) + + +def waiting_sentence(artifact_ids: tuple[int, ...]) -> str: + """How the render's call answers: what it waits for, and not to render again.""" + names = [image_name(i) for i in artifact_ids] + drawn = ( + f"{names[0]} is drawn" + if len(names) == 1 + else f"{', '.join(names[:-1])} and {names[-1]} are drawn" + ) + return ( + f"It renders once {drawn}, and appears in Studio then: tell the user so, " + "and do not render it again." + ) diff --git a/surfsense_local/backend/tests/integration/agent/test_waiting_render.py b/surfsense_local/backend/tests/integration/agent/test_waiting_render.py new file mode 100644 index 0000000000..961f330f63 --- /dev/null +++ b/surfsense_local/backend/tests/integration/agent/test_waiting_render.py @@ -0,0 +1,159 @@ +"""A document that places a picture still being drawn waits for it, on the +server, and renders once its pictures are drawn: a comic drawn slowly on this +computer still ends as one PDF without the user asking again.""" + +import asyncio +import threading +import time + +import pytest +from sqlalchemy import Engine + +from modules.artifacts.models import Artifact +from modules.documents.models import DocumentStatus +from shared.db import create_session_factory +from tests.integration.agent.test_make_image import ( # noqa: F401 - fixtures + Painter, + _artifact_id, + _pictures_in_word, + _placing, + _png, + painter, +) +from tests.integration.agent.tool_endpoint_client import ToolEndpoint +from tests.integration.worker.conftest import stub_model # noqa: F401 +from worker.waiting_renders import settle_every_waiting_render + +pytestmark = [ + pytest.mark.integration, + pytest.mark.usefixtures("stub_model", "model_reads_images"), +] + + +def _status(engine: Engine, artifact_id: int) -> tuple[DocumentStatus, str | None]: + with create_session_factory(engine)() as session: + document = session.get(Artifact, artifact_id).document + return document.status, document.error_message + + +async def _ended(engine: Engine, artifact_id: int) -> tuple[DocumentStatus, str | None]: + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + status, error = _status(engine, artifact_id) + if status not in (DocumentStatus.PENDING, DocumentStatus.PROCESSING): + return status, error + await asyncio.sleep(0.1) + raise AssertionError(f"artifact {artifact_id} never ended") + + +@pytest.fixture +def slow(monkeypatch: pytest.MonkeyPatch, painter: Painter) -> Painter: # noqa: F811 + """The picture call answers queued, and the drawing waits to be let go.""" + monkeypatch.setattr("modules.agent.tool_endpoint.make_image.WAIT_SECONDS", 0.2) + painter.hold = threading.Event() + return painter + + +async def test_a_render_naming_a_picture_still_drawn_waits_and_then_renders( + tools: ToolEndpoint, studio_worker: None, slow: Painter, engine: Engine +) -> None: + """Answered at once, not refused; made once the picture is drawn.""" + workspace_id = await tools.workspace() + queued, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + name = f"image-{_artifact_id(queued)}" + + began = time.monotonic() + text, is_error = await tools.call(workspace_id, "render_document", _placing(name)) + + assert is_error is False, text + assert time.monotonic() - began < 10 + assert text.startswith("Queued artifact ") + assert f"renders once {name} is drawn" in text + render = _artifact_id(text) + assert _status(engine, render)[0] is DocumentStatus.PENDING + + assert slow.hold is not None + slow.hold.set() + + assert (await _ended(engine, render))[0] is DocumentStatus.READY + assert _pictures_in_word(engine, render) == [_png()] + + +async def test_it_waits_only_for_the_pictures_not_yet_drawn( + tools: ToolEndpoint, + studio_worker: None, + painter: Painter, # noqa: F811 - the fixture + engine: Engine, +) -> None: + """Page one is ready, page two is still drawing: it names only page two.""" + workspace_id = await tools.workspace() + first, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page one", "title": "Page 1"} + ) + painter.hold = threading.Event() + with pytest.MonkeyPatch.context() as patch: + patch.setattr("modules.agent.tool_endpoint.make_image.WAIT_SECONDS", 0.2) + second, _ = await tools.call( + workspace_id, "make_image", {"prompt": "Page two", "title": "Page 2"} + ) + one, two = f"image-{_artifact_id(first)}", f"image-{_artifact_id(second)}" + placing = {**_placing(one), "images": [one, two]} + + text, is_error = await tools.call(workspace_id, "render_document", placing) + + assert is_error is False, text + assert f"renders once {two} is drawn" in text + assert one not in text.split("renders once", 1)[1] + painter.hold.set() + assert (await _ended(engine, _artifact_id(text)))[0] is DocumentStatus.READY + + +async def test_a_picture_that_fails_fails_the_render_saying_which( + tools: ToolEndpoint, studio_worker: None, slow: Painter, engine: Engine +) -> None: + """The agent can draw that page again, or render without it.""" + workspace_id = await tools.workspace() + queued, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + name = f"image-{_artifact_id(queued)}" + text, _ = await tools.call(workspace_id, "render_document", _placing(name)) + + slow.fail_with = RuntimeError("the provider refused it") + assert slow.hold is not None + slow.hold.set() + + status, error = await _ended(engine, _artifact_id(text)) + assert status is DocumentStatus.FAILED + assert error is not None and f"{name} failed" in error + + +async def test_a_picture_that_ended_while_the_worker_was_down_is_settled_at_start( + tools: ToolEndpoint, studio_worker: None, slow: Painter, engine: Engine +) -> None: + """The job that would have queued it never ran its end.""" + workspace_id = await tools.workspace() + queued, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + picture = _artifact_id(queued) + text, _ = await tools.call( + workspace_id, "render_document", _placing(f"image-{picture}") + ) + render = _artifact_id(text) + # As a quit leaves it: the picture's run failed as interrupted, unannounced. + with create_session_factory(engine)() as session: + document = session.get(Artifact, picture).document + document.status = DocumentStatus.FAILED + document.error_message = "interrupted when the app closed" + session.commit() + + settle_every_waiting_render() + + status, error = _status(engine, render) + assert status is DocumentStatus.FAILED + assert error is not None and f"image-{picture} failed" in error + assert slow.hold is not None + slow.hold.set() diff --git a/surfsense_local/backend/tests/unit/worker/test_consumer.py b/surfsense_local/backend/tests/unit/worker/test_consumer.py index 1dca29a804..778082986d 100644 --- a/surfsense_local/backend/tests/unit/worker/test_consumer.py +++ b/surfsense_local/backend/tests/unit/worker/test_consumer.py @@ -35,6 +35,7 @@ def run(self) -> None: # Startup steps that read the database, which a unit test has none of. monkeypatch.setattr(consumer, "wait_for_schema", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_documents", lambda _kinds: None) + monkeypatch.setattr(consumer, "settle_every_waiting_render", lambda: None) consumer.consume("ingest") consumer.consume("studio") @@ -64,6 +65,7 @@ def run(self) -> None: monkeypatch.setattr(consumer, "Consumer", FakeConsumer) monkeypatch.setattr(consumer, "wait_for_schema", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_documents", lambda _kinds: None) + monkeypatch.setattr(consumer, "settle_every_waiting_render", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_runs", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_pulls", lambda: None) monkeypatch.setattr(consumer, "remove_leftover_run_folders", lambda: None) @@ -102,6 +104,7 @@ def run(self) -> None: monkeypatch.setattr(consumer, "Consumer", FakeConsumer) monkeypatch.setattr(consumer, "wait_for_schema", lambda: None) monkeypatch.setattr(consumer, "fail_interrupted_documents", lambda _kinds: None) + monkeypatch.setattr(consumer, "settle_every_waiting_render", lambda: None) left = data_dir / "tmp" / "document-scripts" / "0f1e2d3c" (left / "images").mkdir(parents=True) (left / "images" / "7-1.png").write_bytes(b"a figure") diff --git a/surfsense_local/backend/worker/consumer.py b/surfsense_local/backend/worker/consumer.py index 850168748f..361708056f 100644 --- a/surfsense_local/backend/worker/consumer.py +++ b/surfsense_local/backend/worker/consumer.py @@ -12,6 +12,7 @@ from worker.interrupted_documents import fail_interrupted_documents from worker.interrupted_pulls import fail_interrupted_pulls from worker.wait_for_schema import wait_for_schema +from worker.waiting_renders import settle_every_waiting_render # Studio jobs wait on a model: overlap them. Ingest jobs saturate the CPU: one. # ponytail: laptop defaults; becomes a setting for power users. @@ -46,6 +47,8 @@ def consume(name: str) -> None: # No script runs yet, so every run folder is one a killed worker left. remove_leftover_run_folders() fail_interrupted_documents({DocumentType.ARTIFACT}) + # After those: a document waiting on an interrupted picture fails too. + settle_every_waiting_render() logging.getLogger(__name__).info( "%s: worker consuming with %s threads", name, workers ) diff --git a/surfsense_local/backend/worker/studio/job.py b/surfsense_local/backend/worker/studio/job.py index 43bf78b4cb..2d0f623721 100644 --- a/surfsense_local/backend/worker/studio/job.py +++ b/surfsense_local/backend/worker/studio/job.py @@ -76,6 +76,7 @@ from worker.studio.shared import gather, persist, window_budget from worker.studio.shared.artifact import Built, Source from worker.studio.web.html.written import page as written_page +from worker.waiting_renders import settle_after_job logger = logging.getLogger(__name__) @@ -107,6 +108,8 @@ def run(artifact_id: int) -> None: _generate(session, artifact) finally: engine.dispose() + # However it ended: a document waiting on this picture may now go. + settle_after_job(artifact_id) def _generate(session: Session, artifact: Artifact) -> None: diff --git a/surfsense_local/backend/worker/waiting_renders.py b/surfsense_local/backend/worker/waiting_renders.py new file mode 100644 index 0000000000..84c49d7bd5 --- /dev/null +++ b/surfsense_local/backend/worker/waiting_renders.py @@ -0,0 +1,27 @@ +"""Settle the documents waiting on pictures from the worker, and tell open windows.""" + +from modules.artifacts.waiting_renders.settle import settle_waiting_renders +from shared.config import get_storage_settings +from shared.db import create_db_engine, create_session_factory +from worker.notify import notify_artifact_updates + + +def settle_after_job(artifact_id: int) -> None: + """A job ended, or found its artifact gone: what waited on that picture may go.""" + _settle(picture_id=artifact_id) + + +def settle_every_waiting_render() -> None: + """At start: a picture may have ended, or been failed as interrupted, unannounced.""" + _settle() + + +def _settle(picture_id: int | None = None) -> None: + engine = create_db_engine(get_storage_settings().database_path) + try: + with create_session_factory(engine)() as session: + for render in settle_waiting_renders(session, picture_id=picture_id): + session.refresh(render) + notify_artifact_updates(render) + finally: + engine.dispose() From 3fb7bc942ccb264e9c2c3dd88517de4b5663216e Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:51:17 +0530 Subject: [PATCH 06/33] feat(local): Studio edits every picture, and Settings marks the models that edit The Edit box needs only an image model, and says when a change is drawn again because the model takes no picture in; it never sends the user to the image editing models. Image models that take a picture in carry an Edits label, tooltip "Can edit pictures", and the editing models become an optional "Use a different model for edits". Translated into every shipped language. --- .../models/pictures-in/edits-badge.test.tsx | 30 ++++++++++ .../models/pictures-in/edits-badge.tsx | 45 ++++++++++++++ .../models/pictures-in/use-pictures-in.ts | 19 ++++++ .../models/remote/models/server-models.tsx | 17 ++++-- .../models/your-models/in-use-summary.tsx | 2 +- .../image-edit-models-settings.test.tsx | 4 +- .../models/image-edit-models-settings.tsx | 4 +- .../features/studio/artifact-panel.test.tsx | 60 +++++++++++++++---- .../src/features/studio/edit-model.ts | 11 ++-- .../src/features/studio/refine-box.tsx | 22 +++++-- surfsense_local/frontend/translations/de.json | 10 +++- surfsense_local/frontend/translations/en.json | 10 +++- surfsense_local/frontend/translations/es.json | 10 +++- surfsense_local/frontend/translations/fr.json | 10 +++- surfsense_local/frontend/translations/hi.json | 10 +++- surfsense_local/frontend/translations/ja.json | 10 +++- surfsense_local/frontend/translations/ko.json | 10 +++- .../frontend/translations/pt-BR.json | 10 +++- surfsense_local/frontend/translations/ru.json | 10 +++- .../frontend/translations/zh-CN.json | 10 +++- 20 files changed, 252 insertions(+), 62 deletions(-) create mode 100644 surfsense_local/frontend/src/features/models/pictures-in/edits-badge.test.tsx create mode 100644 surfsense_local/frontend/src/features/models/pictures-in/edits-badge.tsx create mode 100644 surfsense_local/frontend/src/features/models/pictures-in/use-pictures-in.ts diff --git a/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.test.tsx b/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.test.tsx new file mode 100644 index 0000000000..c29c1eeb65 --- /dev/null +++ b/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.test.tsx @@ -0,0 +1,30 @@ +import { afterEach, describe, expect, it } from "vitest" +import { cleanup, screen } from "@testing-library/react" +import userEvent from "@testing-library/user-event" + +import { render } from "@/test-utils" + +import { EditsBadge } from "./edits-badge" + +afterEach(cleanup) + +describe("edits badge", () => { + it("names itself in full, since its tooltip is only seen", () => { + render(<EditsBadge />) + + expect(screen.getByText("Edits")).toBeTruthy() + expect( + screen.getByRole("note", { name: "Edits: can edit pictures" }) + ).toBeTruthy() + }) + + it("opens its tooltip on keyboard focus as on hover", async () => { + const user = userEvent.setup() + render(<EditsBadge />) + + await user.tab() + + expect(document.activeElement?.textContent).toBe("Edits") + expect(await screen.findByText("Can edit pictures")).toBeTruthy() + }) +}) diff --git a/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.tsx b/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.tsx new file mode 100644 index 0000000000..48d39cb286 --- /dev/null +++ b/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.tsx @@ -0,0 +1,45 @@ +import { Badge } from "@/components/ui/badge" +import { + Tooltip, + TooltipContent, + TooltipTrigger, +} from "@/components/ui/tooltip" +import { intl } from "@/i18n/intl" + +/** Marks an image model that takes a picture in. Information only: any image + * model can be chosen, and one without it changes a picture by redrawing. */ +export function EditsBadge() { + const label = intl.formatMessage({ + id: "models_edits_badge_label", + defaultMessage: "Edits", + }) + const tip = intl.formatMessage({ + id: "models_edits_badge_tooltip", + defaultMessage: "Can edit pictures", + }) + return ( + <Tooltip> + <TooltipTrigger + render={ + // Focusable, so the tooltip opens from the keyboard too; named in + // full, since the tooltip is only seen. + <Badge + variant="secondary" + tabIndex={0} + role="note" + aria-label={intl.formatMessage( + { + id: "models_edits_badge_aria", + defaultMessage: "{label}: can edit pictures", + }, + { label } + )} + /> + } + > + {label} + </TooltipTrigger> + <TooltipContent side="top">{tip}</TooltipContent> + </Tooltip> + ) +} diff --git a/surfsense_local/frontend/src/features/models/pictures-in/use-pictures-in.ts b/surfsense_local/frontend/src/features/models/pictures-in/use-pictures-in.ts new file mode 100644 index 0000000000..3d18876c76 --- /dev/null +++ b/surfsense_local/frontend/src/features/models/pictures-in/use-pictures-in.ts @@ -0,0 +1,19 @@ +import { useQuery } from "@tanstack/react-query" + +import { requestJson } from "@/lib/api" + +import { MODELS_QUERY_KEY } from "../models-query" + +/** Which chosen model takes a picture in. Null: a change to a picture is drawn + * again from its prompt, by the image model. */ +export type PicturesIn = { source: "edit_model" | "image_model" | null } + +/** Under the models key, so choosing a model reads it again. */ +export function usePicturesIn({ enabled = true }: { enabled?: boolean } = {}) { + return useQuery({ + enabled, + queryKey: [...MODELS_QUERY_KEY, "pictures-in"], + queryFn: ({ signal }) => + requestJson<PicturesIn>("/llm/pictures-in", { signal }), + }) +} diff --git a/surfsense_local/frontend/src/features/models/remote/models/server-models.tsx b/surfsense_local/frontend/src/features/models/remote/models/server-models.tsx index bf54882118..3797860e2f 100644 --- a/surfsense_local/frontend/src/features/models/remote/models/server-models.tsx +++ b/surfsense_local/frontend/src/features/models/remote/models/server-models.tsx @@ -22,6 +22,7 @@ import { cn } from "@/lib/utils" import { intl } from "@/i18n/intl" import type { ModelType } from "../../model-type" +import { EditsBadge } from "../../pictures-in/edits-badge" import type { ModelSelection } from "../../selection/api" import { useSelection } from "../../selection/use-selection" import { ServerVoices } from "../voices/server-voices" @@ -373,11 +374,17 @@ export function ServerModels({ })} </Badge> ) : ( - model.types.map((type) => ( - <Badge key={type} variant="secondary"> - {TYPE_LABELS[type]()} - </Badge> - )) + model.types.map((type) => + // A picture in as well as out: said the same + // way wherever an image model is listed. + type === "image_edit" ? ( + <EditsBadge key={type} /> + ) : ( + <Badge key={type} variant="secondary"> + {TYPE_LABELS[type]()} + </Badge> + ) + ) )} {model.reads_images ? ( <Badge variant="secondary"> diff --git a/surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx b/surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx index 5ef1a9fbc6..a8343ef82e 100644 --- a/surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx +++ b/surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx @@ -47,7 +47,7 @@ export function InUseSummary({ { id: "models_in_use_summary_empty", defaultMessage: - "{slot, select, chat {No chat model chosen} image {No image model chosen} image_edit {No image editing model chosen} video {No video model chosen} audio {No audio model chosen} speech {No speech recognition model chosen} other {No model chosen}}", + "{slot, select, chat {No chat model chosen} image {No image model chosen} image_edit {Same as the image model} video {No video model chosen} audio {No audio model chosen} speech {No speech recognition model chosen} other {No model chosen}}", }, { slot } )} diff --git a/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.test.tsx b/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.test.tsx index 73ef417b50..fe3964f1fc 100644 --- a/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.test.tsx +++ b/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.test.tsx @@ -122,7 +122,9 @@ describe("image editing model settings", () => { render(<ImageEditModelsSettings onModelUnavailable={() => undefined} />) expect( - await screen.findByRole("heading", { name: "Image editing models" }) + await screen.findByRole("heading", { + name: "Use a different model for edits", + }) ).toBeTruthy() await user.click( await screen.findByRole("button", { name: "Use FLUX.2 klein 4B" }) diff --git a/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx b/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx index 40217d2efe..b2bef70eb4 100644 --- a/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx +++ b/surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx @@ -25,12 +25,12 @@ export function ImageEditModelsSettings({ <ModelSlotSettings title={intl.formatMessage({ id: "settings_image_edit_models_title", - defaultMessage: "Image editing models", + defaultMessage: "Use a different model for edits", })} description={intl.formatMessage({ id: "settings_image_edit_models_body", defaultMessage: - "The model that edits images. Run one on this computer, or use one from a server.", + "Optional. Your image model edits pictures when it can, and draws them again when it cannot. Choose a model here only to edit with a different one.", })} slot="image_edit" modelType="image_edit" diff --git a/surfsense_local/frontend/src/features/studio/artifact-panel.test.tsx b/surfsense_local/frontend/src/features/studio/artifact-panel.test.tsx index 62b11384a3..668748b3d6 100644 --- a/surfsense_local/frontend/src/features/studio/artifact-panel.test.tsx +++ b/surfsense_local/frontend/src/features/studio/artifact-panel.test.tsx @@ -909,12 +909,12 @@ describe("artifact panel image edit", () => { expect(onRefine).not.toHaveBeenCalled() }) - it("says an image editing model is needed, and links to its settings", async () => { + it("says an image model is needed when none is chosen, and links to it", async () => { const v1 = picture() vi.stubGlobal( "fetch", vi.fn(async (url: string) => - String(url).includes("/llm/selection/image_edit") + String(url).includes("/llm/selection/image_gen") ? Response.json({ detail: "no selection" }, { status: 404 }) : Response.json({ ...v1, @@ -945,9 +945,7 @@ describe("artifact panel image edit", () => { await user.click( await screen.findByRole("button", { name: "Edit this image" }) ) - expect( - await screen.findByText("Needs an image editing model.") - ).toBeTruthy() + expect(await screen.findByText("Needs an image model.")).toBeTruthy() expect( screen .getByRole("textbox", { name: "How to change this image" }) @@ -955,24 +953,23 @@ describe("artifact panel image edit", () => { ).toBe(true) await user.click(screen.getByRole("button", { name: "Choose a model" })) - expect(onOpenModelSettings).toHaveBeenCalledExactlyOnceWith( - "image-edit-models" - ) + // Never the editing models: every image model can change a picture. + expect(onOpenModelSettings).toHaveBeenCalledExactlyOnceWith("image-models") expect(onImageEdit).not.toHaveBeenCalled() }) - it("holds the instruction until it is known whether the edit model is chosen", async () => { + it("holds the instruction until it is known whether an image model is chosen", async () => { const v1 = picture() let answer = () => {} vi.stubGlobal( "fetch", vi.fn((url: string) => - String(url).includes("/llm/selection/image_edit") + String(url).includes("/llm/selection/image_gen") ? new Promise<Response>((resolve) => { answer = () => resolve( Response.json({ - model_type: "image_edit", + model_type: "image_gen", provider: "sdcpp", connection_id: null, name: "flux-2-klein-4b-Q4_0", @@ -1002,13 +999,52 @@ describe("artifact panel image edit", () => { name: "How to change this image", }) expect(field.hasAttribute("disabled")).toBe(true) - expect(screen.queryByText("Needs an image editing model.")).toBeNull() + expect(screen.queryByText("Needs an image model.")).toBeNull() answer() await vi.waitFor(() => expect(field.hasAttribute("disabled")).toBe(false)) await vi.waitFor(() => expect(document.activeElement).toBe(field)) }) + it("says a change is drawn again when the image model takes no picture in", async () => { + const v1 = picture() + vi.stubGlobal( + "fetch", + vi.fn(async (url: string) => + String(url).includes("/llm/pictures-in") + ? Response.json({ source: null }) + : Response.json({ + ...v1, + format: "summary", + content: "A lighthouse at dusk", + files: [], + quiz_state: null, + flashcard_state: null, + }) + ) + ) + const onImageEdit = vi.fn(async () => {}) + const user = userEvent.setup() + + render(panel([v1], vi.fn(), onImageEdit)) + await user.click( + await screen.findByRole("button", { name: "Edit this image" }) + ) + const field = screen.getByRole("textbox", { + name: "How to change this image", + }) + + // Plain, not a warning: the edit still works, as a redraw. + await vi.waitFor(() => + expect(field.getAttribute("placeholder")).toBe( + "Describe a change: the picture is drawn again" + ) + ) + expect(screen.queryByRole("button", { name: "Choose a model" })).toBeNull() + await user.type(field, "Make it night{Enter}") + expect(onImageEdit).toHaveBeenCalledExactlyOnceWith(40, "Make it night") + }) + it("offers no edit on a picture that is not ready", async () => { const failed = picture({ status: "failed", image_editable: false }) serveDetail(failed) diff --git a/surfsense_local/frontend/src/features/studio/edit-model.ts b/surfsense_local/frontend/src/features/studio/edit-model.ts index ea4b793a7c..cf36d5335d 100644 --- a/surfsense_local/frontend/src/features/studio/edit-model.ts +++ b/surfsense_local/frontend/src/features/studio/edit-model.ts @@ -3,20 +3,21 @@ import { useSelection } from "@/features/models/selection/use-selection" /** What the Refine box changes: a document's spec, or a picture. */ export type RefineSubject = "document" | "image" -/** Where Settings chooses the model each subject's edit needs. */ -export type EditModelSection = "chat-models" | "image-edit-models" +/** Where Settings chooses the model each subject's edit needs. Any image model + * changes a picture, by editing it or drawing it again, so a picture needs one. */ +export type EditModelSection = "chat-models" | "image-models" -const SLOT = { document: "text_gen", image: "image_edit" } as const +const SLOT = { document: "text_gen", image: "image_gen" } as const export const EDIT_MODEL_SECTION = { document: "chat-models", - image: "image-edit-models", + image: "image-models", } as const satisfies Record<RefineSubject, EditModelSection> // The codes the API refuses with when that slot is empty. export const EDIT_MODEL_CODE = { document: "needs_chat", - image: "needs_image_edit", + image: "needs_image", } as const satisfies Record<RefineSubject, string> /** Whether the model this subject's edit needs is chosen; null while that is diff --git a/surfsense_local/frontend/src/features/studio/refine-box.tsx b/surfsense_local/frontend/src/features/studio/refine-box.tsx index acf1153438..35640d4794 100644 --- a/surfsense_local/frontend/src/features/studio/refine-box.tsx +++ b/surfsense_local/frontend/src/features/studio/refine-box.tsx @@ -24,6 +24,8 @@ import { } from "@/components/ui/tooltip" import { intl } from "@/i18n/intl" +import { usePicturesIn } from "@/features/models/pictures-in/use-pictures-in" + import { useEditModelChosen, type RefineSubject } from "./edit-model" import { EditModelNotice } from "./edit-model-notice" import { messageFrom } from "./use-studio" @@ -161,7 +163,10 @@ export function RefineBox({ if (width) box.current?.style.setProperty("--pill-width", `${width + 2}px`) }, [shownVersion]) - const labels = subject === "image" ? imageLabels() : documentLabels() + const picturesIn = usePicturesIn({ enabled: subject === "image" }) + // Said only once known: an answer that is not one leaves the usual words. + const redraws = picturesIn.data?.source === null + const labels = subject === "image" ? imageLabels(redraws) : documentLabels() const openLabel = labels.open const makingLabel = (version: number) => @@ -334,7 +339,7 @@ function documentLabels() { } } -function imageLabels() { +function imageLabels(redraws: boolean) { return { open: intl.formatMessage({ id: "studio_image_edit_open_aria", @@ -344,10 +349,15 @@ function imageLabels() { id: "studio_image_edit_instruction_aria", defaultMessage: "How to change this image", }), - placeholder: intl.formatMessage({ - id: "studio_image_edit_instruction_placeholder", - defaultMessage: "Describe a change, such as a darker background", - }), + placeholder: redraws + ? intl.formatMessage({ + id: "studio_image_edit_redraw_placeholder", + defaultMessage: "Describe a change: the picture is drawn again", + }) + : intl.formatMessage({ + id: "studio_image_edit_instruction_placeholder", + defaultMessage: "Describe a change, such as a darker background", + }), submit: intl.formatMessage({ id: "studio_image_edit_submit_button", defaultMessage: "Edit", diff --git a/surfsense_local/frontend/translations/de.json b/surfsense_local/frontend/translations/de.json index ae08f9ee0e..fdc73aaf0e 100644 --- a/surfsense_local/frontend/translations/de.json +++ b/surfsense_local/frontend/translations/de.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "Die Anfrage ist fehlgeschlagen", "models_download_speech_unsupported_title": "Spracherkennungsmodelle laufen nicht auf diesem Computer", "models_download_video_unsupported_title": "Videomodelle laufen nicht auf diesem Computer", + "models_edits_badge_aria": "{label}: kann Bilder bearbeiten", + "models_edits_badge_label": "Bearbeitet", + "models_edits_badge_tooltip": "Kann Bilder bearbeiten", "models_elevenlabs_dialog_title": "ElevenLabs verbinden", "models_elevenlabs_form_cancel_button": "Abbrechen", "models_elevenlabs_form_connect_button": "ElevenLabs verbinden", @@ -942,7 +945,7 @@ "models_in_use_local_label": "Dieser Computer", "models_in_use_missing_label": "Nicht auf diesem Computer gefunden", "models_in_use_summary_aria": "{slot, select, chat {Aktives Chat-Modell} image {Aktives Bildmodell} image_edit {Aktives Bildbearbeitungsmodell} video {Aktives Videomodell} audio {Aktives Audiomodell} speech {Aktives Spracherkennungsmodell} other {Aktives Modell}}", - "models_in_use_summary_empty": "{slot, select, chat {Kein Chat-Modell gewählt} image {Kein Bildmodell gewählt} image_edit {Kein Bildbearbeitungsmodell gewählt} video {Kein Videomodell gewählt} audio {Kein Audiomodell gewählt} speech {Kein Spracherkennungsmodell gewählt} other {Kein Modell gewählt}}", + "models_in_use_summary_empty": "{slot, select, chat {Kein Chat-Modell gewählt} image {Kein Bildmodell gewählt} image_edit {Wie das Bildmodell} video {Kein Videomodell gewählt} audio {Kein Audiomodell gewählt} speech {Kein Spracherkennungsmodell gewählt} other {Kein Modell gewählt}}", "models_in_use_summary_label": "Aktiv:", "models_in_use_unknown_server_label": "Ein Server", "models_install_cancelled_short_status": "Abgebrochen", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "Wie System", "settings_general_reduce_motion_title": "Bewegung reduzieren", "settings_general_title": "Allgemein", - "settings_image_edit_models_body": "Das Modell, das Bilder bearbeitet. Führe eines auf diesem Computer aus oder nutze eines von einem Server.", - "settings_image_edit_models_title": "Modelle für Bildbearbeitung", + "settings_image_edit_models_body": "Optional. Dein Bildmodell bearbeitet Bilder, wenn es kann, und zeichnet sie sonst neu. Wähle hier nur ein Modell, wenn du mit einem anderen bearbeiten willst.", + "settings_image_edit_models_title": "Anderes Modell für Bearbeitungen", "settings_image_models_body": "Das Modell, das Bilder in Studio erstellt. Führe eines auf diesem Computer aus oder nutze eines von einem Server.", "settings_image_models_title": "Modelle für Bilderzeugung", "settings_models_add_button": "Modell hinzufügen", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "Wie dieses Bild geändert werden soll", "studio_image_edit_instruction_placeholder": "Beschreibe eine Änderung, etwa einen dunkleren Hintergrund", "studio_image_edit_open_aria": "Dieses Bild bearbeiten", + "studio_image_edit_redraw_placeholder": "Beschreibe eine Änderung: Das Bild wird neu gezeichnet", "studio_image_edit_submit_button": "Bearbeiten", "studio_mindmap_viewer_fit_aria": "Mindmap einpassen", "studio_mindmap_viewer_tree_aria": "Mindmap", diff --git a/surfsense_local/frontend/translations/en.json b/surfsense_local/frontend/translations/en.json index bf10ebac99..789f385a85 100644 --- a/surfsense_local/frontend/translations/en.json +++ b/surfsense_local/frontend/translations/en.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "The request failed", "models_download_speech_unsupported_title": "Speech recognition models cannot run on this computer", "models_download_video_unsupported_title": "Video models cannot run on this computer", + "models_edits_badge_aria": "{label}: can edit pictures", + "models_edits_badge_label": "Edits", + "models_edits_badge_tooltip": "Can edit pictures", "models_elevenlabs_dialog_title": "Connect ElevenLabs", "models_elevenlabs_form_cancel_button": "Cancel", "models_elevenlabs_form_connect_button": "Connect ElevenLabs", @@ -942,7 +945,7 @@ "models_in_use_local_label": "This computer", "models_in_use_missing_label": "Not found on this computer", "models_in_use_summary_aria": "{slot, select, chat {chat model in use} image {image model in use} image_edit {image editing model in use} video {video model in use} audio {audio model in use} speech {speech recognition model in use} other {model in use}}", - "models_in_use_summary_empty": "{slot, select, chat {No chat model chosen} image {No image model chosen} image_edit {No image editing model chosen} video {No video model chosen} audio {No audio model chosen} speech {No speech recognition model chosen} other {No model chosen}}", + "models_in_use_summary_empty": "{slot, select, chat {No chat model chosen} image {No image model chosen} image_edit {Same as the image model} video {No video model chosen} audio {No audio model chosen} speech {No speech recognition model chosen} other {No model chosen}}", "models_in_use_summary_label": "In use:", "models_in_use_unknown_server_label": "A server", "models_install_cancelled_short_status": "Cancelled", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "Match system", "settings_general_reduce_motion_title": "Reduce motion", "settings_general_title": "General", - "settings_image_edit_models_body": "The model that edits images. Run one on this computer, or use one from a server.", - "settings_image_edit_models_title": "Image editing models", + "settings_image_edit_models_body": "Optional. Your image model edits pictures when it can, and draws them again when it cannot. Choose a model here only to edit with a different one.", + "settings_image_edit_models_title": "Use a different model for edits", "settings_image_models_body": "The model that makes images in Studio. Run one on this computer, or use one from a server.", "settings_image_models_title": "Image generation models", "settings_models_add_button": "Add model", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "How to change this image", "studio_image_edit_instruction_placeholder": "Describe a change, such as a darker background", "studio_image_edit_open_aria": "Edit this image", + "studio_image_edit_redraw_placeholder": "Describe a change: the picture is drawn again", "studio_image_edit_submit_button": "Edit", "studio_mindmap_viewer_fit_aria": "Fit mind map", "studio_mindmap_viewer_tree_aria": "Mind map", diff --git a/surfsense_local/frontend/translations/es.json b/surfsense_local/frontend/translations/es.json index 4c207d2bd7..0c9d2a724a 100644 --- a/surfsense_local/frontend/translations/es.json +++ b/surfsense_local/frontend/translations/es.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "La solicitud falló", "models_download_speech_unsupported_title": "Los modelos de reconocimiento de voz no pueden ejecutarse en este equipo", "models_download_video_unsupported_title": "Los modelos de video no pueden ejecutarse en este equipo", + "models_edits_badge_aria": "{label}: puede editar imágenes", + "models_edits_badge_label": "Edita", + "models_edits_badge_tooltip": "Puede editar imágenes", "models_elevenlabs_dialog_title": "Conectar ElevenLabs", "models_elevenlabs_form_cancel_button": "Cancelar", "models_elevenlabs_form_connect_button": "Conectar ElevenLabs", @@ -942,7 +945,7 @@ "models_in_use_local_label": "Este equipo", "models_in_use_missing_label": "No se encuentra en este equipo", "models_in_use_summary_aria": "{slot, select, chat {modelo de chat en uso} image {modelo de imagen en uso} image_edit {modelo de edición de imágenes en uso} video {modelo de video en uso} audio {modelo de audio en uso} speech {modelo de reconocimiento de voz en uso} other {modelo en uso}}", - "models_in_use_summary_empty": "{slot, select, chat {Sin modelo de chat elegido} image {Sin modelo de imagen elegido} image_edit {Sin modelo de edición de imágenes elegido} video {Sin modelo de video elegido} audio {Sin modelo de audio elegido} speech {Sin modelo de reconocimiento de voz elegido} other {Sin modelo elegido}}", + "models_in_use_summary_empty": "{slot, select, chat {Sin modelo de chat elegido} image {Sin modelo de imagen elegido} image_edit {Igual que el modelo de imagen} video {Sin modelo de video elegido} audio {Sin modelo de audio elegido} speech {Sin modelo de reconocimiento de voz elegido} other {Sin modelo elegido}}", "models_in_use_summary_label": "En uso:", "models_in_use_unknown_server_label": "Un servidor", "models_install_cancelled_short_status": "Cancelado", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "Igual que el sistema", "settings_general_reduce_motion_title": "Reducir movimiento", "settings_general_title": "General", - "settings_image_edit_models_body": "El modelo que edita imágenes. Ejecuta uno en este equipo o usa uno de un servidor.", - "settings_image_edit_models_title": "Modelos de edición de imágenes", + "settings_image_edit_models_body": "Opcional. Tu modelo de imagen edita las imágenes cuando puede y, si no, las vuelve a dibujar. Elige aquí un modelo solo para editar con otro distinto.", + "settings_image_edit_models_title": "Usar otro modelo para ediciones", "settings_image_models_body": "El modelo que crea imágenes en Studio. Ejecuta uno en este equipo o usa uno de un servidor.", "settings_image_models_title": "Modelos de generación de imágenes", "settings_models_add_button": "Añadir modelo", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "Cómo cambiar esta imagen", "studio_image_edit_instruction_placeholder": "Describe un cambio, como un fondo más oscuro", "studio_image_edit_open_aria": "Editar esta imagen", + "studio_image_edit_redraw_placeholder": "Describe un cambio: la imagen se vuelve a dibujar", "studio_image_edit_submit_button": "Editar", "studio_mindmap_viewer_fit_aria": "Ajustar el mapa mental", "studio_mindmap_viewer_tree_aria": "Mapa mental", diff --git a/surfsense_local/frontend/translations/fr.json b/surfsense_local/frontend/translations/fr.json index dd780e5cf8..377fbc2970 100644 --- a/surfsense_local/frontend/translations/fr.json +++ b/surfsense_local/frontend/translations/fr.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "La requête a échoué", "models_download_speech_unsupported_title": "Les modèles de reconnaissance vocale ne peuvent pas fonctionner sur cet ordinateur", "models_download_video_unsupported_title": "Les modèles vidéo ne peuvent pas fonctionner sur cet ordinateur", + "models_edits_badge_aria": "{label} : peut modifier des images", + "models_edits_badge_label": "Modifie", + "models_edits_badge_tooltip": "Peut modifier des images", "models_elevenlabs_dialog_title": "Connecter ElevenLabs", "models_elevenlabs_form_cancel_button": "Annuler", "models_elevenlabs_form_connect_button": "Connecter ElevenLabs", @@ -942,7 +945,7 @@ "models_in_use_local_label": "Cet ordinateur", "models_in_use_missing_label": "Introuvable sur cet ordinateur", "models_in_use_summary_aria": "{slot, select, chat {modèle de chat actif} image {modèle d’image actif} image_edit {modèle de modification d’images actif} video {modèle vidéo actif} audio {modèle audio actif} speech {modèle de reconnaissance vocale actif} other {modèle actif}}", - "models_in_use_summary_empty": "{slot, select, chat {Aucun modèle de chat choisi} image {Aucun modèle d’image choisi} image_edit {Aucun modèle de modification d’images choisi} video {Aucun modèle vidéo choisi} audio {Aucun modèle audio choisi} speech {Aucun modèle de reconnaissance vocale choisi} other {Aucun modèle choisi}}", + "models_in_use_summary_empty": "{slot, select, chat {Aucun modèle de chat choisi} image {Aucun modèle d’image choisi} image_edit {Identique au modèle d’image} video {Aucun modèle vidéo choisi} audio {Aucun modèle audio choisi} speech {Aucun modèle de reconnaissance vocale choisi} other {Aucun modèle choisi}}", "models_in_use_summary_label": "Actif :", "models_in_use_unknown_server_label": "Un serveur", "models_install_cancelled_short_status": "Annulé", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "Comme le système", "settings_general_reduce_motion_title": "Réduire les animations", "settings_general_title": "Général", - "settings_image_edit_models_body": "Le modèle qui modifie les images. Exécutez-en un sur cet ordinateur ou utilisez-en un depuis un serveur.", - "settings_image_edit_models_title": "Modèles de modification d’images", + "settings_image_edit_models_body": "Facultatif. Votre modèle d’image modifie les images quand il le peut, et les redessine sinon. Choisissez un modèle ici uniquement pour modifier avec un autre.", + "settings_image_edit_models_title": "Utiliser un autre modèle pour les modifications", "settings_image_models_body": "Le modèle qui crée les images dans Studio. Exécutez-en un sur cet ordinateur ou utilisez-en un depuis un serveur.", "settings_image_models_title": "Modèles de génération d’images", "settings_models_add_button": "Ajouter un modèle", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "Comment modifier cette image", "studio_image_edit_instruction_placeholder": "Décrivez une modification, comme un arrière-plan plus sombre", "studio_image_edit_open_aria": "Modifier cette image", + "studio_image_edit_redraw_placeholder": "Décrivez une modification : l’image est redessinée", "studio_image_edit_submit_button": "Modifier", "studio_mindmap_viewer_fit_aria": "Ajuster la carte mentale", "studio_mindmap_viewer_tree_aria": "Carte mentale", diff --git a/surfsense_local/frontend/translations/hi.json b/surfsense_local/frontend/translations/hi.json index e2eb04ec80..1453bd7a08 100644 --- a/surfsense_local/frontend/translations/hi.json +++ b/surfsense_local/frontend/translations/hi.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "अनुरोध विफल रहा", "models_download_speech_unsupported_title": "स्पीच रिकग्निशन मॉडल इस कंप्यूटर पर नहीं चल सकते", "models_download_video_unsupported_title": "वीडियो मॉडल इस कंप्यूटर पर नहीं चल सकते", + "models_edits_badge_aria": "{label}: इमेज एडिट कर सकता है", + "models_edits_badge_label": "एडिटिंग", + "models_edits_badge_tooltip": "इमेज एडिट कर सकता है", "models_elevenlabs_dialog_title": "ElevenLabs कनेक्ट करें", "models_elevenlabs_form_cancel_button": "रद्द करें", "models_elevenlabs_form_connect_button": "ElevenLabs कनेक्ट करें", @@ -942,7 +945,7 @@ "models_in_use_local_label": "यह कंप्यूटर", "models_in_use_missing_label": "इस कंप्यूटर पर नहीं मिला", "models_in_use_summary_aria": "{slot, select, chat {उपयोग में चैट मॉडल} image {उपयोग में इमेज मॉडल} image_edit {उपयोग में इमेज एडिटिंग मॉडल} video {उपयोग में वीडियो मॉडल} audio {उपयोग में ऑडियो मॉडल} speech {उपयोग में स्पीच रिकग्निशन मॉडल} other {उपयोग में मॉडल}}", - "models_in_use_summary_empty": "{slot, select, chat {कोई चैट मॉडल नहीं चुना गया} image {कोई इमेज मॉडल नहीं चुना गया} image_edit {कोई इमेज एडिटिंग मॉडल नहीं चुना गया} video {कोई वीडियो मॉडल नहीं चुना गया} audio {कोई ऑडियो मॉडल नहीं चुना गया} speech {कोई स्पीच रिकग्निशन मॉडल नहीं चुना गया} other {कोई मॉडल नहीं चुना गया}}", + "models_in_use_summary_empty": "{slot, select, chat {कोई चैट मॉडल नहीं चुना गया} image {कोई इमेज मॉडल नहीं चुना गया} image_edit {इमेज मॉडल जैसा ही} video {कोई वीडियो मॉडल नहीं चुना गया} audio {कोई ऑडियो मॉडल नहीं चुना गया} speech {कोई स्पीच रिकग्निशन मॉडल नहीं चुना गया} other {कोई मॉडल नहीं चुना गया}}", "models_in_use_summary_label": "उपयोग में:", "models_in_use_unknown_server_label": "एक सर्वर", "models_install_cancelled_short_status": "रद्द किया गया", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "सिस्टम के अनुसार", "settings_general_reduce_motion_title": "मोशन कम करें", "settings_general_title": "सामान्य", - "settings_image_edit_models_body": "वह मॉडल जो इमेज एडिट करता है। इसे इस कंप्यूटर पर चलाएं, या किसी सर्वर से इस्तेमाल करें।", - "settings_image_edit_models_title": "इमेज एडिटिंग मॉडल", + "settings_image_edit_models_body": "वैकल्पिक। आपका इमेज मॉडल जब कर सकता है तब इमेज एडिट करता है, और नहीं कर सकता तो उन्हें फिर से बनाता है। यहाँ मॉडल तभी चुनें जब आप किसी दूसरे मॉडल से एडिट करना चाहें।", + "settings_image_edit_models_title": "एडिट के लिए दूसरा मॉडल इस्तेमाल करें", "settings_image_models_body": "वह मॉडल जो Studio में इमेज बनाता है। इसे इस कंप्यूटर पर चलाएं, या किसी सर्वर से इस्तेमाल करें।", "settings_image_models_title": "इमेज जनरेशन मॉडल", "settings_models_add_button": "मॉडल जोड़ें", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "इस इमेज को कैसे बदलें", "studio_image_edit_instruction_placeholder": "कोई बदलाव बताएं, जैसे गहरा बैकग्राउंड", "studio_image_edit_open_aria": "इस इमेज को एडिट करें", + "studio_image_edit_redraw_placeholder": "बदलाव बताएं: इमेज फिर से बनाई जाएगी", "studio_image_edit_submit_button": "एडिट करें", "studio_mindmap_viewer_fit_aria": "माइंड मैप फ़िट करें", "studio_mindmap_viewer_tree_aria": "माइंड मैप", diff --git a/surfsense_local/frontend/translations/ja.json b/surfsense_local/frontend/translations/ja.json index 5649f0c6dc..f4eb1a0f4a 100644 --- a/surfsense_local/frontend/translations/ja.json +++ b/surfsense_local/frontend/translations/ja.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "リクエストに失敗しました", "models_download_speech_unsupported_title": "このコンピューターでは音声認識モデルを実行できません", "models_download_video_unsupported_title": "このコンピューターでは動画モデルを実行できません", + "models_edits_badge_aria": "{label}:画像を編集できます", + "models_edits_badge_label": "編集可", + "models_edits_badge_tooltip": "画像を編集できます", "models_elevenlabs_dialog_title": "ElevenLabsに接続", "models_elevenlabs_form_cancel_button": "キャンセル", "models_elevenlabs_form_connect_button": "ElevenLabsに接続", @@ -942,7 +945,7 @@ "models_in_use_local_label": "このコンピューター", "models_in_use_missing_label": "このコンピューターに見つかりません", "models_in_use_summary_aria": "{slot, select, chat {使用中のチャットモデル} image {使用中の画像モデル} image_edit {使用中の画像編集モデル} video {使用中の動画モデル} audio {使用中の音声モデル} speech {使用中の音声認識モデル} other {使用中のモデル}}", - "models_in_use_summary_empty": "{slot, select, chat {チャットモデルが選択されていません} image {画像モデルが選択されていません} image_edit {画像編集モデルが選択されていません} video {動画モデルが選択されていません} audio {音声モデルが選択されていません} speech {音声認識モデルが選択されていません} other {モデルが選択されていません}}", + "models_in_use_summary_empty": "{slot, select, chat {チャットモデルが選択されていません} image {画像モデルが選択されていません} image_edit {画像モデルと同じ} video {動画モデルが選択されていません} audio {音声モデルが選択されていません} speech {音声認識モデルが選択されていません} other {モデルが選択されていません}}", "models_in_use_summary_label": "使用中:", "models_in_use_unknown_server_label": "サーバー", "models_install_cancelled_short_status": "キャンセル済み", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "システムに合わせる", "settings_general_reduce_motion_title": "動きを減らす", "settings_general_title": "一般", - "settings_image_edit_models_body": "画像を編集するモデルです。このコンピューターで実行するか、サーバー上のモデルを使います。", - "settings_image_edit_models_title": "画像編集モデル", + "settings_image_edit_models_body": "任意です。画像モデルは、可能な場合は画像を編集し、できない場合は描き直します。別のモデルで編集したい場合にのみ、ここでモデルを選択してください。", + "settings_image_edit_models_title": "編集に別のモデルを使う", "settings_image_models_body": "Studioで画像を生成するモデルです。このコンピューターで実行するか、サーバー上のモデルを使います。", "settings_image_models_title": "画像生成モデル", "settings_models_add_button": "モデルを追加", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "この画像の変更内容", "studio_image_edit_instruction_placeholder": "変更内容を入力してください(例: 背景を暗くする)", "studio_image_edit_open_aria": "この画像を編集", + "studio_image_edit_redraw_placeholder": "変更内容を入力してください(画像は描き直されます)", "studio_image_edit_submit_button": "編集", "studio_mindmap_viewer_fit_aria": "マインドマップ全体を表示", "studio_mindmap_viewer_tree_aria": "マインドマップ", diff --git a/surfsense_local/frontend/translations/ko.json b/surfsense_local/frontend/translations/ko.json index 3adb0897b6..e9a2dc8678 100644 --- a/surfsense_local/frontend/translations/ko.json +++ b/surfsense_local/frontend/translations/ko.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "요청이 실패했습니다", "models_download_speech_unsupported_title": "이 컴퓨터에서는 음성 인식 모델을 실행할 수 없습니다", "models_download_video_unsupported_title": "이 컴퓨터에서는 동영상 모델을 실행할 수 없습니다", + "models_edits_badge_aria": "{label}: 이미지를 편집할 수 있습니다", + "models_edits_badge_label": "편집 가능", + "models_edits_badge_tooltip": "이미지를 편집할 수 있습니다", "models_elevenlabs_dialog_title": "ElevenLabs 연결", "models_elevenlabs_form_cancel_button": "취소", "models_elevenlabs_form_connect_button": "ElevenLabs 연결", @@ -942,7 +945,7 @@ "models_in_use_local_label": "이 컴퓨터", "models_in_use_missing_label": "이 컴퓨터에서 찾을 수 없음", "models_in_use_summary_aria": "{slot, select, chat {사용 중인 채팅 모델} image {사용 중인 이미지 모델} image_edit {사용 중인 이미지 편집 모델} video {사용 중인 동영상 모델} audio {사용 중인 음성 모델} speech {사용 중인 음성 인식 모델} other {사용 중인 모델}}", - "models_in_use_summary_empty": "{slot, select, chat {선택된 채팅 모델이 없습니다} image {선택된 이미지 모델이 없습니다} image_edit {선택된 이미지 편집 모델이 없습니다} video {선택된 동영상 모델이 없습니다} audio {선택된 음성 모델이 없습니다} speech {선택된 음성 인식 모델이 없습니다} other {선택된 모델이 없습니다}}", + "models_in_use_summary_empty": "{slot, select, chat {선택된 채팅 모델이 없습니다} image {선택된 이미지 모델이 없습니다} image_edit {이미지 모델과 동일} video {선택된 동영상 모델이 없습니다} audio {선택된 음성 모델이 없습니다} speech {선택된 음성 인식 모델이 없습니다} other {선택된 모델이 없습니다}}", "models_in_use_summary_label": "사용 중:", "models_in_use_unknown_server_label": "서버", "models_install_cancelled_short_status": "취소됨", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "시스템에 맞춤", "settings_general_reduce_motion_title": "동작 줄이기", "settings_general_title": "일반", - "settings_image_edit_models_body": "이미지를 편집하는 모델입니다. 이 컴퓨터에서 실행하거나, 서버의 모델을 사용하세요.", - "settings_image_edit_models_title": "이미지 편집 모델", + "settings_image_edit_models_body": "선택 사항입니다. 이미지 모델은 가능하면 이미지를 편집하고, 그렇지 않으면 다시 그립니다. 다른 모델로 편집하려는 경우에만 여기에서 모델을 선택하세요.", + "settings_image_edit_models_title": "편집에 다른 모델 사용", "settings_image_models_body": "Studio에서 이미지를 만드는 모델입니다. 이 컴퓨터에서 실행하거나, 서버의 모델을 사용하세요.", "settings_image_models_title": "이미지 생성 모델", "settings_models_add_button": "모델 추가", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "이 이미지를 바꿀 방법", "studio_image_edit_instruction_placeholder": "배경을 어둡게 하는 등 바꿀 내용을 설명하세요", "studio_image_edit_open_aria": "이 이미지 편집", + "studio_image_edit_redraw_placeholder": "변경 사항을 입력하세요. 이미지가 다시 그려집니다", "studio_image_edit_submit_button": "편집", "studio_mindmap_viewer_fit_aria": "마인드맵 맞춤", "studio_mindmap_viewer_tree_aria": "마인드맵", diff --git a/surfsense_local/frontend/translations/pt-BR.json b/surfsense_local/frontend/translations/pt-BR.json index 5b1c12dc89..f88b1e6472 100644 --- a/surfsense_local/frontend/translations/pt-BR.json +++ b/surfsense_local/frontend/translations/pt-BR.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "A solicitação falhou", "models_download_speech_unsupported_title": "Modelos de reconhecimento de voz não podem ser executados neste computador", "models_download_video_unsupported_title": "Modelos de vídeo não podem ser executados neste computador", + "models_edits_badge_aria": "{label}: pode editar imagens", + "models_edits_badge_label": "Edita", + "models_edits_badge_tooltip": "Pode editar imagens", "models_elevenlabs_dialog_title": "Conectar ElevenLabs", "models_elevenlabs_form_cancel_button": "Cancelar", "models_elevenlabs_form_connect_button": "Conectar ElevenLabs", @@ -942,7 +945,7 @@ "models_in_use_local_label": "Este computador", "models_in_use_missing_label": "Não encontrado neste computador", "models_in_use_summary_aria": "{slot, select, chat {modelo de chat em uso} image {modelo de imagem em uso} image_edit {modelo de edição de imagem em uso} video {modelo de vídeo em uso} audio {modelo de áudio em uso} speech {modelo de reconhecimento de voz em uso} other {modelo em uso}}", - "models_in_use_summary_empty": "{slot, select, chat {Nenhum modelo de chat escolhido} image {Nenhum modelo de imagem escolhido} image_edit {Nenhum modelo de edição de imagem escolhido} video {Nenhum modelo de vídeo escolhido} audio {Nenhum modelo de áudio escolhido} speech {Nenhum modelo de reconhecimento de voz escolhido} other {Nenhum modelo escolhido}}", + "models_in_use_summary_empty": "{slot, select, chat {Nenhum modelo de chat escolhido} image {Nenhum modelo de imagem escolhido} image_edit {Igual ao modelo de imagem} video {Nenhum modelo de vídeo escolhido} audio {Nenhum modelo de áudio escolhido} speech {Nenhum modelo de reconhecimento de voz escolhido} other {Nenhum modelo escolhido}}", "models_in_use_summary_label": "Em uso:", "models_in_use_unknown_server_label": "Um servidor", "models_install_cancelled_short_status": "Cancelado", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "Igual ao sistema", "settings_general_reduce_motion_title": "Reduzir movimento", "settings_general_title": "Geral", - "settings_image_edit_models_body": "O modelo que edita imagens. Execute um neste computador ou use um de um servidor.", - "settings_image_edit_models_title": "Modelos de edição de imagem", + "settings_image_edit_models_body": "Opcional. Seu modelo de imagem edita as imagens quando consegue e, quando não consegue, desenha de novo. Escolha um modelo aqui só para editar com outro.", + "settings_image_edit_models_title": "Usar outro modelo para edições", "settings_image_models_body": "O modelo que cria imagens no Studio. Execute um neste computador ou use um de um servidor.", "settings_image_models_title": "Modelos de geração de imagem", "settings_models_add_button": "Adicionar modelo", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "Como alterar esta imagem", "studio_image_edit_instruction_placeholder": "Descreva uma alteração, como um fundo mais escuro", "studio_image_edit_open_aria": "Editar esta imagem", + "studio_image_edit_redraw_placeholder": "Descreva uma mudança: a imagem será desenhada de novo", "studio_image_edit_submit_button": "Editar", "studio_mindmap_viewer_fit_aria": "Ajustar o mapa mental", "studio_mindmap_viewer_tree_aria": "Mapa mental", diff --git a/surfsense_local/frontend/translations/ru.json b/surfsense_local/frontend/translations/ru.json index eaefde430c..6d3af6ac51 100644 --- a/surfsense_local/frontend/translations/ru.json +++ b/surfsense_local/frontend/translations/ru.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "Запрос не выполнен", "models_download_speech_unsupported_title": "Модели распознавания речи не запускаются на этом компьютере", "models_download_video_unsupported_title": "Модели для видео не запускаются на этом компьютере", + "models_edits_badge_aria": "{label}: может редактировать изображения", + "models_edits_badge_label": "Редактирует", + "models_edits_badge_tooltip": "Может редактировать изображения", "models_elevenlabs_dialog_title": "Подключить ElevenLabs", "models_elevenlabs_form_cancel_button": "Отмена", "models_elevenlabs_form_connect_button": "Подключить ElevenLabs", @@ -942,7 +945,7 @@ "models_in_use_local_label": "Этот компьютер", "models_in_use_missing_label": "Не найдена на этом компьютере", "models_in_use_summary_aria": "{slot, select, chat {используемая модель для чата} image {используемая модель для изображений} image_edit {используемая модель для редактирования изображений} video {используемая модель для видео} audio {используемая модель для аудио} speech {используемая модель распознавания речи} other {используемая модель}}", - "models_in_use_summary_empty": "{slot, select, chat {Модель для чата не выбрана} image {Модель для изображений не выбрана} image_edit {Модель для редактирования изображений не выбрана} video {Модель для видео не выбрана} audio {Модель для аудио не выбрана} speech {Модель распознавания речи не выбрана} other {Модель не выбрана}}", + "models_in_use_summary_empty": "{slot, select, chat {Модель для чата не выбрана} image {Модель для изображений не выбрана} image_edit {Как у модели изображений} video {Модель для видео не выбрана} audio {Модель для аудио не выбрана} speech {Модель распознавания речи не выбрана} other {Модель не выбрана}}", "models_in_use_summary_label": "Используется:", "models_in_use_unknown_server_label": "Сервер", "models_install_cancelled_short_status": "Отменено", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "Как в системе", "settings_general_reduce_motion_title": "Уменьшить движение", "settings_general_title": "Общие", - "settings_image_edit_models_body": "Модель, которая редактирует изображения. Запустите её на этом компьютере или используйте модель с сервера.", - "settings_image_edit_models_title": "Модели редактирования изображений", + "settings_image_edit_models_body": "Необязательно. Ваша модель изображений редактирует изображения, когда может, а иначе рисует их заново. Выбирайте модель здесь, только чтобы редактировать другой моделью.", + "settings_image_edit_models_title": "Другая модель для редактирования", "settings_image_models_body": "Модель, которая создаёт изображения в Studio. Запустите её на этом компьютере или используйте модель с сервера.", "settings_image_models_title": "Модели генерации изображений", "settings_models_add_button": "Добавить модель", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "Как изменить это изображение", "studio_image_edit_instruction_placeholder": "Опишите изменение, например более тёмный фон", "studio_image_edit_open_aria": "Изменить изображение", + "studio_image_edit_redraw_placeholder": "Опишите изменение: изображение будет нарисовано заново", "studio_image_edit_submit_button": "Изменить", "studio_mindmap_viewer_fit_aria": "Вписать интеллект-карту", "studio_mindmap_viewer_tree_aria": "Интеллект-карта", diff --git a/surfsense_local/frontend/translations/zh-CN.json b/surfsense_local/frontend/translations/zh-CN.json index 59e16c69fe..a2501de943 100644 --- a/surfsense_local/frontend/translations/zh-CN.json +++ b/surfsense_local/frontend/translations/zh-CN.json @@ -897,6 +897,9 @@ "models_download_speech_request_error": "请求失败", "models_download_speech_unsupported_title": "这台电脑无法运行语音识别模型", "models_download_video_unsupported_title": "这台电脑无法运行视频模型", + "models_edits_badge_aria": "{label}:可以编辑图像", + "models_edits_badge_label": "可编辑", + "models_edits_badge_tooltip": "可以编辑图像", "models_elevenlabs_dialog_title": "连接 ElevenLabs", "models_elevenlabs_form_cancel_button": "取消", "models_elevenlabs_form_connect_button": "连接 ElevenLabs", @@ -942,7 +945,7 @@ "models_in_use_local_label": "这台电脑", "models_in_use_missing_label": "在这台电脑上找不到", "models_in_use_summary_aria": "{slot, select, chat {使用中的对话模型} image {使用中的图像模型} image_edit {使用中的图像编辑模型} video {使用中的视频模型} audio {使用中的语音模型} speech {使用中的语音识别模型} other {使用中的模型}}", - "models_in_use_summary_empty": "{slot, select, chat {未选择对话模型} image {未选择图像模型} image_edit {未选择图像编辑模型} video {未选择视频模型} audio {未选择语音模型} speech {未选择语音识别模型} other {未选择模型}}", + "models_in_use_summary_empty": "{slot, select, chat {未选择对话模型} image {未选择图像模型} image_edit {与图像模型相同} video {未选择视频模型} audio {未选择语音模型} speech {未选择语音识别模型} other {未选择模型}}", "models_in_use_summary_label": "使用中:", "models_in_use_unknown_server_label": "某个服务器", "models_install_cancelled_short_status": "已取消", @@ -1586,8 +1589,8 @@ "settings_general_reduce_motion_system_label": "跟随系统", "settings_general_reduce_motion_title": "减弱动态效果", "settings_general_title": "通用", - "settings_image_edit_models_body": "编辑图像的模型。可以在这台电脑上运行一个,也可以使用服务器上的。", - "settings_image_edit_models_title": "图像编辑模型", + "settings_image_edit_models_body": "可选。你的图像模型在可以时编辑图像,不能时重新绘制。只有想用其他模型编辑时,才需要在这里选择模型。", + "settings_image_edit_models_title": "使用其他模型进行编辑", "settings_image_models_body": "在 Studio 中生成图像的模型。可以在这台电脑上运行一个,也可以使用服务器上的。", "settings_image_models_title": "图像生成模型", "settings_models_add_button": "添加模型", @@ -1994,6 +1997,7 @@ "studio_image_edit_instruction_aria": "如何修改此图像", "studio_image_edit_instruction_placeholder": "描述一处修改,例如让背景更暗", "studio_image_edit_open_aria": "编辑此图像", + "studio_image_edit_redraw_placeholder": "描述更改:图像将重新绘制", "studio_image_edit_submit_button": "编辑", "studio_mindmap_viewer_fit_aria": "缩放以适应思维导图", "studio_mindmap_viewer_tree_aria": "思维导图", From 1d01695e716bff10bcad5bb9699091666b8b05ac Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:51:34 +0530 Subject: [PATCH 07/33] docs: pictures any image model edits, comics, and documents that wait for their pictures Studio and the agent describe what is now built; ADR 0089 records that any image model changes a picture, amending ADR 0077, and ADR 0090 that a document waits on the server for its pictures, amending ADR 0087. The user docs say what Edit does on each kind of model and how the agent makes a comic. The proposal stays in progress, with where the build differs and what is left. --- .../0077-an-image-edit-is-the-next-version.md | 2 +- ...es-pictures-with-its-own-surfsense-tool.md | 2 +- .../0089-any-image-model-changes-a-picture.md | 23 +++++++++++ ...nt-waits-on-the-server-for-its-pictures.md | 22 +++++++++++ docs/adr/README.md | 2 + docs/architecture/agent.md | 14 ++++--- docs/architecture/local-models/selection.md | 2 +- docs/architecture/studio.md | 39 +++++++++++++++---- docs/proposals/pictures/README.md | 18 ++++++++- www/content/docs/v2/agent.mdx | 8 ++-- www/content/docs/v2/studio.mdx | 6 +-- 11 files changed, 115 insertions(+), 23 deletions(-) create mode 100644 docs/adr/0089-any-image-model-changes-a-picture.md create mode 100644 docs/adr/0090-a-document-waits-on-the-server-for-its-pictures.md diff --git a/docs/adr/0077-an-image-edit-is-the-next-version.md b/docs/adr/0077-an-image-edit-is-the-next-version.md index 60423380cc..12d1700208 100644 --- a/docs/adr/0077-an-image-edit-is-the-next-version.md +++ b/docs/adr/0077-an-image-edit-is-the-next-version.md @@ -1,6 +1,6 @@ # ADR 0077: An image edit is the next version of the same document -- **Status:** Accepted +- **Status:** Accepted; its consequence on models that cannot edit superseded by [ADR 0089](0089-any-image-model-changes-a-picture.md) - **Date:** 2026-10-31 - **Supersedes:** the removal of Studio's earlier image Edit, which made a new artifact per edit - **Source:** maintainer decision, 31 Oct 2026 (no permalink) diff --git a/docs/adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md b/docs/adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md index 5c43055cdb..77e9d1b9ff 100644 --- a/docs/adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md +++ b/docs/adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md @@ -1,6 +1,6 @@ # ADR 0087: The agent makes pictures with its own SurfSense tool, not opencode's -- **Status:** Accepted +- **Status:** Accepted; its waiting for the user's next message superseded by [ADR 0090](0090-a-document-waits-on-the-server-for-its-pictures.md) - **Date:** 2026-11-07 - **Amends:** [ADR 0066](0066-web-pages-are-html-the-model-writes-without-network.md), in part: its rule that nothing is written into a page but the policy - **Source:** maintainer decision, 7 Nov 2026 (no permalink) diff --git a/docs/adr/0089-any-image-model-changes-a-picture.md b/docs/adr/0089-any-image-model-changes-a-picture.md new file mode 100644 index 0000000000..991e398c91 --- /dev/null +++ b/docs/adr/0089-any-image-model-changes-a-picture.md @@ -0,0 +1,23 @@ +# ADR 0089: Any image model changes a picture: one that takes pictures edits it, any other draws it again + +- **Status:** Accepted +- **Date:** 2026-11-10 +- **Supersedes:** in part, [ADR 0077](0077-an-image-edit-is-the-next-version.md): its consequence that a picture drawn by a model that cannot edit needs an edit model chosen before it can be edited +- **Source:** maintainer decision, 10 Nov 2026 (no permalink) + +## Context + +An edit ran only on the `image_edit` selection, which few local models can fill and most users never chose; an empty slot answered "Needs an image editing model" and sent them to Settings. Most remote image models take a picture in, but the image model the user had chosen was never asked. OpenRouter has no `/images/edits`, so an edit through it failed. A comic needs every page to keep page 1's characters, which needs a picture in, not a second model. + +## Decision + +- Whether a model takes a picture in is a fact about the model, read from its connection's model list or the catalog when it is chosen, not a slot the user fills. +- One rule picks the model for a picture in: the `image_edit` selection when chosen, as an override; else the image model when it takes pictures; else none. +- With none, an edit is drawn again by the image model from the prompt the picture was drawn with and the change, and says so in one plain line. No edit is refused, and nothing in Studio or chat points to Settings for it. +- A picture in goes to `/images/edits`, or to `/images` with `input_references` where a server has no edit route, so OpenRouter edits too. + +## Consequences + +- Most users choose one image model and edit with it; the edit slot is optional. +- A redraw does not keep the base's composition; the version's first line and the Edit box's placeholder say so. +- Every drawing keeps the prompt it sent, so a redraw has it, and changes add up from version to version. diff --git a/docs/adr/0090-a-document-waits-on-the-server-for-its-pictures.md b/docs/adr/0090-a-document-waits-on-the-server-for-its-pictures.md new file mode 100644 index 0000000000..f50784e060 --- /dev/null +++ b/docs/adr/0090-a-document-waits-on-the-server-for-its-pictures.md @@ -0,0 +1,22 @@ +# ADR 0090: A document waits on the server for the pictures it places + +- **Status:** Accepted +- **Date:** 2026-11-10 +- **Supersedes:** in part, [ADR 0087](0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md): "A document needing a picture still being drawn waits for the user's next message" +- **Source:** maintainer decision, 10 Nov 2026 (no permalink) + +## Context + +A local picture on a processor takes minutes, past a tool call's limit, so its call answers queued. A render naming it was refused, and the agent could only tell the user to ask again: a four-page comic drawn on this computer never became one PDF on its own. A tool that polls, or a turn kept open, would hold a call past its limit, which ADR 0087 rules out. + +## Decision + +- A render naming a picture still being drawn is made pending with the pictures it waits for, unqueued, and its call answers at once. +- The end of every Studio job queues the renders waiting on that picture once all of theirs are drawn, and fails one whose picture failed, was cancelled or deleted, with a sentence naming it. The worker does the same when it starts. +- Only a render waits. A web page still refuses a picture being drawn, since it is checked and screenshotted as it is saved. + +## Consequences + +- A comic drawn slowly ends as a PDF without the user asking again. +- No process, timer or model stays awake for it: the job that ends is the one event that can make the render ready. +- No one looks at a waiting render's pages, since it finishes after its turn; a comic's PDF is a fixed layout. diff --git a/docs/adr/README.md b/docs/adr/README.md index 7db13256de..aaa7aa90a5 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -117,3 +117,5 @@ The Date is the day the decision was first written down. Source links are permal | 0086 | [The contract redline ships with the Business switch, as an app runner on SurfSense's own Word engine](0086-the-contract-redline-ships-with-the-business-switch-on-the-word-engine.md) | Accepted | | 0087 | [The agent makes pictures with its own SurfSense tool, not opencode's](0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md) | Accepted | | 0088 | [The app draws Word and Excel files with one renderer](0088-the-app-draws-word-and-excel-files-with-one-renderer.md) | Accepted | +| 0089 | [Any image model changes a picture: one that takes pictures edits it, any other draws it again](0089-any-image-model-changes-a-picture.md) | Accepted | +| 0090 | [A document waits on the server for the pictures it places](0090-a-document-waits-on-the-server-for-its-pictures.md) | Accepted | diff --git a/docs/architecture/agent.md b/docs/architecture/agent.md index 9c36c66e42..7582bf4c8e 100644 --- a/docs/architecture/agent.md +++ b/docs/architecture/agent.md @@ -217,13 +217,17 @@ The step links the version in the thread as a render's does (`surfsense_save_pag [`make_image.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/make_image.py) draws a picture from a prompt the agent wrote, with no sources and no chat model ([ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)), through Studio's job ([drawn pictures](studio.md#pictures-the-agent-asks-for)). Within one 190-second deadline it: -1. **Refuses** a missing argument, a prompt over 32,000 characters, only to catch a runaway model, no image model or, for an edit, no edit model, no embedding model, a title this chat already gave a picture of that style, and an `artifact_id` this chat did not make. None is a failed run. +1. **Refuses** a missing argument, a prompt over 32,000 characters, only to catch a runaway model, no image model, no embedding model, a title this chat already gave a picture of that style, an `artifact_id` this chat did not make, and a reference that is not a ready picture of this chat, more than four, or given with `artifact_id`. None is a failed run. 2. **Waits** up to 150 seconds. A failed picture is a failed run, under its own three-failure limit and never retried by the worker, since each try may be billed. A queued picture that fails after its call answered counts when the next one is asked for. Calls in one step are held apart while each checks its title and starts, so two with one title draw one picture. -3. **Answers** with the picture's name, and the picture as a tool result's image when the model reads images. It offers an edit only when an edit model is chosen; without one, it tells the agent to say what is wrong and point the user to Settings › Image edit, so the agent neither calls an edit that would be refused nor bills a redraw. A picture in a format documents cannot place, SVG or GIF, gets no name and says so. +3. **Answers** with the picture's name, where it was drawn ("Drawn on this computer" or "Drawn by <connection>"), and the picture as a tool result's image when the model reads images. It always offers a change: with no model that takes a picture in ([which model](studio.md#which-model-takes-a-picture)), it says the change is drawn again from the prompt; it never points to Settings. A reference drawn from words alone, or a change redrawn, says so in a line. A picture in a format documents cannot place, SVG or GIF, gets no name and says so. + +**Shape and references.** `shape` (`square`, `portrait`, `landscape`) reaches the image model ([shapes](studio.md#shapes-and-references)). `references`, up to four picture names of this chat, send those pictures with the prompt to the model that takes pictures, so a comic's later pages keep page 1's characters; with none, the picture is drawn from its prompt and the result says the characters may differ. The tool's listing stays the same whatever models are chosen, so a local model's prompt cache holds. + +**Comics and sets.** The `surfsense-pictures` skill ([`skills/surfsense-pictures/`](../../surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md)) is the craft beside the tool, as `surfsense-documents` is beside the render: one picture, or a set such as a comic, a storyboard or a picture book. A set is planned with a character sheet, confirmed once, drawn first page alone and the rest with it as reference, and bound into one PDF by the script in `pdf.md`: one full-bleed picture a page, placed as JPEG, with a page's exact lines set in a strip under it when its lettering came out wrong or it was drawn without words. The prompt sends pictures and comics there, and the web page skill's description sends them away. **Prompt length.** As in Studio, nothing limits a prompt per model: the tool's description asks for one self-contained prompt, as short as it says what to draw, and a model's own refusal comes back as a failed picture. -**A slow picture.** A local picture on a processor takes minutes, past a tool call's hard limit, so the call answers `Queued artifact …` with the name it will have, and no tool polls. With a remote chat model the agent answers at once, and a render or page naming the picture is refused as still being drawn until it is ready. With a local chat model, the agent's next step waits in the model endpoint behind [the image's hold](local-models/admission.md#the-image-jobs-hold), so the picture is usually ready when the agent goes on. +**A slow picture.** A local picture on a processor takes minutes, past a tool call's hard limit, so the call answers `Queued artifact …` with the name it will have, and no tool polls. A render naming it is not refused: it is made pending with `waits_for`, unqueued, and answers at once that it renders once the picture is drawn ([`waiting_renders/`](../../surfsense_local/backend/modules/artifacts/waiting_renders/)). The end of every Studio job queues the renders waiting on that picture once all theirs are ready, or fails one whose picture failed, was cancelled or deleted, with a sentence naming it; one update takes `waits_for` off, so two pictures ending together queue it once. The Studio worker does the same when it starts, after failing interrupted jobs. A web page still refuses a picture being drawn, since it is checked and screenshotted as it is saved. With a local chat model, the agent's next step waits in the model endpoint behind [the image's hold](local-models/admission.md#the-image-jobs-hold), so the picture is usually ready when the agent goes on. **Placing one.** `image-<artifact id>` names one version, so it keeps meaning the same picture as edits are made. It works in a render's `images` and in a page ([`names.py`](../../surfsense_local/backend/modules/artifacts/drawn_images/names.py)), only in the chat that made the picture, as a chart's name does. Finding it reads the database; each use then writes it afresh as a PNG to `outputs/images/<name>.png`, outside any transaction, converted if the model answered in another format and refused past 40 million pixels, since the agent can write to its own outputs and what is placed must be what the model drew. A page's size is worked out before any image is written into it, since a small file can name one picture thousands of times. @@ -372,8 +376,8 @@ A document script is another matter: it runs in the Studio worker's runner, whic - The model endpoint neither simplifies tool schemas to what llama.cpp's grammar takes nor turns a tool call a model writes as text into a real one, so small local models stall where a remote one would not. - A file the agent writes to `outputs/` with its own tools does not become an artifact; only `surfsense_render_document`, `surfsense_revise_document`, `surfsense_convert_document`, `surfsense_make_image` and the PDF tools make one. An analysis's tables and charts stay in `outputs/analysis/` too, outside Studio, until a document places them. - An analysis waits on the Studio worker's four threads. Behind four long Studio jobs it is dropped after 60 seconds unstarted, and the call waits out its 190 seconds before saying so. -- A picture still being drawn when its call answers leaves the agent no way to wait or come back: a document that needs it is made on the user's next message. A local chat model's step waiting out a whole local picture through the agent's model endpoint is not measured. -- Lettering an image model draws is not measured, so one picture holding a comic's speech bubbles or a page of text may garble them. The tool's description sends exact words to the document around the picture. +- A local chat model's step waiting out a whole local picture through the agent's model endpoint is not measured. A render that waits on a picture finishes after its turn, so no one looks at its pages. +- A waiting render that fails does not count toward renders' three-failure stop. - A Studio job the agent starts with `surfsense_create_artifact` runs on the same local model as the agent, so the agent's next step waits behind it. - An artifact the agent starts with `surfsense_create_artifact` records neither the thread nor the step that made it, and one it renders records its thread but not the step: the tool call reaches SurfSense without the step. - A thread's folder narrows what its agent can reach to its own sources, but does not close it. Through a hole in opencode, below, an agent can still read what is outside its folder: opencode's tool-output folder is shared by all, and closing it needs a change in opencode. Through SurfSense's own tools it can too: a script `surfsense_render_document` or `surfsense_analyze_data` runs can read any file the user can ([Studio](studio.md#known-gaps)), and the text the render returns shows what it put in the document, another thread's sources included; and `surfsense_read_document` reads any document the agent rendered in the workspace, from any thread and filed in an unticked folder too, and such a script can hold what that thread's agent took from its own sources. diff --git a/docs/architecture/local-models/selection.md b/docs/architecture/local-models/selection.md index 25b611c7a3..16225b7bdc 100644 --- a/docs/architecture/local-models/selection.md +++ b/docs/architecture/local-models/selection.md @@ -22,7 +22,7 @@ type is the slot. |---|---|---|---| | `text_gen` | `llamacpp`, the bundled runtime | `openai_compatible`, with a `connection_id` | chat, titles, Studio's writing | | `image_gen` | `sdcpp`, the bundled sd-server | `openai_compatible`, with a `connection_id` | Studio's `image` and `infographic` | -| `image_edit` | `sdcpp`, a model whose entry names `edit` | `openai_compatible`, with a `connection_id` | Studio's image and infographic edits ([studio](../studio.md#editing-a-picture)) | +| `image_edit` | `sdcpp`, a model whose entry names `edit` | `openai_compatible`, with a `connection_id` | Optional: edits and references with a model other than the image model; empty, the image model takes them when it takes pictures in, and redraws when not ([studio](../studio.md#which-model-takes-a-picture)) | | `audio_gen` | `audiocpp`, the bundled audio.cpp server | `openai_compatible`, with a `connection_id` | Studio's `podcast` | | `video_gen` | `sdcpp`, a model whose entry has a `video` block | `openai_compatible`, with a `connection_id` | nothing yet | | `speech_to_text` | `audiocpp`, a recogniser, run on a second audio.cpp server | none: local only | the transcription routes ([admission](admission.md#speech-recognition)), which a recording's ingest calls ([documents](../documents.md#recordings)) | diff --git a/docs/architecture/studio.md b/docs/architecture/studio.md index 4e1267753c..cf39dfdbbf 100644 --- a/docs/architecture/studio.md +++ b/docs/architecture/studio.md @@ -323,36 +323,59 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i ## The image path -- A drawn `image` or `infographic` resolves the `image_gen` selection, and an edit the `image_edit` selection ([below](#editing-a-picture)). A remote selection loads its connection, which runs the egress check and decrypts the key. An `sdcpp` selection points the same client at the bundled sd-server's loopback URL, with no key and no egress decision, which is what makes an image possible on an offline machine. +- A drawn `image` or `infographic` resolves the `image_gen` selection, and an edit, or a drawing with references, the model that takes a picture in ([below](#which-model-takes-a-picture)). A remote selection loads its connection, which runs the egress check and decrypts the key. An `sdcpp` selection points the same client at the bundled sd-server's loopback URL, with no key and no egress decision, which is what makes an image possible on an offline machine. - sd-server runs only while Studio needs it, because it keeps a model's weights in memory until it exits. The API answers Electron's poll from the jobs ([`local_image_demand.py`](../../surfsense_local/backend/modules/artifacts/local_image_demand.py)): the chosen model of the image type the oldest processing job needs, read from its recipe, so an edit needs `image_edit` though its format is still `image` or `infographic`; else, for 5 minutes after the last such job ended, the same one, so a second image does not reload it; else nothing, and Electron stops it. A cancel ends the window at once, since sd-server cannot stop a generation, and so does any other Studio job running, since a podcast voices in that memory and a text job loads the chat model into it. The job first gives up the text runtime through the API, held for the whole image, because the two share the graphics card and its writing is done by then: the API lets running replies finish, unloads the text models and keeps local text waiting until the image is drawn, and unloads the speech recognition model, starting no recording's window meanwhile ([admission](local-models/admission.md#the-image-jobs-hold)). Then it waits up to 2 minutes for `/sdapi/v1/sd-models` to name its weights before it posts, and fails with a reason if sd-server never comes up. - sd-server is launched with each file of the build on its flag and the model's reviewed `image` defaults as its own (`-W`, `-H`, `--steps`, `--cfg-scale`, `--sampling-method`, `--flow-shift`), since a request that carries none of them starts from the launch settings ([`local-models/catalog.md`](local-models/catalog.md)). -- The client posts `{model, prompt}` to `/images/generations`, falls back once to `/images` only on `404` or `405`, and remembers the route that worked for the life of the process. It accepts `b64_json`, a data URL or a URL, caps the sizes, and checks the bytes against the claimed MIME type ([`connections.md`](connections.md)). +- The client posts `{model, prompt}`, and `size` when a shape is asked, to `/images/generations`, falls back once to `/images` only on `404` or `405`, and remembers the route that worked for the life of the process. It accepts `b64_json`, a data URL or a URL, caps the sizes, and checks the bytes against the claimed MIME type ([`connections.md`](connections.md)). - The `text_gen` model writes first in both formats, except a picture the agent asked for ([below](#pictures-the-agent-asks-for)). The markdown body is the image prompt or the brief, so an image can be found by what it shows. - A drawn picture is v1 of its own document: the job records `version` and `recipe: {"kind": "draft"}` as a Word draft's, so its edits group under it. Regenerate on it still draws from the sources and replaces v1 in place. +- Every drawing, edit and redraw keeps the exact prompt it sent as `drawing_prompt`, so a later redraw starts from it ([`redraw/`](../../surfsense_local/backend/worker/studio/media/visual/redraw/)). + +### Which model takes a picture + +One rule picks the model for any job that sends a picture in, an edit or a drawing with references ([`pictures_in/`](../../surfsense_local/backend/modules/llm/pictures_in/)): + +1. the `image_edit` selection, when one is chosen: an override, for editing with another model; +2. else the `image_gen` selection, when it takes a picture in; +3. else none, and an edit is drawn again ([below](#editing-a-picture)). + +Whether a remote image model takes a picture in is read when it is chosen, from its connection's model list: a model that lists `image` among its `input_modalities` and its outputs, as OpenRouter's do, gets the `image_edit` type beside `image_gen`, as the catalog already gives a model with an image in and out; the choice keeps `takes_pictures` in the selection's `settings`, and another model chosen starts without it. A local model takes pictures when its entry names `edit`. A model chosen unlisted is taken as words only, since a picture it ignores is billed. `GET /llm/pictures-in` answers `{source}`, `edit_model`, `image_model` or `null`, for the app. + +### Shapes and references + +A drawing can ask for a `shape`, `square`, `portrait` or `landscape` ([`picture_shape.py`](../../surfsense_local/backend/modules/llm/providers/picture_shape.py)): + +- **Remote:** `size` `1024x1024`, `1024x1536` or `1536x1024` on `/images/generations` and `/images/edits`; `aspect_ratio` `1:1`, `2:3` or `3:2` on `/images`. A server that refuses one with a `400` is asked once more without it and remembered, separately from `response_format`, since GPT image models refuse that and take a size. +- **Local:** a drawing with a shape goes through sd-server's native `img_gen` with `width` and `height` at the launch's `-W` times `-H` pixels in the shape's proportions, each side a multiple of 64; a drawing with none keeps `/v1/images/generations`. + +References are pictures sent with the prompt, so a new picture keeps their characters and style, as a comic's later pages keep page 1's. They go to the model that takes a picture in, as an edit's one picture does: every reference as an `image[]` part of `/images/edits`, or, where a server answers `404` or `405` there, as OpenRouter does, `input_references` of `{type: "image_url", image_url: {url: <data URL>}}` on `/images`, remembered per server; locally, every one in `ref_images`. A refusal there comes before any drawing, so trying the other route bills nothing; a `5xx` or a timeout is never retried. With no model that takes a picture in, the picture is drawn from its prompt alone and keeps `drawn_from_words`. ### Pictures the agent asks for -`surfsense_make_image` ([agent](agent.md#pictures-the-agent-makes)) creates an `image` or `infographic` with no sources, the chat's `chat_thread_id` and the recipe `{"kind": "image_drawn", "prompt"}` ([`drawn_images/`](../../surfsense_local/backend/modules/artifacts/drawn_images/), [ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). The job paints the prompt as written, asking no chat model; an infographic adds Studio's sketchnote style around it ([`media/visual/drawn/`](../../surfsense_local/backend/worker/studio/media/visual/drawn/)). The body is the title and the prompt, so it can be found by what it shows. +`surfsense_make_image` ([agent](agent.md#pictures-the-agent-makes)) creates an `image` or `infographic` with no sources, the chat's `chat_thread_id` and the recipe `{"kind": "image_drawn", "prompt"}`, with `shape` and `references` (artifact ids) when asked ([`drawn_images/`](../../surfsense_local/backend/modules/artifacts/drawn_images/), [ADR 0087](../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). The job paints the prompt as written, asking no chat model; an infographic adds Studio's sketchnote style around it ([`media/visual/drawn/`](../../surfsense_local/backend/worker/studio/media/visual/drawn/)). The body is the title and the prompt, so it can be found by what it shows. -It needs only `image_gen`, its Regenerate paints the same prompt, and Huey never retries it, since each try may be billed. It is v1 of its own document and takes Studio's Edit box like any picture. Every edit keeps its base's chat, from either box, so that chat can still place it. While a script of its chat names a picture, deleting that picture is refused with `placed_in_document`, since the script names it rather than copying it ([`placed_in.py`](../../surfsense_local/backend/modules/artifacts/drawn_images/placed_in.py)). +It needs only `image_gen`, its Regenerate paints the same prompt at the same shape from the same references, a deleted reference failing it, and Huey never retries it, since each try may be billed. It is v1 of its own document and takes Studio's Edit box like any picture. Every edit keeps its base's chat, from either box, so that chat can still place it. While a script of its chat names a picture, deleting that picture is refused with `placed_in_document`, since the script names it rather than copying it ([`placed_in.py`](../../surfsense_local/backend/modules/artifacts/drawn_images/placed_in.py)). ### Editing a picture `POST /artifacts/{id}/image-edit` with `{instruction}` (1 to 2,000 characters, as Refine's) makes an image's or infographic's next version from the open version's picture ([`image_edits/`](../../surfsense_local/backend/modules/artifacts/image_edits/), [ADR 0077](../adr/0077-an-image-edit-is-the-next-version.md)). -- It refuses, each a `409` whose `detail` is `{message, code}`: a version still being made (`already_generating`); one whose last run failed or was stopped (`image_version_unfinished`, its own code so the sentence says to edit another version, not refine one); anything but an `image` or `infographic` with a still `primary` picture in PNG, JPEG or WebP, since an edit model does not read the SVG a remote model may draw and `/images/edits` takes no GIF (`not_editable`); and, after those, no `image_edit` model it can use (`needs_image_edit`, "Needs an image editing model"), the way Refine answers `needs_chat`: none chosen, a local one not installed or whose entry no longer names `edit`, or a connection that is gone or no longer serves image editing. Egress to its host being off is refused as any call to a host is, with the egress refusal that asks to allow it ([egress](egress.md)). The check reads the selection and asks no server, which would add a round trip to every edit and could start sd-server only to answer; a model that cannot be reached fails the version with its reason, as a refine's does. `ArtifactRead.image_editable` is true for a ready `image` or `infographic` with a `primary` PNG, JPEG or WebP. +- It refuses, each a `409` whose `detail` is `{message, code}`: a version still being made (`already_generating`); one whose last run failed or was stopped (`image_version_unfinished`, its own code so the sentence says to edit another version, not refine one); anything but an `image` or `infographic` with a still `primary` picture in PNG, JPEG or WebP, since an edit model does not read the SVG a remote model may draw and `/images/edits` takes no GIF (`not_editable`); and, after those, no image model at all (`needs_image`, "Needs an image model"): with [no model that takes a picture in](#which-model-takes-a-picture), the image model draws it again, so no edit is refused for want of an edit model. Egress to its host being off is refused as any call to a host is, with the egress refusal that asks to allow it ([egress](egress.md)). The check reads the selection and asks no server, which would add a round trip to every edit and could start sd-server only to answer; a model that cannot be reached fails the version with its reason, as a refine's does. `ArtifactRead.image_editable` is true for a ready `image` or `infographic` with a `primary` PNG, JPEG or WebP. - It creates the next version, `pending`, with the base's title, prompt and `source_document_ids`, and the recipe `{"kind": "image_edit", "instruction", "base_artifact_id"}`, under the `version` keys a refine writes. An image drawn before versions were kept becomes v1 then. The recipe names the base rather than copying its bytes, so a base cannot go while a later version names it: `DELETE /artifacts/{id}`, and the documents route for its document, refuse it with `409` `edited_from`, "A later version is edited from this one: delete the whole document." ([`edited_from.py`](../../surfsense_local/backend/modules/artifacts/image_edits/edited_from.py)). The list deletes a document's versions newest first, each after the last, and an edit is always newer than its base, so it never meets the refusal. -- The job reads the base's own file, so edits chain: editing v3 starts from v3's picture. The instruction goes to the edit model as written, with no chat model call first ([`media/visual/edit/`](../../surfsense_local/backend/worker/studio/media/visual/edit/)). The body is the base's, then `Edited: <instruction>`, so the version stays findable by what it shows and says how it moved; the file keeps the base's name with the extension of what came back. Every failure ends the job without Huey's retry, as a refine's does. -- **Remote:** the client posts multipart `/images/edits` with `image`, `prompt`, `model` and `response_format: b64_json`, sent once more without the last if the model refuses it with a `400`, as a generation is ([connections](connections.md)), and no `input_fidelity`, which only some GPT image models take ([`image_edit.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py)). It reads the answer's `data[]` with the same bounded read, link fetch and MIME check as a generation ([`image_response.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py)). +- The job reads the base's own file, so edits chain: editing v3 starts from v3's picture. The instruction goes to the model that takes the picture as written, with no chat model call first ([`media/visual/edit/`](../../surfsense_local/backend/worker/studio/media/visual/edit/)). The body is the base's, then `Edited: <instruction>`, so the version stays findable by what it shows and says how it moved; the file keeps the base's name with the extension of what came back. Every failure ends the job without Huey's retry, as a refine's does. +- **A redraw.** With no model that takes a picture in, the image model draws `<drawing prompt>\n\nChange: <instruction>` at the base's proportions, with no chat model. The drawing prompt is the base's `drawing_prompt`, or, for a picture drawn before it was kept, its recipe's prompt, or its body under the title, so changes add up from version to version. The version keeps `redrawn: true` and its body starts "Drawn again with your change." ([`redraw/`](../../surfsense_local/backend/worker/studio/media/visual/redraw/)). Studio's Edit box says so in its placeholder, "Describe a change: the picture is drawn again", read from `GET /llm/pictures-in`. +- **Remote:** the client posts multipart `/images/edits` with `image`, `prompt`, `model` and `response_format: b64_json`, sent once more without the last if the model refuses it with a `400`, as a generation is ([connections](connections.md)); where the server has no `/images/edits`, the picture goes to `/images` as a reference ([above](#shapes-and-references)); and no `input_fidelity`, which only some GPT image models take ([`image_edit.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py)). It reads the answer's `data[]` with the same bounded read, link fetch and MIME check as a generation ([`image_response.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py)). - **Local:** inside the same text-runtime hold and model wait as a drawn image, the client posts sd-server's native `POST /sdcpp/v1/img_gen` with `{prompt, ref_images: [<data URL>], width, height, output_format: "png"}` and polls `GET /sdcpp/v1/jobs/{id}` each second, which also looks for a cancel ([`editor.py`](../../surfsense_local/backend/modules/llm/providers/sdcpp/editor.py)). A `failed` job, a `404`, `410` or `429`, or a server that stops answering fails the edit. Steps, guidance and sampler come from the launch, as for a drawn image; the size is the base's own, since an edit model launched at another size would crop or stretch it. sd.cpp picks the reference preset from the model's architecture, so the launch needs no `--ref-image-args`. - **Why `img_gen`.** At the pinned `master-869-07a85c7`, sd-server's `/v1/images/edits` makes the first upload the `init_image` as well as a reference, and `/sdapi/v1/img2img` takes `init_images`: both re-noise the picture at a strength of 0.75 rather than edit it. `ref_images` alone is how every edit model sd.cpp documents is run. - **Cancel:** a queued sd.cpp job is withdrawn with `POST /sdcpp/v1/jobs/{id}/cancel`, given 2 seconds, since the text runtime is still held while it is asked. A generating one cannot be (`cancel_generating: false`); the cancel ends the demand at once and Electron stops sd-server, as for a drawn image. -- **Which models edit:** a local model whose entry names `edit`, today FLUX.2 klein 4B, which fills both image slots from one download, so nothing reloads between drawing and editing; or a remote model that takes an image in and gives one out ([`classifier.py`](../../surfsense_local/backend/modules/llm/catalog/remote/classifier.py)). Models that can only re-noise a picture, SD 1.5, SDXL, Z-Image Turbo, ERNIE-Image Turbo and LongCat-Image, name no `edit` and cannot be chosen for it. +- **Which models edit:** a local model whose entry names `edit`, today FLUX.2 klein 4B, which fills both image slots from one download, so nothing reloads between drawing and editing; or a remote model that takes an image in and gives one out, by its connection's own list or the catalog ([`classifier.py`](../../surfsense_local/backend/modules/llm/catalog/remote/classifier.py)). Settings marks those with an **Edits** label, "Can edit pictures". Models that can only re-noise a picture, SD 1.5, SDXL, Z-Image Turbo, ERNIE-Image Turbo and LongCat-Image, name no `edit`: they cannot be chosen as the edit override, and a picture they drew is redrawn. ## Known gaps - An image edit is measured only on klein Q4_0 through the pinned sd-server on a processor, with no graphics card: a photo's layout held, and an infographic's printed lettering held through a colour change and a changed title, 657 to 962 seconds an edit, about twice a draw. Lettering klein drew itself, a sketchnote's, is not measured, and neither is any time on a graphics card or a remote edit model. - An edit changes the whole picture by instruction; there is no mask or selected region, which sd.cpp and `/images/edits` both take, since drawing one needs its own interface. +- References are measured only on OpenRouter's `/images`, on six models drawing one four-page comic: each held its characters from page 1. OpenAI's `/images/edits` with several pictures, and local references through sd-server, are not measured; nor is a remote server that takes one picture on `/images/edits` and refuses several. +- A remote model's taking pictures is read when it is chosen; a provider that later starts or stops listing it is not seen until the model is chosen again. - The [window budget](#the-window-budget) prices a draft's first call. What only some calls add is not priced: a written web page's design guide (6,235 characters, about 2,100 tokens past the prompt it reserves), the failed reply a repair replays, and a podcast segment's recap. On a small window these can leave the reply short or be refused, which fails the job with the context sentence and no retry. Reasoning is not reserved either: Studio turns thinking off, which the local runtime honours and some remote endpoints do not, whose windows are large. - The budget prices text with admission's estimate, not the model's tokenizer, so scripts it prices at a token a character, such as Cyrillic or Devanagari, are cut further than they need. A long figure list leaves a Word or PDF draft less of its sources' text, since every figure they hold is listed. - Under an image's hold, a local model no request has seated since the app started has no last window, so its job keeps the 24,000-character ceiling and may be refused. diff --git a/docs/proposals/pictures/README.md b/docs/proposals/pictures/README.md index 7289abc25d..d85be3a58e 100644 --- a/docs/proposals/pictures/README.md +++ b/docs/proposals/pictures/README.md @@ -1,5 +1,5 @@ --- -status: proposed +status: in-progress code: - surfsense_local/backend/modules/llm/providers/ - surfsense_local/backend/modules/llm/connections/ @@ -90,6 +90,22 @@ On one branch, `feat/comics`, a commit per step, each test first: 7. Settings' label and Studio's Edit ([05](05-in-the-app.md)). 8. Docs: architecture, ADRs, user docs, and this proposal deleted ([below](#when-it-ships)). +## Built, and where it differs + +Steps 1 to 8 are built on `feat/comics`; what is now true is in [studio](../../architecture/studio.md#which-model-takes-a-picture), [agent](../../architecture/agent.md#pictures-the-agent-makes), [ADR 0089](../../adr/0089-any-image-model-changes-a-picture.md) and [ADR 0090](../../adr/0090-a-document-waits-on-the-server-for-its-pictures.md). Where the build differs from the parts below: + +- **Two protocols, not one.** `ImageGenerator` draws from words and `ImageEditor` takes pictures, each with `shape`, the editor with several pictures; the rule picks which client a picture goes to, so Studio's pipelines kept their shape. +- **No OpenRouter client.** One client sends pictures to `/images/edits` and, on a `404` or `405` there, to `/images` with `input_references`, remembered per server. Any OpenAI-compatible server whose image models take pictures works, not OpenRouter alone. +- **Taking pictures is the `image_edit` type** a model already gets from the catalog, now also from a connection's own `input_modalities`; no new field. +- **A waiting render's failure does not count** toward renders' three-failure stop: renders never counted late failures, as pictures do. +- **The agent's prompt names the skill in one line**, since it was 4 characters short of its 4,000-character cap; the tool's description and the web page skill's carry the rest. + +## Left to do + +- The spikes in [image models](01-image-models.md#spikes-before-code): several pictures on OpenAI's `/images/edits`, and local references and shapes through sd-server. +- Settings: the edit override as its own section, retitled "Use a different model for edits", not yet folded under Image gen; the **Edits** label on server models only, not yet on local models to download. +- The images guide's walkthrough and screenshots, which still choose a model for editing. + ## Decisions - **One image model, not two slots.** Whether a model can take a picture in is a fact about the model, so it is read from the model, not asked of the user. The edit slot stays as an override for the few who want another model for edits. diff --git a/www/content/docs/v2/agent.mdx b/www/content/docs/v2/agent.mdx index 06071820ac..6644e6dd7c 100644 --- a/www/content/docs/v2/agent.mdx +++ b/www/content/docs/v2/agent.mdx @@ -31,12 +31,14 @@ Documents the agent made are changed in its chat, not with Studio's **Refine**. ### Pictures -Ask for a picture, an infographic or a document with a picture in it. The agent reads what it needs from your sources, writes the prompt itself and draws with the model in **Settings › Image gen**. To change a picture, ask in the same chat, which uses **Settings › Image edit** and makes its next version. Each picture appears in [Studio](/docs/v2/studio). +Ask for a picture, an infographic or a document with a picture in it. The agent reads what it needs from your sources, writes the prompt itself and draws with the model in **Settings › Image gen**. To change a picture, ask in the same chat, and it makes its next version. Each picture appears in [Studio](/docs/v2/studio). + +Ask for a comic, a storyboard or a picture book and the agent plans the pages, tells you how many it will draw, and asks once before it starts. It draws one picture per page, keeps the characters from page to page, and puts the pages together in one PDF. With an image model that can't take a picture in, each page is drawn from a description of the characters, so they may look a little different from page to page. - **It draws only when you ask.** A remote model bills each picture. A model on a computer without a graphics card takes several minutes each. -- **A slow picture may finish after the agent replies.** It appears in Studio when it's ready. To place it in a document, ask again in the same chat. +- **A slow picture may finish after the agent replies.** It appears in Studio when it's ready. A document that uses it, such as a comic's PDF, waits for it and appears in Studio once its pictures are drawn. - **Pictures used in a document stay until the document is gone.** If the agent placed a picture in a Word document, PDF, deck or workbook, delete that document before deleting the picture, because SurfSense needs the picture to make the document again. -- **Exact words go in the document.** Image models can garble lettering, so check any words in a picture. Data charts stay real charts. +- **Exact words go in the document.** Image models can garble lettering, so check any words in a picture. When a comic page's speech bubbles come out wrong, the PDF prints that page's lines under it. Data charts stay real charts. ### Start from your own template diff --git a/www/content/docs/v2/studio.mdx b/www/content/docs/v2/studio.mdx index dc5d40dee7..6c5fbc60bf 100644 --- a/www/content/docs/v2/studio.mdx +++ b/www/content/docs/v2/studio.mdx @@ -92,11 +92,11 @@ Documents the agent made are changed in its chat instead: "The agent made this d ## Edit an image -Over a ready image or infographic, the same round button opens an edit box. Describe a change, such as "a darker background" or a new title, and click **Edit**. Studio sends the picture and your instruction to the image editing model, and what comes back is the next version. +Over a ready image or infographic, the same round button opens an edit box. Describe a change, such as "a darker background" or a new title, and click **Edit**. Studio sends the picture and your instruction to your image model, and what comes back is the next version. Each edit starts from the version you have open, so edits chain, and every earlier version stays. There's no way to select one region of the picture yet: an edit applies to the whole image. -Without an image editing model, the box says "Needs an image editing model." with **Choose a model**. +Some image models can't take a picture in. With one of those, the box says "Describe a change: the picture is drawn again": Studio draws the picture again from what it was drawn from and your change, so the rest of the picture may change too. Image models that edit are marked **Edits** in Settings. <Callout type="info"> Editing an image on a computer without a graphics card is slow. In testing, @@ -108,7 +108,7 @@ Without an image editing model, the box says "Needs an image editing model." wit Choose them in Settings: - **Settings › Image gen** for drawing. Run a model on this computer, such as FLUX.2 klein 4B, Z-Image Turbo or Stable Diffusion XL, or use one from an API connection. -- **Settings › Image edit** for editing. FLUX.2 klein 4B both draws and edits from one download. A remote model that takes a picture in and gives one back works too. +- **Settings › Image edit** is optional: your image model edits pictures when it can, and draws them again when it can't. Choose a model there only to edit with a different one. FLUX.2 klein 4B both draws and edits from one download, and most remote image models take a picture in. A local image model runs only while Studio needs it, and works offline. It frees its memory a few minutes after the last image. See [Models](/docs/v2/models). From 00b26db6fb913dc75179f67c82b8cebc8058b27f Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 00:56:40 +0530 Subject: [PATCH 08/33] feat(local): every kind of comic, read and bound its own way, in both engines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit comics.md covers Western comics, graphic novels, bande dessinée, manga, manhua, manhwa and webtoons, comic strips, yonkoma, picture books and storyboards: how each reads, how its pictures are shaped and bound, and how it looks. The binding script reads a manga right to left and stacks a webtoon down one long page. A test builds Claude Code's skills plugin from the shipped skills, so a Claude plan thread gets the pictures skill as opencode does. --- docs/architecture/agent.md | 2 +- .../agent/skills/surfsense-pictures/SKILL.md | 4 +- .../agent/skills/surfsense-pictures/comics.md | 33 +++++-- .../agent/skills/surfsense-pictures/pdf.md | 61 +++++++++---- .../tests/unit/agent/test_pictures_skill.py | 87 ++++++++++++++++++- www/content/docs/v2/agent.mdx | 2 +- 6 files changed, 159 insertions(+), 30 deletions(-) diff --git a/docs/architecture/agent.md b/docs/architecture/agent.md index 7582bf4c8e..fb487337fb 100644 --- a/docs/architecture/agent.md +++ b/docs/architecture/agent.md @@ -223,7 +223,7 @@ The step links the version in the thread as a render's does (`surfsense_save_pag **Shape and references.** `shape` (`square`, `portrait`, `landscape`) reaches the image model ([shapes](studio.md#shapes-and-references)). `references`, up to four picture names of this chat, send those pictures with the prompt to the model that takes pictures, so a comic's later pages keep page 1's characters; with none, the picture is drawn from its prompt and the result says the characters may differ. The tool's listing stays the same whatever models are chosen, so a local model's prompt cache holds. -**Comics and sets.** The `surfsense-pictures` skill ([`skills/surfsense-pictures/`](../../surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md)) is the craft beside the tool, as `surfsense-documents` is beside the render: one picture, or a set such as a comic, a storyboard or a picture book. A set is planned with a character sheet, confirmed once, drawn first page alone and the rest with it as reference, and bound into one PDF by the script in `pdf.md`: one full-bleed picture a page, placed as JPEG, with a page's exact lines set in a strip under it when its lettering came out wrong or it was drawn without words. The prompt sends pictures and comics there, and the web page skill's description sends them away. +**Comics and sets.** The `surfsense-pictures` skill ([`skills/surfsense-pictures/`](../../surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md)) is the craft beside the tool, as `surfsense-documents` is beside the render: one picture, or a set such as a comic, a storyboard or a picture book. A set is planned with a character sheet, confirmed once, drawn first page alone and the rest with it as reference, and bound into one PDF by the script in `pdf.md`: one full-bleed picture a page, or, for a webtoon, the pictures stacked down one long page; marked right to left for a manga; placed as JPEG, with a page's exact lines set in a strip under it when its lettering came out wrong or it was drawn without words. The prompt sends pictures and comics there, and the web page skill's description sends them away. **Prompt length.** As in Studio, nothing limits a prompt per model: the tool's description asks for one self-contained prompt, as short as it says what to draw, and a model's own refusal comes back as a failed picture. diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md index bec0287367..19651c42e7 100644 --- a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md @@ -1,6 +1,6 @@ --- name: surfsense-pictures -description: Draw pictures with surfsense_make_image - one picture or infographic, or a set that belongs together, such as a comic, a manga, a comic strip, a storyboard or a picture book, bound into one PDF. Load it whenever the user asks for a picture, an illustration, or any set of pictures, or for a change to one you drew. Not for charts of numbers, and not for a web page. +description: Draw pictures with surfsense_make_image - one picture or infographic, or a set that belongs together, such as a comic, a manga, a manhwa or webtoon, a comic strip, a graphic novel, a storyboard or a picture book, bound into one PDF. Load it whenever the user asks for a picture, an illustration, or any set of pictures, or for a change to one you drew. Not for charts of numbers, and not for a web page. --- # Pictures @@ -11,7 +11,7 @@ description: Draw pictures with surfsense_make_image - one picture or infographi | File | Read it | |---|---| -| `comics.md` | For a comic, a manga, a strip, a storyboard or a picture book: pages, panels and lettering. | +| `comics.md` | For any kind of comic (Western, manga, manhwa and webtoon, manhua, strip, yonkoma, graphic novel, bande dessinée), a picture book or a storyboard: how each reads, is laid out and is lettered. | | `pdf.md` | When you bind pictures into a PDF: the script to copy and fill. | ## One picture diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md index 5f6b37a0a4..e63ffd130c 100644 --- a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md @@ -1,12 +1,31 @@ # Comics -A comic, a manga, a comic strip, a storyboard or a picture book: one picture per page, every page from the same character sheet and style line, bound into one PDF. Follow the set's steps in [`SKILL.md`](SKILL.md#a-set-of-pictures); this file is what is particular to comics. +A comic of any kind: one picture per page, or per panel for a scrolling one, every picture from the same character sheet and style line, bound into one PDF. Follow the set's steps in [`SKILL.md`](SKILL.md#a-set-of-pictures); this file is what is particular to comics. + +## Which kind + +Take the kind the user names; when they name none, a Western comic. The kind decides how it reads, how its pictures are shaped and bound, and how it looks. A style the user asks for (noir, chibi, watercolour, superhero) goes in the style line on top of it. + +| Kind | Reads | Each picture | Binding (`pdf.md`) | Looks | +|---|---|---|---|---| +| Western comic | left to right | a `portrait` page of 4 to 6 panels | `LAYOUT = "pages"` | full colour, bold inks, caption boxes | +| Graphic novel | left to right | a `portrait` page of 3 to 6 panels | `"pages"` | as a Western comic, with room for quiet, wordless panels | +| Bande dessinée | left to right | a `portrait` page of 6 to 9 smaller panels | `"pages"` | detailed album pages, often clean "ligne claire" lines and flat colour | +| Manga | **right to left**: panel 1 top right, the last bottom left | a `portrait` page of 3 to 6 panels | `"pages"`, `DIRECTION = "rtl"` | black and white with screentone, speed lines, expressive faces | +| Manhua | left to right | a `portrait` page, or one panel a picture to scroll | `"pages"` or `"scroll"` | full colour, often painterly | +| Manhwa, webtoon | top to bottom, by scrolling | one panel or beat a `portrait` picture, no page | `LAYOUT = "scroll"` | full colour, soft shading, wide gutters between beats | +| Comic strip | left to right | one `landscape` picture of 3 or 4 panels in a row | `"pages"`, one page a strip | simple shapes, a punchline in the last panel | +| Yonkoma (4-koma) | top to bottom | one `portrait` picture of 4 equal panels stacked | `"pages"`, one page a strip | a manga strip: setup, development, turn, punchline | +| Picture book | left to right | one illustration a page, no bubbles | `"pages"`, every page's text as its lines | soft colour, one moment a page | +| Storyboard | left to right | one `landscape` frame a shot | `"pages"`, each shot's number and note as its lines | rough, sketchy frames; camera and movement in the note | + +A manga's panels and bubbles still read right to left inside each page: say so in every prompt. Write its lines in the user's language, read left to right inside each bubble; ask for vertical lettering only when the user writes Japanese and asks for it. ## The plan -- **Pages.** Four when the user names no number; at most twelve, since each is billed or takes minutes. A comic strip is one `landscape` picture of three or four panels. -- **Panels.** Two to five a page. For each: where it sits (top, bottom left, wide), what it shows, and its lines. -- **Lines.** One short line a bubble, at most about eight words, two or three bubbles a panel. A caption box for narration. Write every line in the plan, word for word: the PDF may need them. +- **Length.** Four pages, or eight beats for a scrolling comic, when the user names none; at most twelve pictures, since each is billed or takes minutes. A comic strip or a yonkoma is one picture a strip. +- **Panels.** For each: where it sits (top, bottom left, wide), what it shows, and its lines. A scrolling comic has one beat a picture instead. +- **Lines.** One short line a bubble, at most about eight words, two or three bubbles a panel. A caption box for narration. Write every line in the plan, word for word: the PDF may need them. A picture book's and a storyboard's text is never in the picture; it goes under it. ## A page's prompt @@ -23,14 +42,14 @@ Panel 2 (bottom left): NORA taps the dead lamp. NORA says: "THE BULB IS BROKEN!" Panel 3 (bottom right): MILO spots a box of spare bulbs on a high shelf. ``` -From page 2 on, add one line before the page: "Keep the characters looking exactly as in the reference picture: same faces, colours, clothes and drawing style. Draw a new page; do not copy its panels." Every page is `shape: portrait`. +For a manga: "Black-and-white manga page with screentone, read right to left. Panel 1 (top right): …". For a scrolling comic: "One tall webtoon panel, full colour, nothing else in the picture: …", one beat per picture. From the second picture on, add one line before the page: "Keep the characters looking exactly as in the reference picture: same faces, colours, clothes and drawing style. Draw a new page; do not copy its panels." ## Lettering -- **Bubbles drawn by the model.** Write each line in the prompt in quotes, in capitals. +- **Bubbles drawn by the model.** Write each line in the prompt in quotes, in capitals for a Western comic. - **A misspelt line is not redrawn.** Note the page; the PDF sets that page's lines under it, exactly, in a strip ([`pdf.md`](pdf.md)). A redraw would bill another picture for one letter. - **On this computer** (the result says "Drawn on this computer"), **or when you cannot see the pictures**, draw every page without words: leave the bubbles out of the prompts, ask for "no text, no lettering, no speech bubbles", and give every page its strip. You cannot rely on the lettering, or check it. ## Before you bind -Look at every page. Redraw or change one only when it is plainly wrong, such as a missing character, the wrong number of panels, or a third arm, and say which and why. Then bind the PDF, every page's name in `images`, in order. +Look at every picture. Redraw or change one only when it is plainly wrong, such as a missing character, the wrong number of panels, panels in the wrong reading order, or a third arm, and say which and why. Then bind the PDF, every picture's name in `images`, in reading order, with the kind's `LAYOUT` and `DIRECTION`. diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md index 75c114b308..c7a13361d5 100644 --- a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md @@ -1,11 +1,12 @@ # Binding pictures into a PDF -Render this with `surfsense_render_document`, `format: pdf`, every page's name in `images`, in order. Fill in `TITLE` and `PAGES`, and change nothing else unless the user asks: it is the layout every comic and picture book uses. +Render this with `surfsense_render_document`, `format: pdf`, every picture's name in `images`, in reading order. Fill in `TITLE`, `PAGES`, and for the kind of comic `LAYOUT` and `DIRECTION` ([`comics.md`](comics.md#which-kind)); change nothing else unless the user asks. -- One picture per page, full bleed, at comic-book width, each page as tall as its picture. -- A page's lines, when given, are set under its picture in a strip, exactly as written. Give them for a page whose lettering came out wrong, or for every page drawn without words. +- `LAYOUT = "pages"`: one picture per page, full bleed, at comic-book width, each page as tall as its picture. `LAYOUT = "scroll"`: the pictures stacked down one long page, read by scrolling, as a webtoon is; a new page every twelve. +- `DIRECTION = "rtl"` tells a reader the book reads right to left, as a manga does, so its spreads are laid out that way. +- A picture's lines, when given, are set under it in a strip, exactly as written. Give them for a page whose lettering came out wrong, for every page drawn without words, for a picture book's text and a storyboard's notes. - Pictures are placed as JPEG, so a book of pages stays a few megabytes. -- A page still being drawn may be named all the same: the PDF waits for it, and appears in Studio when its last page is drawn. Tell the user so, and stop. +- A picture still being drawn may be named all the same: the PDF waits for it, and appears in Studio when its last picture is drawn. Tell the user so, and stop. ```python import os @@ -18,15 +19,19 @@ from reportlab.pdfbase.pdfmetrics import stringWidth from reportlab.pdfgen import canvas TITLE = "The Lighthouse Cat" -# Each page in order: its picture's name, and the lines to set under it, or none. +# Each picture in reading order: its name, and the lines to set under it, or none. PAGES = [ ("image-12", []), ("image-13", ["NORA: The bulb is broken!"]), ] +LAYOUT = "pages" # "pages", or "scroll" for a webtoon +DIRECTION = "ltr" # "ltr", or "rtl" for a manga WIDTH = 6.625 * inch MARGIN = 0.25 * inch +GUTTER = 0.2 * inch FONT, SIZE, LEADING = "Helvetica", 11, 15 +PER_SCROLL = 12 def wrapped(text, width): @@ -41,24 +46,44 @@ def wrapped(text, width): return lines + [line] if line else lines -pdf = canvas.Canvas(os.environ["OUTPUT_PATH"]) -pdf.setTitle(TITLE) -for name, said in PAGES: - path = os.path.join(os.environ["IMAGES_DIR"], name + ".png") - picture = Image.open(path).convert("RGB") +def block(name, said): + """A picture as JPEG, its height at the page's width, and its strip's lines.""" + picture = Image.open(os.path.join(os.environ["IMAGES_DIR"], name + ".png")) + picture = picture.convert("RGB") jpeg = BytesIO() picture.save(jpeg, "JPEG", quality=90) jpeg.seek(0) - height = WIDTH * picture.height / picture.width lines = [line for text in said for line in wrapped(text, WIDTH - 2 * MARGIN)] strip = len(lines) * LEADING + 2 * MARGIN if lines else 0 - pdf.setPageSize((WIDTH, height + strip)) - pdf.drawImage(ImageReader(jpeg), 0, strip, WIDTH, height) - pdf.setFont(FONT, SIZE) - y = strip - MARGIN - SIZE - for line in lines: - pdf.drawString(MARGIN, y, line) - y -= LEADING + return ImageReader(jpeg), WIDTH * picture.height / picture.width, lines, strip + + +def draw(pdf, blocks, gutter): + """Top to bottom on one page, each picture with its strip under it.""" + height = sum(h + strip for _, h, _, strip in blocks) + gutter * (len(blocks) - 1) + pdf.setPageSize((WIDTH, height)) + top = height + for image, h, lines, strip in blocks: + pdf.drawImage(image, 0, top - h, WIDTH, h) + pdf.setFont(FONT, SIZE) + y = top - h - MARGIN - SIZE + for line in lines: + pdf.drawString(MARGIN, y, line) + y -= LEADING + top -= h + strip + gutter pdf.showPage() + + +pdf = canvas.Canvas(os.environ["OUTPUT_PATH"]) +pdf.setTitle(TITLE) +if DIRECTION == "rtl": + pdf.setViewerPreference("Direction", "R2L") +blocks = [block(name, said) for name, said in PAGES] +if LAYOUT == "scroll": + for start in range(0, len(blocks), PER_SCROLL): + draw(pdf, blocks[start : start + PER_SCROLL], GUTTER) +else: + for one in blocks: + draw(pdf, [one], 0) pdf.save() ``` diff --git a/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py index bd074514c1..d05b84f542 100644 --- a/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py +++ b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py @@ -12,6 +12,7 @@ from PIL import Image from pypdf import PdfReader +from modules.agent.engines.claude_code.skills_plugin import build_skills_plugin from modules.agent.opencode_config import PICTURES_SKILL, skills_folder from modules.agent.tool_endpoint.make_image import LISTING @@ -43,16 +44,100 @@ def test_the_skill_never_sends_the_user_to_settings() -> None: assert "Settings >" not in _file(name), name -def _script(pages: list[tuple[str, list[str]]]) -> str: +def test_a_claude_plan_thread_gets_the_skill_with_its_own_tool_names( + tmp_path: Path, +) -> None: + """Claude Code reads the same skill, its SurfSense tools named as it lists them.""" + plugin = build_skills_plugin(skills_folder(), tmp_path / "plugin") + + assert f"surfsense:{PICTURES_SKILL}" in plugin.names + for name in ("SKILL.md", "comics.md", "pdf.md"): + text = (plugin.skills / PICTURES_SKILL / name).read_text(encoding="utf-8") + assert not re.search( + r"surfsense_(make_image|render_document|read_document)", text + ) + skill = (plugin.skills / PICTURES_SKILL / "SKILL.md").read_text(encoding="utf-8") + assert "mcp__surfsense__make_image" in skill + + +def test_comics_cover_every_kind_the_user_may_ask_for() -> None: + """Each reads and is laid out its own way; a manga is not a Western comic.""" + comics = _file("comics.md").lower() + + for kind in ( + "manga", + "manhwa", + "webtoon", + "manhua", + "comic strip", + "yonkoma", + "graphic novel", + "bande dessinée", + "picture book", + "storyboard", + ): + assert kind in comics, kind + + +def _script( + pages: list[tuple[str, list[str]]], + direction: str = "ltr", + layout: str = "pages", +) -> str: """The script the skill gives, filled as the agent fills it.""" (script,) = re.findall(r"```python\n(.*?)```", _file("pdf.md"), re.S) filled = re.sub( r"PAGES = \[\n.*?\n\]", f"PAGES = {pages!r}", script, count=1, flags=re.S ) + filled = filled.replace('DIRECTION = "ltr"', f"DIRECTION = {direction!r}", 1) + filled = filled.replace('LAYOUT = "pages"', f"LAYOUT = {layout!r}", 1) assert filled != script + assert f"DIRECTION = {direction!r}" in filled + assert f"LAYOUT = {layout!r}" in filled return filled +def _bind(tmp_path: Path, script: str, count: int) -> PdfReader: + images, work = tmp_path / "images", tmp_path / "work" + images.mkdir() + work.mkdir() + # A colour each: ReportLab stores one copy of pictures that are the same. + for n, colour in zip( + range(1, count + 1), ("red", "green", "blue", "white"), strict=False + ): + (images / f"image-{n}.png").write_bytes(_page(colour, (1024, 1536))) + output = tmp_path / "comic.pdf" + subprocess.run( + [sys.executable, "-c", script], + cwd=work, + env={**os.environ, "OUTPUT_PATH": str(output), "IMAGES_DIR": str(images)}, + check=True, + ) + return PdfReader(output) + + +def test_a_manga_reads_right_to_left(tmp_path: Path) -> None: + """A reader then lays its spreads out as a manga's are.""" + pages = [(f"image-{n}", []) for n in range(1, 3)] + + pdf = _bind(tmp_path, _script(pages, direction="rtl"), 2) + + assert pdf.trailer["/Root"]["/ViewerPreferences"]["/Direction"] == "/R2L" + assert len(pdf.pages) == 2 + + +def test_a_webtoon_scrolls_down_one_long_page(tmp_path: Path) -> None: + """Its panels stack in one column, read by scrolling, not page by page.""" + pages = [(f"image-{n}", []) for n in range(1, 4)] + + pdf = _bind(tmp_path, _script(pages, layout="scroll"), 3) + + (page,) = pdf.pages + assert len(page.images) == 3 + width, height = float(page.mediabox.width), float(page.mediabox.height) + assert height > 3 * width * 1.5 + + def _page(colour: str, size: tuple[int, int]) -> bytes: out = io.BytesIO() Image.new("RGB", size, colour).save(out, "PNG") diff --git a/www/content/docs/v2/agent.mdx b/www/content/docs/v2/agent.mdx index 6644e6dd7c..0044bf0775 100644 --- a/www/content/docs/v2/agent.mdx +++ b/www/content/docs/v2/agent.mdx @@ -33,7 +33,7 @@ Documents the agent made are changed in its chat, not with Studio's **Refine**. Ask for a picture, an infographic or a document with a picture in it. The agent reads what it needs from your sources, writes the prompt itself and draws with the model in **Settings › Image gen**. To change a picture, ask in the same chat, and it makes its next version. Each picture appears in [Studio](/docs/v2/studio). -Ask for a comic, a storyboard or a picture book and the agent plans the pages, tells you how many it will draw, and asks once before it starts. It draws one picture per page, keeps the characters from page to page, and puts the pages together in one PDF. With an image model that can't take a picture in, each page is drawn from a description of the characters, so they may look a little different from page to page. +Ask for a comic, a manga, a webtoon, a comic strip, a storyboard or a picture book and the agent plans the pages, tells you how many it will draw, and asks once before it starts. It draws one picture per page, keeps the characters from page to page, and puts the pages together in one PDF. With an image model that can't take a picture in, each page is drawn from a description of the characters, so they may look a little different from page to page. - **It draws only when you ask.** A remote model bills each picture. A model on a computer without a graphics card takes several minutes each. - **A slow picture may finish after the agent replies.** It appears in Studio when it's ready. A document that uses it, such as a comic's PDF, waits for it and appears in Studio once its pictures are drawn. From a5cb98ad616f3b9d53a5a38566679d22240ccc84 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 01:27:38 +0530 Subject: [PATCH 09/33] feat(local): mark image models that edit with the Image editing badge, not an Edits label Beside Image generation it reads as one of the model's types and needs no tooltip, so the Edits component, its tooltip and its strings go. --- docs/architecture/studio.md | 2 +- docs/proposals/pictures/05-in-the-app.md | 5 +-- docs/proposals/pictures/README.md | 4 +- .../models/pictures-in/edits-badge.test.tsx | 30 ------------- .../models/pictures-in/edits-badge.tsx | 45 ------------------- .../models/remote/models/server-models.tsx | 17 +++---- surfsense_local/frontend/translations/de.json | 3 -- surfsense_local/frontend/translations/en.json | 3 -- surfsense_local/frontend/translations/es.json | 3 -- surfsense_local/frontend/translations/fr.json | 3 -- surfsense_local/frontend/translations/hi.json | 3 -- surfsense_local/frontend/translations/ja.json | 3 -- surfsense_local/frontend/translations/ko.json | 3 -- .../frontend/translations/pt-BR.json | 3 -- surfsense_local/frontend/translations/ru.json | 3 -- .../frontend/translations/zh-CN.json | 3 -- www/content/docs/v2/studio.mdx | 2 +- 17 files changed, 10 insertions(+), 125 deletions(-) delete mode 100644 surfsense_local/frontend/src/features/models/pictures-in/edits-badge.test.tsx delete mode 100644 surfsense_local/frontend/src/features/models/pictures-in/edits-badge.tsx diff --git a/docs/architecture/studio.md b/docs/architecture/studio.md index cf39dfdbbf..9dcb71775f 100644 --- a/docs/architecture/studio.md +++ b/docs/architecture/studio.md @@ -368,7 +368,7 @@ It needs only `image_gen`, its Regenerate paints the same prompt at the same sha - **Local:** inside the same text-runtime hold and model wait as a drawn image, the client posts sd-server's native `POST /sdcpp/v1/img_gen` with `{prompt, ref_images: [<data URL>], width, height, output_format: "png"}` and polls `GET /sdcpp/v1/jobs/{id}` each second, which also looks for a cancel ([`editor.py`](../../surfsense_local/backend/modules/llm/providers/sdcpp/editor.py)). A `failed` job, a `404`, `410` or `429`, or a server that stops answering fails the edit. Steps, guidance and sampler come from the launch, as for a drawn image; the size is the base's own, since an edit model launched at another size would crop or stretch it. sd.cpp picks the reference preset from the model's architecture, so the launch needs no `--ref-image-args`. - **Why `img_gen`.** At the pinned `master-869-07a85c7`, sd-server's `/v1/images/edits` makes the first upload the `init_image` as well as a reference, and `/sdapi/v1/img2img` takes `init_images`: both re-noise the picture at a strength of 0.75 rather than edit it. `ref_images` alone is how every edit model sd.cpp documents is run. - **Cancel:** a queued sd.cpp job is withdrawn with `POST /sdcpp/v1/jobs/{id}/cancel`, given 2 seconds, since the text runtime is still held while it is asked. A generating one cannot be (`cancel_generating: false`); the cancel ends the demand at once and Electron stops sd-server, as for a drawn image. -- **Which models edit:** a local model whose entry names `edit`, today FLUX.2 klein 4B, which fills both image slots from one download, so nothing reloads between drawing and editing; or a remote model that takes an image in and gives one out, by its connection's own list or the catalog ([`classifier.py`](../../surfsense_local/backend/modules/llm/catalog/remote/classifier.py)). Settings marks those with an **Edits** label, "Can edit pictures". Models that can only re-noise a picture, SD 1.5, SDXL, Z-Image Turbo, ERNIE-Image Turbo and LongCat-Image, name no `edit`: they cannot be chosen as the edit override, and a picture they drew is redrawn. +- **Which models edit:** a local model whose entry names `edit`, today FLUX.2 klein 4B, which fills both image slots from one download, so nothing reloads between drawing and editing; or a remote model that takes an image in and gives one out, by its connection's own list or the catalog ([`classifier.py`](../../surfsense_local/backend/modules/llm/catalog/remote/classifier.py)). Settings lists those with an **Image editing** badge beside **Image generation**. Models that can only re-noise a picture, SD 1.5, SDXL, Z-Image Turbo, ERNIE-Image Turbo and LongCat-Image, name no `edit`: they cannot be chosen as the edit override, and a picture they drew is redrawn. ## Known gaps diff --git a/docs/proposals/pictures/05-in-the-app.md b/docs/proposals/pictures/05-in-the-app.md index 4cbad846d9..3f73c641a5 100644 --- a/docs/proposals/pictures/05-in-the-app.md +++ b/docs/proposals/pictures/05-in-the-app.md @@ -11,8 +11,7 @@ ## Settings -- **The label.** In Settings › Image models, each model that takes pictures in carries a small **Edits** label beside its name, in the list of a server's models and in the list of local models to download ([`server-models.tsx`](../../../surfsense_local/frontend/src/features/models/remote/models/server-models.tsx), [`download-image-models.tsx`](../../../surfsense_local/frontend/src/features/models/local/image/download-image-models.tsx)). It is information only: choosing a model without it works. -- **The tooltip** on the label: "Can edit pictures". The label is focusable, so the tooltip opens on keyboard focus as on hover, and a screen reader reads the same words. It uses the app's own tooltip, as the rest of Settings does. +- **The badge.** In Settings › Image models, each model that takes pictures in carries the **Image editing** type badge beside **Image generation**, in the list of a server's models and in the list of local models to download ([`server-models.tsx`](../../../surfsense_local/frontend/src/features/models/remote/models/server-models.tsx), [`download-image-models.tsx`](../../../surfsense_local/frontend/src/features/models/local/image/download-image-models.tsx)). It is information only: choosing a model without it works. - **Image editing models** ([`image-edit-models-settings.tsx`](../../../surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx)) becomes an optional override, collapsed under the image models, titled "Use a different model for edits". Its summary reads "Same as the image model" when nothing is chosen there, in place of "No image editing model chosen" ([`in-use-summary.tsx`](../../../surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx)). ## Words @@ -21,8 +20,6 @@ Every new or changed string goes through the `translate` skill, into every langu | Where | Text | |---|---| -| Label | Edits | -| Tooltip | Can edit pictures | | Edit placeholder, a model that edits | Describe a change | | Edit placeholder, a redraw | Describe a change: the picture is drawn again | | A redrawn version | Drawn again with your change. | diff --git a/docs/proposals/pictures/README.md b/docs/proposals/pictures/README.md index d85be3a58e..bc51e0ad9d 100644 --- a/docs/proposals/pictures/README.md +++ b/docs/proposals/pictures/README.md @@ -37,7 +37,7 @@ Asked for a comic, the agent wrote a web page and drew the panels as SVG, or dre | Edit, with a model that takes pictures in | needs the edit slot filled | a real edit by the image model | | Edit, with a model that does not | "Needs an image editing model", and a link to Settings | the picture drawn again from its prompt and the change, said in one neutral line | | Editing through OpenRouter | 404 | works, through `/images` with `input_references` | -| Settings › Image models | no sign of which models edit | an **Edits** label on those that do, tooltip "Can edit pictures" | +| Settings › Image models | no sign of which models edit | an **Image editing** badge on those that do, beside **Image generation** | | "Make a comic about …" | a web page with SVG panels, or one picture | a plan, one confirmation, one picture per page, one PDF | | Characters across pages | redrawn from words each time | page 1 given to every later page as a reference | | A page's shape | the model's default | `portrait`, `landscape` or `square` | @@ -103,7 +103,7 @@ Steps 1 to 8 are built on `feat/comics`; what is now true is in [studio](../../a ## Left to do - The spikes in [image models](01-image-models.md#spikes-before-code): several pictures on OpenAI's `/images/edits`, and local references and shapes through sd-server. -- Settings: the edit override as its own section, retitled "Use a different model for edits", not yet folded under Image gen; the **Edits** label on server models only, not yet on local models to download. +- Settings: the edit override as its own section, retitled "Use a different model for edits", not yet folded under Image gen; the **Image editing** badge on server models only, not yet on local models to download. - The images guide's walkthrough and screenshots, which still choose a model for editing. ## Decisions diff --git a/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.test.tsx b/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.test.tsx deleted file mode 100644 index c29c1eeb65..0000000000 --- a/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.test.tsx +++ /dev/null @@ -1,30 +0,0 @@ -import { afterEach, describe, expect, it } from "vitest" -import { cleanup, screen } from "@testing-library/react" -import userEvent from "@testing-library/user-event" - -import { render } from "@/test-utils" - -import { EditsBadge } from "./edits-badge" - -afterEach(cleanup) - -describe("edits badge", () => { - it("names itself in full, since its tooltip is only seen", () => { - render(<EditsBadge />) - - expect(screen.getByText("Edits")).toBeTruthy() - expect( - screen.getByRole("note", { name: "Edits: can edit pictures" }) - ).toBeTruthy() - }) - - it("opens its tooltip on keyboard focus as on hover", async () => { - const user = userEvent.setup() - render(<EditsBadge />) - - await user.tab() - - expect(document.activeElement?.textContent).toBe("Edits") - expect(await screen.findByText("Can edit pictures")).toBeTruthy() - }) -}) diff --git a/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.tsx b/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.tsx deleted file mode 100644 index 48d39cb286..0000000000 --- a/surfsense_local/frontend/src/features/models/pictures-in/edits-badge.tsx +++ /dev/null @@ -1,45 +0,0 @@ -import { Badge } from "@/components/ui/badge" -import { - Tooltip, - TooltipContent, - TooltipTrigger, -} from "@/components/ui/tooltip" -import { intl } from "@/i18n/intl" - -/** Marks an image model that takes a picture in. Information only: any image - * model can be chosen, and one without it changes a picture by redrawing. */ -export function EditsBadge() { - const label = intl.formatMessage({ - id: "models_edits_badge_label", - defaultMessage: "Edits", - }) - const tip = intl.formatMessage({ - id: "models_edits_badge_tooltip", - defaultMessage: "Can edit pictures", - }) - return ( - <Tooltip> - <TooltipTrigger - render={ - // Focusable, so the tooltip opens from the keyboard too; named in - // full, since the tooltip is only seen. - <Badge - variant="secondary" - tabIndex={0} - role="note" - aria-label={intl.formatMessage( - { - id: "models_edits_badge_aria", - defaultMessage: "{label}: can edit pictures", - }, - { label } - )} - /> - } - > - {label} - </TooltipTrigger> - <TooltipContent side="top">{tip}</TooltipContent> - </Tooltip> - ) -} diff --git a/surfsense_local/frontend/src/features/models/remote/models/server-models.tsx b/surfsense_local/frontend/src/features/models/remote/models/server-models.tsx index 3797860e2f..bf54882118 100644 --- a/surfsense_local/frontend/src/features/models/remote/models/server-models.tsx +++ b/surfsense_local/frontend/src/features/models/remote/models/server-models.tsx @@ -22,7 +22,6 @@ import { cn } from "@/lib/utils" import { intl } from "@/i18n/intl" import type { ModelType } from "../../model-type" -import { EditsBadge } from "../../pictures-in/edits-badge" import type { ModelSelection } from "../../selection/api" import { useSelection } from "../../selection/use-selection" import { ServerVoices } from "../voices/server-voices" @@ -374,17 +373,11 @@ export function ServerModels({ })} </Badge> ) : ( - model.types.map((type) => - // A picture in as well as out: said the same - // way wherever an image model is listed. - type === "image_edit" ? ( - <EditsBadge key={type} /> - ) : ( - <Badge key={type} variant="secondary"> - {TYPE_LABELS[type]()} - </Badge> - ) - ) + model.types.map((type) => ( + <Badge key={type} variant="secondary"> + {TYPE_LABELS[type]()} + </Badge> + )) )} {model.reads_images ? ( <Badge variant="secondary"> diff --git a/surfsense_local/frontend/translations/de.json b/surfsense_local/frontend/translations/de.json index fdc73aaf0e..d4d325837d 100644 --- a/surfsense_local/frontend/translations/de.json +++ b/surfsense_local/frontend/translations/de.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "Die Anfrage ist fehlgeschlagen", "models_download_speech_unsupported_title": "Spracherkennungsmodelle laufen nicht auf diesem Computer", "models_download_video_unsupported_title": "Videomodelle laufen nicht auf diesem Computer", - "models_edits_badge_aria": "{label}: kann Bilder bearbeiten", - "models_edits_badge_label": "Bearbeitet", - "models_edits_badge_tooltip": "Kann Bilder bearbeiten", "models_elevenlabs_dialog_title": "ElevenLabs verbinden", "models_elevenlabs_form_cancel_button": "Abbrechen", "models_elevenlabs_form_connect_button": "ElevenLabs verbinden", diff --git a/surfsense_local/frontend/translations/en.json b/surfsense_local/frontend/translations/en.json index 789f385a85..41d75e13ea 100644 --- a/surfsense_local/frontend/translations/en.json +++ b/surfsense_local/frontend/translations/en.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "The request failed", "models_download_speech_unsupported_title": "Speech recognition models cannot run on this computer", "models_download_video_unsupported_title": "Video models cannot run on this computer", - "models_edits_badge_aria": "{label}: can edit pictures", - "models_edits_badge_label": "Edits", - "models_edits_badge_tooltip": "Can edit pictures", "models_elevenlabs_dialog_title": "Connect ElevenLabs", "models_elevenlabs_form_cancel_button": "Cancel", "models_elevenlabs_form_connect_button": "Connect ElevenLabs", diff --git a/surfsense_local/frontend/translations/es.json b/surfsense_local/frontend/translations/es.json index 0c9d2a724a..0e0f6dd0da 100644 --- a/surfsense_local/frontend/translations/es.json +++ b/surfsense_local/frontend/translations/es.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "La solicitud falló", "models_download_speech_unsupported_title": "Los modelos de reconocimiento de voz no pueden ejecutarse en este equipo", "models_download_video_unsupported_title": "Los modelos de video no pueden ejecutarse en este equipo", - "models_edits_badge_aria": "{label}: puede editar imágenes", - "models_edits_badge_label": "Edita", - "models_edits_badge_tooltip": "Puede editar imágenes", "models_elevenlabs_dialog_title": "Conectar ElevenLabs", "models_elevenlabs_form_cancel_button": "Cancelar", "models_elevenlabs_form_connect_button": "Conectar ElevenLabs", diff --git a/surfsense_local/frontend/translations/fr.json b/surfsense_local/frontend/translations/fr.json index 377fbc2970..77ac9735dc 100644 --- a/surfsense_local/frontend/translations/fr.json +++ b/surfsense_local/frontend/translations/fr.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "La requête a échoué", "models_download_speech_unsupported_title": "Les modèles de reconnaissance vocale ne peuvent pas fonctionner sur cet ordinateur", "models_download_video_unsupported_title": "Les modèles vidéo ne peuvent pas fonctionner sur cet ordinateur", - "models_edits_badge_aria": "{label} : peut modifier des images", - "models_edits_badge_label": "Modifie", - "models_edits_badge_tooltip": "Peut modifier des images", "models_elevenlabs_dialog_title": "Connecter ElevenLabs", "models_elevenlabs_form_cancel_button": "Annuler", "models_elevenlabs_form_connect_button": "Connecter ElevenLabs", diff --git a/surfsense_local/frontend/translations/hi.json b/surfsense_local/frontend/translations/hi.json index 1453bd7a08..24855c0776 100644 --- a/surfsense_local/frontend/translations/hi.json +++ b/surfsense_local/frontend/translations/hi.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "अनुरोध विफल रहा", "models_download_speech_unsupported_title": "स्पीच रिकग्निशन मॉडल इस कंप्यूटर पर नहीं चल सकते", "models_download_video_unsupported_title": "वीडियो मॉडल इस कंप्यूटर पर नहीं चल सकते", - "models_edits_badge_aria": "{label}: इमेज एडिट कर सकता है", - "models_edits_badge_label": "एडिटिंग", - "models_edits_badge_tooltip": "इमेज एडिट कर सकता है", "models_elevenlabs_dialog_title": "ElevenLabs कनेक्ट करें", "models_elevenlabs_form_cancel_button": "रद्द करें", "models_elevenlabs_form_connect_button": "ElevenLabs कनेक्ट करें", diff --git a/surfsense_local/frontend/translations/ja.json b/surfsense_local/frontend/translations/ja.json index f4eb1a0f4a..41f5b1d567 100644 --- a/surfsense_local/frontend/translations/ja.json +++ b/surfsense_local/frontend/translations/ja.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "リクエストに失敗しました", "models_download_speech_unsupported_title": "このコンピューターでは音声認識モデルを実行できません", "models_download_video_unsupported_title": "このコンピューターでは動画モデルを実行できません", - "models_edits_badge_aria": "{label}:画像を編集できます", - "models_edits_badge_label": "編集可", - "models_edits_badge_tooltip": "画像を編集できます", "models_elevenlabs_dialog_title": "ElevenLabsに接続", "models_elevenlabs_form_cancel_button": "キャンセル", "models_elevenlabs_form_connect_button": "ElevenLabsに接続", diff --git a/surfsense_local/frontend/translations/ko.json b/surfsense_local/frontend/translations/ko.json index e9a2dc8678..6fcba4a327 100644 --- a/surfsense_local/frontend/translations/ko.json +++ b/surfsense_local/frontend/translations/ko.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "요청이 실패했습니다", "models_download_speech_unsupported_title": "이 컴퓨터에서는 음성 인식 모델을 실행할 수 없습니다", "models_download_video_unsupported_title": "이 컴퓨터에서는 동영상 모델을 실행할 수 없습니다", - "models_edits_badge_aria": "{label}: 이미지를 편집할 수 있습니다", - "models_edits_badge_label": "편집 가능", - "models_edits_badge_tooltip": "이미지를 편집할 수 있습니다", "models_elevenlabs_dialog_title": "ElevenLabs 연결", "models_elevenlabs_form_cancel_button": "취소", "models_elevenlabs_form_connect_button": "ElevenLabs 연결", diff --git a/surfsense_local/frontend/translations/pt-BR.json b/surfsense_local/frontend/translations/pt-BR.json index f88b1e6472..56e3227918 100644 --- a/surfsense_local/frontend/translations/pt-BR.json +++ b/surfsense_local/frontend/translations/pt-BR.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "A solicitação falhou", "models_download_speech_unsupported_title": "Modelos de reconhecimento de voz não podem ser executados neste computador", "models_download_video_unsupported_title": "Modelos de vídeo não podem ser executados neste computador", - "models_edits_badge_aria": "{label}: pode editar imagens", - "models_edits_badge_label": "Edita", - "models_edits_badge_tooltip": "Pode editar imagens", "models_elevenlabs_dialog_title": "Conectar ElevenLabs", "models_elevenlabs_form_cancel_button": "Cancelar", "models_elevenlabs_form_connect_button": "Conectar ElevenLabs", diff --git a/surfsense_local/frontend/translations/ru.json b/surfsense_local/frontend/translations/ru.json index 6d3af6ac51..23f706ef51 100644 --- a/surfsense_local/frontend/translations/ru.json +++ b/surfsense_local/frontend/translations/ru.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "Запрос не выполнен", "models_download_speech_unsupported_title": "Модели распознавания речи не запускаются на этом компьютере", "models_download_video_unsupported_title": "Модели для видео не запускаются на этом компьютере", - "models_edits_badge_aria": "{label}: может редактировать изображения", - "models_edits_badge_label": "Редактирует", - "models_edits_badge_tooltip": "Может редактировать изображения", "models_elevenlabs_dialog_title": "Подключить ElevenLabs", "models_elevenlabs_form_cancel_button": "Отмена", "models_elevenlabs_form_connect_button": "Подключить ElevenLabs", diff --git a/surfsense_local/frontend/translations/zh-CN.json b/surfsense_local/frontend/translations/zh-CN.json index a2501de943..ec3e1b3562 100644 --- a/surfsense_local/frontend/translations/zh-CN.json +++ b/surfsense_local/frontend/translations/zh-CN.json @@ -897,9 +897,6 @@ "models_download_speech_request_error": "请求失败", "models_download_speech_unsupported_title": "这台电脑无法运行语音识别模型", "models_download_video_unsupported_title": "这台电脑无法运行视频模型", - "models_edits_badge_aria": "{label}:可以编辑图像", - "models_edits_badge_label": "可编辑", - "models_edits_badge_tooltip": "可以编辑图像", "models_elevenlabs_dialog_title": "连接 ElevenLabs", "models_elevenlabs_form_cancel_button": "取消", "models_elevenlabs_form_connect_button": "连接 ElevenLabs", diff --git a/www/content/docs/v2/studio.mdx b/www/content/docs/v2/studio.mdx index 6c5fbc60bf..7b6fdf07e6 100644 --- a/www/content/docs/v2/studio.mdx +++ b/www/content/docs/v2/studio.mdx @@ -96,7 +96,7 @@ Over a ready image or infographic, the same round button opens an edit box. Desc Each edit starts from the version you have open, so edits chain, and every earlier version stays. There's no way to select one region of the picture yet: an edit applies to the whole image. -Some image models can't take a picture in. With one of those, the box says "Describe a change: the picture is drawn again": Studio draws the picture again from what it was drawn from and your change, so the rest of the picture may change too. Image models that edit are marked **Edits** in Settings. +Some image models can't take a picture in. With one of those, the box says "Describe a change: the picture is drawn again": Studio draws the picture again from what it was drawn from and your change, so the rest of the picture may change too. Image models that edit are marked **Image editing** in Settings. <Callout type="info"> Editing an image on a computer without a graphics card is slow. In testing, From 06f4ac8c3213719942dd7e04667d9b37eb07c2c9 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:18:45 +0530 Subject: [PATCH 10/33] feat(local): Studio offers infographic as a type of the one Image card The dialog shows one Image card whose Type switch picks image or infographic and sends that format's key, so edits and the agent's tool see no change. Each format's options now come from its own hook in format-options/, the podcast brief among them. --- .../studio/format-options/format-options.ts | 11 ++++ .../format-options/picture-type-options.tsx | 23 +++++++ .../format-options/picture-type-picker.tsx | 63 +++++++++++++++++++ .../studio/format-options/podcast-options.tsx | 46 ++++++++++++++ .../format-options/use-format-options.ts | 17 +++++ .../src/features/studio/picture-types.ts | 20 ++++++ .../src/features/studio/studio-panel.test.tsx | 63 ++++++++++++++++++- .../src/features/studio/studio-panel.tsx | 53 ++++------------ surfsense_local/frontend/translations/de.json | 2 +- surfsense_local/frontend/translations/en.json | 2 +- surfsense_local/frontend/translations/es.json | 2 +- surfsense_local/frontend/translations/fr.json | 2 +- surfsense_local/frontend/translations/hi.json | 2 +- surfsense_local/frontend/translations/ja.json | 2 +- surfsense_local/frontend/translations/ko.json | 2 +- .../frontend/translations/pt-BR.json | 2 +- surfsense_local/frontend/translations/ru.json | 2 +- .../frontend/translations/zh-CN.json | 2 +- 18 files changed, 263 insertions(+), 53 deletions(-) create mode 100644 surfsense_local/frontend/src/features/studio/format-options/format-options.ts create mode 100644 surfsense_local/frontend/src/features/studio/format-options/picture-type-options.tsx create mode 100644 surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx create mode 100644 surfsense_local/frontend/src/features/studio/format-options/podcast-options.tsx create mode 100644 surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts create mode 100644 surfsense_local/frontend/src/features/studio/picture-types.ts diff --git a/surfsense_local/frontend/src/features/studio/format-options/format-options.ts b/surfsense_local/frontend/src/features/studio/format-options/format-options.ts new file mode 100644 index 0000000000..ea3c445ed8 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/format-options.ts @@ -0,0 +1,11 @@ +import type { ReactNode } from "react" + +import type { StudioJobCreate } from "../api" + +/** What a format adds to the composer and to the job it sends. */ +export type FormatOptions = { + panel: ReactNode + job: Pick<StudioJobCreate, "format" | "options"> + // False while the format still needs something before it can generate. + ready: boolean +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/picture-type-options.tsx b/surfsense_local/frontend/src/features/studio/format-options/picture-type-options.tsx new file mode 100644 index 0000000000..5b76706c34 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/picture-type-options.tsx @@ -0,0 +1,23 @@ +import { useState } from "react" + +import type { StudioFormat } from "../api" +import { pictureTypes } from "../picture-types" +import type { FormatOptions } from "./format-options" +import { PictureTypePicker } from "./picture-type-picker" + +/** The Image card makes the picture type chosen in its dialog, Image first. */ +export function usePictureTypeOptions( + format: string, + catalog: StudioFormat[] +): FormatOptions | null { + const [type, setType] = useState(format) + if (format !== "image") return null + + const types = pictureTypes(catalog) + if (types.length < 2) return null + return { + panel: <PictureTypePicker types={types} value={type} onChange={setType} />, + job: { format: type }, + ready: types.some((entry) => entry.key === type && entry.available), + } +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx new file mode 100644 index 0000000000..b522787b09 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx @@ -0,0 +1,63 @@ +import { SegmentedControl } from "@/components/ui/segmented-control" +import { intl } from "@/i18n/intl" +import { cn } from "@/lib/utils" + +import type { StudioFormat } from "../api" +import { formatLabel } from "../studio-formats" + +/** Image or infographic, as one choice. Both need the same models, so a type is + * closed only if the server ever says so. */ +export function PictureTypePicker({ + types, + value, + onChange, +}: { + types: StudioFormat[] + value: string + onChange: (key: string) => void +}) { + const label = intl.formatMessage({ + id: "studio_picture_type_label", + defaultMessage: "Type", + }) + return ( + <div className="space-y-2"> + <p className="text-xs font-medium text-muted-foreground">{label}</p> + <SegmentedControl + count={types.length} + selectedIndex={Math.max( + 0, + types.findIndex((entry) => entry.key === value) + )} + role="radiogroup" + aria-label={label} + className="flex w-full" + > + {types.map((entry) => ( + <label + key={entry.key} + className={cn( + "relative min-w-0 flex-1", + entry.available + ? "cursor-pointer" + : "cursor-not-allowed opacity-50" + )} + > + <input + type="radio" + name="studio-picture-type" + value={entry.key} + checked={value === entry.key} + disabled={!entry.available} + className="peer sr-only" + onChange={() => onChange(entry.key)} + /> + <span className="relative flex h-8 items-center justify-center truncate rounded-md px-3 text-sm peer-focus-visible:z-10 peer-focus-visible:ring-2 peer-focus-visible:ring-ring peer-focus-visible:outline-none"> + {formatLabel(entry)} + </span> + </label> + ))} + </SegmentedControl> + </div> + ) +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/podcast-options.tsx b/surfsense_local/frontend/src/features/studio/format-options/podcast-options.tsx new file mode 100644 index 0000000000..5f2a5163b1 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/podcast-options.tsx @@ -0,0 +1,46 @@ +import { intl } from "@/i18n/intl" + +import { PodcastBriefForm } from "../podcast-brief-form" +import { usePodcastBrief } from "../use-podcast-brief" +import type { FormatOptions } from "./format-options" + +/** A podcast is generated from its reviewed brief, so it waits for the brief. A + * server model with no voices yet can voice nothing: set them up first. */ +export function usePodcastOptions( + format: string, + workspaceId: number, + onSetUpVoices: () => void +): FormatOptions | null { + const podcast = usePodcastBrief(format === "podcast" ? workspaceId : null) + if (format !== "podcast") return null + + const needsVoices = + podcast.opened?.voices_source === "saved" && + podcast.opened.voices.length === 0 + const panel = + podcast.brief && podcast.opened ? ( + <PodcastBriefForm + brief={podcast.brief} + voices={podcast.opened.voices} + languages={podcast.opened.languages} + voicesSource={podcast.opened.voices_source} + voicesProblem={podcast.opened.problem} + voicedBy={podcast.opened.voiced_by} + onSetUpVoices={onSetUpVoices} + onChange={podcast.setBrief} + /> + ) : ( + <p className="text-sm text-muted-foreground"> + {podcast.error ?? + intl.formatMessage({ + id: "studio_composer_brief_status", + defaultMessage: "Preparing the brief…", + })} + </p> + ) + return { + panel, + job: { format, options: podcast.brief ?? undefined }, + ready: podcast.brief != null && !needsVoices, + } +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts b/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts new file mode 100644 index 0000000000..044cc7f4e9 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts @@ -0,0 +1,17 @@ +import type { StudioFormat } from "../api" +import type { FormatOptions } from "./format-options" +import { usePictureTypeOptions } from "./picture-type-options" +import { usePodcastOptions } from "./podcast-options" + +/** The chosen format's own options, or none: a format with options adds its hook here. */ +export function useFormatOptions( + format: string, + workspaceId: number, + catalog: StudioFormat[], + onSetUpVoices: () => void +): FormatOptions { + // Every hook runs on every render; each answers only for its own format. + const podcast = usePodcastOptions(format, workspaceId, onSetUpVoices) + const picture = usePictureTypeOptions(format, catalog) + return podcast ?? picture ?? { panel: null, job: { format }, ready: true } +} diff --git a/surfsense_local/frontend/src/features/studio/picture-types.ts b/surfsense_local/frontend/src/features/studio/picture-types.ts new file mode 100644 index 0000000000..b9cb393fef --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/picture-types.ts @@ -0,0 +1,20 @@ +import type { StudioFormat } from "./api" + +// An infographic is offered as a type of image, not a card of its own. Each type +// still makes its own format, so its edits and the agent's tool see no change. +export const PICTURE_TYPES = ["image", "infographic"] as const + +export type PictureType = (typeof PICTURE_TYPES)[number] + +/** The catalog's entry for each picture type it offers, in the picker's order. */ +export function pictureTypes(catalog: StudioFormat[]): StudioFormat[] { + return PICTURE_TYPES.flatMap( + (key) => catalog.find((entry) => entry.key === key) ?? [] + ) +} + +/** The cards Studio shows: an infographic only when there is no image to hold it. */ +export function formatCards(catalog: StudioFormat[]): StudioFormat[] { + if (!catalog.some((entry) => entry.key === "image")) return catalog + return catalog.filter((entry) => entry.key !== "infographic") +} diff --git a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx index 01e1314fb0..c963b4e648 100644 --- a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx +++ b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx @@ -181,7 +181,8 @@ describe("studio panel", () => { ) expect(screen.getByRole("button", { name: "Markdown" })).toBeTruthy() - expect(screen.getByRole("button", { name: "Infographic" })).toBeTruthy() + expect(screen.getByRole("button", { name: "Image" })).toBeTruthy() + expect(screen.queryByRole("button", { name: "Infographic" })).toBeNull() expect(document.querySelector("[data-slot=skeleton]")).toBeNull() }) @@ -290,6 +291,66 @@ describe("studio panel", () => { ).toBeTruthy() }) + it("offers an infographic as a type of image, and makes the type chosen", async () => { + const picture = (key: string, label: string) => ({ + key, + label, + requires_model_types: ["image_gen", "text_gen"], + available: true, + unavailable_reason: null, + }) + const fetchMock = vi.fn( + async (input: RequestInfo | URL, init?: RequestInit) => { + const path = String(input) + if (path === "/workspaces/1/studio/formats") { + return Response.json([ + picture("image", "Image"), + picture("infographic", "Infographic"), + ]) + } + if (path === "/workspaces/1/studio/jobs" && init?.method === "POST") { + return Response.json( + { ...pendingArtifact, format: "infographic", title: "Infographic" }, + { status: 201 } + ) + } + if (path === "/workspaces/1/artifacts") return Response.json([]) + return Response.json({ detail: "not found" }, { status: 404 }) + } + ) + vi.stubGlobal("fetch", fetchMock) + const user = userEvent.setup() + + renderStudio() + + const formats = screen.getByRole("region", { name: "Studio formats" }) + await within(formats).findByRole("button", { name: "Image" }) + expect( + within(formats).queryByRole("button", { name: "Infographic" }) + ).toBeNull() + + await user.click(within(formats).getByRole("button", { name: "Image" })) + const dialog = screen.getByRole("dialog", { name: "Image" }) + const type = within(dialog).getByRole("radiogroup", { name: "Type" }) + expect( + (within(type).getByRole("radio", { name: "Image" }) as HTMLInputElement) + .checked + ).toBe(true) + await user.click(within(type).getByRole("radio", { name: "Infographic" })) + await user.click(within(dialog).getByRole("button", { name: /Generate/ })) + + const jobCall = await vi.waitFor(() => + fetchMock.mock.calls.find( + ([path, init]) => + path === "/workspaces/1/studio/jobs" && init?.method === "POST" + ) + ) + expect(JSON.parse(String(jobCall?.[1]?.body))).toEqual({ + format: "infographic", + document_ids: [4], + }) + }) + it("opens the podcast brief for review and sends it with the job", async () => { const brief = { language: "en-US", diff --git a/surfsense_local/frontend/src/features/studio/studio-panel.tsx b/surfsense_local/frontend/src/features/studio/studio-panel.tsx index 7a0e553f3a..d372105fb4 100644 --- a/surfsense_local/frontend/src/features/studio/studio-panel.tsx +++ b/surfsense_local/frontend/src/features/studio/studio-panel.tsx @@ -30,10 +30,10 @@ import { intl } from "@/i18n/intl" import { cn } from "@/lib/utils" import type { StudioFormat, StudioJobCreate } from "./api" -import { PodcastBriefForm } from "./podcast-brief-form" +import { useFormatOptions } from "./format-options/use-format-options" +import { formatCards } from "./picture-types" import { FORMAT_ICONS, formatLabel, studioCatalog } from "./studio-formats" import { translatedUnavailable } from "./studio-unavailable-text" -import { usePodcastBrief } from "./use-podcast-brief" const FORMAT_HINTS: Record<string, () => string> = { summary: () => @@ -92,11 +92,6 @@ const FORMAT_HINTS: Record<string, () => string> = { id: "studio_format_image_tooltip", defaultMessage: "Generate an AI image based on your sources", }), - infographic: () => - intl.formatMessage({ - id: "studio_format_infographic_tooltip", - defaultMessage: "Generate an AI infographic based on your sources", - }), } function catalogFormats(formats: StudioFormat[]) { @@ -136,6 +131,7 @@ function formatHint(entry: StudioFormat) { function Composer({ workspaceId, format, + catalog, documents, selectedDocumentIds, sourceScope, @@ -147,6 +143,7 @@ function Composer({ }: { workspaceId: number format: string + catalog: StudioFormat[] documents: WorkspaceDocument[] selectedDocumentIds: number[] sourceScope?: SourceScope @@ -160,17 +157,9 @@ function Composer({ const selected = new Set(selectedDocumentIds) const [prompt, setPrompt] = useState("") const [view, setView] = useState<"main" | "sources">("main") - const podcast = usePodcastBrief(format === "podcast" ? workspaceId : null) + const options = useFormatOptions(format, workspaceId, catalog, onSetUpVoices) const allSelected = ready.length > 0 && selected.size === ready.length - - // A podcast is generated from its reviewed brief, so it waits for the brief. - // A server model with no voices yet can voice nothing: set them up first. - const needsVoices = - podcast.opened?.voices_source === "saved" && - podcast.opened.voices.length === 0 - const briefReady = - format !== "podcast" || (podcast.brief != null && !needsVoices) - const canGenerate = selected.size > 0 && !isCreating && briefReady + const canGenerate = selected.size > 0 && !isCreating && options.ready return ( // A taller dialog for every format: Generate stays at the bottom, and the @@ -186,28 +175,7 @@ function Composer({ aria-hidden={view !== "main"} > <div className="space-y-3"> - {format === "podcast" ? ( - podcast.brief && podcast.opened ? ( - <PodcastBriefForm - brief={podcast.brief} - voices={podcast.opened.voices} - languages={podcast.opened.languages} - voicesSource={podcast.opened.voices_source} - voicesProblem={podcast.opened.problem} - voicedBy={podcast.opened.voiced_by} - onSetUpVoices={onSetUpVoices} - onChange={podcast.setBrief} - /> - ) : ( - <p className="text-sm text-muted-foreground"> - {podcast.error ?? - intl.formatMessage({ - id: "studio_composer_brief_status", - defaultMessage: "Preparing the brief…", - })} - </p> - ) - ) : null} + {options.panel} {ready.length === 0 ? ( <div className="space-y-2"> @@ -275,11 +243,10 @@ function Composer({ disabled={!canGenerate} onClick={() => { onGenerate({ - format, + ...options.job, document_ids: [...selected], ...(sourceScope ? { source_scope: sourceScope } : {}), prompt: prompt.trim() || undefined, - options: podcast.brief ?? undefined, }) }} > @@ -430,6 +397,7 @@ export const StudioPanel = memo(function StudioPanel({ const [format, setFormat] = useState<string | null>(null) const [formatOpen, setFormatOpen] = useState(false) const catalog = useMemo(() => catalogFormats(formats), [formats]) + const cards = useMemo(() => formatCards(catalog), [catalog]) const selectedFormat = catalog.find((entry) => entry.key === format) const selectFormat = useCallback((key: string) => { setFormat(key) @@ -458,7 +426,7 @@ export const StudioPanel = memo(function StudioPanel({ })} > <div className="grid grid-cols-3 gap-1.5"> - {catalog.map((entry) => ( + {cards.map((entry) => ( <FormatCard key={entry.key} entry={entry} @@ -489,6 +457,7 @@ export const StudioPanel = memo(function StudioPanel({ key={selectedFormat.key} workspaceId={workspaceId} format={selectedFormat.key} + catalog={catalog} documents={documents} selectedDocumentIds={selectedDocumentIds} sourceScope={sourceScope} diff --git a/surfsense_local/frontend/translations/de.json b/surfsense_local/frontend/translations/de.json index d4d325837d..8561fc1a46 100644 --- a/surfsense_local/frontend/translations/de.json +++ b/surfsense_local/frontend/translations/de.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "Bild", "studio_format_image_tooltip": "Bild mit KI aus deinen Quellen erstellen", "studio_format_infographic_label": "Infografik", - "studio_format_infographic_tooltip": "Infografik mit KI aus deinen Quellen erstellen", "studio_format_mindmap_label": "Mindmap", "studio_format_mindmap_tooltip": "Mindmap mit KI aus deinen Quellen erstellen", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "PDF konnte nicht geladen werden", "studio_pdf_viewer_zoom_in_aria": "Vergrößern", "studio_pdf_viewer_zoom_out_aria": "Verkleinern", + "studio_picture_type_label": "Typ", "studio_podcast_brief_add_speaker_button": "Sprecher hinzufügen", "studio_podcast_brief_language_label": "Sprache", "studio_podcast_brief_length_label": "Länge", diff --git a/surfsense_local/frontend/translations/en.json b/surfsense_local/frontend/translations/en.json index 41d75e13ea..6c1ab77359 100644 --- a/surfsense_local/frontend/translations/en.json +++ b/surfsense_local/frontend/translations/en.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "Image", "studio_format_image_tooltip": "Generate an AI image based on your sources", "studio_format_infographic_label": "Infographic", - "studio_format_infographic_tooltip": "Generate an AI infographic based on your sources", "studio_format_mindmap_label": "Mind map", "studio_format_mindmap_tooltip": "Generate an AI mind map based on your sources", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "Failed to load PDF", "studio_pdf_viewer_zoom_in_aria": "Zoom in", "studio_pdf_viewer_zoom_out_aria": "Zoom out", + "studio_picture_type_label": "Type", "studio_podcast_brief_add_speaker_button": "Add speaker", "studio_podcast_brief_language_label": "Language", "studio_podcast_brief_length_label": "Length", diff --git a/surfsense_local/frontend/translations/es.json b/surfsense_local/frontend/translations/es.json index 0e0f6dd0da..ada53c52f6 100644 --- a/surfsense_local/frontend/translations/es.json +++ b/surfsense_local/frontend/translations/es.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "Imagen", "studio_format_image_tooltip": "Genera una imagen con IA a partir de tus fuentes", "studio_format_infographic_label": "Infografía", - "studio_format_infographic_tooltip": "Genera una infografía con IA a partir de tus fuentes", "studio_format_mindmap_label": "Mapa mental", "studio_format_mindmap_tooltip": "Genera un mapa mental con IA a partir de tus fuentes", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "No se pudo cargar el PDF", "studio_pdf_viewer_zoom_in_aria": "Acercar", "studio_pdf_viewer_zoom_out_aria": "Alejar", + "studio_picture_type_label": "Tipo", "studio_podcast_brief_add_speaker_button": "Añadir participante", "studio_podcast_brief_language_label": "Idioma", "studio_podcast_brief_length_label": "Duración", diff --git a/surfsense_local/frontend/translations/fr.json b/surfsense_local/frontend/translations/fr.json index 77ac9735dc..9644bce83e 100644 --- a/surfsense_local/frontend/translations/fr.json +++ b/surfsense_local/frontend/translations/fr.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "Image", "studio_format_image_tooltip": "Générer une image par IA à partir de vos sources", "studio_format_infographic_label": "Infographie", - "studio_format_infographic_tooltip": "Générer une infographie par IA à partir de vos sources", "studio_format_mindmap_label": "Carte mentale", "studio_format_mindmap_tooltip": "Générer une carte mentale par IA à partir de vos sources", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "Impossible de charger le PDF", "studio_pdf_viewer_zoom_in_aria": "Zoom avant", "studio_pdf_viewer_zoom_out_aria": "Zoom arrière", + "studio_picture_type_label": "Type", "studio_podcast_brief_add_speaker_button": "Ajouter un intervenant", "studio_podcast_brief_language_label": "Langue", "studio_podcast_brief_length_label": "Durée", diff --git a/surfsense_local/frontend/translations/hi.json b/surfsense_local/frontend/translations/hi.json index 24855c0776..5072804a5d 100644 --- a/surfsense_local/frontend/translations/hi.json +++ b/surfsense_local/frontend/translations/hi.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "इमेज", "studio_format_image_tooltip": "अपने स्रोतों से AI इमेज जनरेट करें", "studio_format_infographic_label": "इन्फ़ोग्राफ़िक", - "studio_format_infographic_tooltip": "अपने स्रोतों से AI इन्फ़ोग्राफ़िक जनरेट करें", "studio_format_mindmap_label": "माइंड मैप", "studio_format_mindmap_tooltip": "अपने स्रोतों से AI माइंड मैप जनरेट करें", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "PDF लोड नहीं हो सका", "studio_pdf_viewer_zoom_in_aria": "ज़ूम इन", "studio_pdf_viewer_zoom_out_aria": "ज़ूम आउट", + "studio_picture_type_label": "प्रकार", "studio_podcast_brief_add_speaker_button": "स्पीकर जोड़ें", "studio_podcast_brief_language_label": "भाषा", "studio_podcast_brief_length_label": "लंबाई", diff --git a/surfsense_local/frontend/translations/ja.json b/surfsense_local/frontend/translations/ja.json index 41f5b1d567..1e2653d5a5 100644 --- a/surfsense_local/frontend/translations/ja.json +++ b/surfsense_local/frontend/translations/ja.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "画像", "studio_format_image_tooltip": "ソースをもとにAIで画像を生成", "studio_format_infographic_label": "インフォグラフィック", - "studio_format_infographic_tooltip": "ソースをもとにAIでインフォグラフィックを生成", "studio_format_mindmap_label": "マインドマップ", "studio_format_mindmap_tooltip": "ソースをもとにAIでマインドマップを生成", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "PDFを読み込めませんでした", "studio_pdf_viewer_zoom_in_aria": "拡大", "studio_pdf_viewer_zoom_out_aria": "縮小", + "studio_picture_type_label": "種類", "studio_podcast_brief_add_speaker_button": "話者を追加", "studio_podcast_brief_language_label": "言語", "studio_podcast_brief_length_label": "長さ", diff --git a/surfsense_local/frontend/translations/ko.json b/surfsense_local/frontend/translations/ko.json index 6fcba4a327..2f793c82c7 100644 --- a/surfsense_local/frontend/translations/ko.json +++ b/surfsense_local/frontend/translations/ko.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "이미지", "studio_format_image_tooltip": "소스를 바탕으로 AI 이미지 생성", "studio_format_infographic_label": "인포그래픽", - "studio_format_infographic_tooltip": "소스를 바탕으로 AI 인포그래픽 생성", "studio_format_mindmap_label": "마인드맵", "studio_format_mindmap_tooltip": "소스를 바탕으로 AI 마인드맵 생성", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "PDF를 불러오지 못했습니다", "studio_pdf_viewer_zoom_in_aria": "확대", "studio_pdf_viewer_zoom_out_aria": "축소", + "studio_picture_type_label": "유형", "studio_podcast_brief_add_speaker_button": "화자 추가", "studio_podcast_brief_language_label": "언어", "studio_podcast_brief_length_label": "길이", diff --git a/surfsense_local/frontend/translations/pt-BR.json b/surfsense_local/frontend/translations/pt-BR.json index 56e3227918..3e30d0108c 100644 --- a/surfsense_local/frontend/translations/pt-BR.json +++ b/surfsense_local/frontend/translations/pt-BR.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "Imagem", "studio_format_image_tooltip": "Gerar uma imagem com IA a partir das suas fontes", "studio_format_infographic_label": "Infográfico", - "studio_format_infographic_tooltip": "Gerar um infográfico com IA a partir das suas fontes", "studio_format_mindmap_label": "Mapa mental", "studio_format_mindmap_tooltip": "Gerar um mapa mental com IA a partir das suas fontes", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "Não foi possível carregar o PDF", "studio_pdf_viewer_zoom_in_aria": "Ampliar", "studio_pdf_viewer_zoom_out_aria": "Reduzir", + "studio_picture_type_label": "Tipo", "studio_podcast_brief_add_speaker_button": "Adicionar locutor", "studio_podcast_brief_language_label": "Idioma", "studio_podcast_brief_length_label": "Duração", diff --git a/surfsense_local/frontend/translations/ru.json b/surfsense_local/frontend/translations/ru.json index 23f706ef51..300e59d042 100644 --- a/surfsense_local/frontend/translations/ru.json +++ b/surfsense_local/frontend/translations/ru.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "Изображение", "studio_format_image_tooltip": "Создать с помощью ИИ изображение на основе ваших источников", "studio_format_infographic_label": "Инфографика", - "studio_format_infographic_tooltip": "Создать с помощью ИИ инфографику на основе ваших источников", "studio_format_mindmap_label": "Интеллект-карта", "studio_format_mindmap_tooltip": "Создать с помощью ИИ интеллект-карту на основе ваших источников", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "Не удалось загрузить PDF", "studio_pdf_viewer_zoom_in_aria": "Увеличить", "studio_pdf_viewer_zoom_out_aria": "Уменьшить", + "studio_picture_type_label": "Тип", "studio_podcast_brief_add_speaker_button": "Добавить спикера", "studio_podcast_brief_language_label": "Язык", "studio_podcast_brief_length_label": "Длительность", diff --git a/surfsense_local/frontend/translations/zh-CN.json b/surfsense_local/frontend/translations/zh-CN.json index ec3e1b3562..8b9537450e 100644 --- a/surfsense_local/frontend/translations/zh-CN.json +++ b/surfsense_local/frontend/translations/zh-CN.json @@ -1968,7 +1968,6 @@ "studio_format_image_label": "图像", "studio_format_image_tooltip": "根据你的来源用 AI 生成图像", "studio_format_infographic_label": "信息图", - "studio_format_infographic_tooltip": "根据你的来源用 AI 生成信息图", "studio_format_mindmap_label": "思维导图", "studio_format_mindmap_tooltip": "根据你的来源用 AI 生成思维导图", "studio_format_pdf_label": "PDF", @@ -2006,6 +2005,7 @@ "studio_pdf_viewer_unknown_error": "无法加载 PDF", "studio_pdf_viewer_zoom_in_aria": "放大", "studio_pdf_viewer_zoom_out_aria": "缩小", + "studio_picture_type_label": "类型", "studio_podcast_brief_add_speaker_button": "添加说话人", "studio_podcast_brief_language_label": "语言", "studio_podcast_brief_length_label": "时长", From 2cea31f2c18b6f6d8886744d986ac208fd8a01e3 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:18:57 +0530 Subject: [PATCH 11/33] feat(local): the agent asks make_image for an infographic by its type make_image's style argument is now type, image or infographic, as Studio's dialog names it; the pictures skill says when to choose infographic. --- .../agent/skills/surfsense-pictures/SKILL.md | 2 +- .../modules/agent/tool_endpoint/make_image.py | 23 +++++++++++-------- .../integration/agent/test_make_image.py | 8 +++---- 3 files changed, 18 insertions(+), 15 deletions(-) diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md index 19651c42e7..835e2250b9 100644 --- a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md @@ -18,7 +18,7 @@ description: Draw pictures with surfsense_make_image - one picture or infographi 1. Read any sources the picture draws on first. 2. Write one self-contained prompt: the subject, the setting, the style, the light, what is in front and what is behind. As short as it can be while still saying what to draw. -3. Choose `shape` when it matters: `portrait` for a page or a poster, `landscape` for a banner or a strip, `square` for an icon. +3. Choose `shape` when it matters: `portrait` for a page or a poster, `landscape` for a banner or a strip, `square` for an icon. Choose `type: infographic` when the picture sets out facts, in Studio's sketchnote style; leave `type` out for any other picture. 4. Look at what comes back. Image models get hands, counts and lettering wrong. Change it once with its `artifact_id` when it is plainly wrong, and say what you changed; otherwise tell the user what is off. No picture for numbers: a chart stays a chart, in a document. Keep any words in a picture short, a title or a label; text that must be exact goes in the document around it. diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py index ca839506fd..f3cba41a48 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py @@ -100,12 +100,13 @@ "The picture's title, as Studio lists it. Needed for a new picture." ), }, - "style": { + "type": { "type": "string", "enum": sorted(EDITABLE_FORMATS), "description": ( - "infographic to give the prompt's facts Studio's sketchnote " - "style; image, the default, to draw the prompt alone." + "The type of picture: image, the default, draws the prompt " + "alone; infographic gives the prompt's facts Studio's " + "sketchnote style." ), }, "artifact_id": { @@ -143,7 +144,7 @@ class _Request: prompt: str title: str | None - style: str + picture_type: str base_artifact_id: int | None shape: Shape | None = None references: tuple[str, ...] = () @@ -162,7 +163,9 @@ def make( with _starting: if request.base_artifact_id is None: assert request.title is not None # _request asked for it - refuse_same_title(session, scope.thread_id, request.title, request.style) + refuse_same_title( + session, scope.thread_id, request.title, request.picture_type + ) made = _start(session, scope, request) outcome = wait_for_outcome( @@ -199,7 +202,7 @@ def _request(arguments: dict[str, Any]) -> _Request: """The call's arguments, refused in a sentence naming the one to fix.""" prompt, title = arguments.get("prompt"), arguments.get("title") # Null is how many models leave an optional field out. - style = arguments.get("style") or "image" + picture_type = arguments.get("type") or "image" base = arguments.get("artifact_id") if not isinstance(prompt, str) or not prompt.strip(): raise ToolCallError("Give the prompt: what to draw, or how to change it.") @@ -208,9 +211,9 @@ def _request(arguments: dict[str, Any]) -> _Request: f"The prompt is {len(prompt.strip()):,} characters; the limit is " f"{PROMPT_CHARS:,}. Shorten it and call this tool again." ) - if style not in EDITABLE_FORMATS: + if picture_type not in EDITABLE_FORMATS: raise ToolCallError( - f"style must be one of: {', '.join(sorted(EDITABLE_FORMATS))}." + f"type must be one of: {', '.join(sorted(EDITABLE_FORMATS))}." ) if base is not None and (not isinstance(base, int) or isinstance(base, bool)): raise ToolCallError( @@ -237,7 +240,7 @@ def _request(arguments: dict[str, Any]) -> _Request: return _Request( prompt.strip(), title.strip() if isinstance(title, str) else None, - style, + picture_type, base, cast(Shape, shape) if shape is not None else None, tuple(dict.fromkeys(name.strip() for name in references)), @@ -282,7 +285,7 @@ def _new_picture(session: Session, scope: TurnScope, request: _Request) -> Artif scope.workspace_id, title=request.title, prompt=request.prompt, - format=request.style, + format=request.picture_type, chat_thread_id=scope.thread_id, shape=request.shape, references=_reference_ids(session, scope, request.references), diff --git a/surfsense_local/backend/tests/integration/agent/test_make_image.py b/surfsense_local/backend/tests/integration/agent/test_make_image.py index 92dd5b37a2..d7b3a0c0ce 100644 --- a/surfsense_local/backend/tests/integration/agent/test_make_image.py +++ b/surfsense_local/backend/tests/integration/agent/test_make_image.py @@ -128,7 +128,7 @@ async def test_a_picture_is_drawn_from_the_prompt_as_written_with_no_sources( text, images, is_error = await tools.call_content( workspace_id, "make_image", - {"prompt": "A lighthouse at dusk", "title": "Lighthouse", "style": "image"}, + {"prompt": "A lighthouse at dusk", "title": "Lighthouse", "type": "image"}, ) assert is_error is False, text @@ -152,13 +152,13 @@ async def test_a_picture_is_drawn_from_the_prompt_as_written_with_no_sources( async def test_an_infographic_paints_the_agents_facts_in_studios_style( tools: ToolEndpoint, studio_worker: None, painter: Painter ) -> None: - """The style is Studio's own, so both paths draw alike.""" + """An infographic is a type of picture, in Studio's own style, so both paths draw alike.""" workspace_id = await tools.workspace() text, is_error = await tools.call( workspace_id, "make_image", - {"prompt": "Revenue rose 12%", "title": "Q3", "style": "infographic"}, + {"prompt": "Revenue rose 12%", "title": "Q3", "type": "infographic"}, ) assert is_error is False, text @@ -276,7 +276,7 @@ async def test_the_same_title_again_is_refused_so_a_retry_does_not_bill_twice( {"prompt": "x" * 32_001, "title": "Boat"}, "The prompt is 32,001 characters; the limit is 32,000. Shorten it", ), - ({"prompt": "A boat", "title": "Boat", "style": "mural"}, "style must be"), + ({"prompt": "A boat", "title": "Boat", "type": "mural"}, "type must be"), ], ) async def test_a_call_missing_something_says_what_to_fix( From 8ad12748a3aea98d743ec78b44566e480908c29c Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:18:57 +0530 Subject: [PATCH 12/33] feat(local): one binder lays a comic's pictures out as a PDF Page by page at comic-book width or one long scroll, right to left for a manga, pages placed as JPEG, a page's exact lines set in a strip under it. Studio's comics and the agent's will share it. --- .../modules/artifacts/comic/__init__.py | 0 .../modules/artifacts/comic/reading.py | 8 ++ .../tests/unit/worker/test_comic_binding.py | 100 ++++++++++++++++++ .../media/visual/comic_binding/__init__.py | 0 .../studio/media/visual/comic_binding/bind.py | 98 +++++++++++++++++ 5 files changed, 206 insertions(+) create mode 100644 surfsense_local/backend/modules/artifacts/comic/__init__.py create mode 100644 surfsense_local/backend/modules/artifacts/comic/reading.py create mode 100644 surfsense_local/backend/tests/unit/worker/test_comic_binding.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic_binding/__init__.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py diff --git a/surfsense_local/backend/modules/artifacts/comic/__init__.py b/surfsense_local/backend/modules/artifacts/comic/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/modules/artifacts/comic/reading.py b/surfsense_local/backend/modules/artifacts/comic/reading.py new file mode 100644 index 0000000000..60f25ccecd --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/reading.py @@ -0,0 +1,8 @@ +"""How a bound comic reads: page by page or as one scroll, left to right or right to left.""" + +from typing import Literal + +Layout = Literal["pages", "scroll"] +Direction = Literal["ltr", "rtl"] +LAYOUTS: tuple[Layout, ...] = ("pages", "scroll") +DIRECTIONS: tuple[Direction, ...] = ("ltr", "rtl") diff --git a/surfsense_local/backend/tests/unit/worker/test_comic_binding.py b/surfsense_local/backend/tests/unit/worker/test_comic_binding.py new file mode 100644 index 0000000000..8edba24b35 --- /dev/null +++ b/surfsense_local/backend/tests/unit/worker/test_comic_binding.py @@ -0,0 +1,100 @@ +"""Binding a comic's pictures into one PDF: by page or as one scroll, in its reading direction.""" + +import io + +import pytest +from PIL import Image +from pypdf import PdfReader + +from worker.studio.media.visual.comic_binding.bind import BoundPage, bind + +pytestmark = pytest.mark.unit + +POINTS_WIDE = 6.625 * 72 + + +def _png(width: int, height: int, colour: str = "teal") -> bytes: + out = io.BytesIO() + Image.new("RGB", (width, height), colour).save(out, "PNG") + return out.getvalue() + + +def _sizes(pdf: bytes) -> list[tuple[float, float]]: + return [ + (float(page.mediabox.width), float(page.mediabox.height)) + for page in PdfReader(io.BytesIO(pdf)).pages + ] + + +def test_each_picture_is_a_page_at_comic_book_width_as_tall_as_its_picture() -> None: + """No picture is cropped or letterboxed to fit a paper size.""" + pdf = bind( + "The Lighthouse Cat", + [BoundPage(_png(400, 600)), BoundPage(_png(400, 600, "navy"))], + layout="pages", + direction="ltr", + ) + + assert _sizes(pdf) == [(POINTS_WIDE, POINTS_WIDE * 1.5)] * 2 + assert PdfReader(io.BytesIO(pdf)).metadata.title == "The Lighthouse Cat" + + +def test_a_pages_lines_are_set_under_it_exactly() -> None: + """For a misspelt page, or one drawn without words, the strip carries its text.""" + pdf = bind( + "Cat", + [BoundPage(_png(400, 400), ("NORA: The bulb is broken!",))], + layout="pages", + direction="ltr", + ) + + ((width, height),) = _sizes(pdf) + assert height > width + assert ( + "NORA: The bulb is broken!" + in PdfReader(io.BytesIO(pdf)).pages[0].extract_text() + ) + + +def test_a_webtoon_scrolls_down_one_long_page() -> None: + """Its beats stack in one column, read by scrolling, not page by page.""" + pdf = bind( + "Beats", + [BoundPage(_png(400, 600)) for _ in range(3)], + layout="scroll", + direction="ltr", + ) + + ((_, height),) = _sizes(pdf) + assert height > 3 * POINTS_WIDE * 1.5 + + +def test_a_long_scroll_starts_a_new_page_every_twelve() -> None: + """No viewer meets one page tall enough to fail.""" + pdf = bind( + "Beats", + [BoundPage(_png(40, 60)) for _ in range(13)], + layout="scroll", + direction="ltr", + ) + + assert len(_sizes(pdf)) == 2 + + +def test_a_manga_tells_its_reader_it_reads_right_to_left() -> None: + """A reader then lays its spreads out as a manga's are.""" + pdf = bind("Manga", [BoundPage(_png(400, 600))], layout="pages", direction="rtl") + + preferences = PdfReader(io.BytesIO(pdf)).trailer["/Root"]["/ViewerPreferences"] + assert preferences["/Direction"] == "/R2L" + + +def test_pages_are_placed_as_jpeg_so_a_book_stays_small() -> None: + """Four PNG pages made 10 to 28 MB in the spike.""" + noisy = Image.effect_noise((800, 1200), 64).convert("RGB") + out = io.BytesIO() + noisy.save(out, "PNG") + + pdf = bind("Noise", [BoundPage(out.getvalue())], layout="pages", direction="ltr") + + assert len(pdf) < len(out.getvalue()) diff --git a/surfsense_local/backend/worker/studio/media/visual/comic_binding/__init__.py b/surfsense_local/backend/worker/studio/media/visual/comic_binding/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py new file mode 100644 index 0000000000..1122ec211e --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py @@ -0,0 +1,98 @@ +"""A comic's pictures bound into one PDF, as Studio and the agent both bind them.""" + +from dataclasses import dataclass +from io import BytesIO + +from PIL import Image +from reportlab.lib.units import inch +from reportlab.lib.utils import ImageReader +from reportlab.pdfbase.pdfmetrics import stringWidth +from reportlab.pdfgen import canvas + +from modules.artifacts.comic.reading import Direction, Layout + +# Comic-book trim width; each page is as tall as its picture. +WIDTH = 6.625 * inch +MARGIN = 0.25 * inch +GUTTER = 0.2 * inch +FONT, SIZE, LEADING = "Helvetica", 11, 15 +# A scroll starts a new page every twelve pictures, so no viewer meets one huge page. +PER_SCROLL = 12 +# JPEG, not PNG: four PNG pages made 10 to 28 MB in the spike. +JPEG_QUALITY = 90 + + +@dataclass(frozen=True) +class BoundPage: + picture: bytes + # Set under the picture exactly: a misspelt page's, or one drawn without words. + lines: tuple[str, ...] = () + + +@dataclass(frozen=True) +class _Block: + image: ImageReader + height: float + lines: list[str] + strip: float + + +def bind( + title: str, pages: list[BoundPage], *, layout: Layout, direction: Direction +) -> bytes: + out = BytesIO() + pdf = canvas.Canvas(out) + pdf.setTitle(title) + if direction == "rtl": + pdf.setViewerPreference("Direction", "R2L") + blocks = [_block(page) for page in pages] + if layout == "scroll": + for start in range(0, len(blocks), PER_SCROLL): + _draw(pdf, blocks[start : start + PER_SCROLL], GUTTER) + else: + for one in blocks: + _draw(pdf, [one], 0) + pdf.save() + return out.getvalue() + + +def _block(page: BoundPage) -> _Block: + with Image.open(BytesIO(page.picture)) as opened: + picture = opened.convert("RGB") + jpeg = BytesIO() + picture.save(jpeg, "JPEG", quality=JPEG_QUALITY) + jpeg.seek(0) + lines = [line for text in page.lines for line in _wrapped(text, WIDTH - 2 * MARGIN)] + strip = len(lines) * LEADING + 2 * MARGIN if lines else 0 + return _Block( + ImageReader(jpeg), WIDTH * picture.height / picture.width, lines, strip + ) + + +def _wrapped(text: str, width: float) -> list[str]: + lines: list[str] = [] + line = "" + for word in text.split(): + trial = f"{line} {word}".strip() + if line and stringWidth(trial, FONT, SIZE) > width: + lines.append(line) + line = word + else: + line = trial + return [*lines, line] if line else lines + + +def _draw(pdf: canvas.Canvas, blocks: list[_Block], gutter: float) -> None: + """Top to bottom on one page, each picture with its strip under it.""" + height = sum(b.height + b.strip for b in blocks) + gutter * (len(blocks) - 1) + pdf.setPageSize((WIDTH, height)) + top = height + for block in blocks: + pdf.drawImage(block.image, 0, top - block.height, WIDTH, block.height) + pdf.setFont(FONT, SIZE) + y = top - block.height - MARGIN - SIZE + for line in block.lines: + pdf.drawString(MARGIN, y, line) + y -= LEADING + top -= block.height + block.strip + gutter + pdf.showPage() From 9c9643f7811eb9ab6a09e42186a3718b1f40b2a3 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:19:11 +0530 Subject: [PATCH 13/33] feat(local): Studio draws a comic from sources and binds its pages A comic format with options kind and pages, one of ten kinds and 1 to 12 pictures. The chat model plans the cast, style and panels; page 1 is drawn alone and the rest with it as reference; a local model draws no words and every line goes in the strip. A failed comic is never retried, since every page would be billed again. --- .../backend/modules/artifacts/comic/kinds.py | 155 ++++++++++ .../modules/artifacts/comic/options.py | 19 ++ .../backend/modules/artifacts/formats.py | 12 + .../integration/artifacts/test_routes.py | 58 +++- .../tests/integration/worker/test_studio.py | 105 ++++++- .../unit/llm/prompting/test_prompts_ship.py | 4 + .../tests/unit/worker/test_comic_pipeline.py | 265 ++++++++++++++++++ .../unit/worker/test_studio_job_router.py | 10 +- surfsense_local/backend/worker/studio/job.py | 17 +- .../backend/worker/studio/job_router.py | 4 + .../studio/media/visual/comic/__init__.py | 0 .../worker/studio/media/visual/comic/draw.py | 43 +++ .../studio/media/visual/comic/page_prompt.py | 56 ++++ .../studio/media/visual/comic/pipeline.py | 88 ++++++ .../worker/studio/media/visual/comic/plan.py | 76 +++++ .../media/visual/comic/prompts/capable.md | 26 ++ .../media/visual/comic/prompts/compact.md | 25 ++ .../media/visual/comic/prompts/frontier.md | 10 + 18 files changed, 966 insertions(+), 7 deletions(-) create mode 100644 surfsense_local/backend/modules/artifacts/comic/kinds.py create mode 100644 surfsense_local/backend/modules/artifacts/comic/options.py create mode 100644 surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/__init__.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/draw.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/page_prompt.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/plan.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/prompts/capable.md create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/prompts/compact.md create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/prompts/frontier.md diff --git a/surfsense_local/backend/modules/artifacts/comic/kinds.py b/surfsense_local/backend/modules/artifacts/comic/kinds.py new file mode 100644 index 0000000000..0752b856d1 --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/kinds.py @@ -0,0 +1,155 @@ +"""Each kind of comic: how it reads, how its pictures are shaped and bound, and how it looks. + +The agent's skill (`surfsense-pictures/comics.md`) states the same table for the agent. +""" + +from dataclasses import dataclass +from typing import Literal + +Shape = Literal["portrait", "landscape"] + + +@dataclass(frozen=True) +class Kind: + key: str + name: str + # What one picture is, as a page prompt says it. + picture: str + unit: str + shape: Shape + layout: Literal["pages", "scroll"] + direction: Literal["ltr", "rtl"] + looks: str + default_pages: int + # False for a picture book or a storyboard: its text is never in the picture. + bubbles: bool = True + + +KINDS: dict[str, Kind] = { + kind.key: kind + for kind in ( + Kind( + "western", + "Western comic", + "A portrait comic page of 4 to 6 black-bordered panels", + "Page", + "portrait", + "pages", + "ltr", + "Full-colour comic book art, clean bold ink lines, flat colours, white " + "gutters, caption boxes, speech bubbles with clear hand-lettered capitals.", + 4, + ), + Kind( + "graphic_novel", + "Graphic novel", + "A portrait graphic novel page of 3 to 6 panels, with room for quiet, " + "wordless panels", + "Page", + "portrait", + "pages", + "ltr", + "Full-colour graphic novel art, confident ink lines, cinematic framing.", + 4, + ), + Kind( + "bande_dessinee", + "Bande dessinée", + "A detailed portrait album page of 6 to 9 smaller panels", + "Page", + "portrait", + "pages", + "ltr", + "Franco-Belgian bande dessinée, clean 'ligne claire' lines, flat colour, " + "detailed backgrounds.", + 4, + ), + Kind( + "manga", + "Manga", + "A portrait manga page of 3 to 6 panels, read right to left: panel 1 top " + "right, the last bottom left; its bubbles read right to left too", + "Page", + "portrait", + "pages", + "rtl", + "Black-and-white manga with screentone, speed lines and expressive faces.", + 4, + ), + Kind( + "manhua", + "Manhua", + "A portrait manhua page of 3 to 6 panels", + "Page", + "portrait", + "pages", + "ltr", + "Full-colour, painterly manhua art.", + 4, + ), + Kind( + "webtoon", + "Webtoon", + "One tall webtoon panel, one beat of the story, nothing else in the picture", + "Beat", + "portrait", + "scroll", + "ltr", + "Full-colour webtoon art, soft shading, clean digital lines.", + 8, + ), + Kind( + "comic_strip", + "Comic strip", + "One landscape comic strip of 3 or 4 panels in a row, the punchline in the " + "last", + "Strip", + "landscape", + "pages", + "ltr", + "Newspaper comic strip, simple shapes, bold outlines, flat colour.", + 1, + ), + Kind( + "yonkoma", + "Yonkoma (4-koma)", + "One portrait yonkoma of 4 equal panels stacked top to bottom: setup, " + "development, turn, punchline", + "Strip", + "portrait", + "pages", + "ltr", + "Manga-style 4-koma strip, black and white, simple expressive faces.", + 1, + ), + Kind( + "picture_book", + "Picture book", + "One full-page illustration of one moment, no speech bubbles", + "Page", + "portrait", + "pages", + "ltr", + "Children's picture book illustration, soft colour, gentle shapes.", + 4, + bubbles=False, + ), + Kind( + "storyboard", + "Storyboard", + "One landscape storyboard frame of one shot", + "Shot", + "landscape", + "pages", + "ltr", + "Rough, sketchy storyboard frame, grey pencil tones, arrows for camera " + "and movement.", + 6, + bubbles=False, + ), + ) +} + +DEFAULT_KIND = "western" +# Each picture is billed or takes minutes, so a comic stops here. +MAX_PAGES = 12 diff --git a/surfsense_local/backend/modules/artifacts/comic/options.py b/surfsense_local/backend/modules/artifacts/comic/options.py new file mode 100644 index 0000000000..27500ed513 --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/options.py @@ -0,0 +1,19 @@ +"""A comic job's options, the kind and how many pictures, checked when it is created.""" + +from sqlalchemy.orm import Session + +from modules.artifacts.comic.kinds import DEFAULT_KIND, KINDS, MAX_PAGES + + +def validate_options(_session: Session, raw: dict | None) -> dict: + """The Format hook: raises ValueError naming the option to fix.""" + raw = raw or {} + kind = raw.get("kind") or DEFAULT_KIND + if kind not in KINDS: + raise ValueError(f"kind: choose one of {', '.join(KINDS)}") + pages = raw.get("pages", KINDS[kind].default_pages) + if not isinstance(pages, int) or isinstance(pages, bool): + raise ValueError("pages: give a whole number") + if not 1 <= pages <= MAX_PAGES: + raise ValueError(f"pages: from 1 to {MAX_PAGES}") + return {"kind": kind, "pages": pages} diff --git a/surfsense_local/backend/modules/artifacts/formats.py b/surfsense_local/backend/modules/artifacts/formats.py index 8623470e9b..28f39ae601 100644 --- a/surfsense_local/backend/modules/artifacts/formats.py +++ b/surfsense_local/backend/modules/artifacts/formats.py @@ -4,6 +4,7 @@ from sqlalchemy.orm import Session +from modules.artifacts.comic import options as comic_options from modules.artifacts.podcast import brief from modules.llm.model_type import ModelType @@ -31,6 +32,9 @@ class Format: grounding: Grounding = Grounding.PASSAGES # What to search a selection too big to share for, when no prompt says. default_focus: str = "the main points, findings, figures and conclusions" + # Its pipeline also takes the model that draws from a reference picture, as + # `keeper`: None when no chosen model takes pictures in. + draws_with_references: bool = False # worker/studio/job_router.py must name every key here and nothing else @@ -65,6 +69,14 @@ class Format: "Infographic", requires_model_types=(ModelType.IMAGE_GEN, ModelType.TEXT_GEN), ), + Format( + "comic", + "Comic", + requires_model_types=(ModelType.IMAGE_GEN, ModelType.TEXT_GEN), + validate_options=comic_options.validate_options, + draws_with_references=True, + default_focus="the story: who, what happens, and how it ends", + ), ) FORMATS_BY_KEY: dict[str, Format] = {fmt.key: fmt for fmt in FORMATS} diff --git a/surfsense_local/backend/tests/integration/artifacts/test_routes.py b/surfsense_local/backend/tests/integration/artifacts/test_routes.py index 612e9e614b..3cf7db3fb0 100644 --- a/surfsense_local/backend/tests/integration/artifacts/test_routes.py +++ b/surfsense_local/backend/tests/integration/artifacts/test_routes.py @@ -32,7 +32,11 @@ def choose_model(engine: Engine) -> None: """A generation model must be selected before a job can run.""" with create_session_factory(engine)() as session: session.add( - SelectedModel(model_type=ModelType.TEXT_GEN, provider="llamacpp", name="Qwen3-4B-Q4_K_M") + SelectedModel( + model_type=ModelType.TEXT_GEN, + provider="llamacpp", + name="Qwen3-4B-Q4_K_M", + ) ) session.commit() @@ -218,7 +222,9 @@ async def test_a_server_that_lists_no_voices_takes_the_ones_added_for_it( assert "Settings" in refused.json()["detail"] for voice in ("alloy", "echo"): - added = await client.post("/llm/selection/audio_gen/voices", json={"voice": voice}) + added = await client.post( + "/llm/selection/audio_gen/voices", json={"voice": voice} + ) assert added.status_code == 201, added.text opened = (await client.get(brief_url)).json() @@ -671,3 +677,51 @@ async def test_a_ready_artifact_cannot_be_cancelled( response = await client.post(f"/artifacts/{artifact_id}/cancel") assert response.status_code == 409 + + +async def test_a_comic_job_checks_and_stores_its_kind_and_page_count( + client: AsyncClient, engine: Engine, workspace_id: int, choose_model: None +) -> None: + """A kind Studio has no table for, or a thirteenth page, is refused at the door.""" + with create_session_factory(engine)() as session: + connection = ProviderConnection( + label="Images", provider="openai_compatible", base_url="http://127.0.0.1:9" + ) + session.add(connection) + session.flush() + session.add( + SelectedModel( + model_type=ModelType.IMAGE_GEN, + provider="openai_compatible", + connection_id=connection.id, + name="flux", + ) + ) + session.commit() + source_id = make_ready_source(engine, workspace_id) + url = f"/workspaces/{workspace_id}/studio/jobs" + + for options, words in ( + ({"kind": "opera", "pages": 4}, "kind"), + ({"kind": "manga", "pages": 13}, "pages"), + ): + refused = await client.post( + url, + json={"format": "comic", "document_ids": [source_id], "options": options}, + ) + assert refused.status_code == 422, refused.text + assert words in refused.json()["detail"] + + created = await client.post( + url, + json={ + "format": "comic", + "document_ids": [source_id], + "options": {"kind": "manga", "pages": 6}, + }, + ) + assert created.status_code == 201, created.text + with create_session_factory(engine)() as session: + stored = session.get(Artifact, created.json()["id"]) + assert stored.format == "comic" + assert stored.artifact_metadata["options"] == {"kind": "manga", "pages": 6} diff --git a/surfsense_local/backend/tests/integration/worker/test_studio.py b/surfsense_local/backend/tests/integration/worker/test_studio.py index 933ed70147..50c54d23e7 100644 --- a/surfsense_local/backend/tests/integration/worker/test_studio.py +++ b/surfsense_local/backend/tests/integration/worker/test_studio.py @@ -1,16 +1,19 @@ import contextlib +import io import json import time from collections.abc import Iterator import httpx import pytest +from PIL import Image from sqlalchemy import Engine, text from sqlalchemy.orm import Session from modules.artifacts.models import Artifact, ArtifactFileRole from modules.documents.models import Document, DocumentStatus, DocumentType from modules.llm.catalog.local.manifest import load_local_manifest +from modules.llm.pictures_in.model import PictureModel from modules.llm.profile import Fingerprint, Tier from modules.llm.providers.audiocpp.memory import NotEnoughMemoryError from modules.llm.providers.audiocpp.speech import AudioCppSpeech, VoicedModel @@ -119,9 +122,11 @@ def _capture_image(monkeypatch: pytest.MonkeyPatch) -> list[str]: seen: list[str] = [] class FakeImageGenerator: - async def generate(self, _model: str, content: str) -> GeneratedImage: + async def generate( + self, _model: str, content: str, *, shape: str | None = None + ) -> GeneratedImage: seen.append(content) - return GeneratedImage(b"\x89PNG\r\n\x1a\n" + b"\x00" * 8, "image/png") + return GeneratedImage(_tiny_png(), "image/png") selection = type("Selection", (), {"provider": "fake", "name": "flux"})() monkeypatch.setattr( @@ -131,6 +136,13 @@ async def generate(self, _model: str, content: str) -> GeneratedImage: return seen +def _tiny_png() -> bytes: + """A real picture: a comic's binder opens every page it places.""" + out = io.BytesIO() + Image.new("RGB", (8, 12), "teal").save(out, "PNG") + return out.getvalue() + + def _primary_bytes(artifact: Artifact) -> bytes: return ( get_storage_settings().data_dir / artifact.files[0].storage_key @@ -734,6 +746,95 @@ def fail(*_args: object, **_kwargs: object) -> Built: assert "HTTP 500" in (artifact.document.error_message or "") +def test_a_comic_job_binds_its_drawn_pages_into_one_pdf( + session: Session, stub_model: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """The chat model plans it, the image model draws page 1, the model that takes + pictures draws the rest from it, and the PDF is the comic's one file.""" + plan = { + "title": "Cassini's Last Dive", + "style": "Bold ink.", + "characters": "CASSINI, a gold-foil probe with a white dish.", + "pages": [ + { + "panels": [ + { + "where": "top", + "shows": "Cassini reaches Saturn.", + "lines": ["Caption: 2004."], + } + ] + }, + { + "panels": [ + { + "where": "top", + "shows": "Cassini dives.", + "lines": ["Caption: 2017."], + } + ] + }, + ], + } + asked = _capture_model(monkeypatch, json.dumps(plan)) + drawn = _capture_image(monkeypatch) + from_page_one: list[bytes] = [] + + class Keeper: + async def edit( + self, _model: str, _prompt: str, images, *, shape=None + ) -> GeneratedImage: + from_page_one.extend(image.content for image in images) + return GeneratedImage(_tiny_png(), "image/png") + + keeper_selection = type("Selection", (), {"provider": "fake", "name": "flux"})() + monkeypatch.setattr( + "worker.studio.job.pictures_in.picture_model", + lambda _session: PictureModel("image_model", keeper_selection, Keeper()), + ) + artifact = make_artifact( + session, + fmt="comic", + prompt="the finale", + options={"kind": "western", "pages": 2}, + ) + + run(artifact.id) + + session.expire_all() + assert artifact.document.status is DocumentStatus.READY, ( + artifact.document.error_message + ) + assert "the finale" in asked[0] + assert len(drawn) == 1 + assert len(from_page_one) == 1 + assert artifact.document.title == "Cassini's Last Dive" + assert "Cassini dives." in (artifact.document.content or "") + _one_file(artifact, "application/pdf", b"%PDF") + assert artifact.artifact_metadata["comic"]["pages"] == 2 + + +def test_a_failed_comic_is_never_retried_since_every_page_is_billed_again( + session: Session, stub_model: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """A comic's failure ends the task: a retry would draw every page again.""" + + def fail(*_args: object, **_kwargs: object) -> Built: + raise RuntimeError("the image model hung up on page 3") + + monkeypatch.setattr("worker.studio.media.visual.comic.pipeline.render", fail) + monkeypatch.setattr("worker.studio.job.pictures_in.picture_model", lambda _s: None) + artifact = make_artifact( + session, fmt="comic", options={"kind": "western", "pages": 4} + ) + + run(artifact.id) # returns: a raise here would ask Huey for a retry + + session.expire_all() + assert artifact.document.status is DocumentStatus.FAILED + assert "page 3" in (artifact.document.error_message or "") + + def test_persist_stores_a_primary_blob(session: Session, stub_model: None) -> None: """A file format's bytes land on disk with a row that resolves back to them.""" artifact = make_artifact(session) diff --git a/surfsense_local/backend/tests/unit/llm/prompting/test_prompts_ship.py b/surfsense_local/backend/tests/unit/llm/prompting/test_prompts_ship.py index a40f635543..019762347e 100644 --- a/surfsense_local/backend/tests/unit/llm/prompting/test_prompts_ship.py +++ b/surfsense_local/backend/tests/unit/llm/prompting/test_prompts_ship.py @@ -4,6 +4,7 @@ import pytest +from modules.artifacts.comic.kinds import KINDS from modules.artifacts.podcast.brief import Duration, PodcastBrief, Speaker, Style from modules.llm.profile import Tier from modules.llm.prompting import load @@ -13,6 +14,7 @@ from worker.studio.content.summary import pipeline as summary from worker.studio.media.audio.podcast import draft, outline from worker.studio.media.audio.podcast.outline import Segment +from worker.studio.media.visual.comic import plan as comic from worker.studio.media.visual.image import pipeline as image from worker.studio.media.visual.infographic import pipeline as infographic from worker.studio.office import prompt as office @@ -69,6 +71,8 @@ def _every_prompt(tier: Tier) -> dict[str, str]: "html": html.prompt(tier, FOCUS), "image": image.prompt(tier, FOCUS), "infographic": infographic.prompt(tier, FOCUS), + "comic": comic.prompt(tier, FOCUS, KINDS["western"], 4), + "picture book": comic.prompt(tier, FOCUS, KINDS["picture_book"], 4), "outline": outline.prompt(tier, BRIEF, FOCUS), "draft": draft.prompt(tier, BRIEF, SEGMENT, 1, 2, "Sam: Welcome."), } | {spec.key: office.build(tier, spec, FOCUS) for spec in (docx, pptx, xlsx, pdf)} diff --git a/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py b/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py new file mode 100644 index 0000000000..4a8545bd00 --- /dev/null +++ b/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py @@ -0,0 +1,265 @@ +"""A Studio comic: the chat model plans it, the image model draws its pages, the binder makes the PDF.""" + +import io +import json + +import pytest +from PIL import Image +from pypdf import PdfReader + +from modules.artifacts.comic.options import validate_options +from modules.llm.pictures_in.model import PictureModel +from modules.llm.profile import Tier +from modules.llm.providers.protocols import GeneratedImage +from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration +from worker.studio.media.visual.comic import pipeline as comic +from worker.studio.media.visual.drawn.pipeline import DRAWN_FROM_WORDS +from worker.studio.shared.artifact import Source + +pytestmark = pytest.mark.unit + + +def _png(colour: str) -> bytes: + out = io.BytesIO() + Image.new("RGB", (40, 60), colour).save(out, "PNG") + return out.getvalue() + + +PLAN = { + "title": "The Lighthouse Cat", + "style": "Bold ink lines, flat colours.", + "characters": "MILO, a small orange tabby cat with a red scarf.", + "pages": [ + { + "panels": [ + {"where": "top", "shows": "A storm hits the lighthouse.", "lines": []}, + { + "where": "bottom", + "shows": "MILO looks at the dead lamp.", + "lines": ["MILO: The light is out!"], + }, + ] + }, + { + "panels": [ + { + "where": "whole page", + "shows": "MILO finds a bulb.", + "lines": ["MILO: Got it!"], + } + ] + }, + {"panels": [{"where": "whole page", "shows": "The lamp shines.", "lines": []}]}, + ], +} + + +class Artist: + """The image model, and the model that takes page 1 as reference.""" + + def __init__(self) -> None: + self.drawn: list[str] = [] + self.with_reference: list[tuple[str, bytes]] = [] + self.shapes: list[str | None] = [] + + async def generate( + self, _model: str, prompt: str, *, shape: str | None = None + ) -> GeneratedImage: + self.drawn.append(prompt) + self.shapes.append(shape) + return GeneratedImage(_png("teal"), "image/png") + + async def edit( + self, + _model: str, + prompt: str, + images: tuple[GeneratedImage, ...], + *, + shape: str | None = None, + ) -> GeneratedImage: + (reference,) = images + self.with_reference.append((prompt, reference.content)) + self.shapes.append(shape) + return GeneratedImage(_png("navy"), "image/png") + + +def _writer() -> ResolvedGeneration: + selection = type("Selection", (), {"name": "qwen3:8b", "tier": Tier.CAPABLE})() + return ResolvedGeneration(selection, None) + + +def _painter( + artist: Artist, provider: str = "openai_compatible" +) -> ResolvedImageGeneration: + selection = type("Selection", (), {"name": "flux", "provider": provider})() + return ResolvedImageGeneration(selection, artist) + + +def _keeper(artist: Artist) -> PictureModel: + selection = type( + "Selection", (), {"name": "flux", "provider": "openai_compatible"} + )() + return PictureModel("image_model", selection, artist) + + +@pytest.fixture +def asked(monkeypatch: pytest.MonkeyPatch) -> list[str]: + """The chat model answers with PLAN; what it was asked is recorded.""" + seen: list[str] = [] + + def writer(_model: object, system: str, _sources: object) -> str: + seen.append(system) + return json.dumps(PLAN) + + monkeypatch.setattr("worker.studio.shared.generate.run_model", writer) + return seen + + +def _text(pdf: bytes) -> str: + return "\n".join(page.extract_text() for page in PdfReader(io.BytesIO(pdf)).pages) + + +def test_page_one_is_drawn_alone_and_every_later_page_keeps_its_characters( + asked: list[str], +) -> None: + """Page 1 sets the cast; every later page is drawn from it, lettered in its bubbles.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + "make it funny", + {"kind": "western", "pages": 3}, + keeper=_keeper(artist), + ) + + assert "make it funny" in asked[0] + assert "Western comic" in asked[0] + (first,) = artist.drawn + assert "Page 1 of 3" in first + assert "MILO, a small orange tabby cat" in first + assert 'MILO says: "The light is out!"' in first + assert [prompt.split("Page ")[1][:6] for prompt, _ in artist.with_reference] == [ + "2 of 3", + "3 of 3", + ] + assert all(reference == _png("teal") for _, reference in artist.with_reference) + assert all( + "Keep the characters looking exactly as in the reference" in p + for p, _ in artist.with_reference + ) + assert set(artist.shapes) == {"portrait"} + assert built.title == "The Lighthouse Cat" + assert built.primary_mime == "application/pdf" + assert built.primary_filename == "the-lighthouse-cat.pdf" + assert len(PdfReader(io.BytesIO(built.primary)).pages) == 3 + # Drawn in the bubbles, so not set again under the page. + assert "The light is out!" not in _text(built.primary) + assert "The light is out!" in built.markdown + + +def test_a_local_model_draws_no_words_and_every_line_goes_in_the_strip( + asked: list[str], +) -> None: + """Its lettering is not measured, and nobody looks at the page to check it.""" + artist = Artist() + + built = comic.render( + _painter(artist, provider="sdcpp"), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 3}, + keeper=_keeper(artist), + ) + + prompts = artist.drawn + [prompt for prompt, _ in artist.with_reference] + assert all("no text, no lettering, no speech bubbles" in p for p in prompts) + assert not any("The light is out!" in p for p in prompts) + assert "MILO: The light is out!" in _text(built.primary) + + +def test_a_manga_reads_right_to_left_in_its_pages_and_its_book( + asked: list[str], +) -> None: + """Each page's prompt and the bound book both read right to left.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "manga", "pages": 3}, + keeper=_keeper(artist), + ) + + assert "read right to left" in artist.drawn[0] + root = PdfReader(io.BytesIO(built.primary)).trailer["/Root"] + assert root["/ViewerPreferences"]["/Direction"] == "/R2L" + + +def test_with_no_model_that_takes_pictures_every_page_is_drawn_from_the_words( + asked: list[str], +) -> None: + """Nothing is refused; the comic records that its cast may drift.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 3}, + keeper=None, + ) + + assert len(artist.drawn) == 3 + assert artist.with_reference == [] + assert built.metadata is not None and built.metadata[DRAWN_FROM_WORDS] is True + + +def test_a_plan_longer_than_asked_is_cut_to_the_page_count(asked: list[str]) -> None: + """Each page is billed or takes minutes: the count the user chose is the most drawn.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 2}, + keeper=_keeper(artist), + ) + + assert len(artist.drawn) + len(artist.with_reference) == 2 + assert len(PdfReader(io.BytesIO(built.primary)).pages) == 2 + + +def test_a_webtoon_is_one_scroll_of_beats() -> None: + """A webtoon's count is its beats, up to the same twelve.""" + picked = validate_options(None, {"kind": "webtoon", "pages": 8}) # type: ignore[arg-type] + assert picked == {"kind": "webtoon", "pages": 8} + + +def test_options_default_to_a_four_page_western_comic() -> None: + """What the dialog sends when the user changes nothing.""" + assert validate_options(None, None) == {"kind": "western", "pages": 4} # type: ignore[arg-type] + + +@pytest.mark.parametrize( + "raw, words", + [ + ({"kind": "opera", "pages": 4}, "kind"), + ({"kind": "western", "pages": 0}, "pages"), + ({"kind": "western", "pages": 13}, "pages"), + ({"kind": "western", "pages": "four"}, "pages"), + ], +) +def test_options_outside_the_kinds_or_twelve_pages_are_refused( + raw: dict, words: str +) -> None: + """Refused at the door, naming the option to fix, before any picture is billed.""" + with pytest.raises(ValueError, match=words): + validate_options(None, raw) # type: ignore[arg-type] diff --git a/surfsense_local/backend/tests/unit/worker/test_studio_job_router.py b/surfsense_local/backend/tests/unit/worker/test_studio_job_router.py index 7a36e6e4e9..23d4eb7d8a 100644 --- a/surfsense_local/backend/tests/unit/worker/test_studio_job_router.py +++ b/surfsense_local/backend/tests/unit/worker/test_studio_job_router.py @@ -23,8 +23,14 @@ def test_every_kind_has_a_pipeline() -> None: def test_every_pipeline_takes_what_its_format_declares() -> None: """The harness passes requires_model_types positionally, then sources and prompt, - then the checked options for the formats that take them.""" + then the checked options for the formats that take them, then `keeper` for a + format that draws from a reference picture.""" for fmt in FORMATS: render = job_router.pipeline_for(job_router.Kind(fmt.key)) - expected = len(fmt.requires_model_types) + 2 + (fmt.validate_options is not None) + expected = ( + len(fmt.requires_model_types) + + 2 + + (fmt.validate_options is not None) + + fmt.draws_with_references + ) assert len(inspect.signature(render).parameters) == expected, fmt.key diff --git a/surfsense_local/backend/worker/studio/job.py b/surfsense_local/backend/worker/studio/job.py index 2d0f623721..645561e957 100644 --- a/surfsense_local/backend/worker/studio/job.py +++ b/surfsense_local/backend/worker/studio/job.py @@ -230,6 +230,9 @@ def _generate(session: Session, artifact: Artifact) -> None: # The agent's tool reads this FAILED; a retry could bill the picture twice. if image_drawing_of(artifact.artifact_metadata) is not None: return + # Every page of a comic would be drawn, and billed, again. + if artifact.format == job_router.Kind.COMIC: + return raise # Huey retries; a later success clears the message. @@ -278,6 +281,9 @@ def gathered(budget_chars: int | None) -> list[Source]: ) # Options were checked at job creation; only formats that take them get them. extras = [meta.get("options")] if fmt.validate_options else [] + references = ( + {"keeper": _picture_model(session)} if fmt.draws_with_references else {} + ) # Generation runs for minutes; the write lock must not be held across it. session.commit() raise_if_cancelled(session, document) @@ -285,7 +291,9 @@ def gathered(budget_chars: int | None) -> list[Source]: # A cancel hangs up on a model mid-reply; other stages still finish first. with cancellation.watching(lambda: _check_cancelled(session, document)): try: - return job_router.pipeline_for(kind)(*models, sources, prompt, *extras) + return job_router.pipeline_for(kind)( + *models, sources, prompt, *extras, **references + ) except ContextOverflowError as refused: raise window_budget.SourcesDoNotFitError( window_budget.DOES_NOT_FIT @@ -491,6 +499,13 @@ def _window(model: ResolvedGeneration) -> int | None: return None +def _picture_model(session: Session) -> pictures_in.PictureModel | None: + try: + return pictures_in.picture_model(session) + except ModelResolutionError as error: + raise NoModelSelectedError(str(error)) from error + + def _choose_model( session: Session, model_type: ModelType ) -> ResolvedGeneration | ResolvedImageGeneration | TextToSpeech: diff --git a/surfsense_local/backend/worker/studio/job_router.py b/surfsense_local/backend/worker/studio/job_router.py index 8a29be0b84..293791c11e 100644 --- a/surfsense_local/backend/worker/studio/job_router.py +++ b/surfsense_local/backend/worker/studio/job_router.py @@ -14,6 +14,7 @@ from worker.studio.content.quiz import pipeline as quiz from worker.studio.content.summary import pipeline as summary from worker.studio.media.audio.podcast import pipeline as podcast +from worker.studio.media.visual.comic import pipeline as comic from worker.studio.media.visual.image import pipeline as image from worker.studio.media.visual.infographic import pipeline as infographic from worker.studio.office import pipeline as office @@ -40,6 +41,7 @@ class Kind(enum.StrEnum): PDF = "pdf" IMAGE = "image" INFOGRAPHIC = "infographic" + COMIC = "comic" PODCAST = "podcast" @@ -71,5 +73,7 @@ def pipeline_for(kind: Kind) -> Render: return image.render case Kind.INFOGRAPHIC: return infographic.render + case Kind.COMIC: + return comic.render case Kind.PODCAST: return podcast.render diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/__init__.py b/surfsense_local/backend/worker/studio/media/visual/comic/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/draw.py b/surfsense_local/backend/worker/studio/media/visual/comic/draw.py new file mode 100644 index 0000000000..7c36715293 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/draw.py @@ -0,0 +1,43 @@ +"""Page 1 alone, then every later page with page 1 as its reference, so the cast stays the same.""" + +import asyncio +from collections.abc import Awaitable + +from modules.llm.pictures_in.model import PictureModel +from modules.llm.providers.protocols import GeneratedImage +from modules.llm.providers.sdcpp import provider as sdcpp +from modules.llm.resolution import ResolvedImageGeneration + + +async def draw_pages( + painter: ResolvedImageGeneration, + keeper: PictureModel | None, + prompts: list[str], + shape: str, +) -> list[GeneratedImage]: + first = await painter.generator.generate( + painter.selection.name, prompts[0], shape=shape + ) + + def page(prompt: str) -> Awaitable[GeneratedImage]: + if keeper is None: + return painter.generator.generate( + painter.selection.name, prompt, shape=shape + ) + return keeper.editor.edit(keeper.selection.name, prompt, (first,), shape=shape) + + rest = prompts[1:] + # A remote model draws the later pages side by side; this computer's, one at a time. + if _on_this_computer(painter, keeper): + drawn = [await page(prompt) for prompt in rest] + else: + drawn = list(await asyncio.gather(*(page(prompt) for prompt in rest))) + return [first, *drawn] + + +def _on_this_computer( + painter: ResolvedImageGeneration, keeper: PictureModel | None +) -> bool: + return painter.selection.provider == sdcpp.PROVIDER or ( + keeper is not None and keeper.selection.provider == sdcpp.PROVIDER + ) diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/page_prompt.py b/surfsense_local/backend/worker/studio/media/visual/comic/page_prompt.py new file mode 100644 index 0000000000..4679df80c7 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/page_prompt.py @@ -0,0 +1,56 @@ +"""One page's prompt, laid out as image models read a comic page best.""" + +from modules.artifacts.comic.kinds import Kind +from worker.studio.media.visual.comic.plan import Panel, Plan + +KEEP_CHARACTERS = ( + "Keep the characters looking exactly as in the reference picture: same faces, " + "colours, clothes and drawing style. Draw a new page; do not copy its panels." +) +NO_WORDS = "Draw no text, no lettering, no speech bubbles." + + +def page_prompt( + kind: Kind, + plan: Plan, + number: int, + *, + lettered: bool, + referenced: bool, +) -> str: + panels = plan.pages[number - 1] + lines = [" ".join(part for part in (kind.looks, plan.style) if part)] + if plan.characters: + lines.append(f"Characters: {plan.characters}") + if referenced: + lines.append(KEEP_CHARACTERS) + lines.append( + f'{kind.unit} {number} of {len(plan.pages)} of the comic "{plan.title}". ' + f"{kind.picture}:" + ) + lines += [ + _panel(index, panel, lettered) for index, panel in enumerate(panels, start=1) + ] + if not lettered: + lines.append(NO_WORDS) + return "\n".join(lines) + + +def strip_lines(panels: tuple[Panel, ...]) -> tuple[str, ...]: + """A page's lines in reading order, for the strip under a page drawn without words.""" + return tuple(line for panel in panels for line in panel.lines) + + +def _panel(index: int, panel: Panel, lettered: bool) -> str: + where = f" ({panel.where})" if panel.where else "" + said = [_said(line) for line in panel.lines] if lettered else [] + return " ".join([f"Panel {index}{where}: {panel.shows}", *said]) + + +def _said(line: str) -> str: + speaker, colon, words = line.partition(":") + if not colon or not words.strip(): + return f'Caption box: "{line.strip()}"' + if speaker.strip().lower() == "caption": + return f'Caption box: "{words.strip()}"' + return f'{speaker.strip()} says: "{words.strip()}"' diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py new file mode 100644 index 0000000000..471495b4d4 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py @@ -0,0 +1,88 @@ +"""Comic: the chat model plans it from the sources, the image model draws each +page, and the pages are bound into one PDF in the kind's reading order.""" + +import asyncio + +from modules.artifacts.comic.kinds import KINDS +from modules.llm.pictures_in.model import PictureModel +from modules.llm.providers.sdcpp import provider as sdcpp +from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration +from worker.studio.media.visual.comic import plan as planning +from worker.studio.media.visual.comic.draw import draw_pages +from worker.studio.media.visual.comic.page_prompt import page_prompt, strip_lines +from worker.studio.media.visual.comic.plan import Plan +from worker.studio.media.visual.comic_binding.bind import BoundPage, bind +from worker.studio.media.visual.drawn.pipeline import DRAWN_FROM_WORDS +from worker.studio.shared import generate +from worker.studio.shared.artifact import Built, Source, fallback_title +from worker.studio.shared.text import slug + + +def render( + painter: ResolvedImageGeneration, + writer: ResolvedGeneration, + sources: list[Source], + user_prompt: str | None, + options: dict, + *, + keeper: PictureModel | None, +) -> Built: + kind = KINDS[options["kind"]] + plan = planning.parse( + generate.run_model( + writer, + planning.prompt(writer.tier, user_prompt, kind, options["pages"]), + sources, + ), + fallback_title(user_prompt, sources, "Comic"), + options["pages"], + ) + if not plan.pages: + raise ValueError("the writer planned no pages") + # Studio cannot look at a page to check its lettering; a local model's is not + # measured at all, so its pages are drawn without words and lettered below. + lettered = kind.bubbles and painter.selection.provider != sdcpp.PROVIDER + prompts = [ + page_prompt( + kind, + plan, + number, + lettered=lettered, + referenced=keeper is not None and number > 1, + ) + for number in range(1, len(plan.pages) + 1) + ] + pictures = asyncio.run(draw_pages(painter, keeper, prompts, kind.shape)) + pdf = bind( + plan.title, + [ + BoundPage(picture.content, () if lettered else strip_lines(panels)) + for picture, panels in zip(pictures, plan.pages, strict=True) + ], + layout=kind.layout, + direction=kind.direction, + ) + metadata: dict = { + "comic": {"kind": kind.key, "pages": len(plan.pages), "lettered": lettered} + } + if keeper is None and len(plan.pages) > 1: + metadata[DRAWN_FROM_WORDS] = True + return Built( + title=plan.title, + markdown=_markdown(plan, kind.unit), + primary=pdf, + primary_mime="application/pdf", + primary_filename=f"{slug(plan.title, 'comic')}.pdf", + metadata=metadata, + ) + + +def _markdown(plan: Plan, unit: str) -> str: + """Every page's panels and lines, so the comic is found by what it says.""" + parts = [f"# {plan.title}"] + for number, panels in enumerate(plan.pages, start=1): + parts.append(f"\n## {unit} {number}\n") + for index, panel in enumerate(panels, start=1): + parts.append(f"- Panel {index}: {panel.shows}") + parts += [f" - {line}" for line in panel.lines] + return "\n".join(parts) diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/plan.py b/surfsense_local/backend/worker/studio/media/visual/comic/plan.py new file mode 100644 index 0000000000..d7b51e2246 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/plan.py @@ -0,0 +1,76 @@ +"""The comic the chat model plans from the sources: its look, its cast, and every page's panels.""" + +from dataclasses import dataclass + +from modules.artifacts.comic.kinds import Kind +from modules.llm import prompting +from modules.llm.profile import Tier +from worker.studio.shared.text import as_list, as_text, parse_json + +_BUBBLES = ( + "Give each panel its lines: one short line a bubble, at most about eight words, " + 'two or three a panel, each as "NAME: words", or "Caption: words" for narration. ' + "Write every line word for word." +) +_UNDER = ( + "Give each picture one panel. Its lines are the text set under the picture, never " + "in it: a picture book's words, or a storyboard shot's number, camera and movement." +) + + +@dataclass(frozen=True) +class Panel: + where: str + shows: str + lines: tuple[str, ...] + + +@dataclass(frozen=True) +class Plan: + title: str + style: str + characters: str + pages: tuple[tuple[Panel, ...], ...] + + +def prompt(tier: Tier, user_prompt: str | None, kind: Kind, count: int) -> str: + return prompting.load( + __package__, + tier, + focus=prompting.focus(user_prompt), + kind=kind.name, + picture=kind.picture, + count=count, + unit=kind.unit.lower(), + lettering=_BUBBLES if kind.bubbles else _UNDER, + ) + + +def parse(raw: str, untitled: str, count: int) -> Plan: + """The plan, cut to the pages asked for: each one drawn is billed or takes minutes.""" + spec = parse_json(raw) + pages = tuple( + panels + for page in as_list(spec.get("pages"))[:count] + if isinstance(page, dict) and (panels := _panels(page)) + ) + return Plan( + title=as_text(spec.get("title")) or untitled, + style=as_text(spec.get("style")), + characters=as_text(spec.get("characters")), + pages=pages, + ) + + +def _panels(page: dict) -> tuple[Panel, ...]: + return tuple( + Panel( + where=as_text(panel.get("where")), + shows=as_text(panel.get("shows")), + lines=tuple( + line for item in as_list(panel.get("lines")) if (line := as_text(item)) + ), + ) + for panel in as_list(page.get("panels")) + if isinstance(panel, dict) and as_text(panel.get("shows")) + ) diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/prompts/capable.md b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/capable.md new file mode 100644 index 0000000000..0318e32547 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/capable.md @@ -0,0 +1,26 @@ +You plan a comic from source documents: a $kind of $count $unit(s), told from their facts only. + +Task: plan the comic. Each $unit is one picture: $picture. +$focus +Work in this order: + +1. Read every source and pick the story it tells best in $count $unit(s): who, what happens, and how it ends. +2. Write a style line: medium, palette and line weight, in one sentence. +3. Write a character sheet: each character's face, hair, build, clothes and colours in a few fixed words. Every page repeats it word for word, so keep it short and exact. +4. Plan each $unit in reading order: for every panel, where it sits (top, bottom left, wide), what it shows, and its lines. + +$lettering + +Grounding rules: + +- Use only what the sources state. Simplify, but never invent facts, names or figures. +- Plan exactly $count $unit(s), no more. +- A panel shows one moment. Describe what is seen, not what is meant. + +Return only JSON, no prose: {"title": str, "style": str, "characters": str, "pages": [{"panels": [{"where": str, "shows": str, "lines": [str]}]}]} + +Worked example — a different story in the same shape. Copy the structure, never the content: + +{"title": "The Lighthouse Cat", "style": "Bold ink lines, flat warm colours, night-blue sky.", "characters": "MILO, a small orange tabby cat with a white chest, green eyes and a red scarf. NORA, a woman about seventy, short white hair, round glasses, yellow raincoat.", "pages": [{"panels": [{"where": "wide, top", "shows": "A fishing boat tosses on huge waves at night.", "lines": ["Caption: The storm came at midnight."]}, {"where": "bottom left", "shows": "NORA taps the dead lamp.", "lines": ["NORA: The bulb is broken!"]}]}]} + +Return only the JSON. Nothing before it, nothing after it. diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/prompts/compact.md b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/compact.md new file mode 100644 index 0000000000..9ddf1ce0fc --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/compact.md @@ -0,0 +1,25 @@ +You plan a short comic from the sources below. Use their facts only. + +The comic is a $kind. It has exactly $count $unit(s). Each $unit is one picture: $picture. +$focus + +Write: + +- "title": the comic's title. +- "style": one sentence: medium, colours, line weight. +- "characters": each character in a few fixed words: face, hair, clothes, colours. +- "pages": exactly $count items. Each has "panels". Each panel has "where" (top, bottom left, wide), "shows" (what is seen in it), and "lines". + +$lettering + +Rules: + +- Only facts from the sources. Do not invent names or numbers. +- Exactly $count $unit(s). +- One moment per panel. + +Example of the shape only, not the content: + +{"title": "The Lighthouse Cat", "style": "Bold ink lines, flat warm colours.", "characters": "MILO, a small orange tabby cat with a red scarf.", "pages": [{"panels": [{"where": "top", "shows": "A boat in a storm at night.", "lines": ["Caption: The storm came at midnight."]}, {"where": "bottom", "shows": "MILO looks at the dark lamp.", "lines": ["MILO: The light is out!"]}]}]} + +Return only the JSON. Nothing before it, nothing after it. diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/prompts/frontier.md b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/frontier.md new file mode 100644 index 0000000000..242a6cac1e --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/prompts/frontier.md @@ -0,0 +1,10 @@ +Plan a $kind of exactly $count $unit(s) that tells the sources' story faithfully. Each $unit is one picture: $picture. +$focus + +Choose the story the sources tell best at this length. Write a one-sentence style line (medium, palette, line weight) and a short, exact character sheet (each character's face, hair, build, clothes and colours): both are repeated word for word in every picture's prompt, so the cast looks the same throughout. Then plan each $unit in reading order: for every panel, where it sits, what is seen in it, and its lines. + +$lettering + +Use only what the sources state; simplify, never invent facts, names or figures. + +Return only JSON: {"title": str, "style": str, "characters": str, "pages": [{"panels": [{"where": str, "shows": str, "lines": [str]}]}]} From a010dff9f9bdc0ab0fcf5dd35e0c84bd2b20547a Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:19:11 +0530 Subject: [PATCH 14/33] feat(local): the agent draws a comic itself, never through create_artifact create_artifact does not offer comic: the agent draws and looks at each page before it binds them. --- .../backend/modules/agent/tool_endpoint/create_artifact.py | 5 +++-- .../backend/tests/integration/agent/test_make_image.py | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/create_artifact.py b/surfsense_local/backend/modules/agent/tool_endpoint/create_artifact.py index 3c7e312f57..adf902a203 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/create_artifact.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/create_artifact.py @@ -25,11 +25,12 @@ # Office files and PDFs are scripts the agent renders, and a web page it writes # itself (ADR 0066): Studio's draft keeps none to edit. A picture has its own tool, -# since a draft hands the agent nothing to look at or place. +# since a draft hands the agent nothing to look at or place; so does a comic, whose +# pages the agent draws and checks before it binds them. _KEYS = [ fmt.key for fmt in FORMATS - if fmt.key not in (*DOCUMENT_FORMATS, "html", *EDITABLE_FORMATS) + if fmt.key not in (*DOCUMENT_FORMATS, "html", *EDITABLE_FORMATS, "comic") ] # Written out flat, and the same on every turn, so a local model's prompt cache holds. diff --git a/surfsense_local/backend/tests/integration/agent/test_make_image.py b/surfsense_local/backend/tests/integration/agent/test_make_image.py index d7b3a0c0ce..f3b7d0941b 100644 --- a/surfsense_local/backend/tests/integration/agent/test_make_image.py +++ b/surfsense_local/backend/tests/integration/agent/test_make_image.py @@ -373,7 +373,8 @@ async def test_regenerating_paints_the_same_prompt_with_no_chat_model( async def test_the_studio_tool_does_not_offer_a_picture(tools: ToolEndpoint) -> None: - """One way to make a picture: Studio's draft takes sources, and hands nothing back.""" + """One way to make a picture: Studio's draft takes sources, and hands nothing back. + A comic too: the agent draws its pages and looks at each, then binds them.""" workspace_id = await tools.workspace() listed = await tools.request(workspace_id, "tools/list") @@ -385,7 +386,7 @@ async def test_the_studio_tool_does_not_offer_a_picture(tools: ToolEndpoint) -> ) offered = studio["inputSchema"]["properties"]["format"]["enum"] - assert not {"image", "infographic"} & set(offered) + assert not {"image", "infographic", "comic"} & set(offered) assert is_error is True assert text.startswith("Name a format, one of:") From 4db6468b71a0010fe16c5a0472fd6acb4f0a31a1 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:19:12 +0530 Subject: [PATCH 15/33] feat(local): Comic in Studio's dialog, with its kind and page count A Comic card opens a kind and a page count, each kind at its own length, with one line on what the pictures cost; the comic opens in the PDF viewer. Translated, with comic added to the glossary. --- .agents/skills/translate/glossary.md | 1 + .../frontend/src/components/ui/icons.tsx | 2 + .../frontend/src/features/studio/api.ts | 8 +- .../studio/format-options/comic-kinds.ts | 105 ++++++++++++++++++ .../format-options/comic-options-form.tsx | 83 ++++++++++++++ .../studio/format-options/comic-options.tsx | 28 +++++ .../format-options/use-format-options.ts | 6 +- .../src/features/studio/studio-formats.ts | 16 ++- .../src/features/studio/studio-panel.test.tsx | 58 ++++++++++ .../src/features/studio/studio-panel.tsx | 5 + .../src/features/studio/viewers/registry.tsx | 2 + surfsense_local/frontend/translations/de.json | 15 +++ surfsense_local/frontend/translations/en.json | 15 +++ surfsense_local/frontend/translations/es.json | 15 +++ surfsense_local/frontend/translations/fr.json | 15 +++ surfsense_local/frontend/translations/hi.json | 15 +++ surfsense_local/frontend/translations/ja.json | 15 +++ surfsense_local/frontend/translations/ko.json | 15 +++ .../frontend/translations/pt-BR.json | 15 +++ surfsense_local/frontend/translations/ru.json | 15 +++ .../frontend/translations/zh-CN.json | 15 +++ 21 files changed, 458 insertions(+), 6 deletions(-) create mode 100644 surfsense_local/frontend/src/features/studio/format-options/comic-kinds.ts create mode 100644 surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx create mode 100644 surfsense_local/frontend/src/features/studio/format-options/comic-options.tsx diff --git a/.agents/skills/translate/glossary.md b/.agents/skills/translate/glossary.md index e242c26347..e754578086 100644 --- a/.agents/skills/translate/glossary.md +++ b/.agents/skills/translate/glossary.md @@ -41,6 +41,7 @@ SurfSense, Studio, Markdown, llama.cpp, GGUF, Hugging Face, API, MCP, MCP's tran | quiz | クイズ | 퀴즈 | 测验 | क्विज़ | Quiz † | cuestionario | Quiz | Quiz | Тест | | study guide | 学習ガイド | 학습 가이드 | 学习指南 | स्टडी गाइड | Lernzettel † | guía de estudio | Guide de révision | Guia de estudos | конспект | | podcast | ポッドキャスト | 팟캐스트 | 播客 | पॉडकास्ट | Podcast | podcast | Podcast | Podcast | подкаст | +| comic (the Studio format; its kinds keep their own names) | コミック | 만화 | 漫画 | कॉमिक | Comic | cómic | BD | quadrinhos | комикс | | speech (a model reading text aloud) | 音声合成 | 음성 | 语音 | वॉइस | Sprache | voz | voix | voz | речь | | speech recognition (turning recordings and the voice into text, on this computer) | 音声認識 | 음성 인식 | 语音识别 | स्पीच रिकग्निशन | Spracherkennung | reconocimiento de voz | reconnaissance vocale | reconhecimento de voz | распознавание речи | | voice input (dictating into the chat composer) | 音声入力 | 음성 입력 | 语音输入 | वॉइस इनपुट | Spracheingabe | entrada de voz | saisie vocale | entrada de voz | голосовой ввод | diff --git a/surfsense_local/frontend/src/components/ui/icons.tsx b/surfsense_local/frontend/src/components/ui/icons.tsx index e953111ad0..a32fa77152 100644 --- a/surfsense_local/frontend/src/components/ui/icons.tsx +++ b/surfsense_local/frontend/src/components/ui/icons.tsx @@ -16,6 +16,7 @@ import { ChartScatterIcon as ChartScatterIconData, AudioWaveformIcon as AudioWaveformIconData, BanIcon as BanIconData, + BookImageIcon as BookImageIconData, BotIcon as BotIconData, Bug01Icon as Bug01IconData, CancelCircleHalfDotIcon as CancelCircleHalfDotIconData, @@ -154,6 +155,7 @@ export const ArrowRightIcon = createIcon(ArrowRightIconData) export const ArrowRight02Icon = createIcon(ArrowRight02IconData) export const ArrowUp02Icon = createIcon(ArrowUp02IconData) export const BanIcon = createIcon(BanIconData) +export const BookImageIcon = createIcon(BookImageIconData) export const BotIcon = createIcon(BotIconData) export const BugIcon = createIcon(Bug01IconData) export const CancelCircleHalfDotIcon = createIcon(CancelCircleHalfDotIconData) diff --git a/surfsense_local/frontend/src/features/studio/api.ts b/surfsense_local/frontend/src/features/studio/api.ts index 713a1cd012..96de9ce28d 100644 --- a/surfsense_local/frontend/src/features/studio/api.ts +++ b/surfsense_local/frontend/src/features/studio/api.ts @@ -111,7 +111,13 @@ export type StudioJobCreate = { // What the server resolves into the job's sources, folders included. source_scope?: SourceScope prompt?: string - options?: PodcastBrief + options?: PodcastBrief | ComicOptions +} + +/** A comic job's options, checked by the server against its kinds (modules/artifacts/comic/). */ +export type ComicOptions = { + kind: string + pages: number } export const PODCAST_STYLES = [ diff --git a/surfsense_local/frontend/src/features/studio/format-options/comic-kinds.ts b/surfsense_local/frontend/src/features/studio/format-options/comic-kinds.ts new file mode 100644 index 0000000000..98da70b7e4 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/comic-kinds.ts @@ -0,0 +1,105 @@ +import { intl } from "@/i18n/intl" + +// The server's kinds (backend/modules/artifacts/comic/kinds.py), key for key, +// each with the length it starts at. +export const COMIC_KINDS = [ + { + key: "western", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_western_label", + defaultMessage: "Western comic", + }), + }, + { + key: "graphic_novel", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_graphic_novel_label", + defaultMessage: "Graphic novel", + }), + }, + { + key: "bande_dessinee", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_bande_dessinee_label", + defaultMessage: "Bande dessinée", + }), + }, + { + key: "manga", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_manga_label", + defaultMessage: "Manga", + }), + }, + { + key: "manhua", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_manhua_label", + defaultMessage: "Manhua", + }), + }, + { + key: "webtoon", + pages: 8, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_webtoon_label", + defaultMessage: "Webtoon", + }), + }, + { + key: "comic_strip", + pages: 1, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_comic_strip_label", + defaultMessage: "Comic strip", + }), + }, + { + key: "yonkoma", + pages: 1, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_yonkoma_label", + defaultMessage: "Yonkoma (4-koma)", + }), + }, + { + key: "picture_book", + pages: 4, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_picture_book_label", + defaultMessage: "Picture book", + }), + }, + { + key: "storyboard", + pages: 6, + label: () => + intl.formatMessage({ + id: "studio_comic_kind_storyboard_label", + defaultMessage: "Storyboard", + }), + }, +] as const + +export type ComicKind = (typeof COMIC_KINDS)[number]["key"] + +// Each picture is billed or takes minutes, so a comic stops here. +export const MAX_COMIC_PAGES = 12 + +export function startingPages(kind: ComicKind): number { + return COMIC_KINDS.find((entry) => entry.key === kind)?.pages ?? 4 +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx b/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx new file mode 100644 index 0000000000..4a7b806662 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx @@ -0,0 +1,83 @@ +import { useId } from "react" + +import { Select } from "@/components/ui/select" +import { intl } from "@/i18n/intl" + +import { COMIC_KINDS, type ComicKind, MAX_COMIC_PAGES } from "./comic-kinds" + +const PAGE_COUNTS = Array.from({ length: MAX_COMIC_PAGES }, (_, i) => i + 1) + +/** The kind of comic and how many pictures it draws, with what that costs. */ +export function ComicOptionsForm({ + kind, + pages, + onKindChange, + onPagesChange, +}: { + kind: ComicKind + pages: number + onKindChange: (kind: ComicKind) => void + onPagesChange: (pages: number) => void +}) { + const id = useId() + return ( + <div className="space-y-2"> + <div className="grid grid-cols-[2fr_1fr] gap-2"> + <div className="space-y-1"> + <label + htmlFor={`${id}-kind`} + className="text-xs font-medium text-muted-foreground" + > + {intl.formatMessage({ + id: "studio_comic_kind_label", + defaultMessage: "Kind", + })} + </label> + <Select + id={`${id}-kind`} + value={kind} + onChange={(event) => onKindChange(event.target.value as ComicKind)} + > + {COMIC_KINDS.map((entry) => ( + <option key={entry.key} value={entry.key}> + {entry.label()} + </option> + ))} + </Select> + </div> + <div className="space-y-1"> + <label + htmlFor={`${id}-pages`} + className="text-xs font-medium text-muted-foreground" + > + {intl.formatMessage({ + id: "studio_comic_pages_label", + defaultMessage: "Pages", + })} + </label> + <Select + id={`${id}-pages`} + value={String(pages)} + onChange={(event) => onPagesChange(Number(event.target.value))} + > + {PAGE_COUNTS.map((count) => ( + <option key={count} value={count}> + {intl.formatNumber(count)} + </option> + ))} + </Select> + </div> + </div> + <p className="text-xs text-muted-foreground"> + {intl.formatMessage( + { + id: "studio_comic_cost_body", + defaultMessage: + "Draws {count, plural, one {# picture} other {# pictures}} with your image model. Each is billed by its provider, or takes minutes on this computer.", + }, + { count: pages } + )} + </p> + </div> + ) +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/comic-options.tsx b/surfsense_local/frontend/src/features/studio/format-options/comic-options.tsx new file mode 100644 index 0000000000..7246cca6ac --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/comic-options.tsx @@ -0,0 +1,28 @@ +import { useState } from "react" + +import { type ComicKind, startingPages } from "./comic-kinds" +import { ComicOptionsForm } from "./comic-options-form" +import type { FormatOptions } from "./format-options" + +/** A comic starts as a four-page Western comic; a new kind starts at its own length. */ +export function useComicOptions(format: string): FormatOptions | null { + const [kind, setKind] = useState<ComicKind>("western") + const [pages, setPages] = useState(startingPages("western")) + if (format !== "comic") return null + + return { + panel: ( + <ComicOptionsForm + kind={kind} + pages={pages} + onKindChange={(next) => { + setKind(next) + setPages(startingPages(next)) + }} + onPagesChange={setPages} + /> + ), + job: { format, options: { kind, pages } }, + ready: true, + } +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts b/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts index 044cc7f4e9..445a7d246c 100644 --- a/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts +++ b/surfsense_local/frontend/src/features/studio/format-options/use-format-options.ts @@ -1,4 +1,5 @@ import type { StudioFormat } from "../api" +import { useComicOptions } from "./comic-options" import type { FormatOptions } from "./format-options" import { usePictureTypeOptions } from "./picture-type-options" import { usePodcastOptions } from "./podcast-options" @@ -13,5 +14,8 @@ export function useFormatOptions( // Every hook runs on every render; each answers only for its own format. const podcast = usePodcastOptions(format, workspaceId, onSetUpVoices) const picture = usePictureTypeOptions(format, catalog) - return podcast ?? picture ?? { panel: null, job: { format }, ready: true } + const comic = useComicOptions(format) + return ( + podcast ?? picture ?? comic ?? { panel: null, job: { format }, ready: true } + ) } diff --git a/surfsense_local/frontend/src/features/studio/studio-formats.ts b/surfsense_local/frontend/src/features/studio/studio-formats.ts index 6798cb61dc..925dc044f9 100644 --- a/surfsense_local/frontend/src/features/studio/studio-formats.ts +++ b/surfsense_local/frontend/src/features/studio/studio-formats.ts @@ -2,6 +2,7 @@ import type { ComponentType } from "react" import { AiSearchLinesIcon, + BookImageIcon, Cards01Icon, ChartHistogramIcon, File02Icon, @@ -37,6 +38,7 @@ export const FORMAT_ICONS: Record< podcast: PodcastIcon, image: Image01Icon, infographic: ChartHistogramIcon, + comic: BookImageIcon, } // The backend's catalog sends English labels; a known key shows its own. @@ -101,22 +103,28 @@ const FORMAT_LABELS: Record<string, () => string> = { id: "studio_format_infographic_label", defaultMessage: "Infographic", }), + comic: () => + intl.formatMessage({ + id: "studio_format_comic_label", + defaultMessage: "Comic", + }), } export function formatLabel(entry: { key: string; label: string }): string { return FORMAT_LABELS[entry.key]?.() ?? entry.label } +const DRAWN_FORMATS = new Set(["image", "infographic", "comic"]) + // The formats offered before the API answers, and if it never does. The // backend's catalog replaces this as soon as it loads. export function studioCatalog(): StudioFormat[] { return Object.keys(FORMAT_LABELS).map((key) => ({ key, label: FORMAT_LABELS[key](), - requires_model_types: - key === "image" || key === "infographic" - ? ["image_gen", "text_gen"] - : ["text_gen"], + requires_model_types: DRAWN_FORMATS.has(key) + ? ["image_gen", "text_gen"] + : ["text_gen"], available: true, unavailable_reason: null, })) diff --git a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx index c963b4e648..a12abef7fc 100644 --- a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx +++ b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx @@ -351,6 +351,64 @@ describe("studio panel", () => { }) }) + it("makes a comic of the kind and length chosen, and says what it costs", async () => { + const fetchMock = vi.fn( + async (input: RequestInfo | URL, init?: RequestInit) => { + const path = String(input) + if (path === "/workspaces/1/studio/formats") { + return Response.json([ + { + key: "comic", + label: "Comic", + requires_model_types: ["image_gen", "text_gen"], + available: true, + unavailable_reason: null, + }, + ]) + } + if (path === "/workspaces/1/studio/jobs" && init?.method === "POST") { + return Response.json( + { ...pendingArtifact, format: "comic", title: "Comic" }, + { status: 201 } + ) + } + if (path === "/workspaces/1/artifacts") return Response.json([]) + return Response.json({ detail: "not found" }, { status: 404 }) + } + ) + vi.stubGlobal("fetch", fetchMock) + const user = userEvent.setup() + + renderStudio() + + await user.click(await screen.findByRole("button", { name: "Comic" })) + const dialog = screen.getByRole("dialog", { name: "Comic" }) + const kind = within(dialog).getByLabelText("Kind") as HTMLSelectElement + const pages = within(dialog).getByLabelText("Pages") as HTMLSelectElement + expect([kind.value, pages.value]).toEqual(["western", "4"]) + expect(within(dialog).getByText(/Draws 4 pictures/)).toBeTruthy() + + // A webtoon counts beats, and starts at its own length. + await user.selectOptions(kind, "webtoon") + expect(pages.value).toBe("8") + await user.selectOptions(kind, "manga") + await user.selectOptions(pages, "6") + expect(within(dialog).getByText(/Draws 6 pictures/)).toBeTruthy() + await user.click(within(dialog).getByRole("button", { name: /Generate/ })) + + const jobCall = await vi.waitFor(() => + fetchMock.mock.calls.find( + ([path, init]) => + path === "/workspaces/1/studio/jobs" && init?.method === "POST" + ) + ) + expect(JSON.parse(String(jobCall?.[1]?.body))).toEqual({ + format: "comic", + document_ids: [4], + options: { kind: "manga", pages: 6 }, + }) + }) + it("opens the podcast brief for review and sends it with the job", async () => { const brief = { language: "en-US", diff --git a/surfsense_local/frontend/src/features/studio/studio-panel.tsx b/surfsense_local/frontend/src/features/studio/studio-panel.tsx index d372105fb4..01d35e372d 100644 --- a/surfsense_local/frontend/src/features/studio/studio-panel.tsx +++ b/surfsense_local/frontend/src/features/studio/studio-panel.tsx @@ -92,6 +92,11 @@ const FORMAT_HINTS: Record<string, () => string> = { id: "studio_format_image_tooltip", defaultMessage: "Generate an AI image based on your sources", }), + comic: () => + intl.formatMessage({ + id: "studio_format_comic_tooltip", + defaultMessage: "Generate an AI comic based on your sources", + }), } function catalogFormats(formats: StudioFormat[]) { diff --git a/surfsense_local/frontend/src/features/studio/viewers/registry.tsx b/surfsense_local/frontend/src/features/studio/viewers/registry.tsx index 13464503ae..d954e0491b 100644 --- a/surfsense_local/frontend/src/features/studio/viewers/registry.tsx +++ b/surfsense_local/frontend/src/features/studio/viewers/registry.tsx @@ -59,6 +59,8 @@ const ARTIFACT_VIEWERS: Record<string, ComponentType<ArtifactViewerProps>> = { infographic: MediaViewer, xlsx: XlsxViewer, pdf: PdfViewer, + // A comic is its bound pages, one PDF. + comic: PdfViewer, pptx: PptxViewer, html: HtmlViewer, docx: DocxViewer, diff --git a/surfsense_local/frontend/translations/de.json b/surfsense_local/frontend/translations/de.json index 8561fc1a46..4508d90773 100644 --- a/surfsense_local/frontend/translations/de.json +++ b/surfsense_local/frontend/translations/de.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "Erstellung von {name} fehlgeschlagen", "studio_artifact_row_script_failed_tooltip": "Das Dokumentskript ist fehlgeschlagen. Bitte den Agenten, es zu beheben.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Zeichnet {count, plural, one {# Bild} other {# Bilder}} mit deinem Bildmodell. Jedes rechnet dein Anbieter ab, oder es dauert auf diesem Computer einige Minuten.", + "studio_comic_kind_bande_dessinee_label": "Frankobelgischer Comic", + "studio_comic_kind_comic_strip_label": "Comicstrip", + "studio_comic_kind_graphic_novel_label": "Graphic Novel", + "studio_comic_kind_label": "Art", + "studio_comic_kind_manga_label": "Manga", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Bilderbuch", + "studio_comic_kind_storyboard_label": "Storyboard", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "Westlicher Comic", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-Koma)", + "studio_comic_pages_label": "Seiten", "studio_composer_brief_status": "Briefing wird vorbereitet…", "studio_composer_generate_button": "Erstellen", "studio_composer_prompt_label": "Prompt (optional)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "Karten mischen", "studio_flashcards_viewer_shuffle_button": "Mischen", "studio_flashcards_viewer_shuffle_error": "Mischen konnte nicht gespeichert werden", + "studio_format_comic_label": "Comic", + "studio_format_comic_tooltip": "Comic mit KI aus deinen Quellen erstellen", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Word-Dokument mit KI aus deinen Quellen erstellen", "studio_format_flashcards_label": "Karteikarten", diff --git a/surfsense_local/frontend/translations/en.json b/surfsense_local/frontend/translations/en.json index 6c1ab77359..490be64251 100644 --- a/surfsense_local/frontend/translations/en.json +++ b/surfsense_local/frontend/translations/en.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "Generation failed for {name}", "studio_artifact_row_script_failed_tooltip": "The document script failed. Ask the agent to fix it.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Draws {count, plural, one {# picture} other {# pictures}} with your image model. Each is billed by its provider, or takes minutes on this computer.", + "studio_comic_kind_bande_dessinee_label": "Bande dessinée", + "studio_comic_kind_comic_strip_label": "Comic strip", + "studio_comic_kind_graphic_novel_label": "Graphic novel", + "studio_comic_kind_label": "Kind", + "studio_comic_kind_manga_label": "Manga", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Picture book", + "studio_comic_kind_storyboard_label": "Storyboard", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "Western comic", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-koma)", + "studio_comic_pages_label": "Pages", "studio_composer_brief_status": "Preparing the brief…", "studio_composer_generate_button": "Generate", "studio_composer_prompt_label": "Prompt (optional)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "Shuffle cards", "studio_flashcards_viewer_shuffle_button": "Shuffle", "studio_flashcards_viewer_shuffle_error": "Shuffle could not be saved", + "studio_format_comic_label": "Comic", + "studio_format_comic_tooltip": "Generate an AI comic based on your sources", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Generate an AI Word document based on your sources", "studio_format_flashcards_label": "Flashcards", diff --git a/surfsense_local/frontend/translations/es.json b/surfsense_local/frontend/translations/es.json index ada53c52f6..33642ec920 100644 --- a/surfsense_local/frontend/translations/es.json +++ b/surfsense_local/frontend/translations/es.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "Falló la generación de {name}", "studio_artifact_row_script_failed_tooltip": "Falló el script del documento. Pide al agente que lo corrija.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Dibuja {count, plural, one {# imagen} many {# imágenes} other {# imágenes}} con tu modelo de imagen. Cada una la cobra su proveedor, o tarda unos minutos en este equipo.", + "studio_comic_kind_bande_dessinee_label": "Cómic franco-belga", + "studio_comic_kind_comic_strip_label": "Tira cómica", + "studio_comic_kind_graphic_novel_label": "Novela gráfica", + "studio_comic_kind_label": "Tipo", + "studio_comic_kind_manga_label": "Manga", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Libro ilustrado", + "studio_comic_kind_storyboard_label": "Guion gráfico", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "Cómic occidental", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-koma)", + "studio_comic_pages_label": "Páginas", "studio_composer_brief_status": "Preparando las indicaciones…", "studio_composer_generate_button": "Generar", "studio_composer_prompt_label": "Prompt (opcional)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "Mezclar las tarjetas", "studio_flashcards_viewer_shuffle_button": "Mezclar", "studio_flashcards_viewer_shuffle_error": "No se pudo guardar la mezcla", + "studio_format_comic_label": "Cómic", + "studio_format_comic_tooltip": "Genera un cómic con IA a partir de tus fuentes", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Genera un documento de Word con IA a partir de tus fuentes", "studio_format_flashcards_label": "Tarjetas didácticas", diff --git a/surfsense_local/frontend/translations/fr.json b/surfsense_local/frontend/translations/fr.json index 9644bce83e..ee9e765818 100644 --- a/surfsense_local/frontend/translations/fr.json +++ b/surfsense_local/frontend/translations/fr.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "Échec de la génération de {name}", "studio_artifact_row_script_failed_tooltip": "Le script du document a échoué. Demandez à l’agent de le corriger.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Dessine {count, plural, one {# image} many {# images} other {# images}} avec votre modèle d’image. Chacune est facturée par son fournisseur, ou prend quelques minutes sur cet ordinateur.", + "studio_comic_kind_bande_dessinee_label": "BD franco-belge", + "studio_comic_kind_comic_strip_label": "Strip", + "studio_comic_kind_graphic_novel_label": "Roman graphique", + "studio_comic_kind_label": "Genre", + "studio_comic_kind_manga_label": "Manga", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Album illustré", + "studio_comic_kind_storyboard_label": "Storyboard", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "Comics", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-koma)", + "studio_comic_pages_label": "Pages", "studio_composer_brief_status": "Préparation du brief…", "studio_composer_generate_button": "Générer", "studio_composer_prompt_label": "Prompt (facultatif)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "Mélanger les cartes", "studio_flashcards_viewer_shuffle_button": "Mélanger", "studio_flashcards_viewer_shuffle_error": "Le mélange n’a pas pu être enregistré", + "studio_format_comic_label": "BD", + "studio_format_comic_tooltip": "Générer une BD par IA à partir de vos sources", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Générer un document Word par IA à partir de vos sources", "studio_format_flashcards_label": "Cartes mémoire", diff --git a/surfsense_local/frontend/translations/hi.json b/surfsense_local/frontend/translations/hi.json index 5072804a5d..26047170ac 100644 --- a/surfsense_local/frontend/translations/hi.json +++ b/surfsense_local/frontend/translations/hi.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "{name} का जनरेशन विफल रहा", "studio_artifact_row_script_failed_tooltip": "दस्तावेज़ स्क्रिप्ट विफल रही। एजेंट से इसे ठीक करने को कहें।", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "आपके इमेज मॉडल से {count, plural, one {# इमेज} other {# इमेज}} बनाता है। हर इमेज का शुल्क उसका प्रोवाइडर लेता है, या इस कंप्यूटर पर उसमें कुछ मिनट लगते हैं।", + "studio_comic_kind_bande_dessinee_label": "फ़्रेंको-बेल्जियन कॉमिक", + "studio_comic_kind_comic_strip_label": "कॉमिक स्ट्रिप", + "studio_comic_kind_graphic_novel_label": "ग्राफ़िक नॉवेल", + "studio_comic_kind_label": "प्रकार", + "studio_comic_kind_manga_label": "मांगा", + "studio_comic_kind_manhua_label": "मनहुआ", + "studio_comic_kind_picture_book_label": "चित्र पुस्तक", + "studio_comic_kind_storyboard_label": "स्टोरीबोर्ड", + "studio_comic_kind_webtoon_label": "वेबटून", + "studio_comic_kind_western_label": "पश्चिमी कॉमिक", + "studio_comic_kind_yonkoma_label": "योनकोमा (4-कोमा)", + "studio_comic_pages_label": "पेज", "studio_composer_brief_status": "ब्रीफ़ तैयार हो रहा है…", "studio_composer_generate_button": "जनरेट करें", "studio_composer_prompt_label": "प्रॉम्प्ट (वैकल्पिक)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "कार्ड शफ़ल करें", "studio_flashcards_viewer_shuffle_button": "शफ़ल करें", "studio_flashcards_viewer_shuffle_error": "शफ़ल सहेजा नहीं जा सका", + "studio_format_comic_label": "कॉमिक", + "studio_format_comic_tooltip": "अपने स्रोतों से AI कॉमिक जनरेट करें", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "अपने स्रोतों से AI Word दस्तावेज़ जनरेट करें", "studio_format_flashcards_label": "फ़्लैशकार्ड", diff --git a/surfsense_local/frontend/translations/ja.json b/surfsense_local/frontend/translations/ja.json index 1e2653d5a5..678bef1f91 100644 --- a/surfsense_local/frontend/translations/ja.json +++ b/surfsense_local/frontend/translations/ja.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "{name}の生成に失敗しました", "studio_artifact_row_script_failed_tooltip": "ドキュメントのスクリプトが失敗しました。エージェントに修正を依頼してください。", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "画像モデルで{count, plural, other {#枚の画像}}を描きます。1枚ごとにプロバイダーから課金されるか、このコンピューターで数分かかります。", + "studio_comic_kind_bande_dessinee_label": "バンド・デシネ", + "studio_comic_kind_comic_strip_label": "コミックストリップ", + "studio_comic_kind_graphic_novel_label": "グラフィックノベル", + "studio_comic_kind_label": "種類", + "studio_comic_kind_manga_label": "マンガ", + "studio_comic_kind_manhua_label": "マンファ(中国の漫画)", + "studio_comic_kind_picture_book_label": "絵本", + "studio_comic_kind_storyboard_label": "絵コンテ", + "studio_comic_kind_webtoon_label": "ウェブトゥーン", + "studio_comic_kind_western_label": "西洋コミック", + "studio_comic_kind_yonkoma_label": "4コマ漫画", + "studio_comic_pages_label": "ページ数", "studio_composer_brief_status": "概要を準備しています…", "studio_composer_generate_button": "生成", "studio_composer_prompt_label": "プロンプト(任意)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "カードをシャッフル", "studio_flashcards_viewer_shuffle_button": "シャッフル", "studio_flashcards_viewer_shuffle_error": "シャッフルを保存できませんでした", + "studio_format_comic_label": "コミック", + "studio_format_comic_tooltip": "ソースをもとにAIでコミックを生成", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "ソースをもとにAIでWord文書を生成", "studio_format_flashcards_label": "フラッシュカード", diff --git a/surfsense_local/frontend/translations/ko.json b/surfsense_local/frontend/translations/ko.json index 2f793c82c7..397bb4febe 100644 --- a/surfsense_local/frontend/translations/ko.json +++ b/surfsense_local/frontend/translations/ko.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "{name} 생성에 실패했습니다", "studio_artifact_row_script_failed_tooltip": "문서 스크립트가 실패했습니다. 에이전트에게 수정을 요청하세요.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "이미지 모델로 {count, plural, other {이미지 #장}}을 그립니다. 한 장마다 제공업체가 요금을 청구하거나, 이 컴퓨터에서 몇 분이 걸립니다.", + "studio_comic_kind_bande_dessinee_label": "방드 데시네", + "studio_comic_kind_comic_strip_label": "신문 만화", + "studio_comic_kind_graphic_novel_label": "그래픽 노블", + "studio_comic_kind_label": "종류", + "studio_comic_kind_manga_label": "일본 만화", + "studio_comic_kind_manhua_label": "중국 만화", + "studio_comic_kind_picture_book_label": "그림책", + "studio_comic_kind_storyboard_label": "스토리보드", + "studio_comic_kind_webtoon_label": "웹툰", + "studio_comic_kind_western_label": "서양 만화", + "studio_comic_kind_yonkoma_label": "4컷 만화", + "studio_comic_pages_label": "페이지 수", "studio_composer_brief_status": "개요를 준비하는 중…", "studio_composer_generate_button": "생성", "studio_composer_prompt_label": "프롬프트 (선택 사항)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "카드 섞기", "studio_flashcards_viewer_shuffle_button": "섞기", "studio_flashcards_viewer_shuffle_error": "섞기를 저장하지 못했습니다", + "studio_format_comic_label": "만화", + "studio_format_comic_tooltip": "소스를 바탕으로 AI 만화 생성", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "소스를 바탕으로 AI Word 문서 생성", "studio_format_flashcards_label": "플래시카드", diff --git a/surfsense_local/frontend/translations/pt-BR.json b/surfsense_local/frontend/translations/pt-BR.json index 3e30d0108c..ee0c2510ae 100644 --- a/surfsense_local/frontend/translations/pt-BR.json +++ b/surfsense_local/frontend/translations/pt-BR.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "Falha na geração de {name}", "studio_artifact_row_script_failed_tooltip": "O script do documento falhou. Peça ao agente para corrigi-lo.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Desenha {count, plural, one {# imagem} many {# imagens} other {# imagens}} com seu modelo de imagem. Cada uma é cobrada pelo provedor, ou leva alguns minutos neste computador.", + "studio_comic_kind_bande_dessinee_label": "Quadrinho franco-belga", + "studio_comic_kind_comic_strip_label": "Tirinha", + "studio_comic_kind_graphic_novel_label": "Romance gráfico", + "studio_comic_kind_label": "Tipo", + "studio_comic_kind_manga_label": "Mangá", + "studio_comic_kind_manhua_label": "Manhua", + "studio_comic_kind_picture_book_label": "Livro ilustrado", + "studio_comic_kind_storyboard_label": "Storyboard", + "studio_comic_kind_webtoon_label": "Webtoon", + "studio_comic_kind_western_label": "HQ ocidental", + "studio_comic_kind_yonkoma_label": "Yonkoma (4-koma)", + "studio_comic_pages_label": "Páginas", "studio_composer_brief_status": "Preparando o briefing…", "studio_composer_generate_button": "Gerar", "studio_composer_prompt_label": "Prompt (opcional)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "Embaralhar os cartões", "studio_flashcards_viewer_shuffle_button": "Embaralhar", "studio_flashcards_viewer_shuffle_error": "Não foi possível salvar o embaralhamento", + "studio_format_comic_label": "Quadrinhos", + "studio_format_comic_tooltip": "Gerar quadrinhos com IA a partir das suas fontes", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Gerar um documento do Word com IA a partir das suas fontes", "studio_format_flashcards_label": "Flashcards", diff --git a/surfsense_local/frontend/translations/ru.json b/surfsense_local/frontend/translations/ru.json index 300e59d042..06637518ea 100644 --- a/surfsense_local/frontend/translations/ru.json +++ b/surfsense_local/frontend/translations/ru.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "Не удалось создать {name}", "studio_artifact_row_script_failed_tooltip": "Скрипт документа не сработал. Попросите агента исправить его.", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "Рисует {count, plural, one {# изображение} few {# изображения} many {# изображений} other {# изображения}} вашей моделью изображений. Каждое оплачивается у провайдера или занимает несколько минут на этом компьютере.", + "studio_comic_kind_bande_dessinee_label": "Франко-бельгийский комикс", + "studio_comic_kind_comic_strip_label": "Комикс-стрип", + "studio_comic_kind_graphic_novel_label": "Графический роман", + "studio_comic_kind_label": "Вид", + "studio_comic_kind_manga_label": "Манга", + "studio_comic_kind_manhua_label": "Маньхуа", + "studio_comic_kind_picture_book_label": "Книжка с картинками", + "studio_comic_kind_storyboard_label": "Раскадровка", + "studio_comic_kind_webtoon_label": "Вебтун", + "studio_comic_kind_western_label": "Западный комикс", + "studio_comic_kind_yonkoma_label": "Ёнкома (4-кома)", + "studio_comic_pages_label": "Страницы", "studio_composer_brief_status": "Подготовка брифа…", "studio_composer_generate_button": "Создать", "studio_composer_prompt_label": "Промпт (необязательно)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "Перемешать карточки", "studio_flashcards_viewer_shuffle_button": "Перемешать", "studio_flashcards_viewer_shuffle_error": "Не удалось сохранить перемешивание", + "studio_format_comic_label": "Комикс", + "studio_format_comic_tooltip": "Создать с помощью ИИ комикс на основе ваших источников", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "Создать с помощью ИИ документ Word на основе ваших источников", "studio_format_flashcards_label": "Карточки", diff --git a/surfsense_local/frontend/translations/zh-CN.json b/surfsense_local/frontend/translations/zh-CN.json index 8b9537450e..c88300126d 100644 --- a/surfsense_local/frontend/translations/zh-CN.json +++ b/surfsense_local/frontend/translations/zh-CN.json @@ -1891,6 +1891,19 @@ "studio_artifact_row_script_failed_aria": "{name} 生成失败", "studio_artifact_row_script_failed_tooltip": "文档脚本失败。请让智能体修复它。", "studio_artifact_row_version_label": "v{version, number}", + "studio_comic_cost_body": "用你的图像模型绘制 {count, plural, other {# 张图片}}。每张由提供商计费,或在这台电脑上需要几分钟。", + "studio_comic_kind_bande_dessinee_label": "法比漫画", + "studio_comic_kind_comic_strip_label": "漫画条", + "studio_comic_kind_graphic_novel_label": "图像小说", + "studio_comic_kind_label": "类型", + "studio_comic_kind_manga_label": "日本漫画", + "studio_comic_kind_manhua_label": "国漫", + "studio_comic_kind_picture_book_label": "绘本", + "studio_comic_kind_storyboard_label": "分镜", + "studio_comic_kind_webtoon_label": "条漫", + "studio_comic_kind_western_label": "欧美漫画", + "studio_comic_kind_yonkoma_label": "四格漫画", + "studio_comic_pages_label": "页数", "studio_composer_brief_status": "正在准备播客设定…", "studio_composer_generate_button": "生成", "studio_composer_prompt_label": "提示词(可选)", @@ -1958,6 +1971,8 @@ "studio_flashcards_viewer_shuffle_aria": "打乱卡片", "studio_flashcards_viewer_shuffle_button": "打乱", "studio_flashcards_viewer_shuffle_error": "无法保存打乱结果", + "studio_format_comic_label": "漫画", + "studio_format_comic_tooltip": "根据你的来源用 AI 生成漫画", "studio_format_docx_label": "Word", "studio_format_docx_tooltip": "根据你的来源用 AI 生成 Word 文档", "studio_format_flashcards_label": "记忆卡片", From 06c50c90c5f4d83e7645fe1ea7101d3010ac8bcb Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:20:22 +0530 Subject: [PATCH 16/33] feat(local): the agent binds its drawn pages into one comic with surfsense_bind_comic bind_comic takes the pictures' names in reading order, with lines for a page's strip, a layout and a direction, and artifact_id for the comic's next version. A page still being drawn makes the comic wait. The job binds with the shared binder and asks no model, so it starts no local image model. The skill binds with it, and pdf.md's script is gone. --- .../modules/agent/agent_threads/steps.py | 4 + .../agent/skills/surfsense-pictures/SKILL.md | 11 +- .../agent/skills/surfsense-pictures/comics.md | 16 +- .../agent/skills/surfsense-pictures/pdf.md | 89 -------- .../modules/agent/tool_endpoint/bind_comic.py | 214 ++++++++++++++++++ .../agent/tool_endpoint/offered_tools.py | 2 + .../tool_endpoint/same_title_documents.py | 2 +- .../modules/artifacts/comic/binding_recipe.py | 54 +++++ .../modules/artifacts/comic/bound_comic.py | 172 ++++++++++++++ .../modules/artifacts/local_image_demand.py | 8 +- .../integration/agent/test_bind_comic.py | 205 +++++++++++++++++ .../integration/agent/test_tool_endpoint.py | 2 + .../tests/unit/agent/test_pictures_skill.py | 137 +---------- surfsense_local/backend/worker/studio/job.py | 31 ++- .../media/visual/comic_binding/bound_pages.py | 32 +++ 15 files changed, 747 insertions(+), 232 deletions(-) delete mode 100644 surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md create mode 100644 surfsense_local/backend/modules/agent/tool_endpoint/bind_comic.py create mode 100644 surfsense_local/backend/modules/artifacts/comic/binding_recipe.py create mode 100644 surfsense_local/backend/modules/artifacts/comic/bound_comic.py create mode 100644 surfsense_local/backend/tests/integration/agent/test_bind_comic.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py diff --git a/surfsense_local/backend/modules/agent/agent_threads/steps.py b/surfsense_local/backend/modules/agent/agent_threads/steps.py index b390b9dc05..9e3725f794 100644 --- a/surfsense_local/backend/modules/agent/agent_threads/steps.py +++ b/surfsense_local/backend/modules/agent/agent_threads/steps.py @@ -3,6 +3,7 @@ from dataclasses import asdict from typing import Any +from modules.agent.tool_endpoint.bind_comic import TOOL_NAME as BIND_COMIC_TOOL from modules.agent.tool_endpoint.confirm_job import TOOL_NAME as CONFIRM_JOB_TOOL from modules.agent.tool_endpoint.convert_document import TOOL_NAME as CONVERT_TOOL from modules.agent.tool_endpoint.make_image import TOOL_NAME as MAKE_IMAGE_TOOL @@ -32,6 +33,8 @@ SAVE_PAGE_STEP = f"{SERVER}_{SAVE_PAGE_TOOL}" # So does a picture's. MAKE_IMAGE_STEP = f"{SERVER}_{MAKE_IMAGE_TOOL}" +# And a bound comic's. +BIND_COMIC_STEP = f"{SERVER}_{BIND_COMIC_TOOL}" # A job run opens the revised copy it filled in, and a confirmation the version it made. RUN_JOB_STEP = f"{SERVER}_{RUN_JOB_TOOL}" CONFIRM_JOB_STEP = f"{SERVER}_{CONFIRM_JOB_TOOL}" @@ -41,6 +44,7 @@ CONVERT_STEP, SAVE_PAGE_STEP, MAKE_IMAGE_STEP, + BIND_COMIC_STEP, RUN_JOB_STEP, CONFIRM_JOB_STEP, ) diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md index 835e2250b9..06a22056f8 100644 --- a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md @@ -1,6 +1,6 @@ --- name: surfsense-pictures -description: Draw pictures with surfsense_make_image - one picture or infographic, or a set that belongs together, such as a comic, a manga, a manhwa or webtoon, a comic strip, a graphic novel, a storyboard or a picture book, bound into one PDF. Load it whenever the user asks for a picture, an illustration, or any set of pictures, or for a change to one you drew. Not for charts of numbers, and not for a web page. +description: Draw pictures with surfsense_make_image - one picture or infographic, or a set that belongs together, such as a comic, a manga, a manhwa or webtoon, a comic strip, a graphic novel, a storyboard or a picture book, bound into one comic with surfsense_bind_comic. Load it whenever the user asks for a picture, an illustration, or any set of pictures, or for a change to one you drew. Not for charts of numbers, and not for a web page. --- # Pictures @@ -11,8 +11,7 @@ description: Draw pictures with surfsense_make_image - one picture or infographi | File | Read it | |---|---| -| `comics.md` | For any kind of comic (Western, manga, manhwa and webtoon, manhua, strip, yonkoma, graphic novel, bande dessinée), a picture book or a storyboard: how each reads, is laid out and is lettered. | -| `pdf.md` | When you bind pictures into a PDF: the script to copy and fill. | +| `comics.md` | For any kind of comic (Western, manga, manhwa and webtoon, manhua, strip, yonkoma, graphic novel, bande dessinée), a picture book or a storyboard: how each reads, is laid out, is lettered and is bound. | ## One picture @@ -32,8 +31,8 @@ A comic, a storyboard, a picture book, or any pictures that must look like one s 3. **Draw the first alone,** and look at it. When it is wrong, change it before going on: every later picture copies its characters. 4. **Draw the rest with the first as their reference**: `references: ["image-<first>"]`. Give each its own call, all in one step, so a remote model draws them side by side. A character picture the user gave, or a character sheet they asked for first, is the reference instead. 5. **Title each** "<title> · page <n>", or "· scene <n>": a chat cannot reuse a picture's title. -6. **Bind them** into one PDF with `surfsense_render_document` ([`pdf.md`](pdf.md)). The PDF is what the user keeps. -7. **Answer** with the PDF and one line per picture, so the user can ask for "page 3 again". +6. **Bind them** with `surfsense_bind_comic`, every picture's name in reading order, with the `layout` and `direction` [`comics.md`](comics.md#which-kind) gives its kind. The comic is what the user keeps: one artifact in Studio, a PDF. +7. **Answer** with the comic and one line per picture, so the user can ask for "page 3 again". ## When the model cannot take a picture in @@ -46,4 +45,4 @@ Never send the user to Settings for this. ## Changing a set later -Change one picture with its `artifact_id`, then read the PDF's script with `surfsense_read_document`, put the new name in place of the old, and render the PDF's next version. To add a picture, draw it with the first as reference and render again. +Change one picture with its `artifact_id`, then call `surfsense_bind_comic` with the comic's `artifact_id` and every page, the new picture's name in place of the old: that is the comic's next version. To add a picture, draw it with the first as reference and bind again. diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md index e63ffd130c..ab4c74cf12 100644 --- a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/comics.md @@ -1,19 +1,19 @@ # Comics -A comic of any kind: one picture per page, or per panel for a scrolling one, every picture from the same character sheet and style line, bound into one PDF. Follow the set's steps in [`SKILL.md`](SKILL.md#a-set-of-pictures); this file is what is particular to comics. +A comic of any kind: one picture per page, or per panel for a scrolling one, every picture from the same character sheet and style line, bound into one comic with `surfsense_bind_comic`. Follow the set's steps in [`SKILL.md`](SKILL.md#a-set-of-pictures); this file is what is particular to comics. ## Which kind Take the kind the user names; when they name none, a Western comic. The kind decides how it reads, how its pictures are shaped and bound, and how it looks. A style the user asks for (noir, chibi, watercolour, superhero) goes in the style line on top of it. -| Kind | Reads | Each picture | Binding (`pdf.md`) | Looks | +| Kind | Reads | Each picture | Binding (`surfsense_bind_comic`) | Looks | |---|---|---|---|---| -| Western comic | left to right | a `portrait` page of 4 to 6 panels | `LAYOUT = "pages"` | full colour, bold inks, caption boxes | +| Western comic | left to right | a `portrait` page of 4 to 6 panels | `layout: "pages"` | full colour, bold inks, caption boxes | | Graphic novel | left to right | a `portrait` page of 3 to 6 panels | `"pages"` | as a Western comic, with room for quiet, wordless panels | | Bande dessinée | left to right | a `portrait` page of 6 to 9 smaller panels | `"pages"` | detailed album pages, often clean "ligne claire" lines and flat colour | -| Manga | **right to left**: panel 1 top right, the last bottom left | a `portrait` page of 3 to 6 panels | `"pages"`, `DIRECTION = "rtl"` | black and white with screentone, speed lines, expressive faces | +| Manga | **right to left**: panel 1 top right, the last bottom left | a `portrait` page of 3 to 6 panels | `"pages"`, `direction: "rtl"` | black and white with screentone, speed lines, expressive faces | | Manhua | left to right | a `portrait` page, or one panel a picture to scroll | `"pages"` or `"scroll"` | full colour, often painterly | -| Manhwa, webtoon | top to bottom, by scrolling | one panel or beat a `portrait` picture, no page | `LAYOUT = "scroll"` | full colour, soft shading, wide gutters between beats | +| Manhwa, webtoon | top to bottom, by scrolling | one panel or beat a `portrait` picture, no page | `layout: "scroll"` | full colour, soft shading, wide gutters between beats | | Comic strip | left to right | one `landscape` picture of 3 or 4 panels in a row | `"pages"`, one page a strip | simple shapes, a punchline in the last panel | | Yonkoma (4-koma) | top to bottom | one `portrait` picture of 4 equal panels stacked | `"pages"`, one page a strip | a manga strip: setup, development, turn, punchline | | Picture book | left to right | one illustration a page, no bubbles | `"pages"`, every page's text as its lines | soft colour, one moment a page | @@ -25,7 +25,7 @@ A manga's panels and bubbles still read right to left inside each page: say so i - **Length.** Four pages, or eight beats for a scrolling comic, when the user names none; at most twelve pictures, since each is billed or takes minutes. A comic strip or a yonkoma is one picture a strip. - **Panels.** For each: where it sits (top, bottom left, wide), what it shows, and its lines. A scrolling comic has one beat a picture instead. -- **Lines.** One short line a bubble, at most about eight words, two or three bubbles a panel. A caption box for narration. Write every line in the plan, word for word: the PDF may need them. A picture book's and a storyboard's text is never in the picture; it goes under it. +- **Lines.** One short line a bubble, at most about eight words, two or three bubbles a panel. A caption box for narration. Write every line in the plan, word for word: the binding may need them. A picture book's and a storyboard's text is never in the picture; it goes under it, as that page's `lines`. ## A page's prompt @@ -47,9 +47,9 @@ For a manga: "Black-and-white manga page with screentone, read right to left. Pa ## Lettering - **Bubbles drawn by the model.** Write each line in the prompt in quotes, in capitals for a Western comic. -- **A misspelt line is not redrawn.** Note the page; the PDF sets that page's lines under it, exactly, in a strip ([`pdf.md`](pdf.md)). A redraw would bill another picture for one letter. +- **A misspelt line is not redrawn.** Note the page, and give its lines as that page's `lines` when you bind: they are set under it, exactly, in a strip. A redraw would bill another picture for one letter. - **On this computer** (the result says "Drawn on this computer"), **or when you cannot see the pictures**, draw every page without words: leave the bubbles out of the prompts, ask for "no text, no lettering, no speech bubbles", and give every page its strip. You cannot rely on the lettering, or check it. ## Before you bind -Look at every picture. Redraw or change one only when it is plainly wrong, such as a missing character, the wrong number of panels, panels in the wrong reading order, or a third arm, and say which and why. Then bind the PDF, every picture's name in `images`, in reading order, with the kind's `LAYOUT` and `DIRECTION`. +Look at every picture. Redraw or change one only when it is plainly wrong, such as a missing character, the wrong number of panels, panels in the wrong reading order, or a third arm, and say which and why. Then call `surfsense_bind_comic`: every picture's name in `pages`, in reading order, each with its `lines` when it needs a strip, and the kind's `layout` and `direction`. A page still being drawn may be named: the comic waits for it, and appears in Studio when its last page is drawn. Tell the user so, and stop. diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md deleted file mode 100644 index c7a13361d5..0000000000 --- a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/pdf.md +++ /dev/null @@ -1,89 +0,0 @@ -# Binding pictures into a PDF - -Render this with `surfsense_render_document`, `format: pdf`, every picture's name in `images`, in reading order. Fill in `TITLE`, `PAGES`, and for the kind of comic `LAYOUT` and `DIRECTION` ([`comics.md`](comics.md#which-kind)); change nothing else unless the user asks. - -- `LAYOUT = "pages"`: one picture per page, full bleed, at comic-book width, each page as tall as its picture. `LAYOUT = "scroll"`: the pictures stacked down one long page, read by scrolling, as a webtoon is; a new page every twelve. -- `DIRECTION = "rtl"` tells a reader the book reads right to left, as a manga does, so its spreads are laid out that way. -- A picture's lines, when given, are set under it in a strip, exactly as written. Give them for a page whose lettering came out wrong, for every page drawn without words, for a picture book's text and a storyboard's notes. -- Pictures are placed as JPEG, so a book of pages stays a few megabytes. -- A picture still being drawn may be named all the same: the PDF waits for it, and appears in Studio when its last picture is drawn. Tell the user so, and stop. - -```python -import os -from io import BytesIO - -from PIL import Image -from reportlab.lib.units import inch -from reportlab.lib.utils import ImageReader -from reportlab.pdfbase.pdfmetrics import stringWidth -from reportlab.pdfgen import canvas - -TITLE = "The Lighthouse Cat" -# Each picture in reading order: its name, and the lines to set under it, or none. -PAGES = [ - ("image-12", []), - ("image-13", ["NORA: The bulb is broken!"]), -] -LAYOUT = "pages" # "pages", or "scroll" for a webtoon -DIRECTION = "ltr" # "ltr", or "rtl" for a manga - -WIDTH = 6.625 * inch -MARGIN = 0.25 * inch -GUTTER = 0.2 * inch -FONT, SIZE, LEADING = "Helvetica", 11, 15 -PER_SCROLL = 12 - - -def wrapped(text, width): - lines, line = [], "" - for word in text.split(): - trial = f"{line} {word}".strip() - if line and stringWidth(trial, FONT, SIZE) > width: - lines.append(line) - line = word - else: - line = trial - return lines + [line] if line else lines - - -def block(name, said): - """A picture as JPEG, its height at the page's width, and its strip's lines.""" - picture = Image.open(os.path.join(os.environ["IMAGES_DIR"], name + ".png")) - picture = picture.convert("RGB") - jpeg = BytesIO() - picture.save(jpeg, "JPEG", quality=90) - jpeg.seek(0) - lines = [line for text in said for line in wrapped(text, WIDTH - 2 * MARGIN)] - strip = len(lines) * LEADING + 2 * MARGIN if lines else 0 - return ImageReader(jpeg), WIDTH * picture.height / picture.width, lines, strip - - -def draw(pdf, blocks, gutter): - """Top to bottom on one page, each picture with its strip under it.""" - height = sum(h + strip for _, h, _, strip in blocks) + gutter * (len(blocks) - 1) - pdf.setPageSize((WIDTH, height)) - top = height - for image, h, lines, strip in blocks: - pdf.drawImage(image, 0, top - h, WIDTH, h) - pdf.setFont(FONT, SIZE) - y = top - h - MARGIN - SIZE - for line in lines: - pdf.drawString(MARGIN, y, line) - y -= LEADING - top -= h + strip + gutter - pdf.showPage() - - -pdf = canvas.Canvas(os.environ["OUTPUT_PATH"]) -pdf.setTitle(TITLE) -if DIRECTION == "rtl": - pdf.setViewerPreference("Direction", "R2L") -blocks = [block(name, said) for name, said in PAGES] -if LAYOUT == "scroll": - for start in range(0, len(blocks), PER_SCROLL): - draw(pdf, blocks[start : start + PER_SCROLL], GUTTER) -else: - for one in blocks: - draw(pdf, [one], 0) -pdf.save() -``` diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/bind_comic.py b/surfsense_local/backend/modules/agent/tool_endpoint/bind_comic.py new file mode 100644 index 0000000000..0848ea0985 --- /dev/null +++ b/surfsense_local/backend/modules/agent/tool_endpoint/bind_comic.py @@ -0,0 +1,214 @@ +"""Bind the pictures the agent drew into one comic, kept in Studio as a PDF.""" + +import time +from dataclasses import dataclass +from typing import Any, cast + +from sqlalchemy.orm import Session + +from modules.agent.tool_endpoint.failed_renders import FailedRunError +from modules.agent.tool_endpoint.job_outcome import wait_for_outcome +from modules.agent.tool_endpoint.registration import TOOL_CALL_SECONDS +from modules.agent.tool_endpoint.rendered_label import ( + RenderedArtifact, + first_line, + queued_line, +) +from modules.agent.tool_endpoint.same_title_documents import refuse_same_title +from modules.agent.tool_endpoint.tool import Tool, ToolCallError +from modules.agent.tool_endpoint.turn_scope import TurnScope +from modules.artifacts.comic.bound_comic import ( + FORMAT, + ComicRefusedError, + create_bound_comic, +) +from modules.artifacts.comic.kinds import MAX_PAGES +from modules.artifacts.comic.reading import DIRECTIONS, LAYOUTS, Direction, Layout +from modules.artifacts.script_documents.version import version_of +from modules.artifacts.waiting_renders.waits import waiting_sentence, waits_for +from modules.documents.models import DocumentStatus + +TOOL_NAME = "bind_comic" + +# Binding reads files and writes one PDF: seconds, unless Studio is busy. +WAIT_SECONDS = 60 +CALL_SECONDS = TOOL_CALL_SECONDS - 10 +LINE_CHARS = 2_000 + +LISTING: dict[str, Any] = { + "name": TOOL_NAME, + "description": ( + "Bind pictures you drew with surfsense_make_image in this chat into one " + "comic, kept in Studio as a PDF: a comic of any kind, a picture book or a " + "storyboard. Follow the surfsense-pictures skill's comics.md for the kind's " + "layout and direction. Give every page in reading order. A page still being " + "drawn may be named: the comic waits for it. To change a page, change its " + "picture, then call this again with the comic's artifact_id and every page." + ), + "inputSchema": { + "type": "object", + "properties": { + "title": {"type": "string", "description": "The comic's title."}, + "pages": { + "type": "array", + "maxItems": MAX_PAGES, + "items": { + "type": "object", + "properties": { + "image": { + "type": "string", + "description": "The picture's name, image-<id>.", + }, + "lines": { + "type": "array", + "items": {"type": "string"}, + "description": ( + "Text set under the picture exactly: for a page " + "drawn without words or misspelt, a picture book's " + "text, a storyboard shot's note. Left out, none." + ), + }, + }, + "required": ["image"], + }, + "description": "Every page, in reading order.", + }, + "layout": { + "type": "string", + "enum": list(LAYOUTS), + "description": ( + "pages, the default: one picture a page. scroll: stacked down " + "one long page, as a webtoon reads." + ), + }, + "direction": { + "type": "string", + "enum": list(DIRECTIONS), + "description": "rtl for a manga; ltr, the default, otherwise.", + }, + "artifact_id": { + "type": "integer", + "description": ( + "To change a comic you bound in this chat: its artifact id. The " + "result is its next version. Leave out for a new comic." + ), + }, + }, + "required": ["title", "pages"], + }, +} + + +@dataclass(frozen=True) +class _Request: + title: str + pages: list[tuple[str, tuple[str, ...]]] + layout: Layout + direction: Direction + base_artifact_id: int | None + + +def bind(session: Session, scope: TurnScope, arguments: dict[str, Any]) -> str: + """Create the comic's version in one short transaction, then wait for its job.""" + began = time.monotonic() + scope.require_known() + request = _request(arguments) + if request.base_artifact_id is None: + refuse_same_title(session, scope.thread_id, request.title, FORMAT) + try: + artifact = create_bound_comic( + session, + scope.workspace_id, + scope.thread_id, + title=request.title, + pages=request.pages, + layout=request.layout, + direction=request.direction, + base_artifact_id=request.base_artifact_id, + ) + except ComicRefusedError as refused: + session.rollback() + raise ToolCallError(str(refused)) from refused + version = version_of(artifact.artifact_metadata) + made = RenderedArtifact( + artifact.id, + " ".join(request.title.split()), + version.number if version is not None else 1, + ) + waiting = waits_for(artifact.artifact_metadata) + session.commit() + if waiting: + return f"{queued_line(made)}\n{waiting_sentence(waiting)}" + + wait = min(WAIT_SECONDS, began + CALL_SECONDS - time.monotonic()) + outcome = wait_for_outcome(session, made.id, wait) + if outcome is None: + raise ToolCallError(f"Artifact {made.id} was deleted before it was ready.") + if outcome.status is DocumentStatus.READY: + count = len(request.pages) + return ( + f"{first_line(made)}\n" + f"A comic of {count} {'page' if count == 1 else 'pages'}, bound as one " + "PDF in Studio. To change a page, change its picture with " + f"surfsense_make_image, then call this tool again with artifact_id " + f"{made.id} and every page." + ) + if outcome.status is DocumentStatus.FAILED: + raise FailedRunError( + f'The comic "{made.title}" (artifact {made.id}) failed: ' + f"{outcome.error_message or 'no reason given'}\n\nTell the user what failed." + ) + if outcome.status is DocumentStatus.CANCELLED: + raise ToolCallError( + f"Artifact {made.id} was cancelled in Studio. Ask the user before " + "binding it again." + ) + return ( + f"{queued_line(made)}\n" + f"The comic is still being bound after {time.monotonic() - began:.0f} s, " + "behind other Studio work. It appears in Studio when it is ready: tell the " + "user so, and do not bind it again." + ) + + +def _request(arguments: dict[str, Any]) -> _Request: + """The call's arguments, refused in a sentence naming the one to fix.""" + title = arguments.get("title") + if not isinstance(title, str) or not title.strip(): + raise ToolCallError("Give the comic a title.") + pages = arguments.get("pages") + if not isinstance(pages, list) or not pages: + raise ToolCallError("Give the pages: every picture's name, in reading order.") + if len(pages) > MAX_PAGES: + raise ToolCallError(f"Bind at most {MAX_PAGES} pages.") + layout = arguments.get("layout") or "pages" + if layout not in LAYOUTS: + raise ToolCallError(f"layout must be one of: {', '.join(LAYOUTS)}.") + direction = arguments.get("direction") or "ltr" + if direction not in DIRECTIONS: + raise ToolCallError(f"direction must be one of: {', '.join(DIRECTIONS)}.") + base = arguments.get("artifact_id") + if base is not None and (not isinstance(base, int) or isinstance(base, bool)): + raise ToolCallError( + "artifact_id must be the number of a comic you bound, or left out." + ) + return _Request( + title.strip(), + [_page(page) for page in pages], + cast(Layout, layout), + cast(Direction, direction), + base, + ) + + +def _page(page: object) -> tuple[str, tuple[str, ...]]: + if not isinstance(page, dict) or not isinstance(page.get("image"), str): + raise ToolCallError('Each page is {"image": "image-<id>"}, with lines if any.') + lines = page.get("lines") or [] + if not isinstance(lines, list) or not all(isinstance(line, str) for line in lines): + raise ToolCallError("A page's lines must be a list of strings.") + kept = tuple(line.strip()[:LINE_CHARS] for line in lines if line.strip()) + return page["image"].strip(), kept + + +BIND_COMIC = Tool(listing=LISTING, run=bind, waits=True) diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py b/surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py index 9f7e41aa45..7dc23b36e9 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py @@ -10,6 +10,7 @@ from api.dependencies import transact from modules.agent.tool_endpoint import replies from modules.agent.tool_endpoint.analyze_data import ANALYZE_DATA +from modules.agent.tool_endpoint.bind_comic import BIND_COMIC from modules.agent.tool_endpoint.confirm_job import CONFIRM_JOB, confirm_listing from modules.agent.tool_endpoint.convert_document import CONVERT_DOCUMENT from modules.agent.tool_endpoint.create_artifact import CREATE_ARTIFACT @@ -54,6 +55,7 @@ RENDER_DOCUMENT, SAVE_PAGE, MAKE_IMAGE, + BIND_COMIC, READ_DOCUMENT, LIST_IMAGES, SOURCE_PAGES, diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py b/surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py index f8839c85cc..814697e272 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/same_title_documents.py @@ -22,7 +22,7 @@ def refuse_same_title( session: Session, thread_id: int, title: str, format: object ) -> None: """Refuse a new document whose title and format match one the thread rendered.""" - if format not in (*DOCUMENT_FORMATS, "html", *EDITABLE_FORMATS): + if format not in (*DOCUMENT_FORMATS, "html", *EDITABLE_FORMATS, "comic"): return # the service refuses it, naming the formats wanted = _key(title) rendered = session.execute( diff --git a/surfsense_local/backend/modules/artifacts/comic/binding_recipe.py b/surfsense_local/backend/modules/artifacts/comic/binding_recipe.py new file mode 100644 index 0000000000..0098efbdbc --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/binding_recipe.py @@ -0,0 +1,54 @@ +"""How the agent's comic is made: the pictures it drew, in reading order, bound as it asked. + +Kept so Regenerate binds the same pages again, and a later version can change one. +""" + +from dataclasses import dataclass +from typing import Any, cast + +from modules.artifacts.comic.reading import DIRECTIONS, LAYOUTS, Direction, Layout +from modules.artifacts.studio_documents.recipe import RECIPE_KEY + +COMIC_BOUND = "comic_bound" + + +@dataclass(frozen=True) +class ComicPage: + # The picture's artifact id. + picture: int + lines: tuple[str, ...] = () + + +@dataclass(frozen=True) +class ComicBinding: + pages: tuple[ComicPage, ...] + layout: Layout + direction: Direction + + +def comic_bound(binding: ComicBinding) -> dict[str, Any]: + return { + "kind": COMIC_BOUND, + "pages": [ + {"picture": page.picture, "lines": list(page.lines)} + for page in binding.pages + ], + "layout": binding.layout, + "direction": binding.direction, + } + + +def comic_binding_of(metadata: dict[str, Any] | None) -> ComicBinding | None: + recipe = (metadata or {}).get(RECIPE_KEY) + if not isinstance(recipe, dict) or recipe.get("kind") != COMIC_BOUND: + return None + layout = recipe.get("layout") + direction = recipe.get("direction") + return ComicBinding( + pages=tuple( + ComicPage(int(page["picture"]), tuple(page.get("lines") or ())) + for page in recipe.get("pages") or () + ), + layout=cast(Layout, layout) if layout in LAYOUTS else "pages", + direction=cast(Direction, direction) if direction in DIRECTIONS else "ltr", + ) diff --git a/surfsense_local/backend/modules/artifacts/comic/bound_comic.py b/surfsense_local/backend/modules/artifacts/comic/bound_comic.py new file mode 100644 index 0000000000..7f05a16647 --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/bound_comic.py @@ -0,0 +1,172 @@ +"""Create the agent's comic: a pending `comic` artifact that binds pictures drawn in its chat. + +Commits before enqueueing: the worker is another process and must find the row. +""" + +import logging + +from sqlalchemy.orm import Session + +from modules.artifacts.comic.binding_recipe import ( + ComicBinding, + ComicPage, + comic_bound, +) +from modules.artifacts.comic.kinds import MAX_PAGES +from modules.artifacts.comic.reading import Direction, Layout +from modules.artifacts.drawn_images.names import ( + ImageStillDrawingError, + is_image_name, + kept_picture, +) +from modules.artifacts.models import Artifact +from modules.artifacts.queued_job import queue_job +from modules.artifacts.script_documents.version import ( + ArtifactVersion, + next_version_number, + version_of, +) +from modules.artifacts.studio_documents.recipe import RECIPE_KEY +from modules.artifacts.waiting_renders.settle import settle_waiting_renders +from modules.artifacts.waiting_renders.waits import KEY as WAITS_FOR +from modules.documents.models import Document, DocumentStatus, DocumentType +from modules.embedding.active import EmbeddingNotChosenError, require_active_index + +logger = logging.getLogger(__name__) + +TITLE_CHARS = 200 +FORMAT = "comic" + + +class ComicRefusedError(ValueError): + """Why the comic cannot be bound, in a sentence the agent can act on.""" + + +def create_bound_comic( + session: Session, + workspace_id: int, + chat_thread_id: int, + *, + title: str, + pages: list[tuple[str, tuple[str, ...]]], + layout: Layout, + direction: Direction, + base_artifact_id: int | None, +) -> Artifact: + """`pages` are picture names in reading order, each with the lines set under it.""" + title = title.strip() + if not 1 <= len(title) <= TITLE_CHARS: + raise ComicRefusedError( + f"Give the comic a title of 1 to {TITLE_CHARS} characters." + ) + if not 1 <= len(pages) <= MAX_PAGES: + raise ComicRefusedError( + f"Bind 1 to {MAX_PAGES} pages: at most {MAX_PAGES} pages." + ) + try: + require_active_index(session) + except EmbeddingNotChosenError as error: + raise ComicRefusedError( + "Studio is not ready on this computer: no embedding model is chosen yet." + ) from error + + waiting: list[int] = [] + for name, _ in pages: + try: + kept_picture(session, workspace_id, chat_thread_id, name) + except ImageStillDrawingError: + waiting.append(int(name.removeprefix("image-"))) + except LookupError as error: + raise ComicRefusedError( + f'This chat has no picture named "{name}" to bind: use a name ' + "surfsense_make_image gave in this chat." + ) from error + base = ( + None + if base_artifact_id is None + else _base_version(session, workspace_id, chat_thread_id, base_artifact_id) + ) + + document = Document( + workspace_id=workspace_id, + title=title, + document_type=DocumentType.ARTIFACT, + status=DocumentStatus.PENDING, + ) + session.add(document) + session.flush() + artifact = Artifact( + document_id=document.id, + workspace_id=workspace_id, + format=FORMAT, + chat_thread_id=chat_thread_id, + ) + session.add(artifact) + session.flush() + version = ( + ArtifactVersion(root=artifact.id, number=1, parent=None) + if base is None + else ArtifactVersion( + root=base.root, + number=next_version_number(session, workspace_id, base.root), + parent=base_artifact_id, + ) + ) + binding = ComicBinding( + pages=tuple(ComicPage(picture_id(name), lines) for name, lines in pages), + layout=layout, + direction=direction, + ) + artifact.artifact_metadata = { + RECIPE_KEY: comic_bound(binding), + "version": version.as_metadata(), + "source_document_ids": [], + "prompt": None, + **({WAITS_FOR: list(dict.fromkeys(waiting))} if waiting else {}), + } + if waiting: + session.commit() + # A picture may have ended between the check and the commit. + settle_waiting_renders(session, render_id=artifact.id) + else: + queue_job(session, artifact) + logger.info( + "studio: enqueued bound comic %s of %s pages, root=%s v%s", + artifact.id, + len(pages), + version.root, + version.number, + ) + return artifact + + +def picture_id(name: str) -> int: + """The artifact id in a picture's name; ComicRefusedError for any other name.""" + if not is_image_name(name): + raise ComicRefusedError( + f'"{name}" is not a picture\'s name: use a name surfsense_make_image gave.' + ) + return int(name.removeprefix("image-")) + + +def _base_version( + session: Session, workspace_id: int, chat_thread_id: int, base_id: int +) -> ArtifactVersion: + base = session.get(Artifact, base_id) + if ( + base is None + or base.workspace_id != workspace_id + or base.chat_thread_id != chat_thread_id + or base.format != FORMAT + ): + raise ComicRefusedError( + f"This chat bound no comic with artifact id {base_id}: name one " + "surfsense_bind_comic gave in this chat." + ) + version = version_of(base.artifact_metadata) + if version is None: + raise ComicRefusedError( + f"Comic {base_id} was made in Studio and keeps no pages to bind again: " + "bind a new comic without artifact_id." + ) + return version diff --git a/surfsense_local/backend/modules/artifacts/local_image_demand.py b/surfsense_local/backend/modules/artifacts/local_image_demand.py index d269dd80cb..0f1bc3b6a4 100644 --- a/surfsense_local/backend/modules/artifacts/local_image_demand.py +++ b/surfsense_local/backend/modules/artifacts/local_image_demand.py @@ -6,6 +6,7 @@ from sqlalchemy import Row, select from sqlalchemy.orm import Session +from modules.artifacts.comic.binding_recipe import comic_binding_of from modules.artifacts.formats import FORMATS, Format from modules.artifacts.image_edits.recipe import image_edit_of from modules.artifacts.models import Artifact @@ -57,10 +58,13 @@ def local_image_demand(session: Session, now: datetime) -> ModelType | None: return _job_type(last, needs) if now - last.updated_at < IDLE else None -def _job_type(job: Row, needs: dict[str, ModelType]) -> ModelType: - """By the job, not its format: an edit is still an `image` or an `infographic`.""" +def _job_type(job: Row, needs: dict[str, ModelType]) -> ModelType | None: + """By the job, not its format: an edit is still an `image` or an `infographic`, + and a comic the agent bound draws nothing.""" if image_edit_of(job.artifact_metadata) is not None: return ModelType.IMAGE_EDIT + if comic_binding_of(job.artifact_metadata) is not None: + return None return needs[job.format] diff --git a/surfsense_local/backend/tests/integration/agent/test_bind_comic.py b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py new file mode 100644 index 0000000000..eb3ca6928e --- /dev/null +++ b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py @@ -0,0 +1,205 @@ +"""The agent binds the pages it drew into a comic: one Studio artifact, a PDF in reading order.""" + +import asyncio +import io +import threading +import time +from datetime import UTC, datetime + +import pytest +from pypdf import PdfReader +from sqlalchemy import Engine + +from modules.artifacts.local_image_demand import local_image_demand +from modules.artifacts.models import Artifact, ArtifactFileRole +from modules.documents.models import DocumentStatus +from shared.config import get_storage_settings +from shared.db import create_session_factory +from tests.integration.agent.test_make_image import ( # noqa: F401 - fixtures + Painter, + _artifact_id, + painter, +) +from tests.integration.agent.tool_endpoint_client import ToolEndpoint +from tests.integration.worker.conftest import stub_model # noqa: F401 + +pytestmark = [ + pytest.mark.integration, + pytest.mark.usefixtures("stub_model", "model_reads_images", "studio_worker"), +] + + +async def _picture(tools: ToolEndpoint, workspace_id: int, title: str) -> str: + text, is_error = await tools.call( + workspace_id, "make_image", {"prompt": f"A page: {title}", "title": title} + ) + assert is_error is False, text + return f"image-{_artifact_id(text)}" + + +def _comic(engine: Engine, artifact_id: int) -> tuple[str, dict, PdfReader]: + """Its format, its metadata and its PDF.""" + with create_session_factory(engine)() as session: + artifact = session.get(Artifact, artifact_id) + assert artifact is not None + primary = next(f for f in artifact.files if f.role is ArtifactFileRole.PRIMARY) + pdf = (get_storage_settings().data_dir / primary.storage_key).read_bytes() + return ( + artifact.format, + dict(artifact.artifact_metadata or {}), + PdfReader(io.BytesIO(pdf)), + ) + + +async def _ended(engine: Engine, artifact_id: int) -> DocumentStatus: + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + with create_session_factory(engine)() as session: + status = session.get(Artifact, artifact_id).document.status + if status not in (DocumentStatus.PENDING, DocumentStatus.PROCESSING): + return status + await asyncio.sleep(0.1) + raise AssertionError(f"artifact {artifact_id} never ended") + + +async def test_the_pages_become_one_comic_in_studio( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """One artifact in Studio, its pages in the order given, read as asked.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + second = await _picture(tools, workspace_id, "Cat · page 2") + + text, is_error = await tools.call( + workspace_id, + "bind_comic", + { + "title": "The Lighthouse Cat", + "pages": [ + {"image": first}, + {"image": second, "lines": ["NORA: The bulb is broken!"]}, + ], + "direction": "rtl", + }, + ) + + assert is_error is False, text + comic_id = _artifact_id(text) + assert text.splitlines()[0] == ( + f"Rendered artifact {comic_id}, version 1: The Lighthouse Cat" + ) + assert "2 pages" in text + format, _, pdf = _comic(engine, comic_id) + assert format == "comic" + assert len(pdf.pages) == 2 + assert "NORA: The bulb is broken!" in pdf.pages[1].extract_text() + assert pdf.trailer["/Root"]["/ViewerPreferences"]["/Direction"] == "/R2L" + listed = (await tools.client.get(f"/workspaces/{workspace_id}/artifacts")).json() + row = next(a for a in listed if a["id"] == comic_id) + assert (row["format"], row["status"]) == ("comic", "ready") + + +async def test_binding_again_with_its_id_makes_the_comics_next_version( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """A page is changed with make_image, then the comic is bound again around it.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + made, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": first}]} + ) + second = await _picture(tools, workspace_id, "Cat · page 2") + + text, is_error = await tools.call( + workspace_id, + "bind_comic", + { + "title": "Cat", + "pages": [{"image": first}, {"image": second}], + "artifact_id": _artifact_id(made), + }, + ) + + assert is_error is False, text + assert ", version 2: Cat" in text.splitlines()[0] + _, metadata, pdf = _comic(engine, _artifact_id(text)) + assert metadata["version"]["parent"] == _artifact_id(made) + assert len(pdf.pages) == 2 + + +async def test_a_page_still_being_drawn_is_waited_for( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A comic drawn slowly on this computer still ends as one PDF, without asking again.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + monkeypatch.setattr("modules.agent.tool_endpoint.make_image.WAIT_SECONDS", 0.2) + painter.hold = threading.Event() + second = await _picture(tools, workspace_id, "Cat · page 2") + + text, is_error = await tools.call( + workspace_id, + "bind_comic", + {"title": "Cat", "pages": [{"image": first}, {"image": second}]}, + ) + + assert is_error is False, text + assert text.startswith("Queued artifact ") + assert f"renders once {second} is drawn" in text + painter.hold.set() + assert await _ended(engine, _artifact_id(text)) is DocumentStatus.READY + + +@pytest.mark.parametrize( + "arguments, words", + [ + ({"pages": [{"image": "image-1"}]}, "Give the comic a title"), + ({"title": "Cat", "pages": []}, "Give the pages"), + ({"title": "Cat", "pages": [{"image": "image-999"}]}, "no picture named"), + ( + {"title": "Cat", "pages": [{"image": "image-1"}], "layout": "spiral"}, + "layout must be one of: pages, scroll", + ), + ( + {"title": "Cat", "pages": [{"image": "image-1"}] * 13}, + "at most 12 pages", + ), + ], +) +async def test_a_call_missing_something_says_what_to_fix( + tools: ToolEndpoint, arguments: dict, words: str +) -> None: + """A small model may leave arguments out; the refusal names the one to fix.""" + workspace_id = await tools.workspace() + + text, is_error = await tools.call(workspace_id, "bind_comic", arguments) + + assert is_error is True + assert words in text + + +async def test_binding_a_comic_does_not_start_the_local_image_model( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """Binding draws nothing: the image model's memory stays the chat model's.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + made, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": first}]} + ) + with create_session_factory(engine)() as session: + comic = session.get(Artifact, _artifact_id(made)) + comic.document.status = DocumentStatus.PROCESSING + session.commit() + + assert local_image_demand(session, datetime.now(UTC)) is None + diff --git a/surfsense_local/backend/tests/integration/agent/test_tool_endpoint.py b/surfsense_local/backend/tests/integration/agent/test_tool_endpoint.py index d9c2f84da8..fc184a5ec4 100644 --- a/surfsense_local/backend/tests/integration/agent/test_tool_endpoint.py +++ b/surfsense_local/backend/tests/integration/agent/test_tool_endpoint.py @@ -88,6 +88,7 @@ async def test_it_offers_its_tools_with_flat_schemas(tools: ToolEndpoint) -> Non "render_document", "save_page", "make_image", + "bind_comic", "read_document", "list_images", "convert_document", @@ -174,6 +175,7 @@ async def test_a_model_that_reads_images_is_also_offered_source_pages( "render_document", "save_page", "make_image", + "bind_comic", "read_document", "list_images", "source_pages", diff --git a/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py index d05b84f542..fef4079f43 100644 --- a/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py +++ b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py @@ -1,16 +1,9 @@ -"""The pictures skill as the agent reads it, and its binding script as a render -runs it: alone, from an empty folder, with the pictures named in IMAGES_DIR.""" +"""The pictures skill as the agent reads it.""" -import io -import os import re -import subprocess -import sys from pathlib import Path import pytest -from PIL import Image -from pypdf import PdfReader from modules.agent.engines.claude_code.skills_plugin import build_skills_plugin from modules.agent.opencode_config import PICTURES_SKILL, skills_folder @@ -38,7 +31,7 @@ def test_the_tool_sends_a_set_of_pictures_to_the_skill() -> None: def test_the_skill_never_sends_the_user_to_settings() -> None: """A model that takes no picture in is said plainly, never set up mid-comic.""" - for name in ("SKILL.md", "comics.md", "pdf.md"): + for name in ("SKILL.md", "comics.md"): # The app writes the path with a chevron; the agent's prompts with ">". assert "Settings \u203a" not in _file(name), name assert "Settings >" not in _file(name), name @@ -51,10 +44,10 @@ def test_a_claude_plan_thread_gets_the_skill_with_its_own_tool_names( plugin = build_skills_plugin(skills_folder(), tmp_path / "plugin") assert f"surfsense:{PICTURES_SKILL}" in plugin.names - for name in ("SKILL.md", "comics.md", "pdf.md"): + for name in ("SKILL.md", "comics.md"): text = (plugin.skills / PICTURES_SKILL / name).read_text(encoding="utf-8") assert not re.search( - r"surfsense_(make_image|render_document|read_document)", text + r"surfsense_(make_image|bind_comic|render_document|read_document)", text ) skill = (plugin.skills / PICTURES_SKILL / "SKILL.md").read_text(encoding="utf-8") assert "mcp__surfsense__make_image" in skill @@ -79,117 +72,11 @@ def test_comics_cover_every_kind_the_user_may_ask_for() -> None: assert kind in comics, kind -def _script( - pages: list[tuple[str, list[str]]], - direction: str = "ltr", - layout: str = "pages", -) -> str: - """The script the skill gives, filled as the agent fills it.""" - (script,) = re.findall(r"```python\n(.*?)```", _file("pdf.md"), re.S) - filled = re.sub( - r"PAGES = \[\n.*?\n\]", f"PAGES = {pages!r}", script, count=1, flags=re.S - ) - filled = filled.replace('DIRECTION = "ltr"', f"DIRECTION = {direction!r}", 1) - filled = filled.replace('LAYOUT = "pages"', f"LAYOUT = {layout!r}", 1) - assert filled != script - assert f"DIRECTION = {direction!r}" in filled - assert f"LAYOUT = {layout!r}" in filled - return filled - - -def _bind(tmp_path: Path, script: str, count: int) -> PdfReader: - images, work = tmp_path / "images", tmp_path / "work" - images.mkdir() - work.mkdir() - # A colour each: ReportLab stores one copy of pictures that are the same. - for n, colour in zip( - range(1, count + 1), ("red", "green", "blue", "white"), strict=False - ): - (images / f"image-{n}.png").write_bytes(_page(colour, (1024, 1536))) - output = tmp_path / "comic.pdf" - subprocess.run( - [sys.executable, "-c", script], - cwd=work, - env={**os.environ, "OUTPUT_PATH": str(output), "IMAGES_DIR": str(images)}, - check=True, - ) - return PdfReader(output) - - -def test_a_manga_reads_right_to_left(tmp_path: Path) -> None: - """A reader then lays its spreads out as a manga's are.""" - pages = [(f"image-{n}", []) for n in range(1, 3)] - - pdf = _bind(tmp_path, _script(pages, direction="rtl"), 2) - - assert pdf.trailer["/Root"]["/ViewerPreferences"]["/Direction"] == "/R2L" - assert len(pdf.pages) == 2 - - -def test_a_webtoon_scrolls_down_one_long_page(tmp_path: Path) -> None: - """Its panels stack in one column, read by scrolling, not page by page.""" - pages = [(f"image-{n}", []) for n in range(1, 4)] - - pdf = _bind(tmp_path, _script(pages, layout="scroll"), 3) - - (page,) = pdf.pages - assert len(page.images) == 3 - width, height = float(page.mediabox.width), float(page.mediabox.height) - assert height > 3 * width * 1.5 - - -def _page(colour: str, size: tuple[int, int]) -> bytes: - out = io.BytesIO() - Image.new("RGB", size, colour).save(out, "PNG") - return out.getvalue() - - -def test_the_pages_are_bound_one_a_page_as_jpeg(tmp_path: Path) -> None: - """Four comic pages in a PDF of a few megabytes, not the tens PNG made.""" - images, work = tmp_path / "images", tmp_path / "work" - images.mkdir() - work.mkdir() - noisy = Image.effect_noise((1024, 1536), 64).convert("RGB") - for n in range(1, 5): - out = io.BytesIO() - noisy.save(out, "PNG") - (images / f"image-{n}.png").write_bytes(out.getvalue()) - output = tmp_path / "comic.pdf" - - subprocess.run( - [sys.executable, "-c", _script([(f"image-{n}", []) for n in range(1, 5)])], - cwd=work, - env={**os.environ, "OUTPUT_PATH": str(output), "IMAGES_DIR": str(images)}, - check=True, - ) - - pdf = PdfReader(output) - assert len(pdf.pages) == 4 - for page in pdf.pages: - (picture,) = page.images - assert picture.name.endswith(".jpg"), picture.name - assert output.stat().st_size < 5 * 1024 * 1024 - - -def test_a_page_with_lines_gets_them_exactly_in_a_strip_below(tmp_path: Path) -> None: - """A page whose lettering came out wrong keeps the exact words, unredrawn.""" - images, work = tmp_path / "images", tmp_path / "work" - images.mkdir() - work.mkdir() - (images / "image-1.png").write_bytes(_page("white", (1024, 1536))) - (images / "image-2.png").write_bytes(_page("white", (1024, 1536))) - output = tmp_path / "comic.pdf" - said = "NORA: The bulb is broken!" - - subprocess.run( - [sys.executable, "-c", _script([("image-1", []), ("image-2", [said])])], - cwd=work, - env={**os.environ, "OUTPUT_PATH": str(output), "IMAGES_DIR": str(images)}, - check=True, - ) - - plain, lettered = PdfReader(output).pages - assert said in lettered.extract_text() - assert said not in plain.extract_text() - assert float(lettered.mediabox.height) > float(plain.mediabox.height) - assert float(lettered.mediabox.width) == float(plain.mediabox.width) +def test_a_comic_is_bound_by_its_own_tool_not_a_script() -> None: + """The binder is the app's; the agent no longer copies a script to fill.""" + assert not (skills_folder() / PICTURES_SKILL / "pdf.md").exists() + for name in ("SKILL.md", "comics.md"): + text = _file(name) + assert "surfsense_bind_comic" in text, name + assert "surfsense_render_document" not in text, name + diff --git a/surfsense_local/backend/worker/studio/job.py b/surfsense_local/backend/worker/studio/job.py index 645561e957..9be2646d0b 100644 --- a/surfsense_local/backend/worker/studio/job.py +++ b/surfsense_local/backend/worker/studio/job.py @@ -1,12 +1,14 @@ import asyncio import logging import time +from pathlib import Path from typing import cast import httpx from sqlalchemy.exc import OperationalError from sqlalchemy.orm import Session +from modules.artifacts.comic.binding_recipe import ComicBinding, comic_binding_of from modules.artifacts.converted_documents.conversion import ( Conversion, conversion_of, @@ -17,7 +19,7 @@ from modules.artifacts.image_edits.editable import EDITABLE_FORMATS from modules.artifacts.image_edits.recipe import ImageEdit, image_edit_of from modules.artifacts.made_file import made_by -from modules.artifacts.models import Artifact +from modules.artifacts.models import Artifact, ArtifactFileRole from modules.artifacts.revised_copies.revision import revision_of from modules.artifacts.script_documents.script_error import PREFIX, script_error from modules.artifacts.script_documents.spec import ( @@ -60,6 +62,7 @@ from worker.studio.converted_document import pipeline as converted_document from worker.studio.job_run import pipeline as job_run from worker.studio.made_file import pipeline as made_file +from worker.studio.media.visual.comic_binding import bound_pages from worker.studio.media.visual.drawn import pipeline as drawn_image from worker.studio.media.visual.drawn.references import reference_pictures from worker.studio.media.visual.edit import pipeline as image_edit @@ -125,6 +128,7 @@ def _generate(session: Session, artifact: Artifact) -> None: refining = refinement(meta) editing = image_edit_of(meta) drawing = image_drawing_of(meta) + binding = comic_binding_of(meta) script = document_script(meta) if renders_as_stored(meta) else None converting = conversion_of(meta) revised: revised_copy.Made | None = None @@ -139,6 +143,8 @@ def _generate(session: Session, artifact: Artifact) -> None: built = _edit_image(session, artifact, document, editing) elif drawing is not None: built = _draw_image(session, artifact, document, drawing) + elif binding is not None: + built = _bind_comic(session, document, binding) elif made_by(meta) is not None: built = made_file.built(artifact, document.title) elif script is not None: @@ -425,6 +431,29 @@ def _draw_image( ) +def _bind_comic(session: Session, document: Document, binding: ComicBinding) -> Built: + """The agent's pages, as drawn when it bound them; no model is asked.""" + pictures = [_picture_file(session, page.picture) for page in binding.pages] + title = document.title + session.commit() + raise_if_cancelled(session, document) + return bound_pages.render(title, binding, pictures) + + +def _picture_file(session: Session, picture_id: int) -> Path: + picture = session.get(Artifact, picture_id) + primary = ( + None + if picture is None + else next( + (f for f in picture.files if f.role is ArtifactFileRole.PRIMARY), None + ) + ) + if primary is None: + raise ValueError(f"image-{picture_id} is gone, so its page cannot be bound") + return get_storage_settings().data_dir / primary.storage_key + + def _keep_spec(artifact: Artifact, spec: dict) -> None: """A Studio draft's spec becomes v1 of its own document; a refine already has its version.""" meta = dict(artifact.artifact_metadata or {}) diff --git a/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py new file mode 100644 index 0000000000..41a414aa63 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py @@ -0,0 +1,32 @@ +"""The agent's comic: the pictures it drew, read from Studio's store and bound as it asked.""" + +from pathlib import Path + +from modules.artifacts.comic.binding_recipe import ComicBinding +from worker.studio.media.visual.comic_binding.bind import BoundPage, bind +from worker.studio.shared.artifact import Built +from worker.studio.shared.text import slug + + +def render(title: str, binding: ComicBinding, pictures: list[Path]) -> Built: + """`pictures` are the pages' files, in the binding's order.""" + pdf = bind( + title, + [ + BoundPage(path.read_bytes(), page.lines) + for path, page in zip(pictures, binding.pages, strict=True) + ], + layout=binding.layout, + direction=binding.direction, + ) + body = [f"# {title}"] + for number, page in enumerate(binding.pages, start=1): + body.append(f"\n## Page {number}\n") + body += [f"- {line}" for line in page.lines] + return Built( + title=title, + markdown="\n".join(body), + primary=pdf, + primary_mime="application/pdf", + primary_filename=f"{slug(title, 'comic')}.pdf", + ) From 160f3c19d79acd0538c9376ed1277946a28a7f0b Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:20:22 +0530 Subject: [PATCH 17/33] feat(local): the thread names a bound comic's step and opens the comic --- .../src/features/agent/agent-steps.test.tsx | 44 +++++++++++++++++++ .../src/features/agent/step-label.tsx | 35 +++++++++++++++ surfsense_local/frontend/translations/de.json | 4 ++ surfsense_local/frontend/translations/en.json | 4 ++ surfsense_local/frontend/translations/es.json | 4 ++ surfsense_local/frontend/translations/fr.json | 4 ++ surfsense_local/frontend/translations/hi.json | 4 ++ surfsense_local/frontend/translations/ja.json | 4 ++ surfsense_local/frontend/translations/ko.json | 4 ++ .../frontend/translations/pt-BR.json | 4 ++ surfsense_local/frontend/translations/ru.json | 4 ++ .../frontend/translations/zh-CN.json | 4 ++ 12 files changed, 119 insertions(+) diff --git a/surfsense_local/frontend/src/features/agent/agent-steps.test.tsx b/surfsense_local/frontend/src/features/agent/agent-steps.test.tsx index bd8100dfc7..3c5eae02d3 100644 --- a/surfsense_local/frontend/src/features/agent/agent-steps.test.tsx +++ b/surfsense_local/frontend/src/features/agent/agent-steps.test.tsx @@ -232,6 +232,50 @@ describe("agent steps", () => { expect(screen.getByText("Asked for a picture")).toBeTruthy() }) + it("opens the comic a bind_comic step made", async () => { + const openArtifact = vi.fn() + const user = userEvent.setup() + render( + <OpenArtifactContext.Provider value={openArtifact}> + <AgentSteps + steps={[ + rendered({ + tool: "surfsense_bind_comic", + input: { title: "The Lighthouse Cat", pages: [] }, + artifact: { id: 70, title: "The Lighthouse Cat", version: 2 }, + }), + ]} + /> + </OpenArtifactContext.Provider> + ) + + await user.click( + screen.getByRole("button", { + name: "Updated The Lighthouse Cat to v2", + }) + ) + expect(openArtifact).toHaveBeenCalledExactlyOnceWith(70) + }) + + it("names a comic still waiting for its pages by its title", () => { + render( + <AgentSteps + steps={[ + rendered({ + tool: "surfsense_bind_comic", + input: { title: "The Lighthouse Cat", pages: [] }, + artifact: undefined, + output: "Queued artifact 70, version 1: The Lighthouse Cat", + }), + ]} + /> + ) + + expect(screen.getByText(/Binding the comic/).textContent).toBe( + "Binding the comic The Lighthouse Cat" + ) + }) + it("names reading a document’s script and listing source images", () => { render( <AgentSteps diff --git a/surfsense_local/frontend/src/features/agent/step-label.tsx b/surfsense_local/frontend/src/features/agent/step-label.tsx index 90674b4025..33d8f405ff 100644 --- a/surfsense_local/frontend/src/features/agent/step-label.tsx +++ b/surfsense_local/frontend/src/features/agent/step-label.tsx @@ -224,6 +224,41 @@ export function stepLabel( defaultMessage: "Asked for a picture", }) } + case "surfsense_bind_comic": { + const made = step.artifact + if (made) { + return (made.created ?? made.version === 1) + ? intl.formatMessage( + { + id: "agent_steps_bind_comic_created_label", + defaultMessage: "Created <doc>{title}</doc> v{version, number}", + }, + { title: made.title, version: made.version, doc: documentTitle } + ) + : intl.formatMessage( + { + id: "agent_steps_bind_comic_updated_label", + defaultMessage: + "Updated <doc>{title}</doc> to v{version, number}", + }, + { title: made.title, version: made.version, doc: documentTitle } + ) + } + // Running, waiting for its pages, refused or failed: the title it was given. + const title = text(input.title) + return title + ? intl.formatMessage( + { + id: "agent_steps_bind_comic_label", + defaultMessage: "Binding the comic <doc>{title}</doc>", + }, + { title, doc: documentTitle } + ) + : intl.formatMessage({ + id: "agent_steps_bind_comic_unnamed_label", + defaultMessage: "Binding a comic", + }) + } case "surfsense_convert_document": { const made = step.artifact return made diff --git a/surfsense_local/frontend/translations/de.json b/surfsense_local/frontend/translations/de.json index 4508d90773..d8b302f9e2 100644 --- a/surfsense_local/frontend/translations/de.json +++ b/surfsense_local/frontend/translations/de.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "{hosts} erlauben", "agent_steps_analyze_data_label": "Analyse <doc>{title}</doc> ausgeführt", "agent_steps_analyze_data_untitled_label": "Daten analysiert", + "agent_steps_bind_comic_created_label": "<doc>{title}</doc> v{version, number} erstellt", + "agent_steps_bind_comic_label": "Comic <doc>{title}</doc> wird gebunden", + "agent_steps_bind_comic_unnamed_label": "Comic wird gebunden", + "agent_steps_bind_comic_updated_label": "<doc>{title}</doc> auf v{version, number} aktualisiert", "agent_steps_confirm_job_label": "Antwortvorschläge in <doc>{title}</doc> bestätigt", "agent_steps_confirm_job_pending_label": "Bestätigung der Antwortvorschläge gestartet", "agent_steps_convert_label": "<doc>{title}</doc> in PDF umgewandelt", diff --git a/surfsense_local/frontend/translations/en.json b/surfsense_local/frontend/translations/en.json index 490be64251..a71f5e4823 100644 --- a/surfsense_local/frontend/translations/en.json +++ b/surfsense_local/frontend/translations/en.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Allow {hosts}", "agent_steps_analyze_data_label": "Ran the analysis <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "Analysed data", + "agent_steps_bind_comic_created_label": "Created <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Binding the comic <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Binding a comic", + "agent_steps_bind_comic_updated_label": "Updated <doc>{title}</doc> to v{version, number}", "agent_steps_confirm_job_label": "Confirmed suggested answers in <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "Started confirming suggested answers", "agent_steps_convert_label": "Converted <doc>{title}</doc> to PDF", diff --git a/surfsense_local/frontend/translations/es.json b/surfsense_local/frontend/translations/es.json index 33642ec920..27cda535db 100644 --- a/surfsense_local/frontend/translations/es.json +++ b/surfsense_local/frontend/translations/es.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Permitir {hosts}", "agent_steps_analyze_data_label": "Ejecutó el análisis <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "Analizó datos", + "agent_steps_bind_comic_created_label": "Creó <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Encuadernando el cómic <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Encuadernando un cómic", + "agent_steps_bind_comic_updated_label": "Actualizó <doc>{title}</doc> a la v{version, number}", "agent_steps_confirm_job_label": "Confirmó respuestas sugeridas en <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "Empezó a confirmar respuestas sugeridas", "agent_steps_convert_label": "Convirtió <doc>{title}</doc> a PDF", diff --git a/surfsense_local/frontend/translations/fr.json b/surfsense_local/frontend/translations/fr.json index ee9e765818..8cd46d9448 100644 --- a/surfsense_local/frontend/translations/fr.json +++ b/surfsense_local/frontend/translations/fr.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Autoriser {hosts}", "agent_steps_analyze_data_label": "A exécuté l’analyse <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "A analysé des données", + "agent_steps_bind_comic_created_label": "A créé <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Assemblage de la BD <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Assemblage d’une BD", + "agent_steps_bind_comic_updated_label": "A mis à jour <doc>{title}</doc> en v{version, number}", "agent_steps_confirm_job_label": "A confirmé des réponses suggérées dans <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "A commencé à confirmer des réponses suggérées", "agent_steps_convert_label": "<doc>{title}</doc> converti en PDF", diff --git a/surfsense_local/frontend/translations/hi.json b/surfsense_local/frontend/translations/hi.json index 26047170ac..7232b1832c 100644 --- a/surfsense_local/frontend/translations/hi.json +++ b/surfsense_local/frontend/translations/hi.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "{hosts} को अनुमति दें", "agent_steps_analyze_data_label": "<doc>{title}</doc> विश्लेषण चलाया", "agent_steps_analyze_data_untitled_label": "डेटा का विश्लेषण किया", + "agent_steps_bind_comic_created_label": "<doc>{title}</doc> v{version, number} बनाया", + "agent_steps_bind_comic_label": "कॉमिक <doc>{title}</doc> जोड़ा जा रहा है", + "agent_steps_bind_comic_unnamed_label": "कॉमिक जोड़ा जा रहा है", + "agent_steps_bind_comic_updated_label": "<doc>{title}</doc> को v{version, number} में अपडेट किया", "agent_steps_confirm_job_label": "<doc>{title}</doc> में सुझाए गए जवाबों की पुष्टि की", "agent_steps_confirm_job_pending_label": "सुझाए गए जवाबों की पुष्टि शुरू की", "agent_steps_convert_label": "<doc>{title}</doc> को PDF में बदला", diff --git a/surfsense_local/frontend/translations/ja.json b/surfsense_local/frontend/translations/ja.json index 678bef1f91..375706c536 100644 --- a/surfsense_local/frontend/translations/ja.json +++ b/surfsense_local/frontend/translations/ja.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "{hosts}を許可", "agent_steps_analyze_data_label": "分析<doc>{title}</doc>を実行", "agent_steps_analyze_data_untitled_label": "データを分析", + "agent_steps_bind_comic_created_label": "<doc>{title}</doc> v{version, number}を作成", + "agent_steps_bind_comic_label": "コミック<doc>{title}</doc>を製本中", + "agent_steps_bind_comic_unnamed_label": "コミックを製本中", + "agent_steps_bind_comic_updated_label": "<doc>{title}</doc>をv{version, number}に更新", "agent_steps_confirm_job_label": "<doc>{title}</doc>の回答案を確定", "agent_steps_confirm_job_pending_label": "回答案の確定を開始", "agent_steps_convert_label": "<doc>{title}</doc>をPDFに変換", diff --git a/surfsense_local/frontend/translations/ko.json b/surfsense_local/frontend/translations/ko.json index 397bb4febe..f2c0882972 100644 --- a/surfsense_local/frontend/translations/ko.json +++ b/surfsense_local/frontend/translations/ko.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "{hosts} 허용", "agent_steps_analyze_data_label": "<doc>{title}</doc> 분석 실행함", "agent_steps_analyze_data_untitled_label": "데이터를 분석함", + "agent_steps_bind_comic_created_label": "<doc>{title}</doc> v{version, number} 생성함", + "agent_steps_bind_comic_label": "만화 <doc>{title}</doc> 제본 중", + "agent_steps_bind_comic_unnamed_label": "만화 제본 중", + "agent_steps_bind_comic_updated_label": "<doc>{title}</doc>을(를) v{version, number}(으)로 업데이트함", "agent_steps_confirm_job_label": "<doc>{title}</doc>의 제안된 답변을 확정함", "agent_steps_confirm_job_pending_label": "제안된 답변 확정을 시작함", "agent_steps_convert_label": "<doc>{title}</doc>을(를) PDF로 변환", diff --git a/surfsense_local/frontend/translations/pt-BR.json b/surfsense_local/frontend/translations/pt-BR.json index ee0c2510ae..dd37c87240 100644 --- a/surfsense_local/frontend/translations/pt-BR.json +++ b/surfsense_local/frontend/translations/pt-BR.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Permitir {hosts}", "agent_steps_analyze_data_label": "Executou a análise <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "Analisou dados", + "agent_steps_bind_comic_created_label": "Criou <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Montando os quadrinhos <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Montando quadrinhos", + "agent_steps_bind_comic_updated_label": "Atualizou <doc>{title}</doc> para a v{version, number}", "agent_steps_confirm_job_label": "Confirmou respostas sugeridas em <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "Começou a confirmar respostas sugeridas", "agent_steps_convert_label": "Converteu <doc>{title}</doc> em PDF", diff --git a/surfsense_local/frontend/translations/ru.json b/surfsense_local/frontend/translations/ru.json index 06637518ea..24979dbe15 100644 --- a/surfsense_local/frontend/translations/ru.json +++ b/surfsense_local/frontend/translations/ru.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "Разрешить {hosts}", "agent_steps_analyze_data_label": "Выполнил анализ <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "Проанализировал данные", + "agent_steps_bind_comic_created_label": "Создал <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "Сборка комикса <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "Сборка комикса", + "agent_steps_bind_comic_updated_label": "Обновил <doc>{title}</doc> до v{version, number}", "agent_steps_confirm_job_label": "Подтвердил предложенные ответы в <doc>{title}</doc>", "agent_steps_confirm_job_pending_label": "Начал подтверждать предложенные ответы", "agent_steps_convert_label": "Преобразовал <doc>{title}</doc> в PDF", diff --git a/surfsense_local/frontend/translations/zh-CN.json b/surfsense_local/frontend/translations/zh-CN.json index c88300126d..1e0ac935ad 100644 --- a/surfsense_local/frontend/translations/zh-CN.json +++ b/surfsense_local/frontend/translations/zh-CN.json @@ -69,6 +69,10 @@ "agent_steps_allow_hosts_button": "允许 {hosts}", "agent_steps_analyze_data_label": "运行了分析 <doc>{title}</doc>", "agent_steps_analyze_data_untitled_label": "分析了数据", + "agent_steps_bind_comic_created_label": "创建了 <doc>{title}</doc> v{version, number}", + "agent_steps_bind_comic_label": "正在装订漫画 <doc>{title}</doc>", + "agent_steps_bind_comic_unnamed_label": "正在装订漫画", + "agent_steps_bind_comic_updated_label": "将 <doc>{title}</doc> 更新到 v{version, number}", "agent_steps_confirm_job_label": "确认了 <doc>{title}</doc> 中的建议答案", "agent_steps_confirm_job_pending_label": "开始确认建议答案", "agent_steps_convert_label": "已将 <doc>{title}</doc> 转换为 PDF", From 165be0f3af681a833033e603b6fc32a649d02ced Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:21:36 +0530 Subject: [PATCH 18/33] feat(local): a comic's pages stay out of Studio's list and go with the comic make_image takes comic_page for each page, binding marks any page drawn without it, and a page's edits keep the mark. The artifacts list leaves pages out, so Studio, mentions and the jobs dialog show the comic alone; a page still opens from the step that drew it. Deleting a comic deletes the pages and their versions that no other comic binds and no document places. --- .../agent/skills/surfsense-pictures/SKILL.md | 2 +- .../modules/agent/tool_endpoint/make_image.py | 11 ++ .../modules/artifacts/comic/bound_comic.py | 2 + .../backend/modules/artifacts/comic/pages.py | 24 ++++ .../modules/artifacts/comic/unused_pages.py | 76 ++++++++++ .../modules/artifacts/drawn_images/service.py | 3 + .../modules/artifacts/image_edits/service.py | 3 + .../backend/modules/artifacts/router.py | 24 +++- .../integration/agent/test_bind_comic.py | 132 ++++++++++++++++++ .../tests/unit/agent/test_pictures_skill.py | 4 + 10 files changed, 274 insertions(+), 7 deletions(-) create mode 100644 surfsense_local/backend/modules/artifacts/comic/pages.py create mode 100644 surfsense_local/backend/modules/artifacts/comic/unused_pages.py diff --git a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md index 06a22056f8..d3c5f21ee1 100644 --- a/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md +++ b/surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md @@ -30,7 +30,7 @@ A comic, a storyboard, a picture book, or any pictures that must look like one s 2. **Confirm once.** Say how many pictures, the plan in short, and that each is billed by their provider, or takes minutes on their computer. On yes, draw every picture without asking again. A single picture needs no confirmation. 3. **Draw the first alone,** and look at it. When it is wrong, change it before going on: every later picture copies its characters. 4. **Draw the rest with the first as their reference**: `references: ["image-<first>"]`. Give each its own call, all in one step, so a remote model draws them side by side. A character picture the user gave, or a character sheet they asked for first, is the reference instead. -5. **Title each** "<title> · page <n>", or "· scene <n>": a chat cannot reuse a picture's title. +5. **Title each** "<title> · page <n>", or "· scene <n>": a chat cannot reuse a picture's title. For a comic, a storyboard or a picture book, give every page `comic_page: true`, from page 1 on: Studio lists the comic, never its pages. 6. **Bind them** with `surfsense_bind_comic`, every picture's name in reading order, with the `layout` and `direction` [`comics.md`](comics.md#which-kind) gives its kind. The comic is what the user keeps: one artifact in Studio, a PDF. 7. **Answer** with the comic and one line per picture, so the user can ask for "page 3 again". diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py index f3cba41a48..39c576d90b 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py @@ -124,6 +124,14 @@ "Left out, the model's own." ), }, + "comic_page": { + "type": "boolean", + "description": ( + "true for a page of a comic, storyboard or picture book you " + "will bind with surfsense_bind_comic: Studio lists the comic, " + "not its pages." + ), + }, "references": { "type": "array", "items": {"type": "string"}, @@ -148,6 +156,7 @@ class _Request: base_artifact_id: int | None shape: Shape | None = None references: tuple[str, ...] = () + comic_page: bool = False def make( @@ -244,6 +253,7 @@ def _request(arguments: dict[str, Any]) -> _Request: base, cast(Shape, shape) if shape is not None else None, tuple(dict.fromkeys(name.strip() for name in references)), + arguments.get("comic_page") is True, ) @@ -289,6 +299,7 @@ def _new_picture(session: Session, scope: TurnScope, request: _Request) -> Artif chat_thread_id=scope.thread_id, shape=request.shape, references=_reference_ids(session, scope, request.references), + comic_page=request.comic_page, ) diff --git a/surfsense_local/backend/modules/artifacts/comic/bound_comic.py b/surfsense_local/backend/modules/artifacts/comic/bound_comic.py index 7f05a16647..0d30400239 100644 --- a/surfsense_local/backend/modules/artifacts/comic/bound_comic.py +++ b/surfsense_local/backend/modules/artifacts/comic/bound_comic.py @@ -13,6 +13,7 @@ comic_bound, ) from modules.artifacts.comic.kinds import MAX_PAGES +from modules.artifacts.comic.pages import mark_comic_pages from modules.artifacts.comic.reading import Direction, Layout from modules.artifacts.drawn_images.names import ( ImageStillDrawingError, @@ -117,6 +118,7 @@ def create_bound_comic( layout=layout, direction=direction, ) + mark_comic_pages(session, [page.picture for page in binding.pages]) artifact.artifact_metadata = { RECIPE_KEY: comic_bound(binding), "version": version.as_metadata(), diff --git a/surfsense_local/backend/modules/artifacts/comic/pages.py b/surfsense_local/backend/modules/artifacts/comic/pages.py new file mode 100644 index 0000000000..d5ddd4e2ad --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/pages.py @@ -0,0 +1,24 @@ +"""A picture drawn as a comic's page: kept for the agent, never listed apart from its comic.""" + +from typing import Any + +from sqlalchemy.orm import Session + +from modules.artifacts.models import Artifact + +COMIC_PAGE = "comic_page" + + +def is_comic_page(metadata: dict[str, Any] | None) -> bool: + return bool((metadata or {}).get(COMIC_PAGE)) + + +def mark_comic_pages(session: Session, picture_ids: list[int]) -> None: + """Marked again when bound, for a page the agent drew without the mark.""" + for picture_id in picture_ids: + picture = session.get(Artifact, picture_id) + if picture is not None and not is_comic_page(picture.artifact_metadata): + picture.artifact_metadata = { + **(picture.artifact_metadata or {}), + COMIC_PAGE: True, + } diff --git a/surfsense_local/backend/modules/artifacts/comic/unused_pages.py b/surfsense_local/backend/modules/artifacts/comic/unused_pages.py new file mode 100644 index 0000000000..483960cf8d --- /dev/null +++ b/surfsense_local/backend/modules/artifacts/comic/unused_pages.py @@ -0,0 +1,76 @@ +"""The pages that go with a deleted comic: those no other comic binds and no document places. + +A page is never listed, so nothing else would remove it. Every version of a page +goes with it, newest first, since an edit names the version it was made from. +""" + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from modules.artifacts.comic.binding_recipe import comic_binding_of +from modules.artifacts.comic.pages import is_comic_page +from modules.artifacts.models import Artifact +from modules.artifacts.script_documents.spec import DOCUMENT_FORMATS, document_script +from modules.artifacts.script_documents.version import version_of + + +def unused_pages(session: Session, comic: Artifact) -> list[Artifact]: + binding = comic_binding_of(comic.artifact_metadata) + if binding is None: + return [] + still_used = _used_elsewhere(session, comic) + gone: dict[int, Artifact] = {} + for page in binding.pages: + picture = session.get(Artifact, page.picture) + if picture is None or not is_comic_page(picture.artifact_metadata): + continue + versions = _versions(session, picture) + if any(version.id in still_used for version in versions): + continue + gone.update({version.id: version for version in versions}) + return sorted(gone.values(), key=lambda artifact: artifact.id, reverse=True) + + +def _used_elsewhere(session: Session, comic: Artifact) -> set[int]: + """Pictures another comic binds, or a document of the workspace places.""" + others = session.scalars( + select(Artifact).where( + Artifact.workspace_id == comic.workspace_id, + Artifact.id != comic.id, + Artifact.format.in_(("comic", *DOCUMENT_FORMATS)), + ) + ) + used: set[int] = set() + for other in others: + binding = comic_binding_of(other.artifact_metadata) + if binding is not None: + used.update(page.picture for page in binding.pages) + script = document_script(other.artifact_metadata) + if script is not None: + used.update( + int(name.removeprefix("image-")) + for name in script.images + if name.removeprefix("image-").isdigit() + ) + return used + + +def _versions(session: Session, picture: Artifact) -> list[Artifact]: + """Every comic page in the picture's line of versions, or the picture alone.""" + version = version_of(picture.artifact_metadata) + if version is None: + return [picture] + siblings = session.scalars( + select(Artifact).where( + Artifact.workspace_id == picture.workspace_id, + Artifact.chat_thread_id == picture.chat_thread_id, + Artifact.format == picture.format, + ) + ) + return [ + sibling + for sibling in siblings + if is_comic_page(sibling.artifact_metadata) + and (found := version_of(sibling.artifact_metadata)) is not None + and found.root == version.root + ] or [picture] diff --git a/surfsense_local/backend/modules/artifacts/drawn_images/service.py b/surfsense_local/backend/modules/artifacts/drawn_images/service.py index 1f6db1a469..f8f83b3dda 100644 --- a/surfsense_local/backend/modules/artifacts/drawn_images/service.py +++ b/surfsense_local/backend/modules/artifacts/drawn_images/service.py @@ -4,6 +4,7 @@ from sqlalchemy.orm import Session +from modules.artifacts.comic.pages import COMIC_PAGE from modules.artifacts.drawn_images.model import require_image_model from modules.artifacts.drawn_images.recipe import image_drawn from modules.artifacts.models import Artifact @@ -26,6 +27,7 @@ def create_drawn_image( chat_thread_id: int | None, shape: Shape | None = None, references: tuple[int, ...] = (), + comic_page: bool = False, ) -> Artifact: """Pending until Studio's job paints it. Commits before enqueueing: the worker is another process and must find the row.""" @@ -49,6 +51,7 @@ def create_drawn_image( RECIPE_KEY: image_drawn(prompt, shape, references), "source_document_ids": [], "prompt": prompt, + **({COMIC_PAGE: True} if comic_page else {}), }, ) session.add(artifact) diff --git a/surfsense_local/backend/modules/artifacts/image_edits/service.py b/surfsense_local/backend/modules/artifacts/image_edits/service.py index 1563b83853..95a8e047fd 100644 --- a/surfsense_local/backend/modules/artifacts/image_edits/service.py +++ b/surfsense_local/backend/modules/artifacts/image_edits/service.py @@ -5,6 +5,7 @@ from fastapi import HTTPException, status from sqlalchemy.orm import Session +from modules.artifacts.comic.pages import COMIC_PAGE, is_comic_page from modules.artifacts.image_edits.editable import EDITABLE_FORMATS, has_picture from modules.artifacts.image_edits.recipe import image_edited from modules.artifacts.models import Artifact @@ -99,6 +100,8 @@ def create_image_edit_version( "version": version.as_metadata(), "source_document_ids": list(meta.get("source_document_ids") or []), "prompt": meta.get("prompt"), + # A comic's page changed is its page still. + **({COMIC_PAGE: True} if is_comic_page(meta) else {}), } queue_job(session, artifact) logger.info( diff --git a/surfsense_local/backend/modules/artifacts/router.py b/surfsense_local/backend/modules/artifacts/router.py index c4490692e8..6a047d7a0c 100644 --- a/surfsense_local/backend/modules/artifacts/router.py +++ b/surfsense_local/backend/modules/artifacts/router.py @@ -8,6 +8,8 @@ from sqlalchemy.orm import Session from api.dependencies import SessionDep, transact +from modules.artifacts.comic.pages import is_comic_page +from modules.artifacts.comic.unused_pages import unused_pages from modules.artifacts.dependencies import ArtifactDep, get_artifact from modules.artifacts.drawn_images.placed_in import refuse_deleting_a_placed_picture from modules.artifacts.flashcard_progress import ( @@ -106,7 +108,12 @@ def list_artifacts( ) .order_by(Artifact.created_at.desc()) ).all() - return [ArtifactRead.of(artifact) for artifact in artifacts] + # A comic's pages are its own: the comic is listed, they are not. + return [ + ArtifactRead.of(artifact) + for artifact in artifacts + if not is_comic_page(artifact.artifact_metadata) + ] @router.get( @@ -398,11 +405,16 @@ def delete_artifact(artifact: ArtifactDep, session: SessionDep) -> Response: refuse_deleting_a_placed_picture(session, artifact) # The ARTIFACT document is the root: deleting it cascades the sidecar, its # files and its chunks. Only the blobs live beyond the database. - directory = get_storage_settings().artifact_dir(artifact.workspace_id, artifact.id) - document = artifact.document - - session.delete(document) + storage = get_storage_settings() + # A comic's pages go with it, once nothing else uses them: nothing lists them. + gone = [artifact, *unused_pages(session, artifact)] + directories = [storage.artifact_dir(a.workspace_id, a.id) for a in gone] + + for each in gone: + session.delete(each.document) + session.flush() session.commit() - shutil.rmtree(directory, ignore_errors=True) + for directory in directories: + shutil.rmtree(directory, ignore_errors=True) return Response(status_code=status.HTTP_204_NO_CONTENT) diff --git a/surfsense_local/backend/tests/integration/agent/test_bind_comic.py b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py index eb3ca6928e..770ef101c3 100644 --- a/surfsense_local/backend/tests/integration/agent/test_bind_comic.py +++ b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py @@ -203,3 +203,135 @@ async def test_binding_a_comic_does_not_start_the_local_image_model( assert local_image_demand(session, datetime.now(UTC)) is None + +async def _listed_ids(tools: ToolEndpoint, workspace_id: int) -> set[int]: + listed = await tools.client.get(f"/workspaces/{workspace_id}/artifacts") + return {row["id"] for row in listed.json()} + + +async def _page(tools: ToolEndpoint, workspace_id: int, title: str) -> str: + text, is_error = await tools.call( + workspace_id, + "make_image", + {"prompt": f"A page: {title}", "title": title, "comic_page": True}, + ) + assert is_error is False, text + return f"image-{_artifact_id(text)}" + + +def _id(name: str) -> int: + return int(name.removeprefix("image-")) + + +async def test_only_the_comic_is_listed_never_its_pages( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 +) -> None: + """A comic is one artifact in Studio, from its first page drawn to its binding.""" + workspace_id = await tools.workspace() + first = await _page(tools, workspace_id, "Cat · page 1") + assert _id(first) not in await _listed_ids(tools, workspace_id) + second = await _page(tools, workspace_id, "Cat · page 2") + + made, _ = await tools.call( + workspace_id, + "bind_comic", + {"title": "Cat", "pages": [{"image": first}, {"image": second}]}, + ) + + assert await _listed_ids(tools, workspace_id) == {_artifact_id(made)} + + +async def test_a_page_the_agent_changes_stays_out_of_the_list( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 +) -> None: + """Page 3 again is the page's next version, and a page all the same.""" + workspace_id = await tools.workspace() + page = await _page(tools, workspace_id, "Cat · page 1") + + changed, is_error = await tools.call( + workspace_id, + "make_image", + {"prompt": "Make it night", "artifact_id": _id(page)}, + ) + + assert is_error is False, changed + assert await _listed_ids(tools, workspace_id) == set() + + +async def test_binding_hides_a_page_drawn_without_the_mark( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 +) -> None: + """A page the agent forgot to mark leaves the list once its comic binds it.""" + workspace_id = await tools.workspace() + page = await _picture(tools, workspace_id, "Cat · page 1") + + made, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": page}]} + ) + + assert await _listed_ids(tools, workspace_id) == {_artifact_id(made)} + + +async def test_deleting_a_comic_deletes_the_pages_no_other_version_binds( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """Pages cannot be deleted from the list, so they go with the comic, last of all.""" + workspace_id = await tools.workspace() + first = await _page(tools, workspace_id, "Cat · page 1") + v1, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": first}]} + ) + second = await _page(tools, workspace_id, "Cat · page 2") + v2, _ = await tools.call( + workspace_id, + "bind_comic", + { + "title": "Cat", + "pages": [{"image": first}, {"image": second}], + "artifact_id": _artifact_id(v1), + }, + ) + + deleted = await tools.client.delete(f"/artifacts/{_artifact_id(v2)}") + assert deleted.status_code == 204, deleted.text + with create_session_factory(engine)() as session: + assert session.get(Artifact, _id(second)) is None + assert session.get(Artifact, _id(first)) is not None + + deleted = await tools.client.delete(f"/artifacts/{_artifact_id(v1)}") + assert deleted.status_code == 204, deleted.text + with create_session_factory(engine)() as session: + assert session.get(Artifact, _id(first)) is None + + +async def test_deleting_a_comic_takes_a_pages_earlier_versions_too( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """A changed page's first drawing is still kept: nothing else would remove it.""" + workspace_id = await tools.workspace() + page = await _page(tools, workspace_id, "Cat · page 1") + changed, _ = await tools.call( + workspace_id, + "make_image", + {"prompt": "Make it night", "artifact_id": _id(page)}, + ) + made, _ = await tools.call( + workspace_id, + "bind_comic", + {"title": "Cat", "pages": [{"image": f"image-{_artifact_id(changed)}"}]}, + ) + + deleted = await tools.client.delete(f"/artifacts/{_artifact_id(made)}") + + assert deleted.status_code == 204, deleted.text + with create_session_factory(engine)() as session: + assert session.get(Artifact, _id(page)) is None + assert session.get(Artifact, _artifact_id(changed)) is None + diff --git a/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py index fef4079f43..ca3ccc0982 100644 --- a/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py +++ b/surfsense_local/backend/tests/unit/agent/test_pictures_skill.py @@ -80,3 +80,7 @@ def test_a_comic_is_bound_by_its_own_tool_not_a_script() -> None: assert "surfsense_bind_comic" in text, name assert "surfsense_render_document" not in text, name + +def test_a_comics_pages_are_drawn_as_its_pages() -> None: + """Marked from the first, a page is never listed in Studio apart from its comic.""" + assert "comic_page: true" in _file("SKILL.md") From 53a9ef32e7984943adddad46d90d11f4c1732e36 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:21:36 +0530 Subject: [PATCH 19/33] feat(local): search finds a comic as one document, by what its pages were drawn from A comic's page is not indexed, and binding takes a page drawn without the mark out of the index. The comic's text is its title, each page's drawing prompt with a line every page repeats kept once, and the lines under its pages, since its bubbles are drawn and its PDF holds no text for them. --- .../backend/modules/artifacts/comic/pages.py | 9 +- .../integration/agent/test_bind_comic.py | 87 ++++++++++++++++++- surfsense_local/backend/worker/studio/job.py | 16 ++-- .../media/visual/comic_binding/bound_pages.py | 38 ++++++-- .../backend/worker/studio/shared/persist.py | 5 +- 5 files changed, 137 insertions(+), 18 deletions(-) diff --git a/surfsense_local/backend/modules/artifacts/comic/pages.py b/surfsense_local/backend/modules/artifacts/comic/pages.py index d5ddd4e2ad..d18c881ac6 100644 --- a/surfsense_local/backend/modules/artifacts/comic/pages.py +++ b/surfsense_local/backend/modules/artifacts/comic/pages.py @@ -2,9 +2,11 @@ from typing import Any +from sqlalchemy import delete from sqlalchemy.orm import Session from modules.artifacts.models import Artifact +from modules.chunks.models import Chunk COMIC_PAGE = "comic_page" @@ -14,7 +16,8 @@ def is_comic_page(metadata: dict[str, Any] | None) -> bool: def mark_comic_pages(session: Session, picture_ids: list[int]) -> None: - """Marked again when bound, for a page the agent drew without the mark.""" + """Marked again when bound, for a page the agent drew without the mark, and + taken out of search: a comic is found by its own text, as a PDF is.""" for picture_id in picture_ids: picture = session.get(Artifact, picture_id) if picture is not None and not is_comic_page(picture.artifact_metadata): @@ -22,3 +25,7 @@ def mark_comic_pages(session: Session, picture_ids: list[int]) -> None: **(picture.artifact_metadata or {}), COMIC_PAGE: True, } + # The delete trigger clears both search indexes for these rows. + session.execute( + delete(Chunk).where(Chunk.document_id == picture.document_id) + ) diff --git a/surfsense_local/backend/tests/integration/agent/test_bind_comic.py b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py index 770ef101c3..44c4baeadd 100644 --- a/surfsense_local/backend/tests/integration/agent/test_bind_comic.py +++ b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py @@ -8,13 +8,14 @@ import pytest from pypdf import PdfReader -from sqlalchemy import Engine +from sqlalchemy import Engine, select from modules.artifacts.local_image_demand import local_image_demand from modules.artifacts.models import Artifact, ArtifactFileRole from modules.documents.models import DocumentStatus from shared.config import get_storage_settings from shared.db import create_session_factory +from shared.search import retrieve from tests.integration.agent.test_make_image import ( # noqa: F401 - fixtures Painter, _artifact_id, @@ -335,3 +336,87 @@ async def test_deleting_a_comic_takes_a_pages_earlier_versions_too( assert session.get(Artifact, _id(page)) is None assert session.get(Artifact, _artifact_id(changed)) is None + +def _found(engine: Engine, workspace_id: int, query: str) -> set[int]: + """The artifacts a workspace search finds, as a chat's sources would.""" + with create_session_factory(engine)() as session: + hits = retrieve(session, workspace_id, query, top_k=20) + return { + artifact.id + for hit in hits + if ( + artifact := session.scalar( + select(Artifact).where(Artifact.document_id == hit.document_id) + ) + ) + is not None + } + + +async def test_search_finds_the_comic_never_its_pages( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """A comic is found as a PDF is, by what it says, not page by page by its prompts.""" + workspace_id = await tools.workspace() + marked = await _page(tools, workspace_id, "Lighthouse storm page") + unmarked = await _picture(tools, workspace_id, "Lighthouse keeper page") + + made, _ = await tools.call( + workspace_id, + "bind_comic", + { + "title": "Lighthouse", + "pages": [ + {"image": marked}, + {"image": unmarked, "lines": ["NORA: The lighthouse is dark!"]}, + ], + }, + ) + + assert _found(engine, workspace_id, "lighthouse") == {_artifact_id(made)} + + +async def test_a_comic_is_found_by_what_its_pages_were_drawn_from( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """Its story and bubbles are in its pages' prompts, not in any text the PDF holds.""" + workspace_id = await tools.workspace() + style = "Bold ink lines, flat colours.\nCharacters: NORA, seventy, yellow raincoat." + names = [] + for number, panel in enumerate( + ( + 'NORA taps the dead lamp. NORA says: "THE BULB IS BROKEN!"', + "MILO finds spare bulbs on a high shelf.", + ), + start=1, + ): + text, _ = await tools.call( + workspace_id, + "make_image", + { + "prompt": f"{style}\nPage {number} of 2.\nPanel 1: {panel}", + "title": f"Cat · page {number}", + "comic_page": True, + }, + ) + names.append(f"image-{_artifact_id(text)}") + + made, _ = await tools.call( + workspace_id, + "bind_comic", + {"title": "The Lighthouse Cat", "pages": [{"image": n} for n in names]}, + ) + + comic = _artifact_id(made) + assert _found(engine, workspace_id, "broken bulb") == {comic} + assert _found(engine, workspace_id, "shelf") == {comic} + with create_session_factory(engine)() as session: + body = session.get(Artifact, comic).document.content or "" + assert 'NORA says: "THE BULB IS BROKEN!"' in body + assert "MILO finds spare bulbs on a high shelf." in body + # Said once, though every page's prompt repeats it. + assert body.count("yellow raincoat") == 1 diff --git a/surfsense_local/backend/worker/studio/job.py b/surfsense_local/backend/worker/studio/job.py index 9be2646d0b..2c5d1dd96d 100644 --- a/surfsense_local/backend/worker/studio/job.py +++ b/surfsense_local/backend/worker/studio/job.py @@ -1,7 +1,6 @@ import asyncio import logging import time -from pathlib import Path from typing import cast import httpx @@ -68,6 +67,7 @@ from worker.studio.media.visual.edit import pipeline as image_edit from worker.studio.media.visual.edit.base import base_image from worker.studio.media.visual.redraw import pipeline as image_redraw +from worker.studio.media.visual.redraw.drawing_prompt import drawing_prompt_of from worker.studio.office.document import figure_shelf from worker.studio.office.document.refine import refine from worker.studio.office.docx import docx @@ -433,14 +433,15 @@ def _draw_image( def _bind_comic(session: Session, document: Document, binding: ComicBinding) -> Built: """The agent's pages, as drawn when it bound them; no model is asked.""" - pictures = [_picture_file(session, page.picture) for page in binding.pages] + pages = [_kept_page(session, page.picture) for page in binding.pages] title = document.title session.commit() raise_if_cancelled(session, document) - return bound_pages.render(title, binding, pictures) + return bound_pages.render(title, binding, pages) -def _picture_file(session: Session, picture_id: int) -> Path: +def _kept_page(session: Session, picture_id: int) -> bound_pages.KeptPage: + """Its file to bind, and the words it was drawn from, for the comic's search.""" picture = session.get(Artifact, picture_id) primary = ( None @@ -449,9 +450,12 @@ def _picture_file(session: Session, picture_id: int) -> Path: (f for f in picture.files if f.role is ArtifactFileRole.PRIMARY), None ) ) - if primary is None: + if picture is None or primary is None: raise ValueError(f"image-{picture_id} is gone, so its page cannot be bound") - return get_storage_settings().data_dir / primary.storage_key + return bound_pages.KeptPage( + get_storage_settings().data_dir / primary.storage_key, + drawing_prompt_of(picture), + ) def _keep_spec(artifact: Artifact, spec: dict) -> None: diff --git a/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py index 41a414aa63..bc456708c3 100644 --- a/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py +++ b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py @@ -1,5 +1,6 @@ """The agent's comic: the pictures it drew, read from Studio's store and bound as it asked.""" +from dataclasses import dataclass from pathlib import Path from modules.artifacts.comic.binding_recipe import ComicBinding @@ -8,25 +9,44 @@ from worker.studio.shared.text import slug -def render(title: str, binding: ComicBinding, pictures: list[Path]) -> Built: - """`pictures` are the pages' files, in the binding's order.""" +@dataclass(frozen=True) +class KeptPage: + picture: Path + # The words the page was drawn from: its panels and bubbles, as Studio's plan holds them. + drawn_from: str + + +def render(title: str, binding: ComicBinding, pages: list[KeptPage]) -> Built: + """`pages` are the binding's pages' files and prompts, in its order.""" pdf = bind( title, [ - BoundPage(path.read_bytes(), page.lines) - for path, page in zip(pictures, binding.pages, strict=True) + BoundPage(kept.picture.read_bytes(), page.lines) + for kept, page in zip(pages, binding.pages, strict=True) ], layout=binding.layout, direction=binding.direction, ) - body = [f"# {title}"] - for number, page in enumerate(binding.pages, start=1): - body.append(f"\n## Page {number}\n") - body += [f"- {line}" for line in page.lines] return Built( title=title, - markdown="\n".join(body), + markdown=_searchable(title, binding, pages), primary=pdf, primary_mime="application/pdf", primary_filename=f"{slug(title, 'comic')}.pdf", ) + + +def _searchable(title: str, binding: ComicBinding, pages: list[KeptPage]) -> str: + """Each page's prompt and lines, a line every page repeats (its style, its + cast) kept the first time only, so it does not outweigh what the pages say.""" + seen: set[str] = set() + body = [f"# {title}"] + for number, (kept, page) in enumerate(zip(pages, binding.pages, strict=True), 1): + body.append(f"\n## Page {number}\n") + for line in kept.drawn_from.splitlines(): + line = line.strip() + if line and line not in seen: + seen.add(line) + body.append(line) + body += [f"- {line}" for line in page.lines] + return "\n".join(body) diff --git a/surfsense_local/backend/worker/studio/shared/persist.py b/surfsense_local/backend/worker/studio/shared/persist.py index 5be120ef8f..1e755f1b1f 100644 --- a/surfsense_local/backend/worker/studio/shared/persist.py +++ b/surfsense_local/backend/worker/studio/shared/persist.py @@ -4,6 +4,7 @@ from sqlalchemy.orm import Session +from modules.artifacts.comic.pages import is_comic_page from modules.artifacts.models import Artifact, ArtifactFile, ArtifactFileRole from modules.documents.models import Document from modules.embedding.active import ActiveIndex, require_active_index @@ -38,7 +39,9 @@ def persist( source; the sidecar's files hold only the deliverable bytes. """ logger.info("studio: persist artifact %s indexing", artifact.id) - index, passages, vectors = _embed(session, built.markdown) + # A comic's page is found through its comic, as a PDF's pages are. + searchable = "" if is_comic_page(artifact.artifact_metadata) else built.markdown + index, passages, vectors = _embed(session, searchable) document.title = built.title document.content = built.markdown indexing.replace_chunks(session, document, index, passages, vectors) From 8145fd31d76198cbbd246cd471bff2f09abbdc81 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 21:21:37 +0530 Subject: [PATCH 20/33] docs: comics as their own artifact, and the infographic as an image type The comic artifact proposal, Studio's comic format and the agent's binding in the architecture docs, and the Comic format, its kinds and Image's Type in the user docs. --- docs/architecture/agent.md | 5 ++-- docs/architecture/studio.md | 16 ++++++++++-- docs/proposals/pictures/06-comic-artifact.md | 26 ++++++++++++++++++++ docs/proposals/pictures/README.md | 1 + www/content/docs/v2/agent.mdx | 6 ++--- www/content/docs/v2/guides/images.mdx | 4 +-- www/content/docs/v2/studio.mdx | 15 ++++++++--- 7 files changed, 61 insertions(+), 12 deletions(-) create mode 100644 docs/proposals/pictures/06-comic-artifact.md diff --git a/docs/architecture/agent.md b/docs/architecture/agent.md index fb487337fb..dc5d22f882 100644 --- a/docs/architecture/agent.md +++ b/docs/architecture/agent.md @@ -164,7 +164,8 @@ Deleting a thread aborts and deletes its session, disposes its opencode instance | `surfsense_create_artifact` | `format`, one of Studio's formats except `docx`, `pdf`, `pptx`, `xlsx`, `html`, `image` and `infographic`; `source_ids`, the numbers at the end of the source files' names; `instructions`, at most 2,000 characters | Studio's own `create_artifact_job()`, which returns at once with the artifact pending. A source not ticked for the turn is refused. Asked for one of the four Office formats, it refuses and names `surfsense_render_document`, because Studio's draft keeps no script to edit; asked for `html`, it names `surfsense_save_page` and the HTML skill | | `surfsense_render_document` | `title`; `format`, `docx`, `pdf`, `pptx` or `xlsx`; `script`; `artifact_id`, to make the next version of, refused for any version of a document Studio made, which Refine rewrites in Studio, and for a revised copy or a converted PDF, which keep no script and have no Refine, naming `surfsense_revise_document` or the Word version or deck to change and convert again ([`unscripted_base.py`](../../surfsense_local/backend/modules/artifacts/script_documents/unscripted_base.py)); `images`, names from `surfsense_list_images`, of ticked sources only; `template_source_id`, a ticked source's own `.docx` for `docx` or `.pptx` for `pptx`, kept by later versions | runs the script as a new version in Studio and waits for it ([below](#documents-the-agent-makes)) | | `surfsense_save_page` | `path`, the page's file under the thread's `outputs/`; `title`; `artifact_id`, to make a page's next version | keeps the page as a version in Studio and waits for it to be indexed ([web pages the agent makes](#web-pages-the-agent-makes)): the `Rendered artifact <id>, version <n>: <title>` line, the page's size and opening text, the outside resources the viewer will block, up to four screenshots, and the working file to edit next | -| `surfsense_make_image` | `prompt`; `title`, needed for a new picture; `style`, `image` or `infographic`; `artifact_id`, to edit a picture this chat made | a picture kept in Studio and waited for ([pictures the agent makes](#pictures-the-agent-makes)): the `Rendered artifact <id>, version <n>: <title>` line, its name `image-<id>`, and the picture itself when the model reads images; or `Queued artifact …` | +| `surfsense_bind_comic` | `title`; `pages`, each `{image, lines}` in reading order; `layout`, `pages` or `scroll`; `direction`, `ltr` or `rtl`; `artifact_id`, for a comic's next version | a `comic` in Studio, its pages bound into one PDF ([comics the agent binds](studio.md#comics-the-agent-binds)): the `Rendered artifact <id>, version <n>: <title>` line; or `Queued artifact …` while a page is still being drawn | +| `surfsense_make_image` | `prompt`; `title`, needed for a new picture; `type`, `image` or `infographic`; `comic_page`, for a page of a comic, which Studio does not list; `artifact_id`, to edit a picture this chat made | a picture kept in Studio and waited for ([pictures the agent makes](#pictures-the-agent-makes)): the `Rendered artifact <id>, version <n>: <title>` line, its name `image-<id>`, and the picture itself when the model reads images; or `Queued artifact …` | | `surfsense_read_document` | `artifact_id`, of any version, or `document_id`, a workbook source; `sheet`, a workbook's sheet; `offset`, an optional 1-based line, or a workbook's row, to start from | the agent's newest version's number (a lineage it continued from a Studio draft before such renders were refused skips Studio's later versions), artifact id, title, format, status and the images it places, and its script in pages of at most 1,000 lines and 40,000 bytes under "Script, lines a-b of n", ending with the offset of the next page, so a result stays under opencode's truncation limits ([`script_page.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/script_page.py)). A document Studio made is refused, as a render continuing one is: "This document was made in Studio: refine it there, or make a new document." Given the `artifact_id` of a revised copy, it reads the copy's newest ready version as it reads now ([`read_revised_copy.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/read_revised_copy.py)): a Word copy's body with every author's tracked changes marked inline as `<ins author="…">` and `<del author="…">`, `[comment N]` after the text each comment is on, then each comment with its author, whether it is internal, and that text ([`word_marked_text.py`](../../surfsense_local/backend/modules/artifacts/revised_copies/engines/word_marked_text.py)), under its counts of tracked changes and comments; a deck's text slide by slide; a workbook's cells as below. Word and deck text come in pages of lines as a script does. Excel and PowerPoint keep no tracked changes, so their opening counts the edits applied over the copy's versions instead. Every answer ends its opening with the `artifact_id` to revise. Given `document_id`, a ticked `.xlsx` or `.xlsm` source, or the `artifact_id` of a revised copy of one, it returns the workbook's cells instead, read from the file itself (the source's original, or the copy's newest ready version) with openpyxl in read-only mode and formulas as written: each sheet's name and used range, then each non-empty cell as `B7 = 1250.5`, `A1 = "text"` or `C3 = =SUM(C2:C2)`, under the same 40,000 bytes ([`read_workbook.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/read_workbook.py), [`cell_map.py`](../../surfsense_local/backend/modules/artifacts/revised_copies/cell_map.py)). `sheet` reads one sheet and `offset` is then the row to start from; a cut map ends with the row and the call that reads on, or, when one row holds more cells than a map shows, the first cell left out and the next row to read on from. The file is read once the tool's database reads have ended, so a big workbook's seconds of reading never hold SQLite's write lock. Sheet headings take at most a quarter of the map; the sheets past that are only named, up to 4,000 bytes, then counted. The indexed text of a workbook is Docling's tables, without sheet names, addresses or formulas, so the revise tool and the revisions skill send the agent here first. Any other source is refused, pointing at `sources/`. | | `surfsense_list_images` | `source_ids` | each source's figures, one line each: its name, size in pixels, page and caption when known, and its copy under `sources/figures/` to open with `read`; or that the source has none, is not in this workspace, or has its figures being extracted, which queues the figures-only pass ([documents](documents.md#figures)). A source not ticked for the turn is refused | | `surfsense_source_pages` | `document_id`; `pages`, at most four page or slide numbers from 1, the first ones when left out | a PDF, Word or PowerPoint source's own pages as images with the result, in ascending order, with the source's page or slide count; copies at most 1,000 pixels on their long side stay in `sources/pages/<id>-p<n>.png` for a closer look ([below](#source-pages)). Offered only when `opencode.json` declares image input. A source not ticked for the turn, an image, a note or any other kind of file is refused in a sentence saying what to use instead | @@ -223,7 +224,7 @@ The step links the version in the thread as a render's does (`surfsense_save_pag **Shape and references.** `shape` (`square`, `portrait`, `landscape`) reaches the image model ([shapes](studio.md#shapes-and-references)). `references`, up to four picture names of this chat, send those pictures with the prompt to the model that takes pictures, so a comic's later pages keep page 1's characters; with none, the picture is drawn from its prompt and the result says the characters may differ. The tool's listing stays the same whatever models are chosen, so a local model's prompt cache holds. -**Comics and sets.** The `surfsense-pictures` skill ([`skills/surfsense-pictures/`](../../surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md)) is the craft beside the tool, as `surfsense-documents` is beside the render: one picture, or a set such as a comic, a storyboard or a picture book. A set is planned with a character sheet, confirmed once, drawn first page alone and the rest with it as reference, and bound into one PDF by the script in `pdf.md`: one full-bleed picture a page, or, for a webtoon, the pictures stacked down one long page; marked right to left for a manga; placed as JPEG, with a page's exact lines set in a strip under it when its lettering came out wrong or it was drawn without words. The prompt sends pictures and comics there, and the web page skill's description sends them away. +**Comics and sets.** The `surfsense-pictures` skill ([`skills/surfsense-pictures/`](../../surfsense_local/backend/modules/agent/skills/surfsense-pictures/SKILL.md)) is the craft beside the tool, as `surfsense-documents` is beside the render: one picture, or a set such as a comic, a storyboard or a picture book. A set is planned with a character sheet, confirmed once, drawn first page alone and the rest with it as reference, and bound by `surfsense_bind_comic` into one `comic` in Studio, with the binder Studio's comics use: one full-bleed picture a page, or, for a webtoon, the pictures stacked down one long page; marked right to left for a manga; placed as JPEG, with a page's exact lines set in a strip under it when its lettering came out wrong or it was drawn without words. `create_artifact` does not offer `comic`, so the agent never starts a comic whose pages it cannot look at. The prompt sends pictures and comics there, and the web page skill's description sends them away. **Prompt length.** As in Studio, nothing limits a prompt per model: the tool's description asks for one self-contained prompt, as short as it says what to draw, and a model's own refusal comes back as a failed picture. diff --git a/docs/architecture/studio.md b/docs/architecture/studio.md index 9dcb71775f..d34b55d2f9 100644 --- a/docs/architecture/studio.md +++ b/docs/architecture/studio.md @@ -21,7 +21,7 @@ Status lives on the document, not the sidecar, because ADR 0003 keeps indexing s ## Formats -The catalog is a tuple of twelve `Format` rows in [`formats.py`](../../surfsense_local/backend/modules/artifacts/formats.py), computed into availability on each request and never stored. +The catalog is a tuple of thirteen `Format` rows in [`formats.py`](../../surfsense_local/backend/modules/artifacts/formats.py), computed into availability on each request and never stored. | Key | Label | Needs | Worker path | Stored file | |---|---|---|---|---| @@ -37,6 +37,9 @@ The catalog is a tuple of twelve `Format` rows in [`formats.py`](../../surfsense | `podcast` | Podcast | text_gen, audio_gen | `media/audio/podcast/` | MP3 audio, or WAV without a working encoder | | `image` | Image | image_gen, text_gen | `media/visual/image/` | the image | | `infographic` | Infographic | image_gen, text_gen | `media/visual/infographic/` | the image | +| `comic` | Comic | image_gen, text_gen | `media/visual/comic/` | `.pdf`, the bound pages | + +- The panel shows `infographic` as a type of `image`: one Image card, whose dialog chooses Image or Infographic and sends that format's key ([`picture-types.ts`](../../surfsense_local/frontend/src/features/studio/picture-types.ts)). An infographic card shows only when the catalog has no `image`. A format's own options in the dialog, the podcast brief or the picture type, come from [`format-options/`](../../surfsense_local/frontend/src/features/studio/format-options/). - `requires_model_types` is a tuple in the order the pipeline's `render()` takes its models. A format is available when every required model type has a selection; otherwise the reason names every missing one in a fixed reading order: "Needs a chat model", "Needs an image model", "Needs an audio model", or two of them joined, as "Needs a chat model and an audio model". Naming only the first missing type made selecting it look like the gate moving to the other. - `podcast` runs with an `audio_gen` model on this computer or on a server. The format list and the brief read the model's voices without calling it, so neither needs the server's host allowed; the job does. Opening a podcast brief without an audio model answers `409` with "Needs an audio model" and the code `needs_audio`, in the same `{message, code}` detail as a refused create. The app ships Kokoro and chooses it at startup when no audio model is chosen, so that answer needs the choice cleared since the last start, or a build without audio.cpp ([`local-models/selection.md`](local-models/selection.md)). @@ -62,6 +65,7 @@ Eight formats follow it: - **podcast**: the model outlines the episode from the brief, then drafts it segment by segment. Every segment sends the same one-line system prompt and the sources, and its whole draft prompt after them, because llama-server reuses a prompt only up to its first changed token. Measured through the drafting code on Qwen3 1.7B with about 6,200 tokens of sources: with the draft prompt first, every segment re-read the sources; after them, segments 2 to 4 read about 230 tokens each. The draft prompt goes after them whole: split, with what the segments share left ahead of the sources, the same model wrote half the dialogue, in long lines. Each reply is capped at 12 tokens per word of the segment's target, and never under a planned 250-word segment's worth. Uncapped, Qwen3 1.7B looped on a 225-word segment until its 40,960-token window was full, and the JSON retry, which replays the failed reply, could not fit. The target is the outline's own guess: Qwen3 1.7B once gave a segment 20 words, wrote past them, and a 240-token cap ended both replies mid-JSON. The chosen audio model voices every line through audio.cpp's server, and the transcript is the body. - **image**: the model writes a title and an image prompt, and the image model paints the prompt. - **infographic**: the model writes a factual brief (a title, a summary and up to 8 sections of label, value and detail), and the image model paints a prompt built from it in a fixed sketchnote style. +- **comic**: options `{kind, pages}`, one of ten kinds ([`kinds.py`](../../surfsense_local/backend/modules/artifacts/comic/kinds.py)) and 1 to 12 pictures, checked when the job is created. The model plans a title, a style line, a character sheet and each page's panels and lines; the plan is cut to the count asked for, since each picture is billed or takes minutes. Page 1 is drawn alone, the rest with page 1 as their reference by [the model that takes a picture in](#which-model-takes-a-picture), side by side on a remote model and one at a time on this computer; with none, each is drawn from its words and the version keeps `drawn_from_words`. A remote model letters each page's lines in bubbles; a local one draws no words, its lettering being unmeasured, and every page's lines are set in a strip under it, as a picture book's and a storyboard's always are. The pages are bound by [`comic_binding/`](../../surfsense_local/backend/worker/studio/media/visual/comic_binding/), which the agent's comics share; the body is every page's panels and lines. A failed comic is never retried by the worker, since every page would be drawn again. No page is kept on its own, so a Studio comic has no Edit: Regenerate draws it again. Two formats break it. PPTX and XLSX go through `office/`: the tier prompt asks the model to "write one standalone Python script" for the format's library (python-pptx or xlsxwriter), with a `SKILL.md` of authoring guidance beside each format, to the runner's contract: start with a `# title:` comment and save the file at `OUTPUT_PATH`. [`office/pipeline.py`](../../surfsense_local/backend/worker/studio/office/pipeline.py) runs the reply through [the runner](#script-documents), in its own process, never with `exec()` in the worker's. A failing script goes back to the model with its error and the traceback's end, for up to three attempts; one longer than 48,000 characters ran on rather than broke on a line, so it is not shown back but asked for again, shorter, since replayed it overflowed the window ([`script_attempts.py`](../../surfsense_local/backend/worker/studio/office/script_attempts.py)). On a local model each draft stops at 16,384 tokens: a deck's script measured about 3,000, and a 1.7B that looped wrote 130,171 characters for 272 s first. Its failure reason does not start `Script error: `, as for [Word and PDF](#word-and-pdf). The title is the `# title:` comment, else the prompt, the sources or the format; the body is the file's own text, read as a script document's is. No spec is kept: a deck or workbook Studio drafts has no versions and no Refine. @@ -240,7 +244,7 @@ worker/studio/ ├── web/html/ └── media/ ├── audio/podcast/ outline, draft, roster - └── visual/ image, infographic, drawn: a prompt the agent wrote, and edit: any picture changed to an instruction + └── visual/ image, infographic, comic, drawn: a prompt the agent wrote, edit: any picture changed to an instruction, and comic_binding: pages bound into a PDF ``` Every pipeline returns a `Built`: a `title`, the `markdown` that is always the indexed body, and an optional `primary` (and `preview`) with its MIME type and filename. Adding a format is a folder with a `render`, a `case` in `job_router.py` and a row in `formats.py`. Frontend entries below customize its translated label, icon, tooltip and viewer; an unknown key still appears with the server's label, a file icon, the generic tooltip and the document viewer. @@ -356,6 +360,12 @@ References are pictures sent with the prompt, so a new picture keeps their chara It needs only `image_gen`, its Regenerate paints the same prompt at the same shape from the same references, a deleted reference failing it, and Huey never retries it, since each try may be billed. It is v1 of its own document and takes Studio's Edit box like any picture. Every edit keeps its base's chat, from either box, so that chat can still place it. While a script of its chat names a picture, deleting that picture is refused with `placed_in_document`, since the script names it rather than copying it ([`placed_in.py`](../../surfsense_local/backend/modules/artifacts/drawn_images/placed_in.py)). +### Comics the agent binds + +`surfsense_bind_comic` ([agent](agent.md#pictures-the-agent-makes)) creates a `comic` with the recipe `{"kind": "comic_bound", "pages": [{"picture", "lines"}], "layout", "direction"}`, naming the pictures it drew in its chat in reading order ([`comic/bound_comic.py`](../../surfsense_local/backend/modules/artifacts/comic/bound_comic.py)). A page still being drawn makes it wait with `waits_for`, as a document does ([`waiting_renders/`](../../surfsense_local/backend/modules/artifacts/waiting_renders/)). The job binds the pictures' own files with the shared binder and asks no model, so it asks for no local image model either. Called again with the comic's `artifact_id`, it makes the comic's next version; a Studio comic keeps no pages, so it cannot be continued that way. + +Its pages are never listed: a picture drawn with `comic_page`, every later version of it, and any picture a comic binds keep `comic_page: true`, and `GET /workspaces/{id}/artifacts` leaves them out ([`comic/pages.py`](../../surfsense_local/backend/modules/artifacts/comic/pages.py)), so Studio, mentions and the jobs dialog show the comic alone. A page is not indexed either, and binding takes a page drawn without the mark out of the index, so search finds the comic alone, as it finds a PDF. Its text is its title, each page's drawing prompt, a line every page repeats (its style, its cast) kept once, and the lines set under its pages ([`bound_pages.py`](../../surfsense_local/backend/worker/studio/media/visual/comic_binding/bound_pages.py)): its bubbles are drawn, so the PDF holds no text to read them from. A page still opens by its id from the chat step that drew it. Deleting a comic deletes the pages no other comic binds and no document places, every version of each, newest first ([`unused_pages.py`](../../surfsense_local/backend/modules/artifacts/comic/unused_pages.py)); a page drawn and never bound stays, unlisted. + ### Editing a picture `POST /artifacts/{id}/image-edit` with `{instruction}` (1 to 2,000 characters, as Refine's) makes an image's or infographic's next version from the open version's picture ([`image_edits/`](../../surfsense_local/backend/modules/artifacts/image_edits/), [ADR 0077](../adr/0077-an-image-edit-is-the-next-version.md)). @@ -372,6 +382,8 @@ It needs only `image_gen`, its Regenerate paints the same prompt at the same sha ## Known gaps +- A Studio comic keeps only its PDF, so one page cannot be changed: Regenerate draws every page again. +- A page the agent drew and never bound is kept unlisted, until its chat's comic binds it; nothing deletes it. - An image edit is measured only on klein Q4_0 through the pinned sd-server on a processor, with no graphics card: a photo's layout held, and an infographic's printed lettering held through a colour change and a changed title, 657 to 962 seconds an edit, about twice a draw. Lettering klein drew itself, a sketchnote's, is not measured, and neither is any time on a graphics card or a remote edit model. - An edit changes the whole picture by instruction; there is no mask or selected region, which sd.cpp and `/images/edits` both take, since drawing one needs its own interface. - References are measured only on OpenRouter's `/images`, on six models drawing one four-page comic: each held its characters from page 1. OpenAI's `/images/edits` with several pictures, and local references through sd-server, are not measured; nor is a remote server that takes one picture on `/images/edits` and refuses several. diff --git a/docs/proposals/pictures/06-comic-artifact.md b/docs/proposals/pictures/06-comic-artifact.md new file mode 100644 index 0000000000..dfd8b90148 --- /dev/null +++ b/docs/proposals/pictures/06-comic-artifact.md @@ -0,0 +1,26 @@ +# The comic artifact + +> A comic is its own Studio format, `comic`: one artifact whose file is the bound PDF. Studio makes one from sources in its dialog, and the agent binds the pages it drew into one with `surfsense_bind_comic`. Both bind with the same code, so the agent no longer copies a script to fill. Image and infographic stay one Image card in the dialog, the infographic a type of image. + +## Why + +A comic was a PDF the agent rendered from a script in `pdf.md` ([03](03-comics.md)), listed in Studio as a PDF beside its pages, and Studio could not make one. The agent copied a hundred-line ReportLab script for every comic, which a small model can break. + +## Studio + +- **Format** `comic`, needing `image_gen` and `text_gen`, with options `{kind, pages}`: one of the ten kinds in `comics.md`, and 1 to 12 pictures, each kind starting at its own length. Checked when the job is created ([`kinds.py`](../../../surfsense_local/backend/modules/artifacts/comic/kinds.py)). +- **Plan, draw, bind** ([`media/visual/comic/`](../../../surfsense_local/backend/worker/studio/media/visual/comic/)): the chat model plans a title, a style line, a character sheet and each page's panels and lines; page 1 is drawn alone and the rest with page 1 as reference; the binder makes the PDF. +- **Lettering.** Studio cannot look at a page. A remote model letters bubbles; a local model's lettering is not measured, so it draws no words and every line goes in the strip under its page. +- **No retry, no Edit.** A retry would bill every page again. One artifact holds one primary file and keeping pages would need a schema change, so a Studio comic is changed by Regenerate. +- **The dialog** gives each format's options their own slot (`format-options/`): the podcast brief, the picture type, and the comic's kind and pages with what they cost. + +## The agent + +- **`surfsense_bind_comic`** takes the pictures' names in reading order, each with lines for its strip, a `layout` and a `direction`, and `artifact_id` for the comic's next version. A page still being drawn makes it wait, as a document does ([04](04-waiting-render.md)). +- **`create_artifact`** does not offer `comic`: the agent draws its pages and looks at each before it binds them. +- **The skill** drops `pdf.md`; `comics.md` gives each kind's `layout` and `direction`. +- **Pages are not listed.** `make_image` takes `comic_page: true` for every page, binding marks any page drawn without it, and a page's edits keep the mark. Studio lists the comic alone; deleting it deletes the pages nothing else uses. + +## Not in this + +- Changing one page of a Studio comic. diff --git a/docs/proposals/pictures/README.md b/docs/proposals/pictures/README.md index bc51e0ad9d..b52b2540ec 100644 --- a/docs/proposals/pictures/README.md +++ b/docs/proposals/pictures/README.md @@ -52,6 +52,7 @@ Asked for a comic, the agent wrote a web page and drew the panels as SVG, or dre | **Comics** | [`03-comics.md`](03-comics.md) | the `surfsense-pictures` skill, the routing, the PDF | | **The render that waits** | [`04-waiting-render.md`](04-waiting-render.md) | a document that renders once its pictures are drawn | | **In the app** | [`05-in-the-app.md`](05-in-the-app.md) | Settings' label, Studio's Edit, the words users see | +| **The comic artifact** | [`06-comic-artifact.md`](06-comic-artifact.md) | `comic` as a Studio format, `surfsense_bind_comic`, one binder, Image's infographic type | ## The spike diff --git a/www/content/docs/v2/agent.mdx b/www/content/docs/v2/agent.mdx index 0044bf0775..7ca2d9c944 100644 --- a/www/content/docs/v2/agent.mdx +++ b/www/content/docs/v2/agent.mdx @@ -33,12 +33,12 @@ Documents the agent made are changed in its chat, not with Studio's **Refine**. Ask for a picture, an infographic or a document with a picture in it. The agent reads what it needs from your sources, writes the prompt itself and draws with the model in **Settings › Image gen**. To change a picture, ask in the same chat, and it makes its next version. Each picture appears in [Studio](/docs/v2/studio). -Ask for a comic, a manga, a webtoon, a comic strip, a storyboard or a picture book and the agent plans the pages, tells you how many it will draw, and asks once before it starts. It draws one picture per page, keeps the characters from page to page, and puts the pages together in one PDF. With an image model that can't take a picture in, each page is drawn from a description of the characters, so they may look a little different from page to page. +Ask for a comic, a manga, a webtoon, a comic strip, a storyboard or a picture book and the agent plans the pages, tells you how many it will draw, and asks once before it starts. It draws one picture per page, keeps the characters from page to page, and binds the pages into one comic in Studio, a PDF. Studio lists the comic, not its pages; deleting the comic deletes them too. To change a page, ask, and it redraws that page and binds the comic again. With an image model that can't take a picture in, each page is drawn from a description of the characters, so they may look a little different from page to page. - **It draws only when you ask.** A remote model bills each picture. A model on a computer without a graphics card takes several minutes each. -- **A slow picture may finish after the agent replies.** It appears in Studio when it's ready. A document that uses it, such as a comic's PDF, waits for it and appears in Studio once its pictures are drawn. +- **A slow picture may finish after the agent replies.** It appears in Studio when it's ready. A document that uses it, such as a comic, waits for it and appears in Studio once its pictures are drawn. - **Pictures used in a document stay until the document is gone.** If the agent placed a picture in a Word document, PDF, deck or workbook, delete that document before deleting the picture, because SurfSense needs the picture to make the document again. -- **Exact words go in the document.** Image models can garble lettering, so check any words in a picture. When a comic page's speech bubbles come out wrong, the PDF prints that page's lines under it. Data charts stay real charts. +- **Exact words go in the document.** Image models can garble lettering, so check any words in a picture. When a comic page's speech bubbles come out wrong, the comic prints that page's lines under it. Data charts stay real charts. ### Start from your own template diff --git a/www/content/docs/v2/guides/images.mdx b/www/content/docs/v2/guides/images.mdx index 229537cbdd..dd1e6a74cc 100644 --- a/www/content/docs/v2/guides/images.mdx +++ b/www/content/docs/v2/guides/images.mdx @@ -7,7 +7,7 @@ import { Step, Steps } from 'fumadocs-ui/components/steps'; <GuideVideo slug="images" poster="/guides/images/poster.webp" title="Downloading an image model, drawing a picture and editing it" /> -**Image** and **Infographic** in Studio need an image model. A model on your computer works offline and runs only while Studio needs it; with a graphics card a picture takes seconds, without one several minutes. See [Image models](/docs/v2/studio#image-models). +**Image** in Studio needs an image model. A model on your computer works offline and runs only while Studio needs it; with a graphics card a picture takes seconds, without one several minutes. See [Image models](/docs/v2/studio#image-models). <Steps> <Step> @@ -32,7 +32,7 @@ When the download is done, click **Use**. FLUX.2 klein 4B both draws and edits f ### Draw a picture -In Studio, click **Image** or **Infographic**, describe what you want under **Prompt (optional)**, and click **Generate**. Open it when it's ready. +In Studio, click **Image**. Under **Type**, keep **Image**, or choose **Infographic** for a picture that lays out your sources' key points. Describe what you want under **Prompt (optional)**, and click **Generate**. Open it when it's ready. ![The picture Studio drew, open in the right panel](/guides/images/03-picture.webp) diff --git a/www/content/docs/v2/studio.mdx b/www/content/docs/v2/studio.mdx index 7b6fdf07e6..6c29348b00 100644 --- a/www/content/docs/v2/studio.mdx +++ b/www/content/docs/v2/studio.mdx @@ -32,8 +32,8 @@ The result appears in the **Artifacts** list below, and a notice says when it's | **Flashcards** | A deck you go through one card at a time, marking **Got it** or **Needs review**, with **Shuffle** | none | | **Quiz** | Questions with explanations, a score, and **Retake quiz** for all or only the missed questions | none | | **Podcast** | An audio episode with its transcript | `.mp3` | -| **Image** | A picture drawn from your sources | the image | -| **Infographic** | A picture that lays out your sources' key points | the image | +| **Image** | A picture drawn from your sources; choose **Infographic** under **Type** for one that lays out their key points | the image | +| **Comic** | A comic, manga, webtoon, strip, picture book or storyboard drawn from your sources, its pages in one PDF | `.pdf` | Word, Slides and Spreadsheet files are real Office files: open and edit them in Word, PowerPoint, Excel or LibreOffice. Flashcard marks and quiz answers are saved as you go. @@ -42,7 +42,7 @@ Word, Slides and Spreadsheet files are real Office files: open and edit them in Every format needs a chat model. A format that can't run says what it needs, such as "Needs an image model." - **Podcast** also needs an audio model. SurfSense ships one and picks it for you. -- **Image** and **Infographic** also need an image model. See [Image models](#image-models). +- **Image**, of either type, and **Comic** also need an image model. See [Image models](#image-models). Studio uses the chat model selected in Settings at the time you click **Generate**. Strong models make Word documents and PDFs with richer layout; small local models make simpler ones from Markdown. @@ -59,6 +59,15 @@ Click **Podcast** to open its brief before you generate: See [Voice](/docs/v2/voice#podcast-voices) for voices and audio models. +### Comics + +Click **Comic**, then choose: + +- **Kind**: Western comic, graphic novel, bande dessinée, manga, manhua, webtoon, comic strip, yonkoma, picture book or storyboard. A manga reads right to left, and a webtoon scrolls down one long page. +- **Pages**: 1 to 12. Each kind starts at its own length, such as 4 pages, or 8 for a webtoon. + +Studio draws one picture per page, keeping the characters from the first page. Each picture is billed by your image model's provider, or takes minutes on this computer. With a model on this computer, the pages are drawn without words and every line is printed under its page. To change a comic, regenerate it, which draws every page again, or ask the agent, which can change one page. + ## Open, download and delete Click an artifact to open it in the right panel. From c20dcb48608cc2495db64a701e886899201ce757 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 22:00:34 +0530 Subject: [PATCH 21/33] feat(local): Image's types are illustrated cards, not a switch Image and Infographic are cards with their own pictures, the chosen one bordered and ticked, as a radio group still. The pictures are drawn on the theme's tokens and registered as icons, ImageTypeIcon and InfographicTypeIcon. --- .../frontend/src/components/ui/icons.tsx | 3 + .../src/components/ui/picture-type-icons.tsx | 183 ++++++++++++++++++ .../format-options/picture-type-picker.tsx | 93 +++++---- .../src/features/studio/studio-panel.test.tsx | 4 + 4 files changed, 248 insertions(+), 35 deletions(-) create mode 100644 surfsense_local/frontend/src/components/ui/picture-type-icons.tsx diff --git a/surfsense_local/frontend/src/components/ui/icons.tsx b/surfsense_local/frontend/src/components/ui/icons.tsx index a32fa77152..10ecf59c21 100644 --- a/surfsense_local/frontend/src/components/ui/icons.tsx +++ b/surfsense_local/frontend/src/components/ui/icons.tsx @@ -258,3 +258,6 @@ export const Xls01Icon = createIcon(Xls01IconData) export const YoutubeIcon = createIcon(YoutubeIconData) export const ZoomInIcon = createIcon(ZoomInIconData) export const ZoomOutIcon = createIcon(ZoomOutIconData) + +// Studio's picture types, drawn for their cards rather than taken from the icon set. +export { ImageTypeIcon, InfographicTypeIcon } from "./picture-type-icons" diff --git a/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx b/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx new file mode 100644 index 0000000000..c9ea6d2b35 --- /dev/null +++ b/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx @@ -0,0 +1,183 @@ +import type { SVGProps } from "react" + +import { cn } from "@/lib/utils" + +// Studio's picture types as small pictures for their cards, drawn on the +// theme's tokens so they follow light and dark. 3:2, the card's thumbnail. +const VIEW_BOX = "0 0 96 64" + +/** A framed picture: hills under a sun. */ +export function ImageTypeIcon({ + className, + ...props +}: SVGProps<SVGSVGElement>) { + return ( + <svg + viewBox={VIEW_BOX} + fill="none" + aria-hidden="true" + data-slot="picture-type-art" + className={cn("size-full", className)} + {...props} + > + <rect + x="14" + y="8" + width="68" + height="48" + rx="5" + className="fill-card stroke-foreground" + strokeWidth="1.6" + /> + <circle + cx="64" + cy="21" + r="5.5" + className="fill-secondary stroke-foreground" + strokeWidth="1.4" + /> + <path + d="M15 45 L36 26 L52 41 L60 34 L81 51 V51.5 A3.5 3.5 0 0 1 77.5 55 H18.5 A3.5 3.5 0 0 1 15 51.5 Z" + className="fill-muted stroke-foreground" + strokeWidth="1.4" + strokeLinejoin="round" + /> + <path + d="M15 52 L30 41 L44 50 L50 46 L64 55" + className="stroke-muted-foreground" + strokeWidth="1.2" + strokeLinecap="round" + strokeLinejoin="round" + /> + <path + d="M31 33 l3 -2.5 M39 30 l2.5 2" + className="stroke-muted-foreground" + strokeWidth="1.1" + strokeLinecap="round" + /> + </svg> + ) +} + +/** A sketchnote: a heading, two notes joined by an arrow, and a small chart. */ +export function InfographicTypeIcon({ + className, + ...props +}: SVGProps<SVGSVGElement>) { + return ( + <svg + viewBox={VIEW_BOX} + fill="none" + aria-hidden="true" + data-slot="picture-type-art" + className={cn("size-full", className)} + {...props} + > + <rect + x="10" + y="6" + width="76" + height="52" + rx="5" + className="fill-card stroke-foreground" + strokeWidth="1.6" + /> + <rect + x="18" + y="12" + width="34" + height="5" + rx="2.5" + className="fill-foreground" + /> + <rect + x="56" + y="13" + width="14" + height="3" + rx="1.5" + className="fill-muted-foreground" + /> + <rect + x="17" + y="23" + width="26" + height="14" + rx="3" + className="fill-muted stroke-foreground" + strokeWidth="1.2" + /> + <path + d="M21 28 H38 M21 32 H33" + className="stroke-muted-foreground" + strokeWidth="1.2" + strokeLinecap="round" + /> + <path + d="M30 37 C30 43 34 45 40 45" + className="stroke-foreground" + strokeWidth="1.2" + strokeLinecap="round" + /> + <path + d="M37.5 42.5 L40.5 45 L37.5 47.5" + className="stroke-foreground" + strokeWidth="1.2" + strokeLinecap="round" + strokeLinejoin="round" + /> + <rect + x="43" + y="40" + width="20" + height="11" + rx="3" + className="fill-secondary stroke-foreground" + strokeWidth="1.2" + /> + <path + d="M47 45.5 H59" + className="stroke-muted-foreground" + strokeWidth="1.2" + strokeLinecap="round" + /> + <path + d="M66 50 H80" + className="stroke-foreground" + strokeWidth="1.2" + strokeLinecap="round" + /> + <rect + x="68" + y="39" + width="3.5" + height="11" + rx="1" + className="fill-chart-3" + /> + <rect + x="73" + y="31" + width="3.5" + height="19" + rx="1" + className="fill-chart-1" + /> + <rect + x="78" + y="35" + width="3.5" + height="15" + rx="1" + className="fill-chart-4" + /> + <circle + cx="74" + cy="24.5" + r="3" + className="fill-card stroke-foreground" + strokeWidth="1.1" + /> + </svg> + ) +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx index b522787b09..b44b048566 100644 --- a/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx +++ b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx @@ -1,12 +1,23 @@ -import { SegmentedControl } from "@/components/ui/segmented-control" +import type { ComponentType } from "react" + +import { + CheckIcon, + ImageTypeIcon, + InfographicTypeIcon, +} from "@/components/ui/icons" import { intl } from "@/i18n/intl" import { cn } from "@/lib/utils" import type { StudioFormat } from "../api" import { formatLabel } from "../studio-formats" -/** Image or infographic, as one choice. Both need the same models, so a type is - * closed only if the server ever says so. */ +const TYPE_ART: Record<string, ComponentType<{ className?: string }>> = { + image: ImageTypeIcon, + infographic: InfographicTypeIcon, +} + +/** Image or infographic as cards, each shown by its picture, the chosen one ticked. + * Both need the same models, so a type is closed only if the server ever says so. */ export function PictureTypePicker({ types, value, @@ -23,41 +34,53 @@ export function PictureTypePicker({ return ( <div className="space-y-2"> <p className="text-xs font-medium text-muted-foreground">{label}</p> - <SegmentedControl - count={types.length} - selectedIndex={Math.max( - 0, - types.findIndex((entry) => entry.key === value) - )} + <div role="radiogroup" aria-label={label} - className="flex w-full" + className="grid grid-cols-2 gap-2" > - {types.map((entry) => ( - <label - key={entry.key} - className={cn( - "relative min-w-0 flex-1", - entry.available - ? "cursor-pointer" - : "cursor-not-allowed opacity-50" - )} - > - <input - type="radio" - name="studio-picture-type" - value={entry.key} - checked={value === entry.key} - disabled={!entry.available} - className="peer sr-only" - onChange={() => onChange(entry.key)} - /> - <span className="relative flex h-8 items-center justify-center truncate rounded-md px-3 text-sm peer-focus-visible:z-10 peer-focus-visible:ring-2 peer-focus-visible:ring-ring peer-focus-visible:outline-none"> - {formatLabel(entry)} - </span> - </label> - ))} - </SegmentedControl> + {types.map((entry) => { + const Art = TYPE_ART[entry.key] + const checked = value === entry.key + return ( + <label + key={entry.key} + className={cn( + "group relative flex min-w-0 flex-col gap-1.5", + entry.available + ? "cursor-pointer" + : "cursor-not-allowed opacity-50" + )} + > + <input + type="radio" + name="studio-picture-type" + value={entry.key} + checked={checked} + disabled={!entry.available} + className="peer sr-only" + onChange={() => onChange(entry.key)} + /> + <span + className={cn( + "relative flex aspect-3/2 items-center justify-center overflow-hidden rounded-lg border bg-muted/40 px-5 py-3 transition-colors duration-150 ease-out peer-focus-visible:ring-2 peer-focus-visible:ring-ring motion-reduce:transition-none", + checked + ? "border-primary bg-primary/5" + : "border-border group-hover:bg-accent" + )} + > + {Art ? <Art /> : null} + {checked ? ( + <span className="absolute top-1.5 right-1.5 flex size-5 items-center justify-center rounded-full bg-primary text-primary-foreground"> + <CheckIcon className="size-3" strokeWidth={2.5} /> + </span> + ) : null} + </span> + <span className="truncate text-sm">{formatLabel(entry)}</span> + </label> + ) + })} + </div> </div> ) } diff --git a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx index a12abef7fc..aa136673a9 100644 --- a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx +++ b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx @@ -332,6 +332,10 @@ describe("studio panel", () => { await user.click(within(formats).getByRole("button", { name: "Image" })) const dialog = screen.getByRole("dialog", { name: "Image" }) const type = within(dialog).getByRole("radiogroup", { name: "Type" }) + // A card for each type, shown by its own picture. + expect(type.querySelectorAll("[data-slot=picture-type-art]")).toHaveLength( + 2 + ) expect( (within(type).getByRole("radio", { name: "Image" }) as HTMLInputElement) .checked From 2160fa924063444b4674db80d814b6e8328b1eac Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 22:38:29 +0530 Subject: [PATCH 22/33] feat(local): colourful type pictures on their own backdrops Image is a photo print over another, a landscape under a sun; Infographic a sheet with a donut, a bar chart and a trend line. Each carries its own colours and a gradient backdrop that fills its card, so it reads the same in light and dark; the card's hover and selection moved to its border. --- .../src/components/ui/picture-type-icons.tsx | 290 ++++++++++-------- .../format-options/picture-type-picker.tsx | 6 +- 2 files changed, 164 insertions(+), 132 deletions(-) diff --git a/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx b/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx index c9ea6d2b35..92fd91798b 100644 --- a/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx +++ b/surfsense_local/frontend/src/components/ui/picture-type-icons.tsx @@ -1,16 +1,22 @@ -import type { SVGProps } from "react" +import { type SVGProps, useId } from "react" import { cn } from "@/lib/utils" -// Studio's picture types as small pictures for their cards, drawn on the -// theme's tokens so they follow light and dark. 3:2, the card's thumbnail. -const VIEW_BOX = "0 0 96 64" +// Studio's picture types as small pictures for their cards. They carry their +// own colours and backdrop, as a thumbnail does, so they read the same in light and dark. +// 3:2, the card's thumbnail. +const VIEW_BOX = "0 0 120 80" +const INK = "#1f2a44" -/** A framed picture: hills under a sun. */ +/** A photo print over another: a landscape under a sun, and sparks for AI. */ export function ImageTypeIcon({ className, ...props }: SVGProps<SVGSVGElement>) { + const id = useId() + const photo = `${id}-photo` + const sky = `${id}-sky` + const backdrop = `${id}-backdrop` return ( <svg viewBox={VIEW_BOX} @@ -20,50 +26,96 @@ export function ImageTypeIcon({ className={cn("size-full", className)} {...props} > + <defs> + <clipPath id={photo}> + <rect x="33" y="19" width="56" height="38" rx="2.5" /> + </clipPath> + <linearGradient id={backdrop} x1="0" y1="0" x2="1" y2="1"> + <stop offset="0" stopColor="#ffe3ee" /> + <stop offset="1" stopColor="#e2e6ff" /> + </linearGradient> + <linearGradient id={sky} x1="0" y1="0" x2="0" y2="1"> + <stop offset="0" stopColor="#9fd3ff" /> + <stop offset="1" stopColor="#e3f3ff" /> + </linearGradient> + </defs> + <rect width="120" height="80" fill={`url(#${backdrop})`} /> + <circle cx="12" cy="10" r="16" fill="#ffffff" opacity="0.45" /> + <circle cx="112" cy="74" r="20" fill="#ffffff" opacity="0.35" /> + <ellipse cx="61" cy="73" rx="34" ry="3" fill="#000" opacity="0.12" /> <rect - x="14" - y="8" - width="68" - height="48" - rx="5" - className="fill-card stroke-foreground" - strokeWidth="1.6" - /> - <circle - cx="64" - cy="21" - r="5.5" - className="fill-secondary stroke-foreground" - strokeWidth="1.4" - /> - <path - d="M15 45 L36 26 L52 41 L60 34 L81 51 V51.5 A3.5 3.5 0 0 1 77.5 55 H18.5 A3.5 3.5 0 0 1 15 51.5 Z" - className="fill-muted stroke-foreground" - strokeWidth="1.4" - strokeLinejoin="round" - /> + x="30" + y="11" + width="62" + height="52" + rx="4" + transform="rotate(-9 61 37)" + fill="#ffd3e2" + stroke={INK} + strokeOpacity="0.35" + strokeWidth="1" + /> + <g transform="rotate(4 61 41)"> + <rect + x="28" + y="14" + width="66" + height="56" + rx="4" + fill="#ffffff" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1.1" + /> + <g clipPath={`url(#${photo})`}> + <rect x="33" y="19" width="56" height="38" fill={`url(#${sky})`} /> + <circle cx="75" cy="29" r="5.5" fill="#ffc83d" /> + <path + d="M54 27.5 a3 3 0 0 1 5.6 -1.2 a2.4 2.4 0 0 1 3.4 2.2 h-9 Z" + fill="#ffffff" + /> + <path + d="M33 50 L48 32 L58 43 L67 35 L89 53 V57 H33 Z" + fill="#7c8cf8" + /> + <path + d="M44.6 36.9 L48 32 L51.6 36.3 L49.6 35.3 L47.4 37.6 Z" + fill="#ffffff" + /> + <path d="M60 41 L67 35 L89 53 V57 H60 Z" fill="#5664e0" /> + <path + d="M33 57 V50 C45 44 56 47 64 50 C74 54 82 49 89 47 V57 Z" + fill="#4cc38a" + /> + <path + d="M33 57 V53 C44 50 52 52 60 55 C66 57 72 55 76 57 Z" + fill="#2fa56e" + /> + </g> + <rect x="33" y="62" width="22" height="2.6" rx="1.3" fill="#e4e7ee" /> + </g> <path - d="M15 52 L30 41 L44 50 L50 46 L64 55" - className="stroke-muted-foreground" - strokeWidth="1.2" - strokeLinecap="round" - strokeLinejoin="round" + d="M98 12 C98.6 15.8 99.8 17 103.5 17.5 C99.8 18 98.6 19.2 98 23 C97.4 19.2 96.2 18 92.5 17.5 C96.2 17 97.4 15.8 98 12 Z" + fill="#ff6fa8" /> <path - d="M31 33 l3 -2.5 M39 30 l2.5 2" - className="stroke-muted-foreground" - strokeWidth="1.1" - strokeLinecap="round" + d="M22 50 C22.3 52 23 52.7 25 53 C23 53.3 22.3 54 22 56 C21.7 54 21 53.3 19 53 C21 52.7 21.7 52 22 50 Z" + fill="#ffc83d" /> + <circle cx="104" cy="28" r="1.4" fill="#7c8cf8" /> </svg> ) } -/** A sketchnote: a heading, two notes joined by an arrow, and a small chart. */ +/** A poster of figures: a title, a donut with its legend, and a bar chart. */ export function InfographicTypeIcon({ className, ...props }: SVGProps<SVGSVGElement>) { + const id = useId() + const sheet = `${id}-sheet` + const header = `${id}-header` + const backdrop = `${id}-backdrop` return ( <svg viewBox={VIEW_BOX} @@ -73,110 +125,90 @@ export function InfographicTypeIcon({ className={cn("size-full", className)} {...props} > + <defs> + <clipPath id={sheet}> + <rect x="22" y="8" width="76" height="62" rx="5" /> + </clipPath> + <linearGradient id={backdrop} x1="0" y1="0" x2="1" y2="1"> + <stop offset="0" stopColor="#ddf7ea" /> + <stop offset="1" stopColor="#dcebff" /> + </linearGradient> + <linearGradient id={header} x1="0" y1="0" x2="1" y2="0"> + <stop offset="0" stopColor="#8b5cf6" /> + <stop offset="1" stopColor="#3b82f6" /> + </linearGradient> + </defs> + <rect width="120" height="80" fill={`url(#${backdrop})`} /> + <circle cx="108" cy="12" r="18" fill="#ffffff" opacity="0.45" /> + <circle cx="10" cy="70" r="16" fill="#ffffff" opacity="0.35" /> + <ellipse cx="60" cy="74" rx="36" ry="3" fill="#000" opacity="0.12" /> + <rect x="22" y="8" width="76" height="62" rx="5" fill="#ffffff" /> <rect - x="10" - y="6" + x="22" + y="8" width="76" - height="52" - rx="5" - className="fill-card stroke-foreground" - strokeWidth="1.6" - /> - <rect - x="18" - y="12" - width="34" - height="5" - rx="2.5" - className="fill-foreground" - /> - <rect - x="56" - y="13" - width="14" - height="3" - rx="1.5" - className="fill-muted-foreground" + height="7" + clipPath={`url(#${sheet})`} + fill={`url(#${header})`} /> <rect - x="17" - y="23" - width="26" - height="14" - rx="3" - className="fill-muted stroke-foreground" - strokeWidth="1.2" + x="22" + y="8" + width="76" + height="62" + rx="5" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1.1" /> - <path - d="M21 28 H38 M21 32 H33" - className="stroke-muted-foreground" - strokeWidth="1.2" - strokeLinecap="round" + <rect x="30" y="20" width="32" height="4" rx="2" fill={INK} /> + <rect x="30" y="27" width="22" height="2.5" rx="1.25" fill="#a3acc2" /> + <circle cx="42" cy="49" r="9" strokeWidth="5" stroke="#e8ebf2" /> + <circle + cx="42" + cy="49" + r="9" + strokeWidth="5" + strokeDasharray="24 56.5" + transform="rotate(-90 42 49)" + stroke="#ff7a59" /> - <path - d="M30 37 C30 43 34 45 40 45" - className="stroke-foreground" - strokeWidth="1.2" - strokeLinecap="round" + <circle + cx="42" + cy="49" + r="9" + strokeWidth="5" + strokeDasharray="17 56.5" + strokeDashoffset="-24" + transform="rotate(-90 42 49)" + stroke="#ffc83d" /> + <circle + cx="42" + cy="49" + r="9" + strokeWidth="5" + strokeDasharray="15.5 56.5" + strokeDashoffset="-41" + transform="rotate(-90 42 49)" + stroke="#3b82f6" + /> + <circle cx="72" cy="21" r="2" fill="#ff7a59" /> + <rect x="76" y="20" width="14" height="2" rx="1" fill="#e4e7ee" /> + <circle cx="72" cy="27" r="2" fill="#3b82f6" /> + <rect x="76" y="26" width="10" height="2" rx="1" fill="#e4e7ee" /> + <rect x="60" y="49" width="5" height="12" rx="1" fill="#3b82f6" /> + <rect x="67.5" y="43" width="5" height="18" rx="1" fill="#22c55e" /> + <rect x="75" y="46" width="5" height="15" rx="1" fill="#ffc83d" /> + <rect x="82.5" y="37" width="5" height="24" rx="1" fill="#ff7a59" /> + <path d="M58 61.5 H90" stroke={INK} strokeOpacity="0.4" strokeWidth="1" /> <path - d="M37.5 42.5 L40.5 45 L37.5 47.5" - className="stroke-foreground" + d="M60 40 L67 35 L74 38 L84 31" + stroke="#8b5cf6" strokeWidth="1.2" strokeLinecap="round" strokeLinejoin="round" - /> - <rect - x="43" - y="40" - width="20" - height="11" - rx="3" - className="fill-secondary stroke-foreground" - strokeWidth="1.2" - /> - <path - d="M47 45.5 H59" - className="stroke-muted-foreground" - strokeWidth="1.2" - strokeLinecap="round" - /> - <path - d="M66 50 H80" - className="stroke-foreground" - strokeWidth="1.2" - strokeLinecap="round" - /> - <rect - x="68" - y="39" - width="3.5" - height="11" - rx="1" - className="fill-chart-3" - /> - <rect - x="73" - y="31" - width="3.5" - height="19" - rx="1" - className="fill-chart-1" - /> - <rect - x="78" - y="35" - width="3.5" - height="15" - rx="1" - className="fill-chart-4" - /> - <circle - cx="74" - cy="24.5" - r="3" - className="fill-card stroke-foreground" - strokeWidth="1.1" + strokeDasharray="2 2" /> </svg> ) diff --git a/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx index b44b048566..8e93100fc7 100644 --- a/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx +++ b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx @@ -63,10 +63,10 @@ export function PictureTypePicker({ /> <span className={cn( - "relative flex aspect-3/2 items-center justify-center overflow-hidden rounded-lg border bg-muted/40 px-5 py-3 transition-colors duration-150 ease-out peer-focus-visible:ring-2 peer-focus-visible:ring-ring motion-reduce:transition-none", + "relative flex aspect-3/2 items-center justify-center overflow-hidden rounded-lg border transition-colors duration-150 ease-out peer-focus-visible:ring-2 peer-focus-visible:ring-ring motion-reduce:transition-none", checked - ? "border-primary bg-primary/5" - : "border-border group-hover:bg-accent" + ? "border-primary ring-1 ring-primary" + : "border-border group-hover:border-foreground/30" )} > {Art ? <Art /> : null} From b6e1ce492d564561fac718ca763e6f3fb1e83d1d Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 23:20:54 +0530 Subject: [PATCH 23/33] feat(local): ScrollFade fades a row that scrolls sideways orientation="horizontal" fades the start and end edges with shadcn's scroll-fade-x; vertical stays the default. --- surfsense_local/frontend/src/components/ui/scroll-fade.tsx | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/surfsense_local/frontend/src/components/ui/scroll-fade.tsx b/surfsense_local/frontend/src/components/ui/scroll-fade.tsx index 7e8abca926..504c923670 100644 --- a/surfsense_local/frontend/src/components/ui/scroll-fade.tsx +++ b/surfsense_local/frontend/src/components/ui/scroll-fade.tsx @@ -9,6 +9,7 @@ export function ScrollFade({ className, viewportClassName, scroll = true, + orientation = "vertical", ...viewportProps }: { children: ReactNode @@ -18,6 +19,8 @@ export function ScrollFade({ // dialog section that scrolls its heading and body together). Renders // children at their natural height instead of a clipped, scrolling box. scroll?: boolean + // Horizontal fades the start and end edges, for a row that scrolls sideways. + orientation?: "vertical" | "horizontal" } & Omit<ComponentProps<"div">, "children" | "className">) { // The rest, `ref` included, land on the viewport: it is the element that // scrolls, so it is the one a caller measures, labels or listens to. @@ -35,7 +38,9 @@ export function ScrollFade({ {...viewportProps} data-slot="scroll-fade-viewport" className={cn( - "h-full min-h-0 scroll-fade overflow-y-auto overscroll-contain [--scroll-fade-reveal:24px] scroll-fade-6", + orientation === "horizontal" + ? "scroll-fade-x overflow-x-auto overscroll-x-contain [--scroll-fade-reveal:24px] scroll-fade-6" + : "h-full min-h-0 scroll-fade overflow-y-auto overscroll-contain [--scroll-fade-reveal:24px] scroll-fade-6", viewportClassName )} > From b23b27ae70158f99e8481d2881d2bfdbc2c40bf8 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 23:21:00 +0530 Subject: [PATCH 24/33] feat(local): comic kinds are illustrated cards in a scrolling row Each of the ten kinds has its own picture, drawn on a shared frame and registered as icons. Kind is a row of cards that scrolls inside the dialog, with a fade at either edge and arrows only where more kinds lie; Pages stays a dropdown beside the cost line. Image's types and the kinds share one card. --- .../src/components/ui/comic-kind-icons.tsx | 497 ++++++++++++++++++ .../frontend/src/components/ui/icons.tsx | 15 + .../src/components/ui/picture-art.tsx | 123 +++++ .../format-options/comic-kind-picker.tsx | 170 ++++++ .../format-options/comic-options-form.tsx | 94 ++-- .../studio/format-options/picture-card.tsx | 62 +++ .../format-options/picture-type-picker.tsx | 61 +-- .../src/features/studio/studio-panel.test.tsx | 15 +- surfsense_local/frontend/translations/de.json | 2 + surfsense_local/frontend/translations/en.json | 2 + surfsense_local/frontend/translations/es.json | 2 + surfsense_local/frontend/translations/fr.json | 2 + surfsense_local/frontend/translations/hi.json | 2 + surfsense_local/frontend/translations/ja.json | 2 + surfsense_local/frontend/translations/ko.json | 2 + .../frontend/translations/pt-BR.json | 2 + surfsense_local/frontend/translations/ru.json | 2 + .../frontend/translations/zh-CN.json | 2 + 18 files changed, 950 insertions(+), 107 deletions(-) create mode 100644 surfsense_local/frontend/src/components/ui/comic-kind-icons.tsx create mode 100644 surfsense_local/frontend/src/components/ui/picture-art.tsx create mode 100644 surfsense_local/frontend/src/features/studio/format-options/comic-kind-picker.tsx create mode 100644 surfsense_local/frontend/src/features/studio/format-options/picture-card.tsx diff --git a/surfsense_local/frontend/src/components/ui/comic-kind-icons.tsx b/surfsense_local/frontend/src/components/ui/comic-kind-icons.tsx new file mode 100644 index 0000000000..fa35706ceb --- /dev/null +++ b/surfsense_local/frontend/src/components/ui/comic-kind-icons.tsx @@ -0,0 +1,497 @@ +import { type SVGProps, useId } from "react" + +import { + Bubble, + Ground, + INK, + Panel, + PictureArt, + Sheet, + burst, +} from "./picture-art" + +// Each kind of comic Studio draws, as a picture for its card. +// SVG's own from and to animate; here they name a picture's backdrop colours. +type ArtProps = Omit<SVGProps<SVGSVGElement>, "from" | "to"> + +/** A page of colour panels, a speech bubble and a POW burst. */ +export function WesternComicIcon(props: ArtProps) { + return ( + <PictureArt from="#dbeafe" to="#e0e7ff" {...props}> + <Ground /> + <Sheet x="34" y="8" width="52" height="62" /> + <Panel x="38" y="12" width="44" height="20" fill="#93c5fd" /> + <path + d="M38 32 V26 H43 V22 H48 V27 H52 V20 H57 V25 H62 V32 Z" + fill="#3b82f6" + /> + <Bubble cx={70} cy={19} /> + <Panel x="38" y="35" width="20" height="31" fill="#fca5a5" /> + <circle + cx="48" + cy="47" + r="5" + fill="#fde68a" + stroke={INK} + strokeWidth="0.7" + /> + <path d="M43 66 C43 58 53 58 53 66 Z" fill="#ef4444" /> + <Panel x="61" y="35" width="21" height="31" fill="#fde68a" /> + <path d="M64 63 L71 41 L78 63 Z" fill="#fb923c" opacity="0.8" /> + <polygon + points={burst(92, 27, 13, 7.5)} + fill="#facc15" + stroke={INK} + strokeWidth="0.8" + strokeLinejoin="round" + /> + <polygon points={burst(92, 27, 7, 4)} fill="#ef4444" /> + </PictureArt> + ) +} + +/** A hardcover: a moonlit cover on a thick block of pages. */ +export function GraphicNovelIcon(props: ArtProps) { + const cover = `${useId()}-cover` + return ( + <PictureArt from="#ede9fe" to="#fae8ff" {...props}> + <defs> + <linearGradient id={cover} x1="0" y1="0" x2="0" y2="1"> + <stop offset="0" stopColor="#7c3aed" /> + <stop offset="1" stopColor="#4c1d95" /> + </linearGradient> + </defs> + <Ground /> + <rect + x="40" + y="10" + width="44" + height="60" + rx="2" + fill="#f5f5f4" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1" + /> + <path d="M81 12 V68 M83 13 V67" stroke="#d6d3d1" strokeWidth="0.6" /> + <rect + x="36" + y="8" + width="44" + height="60" + rx="2.5" + fill={`url(#${cover})`} + stroke={INK} + strokeOpacity="0.6" + strokeWidth="1" + /> + <rect x="36" y="8" width="5" height="60" rx="1.5" fill="#3b0764" /> + <circle cx="66" cy="22" r="6" fill="#fde68a" /> + <circle cx="68.5" cy="20.5" r="5" fill="#6d28d9" /> + <path d="M41 52 L52 40 L60 47 L69 36 L80 46 V68 H41 Z" fill="#2e1065" /> + <rect + x="46" + y="56" + width="26" + height="3" + rx="1.5" + fill="#ffffff" + opacity="0.9" + /> + <rect + x="50" + y="61" + width="18" + height="2" + rx="1" + fill="#ffffff" + opacity="0.55" + /> + <circle cx="49" cy="17" r="0.9" fill="#ffffff" /> + <circle cx="56" cy="26" r="0.7" fill="#ffffff" /> + <circle cx="74" cy="32" r="0.7" fill="#ffffff" /> + </PictureArt> + ) +} + +/** An album page in clean lines and flat colour, nine small panels. */ +export function BandeDessineeIcon(props: ArtProps) { + const fills = [ + "#93c5fd", + "#fde68a", + "#fca5a5", + "#86efac", + "#ffffff", + "#93c5fd", + "#fdba74", + "#c4b5fd", + "#fde68a", + ] + return ( + <PictureArt from="#ffedd5" to="#fef3c7" {...props}> + <Ground /> + <Sheet x="31" y="7" width="58" height="64" /> + {fills.map((fill, i) => ( + <Panel + key={fill + i} + x={35 + (i % 3) * 17.3} + y={11 + Math.floor(i / 3) * 19.6} + width="15.3" + height="17.6" + fill={fill} + /> + ))} + <Bubble cx={60.5} cy={35.5} rx={5} ry={3} /> + <circle cx="43" cy="21" r="3" fill="#fbbf24" /> + <path + d="M35 28.6 L41 23 L45 26 L50.3 21.5 V28.6 Z" + fill="#2563eb" + opacity="0.7" + /> + <path d="M78 66 l2.5 -5 l2.5 5 Z" fill="#ef4444" /> + </PictureArt> + ) +} + +/** A black-and-white page: screentone, speed lines and a wide eye, read right to left. */ +export function MangaIcon(props: ArtProps) { + const tone = `${useId()}-tone` + return ( + <PictureArt from="#fce7f3" to="#f5f3ff" {...props}> + <defs> + <pattern + id={tone} + width="2.6" + height="2.6" + patternUnits="userSpaceOnUse" + > + <circle cx="1.3" cy="1.3" r="0.55" fill={INK} opacity="0.55" /> + </pattern> + </defs> + <Ground /> + <Sheet x="36" y="8" width="48" height="62" /> + <Panel x="40" y="12" width="40" height="24" fill="#ffffff" /> + <path + d="M41 13 L60 24 M41 22 L60 24 M41 34 L60 24 M79 13 L60 24 M79 30 L60 24 M70 35 L60 24 M50 13 L60 24" + stroke={INK} + strokeOpacity="0.35" + strokeWidth="0.6" + /> + <path + d="M49 24 C53 18 67 18 71 24 C67 30 53 30 49 24 Z" + fill="#ffffff" + stroke={INK} + strokeWidth="1" + /> + <circle cx="60" cy="24" r="4.2" fill={INK} /> + <circle cx="61.4" cy="22.6" r="1.3" fill="#ffffff" /> + <Panel x="61" y="39" width="19" height="27" fill={`url(#${tone})`} /> + <Panel x="40" y="39" width="18" height="27" fill="#ffffff" /> + <ellipse + cx="49" + cy="50" + rx="5.5" + ry="7.5" + fill="#ffffff" + stroke={INK} + strokeWidth="0.7" + /> + <path + d="M47 46 V54 M50.5 45 V55" + stroke={INK} + strokeWidth="0.9" + strokeLinecap="round" + /> + <path + d="M100 15 H90 M93 12 L90 15 L93 18" + stroke="#ec4899" + strokeWidth="1.5" + strokeLinecap="round" + strokeLinejoin="round" + /> + </PictureArt> + ) +} + +/** A painterly page: a sunset over mountains, and a red lantern. */ +export function ManhuaIcon(props: ArtProps) { + const sunset = `${useId()}-sunset` + return ( + <PictureArt from="#fee2e2" to="#ffedd5" {...props}> + <defs> + <linearGradient id={sunset} x1="0" y1="0" x2="0" y2="1"> + <stop offset="0" stopColor="#fb923c" /> + <stop offset="1" stopColor="#f43f5e" /> + </linearGradient> + </defs> + <Ground /> + <Sheet x="36" y="8" width="48" height="62" /> + <Panel x="40" y="12" width="40" height="28" fill={`url(#${sunset})`} /> + <circle cx="66" cy="22" r="5" fill="#fde68a" opacity="0.9" /> + <path d="M40.4 40 L48 27 L55 34 L63 24 L79.6 39.6 V40 Z" fill="#9f1239" /> + <path d="M40.4 40 L46 34 L53 38 L58 33 L68 40 Z" fill="#4c0519" /> + <Panel x="40" y="43" width="40" height="23" fill="#fef3c7" /> + <path d="M51 43 V47" stroke="#b45309" strokeWidth="0.8" /> + <rect x="47.5" y="47" width="7" height="2" rx="0.6" fill="#f59e0b" /> + <ellipse cx="51" cy="54" rx="6" ry="5.5" fill="#dc2626" /> + <path + d="M48 50 Q51 54 48 58 M54 50 Q51 54 54 58" + stroke="#991b1b" + strokeWidth="0.6" + /> + <rect x="47.5" y="58.5" width="7" height="2" rx="0.6" fill="#f59e0b" /> + <path d="M51 60.5 V64" stroke="#f59e0b" strokeWidth="1" /> + <path + d="M62 49 H75 M62 53 H72 M62 57 H74" + stroke="#d6b88a" + strokeWidth="1.2" + strokeLinecap="round" + /> + </PictureArt> + ) +} + +/** A phone of tall panels stacked to scroll down. */ +export function WebtoonIcon(props: ArtProps) { + const screen = `${useId()}-screen` + return ( + <PictureArt from="#d1fae5" to="#cffafe" {...props}> + <defs> + <clipPath id={screen}> + <rect x="46.5" y="10" width="27" height="58" rx="2" /> + </clipPath> + </defs> + <Ground rx={22} /> + <rect x="44" y="6" width="32" height="66" rx="5" fill={INK} /> + <rect x="46.5" y="10" width="27" height="58" rx="2" fill="#ffffff" /> + <g clipPath={`url(#${screen})`}> + <Panel x="48.5" y="12" width="23" height="17" fill="#a7f3d0" /> + <circle cx="64" cy="18" r="3.5" fill="#fbbf24" /> + <path d="M48.5 29 L55 22 L61 29 Z" fill="#10b981" /> + <Panel x="48.5" y="32" width="23" height="17" fill="#bfdbfe" /> + <Bubble cx={57} cy={38} rx={5} ry={3} /> + <Panel x="48.5" y="52" width="23" height="17" fill="#fbcfe8" /> + <circle cx="60" cy="61" r="4" fill="#f472b6" /> + </g> + <rect x="55" y="7.5" width="10" height="1.5" rx="0.75" fill="#475569" /> + <path + d="M88 22 V54 M84 50 L88 55 L92 50" + stroke="#10b981" + strokeWidth="1.8" + strokeLinecap="round" + strokeLinejoin="round" + /> + <path + d="M88 22 V54" + stroke="#10b981" + strokeWidth="1.8" + strokeDasharray="3 3" + strokeLinecap="round" + /> + </PictureArt> + ) +} + +/** Three panels in a row, the punchline last. */ +export function ComicStripIcon(props: ArtProps) { + return ( + <PictureArt from="#fef9c3" to="#fde68a" {...props}> + <Ground rx={46} /> + <Sheet x="12" y="22" width="96" height="38" /> + <Panel x="15.5" y="25.5" width="28" height="31" fill="#bae6fd" /> + <Panel x="46" y="25.5" width="28" height="31" fill="#c7d2fe" /> + <Panel x="76.5" y="25.5" width="28" height="31" fill="#fecaca" /> + <Bubble cx={26} cy={32} rx={6} ry={3.2} /> + <circle + cx="31" + cy="48" + r="4.5" + fill="#fde68a" + stroke={INK} + strokeWidth="0.7" + /> + <Bubble cx={57} cy={32} rx={6} ry={3.2} /> + <circle + cx="62" + cy="48" + r="4.5" + fill="#fde68a" + stroke={INK} + strokeWidth="0.7" + /> + <polygon + points={burst(90, 36, 7, 3.5, 8)} + fill="#facc15" + stroke={INK} + strokeWidth="0.7" + strokeLinejoin="round" + /> + <circle + cx="92" + cy="49" + r="4.5" + fill="#fde68a" + stroke={INK} + strokeWidth="0.7" + /> + <path + d="M90 50 Q92 52 94 50" + stroke={INK} + strokeWidth="0.7" + strokeLinecap="round" + /> + </PictureArt> + ) +} + +/** Four equal panels stacked top to bottom, its title down the side. */ +export function YonkomaIcon(props: ArtProps) { + const fills = ["#fde68a", "#bbf7d0", "#bfdbfe", "#fecaca"] + return ( + <PictureArt from="#ede9fe" to="#e0e7ff" {...props}> + <Ground rx={20} /> + <Sheet x="45" y="5" width="30" height="67" /> + {fills.map((fill, i) => ( + <Panel + key={fill} + x="48.5" + y={8.5 + i * 15.6} + width="23" + height="13.6" + fill={fill} + /> + ))} + <circle cx="56" cy="15" r="2.6" fill="#fb923c" /> + <circle cx="62" cy="31" r="2.6" fill="#22c55e" /> + <circle cx="58" cy="46" r="2.6" fill="#3b82f6" /> + <polygon points={burst(60, 62, 4, 2, 6)} fill="#ef4444" /> + <rect + x="78" + y="9" + width="3.5" + height="26" + rx="1.75" + fill={INK} + opacity="0.75" + /> + <rect x="78" y="38" width="3.5" height="10" rx="1.75" fill="#a78bfa" /> + </PictureArt> + ) +} + +/** An open book: a sunny picture on one page, its words on the other. */ +export function PictureBookIcon(props: ArtProps) { + return ( + <PictureArt from="#dcfce7" to="#ecfccb" {...props}> + <Ground rx={42} /> + <path + d="M60 66 C50 61 33 61 18 64 V16 C33 13 50 13 60 18 C70 13 87 13 102 16 V64 C87 61 70 61 60 66 Z" + fill="#f97316" + /> + <path + d="M60 62 C50 57 34 57 21 60 V14 C34 11 50 11 60 16 Z" + fill="#ffffff" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1" + strokeLinejoin="round" + /> + <path + d="M60 62 C70 57 86 57 99 60 V14 C86 11 70 11 60 16 Z" + fill="#ffffff" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1" + strokeLinejoin="round" + /> + <path + d="M24 50 C33 44 46 46 57 51 V57 C46 54 34 54 24 56 Z" + fill="#86efac" + /> + <circle cx="47" cy="24" r="4.5" fill="#fbbf24" /> + <rect x="31" y="38" width="2.6" height="12" rx="1" fill="#a16207" /> + <circle cx="32.3" cy="35" r="6" fill="#22c55e" /> + <circle cx="29" cy="33" r="1" fill="#f472b6" /> + <path + d="M66 22 H92 M66 28 H90 M66 34 H93 M66 40 H86 M66 46 H91" + stroke="#cbd5e1" + strokeWidth="2" + strokeLinecap="round" + /> + <path + d="M85 53 C85 51 88 51 88 53 C88 51 91 51 91 53 C91 55.5 88 57 88 57 C88 57 85 55.5 85 53 Z" + fill="#f472b6" + /> + </PictureArt> + ) +} + +/** A sheet of rough frames, each a shot, with arrows and a camera. */ +export function StoryboardIcon(props: ArtProps) { + const frames: [number, number][] = [ + [29, 13], + [62, 13], + [29, 41], + [62, 41], + ] + return ( + <PictureArt from="#e2e8f0" to="#dbeafe" {...props}> + <Ground rx={38} /> + <Sheet x="24" y="8" width="72" height="62" /> + {frames.map(([x, y]) => ( + <g key={`${x}-${y}`}> + <rect + x={x} + y={y} + width="29" + height="18" + rx="1" + fill="#f1f5f9" + stroke={INK} + strokeOpacity="0.7" + strokeWidth="0.8" + /> + <path + d={`M${x + 3} ${y + 15} C${x + 9} ${y + 9} ${x + 15} ${y + 13} ${x + 26} ${y + 7}`} + stroke="#64748b" + strokeWidth="0.8" + strokeLinecap="round" + /> + <circle + cx={x + 10} + cy={y + 7} + r="2.6" + stroke="#64748b" + strokeWidth="0.8" + /> + <rect x={x} y={y + 20} width="18" height="2" rx="1" fill="#cbd5e1" /> + </g> + ))} + <path + d="M58.5 22 H61.5 M59.8 20.5 L61.5 22 L59.8 23.5" + stroke="#3b82f6" + strokeWidth="1.1" + strokeLinecap="round" + strokeLinejoin="round" + /> + <path + d="M76 33.5 V39 M74.5 37.5 L76 39 L77.5 37.5" + stroke="#3b82f6" + strokeWidth="1.1" + strokeLinecap="round" + strokeLinejoin="round" + /> + <rect x="88" y="4" width="20" height="13" rx="3" fill={INK} /> + <rect x="92" y="2.5" width="7" height="3" rx="1" fill={INK} /> + <circle + cx="98" + cy="10.5" + r="4.2" + fill="#60a5fa" + stroke="#ffffff" + strokeWidth="1" + /> + <circle cx="98" cy="10.5" r="1.6" fill="#1e3a8a" /> + </PictureArt> + ) +} diff --git a/surfsense_local/frontend/src/components/ui/icons.tsx b/surfsense_local/frontend/src/components/ui/icons.tsx index 10ecf59c21..33eb80f361 100644 --- a/surfsense_local/frontend/src/components/ui/icons.tsx +++ b/surfsense_local/frontend/src/components/ui/icons.tsx @@ -29,6 +29,7 @@ import { CheckmarkBadge02Icon as CheckmarkBadge02IconData, CheckmarkCircle02Icon, ChevronDownIcon as ChevronDownIconData, + ChevronLeftIcon as ChevronLeftIconData, ChevronRightIcon as ChevronRightIconData, Clock01Icon as Clock01IconData, CircleArrowUp02Icon as CircleArrowUp02IconData, @@ -168,6 +169,7 @@ export const CheckCircle2Icon = createIcon(CheckmarkCircle02Icon) export const CheckIcon = createIcon(CheckIconData) export const CheckmarkBadgeIcon = createIcon(CheckmarkBadge02IconData) export const ChevronDownIcon = createIcon(ChevronDownIconData) +export const ChevronLeftIcon = createIcon(ChevronLeftIconData) export const ChevronRightIcon = createIcon(ChevronRightIconData) export const ClockIcon = createIcon(Clock01IconData) export const CircleArrowUp02Icon = createIcon(CircleArrowUp02IconData) @@ -261,3 +263,16 @@ export const ZoomOutIcon = createIcon(ZoomOutIconData) // Studio's picture types, drawn for their cards rather than taken from the icon set. export { ImageTypeIcon, InfographicTypeIcon } from "./picture-type-icons" +// The kinds of comic Studio draws, drawn for their cards in the same way. +export { + BandeDessineeIcon, + ComicStripIcon, + GraphicNovelIcon, + MangaIcon, + ManhuaIcon, + PictureBookIcon, + StoryboardIcon, + WebtoonIcon, + WesternComicIcon, + YonkomaIcon, +} from "./comic-kind-icons" diff --git a/surfsense_local/frontend/src/components/ui/picture-art.tsx b/surfsense_local/frontend/src/components/ui/picture-art.tsx new file mode 100644 index 0000000000..1e031a41b2 --- /dev/null +++ b/surfsense_local/frontend/src/components/ui/picture-art.tsx @@ -0,0 +1,123 @@ +import { type ReactNode, type SVGProps, useId } from "react" + +import { cn } from "@/lib/utils" + +// The frame every Studio card picture shares: a 3:2 canvas on its own +// gradient backdrop, so it reads the same in light and dark. +export const INK = "#1f2a44" + +export function PictureArt({ + from, + to, + className, + children, + ...props +}: Omit<SVGProps<SVGSVGElement>, "from" | "to"> & { + from: string + to: string + children: ReactNode +}) { + const backdrop = `${useId()}-backdrop` + return ( + <svg + viewBox="0 0 120 80" + fill="none" + aria-hidden="true" + data-slot="picture-type-art" + className={cn("size-full", className)} + {...props} + > + <defs> + <linearGradient id={backdrop} x1="0" y1="0" x2="1" y2="1"> + <stop offset="0" stopColor={from} /> + <stop offset="1" stopColor={to} /> + </linearGradient> + </defs> + <rect width="120" height="80" fill={`url(#${backdrop})`} /> + <circle cx="12" cy="10" r="16" fill="#ffffff" opacity="0.45" /> + <circle cx="112" cy="74" r="20" fill="#ffffff" opacity="0.35" /> + {children} + </svg> + ) +} + +/** The soft shadow a picture's object stands on. */ +export function Ground({ cx = 60, rx = 32 }: { cx?: number; rx?: number }) { + return <ellipse cx={cx} cy="74" rx={rx} ry="3" fill="#000" opacity="0.12" /> +} + +/** A white sheet with the ink edge every printed object has. */ +export function Sheet(props: SVGProps<SVGRectElement>) { + return ( + <rect + rx="3" + fill="#ffffff" + stroke={INK} + strokeOpacity="0.45" + strokeWidth="1.1" + {...props} + /> + ) +} + +/** A comic panel: a flat colour inside a thin ink border. */ +export function Panel(props: SVGProps<SVGRectElement>) { + return ( + <rect + rx="1" + stroke={INK} + strokeOpacity="0.7" + strokeWidth="0.8" + {...props} + /> + ) +} + +/** A speech bubble with its tail pointing down and left. */ +export function Bubble({ + cx, + cy, + rx = 6, + ry = 3.5, +}: { + cx: number + cy: number + rx?: number + ry?: number +}) { + return ( + <g> + <path + d={`M${cx - rx * 0.4} ${cy + ry * 0.7} L${cx - rx * 0.75} ${cy + ry + 2.5} L${cx - rx * 0.05} ${cy + ry * 0.85} Z`} + fill="#ffffff" + stroke={INK} + strokeWidth="0.7" + strokeLinejoin="round" + /> + <ellipse + cx={cx} + cy={cy} + rx={rx} + ry={ry} + fill="#ffffff" + stroke={INK} + strokeWidth="0.7" + /> + </g> + ) +} + +/** A star burst's points, as an SVG polygon takes them. */ +export function burst( + cx: number, + cy: number, + outer: number, + inner: number, + points = 10 +): string { + return Array.from({ length: points * 2 }, (_, i) => { + const radius = i % 2 === 0 ? outer : inner + const angle = (Math.PI * i) / points - Math.PI / 2 + return `${(cx + radius * Math.cos(angle)).toFixed(1)},${(cy + radius * Math.sin(angle)).toFixed(1)}` + }).join(" ") +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/comic-kind-picker.tsx b/surfsense_local/frontend/src/features/studio/format-options/comic-kind-picker.tsx new file mode 100644 index 0000000000..ecacf58d76 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/comic-kind-picker.tsx @@ -0,0 +1,170 @@ +import { + type ComponentType, + useCallback, + useEffect, + useRef, + useState, +} from "react" + +import { Button } from "@/components/ui/button" +import { + BandeDessineeIcon, + ChevronLeftIcon, + ChevronRightIcon, + ComicStripIcon, + GraphicNovelIcon, + MangaIcon, + ManhuaIcon, + PictureBookIcon, + StoryboardIcon, + WebtoonIcon, + WesternComicIcon, + YonkomaIcon, +} from "@/components/ui/icons" +import { ScrollFade } from "@/components/ui/scroll-fade" +import { intl } from "@/i18n/intl" +import { cn } from "@/lib/utils" + +import { COMIC_KINDS, type ComicKind } from "./comic-kinds" +import { PictureCard } from "./picture-card" + +const KIND_ART: Record<ComicKind, ComponentType<{ className?: string }>> = { + western: WesternComicIcon, + graphic_novel: GraphicNovelIcon, + bande_dessinee: BandeDessineeIcon, + manga: MangaIcon, + manhua: ManhuaIcon, + webtoon: WebtoonIcon, + comic_strip: ComicStripIcon, + yonkoma: YonkomaIcon, + picture_book: PictureBookIcon, + storyboard: StoryboardIcon, +} + +/** Every kind of comic as a card in one row that scrolls sideways; an arrow + * shows on a side only while more kinds lie that way. */ +export function ComicKindPicker({ + value, + onChange, +}: { + value: ComicKind + onChange: (kind: ComicKind) => void +}) { + const row = useRef<HTMLDivElement>(null) + const [more, setMore] = useState({ back: false, ahead: false }) + + const measure = useCallback(() => { + const element = row.current + if (!element) return + setMore({ + // A few pixels of slack: the snap settles within the row's padding. + back: element.scrollLeft > 4, + ahead: element.scrollLeft + element.clientWidth < element.scrollWidth - 4, + }) + }, []) + + useEffect(() => { + measure() + const element = row.current + if (!element || typeof ResizeObserver === "undefined") return + const observer = new ResizeObserver(measure) + observer.observe(element) + return () => observer.disconnect() + }, [measure]) + + const page = (direction: 1 | -1) => { + const element = row.current + if (!element) return + const still = window.matchMedia?.( + "(prefers-reduced-motion: reduce)" + ).matches + element.scrollBy({ + left: direction * element.clientWidth * 0.8, + behavior: still ? "auto" : "smooth", + }) + } + + const label = intl.formatMessage({ + id: "studio_comic_kind_label", + defaultMessage: "Kind", + }) + return ( + <div className="space-y-2"> + <p className="text-xs font-medium text-muted-foreground">{label}</p> + {/* Sized by the dialog, not by its ten cards: a grid or flex parent would + otherwise grow to fit the whole row instead of letting it scroll. */} + <div className="relative [contain:inline-size]"> + <ScrollFade + orientation="horizontal" + ref={row} + role="radiogroup" + aria-label={label} + onScroll={measure} + viewportClassName="flex snap-x scroll-px-0.5 [scrollbar-width:none] gap-2 p-0.5 [&::-webkit-scrollbar]:hidden" + > + {COMIC_KINDS.map((kind) => ( + <PictureCard + key={kind.key} + name="studio-comic-kind" + value={kind.key} + label={kind.label()} + Art={KIND_ART[kind.key]} + checked={value === kind.key} + className="w-32 shrink-0 snap-start" + onChange={(event) => { + onChange(kind.key) + // A kind reached by the arrow keys comes into view. + event.currentTarget + .closest("label") + ?.scrollIntoView?.({ block: "nearest", inline: "nearest" }) + }} + /> + ))} + </ScrollFade> + <ScrollArrow side="back" shown={more.back} onClick={() => page(-1)} /> + <ScrollArrow side="ahead" shown={more.ahead} onClick={() => page(1)} /> + </div> + </div> + ) +} + +function ScrollArrow({ + side, + shown, + onClick, +}: { + side: "back" | "ahead" + shown: boolean + onClick: () => void +}) { + if (!shown) return null + const Icon = side === "back" ? ChevronLeftIcon : ChevronRightIcon + return ( + <Button + type="button" + variant="default" + size="icon-sm" + // Pointer only: the arrow keys already move through the kinds. + tabIndex={-1} + aria-label={ + side === "back" + ? intl.formatMessage({ + id: "studio_comic_kind_scroll_back_aria", + defaultMessage: "Show earlier kinds", + }) + : intl.formatMessage({ + id: "studio_comic_kind_scroll_ahead_aria", + defaultMessage: "Show more kinds", + }) + } + onClick={onClick} + className={cn( + // Centred on the cards' pictures, which are 2/3 of a 128 px card wide. + "absolute top-[calc(128px/3-12px)] rounded-full shadow-md", + side === "back" ? "-left-2" : "-right-2" + )} + > + <Icon /> + </Button> + ) +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx b/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx index 4a7b806662..8b759f6b8e 100644 --- a/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx +++ b/surfsense_local/frontend/src/features/studio/format-options/comic-options-form.tsx @@ -3,7 +3,8 @@ import { useId } from "react" import { Select } from "@/components/ui/select" import { intl } from "@/i18n/intl" -import { COMIC_KINDS, type ComicKind, MAX_COMIC_PAGES } from "./comic-kinds" +import { ComicKindPicker } from "./comic-kind-picker" +import { type ComicKind, MAX_COMIC_PAGES } from "./comic-kinds" const PAGE_COUNTS = Array.from({ length: MAX_COMIC_PAGES }, (_, i) => i + 1) @@ -21,63 +22,44 @@ export function ComicOptionsForm({ }) { const id = useId() return ( - <div className="space-y-2"> - <div className="grid grid-cols-[2fr_1fr] gap-2"> - <div className="space-y-1"> - <label - htmlFor={`${id}-kind`} - className="text-xs font-medium text-muted-foreground" - > - {intl.formatMessage({ - id: "studio_comic_kind_label", - defaultMessage: "Kind", - })} - </label> - <Select - id={`${id}-kind`} - value={kind} - onChange={(event) => onKindChange(event.target.value as ComicKind)} - > - {COMIC_KINDS.map((entry) => ( - <option key={entry.key} value={entry.key}> - {entry.label()} - </option> - ))} - </Select> - </div> - <div className="space-y-1"> - <label - htmlFor={`${id}-pages`} - className="text-xs font-medium text-muted-foreground" - > - {intl.formatMessage({ - id: "studio_comic_pages_label", - defaultMessage: "Pages", - })} - </label> - <Select - id={`${id}-pages`} - value={String(pages)} - onChange={(event) => onPagesChange(Number(event.target.value))} - > - {PAGE_COUNTS.map((count) => ( - <option key={count} value={count}> - {intl.formatNumber(count)} - </option> - ))} - </Select> + <div className="space-y-3"> + <ComicKindPicker value={kind} onChange={onKindChange} /> + <div className="space-y-1"> + <label + htmlFor={`${id}-pages`} + className="text-xs font-medium text-muted-foreground" + > + {intl.formatMessage({ + id: "studio_comic_pages_label", + defaultMessage: "Pages", + })} + </label> + <div className="flex items-center gap-3"> + <div className="w-20 shrink-0"> + <Select + id={`${id}-pages`} + value={String(pages)} + onChange={(event) => onPagesChange(Number(event.target.value))} + > + {PAGE_COUNTS.map((count) => ( + <option key={count} value={count}> + {intl.formatNumber(count)} + </option> + ))} + </Select> + </div> + <p className="text-xs text-muted-foreground"> + {intl.formatMessage( + { + id: "studio_comic_cost_body", + defaultMessage: + "Draws {count, plural, one {# picture} other {# pictures}} with your image model. Each is billed by its provider, or takes minutes on this computer.", + }, + { count: pages } + )} + </p> </div> </div> - <p className="text-xs text-muted-foreground"> - {intl.formatMessage( - { - id: "studio_comic_cost_body", - defaultMessage: - "Draws {count, plural, one {# picture} other {# pictures}} with your image model. Each is billed by its provider, or takes minutes on this computer.", - }, - { count: pages } - )} - </p> </div> ) } diff --git a/surfsense_local/frontend/src/features/studio/format-options/picture-card.tsx b/surfsense_local/frontend/src/features/studio/format-options/picture-card.tsx new file mode 100644 index 0000000000..882fa804d3 --- /dev/null +++ b/surfsense_local/frontend/src/features/studio/format-options/picture-card.tsx @@ -0,0 +1,62 @@ +import type { ChangeEvent, ComponentType } from "react" + +import { CheckIcon } from "@/components/ui/icons" +import { cn } from "@/lib/utils" + +/** One choice shown by its picture, its label under it, the chosen one ticked. + * A radio underneath, so a group of them keeps the keyboard and screen readers. */ +export function PictureCard({ + name, + value, + label, + Art, + checked, + disabled = false, + className, + onChange, +}: { + name: string + value: string + label: string + Art: ComponentType<{ className?: string }> | undefined + checked: boolean + disabled?: boolean + className?: string + onChange: (event: ChangeEvent<HTMLInputElement>) => void +}) { + return ( + <label + className={cn( + "group relative flex min-w-0 flex-col gap-1.5", + disabled ? "cursor-not-allowed opacity-50" : "cursor-pointer", + className + )} + > + <input + type="radio" + name={name} + value={value} + checked={checked} + disabled={disabled} + className="peer sr-only" + onChange={onChange} + /> + <span + className={cn( + "relative flex aspect-3/2 items-center justify-center overflow-hidden rounded-lg border transition-colors duration-150 ease-out peer-focus-visible:ring-2 peer-focus-visible:ring-ring motion-reduce:transition-none", + checked + ? "border-primary ring-1 ring-primary" + : "border-border group-hover:border-foreground/30" + )} + > + {Art ? <Art /> : null} + {checked ? ( + <span className="absolute top-1.5 right-1.5 flex size-5 items-center justify-center rounded-full bg-primary text-primary-foreground"> + <CheckIcon className="size-3" strokeWidth={2.5} /> + </span> + ) : null} + </span> + <span className="truncate text-sm">{label}</span> + </label> + ) +} diff --git a/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx index 8e93100fc7..37607582f9 100644 --- a/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx +++ b/surfsense_local/frontend/src/features/studio/format-options/picture-type-picker.tsx @@ -1,15 +1,11 @@ import type { ComponentType } from "react" -import { - CheckIcon, - ImageTypeIcon, - InfographicTypeIcon, -} from "@/components/ui/icons" +import { ImageTypeIcon, InfographicTypeIcon } from "@/components/ui/icons" import { intl } from "@/i18n/intl" -import { cn } from "@/lib/utils" import type { StudioFormat } from "../api" import { formatLabel } from "../studio-formats" +import { PictureCard } from "./picture-card" const TYPE_ART: Record<string, ComponentType<{ className?: string }>> = { image: ImageTypeIcon, @@ -39,47 +35,18 @@ export function PictureTypePicker({ aria-label={label} className="grid grid-cols-2 gap-2" > - {types.map((entry) => { - const Art = TYPE_ART[entry.key] - const checked = value === entry.key - return ( - <label - key={entry.key} - className={cn( - "group relative flex min-w-0 flex-col gap-1.5", - entry.available - ? "cursor-pointer" - : "cursor-not-allowed opacity-50" - )} - > - <input - type="radio" - name="studio-picture-type" - value={entry.key} - checked={checked} - disabled={!entry.available} - className="peer sr-only" - onChange={() => onChange(entry.key)} - /> - <span - className={cn( - "relative flex aspect-3/2 items-center justify-center overflow-hidden rounded-lg border transition-colors duration-150 ease-out peer-focus-visible:ring-2 peer-focus-visible:ring-ring motion-reduce:transition-none", - checked - ? "border-primary ring-1 ring-primary" - : "border-border group-hover:border-foreground/30" - )} - > - {Art ? <Art /> : null} - {checked ? ( - <span className="absolute top-1.5 right-1.5 flex size-5 items-center justify-center rounded-full bg-primary text-primary-foreground"> - <CheckIcon className="size-3" strokeWidth={2.5} /> - </span> - ) : null} - </span> - <span className="truncate text-sm">{formatLabel(entry)}</span> - </label> - ) - })} + {types.map((entry) => ( + <PictureCard + key={entry.key} + name="studio-picture-type" + value={entry.key} + label={formatLabel(entry)} + Art={TYPE_ART[entry.key]} + checked={value === entry.key} + disabled={!entry.available} + onChange={() => onChange(entry.key)} + /> + ))} </div> </div> ) diff --git a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx index aa136673a9..f07f8aeaf0 100644 --- a/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx +++ b/surfsense_local/frontend/src/features/studio/studio-panel.test.tsx @@ -387,15 +387,22 @@ describe("studio panel", () => { await user.click(await screen.findByRole("button", { name: "Comic" })) const dialog = screen.getByRole("dialog", { name: "Comic" }) - const kind = within(dialog).getByLabelText("Kind") as HTMLSelectElement + // Every kind is a card shown by its own picture, a Western comic first. + const kind = within(dialog).getByRole("radiogroup", { name: "Kind" }) + expect(within(kind).getAllByRole("radio")).toHaveLength(10) + expect(kind.querySelectorAll("[data-slot=picture-type-art]")).toHaveLength( + 10 + ) + const western = within(kind).getByRole("radio", { name: "Western comic" }) + expect((western as HTMLInputElement).checked).toBe(true) const pages = within(dialog).getByLabelText("Pages") as HTMLSelectElement - expect([kind.value, pages.value]).toEqual(["western", "4"]) + expect(pages.value).toBe("4") expect(within(dialog).getByText(/Draws 4 pictures/)).toBeTruthy() // A webtoon counts beats, and starts at its own length. - await user.selectOptions(kind, "webtoon") + await user.click(within(kind).getByRole("radio", { name: "Webtoon" })) expect(pages.value).toBe("8") - await user.selectOptions(kind, "manga") + await user.click(within(kind).getByRole("radio", { name: "Manga" })) await user.selectOptions(pages, "6") expect(within(dialog).getByText(/Draws 6 pictures/)).toBeTruthy() await user.click(within(dialog).getByRole("button", { name: /Generate/ })) diff --git a/surfsense_local/frontend/translations/de.json b/surfsense_local/frontend/translations/de.json index d8b302f9e2..becd05a3d1 100644 --- a/surfsense_local/frontend/translations/de.json +++ b/surfsense_local/frontend/translations/de.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "Manga", "studio_comic_kind_manhua_label": "Manhua", "studio_comic_kind_picture_book_label": "Bilderbuch", + "studio_comic_kind_scroll_ahead_aria": "Weitere Arten zeigen", + "studio_comic_kind_scroll_back_aria": "Frühere Arten zeigen", "studio_comic_kind_storyboard_label": "Storyboard", "studio_comic_kind_webtoon_label": "Webtoon", "studio_comic_kind_western_label": "Westlicher Comic", diff --git a/surfsense_local/frontend/translations/en.json b/surfsense_local/frontend/translations/en.json index a71f5e4823..03143ab211 100644 --- a/surfsense_local/frontend/translations/en.json +++ b/surfsense_local/frontend/translations/en.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "Manga", "studio_comic_kind_manhua_label": "Manhua", "studio_comic_kind_picture_book_label": "Picture book", + "studio_comic_kind_scroll_ahead_aria": "Show more kinds", + "studio_comic_kind_scroll_back_aria": "Show earlier kinds", "studio_comic_kind_storyboard_label": "Storyboard", "studio_comic_kind_webtoon_label": "Webtoon", "studio_comic_kind_western_label": "Western comic", diff --git a/surfsense_local/frontend/translations/es.json b/surfsense_local/frontend/translations/es.json index 27cda535db..f37fbdf2a7 100644 --- a/surfsense_local/frontend/translations/es.json +++ b/surfsense_local/frontend/translations/es.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "Manga", "studio_comic_kind_manhua_label": "Manhua", "studio_comic_kind_picture_book_label": "Libro ilustrado", + "studio_comic_kind_scroll_ahead_aria": "Mostrar más tipos", + "studio_comic_kind_scroll_back_aria": "Mostrar tipos anteriores", "studio_comic_kind_storyboard_label": "Guion gráfico", "studio_comic_kind_webtoon_label": "Webtoon", "studio_comic_kind_western_label": "Cómic occidental", diff --git a/surfsense_local/frontend/translations/fr.json b/surfsense_local/frontend/translations/fr.json index 8cd46d9448..faabef039e 100644 --- a/surfsense_local/frontend/translations/fr.json +++ b/surfsense_local/frontend/translations/fr.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "Manga", "studio_comic_kind_manhua_label": "Manhua", "studio_comic_kind_picture_book_label": "Album illustré", + "studio_comic_kind_scroll_ahead_aria": "Afficher plus de genres", + "studio_comic_kind_scroll_back_aria": "Afficher les genres précédents", "studio_comic_kind_storyboard_label": "Storyboard", "studio_comic_kind_webtoon_label": "Webtoon", "studio_comic_kind_western_label": "Comics", diff --git a/surfsense_local/frontend/translations/hi.json b/surfsense_local/frontend/translations/hi.json index 7232b1832c..80fdb509c7 100644 --- a/surfsense_local/frontend/translations/hi.json +++ b/surfsense_local/frontend/translations/hi.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "मांगा", "studio_comic_kind_manhua_label": "मनहुआ", "studio_comic_kind_picture_book_label": "चित्र पुस्तक", + "studio_comic_kind_scroll_ahead_aria": "और प्रकार दिखाएं", + "studio_comic_kind_scroll_back_aria": "पिछले प्रकार दिखाएं", "studio_comic_kind_storyboard_label": "स्टोरीबोर्ड", "studio_comic_kind_webtoon_label": "वेबटून", "studio_comic_kind_western_label": "पश्चिमी कॉमिक", diff --git a/surfsense_local/frontend/translations/ja.json b/surfsense_local/frontend/translations/ja.json index 375706c536..880a3cd5bd 100644 --- a/surfsense_local/frontend/translations/ja.json +++ b/surfsense_local/frontend/translations/ja.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "マンガ", "studio_comic_kind_manhua_label": "マンファ(中国の漫画)", "studio_comic_kind_picture_book_label": "絵本", + "studio_comic_kind_scroll_ahead_aria": "他の種類を表示", + "studio_comic_kind_scroll_back_aria": "前の種類を表示", "studio_comic_kind_storyboard_label": "絵コンテ", "studio_comic_kind_webtoon_label": "ウェブトゥーン", "studio_comic_kind_western_label": "西洋コミック", diff --git a/surfsense_local/frontend/translations/ko.json b/surfsense_local/frontend/translations/ko.json index f2c0882972..205a8104d6 100644 --- a/surfsense_local/frontend/translations/ko.json +++ b/surfsense_local/frontend/translations/ko.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "일본 만화", "studio_comic_kind_manhua_label": "중국 만화", "studio_comic_kind_picture_book_label": "그림책", + "studio_comic_kind_scroll_ahead_aria": "더 많은 종류 보기", + "studio_comic_kind_scroll_back_aria": "이전 종류 보기", "studio_comic_kind_storyboard_label": "스토리보드", "studio_comic_kind_webtoon_label": "웹툰", "studio_comic_kind_western_label": "서양 만화", diff --git a/surfsense_local/frontend/translations/pt-BR.json b/surfsense_local/frontend/translations/pt-BR.json index dd37c87240..dd4960937e 100644 --- a/surfsense_local/frontend/translations/pt-BR.json +++ b/surfsense_local/frontend/translations/pt-BR.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "Mangá", "studio_comic_kind_manhua_label": "Manhua", "studio_comic_kind_picture_book_label": "Livro ilustrado", + "studio_comic_kind_scroll_ahead_aria": "Mostrar mais tipos", + "studio_comic_kind_scroll_back_aria": "Mostrar tipos anteriores", "studio_comic_kind_storyboard_label": "Storyboard", "studio_comic_kind_webtoon_label": "Webtoon", "studio_comic_kind_western_label": "HQ ocidental", diff --git a/surfsense_local/frontend/translations/ru.json b/surfsense_local/frontend/translations/ru.json index 24979dbe15..6ac10892de 100644 --- a/surfsense_local/frontend/translations/ru.json +++ b/surfsense_local/frontend/translations/ru.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "Манга", "studio_comic_kind_manhua_label": "Маньхуа", "studio_comic_kind_picture_book_label": "Книжка с картинками", + "studio_comic_kind_scroll_ahead_aria": "Показать больше видов", + "studio_comic_kind_scroll_back_aria": "Показать предыдущие виды", "studio_comic_kind_storyboard_label": "Раскадровка", "studio_comic_kind_webtoon_label": "Вебтун", "studio_comic_kind_western_label": "Западный комикс", diff --git a/surfsense_local/frontend/translations/zh-CN.json b/surfsense_local/frontend/translations/zh-CN.json index 1e0ac935ad..8ce5fa13a6 100644 --- a/surfsense_local/frontend/translations/zh-CN.json +++ b/surfsense_local/frontend/translations/zh-CN.json @@ -1903,6 +1903,8 @@ "studio_comic_kind_manga_label": "日本漫画", "studio_comic_kind_manhua_label": "国漫", "studio_comic_kind_picture_book_label": "绘本", + "studio_comic_kind_scroll_ahead_aria": "显示更多类型", + "studio_comic_kind_scroll_back_aria": "显示前面的类型", "studio_comic_kind_storyboard_label": "分镜", "studio_comic_kind_webtoon_label": "条漫", "studio_comic_kind_western_label": "欧美漫画", From c73526d8d97844c2624056bed8175948865037cd Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 23:21:00 +0530 Subject: [PATCH 25/33] feat(local): the Comic card shows the canvas icon --- surfsense_local/frontend/src/components/ui/icons.tsx | 4 ++-- .../frontend/src/features/studio/studio-formats.ts | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/surfsense_local/frontend/src/components/ui/icons.tsx b/surfsense_local/frontend/src/components/ui/icons.tsx index 33eb80f361..2cf931de01 100644 --- a/surfsense_local/frontend/src/components/ui/icons.tsx +++ b/surfsense_local/frontend/src/components/ui/icons.tsx @@ -16,7 +16,6 @@ import { ChartScatterIcon as ChartScatterIconData, AudioWaveformIcon as AudioWaveformIconData, BanIcon as BanIconData, - BookImageIcon as BookImageIconData, BotIcon as BotIconData, Bug01Icon as Bug01IconData, CancelCircleHalfDotIcon as CancelCircleHalfDotIconData, @@ -29,6 +28,7 @@ import { CheckmarkBadge02Icon as CheckmarkBadge02IconData, CheckmarkCircle02Icon, ChevronDownIcon as ChevronDownIconData, + CanvasIcon as CanvasIconData, ChevronLeftIcon as ChevronLeftIconData, ChevronRightIcon as ChevronRightIconData, Clock01Icon as Clock01IconData, @@ -156,7 +156,6 @@ export const ArrowRightIcon = createIcon(ArrowRightIconData) export const ArrowRight02Icon = createIcon(ArrowRight02IconData) export const ArrowUp02Icon = createIcon(ArrowUp02IconData) export const BanIcon = createIcon(BanIconData) -export const BookImageIcon = createIcon(BookImageIconData) export const BotIcon = createIcon(BotIconData) export const BugIcon = createIcon(Bug01IconData) export const CancelCircleHalfDotIcon = createIcon(CancelCircleHalfDotIconData) @@ -169,6 +168,7 @@ export const CheckCircle2Icon = createIcon(CheckmarkCircle02Icon) export const CheckIcon = createIcon(CheckIconData) export const CheckmarkBadgeIcon = createIcon(CheckmarkBadge02IconData) export const ChevronDownIcon = createIcon(ChevronDownIconData) +export const CanvasIcon = createIcon(CanvasIconData) export const ChevronLeftIcon = createIcon(ChevronLeftIconData) export const ChevronRightIcon = createIcon(ChevronRightIconData) export const ClockIcon = createIcon(Clock01IconData) diff --git a/surfsense_local/frontend/src/features/studio/studio-formats.ts b/surfsense_local/frontend/src/features/studio/studio-formats.ts index 925dc044f9..8680444bf5 100644 --- a/surfsense_local/frontend/src/features/studio/studio-formats.ts +++ b/surfsense_local/frontend/src/features/studio/studio-formats.ts @@ -2,7 +2,7 @@ import type { ComponentType } from "react" import { AiSearchLinesIcon, - BookImageIcon, + CanvasIcon, Cards01Icon, ChartHistogramIcon, File02Icon, @@ -38,7 +38,7 @@ export const FORMAT_ICONS: Record< podcast: PodcastIcon, image: Image01Icon, infographic: ChartHistogramIcon, - comic: BookImageIcon, + comic: CanvasIcon, } // The backend's catalog sends English labels; a known key shows its own. From 0a0aa6a3308fe90927a1389c3a2dc9f10298c4d2 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 11 Nov 2026 23:48:33 +0530 Subject: [PATCH 26/33] fix(local): a refused image request counts even when its body is cut off OpenRouter has no /images/edits and sometimes hangs up partway through the body of its 404. Both image clients read the whole reply before its status, so that 404 failed a comic's later pages instead of sending them to /images. A refusal is now decided by its status alone; only a success must arrive whole. --- docs/architecture/studio.md | 2 +- .../llm/providers/openai_compatible/image.py | 13 +----- .../providers/openai_compatible/image_edit.py | 11 +---- .../openai_compatible/image_response.py | 13 ++++++ .../llm/providers/test_remote_image_edit.py | 44 +++++++++++++++++++ .../tests/unit/llm/test_openai_compatible.py | 40 +++++++++++++++++ 6 files changed, 102 insertions(+), 21 deletions(-) diff --git a/docs/architecture/studio.md b/docs/architecture/studio.md index d34b55d2f9..329e3be0bf 100644 --- a/docs/architecture/studio.md +++ b/docs/architecture/studio.md @@ -330,7 +330,7 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i - A drawn `image` or `infographic` resolves the `image_gen` selection, and an edit, or a drawing with references, the model that takes a picture in ([below](#which-model-takes-a-picture)). A remote selection loads its connection, which runs the egress check and decrypts the key. An `sdcpp` selection points the same client at the bundled sd-server's loopback URL, with no key and no egress decision, which is what makes an image possible on an offline machine. - sd-server runs only while Studio needs it, because it keeps a model's weights in memory until it exits. The API answers Electron's poll from the jobs ([`local_image_demand.py`](../../surfsense_local/backend/modules/artifacts/local_image_demand.py)): the chosen model of the image type the oldest processing job needs, read from its recipe, so an edit needs `image_edit` though its format is still `image` or `infographic`; else, for 5 minutes after the last such job ended, the same one, so a second image does not reload it; else nothing, and Electron stops it. A cancel ends the window at once, since sd-server cannot stop a generation, and so does any other Studio job running, since a podcast voices in that memory and a text job loads the chat model into it. The job first gives up the text runtime through the API, held for the whole image, because the two share the graphics card and its writing is done by then: the API lets running replies finish, unloads the text models and keeps local text waiting until the image is drawn, and unloads the speech recognition model, starting no recording's window meanwhile ([admission](local-models/admission.md#the-image-jobs-hold)). Then it waits up to 2 minutes for `/sdapi/v1/sd-models` to name its weights before it posts, and fails with a reason if sd-server never comes up. - sd-server is launched with each file of the build on its flag and the model's reviewed `image` defaults as its own (`-W`, `-H`, `--steps`, `--cfg-scale`, `--sampling-method`, `--flow-shift`), since a request that carries none of them starts from the launch settings ([`local-models/catalog.md`](local-models/catalog.md)). -- The client posts `{model, prompt}`, and `size` when a shape is asked, to `/images/generations`, falls back once to `/images` only on `404` or `405`, and remembers the route that worked for the life of the process. It accepts `b64_json`, a data URL or a URL, caps the sizes, and checks the bytes against the claimed MIME type ([`connections.md`](connections.md)). +- The client posts `{model, prompt}`, and `size` when a shape is asked, to `/images/generations`, falls back once to `/images` only on `404` or `405`, and remembers the route that worked for the life of the process. A refusal is decided by its status alone, so one whose body is cut off still counts: OpenRouter has hung up partway through the body of a `404`, which failed a comic's pages before ([`image_response.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py)). Only a success must arrive whole. It accepts `b64_json`, a data URL or a URL, caps the sizes, and checks the bytes against the claimed MIME type ([`connections.md`](connections.md)). - The `text_gen` model writes first in both formats, except a picture the agent asked for ([below](#pictures-the-agent-asks-for)). The markdown body is the image prompt or the brief, so an image can be found by what it shows. - A drawn picture is v1 of its own document: the job records `version` and `recipe: {"kind": "draft"}` as a Word draft's, so its edits group under it. Regenerate on it still draws from the sources and replaces v1 in place. - Every drawing, edit and redraw keeps the exact prompt it sent as `drawing_prompt`, so a later redraw starts from it ([`redraw/`](../../surfsense_local/backend/worker/studio/media/visual/redraw/)). diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py index e353fa95eb..0062f478fb 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py @@ -4,11 +4,10 @@ from modules.llm.connections.key_headers import key_headers from modules.llm.providers.openai_compatible.image_response import ( - MAX_RESPONSE_BYTES, PICTURE_ITSELF, NonRetryableImageError, - bounded_body, read_image, + reply_payload, ) from modules.llm.providers.picture_shape import SIZE, Shape from modules.llm.providers.protocols import GeneratedImage @@ -117,12 +116,4 @@ async def _post( }, ) as reply, ): - content = await bounded_body(reply, MAX_RESPONSE_BYTES) - try: - payload = json.loads(content) - except json.JSONDecodeError: - if reply.status_code >= 400: - payload = {} - else: - raise - return reply.status_code, payload + return reply.status_code, await reply_payload(reply) diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py index 8cb9559fa3..4cb753c73b 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py @@ -9,11 +9,10 @@ from modules.llm.connections.key_headers import key_headers from modules.llm.providers.openai_compatible.image_response import ( - MAX_RESPONSE_BYTES, PICTURE_ITSELF, NonRetryableImageError, - bounded_body, read_image, + reply_payload, ) from modules.llm.providers.picture_shape import ASPECT_RATIO, SIZE, Shape from modules.llm.providers.protocols import GeneratedImage @@ -114,13 +113,7 @@ async def _post( files=_edits_files(images), ) async with request as reply: - content = await bounded_body(reply, MAX_RESPONSE_BYTES) - try: - return reply.status_code, json.loads(content) - except json.JSONDecodeError: - if reply.status_code >= 400: - return reply.status_code, {} - raise + return reply.status_code, await reply_payload(reply) def _edits_fields( diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py index c001b53d28..99a39d28b0 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py @@ -2,6 +2,7 @@ import base64 import binascii +import json from urllib.parse import urlsplit import httpx @@ -42,6 +43,18 @@ async def read_image(payload: object) -> GeneratedImage: return await download_image(url) +async def reply_payload(reply: httpx.Response) -> object: + """The reply's JSON. A refusal is decided by its status, so one whose body is + cut off, or is not JSON, still answers with its status: OpenRouter has hung up + partway through a 404's body. Only a success must arrive whole.""" + if reply.status_code < 400: + return json.loads(await bounded_body(reply, MAX_RESPONSE_BYTES)) + try: + return json.loads(await bounded_body(reply, MAX_RESPONSE_BYTES)) + except (httpx.TransportError, ValueError): + return {} + + async def bounded_body(reply: httpx.Response, limit: int) -> bytes: length = reply.headers.get("content-length") if length is not None and length.isdigit() and int(length) > limit: diff --git a/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py b/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py index df1fe1d6bc..867f5f2ea3 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py +++ b/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py @@ -178,3 +178,47 @@ async def test_a_server_error_on_the_edit_route_is_not_tried_elsewhere() -> None with pytest.raises(NonRetryableImageError, match="502"): await _editor(httpx.Response(502, json={}), asked).edit("m", "x", (BASE,)) assert len(asked) == 1 + + +class _CutOff(httpx.AsyncByteStream): + """A body the server stops sending partway, as OpenRouter did after a 404.""" + + async def __aiter__(self): + yield b'{"error": {"message": "Not Fou' + raise httpx.RemoteProtocolError( + "peer closed connection without sending complete message body " + "(incomplete chunked read)" + ) + + +async def test_a_missing_edit_route_cut_off_mid_reply_still_sends_references() -> None: + """Its 404 says there is no such route before any work; the body adds nothing, + so a server that hangs up while sending it is no reason to fail the page.""" + _routes.clear() + asked: list[httpx.Request] = [] + + def answer(request: httpx.Request) -> httpx.Response: + asked.append(request) + if request.url.path.endswith("/images/edits"): + return httpx.Response(404, stream=_CutOff()) + return httpx.Response(200, json=DONE) + + editor = OpenAICompatibleImageEditor( + "https://cutoff.test/v1", "sk-test", transport=httpx.MockTransport(answer) + ) + image = await editor.edit("google/gemini-2.5-flash-image", "page two", (PAGE_ONE,)) + + assert image.content == EDITED + assert [r.url.path for r in asked] == ["/v1/images/edits", "/v1/images"] + + +async def test_a_picture_cut_off_mid_reply_still_fails() -> None: + """A success must arrive whole: half a picture is no picture.""" + _routes.clear() + asked: list[httpx.Request] = [] + + with pytest.raises(NonRetryableImageError, match="incomplete chunked read"): + await _editor(httpx.Response(200, stream=_CutOff()), asked).edit( + "m", "x", (BASE,) + ) + assert len(asked) == 1 diff --git a/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py b/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py index 87d7f71343..44a4903f45 100644 --- a/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py +++ b/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py @@ -1,5 +1,6 @@ import base64 +import httpx import pytest from modules.llm.connections.service import normalize_base_url, parse_models @@ -332,3 +333,42 @@ async def post(_route, _model, _prompt, _picture, size=None): await provider.generate("gpt-image-1", "a picture") assert sizes == ["1536x1024", None] + + +class _CutOff(httpx.AsyncByteStream): + """A body the server stops sending partway, as OpenRouter did after a 404.""" + + async def __aiter__(self): + yield b'{"error": {"message": "Not Fou' + raise httpx.RemoteProtocolError( + "peer closed connection without sending complete message body " + "(incomplete chunked read)" + ) + + +async def test_a_missing_route_cut_off_mid_reply_still_falls_back( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The 404 is the answer; a server that hangs up while sending its body has + still said the route is missing, so the picture goes to the other route.""" + _route_cache.clear() + asked: list[str] = [] + + def answer(request: httpx.Request) -> httpx.Response: + asked.append(request.url.path) + if request.url.path.endswith("/images/generations"): + return httpx.Response(404, stream=_CutOff()) + return httpx.Response(200, json=PAYLOAD) + + real_client = httpx.AsyncClient + + def client(**kwargs: object) -> httpx.AsyncClient: + return real_client(transport=httpx.MockTransport(answer), **kwargs) + + monkeypatch.setattr( + "modules.llm.providers.openai_compatible.image.httpx.AsyncClient", client + ) + image = await _provider(5).generate("image", "draw") + + assert image.content == PNG + assert asked == ["/v1/images/generations", "/v1/images"] From 29aeb1e4a4de65ad4fd280ab25646fa06bf586b8 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Thu, 12 Nov 2026 01:01:48 +0530 Subject: [PATCH 27/33] fix(local): a Studio job lets the API write while it reads the model's window The job held SQLite's write lock while asking the API for the window, and the API needs that lock to answer, so every lookup timed out as database is locked and failed the API's other requests with it. --- .../tests/integration/worker/test_studio.py | 41 +++++++++++++++++++ surfsense_local/backend/worker/studio/job.py | 2 + 2 files changed, 43 insertions(+) diff --git a/surfsense_local/backend/tests/integration/worker/test_studio.py b/surfsense_local/backend/tests/integration/worker/test_studio.py index 50c54d23e7..f7ed807be6 100644 --- a/surfsense_local/backend/tests/integration/worker/test_studio.py +++ b/surfsense_local/backend/tests/integration/worker/test_studio.py @@ -707,6 +707,47 @@ def api_keeps_writing() -> None: assert all(a.document.status is DocumentStatus.READY for a in artifacts) +def test_the_api_can_write_while_the_job_reads_the_models_window( + session: Session, engine: Engine, stub_model: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """The window is read over HTTP from the API, which writes in the same file.""" + import sqlite3 + + api_could_write: list[bool] = [] + + class Generator: + async def context_tokens(self, _model: str) -> int: + api = sqlite3.connect(str(engine.url.database), timeout=0.1) + try: + api.execute("BEGIN IMMEDIATE") + api.rollback() + api_could_write.append(True) + except sqlite3.OperationalError: + api_could_write.append(False) + finally: + api.close() + return 32_000 + + selection = type( + "Selection", + (), + {"provider": "fake", "name": "fake", "connection": None, "tier": Tier.COMPACT}, + )() + + def resolve(job_session: Session) -> ResolvedGeneration: + # The real lookup reads the selection, which opens the job's transaction. + job_session.execute(text("SELECT 1")) + return ResolvedGeneration(selection, Generator()) + + monkeypatch.setattr("worker.studio.job.resolve_routed_generation", resolve) + _capture_model(monkeypatch, SUMMARY) + artifact = make_artifact(session) + + run(artifact.id) + + assert api_could_write == [True] + + def test_a_generation_failure_leaves_a_reason( session: Session, stub_model: None, monkeypatch: pytest.MonkeyPatch ) -> None: diff --git a/surfsense_local/backend/worker/studio/job.py b/surfsense_local/backend/worker/studio/job.py index 2c5d1dd96d..22ae2ac4fe 100644 --- a/surfsense_local/backend/worker/studio/job.py +++ b/surfsense_local/backend/worker/studio/job.py @@ -254,6 +254,8 @@ def _draft(session: Session, artifact: Artifact, document: Document) -> Built: writer = cast( ResolvedGeneration, models[fmt.requires_model_types.index(ModelType.TEXT_GEN)] ) + # The window is asked of the API, which needs the write lock to answer. + session.commit() window = _window(writer) # The prompt is what to search for, where the format reads passages. query = prompt if fmt.grounding is Grounding.PASSAGES else None From 697043222e84d78cd89d3fe1e1d980cffaf05c8a Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Thu, 12 Nov 2026 01:01:48 +0530 Subject: [PATCH 28/33] fix(local): a comic plan is asked for in its JSON shape, and a refusal says so The plan request carries a schema, a reply that is not a plan is asked for once more before any page is drawn, and a model that declines fails the job with its own words instead of a JSON parse error. --- docs/architecture/studio.md | 4 +- .../tests/unit/worker/test_comic_pipeline.py | 75 ++++++++++++++++++- .../studio/media/visual/comic/asking.py | 59 +++++++++++++++ .../studio/media/visual/comic/pipeline.py | 16 ++-- .../studio/media/visual/comic/schema.py | 33 ++++++++ 5 files changed, 175 insertions(+), 12 deletions(-) create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/asking.py create mode 100644 surfsense_local/backend/worker/studio/media/visual/comic/schema.py diff --git a/docs/architecture/studio.md b/docs/architecture/studio.md index 329e3be0bf..811967ba44 100644 --- a/docs/architecture/studio.md +++ b/docs/architecture/studio.md @@ -65,7 +65,7 @@ Eight formats follow it: - **podcast**: the model outlines the episode from the brief, then drafts it segment by segment. Every segment sends the same one-line system prompt and the sources, and its whole draft prompt after them, because llama-server reuses a prompt only up to its first changed token. Measured through the drafting code on Qwen3 1.7B with about 6,200 tokens of sources: with the draft prompt first, every segment re-read the sources; after them, segments 2 to 4 read about 230 tokens each. The draft prompt goes after them whole: split, with what the segments share left ahead of the sources, the same model wrote half the dialogue, in long lines. Each reply is capped at 12 tokens per word of the segment's target, and never under a planned 250-word segment's worth. Uncapped, Qwen3 1.7B looped on a 225-word segment until its 40,960-token window was full, and the JSON retry, which replays the failed reply, could not fit. The target is the outline's own guess: Qwen3 1.7B once gave a segment 20 words, wrote past them, and a 240-token cap ended both replies mid-JSON. The chosen audio model voices every line through audio.cpp's server, and the transcript is the body. - **image**: the model writes a title and an image prompt, and the image model paints the prompt. - **infographic**: the model writes a factual brief (a title, a summary and up to 8 sections of label, value and detail), and the image model paints a prompt built from it in a fixed sketchnote style. -- **comic**: options `{kind, pages}`, one of ten kinds ([`kinds.py`](../../surfsense_local/backend/modules/artifacts/comic/kinds.py)) and 1 to 12 pictures, checked when the job is created. The model plans a title, a style line, a character sheet and each page's panels and lines; the plan is cut to the count asked for, since each picture is billed or takes minutes. Page 1 is drawn alone, the rest with page 1 as their reference by [the model that takes a picture in](#which-model-takes-a-picture), side by side on a remote model and one at a time on this computer; with none, each is drawn from its words and the version keeps `drawn_from_words`. A remote model letters each page's lines in bubbles; a local one draws no words, its lettering being unmeasured, and every page's lines are set in a strip under it, as a picture book's and a storyboard's always are. The pages are bound by [`comic_binding/`](../../surfsense_local/backend/worker/studio/media/visual/comic_binding/), which the agent's comics share; the body is every page's panels and lines. A failed comic is never retried by the worker, since every page would be drawn again. No page is kept on its own, so a Studio comic has no Edit: Regenerate draws it again. +- **comic**: options `{kind, pages}`, one of ten kinds ([`kinds.py`](../../surfsense_local/backend/modules/artifacts/comic/kinds.py)) and 1 to 12 pictures, checked when the job is created. The model plans a title, a style line, a character sheet and each page's panels and lines, asked for in [`schema.py`](../../surfsense_local/backend/worker/studio/media/visual/comic/schema.py)'s shape; the plan is cut to the count asked for, since each picture is billed or takes minutes. A reply that is not a plan is asked for once more with the reply replayed ([`asking.py`](../../surfsense_local/backend/worker/studio/media/visual/comic/asking.py)), which costs no picture. A second reply with no JSON in it is the model declining, often over the sources themselves, so the job fails with the model's first line rather than a parse error. Page 1 is drawn alone, the rest with page 1 as their reference by [the model that takes a picture in](#which-model-takes-a-picture), side by side on a remote model and one at a time on this computer; with none, each is drawn from its words and the version keeps `drawn_from_words`. A remote model letters each page's lines in bubbles; a local one draws no words, its lettering being unmeasured, and every page's lines are set in a strip under it, as a picture book's and a storyboard's always are. The pages are bound by [`comic_binding/`](../../surfsense_local/backend/worker/studio/media/visual/comic_binding/), which the agent's comics share; the body is every page's panels and lines. A failed comic is never retried by the worker, since every page would be drawn again. No page is kept on its own, so a Studio comic has no Edit: Regenerate draws it again. Two formats break it. PPTX and XLSX go through `office/`: the tier prompt asks the model to "write one standalone Python script" for the format's library (python-pptx or xlsxwriter), with a `SKILL.md` of authoring guidance beside each format, to the runner's contract: start with a `# title:` comment and save the file at `OUTPUT_PATH`. [`office/pipeline.py`](../../surfsense_local/backend/worker/studio/office/pipeline.py) runs the reply through [the runner](#script-documents), in its own process, never with `exec()` in the worker's. A failing script goes back to the model with its error and the traceback's end, for up to three attempts; one longer than 48,000 characters ran on rather than broke on a line, so it is not shown back but asked for again, shorter, since replayed it overflowed the window ([`script_attempts.py`](../../surfsense_local/backend/worker/studio/office/script_attempts.py)). On a local model each draft stops at 16,384 tokens: a deck's script measured about 3,000, and a 1.7B that looped wrote 130,171 characters for 272 s first. Its failure reason does not start `Script error: `, as for [Word and PDF](#word-and-pdf). The title is the `# title:` comment, else the prompt, the sources or the format; the body is the file's own text, read as a script document's is. No spec is kept: a deck or workbook Studio drafts has no versions and no Refine. @@ -252,7 +252,7 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i ## Jobs - `studio_job` runs on the `studio` Huey queue in `huey.db` with `retries=1`. Every path that makes a version commits its row and then queues its job through one helper ([`queued_job.py`](../../surfsense_local/backend/modules/artifacts/queued_job.py)), since the worker is another process that must find the row. A job that cannot be queued fails its version with "This could not be queued. Retry it." rather than leaving it `pending` with no job to end it, which would refuse every Retry and edit. `worker-studio` drains it with four threads, because the work mostly waits on a model ([ADR 0008](../adr/0008-two-job-queues.md)). -- A job ([`job.py`](../../surfsense_local/backend/worker/studio/job.py)) marks the document `processing` unless it was cancelled, resolves one model per required model type, gathers the sources to the text model's window, and commits before rendering, so no write lock is held across a generation that can take minutes. It checks for a cancel before and after rendering. +- A job ([`job.py`](../../surfsense_local/backend/worker/studio/job.py)) marks the document `processing` unless it was cancelled, resolves one model per required model type, commits, asks the API for the text model's window, gathers the sources to it, and commits again before rendering, so no write lock is held across a generation that can take minutes. The first commit is because the API needs the write lock to answer: held, every window lookup waited out the busy timeout, failed as `database is locked`, and failed the API's other requests with it. It checks for a cancel before and after rendering. - [`persist.py`](../../surfsense_local/backend/worker/studio/shared/persist.py) chunks and embeds the body with the ingest code, with no transaction open, then sets the document's title and markdown and indexes that body, so the artifact is searchable and citable. If the format has a file, it clears the artifact's folder and file rows and writes the file named by its role, recording its size and SHA-256. No pipeline writes a `preview` yet. - The document is created without a `dedup_key`, so it is never deduplicated against another document. - A failure rolls back and writes `failed` with a reason cut to 500 characters: the error's first line, or for an HTTP error its whole message after "The model could not be reached: ". A model that refused a request past its window was reached, so that refusal (llama-server's `exceed_context_size_error`, or the `context_length_exceeded` OpenAI and servers copying it send) reads, for a draft, as the [window budget](#the-window-budget)'s "don't fit" sentence, and for a refine or a job run, "The request was longer than the model's context window. Choose a model with a larger context window." The job is then re-raised for Huey's one retry, except an image error, since the endpoint may already have generated, and billed, an image, a podcast refused for memory, since a retry would draft the episode again and refuse again, a request past the model's window or sources it cannot hold, since the same request is refused the same way, any failure of a script document ([above](#script-documents)) of a converted PDF ([above](#converted-pdfs)) or of a revised copy ([above](#revised-copies)), and any failure of a refine ([Refine](#refine)) or an image edit ([editing a picture](#editing-a-picture)), which Retry asks again. diff --git a/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py b/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py index 4a8545bd00..6729cdead4 100644 --- a/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py +++ b/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py @@ -107,7 +107,7 @@ def asked(monkeypatch: pytest.MonkeyPatch) -> list[str]: """The chat model answers with PLAN; what it was asked is recorded.""" seen: list[str] = [] - def writer(_model: object, system: str, _sources: object) -> str: + def writer(_model: object, system: str, _sources: object, **_asked: object) -> str: seen.append(system) return json.dumps(PLAN) @@ -237,6 +237,79 @@ def test_a_plan_longer_than_asked_is_cut_to_the_page_count(asked: list[str]) -> assert len(PdfReader(io.BytesIO(built.primary)).pages) == 2 +REFUSAL = ( + "I can't turn this document into a comic.\n\n" + "It glorifies real violence, and a dramatised version would spread it." +) + + +def _replying(monkeypatch: pytest.MonkeyPatch, *replies: str) -> list[dict]: + """The chat model answers `replies` in turn; each call's keywords are recorded.""" + calls: list[dict] = [] + + def writer(_model: object, _system: str, _sources: object, **asked: object) -> str: + calls.append(asked) + return replies[len(calls) - 1] + + monkeypatch.setattr("worker.studio.shared.generate.run_model", writer) + return calls + + +def _render_three(artist: Artist) -> object: + return comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 3}, + keeper=_keeper(artist), + ) + + +def test_the_plan_is_asked_for_in_its_json_shape( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """An endpoint that takes a schema cannot answer outside it.""" + calls = _replying(monkeypatch, json.dumps(PLAN)) + + _render_three(Artist()) + + schema = calls[0]["json_schema"] + assert schema["required"] == ["title", "style", "characters", "pages"] + + +def test_a_plan_that_is_not_json_is_asked_for_once_more( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Nothing is drawn yet, so asking again costs one text reply, not pages.""" + calls = _replying( + monkeypatch, "Here is the plan: title The Lighthouse Cat", json.dumps(PLAN) + ) + artist = Artist() + + built = _render_three(artist) + + assert calls[1]["repair"].reply == "Here is the plan: title The Lighthouse Cat" + assert built.title == "The Lighthouse Cat" + + +def test_a_model_that_declines_twice_fails_with_what_it_said( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A refusal reads as one, not as a JSON parse error, and no page is drawn.""" + _replying(monkeypatch, REFUSAL, REFUSAL) + artist = Artist() + + with pytest.raises(ValueError) as failed: + _render_three(artist) + + assert str(failed.value).splitlines()[0] == ( + "The model did not plan the comic. It said: " + "I can't turn this document into a comic." + ) + assert artist.drawn == [] + + def test_a_webtoon_is_one_scroll_of_beats() -> None: """A webtoon's count is its beats, up to the same twelve.""" picked = validate_options(None, {"kind": "webtoon", "pages": 8}) # type: ignore[arg-type] diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/asking.py b/surfsense_local/backend/worker/studio/media/visual/comic/asking.py new file mode 100644 index 0000000000..486eac671d --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/asking.py @@ -0,0 +1,59 @@ +"""Asking the chat model for the comic's plan, once more if its reply is not one.""" + +import logging + +from modules.artifacts.comic.kinds import Kind +from modules.llm.resolution import ResolvedGeneration +from worker.studio.media.visual.comic import plan as planning +from worker.studio.media.visual.comic.schema import REPLY +from worker.studio.shared import generate +from worker.studio.shared.artifact import Source + +logger = logging.getLogger(__name__) + +_JSON_NUDGE = ( + "Your previous reply was not the plan as JSON. Return only the JSON object." +) +# Enough of a reply to tell a refusal from broken JSON in the log. +_LOGGED_CHARS = 500 + + +def ask_for_plan( + writer: ResolvedGeneration, + sources: list[Source], + user_prompt: str | None, + kind: Kind, + count: int, + untitled: str, +) -> planning.Plan: + """Nothing is drawn yet, so a second ask costs one text reply, not pages.""" + system = planning.prompt(writer.tier, user_prompt, kind, count) + reply = generate.run_model(writer, system, sources, json_schema=REPLY) + try: + return planning.parse(reply, untitled, count) + except ValueError as first: + logger.warning( + "studio: comic plan unreadable (%s); asking again. It began: %r", + first, + reply[:_LOGGED_CHARS], + ) + retry = generate.run_model( + writer, + system, + sources, + json_schema=REPLY, + repair=generate.Repair(reply, _JSON_NUDGE), + ) + try: + return planning.parse(retry, untitled, count) + except ValueError as error: + logger.warning("studio: comic plan unreadable again: %r", retry[:_LOGGED_CHARS]) + raise _unplanned(retry, error) from error + + +def _unplanned(reply: str, error: ValueError) -> ValueError: + """A reply with no JSON in it is the model saying no; the user reads its words.""" + said = next((line.strip() for line in reply.splitlines() if line.strip()), "") + if said and "{" not in reply: + return ValueError(f"The model did not plan the comic. It said: {said}") + return ValueError(f"The model's comic plan could not be read: {error}") diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py index 471495b4d4..1373b6c9a3 100644 --- a/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py @@ -7,13 +7,12 @@ from modules.llm.pictures_in.model import PictureModel from modules.llm.providers.sdcpp import provider as sdcpp from modules.llm.resolution import ResolvedGeneration, ResolvedImageGeneration -from worker.studio.media.visual.comic import plan as planning +from worker.studio.media.visual.comic.asking import ask_for_plan from worker.studio.media.visual.comic.draw import draw_pages from worker.studio.media.visual.comic.page_prompt import page_prompt, strip_lines from worker.studio.media.visual.comic.plan import Plan from worker.studio.media.visual.comic_binding.bind import BoundPage, bind from worker.studio.media.visual.drawn.pipeline import DRAWN_FROM_WORDS -from worker.studio.shared import generate from worker.studio.shared.artifact import Built, Source, fallback_title from worker.studio.shared.text import slug @@ -28,14 +27,13 @@ def render( keeper: PictureModel | None, ) -> Built: kind = KINDS[options["kind"]] - plan = planning.parse( - generate.run_model( - writer, - planning.prompt(writer.tier, user_prompt, kind, options["pages"]), - sources, - ), - fallback_title(user_prompt, sources, "Comic"), + plan = ask_for_plan( + writer, + sources, + user_prompt, + kind, options["pages"], + fallback_title(user_prompt, sources, "Comic"), ) if not plan.pages: raise ValueError("the writer planned no pages") diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/schema.py b/surfsense_local/backend/worker/studio/media/visual/comic/schema.py new file mode 100644 index 0000000000..58d9fd06e8 --- /dev/null +++ b/surfsense_local/backend/worker/studio/media/visual/comic/schema.py @@ -0,0 +1,33 @@ +"""The plan reply's shape, enforced as a grammar where the endpoint takes one. + +Only types and `required`, like the deck's: the page count stays the prompt's +to ask for and the plan's to cut. +""" + +_PANEL = { + "type": "object", + "properties": { + "where": {"type": "string"}, + "shows": {"type": "string"}, + "lines": {"type": "array", "items": {"type": "string"}}, + }, + "required": ["where", "shows", "lines"], +} + +REPLY = { + "type": "object", + "properties": { + "title": {"type": "string"}, + "style": {"type": "string"}, + "characters": {"type": "string"}, + "pages": { + "type": "array", + "items": { + "type": "object", + "properties": {"panels": {"type": "array", "items": _PANEL}}, + "required": ["panels"], + }, + }, + }, + "required": ["title", "style", "characters", "pages"], +} From c870cbc3a954f51be870b205c0682937a1c4037f Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Thu, 12 Nov 2026 01:04:37 +0530 Subject: [PATCH 29/33] docs: comics and picture edits shipped, so their proposal goes What is still open moves to the Known gaps of studio, agent and model selection: the sd-server shape spike, the images guide's edit-model walkthrough, the Settings edit override and local badge, and the per-turn picture limit. --- docs/architecture/agent.md | 1 + docs/architecture/local-models/selection.md | 1 + docs/architecture/studio.md | 3 +- docs/proposals/pictures/01-image-models.md | 79 ----------- docs/proposals/pictures/02-make-image-tool.md | 46 ------ docs/proposals/pictures/03-comics.md | 59 -------- docs/proposals/pictures/04-waiting-render.md | 32 ----- docs/proposals/pictures/05-in-the-app.md | 38 ----- docs/proposals/pictures/06-comic-artifact.md | 26 ---- docs/proposals/pictures/README.md | 131 ------------------ 10 files changed, 4 insertions(+), 412 deletions(-) delete mode 100644 docs/proposals/pictures/01-image-models.md delete mode 100644 docs/proposals/pictures/02-make-image-tool.md delete mode 100644 docs/proposals/pictures/03-comics.md delete mode 100644 docs/proposals/pictures/04-waiting-render.md delete mode 100644 docs/proposals/pictures/05-in-the-app.md delete mode 100644 docs/proposals/pictures/06-comic-artifact.md delete mode 100644 docs/proposals/pictures/README.md diff --git a/docs/architecture/agent.md b/docs/architecture/agent.md index dc5d22f882..fd82c947ed 100644 --- a/docs/architecture/agent.md +++ b/docs/architecture/agent.md @@ -379,6 +379,7 @@ A document script is another matter: it runs in the Studio worker's runner, whic - An analysis waits on the Studio worker's four threads. Behind four long Studio jobs it is dropped after 60 seconds unstarted, and the call waits out its 190 seconds before saying so. - A local chat model's step waiting out a whole local picture through the agent's model endpoint is not measured. A render that waits on a picture finishes after its turn, so no one looks at its pages. - A waiting render that fails does not count toward renders' three-failure stop. +- Nothing on the server limits pictures per turn; only the comics skill's twelve pages do. - A Studio job the agent starts with `surfsense_create_artifact` runs on the same local model as the agent, so the agent's next step waits behind it. - An artifact the agent starts with `surfsense_create_artifact` records neither the thread nor the step that made it, and one it renders records its thread but not the step: the tool call reaches SurfSense without the step. - A thread's folder narrows what its agent can reach to its own sources, but does not close it. Through a hole in opencode, below, an agent can still read what is outside its folder: opencode's tool-output folder is shared by all, and closing it needs a change in opencode. Through SurfSense's own tools it can too: a script `surfsense_render_document` or `surfsense_analyze_data` runs can read any file the user can ([Studio](studio.md#known-gaps)), and the text the render returns shows what it put in the document, another thread's sources included; and `surfsense_read_document` reads any document the agent rendered in the workspace, from any thread and filed in an unticked folder too, and such a script can hold what that thread's agent took from its own sources. diff --git a/docs/architecture/local-models/selection.md b/docs/architecture/local-models/selection.md index 16225b7bdc..6ce168fdc2 100644 --- a/docs/architecture/local-models/selection.md +++ b/docs/architecture/local-models/selection.md @@ -345,5 +345,6 @@ language, and `test_connections.py` that no connection fills it. ## Known gaps +- Settings shows the edit override as its own section, "Use a different model for edits", not folded under Image gen; and the **Image editing** badge is on server models only, not on local models to download. - Only the quiz and flashcards pass `json_schema`: mind map, HTML, image, infographic and the podcast's outline and draft still ask for JSON in the prompt alone, so their format compliance depends on it. - Nothing measures whether three tiers are still needed; once constrained decoding carries format compliance, a tier would carry reasoning depth only, which plausibly collapses three tiers to two. diff --git a/docs/architecture/studio.md b/docs/architecture/studio.md index 811967ba44..847384b6fb 100644 --- a/docs/architecture/studio.md +++ b/docs/architecture/studio.md @@ -386,7 +386,8 @@ Its pages are never listed: a picture drawn with `comic_page`, every later versi - A page the agent drew and never bound is kept unlisted, until its chat's comic binds it; nothing deletes it. - An image edit is measured only on klein Q4_0 through the pinned sd-server on a processor, with no graphics card: a photo's layout held, and an infographic's printed lettering held through a colour change and a changed title, 657 to 962 seconds an edit, about twice a draw. Lettering klein drew itself, a sketchnote's, is not measured, and neither is any time on a graphics card or a remote edit model. - An edit changes the whole picture by instruction; there is no mask or selected region, which sd.cpp and `/images/edits` both take, since drawing one needs its own interface. -- References are measured only on OpenRouter's `/images`, on six models drawing one four-page comic: each held its characters from page 1. OpenAI's `/images/edits` with several pictures, and local references through sd-server, are not measured; nor is a remote server that takes one picture on `/images/edits` and refuses several. +- References are measured only on OpenRouter's `/images`, on six models drawing one four-page comic: each held its characters from page 1. OpenAI's `/images/edits` with several pictures, and local references and shapes through sd-server, are not measured; nor is a remote server that takes one picture on `/images/edits` and refuses several. +- The [images guide](../../www/content/docs/v2/guides/images.mdx)'s walkthrough and screenshots still choose a separate model for editing, from before the image model edits. - A remote model's taking pictures is read when it is chosen; a provider that later starts or stops listing it is not seen until the model is chosen again. - The [window budget](#the-window-budget) prices a draft's first call. What only some calls add is not priced: a written web page's design guide (6,235 characters, about 2,100 tokens past the prompt it reserves), the failed reply a repair replays, and a podcast segment's recap. On a small window these can leave the reply short or be refused, which fails the job with the context sentence and no retry. Reasoning is not reserved either: Studio turns thinking off, which the local runtime honours and some remote endpoints do not, whose windows are large. - The budget prices text with admission's estimate, not the model's tokenizer, so scripts it prices at a token a character, such as Cyrillic or Devanagari, are cut further than they need. A long figure list leaves a Word or PDF draft less of its sources' text, since every figure they hold is listed. diff --git a/docs/proposals/pictures/01-image-models.md b/docs/proposals/pictures/01-image-models.md deleted file mode 100644 index 304405b4cc..0000000000 --- a/docs/proposals/pictures/01-image-models.md +++ /dev/null @@ -1,79 +0,0 @@ -# Image models - -> Each image model says whether it can take a picture in. One rule picks the model for any job that sends a picture: the user's edit model if they chose one, else the image model if it can, else none, and then the job redraws. One interface draws, edits and keeps references, and each provider's client knows its own routes. - -## What a model can do - -Each image model carries `takes_pictures`: whether it draws from a picture as well as from words. - -- **Remote:** from what the connection's `/models` publishes, read live. Discovery already reads each entry's `output_modalities` to find image models ([`service.py`](../../../surfsense_local/backend/modules/llm/connections/service.py)); it reads `input_modalities` beside it, from the entry or its `architecture`, and `DiscoveredModel` gains `takes_pictures`. OpenRouter lists `text,image` for every model the spike drew with. The bundled manifest's `modalities.input` answers when the endpoint publishes nothing; it lacks most OpenRouter image models, so it is the fallback, not the source. -- **Local:** the sd.cpp catalog's `edit` type, which FLUX.2 klein has today ([selection](../../architecture/local-models/selection.md)). -- **Kept with the choice.** When the user chooses an image model, its `takes_pictures` is kept in the selection's existing `settings` JSON, and refreshed whenever that connection's model list is read again. No migration: the column is there ([`models.py`](../../../surfsense_local/backend/modules/llm/models.py)). -- **Unknown is no.** A model whose endpoint says nothing and that the manifest does not carry gets `takes_pictures: false`, so its edits redraw. Sending a picture to a model that ignores it would bill a picture that keeps nothing. - -## Which model handles a picture - -One function, `picture_model(session)`, beside the others in [`resolution.py`](../../../surfsense_local/backend/modules/llm/resolution.py), answers every job that sends a picture in, an edit or a comic page with references: - -1. the `image_edit` selection, when the user chose one: the override; -2. else the `image_gen` selection, when it `takes_pictures`; -3. else none. - -The API exposes the answer once, as `pictures_in: "edit_model" | "image_model" | null`, beside the selections it is worked out from. Studio, the agent's tool list and the skill all read that, so no two places can disagree. - -## One interface - -The two protocols, `ImageGenerator` and `ImageEditor` ([`protocols.py`](../../../surfsense_local/backend/modules/llm/providers/protocols.py)), become one: - -```python -class ImageModel(Protocol): - async def draw( - self, - model: str, - prompt: str, - *, - shape: Shape | None = None, - references: Sequence[GeneratedImage] = (), - ) -> GeneratedImage: ... -``` - -An edit is a drawing with the picture as its one reference and the instruction as its prompt. `Shape` is `square`, `portrait` or `landscape`; left out, the model's default, as today. - -## One client per provider - -| Provider | No references | With references | -|---|---|---| -| **OpenRouter** (`catalog_provider` `openrouter`) | `POST /images/generations` with `size` | `POST /images` with `input_references` and `aspect_ratio` | -| **Any other OpenAI-compatible server** | `POST /images/generations` with `size`, then `/images` as today's route cache does | `POST /images/edits`, multipart, one `image[]` part per reference, and `size` | -| **Local sd-server** | native `POST /sdcpp/v1/img_gen` with `width` and `height` | the same, with `ref_images` holding every reference | - -- **Sizes.** `square` is `1024x1024` or `1:1`, `portrait` `1024x1536` or `2:3`, `landscape` `1536x1024` or `3:2`. Locally, the launch's default pixel count at that proportion, each side a multiple of 64. -- **OpenRouter gets its own client** because it answers 404 on `/images/edits` and its pure image models refuse `/chat/completions` ([the spike](README.md#the-spike)). The client is chosen by the connection's `catalog_provider`, which every connection already records. -- **Local drawing moves to the native route** only when a shape is asked; a drawing with no shape keeps today's `/v1/images/generations`. The native route is the one the edit client already uses, because sd-server's OpenAI and A1111 edit routes re-noise the picture ([ADR 0077](../../adr/0077-an-image-edit-is-the-next-version.md)). -- **Retries, the same for every client.** A `400` to an optional field (`size`, `aspect_ratio`, `response_format`) is sent once more without it, remembered per connection, as `response_format` already is: a `400` comes before any drawing. A timeout or a `5xx` is never retried, since the provider may still be drawing and a second request bills a second picture. -- **Several references refused.** A server that answers `400` to several `image[]` parts gets the first alone, once, and the result says only one reached the model. The spike below decides whether that is kept. - -## Edits and redraws - -An edit of a picture, from Studio's Edit box or from the agent, stays the next version of the same document, with the recipe ADR 0077 gives it. What runs depends on the rule: - -- **A model takes pictures:** the edit draws with the base version's picture as its one reference and the instruction as its prompt, at the base's proportions. The same as today, through the rule instead of the slot. -- **None does: a redraw.** The image model draws again from the prompt that drew the base and the instruction: `"<prompt>\n\nChange: <instruction>"`, at the base's proportions, with no chat model in between. The recipe records `"redrawn": true`, and the version's body starts with one line, "Drawn again with your change." It is billed as one picture, as an edit is. -- **The prompt a redraw starts from** is the one the base was drawn with. A picture the agent drew keeps it as `prompt` already. A Studio draft keeps it only in its body under the title ([`image/pipeline.py`](../../../surfsense_local/backend/worker/studio/media/visual/image/pipeline.py)); from now on every drawing also keeps it as `drawing_prompt` in its metadata, and a redraw of an older picture reads it from the body. A redraw of a redraw starts from the redrawn version's own prompt, so changes add up. -- **The `needs_image_edit` refusal goes.** No edit is refused for want of a model; only for want of an image model at all, which is `needs_image` as today. - -## Spikes before code - -Each runs in the scratchpad against the real endpoint or binary, and its result is posted on the implementation PR. - -- **OpenAI's `/images/edits` with two and four `image[]` parts and `size=1024x1536`,** on gpt-image. Decides the one-reference fallback. -- **Local references:** the pinned sd-server with FLUX.2 klein, `ref_images` of one and two pictures at a portrait `width` and `height`: whether a face holds over four pages, and the seconds a page on a processor. -- **Local shape:** the native route's drawing at a portrait size against today's route's square one, for time and quality. -- **`input_modalities` across the providers the catalog connects to,** to see which publish it and how often the manifest must answer instead. - -## Checks - -- Discovery: an OpenRouter-shaped `/models` entry with `input_modalities` `text,image` gives `takes_pictures`; one with `text` alone does not; one with neither falls to the manifest, then to no. -- The rule: each of its three answers, and `pictures_in` in the API. -- Clients: the request each sends with and without a shape and references, the `400` retries, no retry on a timeout or a `5xx`, against a stand-in server. -- Edits: a real edit through the image model; a redraw through a model that takes no pictures, its recipe and its first line; a redraw of a Studio draft made before `drawing_prompt`; no `needs_image_edit` anywhere. diff --git a/docs/proposals/pictures/02-make-image-tool.md b/docs/proposals/pictures/02-make-image-tool.md deleted file mode 100644 index 6398c28f58..0000000000 --- a/docs/proposals/pictures/02-make-image-tool.md +++ /dev/null @@ -1,46 +0,0 @@ -# The agent's picture tool - -> `surfsense_make_image` gains `shape` and `references`, its description stops steering away from pictures with words or from several pictures, and an edit or a reference that no model can take is drawn from words instead of refused. - -Everything else in [`make_image.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/make_image.py) stays: the wait, the three-failure limit, the `image-<id>` names, placing, and the one-title-per-chat refusal. - -## New arguments - -| Argument | Type | Meaning | -|---|---|---| -| `shape` | `square`, `portrait` or `landscape` | The picture's proportions. Left out, the model's default. | -| `references` | up to 4 picture names | Pictures this chat made, each ready, for the new picture to keep: the same characters and style. Only for a new picture, not with `artifact_id`. | - -- **References are checked as placing is** ([`names.py`](../../../surfsense_local/backend/modules/artifacts/drawn_images/names.py)): a name of this chat, ready, a still picture. A picture still being drawn is refused with "Draw it first and wait for it", and none of these refusals is a failed run. Four bounds what one call uploads; the local spike may lower it. -- **A picture with references runs on the model [the rule](01-image-models.md#which-model-handles-a-picture) picks.** When it picks none, the picture is drawn from its prompt alone and the result says so: "Drawn from words alone: the image model cannot take a picture in, so the characters may differ from the reference." Nothing is refused, so the agent never loops on a setting it cannot change. -- **An edit with `artifact_id`** runs as Studio's does ([edits and redraws](01-image-models.md#edits-and-redraws)): a real edit, or a redraw that the result names as one. The tool's refusal for want of an edit model, and its text sending the user to Settings › Image edit, go. -- **The recipe** of a new picture records `shape` and `references` as artifact ids: `{"kind": "image_drawn", "prompt", "shape", "references"}`. Regenerate repeats it with the same references. - -## The listing stays fixed - -Tool listings are fixed and in one order, so a local model's prompt cache holds from turn to turn ([`offered_tools.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/offered_tools.py)). So `references` is always listed, and what happens without a model is in the result, not in the listing. - -## The description - -Rewritten, as short as it can be: - -- "Only when the user asks for one" stays, and so does "Draw no data chart". -- "Confirm before making more than one" becomes "When a request needs more than one picture, confirm how many once, before the first: each is billed, or takes minutes on their computer." -- "Words that must be exact belong in the document around the picture" becomes "Lettering may come out misspelt: keep words in a picture short, and put text that must be exact in the document around it." -- One line: "For a comic, a storyboard or any set of pictures, load the `surfsense-pictures` skill first." - -## The result - -Adds, to what it says today: - -- **Where it was drawn:** "Drawn on this computer" or "Drawn by <connection name>". The skill draws wordless pages on this computer ([comics](03-comics.md#lettering)). -- **How it was drawn,** when it was not as asked: from words alone, a redraw, or a shape or reference the model did not take. - -## Checks - -[`test_make_image.py`](../../../surfsense_local/backend/tests/integration/agent/test_make_image.py), against the stand-in painter: - -- `shape` reaches the model; references reach it in order; a reference from another chat, a picture still drawing, a fifth reference, and references with `artifact_id` are refused, unbilled. -- With no model that takes pictures, references draw from words and the result says so; an edit redraws and the result says so. -- The result names where the picture was drawn. -- The listing is the same, byte for byte, whatever models are chosen. diff --git a/docs/proposals/pictures/03-comics.md b/docs/proposals/pictures/03-comics.md deleted file mode 100644 index 534ffdaf87..0000000000 --- a/docs/proposals/pictures/03-comics.md +++ /dev/null @@ -1,59 +0,0 @@ -# Comics - -> A comic, a storyboard or a picture book loads one skill, `surfsense-pictures`. The agent plans the pages, confirms the count once, draws page 1, draws the rest with page 1 as their reference, and binds every page into one PDF. The PDF is the comic. - -## Routing - -- **The prompt.** The picture line in [`agent.md`](../../../surfsense_local/backend/modules/agent/prompts/agent.md) becomes: "For a picture, an infographic, a comic, a storyboard, a picture book or any set of pictures, load the `surfsense-pictures` skill, then use `surfsense_make_image`." -- **The web page skill** ([`SKILL.md`](../../../surfsense_local/backend/modules/agent/skills/surfsense-html/SKILL.md)) adds to its description: "Not for a comic, a storyboard or a picture: those are drawn pictures." A page that holds a finished comic's pictures stays possible when the user asks for a web page by name. -- **Both engines** load skills the same way, so opencode threads and Claude plan threads get it alike. - -## The skill - -`surfsense-pictures` is the craft beside the tool's contract, as `surfsense-documents` is beside `surfsense_render_document`. It sits with the other skills ([`skills/`](../../../surfsense_local/backend/modules/agent/skills/)): - -| File | Holds | -|---|---| -| `SKILL.md` | One picture or a set; writing a prompt; lettering; the series steps below; what to say when a model cannot take pictures | -| `comics.md` | Pages and panels, bubbles and captions, the comic's PDF | -| `pdf.md` | The binding script, to copy and fill | - -A storyboard or a picture book later is one more file beside `comics.md`, with no new skill and no prompt change. - -## The steps - -1. **Plan.** A title, the page count, and for each page its panels, what each shows and its lines. A **character sheet**: each character's face, hair, build, clothes and colours, in a few fixed words. A **style line**: medium, palette, line weight, panel borders. Four pages when the user names no number; at most twelve, since each is billed or takes minutes. -2. **Confirm once.** One message: the page count, the plan in short, and what it costs: billed once a page by their provider, or minutes a page on this computer. When the image model cannot take pictures in, one more plain line: "Each page is drawn from the description, so the characters may differ a little between pages." On yes, every page is drawn without asking again. A one-page comic or strip needs no confirmation, as a single picture needs none. -3. **Page 1 alone,** `shape: portrait`. Its prompt is the style line, the character sheet and page 1's panels, laid out as "A comic page of N panels: panel 1, top left: …". Look at it; when it is wrong, edit it before going on, since every later page copies its characters. -4. **The rest, with page 1 as their reference.** Each is its own call, with `references: ["image-<page 1>"]`, the same style line and character sheet, and its panels, all in one step so a remote model draws them side by side. A character picture the user gave, or a character sheet they asked for first, is the reference instead. -5. **Look at every page.** Redraw or edit one only when it is plainly wrong, and say so; a misspelt word is not a reason ([below](#lettering)). -6. **Bind the PDF** ([below](#the-pdf)). -7. **Answer** with the PDF and one line per page, so the user can ask for "page 3 again". -8. **A later change.** Edit the page with its `artifact_id`, read the PDF's script with `surfsense_read_document`, and render its next version with the new name in place of the old. To add a page, draw it with page 1 as reference and render again. - -Each page is its own picture, titled "<comic title> · page <n>", so the one-title-per-chat refusal never meets two pages of one comic. - -## Lettering - -- **Bubbles are drawn by the model.** Each holds one short line, at most about eight words, two or three to a panel, written in the prompt in quotes. The best three models in [the spike](README.md#the-spike) spelled every line right. -- **A misspelt page is not redrawn.** The PDF typesets that page's exact lines in a strip under it, so a bad letter never bills a second picture. -- **On this computer, or when the agent cannot see pictures,** every page is drawn without words and every page gets its strip. Lettering a local model draws itself is not measured ([studio, Known gaps](../../architecture/studio.md#known-gaps)), and a page the agent cannot see it cannot check. - -## The PDF - -Made with `surfsense_render_document`, `format: pdf`, every page's name in `images`, from the script in `pdf.md`: - -- One picture per page, full bleed, in order, at comic-book width (6.625 in) and the page's own height. A title page only when the user asks for one. -- **Pages placed as JPEG at quality 90**, not PNG: four PNG pages made 10 to 28 MB in the spike. Pillow converts each before ReportLab places it; a page is drawn once, so the loss is once. -- A page with a lettering strip is a little taller, the strip below the picture, the lines in the PDF's own font. -- A page still being drawn is named all the same ([the render that waits](04-waiting-render.md)). - -## Which models to recommend - -From [the spike](README.md#the-spike): `gpt-image-2.6`, Gemini's image models and Seedream letter well and hold characters, and the user docs name them for comics. FLUX.2 klein, remote or local, draws fast and holds characters but garbles lettering, so its pages get strips. The app recommends nothing in the moment: the user's chosen model is used as it is. - -## Checks - -- An agent test, with the stand-in model, as [`test_agent_makes_image.py`](../../../surfsense_local/backend/tests/integration/agent/test_agent_makes_image.py) does for one picture: "make a comic" loads `surfsense-pictures`, draws page 1, then the rest with `references`, then renders one PDF naming every page. -- `pdf.md`'s script, run as the render runs it, on four stand-in pages: four PDF pages, JPEG images, under 5 MB, and a strip page when asked. -- Manual: a four-page comic on a remote model that takes pictures, on one that does not, and on a local model, each read as a PDF, with page 3 changed after. diff --git a/docs/proposals/pictures/04-waiting-render.md b/docs/proposals/pictures/04-waiting-render.md deleted file mode 100644 index 30734195cf..0000000000 --- a/docs/proposals/pictures/04-waiting-render.md +++ /dev/null @@ -1,32 +0,0 @@ -# The render that waits - -> A document that names a picture still being drawn is not refused. It becomes a pending version that the last of its pictures' jobs queues, so a comic drawn on a slow local model still ends as a PDF without the user asking again. - -Amends [ADR 0087](../../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)'s "A document needing a picture still being drawn waits for the user's next message". - -## Today - -A local picture takes minutes, and an edit 657 to 962 seconds on a processor ([studio](../../architecture/studio.md#editing-a-picture)). `surfsense_make_image` answers `Queued` after 150 seconds, and a render naming that picture is refused while it is drawing ([`script_documents/service.py`](../../../surfsense_local/backend/modules/artifacts/script_documents/service.py)), so the agent can only tell the user to ask again. - -## Design - -- **The render is accepted.** `surfsense_render_document` with a picture name still being drawn creates the pending version as today, with `waits_for: [<artifact ids>]` in its metadata, and does not queue its job. The call answers at once: "Queued artifact <id>, version <n>: it renders once image-12 and image-14 are drawn. Do not render it again." The agent tells the user the PDF will appear in Studio. -- **A picture's job ending queues what waits on it.** At the end of every Studio job, where the worker already notifies the app ([`job.py`](../../../surfsense_local/backend/worker/studio/job.py)), it finds the pending renders in that picture's thread whose `waits_for` names it. Each one whose pictures have all ended is queued then, through `queue_job` as every version is ([`queued_job.py`](../../../surfsense_local/backend/modules/artifacts/queued_job.py)). -- **A picture that failed or was cancelled** fails the waiting render with a sentence naming it: "image-14 failed, so this PDF was not made. Draw that page again, or render without it." That failure counts under renders' three-failure limit when the next render is asked for, as a queued render's late failure does today ([`failed_renders.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/failed_renders.py)). -- **Only drawn pictures wait.** A source's figure or a chat's chart is there or not, so a missing one is refused as today. A web page saved with `surfsense_save_page` still refuses a picture being drawn, since a page is checked and screenshotted as it is saved. -- **A restart** settles waiting renders when the worker starts, beside the interrupted documents ([`interrupted_documents.py`](../../../surfsense_local/backend/worker/interrupted_documents.py)): one whose pictures have all ended is queued, one still waiting stays pending, and one waiting on a picture that was itself interrupted fails with the same sentence. -- **Cancel** in Studio works as on any pending version. -- **No one checks the previews** of a render that finishes after its turn. The comic's PDF is a fixed layout, one full-bleed picture a page, so little can go wrong, and the open thread links the PDF once it is ready, as it does for any queued version ([agent](../../architecture/agent.md#pictures-the-agent-makes)). - -## Decisions - -- **On the server, not in the agent.** A tool that polls, or a turn kept open, holds a call past its limit or a chat open for an hour; ADR 0087 rules out both. A pending version that the last picture's job queues needs no new process, no polling and no model awake. -- **In the job's end, not a timer.** The job that ends is the one event that can make a render ready, so nothing runs in between. - -## Checks - -- A render naming a picture still drawing answers `Queued` and is not queued; it renders once the picture is ready. -- One of two pictures ending leaves it waiting; the second queues it. -- A failed or a cancelled picture fails it, with the sentence, and the failure counts toward the next render. -- A worker restart queues it, keeps it, or fails it, as above. -- A web page naming a picture still drawing is still refused. diff --git a/docs/proposals/pictures/05-in-the-app.md b/docs/proposals/pictures/05-in-the-app.md deleted file mode 100644 index 3f73c641a5..0000000000 --- a/docs/proposals/pictures/05-in-the-app.md +++ /dev/null @@ -1,38 +0,0 @@ -# In the app - -> Edit works on every picture. Settings marks the image models that can edit, with a small label and a three-word tooltip. Nothing in Studio or chat sends the user to Settings for pictures, and a model that cannot do something is described in a neutral line, never a warning. - -## Studio's Edit - -- **The Edit box shows over every ready picture**, as it does over a document's Refine ([`artifact-panel.tsx`](../../../surfsense_local/frontend/src/features/studio/artifact-panel.tsx)). It no longer checks the `image_edit` slot ([`edit-model.ts`](../../../surfsense_local/frontend/src/features/studio/edit-model.ts)): every picture can be edited or redrawn ([edits and redraws](../pictures/01-image-models.md#edits-and-redraws)). -- **Its placeholder** says what will happen: "Describe a change" when the model edits, "Describe a change: the picture is drawn again" when it redraws. The box reads `pictures_in` from the API to choose. -- **A redrawn version** shows its body's first line, "Drawn again with your change.", in the viewer's normal text. -- **`needs_image_edit` goes**, with its message "Needs an image editing model." and its link to Settings › Image editing models. Only an image picture with no image model at all says "Needs an image model.", as today. - -## Settings - -- **The badge.** In Settings › Image models, each model that takes pictures in carries the **Image editing** type badge beside **Image generation**, in the list of a server's models and in the list of local models to download ([`server-models.tsx`](../../../surfsense_local/frontend/src/features/models/remote/models/server-models.tsx), [`download-image-models.tsx`](../../../surfsense_local/frontend/src/features/models/local/image/download-image-models.tsx)). It is information only: choosing a model without it works. -- **Image editing models** ([`image-edit-models-settings.tsx`](../../../surfsense_local/frontend/src/features/settings/models/image-edit-models-settings.tsx)) becomes an optional override, collapsed under the image models, titled "Use a different model for edits". Its summary reads "Same as the image model" when nothing is chosen there, in place of "No image editing model chosen" ([`in-use-summary.tsx`](../../../surfsense_local/frontend/src/features/models/your-models/in-use-summary.tsx)). - -## Words - -Every new or changed string goes through the `translate` skill, into every language the app ships, short enough not to overflow: - -| Where | Text | -|---|---| -| Edit placeholder, a model that edits | Describe a change | -| Edit placeholder, a redraw | Describe a change: the picture is drawn again | -| A redrawn version | Drawn again with your change. | -| The override's title | Use a different model for edits | -| The override's summary, unset | Same as the image model | - -## The user docs - -[`guides/images.mdx`](../../../www/content/docs/v2/guides/images.mdx) gains making a comic and what Edit does on each kind of model; [`studio.mdx`](../../../www/content/docs/v2/studio.mdx) the redraw; the image model pages under `models/` the label and the optional edit model; and the models that letter well for comics are named there. - -## Checks - -- The Edit box over a picture with each of `pictures_in`'s three answers: its placeholder, and that it sends. -- No `needs_image_edit` and no link to the edit settings anywhere in Studio. -- The label shows only on models that take pictures in; its tooltip opens on hover and on focus. -- The override collapsed and summarised as "Same as the image model" when unset. diff --git a/docs/proposals/pictures/06-comic-artifact.md b/docs/proposals/pictures/06-comic-artifact.md deleted file mode 100644 index dfd8b90148..0000000000 --- a/docs/proposals/pictures/06-comic-artifact.md +++ /dev/null @@ -1,26 +0,0 @@ -# The comic artifact - -> A comic is its own Studio format, `comic`: one artifact whose file is the bound PDF. Studio makes one from sources in its dialog, and the agent binds the pages it drew into one with `surfsense_bind_comic`. Both bind with the same code, so the agent no longer copies a script to fill. Image and infographic stay one Image card in the dialog, the infographic a type of image. - -## Why - -A comic was a PDF the agent rendered from a script in `pdf.md` ([03](03-comics.md)), listed in Studio as a PDF beside its pages, and Studio could not make one. The agent copied a hundred-line ReportLab script for every comic, which a small model can break. - -## Studio - -- **Format** `comic`, needing `image_gen` and `text_gen`, with options `{kind, pages}`: one of the ten kinds in `comics.md`, and 1 to 12 pictures, each kind starting at its own length. Checked when the job is created ([`kinds.py`](../../../surfsense_local/backend/modules/artifacts/comic/kinds.py)). -- **Plan, draw, bind** ([`media/visual/comic/`](../../../surfsense_local/backend/worker/studio/media/visual/comic/)): the chat model plans a title, a style line, a character sheet and each page's panels and lines; page 1 is drawn alone and the rest with page 1 as reference; the binder makes the PDF. -- **Lettering.** Studio cannot look at a page. A remote model letters bubbles; a local model's lettering is not measured, so it draws no words and every line goes in the strip under its page. -- **No retry, no Edit.** A retry would bill every page again. One artifact holds one primary file and keeping pages would need a schema change, so a Studio comic is changed by Regenerate. -- **The dialog** gives each format's options their own slot (`format-options/`): the podcast brief, the picture type, and the comic's kind and pages with what they cost. - -## The agent - -- **`surfsense_bind_comic`** takes the pictures' names in reading order, each with lines for its strip, a `layout` and a `direction`, and `artifact_id` for the comic's next version. A page still being drawn makes it wait, as a document does ([04](04-waiting-render.md)). -- **`create_artifact`** does not offer `comic`: the agent draws its pages and looks at each before it binds them. -- **The skill** drops `pdf.md`; `comics.md` gives each kind's `layout` and `direction`. -- **Pages are not listed.** `make_image` takes `comic_page: true` for every page, binding marks any page drawn without it, and a page's edits keep the mark. Studio lists the comic alone; deleting it deletes the pages nothing else uses. - -## Not in this - -- Changing one page of a Studio comic. diff --git a/docs/proposals/pictures/README.md b/docs/proposals/pictures/README.md deleted file mode 100644 index b52b2540ec..0000000000 --- a/docs/proposals/pictures/README.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -status: in-progress -code: - - surfsense_local/backend/modules/llm/providers/ - - surfsense_local/backend/modules/llm/connections/ - - surfsense_local/backend/modules/llm/resolution.py - - surfsense_local/backend/modules/artifacts/drawn_images/ - - surfsense_local/backend/modules/artifacts/image_edits/ - - surfsense_local/backend/modules/artifacts/script_documents/ - - surfsense_local/backend/modules/agent/prompts/agent.md - - surfsense_local/backend/modules/agent/skills/ - - surfsense_local/backend/modules/agent/tool_endpoint/ - - surfsense_local/backend/worker/studio/ - - surfsense_local/frontend/src/features/studio/ - - surfsense_local/frontend/src/features/models/ - - surfsense_local/frontend/src/features/settings/models/ ---- - -# Pictures: one image model, edits for every model, and comics - -> The user chooses one image model. It draws, and when it can take a picture in it also edits and keeps characters the same from page to page. A model that cannot gets its edits redrawn instead, so Edit works for every model and nothing asks for a second model. On that, the agent makes comics: it plans the pages, confirms the count once, draws one picture per page and binds them into one PDF. - -## Today - -Asked for a comic, the agent wrote a web page and drew the panels as SVG, or drew one picture and stopped. Under that sit four limits in how images work: - -- **Nothing makes a comic a picture job.** The prompt sends only "a picture or infographic" to `surfsense_make_image` ([`agent.md`](../../../surfsense_local/backend/modules/agent/prompts/agent.md)); the web page skill's description, "an explainer … a figure that shows how something works", drawn "as inline SVG", is the nearest match; and the tool's own description says "confirm before making more than one" and sends exact words "to the document around the picture" ([`make_image.py`](../../../surfsense_local/backend/modules/agent/tool_endpoint/make_image.py)). -- **Editing needs a second model most users never choose.** Studio's Edit and the agent's edit run only on the `image_edit` slot ([ADR 0077](../../adr/0077-an-image-edit-is-the-next-version.md)). Few local models can fill it, and an empty slot answers "Needs an image editing model" with a link to Settings. The image model the user did choose is never asked, though most remote ones take a picture in. -- **Editing through OpenRouter fails.** A remote edit always posts `/images/edits` ([`image_edit.py`](../../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py)), and OpenRouter has no such route: it answers 404. -- **Every picture is the model's default shape, and drawn from words alone.** No request sends a size or a reference picture, so a comic page comes out square and its characters change from page to page. A local page is also past a tool call's limit, and a document naming a picture still being drawn is refused until the user's next message ([agent, Known gaps](../../architecture/agent.md#known-gaps)). - -## What changes - -| | Today | With this | -|---|---|---| -| Models the user chooses for pictures | an image model, and an edit model for edits | one image model; a different edit model only if they want one | -| Edit, with a model that takes pictures in | needs the edit slot filled | a real edit by the image model | -| Edit, with a model that does not | "Needs an image editing model", and a link to Settings | the picture drawn again from its prompt and the change, said in one neutral line | -| Editing through OpenRouter | 404 | works, through `/images` with `input_references` | -| Settings › Image models | no sign of which models edit | an **Image editing** badge on those that do, beside **Image generation** | -| "Make a comic about …" | a web page with SVG panels, or one picture | a plan, one confirmation, one picture per page, one PDF | -| Characters across pages | redrawn from words each time | page 1 given to every later page as a reference | -| A page's shape | the model's default | `portrait`, `landscape` or `square` | -| A document waiting on a picture still being drawn | refused until the user's next message | a pending version that renders when its pictures are drawn | - -## Parts - -| Part | File | Owns | -|---|---|---| -| **Image models** | [`01-image-models.md`](01-image-models.md) | what a model can do, which model handles a picture, one client per provider, edits and redraws | -| **The agent's picture tool** | [`02-make-image-tool.md`](02-make-image-tool.md) | `shape` and `references` on `surfsense_make_image`, and its description | -| **Comics** | [`03-comics.md`](03-comics.md) | the `surfsense-pictures` skill, the routing, the PDF | -| **The render that waits** | [`04-waiting-render.md`](04-waiting-render.md) | a document that renders once its pictures are drawn | -| **In the app** | [`05-in-the-app.md`](05-in-the-app.md) | Settings' label, Studio's Edit, the words users see | -| **The comic artifact** | [`06-comic-artifact.md`](06-comic-artifact.md) | `comic` as a Studio format, `surfsense_bind_comic`, one binder, Image's infographic type | - -## The spike - -Run on 10 Nov 2026 against OpenRouter with one four-page story, "The Lighthouse Cat": page 1 drawn from words with a portrait size, pages 2 to 4 drawn with page 1 as their reference, the four bound into a PDF. Six models, 24 pictures. The script and its outputs stayed in the scratchpad. - -| Model | Lettering | Characters held | Follows the script | Seconds a page | -|---|---|---|---|---| -| `openai/gpt-image-2.6-flare` | exact | yes | every line and panel | 45–75 | -| `google/gemini-nano-banana-2.1` | exact | yes | adds lines of its own | fast | -| `bytedance-seed/seedream-5-0-pro` | exact | yes | close; adds a page number on some pages | fast | -| `qwen/qwen-image-3-pro` | good | yes | repeats a caption | 130–195, one upstream 504 | -| `black-forest-labs/flux-3-image` | good | mostly | draws two cats in one panel | 65–80 | -| `black-forest-labs/flux-3-image-klein` | garbled ("WENT CUT", labels unreadable) | yes | repeats captions | 30–50 | - -What it settled: - -- **OpenRouter has no `/images/edits`.** It has two kinds of image model. Pure image models (GPT image, FLUX, Qwen) refuse `/chat/completions` with a 404 that names `/api/v1/images`; chat image models (Gemini, Seedream) take it. `POST /images` with `input_references`, a list of `{"type": "image_url", "image_url": {"url": <data URL>}}`, and `aspect_ratio` worked for every pure image model tried, so it is the one route for a picture in. -- **`size` works.** `1024x1536` on `/images/generations` came back portrait from all six; FLUX gave its own 880 x 1184. -- **A reference holds a character on every model tried**, with no edit model involved. -- **Lettering is good enough on the best three** to let the model draw speech bubbles; klein shows why the PDF also needs a typed fallback. -- **A page PDF of PNGs is 10 to 28 MB** for four pages. JPEG pages are needed. -- **A read timeout must not be retried**: the provider may still be drawing, and a second request bills a second picture. - -Not spiked: a local model with references (no image model is installed on the machine the spike ran on), and providers other than OpenRouter. Both are spikes before their part's code ([image models](01-image-models.md#spikes-before-code)). - -## Build order - -On one branch, `feat/comics`, a commit per step, each test first: - -1. Image models: the capability, the rule, the clients ([01](01-image-models.md)). -2. Edits for every model: the edit job on the rule, and redraws ([01](01-image-models.md#edits-and-redraws)). -3. `shape` and `references` on the tool ([02](02-make-image-tool.md)). -4. The `surfsense-pictures` skill and its routing ([03](03-comics.md)). -5. The comic PDF ([03](03-comics.md#the-pdf)). -6. The render that waits ([04](04-waiting-render.md)). -7. Settings' label and Studio's Edit ([05](05-in-the-app.md)). -8. Docs: architecture, ADRs, user docs, and this proposal deleted ([below](#when-it-ships)). - -## Built, and where it differs - -Steps 1 to 8 are built on `feat/comics`; what is now true is in [studio](../../architecture/studio.md#which-model-takes-a-picture), [agent](../../architecture/agent.md#pictures-the-agent-makes), [ADR 0089](../../adr/0089-any-image-model-changes-a-picture.md) and [ADR 0090](../../adr/0090-a-document-waits-on-the-server-for-its-pictures.md). Where the build differs from the parts below: - -- **Two protocols, not one.** `ImageGenerator` draws from words and `ImageEditor` takes pictures, each with `shape`, the editor with several pictures; the rule picks which client a picture goes to, so Studio's pipelines kept their shape. -- **No OpenRouter client.** One client sends pictures to `/images/edits` and, on a `404` or `405` there, to `/images` with `input_references`, remembered per server. Any OpenAI-compatible server whose image models take pictures works, not OpenRouter alone. -- **Taking pictures is the `image_edit` type** a model already gets from the catalog, now also from a connection's own `input_modalities`; no new field. -- **A waiting render's failure does not count** toward renders' three-failure stop: renders never counted late failures, as pictures do. -- **The agent's prompt names the skill in one line**, since it was 4 characters short of its 4,000-character cap; the tool's description and the web page skill's carry the rest. - -## Left to do - -- The spikes in [image models](01-image-models.md#spikes-before-code): several pictures on OpenAI's `/images/edits`, and local references and shapes through sd-server. -- Settings: the edit override as its own section, retitled "Use a different model for edits", not yet folded under Image gen; the **Image editing** badge on server models only, not yet on local models to download. -- The images guide's walkthrough and screenshots, which still choose a model for editing. - -## Decisions - -- **One image model, not two slots.** Whether a model can take a picture in is a fact about the model, so it is read from the model, not asked of the user. The edit slot stays as an override for the few who want another model for edits. -- **Every model can be edited.** A model that cannot take a picture in redraws from the picture's prompt and the change. It is not an edit, and the version says so, but the user never meets a dead end or a setting. -- **The agent's own tool, never opencode's.** opencode's `experimental_image_generate` finishes after the turn ends and needs opencode 2.2 ([ADR 0087](../../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md)). It is not used, now or as a fallback. -- **Comics are pictures bound into a PDF.** Not a web page, not a single picture, not a new Studio format. -- **Nothing in Studio or chat points to Settings** for pictures. A model that cannot do something is described in a neutral line, never as a warning. - -## Not decided here - -- Grouping a comic's pages under its PDF in Studio, so twelve pages are not twelve rows. -- A comic button in Studio, which would need a chat model to write the plan from sources. -- Panels lettered by SurfSense from wordless pictures, with speech bubbles the PDF script draws. -- A server limit on pictures per turn, beyond the skill's twelve pages. - -## When it ships - -- [studio](../../architecture/studio.md): image models and the rule, the clients, edits and redraws, `shape` and `references` in the recipe. -- [agent](../../architecture/agent.md): the skill, the tool's arguments, the render that waits, and the Known gaps on slow pictures and lettering deleted. -- [selection](../../architecture/local-models/selection.md): the edit slot as an override. -- ADRs: one amending [ADR 0077](../../adr/0077-an-image-edit-is-the-next-version.md) (the image model edits; a model that cannot redraws), one amending [ADR 0087](../../adr/0087-the-agent-makes-pictures-with-its-own-surfsense-tool.md) (a document waits on its pictures on the server). -- The user docs: [`guides/images.mdx`](../../../www/content/docs/v2/guides/images.mdx), [`studio.mdx`](../../../www/content/docs/v2/studio.mdx), [`agent.mdx`](../../../www/content/docs/v2/agent.mdx) and the image model pages under `models/`. From 66a09b55634dc09a51ce83009b9683478ae07b8f Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Thu, 12 Nov 2026 13:14:36 +0530 Subject: [PATCH 30/33] fix(local): pictures keep their shape, model and style past the edge cases /images takes a shape as aspect_ratio, a refused edit is resent only without a field it carried, a picture running beside a bound comic keeps sd-server, and an older infographic redraws in Studio's style. --- .../modules/artifacts/local_image_demand.py | 14 +++++---- .../llm/providers/openai_compatible/image.py | 12 ++++++-- .../providers/openai_compatible/image_edit.py | 10 ++++++- .../integration/agent/test_bind_comic.py | 28 ++++++++++++++++- .../llm/providers/test_remote_image_edit.py | 22 ++++++++++++++ .../tests/unit/llm/test_openai_compatible.py | 30 +++++++++++++++++++ .../tests/unit/worker/test_drawing_prompt.py | 28 +++++++++++++++++ .../media/visual/redraw/drawing_prompt.py | 15 ++++++---- 8 files changed, 143 insertions(+), 16 deletions(-) create mode 100644 surfsense_local/backend/tests/unit/worker/test_drawing_prompt.py diff --git a/surfsense_local/backend/modules/artifacts/local_image_demand.py b/surfsense_local/backend/modules/artifacts/local_image_demand.py index 0f1bc3b6a4..cfc5e6b50f 100644 --- a/surfsense_local/backend/modules/artifacts/local_image_demand.py +++ b/surfsense_local/backend/modules/artifacts/local_image_demand.py @@ -36,12 +36,14 @@ def local_image_demand(session: Session, now: datetime) -> ModelType | None: ).join(Document, Artifact.document_id == Document.id) jobs = studio.where(Artifact.format.in_(needs)) running = session.execute( - jobs.where(Document.status == DocumentStatus.PROCESSING) - .order_by(Document.updated_at) - .limit(1) - ).first() - if running is not None: - return _job_type(running, needs) + jobs.where(Document.status == DocumentStatus.PROCESSING).order_by( + Document.updated_at + ) + ).all() + # A bound comic draws nothing, so a picture running beside it still decides. + drawing = (slot for job in running if (slot := _job_type(job, needs))) + if (slot := next(drawing, None)) is not None: + return slot # A podcast voices on the processor and a text job loads the chat model; # either needs the memory more than a second image would. if session.execute( diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py index 0062f478fb..082e748086 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image.py @@ -9,7 +9,7 @@ read_image, reply_payload, ) -from modules.llm.providers.picture_shape import SIZE, Shape +from modules.llm.providers.picture_shape import ASPECT_RATIO, SHAPES, SIZE, Shape from modules.llm.providers.protocols import GeneratedImage __all__ = ["NonRetryableImageError", "OpenAICompatibleImageProvider"] @@ -23,6 +23,8 @@ _asks_for_picture: dict[tuple[int, str], bool] = {} # Endpoints that refuse a size: they draw their own. _refuses_size: set[tuple[int, str]] = set() +# OpenRouter's /images names a shape by its ratio, as it does for references. +_RATIO_OF_SIZE = {SIZE[shape]: ASPECT_RATIO[shape] for shape in SHAPES} class OpenAICompatibleImageProvider: @@ -112,8 +114,14 @@ async def _post( "model": model, "prompt": prompt, **(PICTURE_ITSELF if picture else {}), - **({"size": size} if size else {}), + **(_shape_field(route, size) if size else {}), }, ) as reply, ): return reply.status_code, await reply_payload(reply) + + +def _shape_field(route: str, size: str) -> dict[str, str]: + if route == EXTENSION_ROUTE: + return {"aspect_ratio": _RATIO_OF_SIZE[size]} + return {"size": size} diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py index 4cb753c73b..4c6c25521f 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/image_edit.py @@ -68,7 +68,7 @@ async def edit( ) # Without each optional field in turn if refused: a 400 comes before any work. for field in _OPTIONAL: - if status != 400 or field in refused: + if status != 400 or field not in _sent(route, shape, refused): continue refused = refused | {field} status, payload = await self._post( @@ -116,6 +116,14 @@ async def _post( return reply.status_code, await reply_payload(reply) +def _sent(route: str, shape: Shape | None, refused: frozenset[str]) -> set[str]: + """The optional fields a request carries: dropping one it lacks resends it unchanged.""" + carried = {PICTURE} if route == EDITS else set() + if shape is not None: + carried.add(SHAPE) + return carried - refused + + def _edits_fields( model: str, prompt: str, shape: Shape | None, refused: frozenset[str] ) -> dict[str, str]: diff --git a/surfsense_local/backend/tests/integration/agent/test_bind_comic.py b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py index 44c4baeadd..be16b596eb 100644 --- a/surfsense_local/backend/tests/integration/agent/test_bind_comic.py +++ b/surfsense_local/backend/tests/integration/agent/test_bind_comic.py @@ -4,7 +4,7 @@ import io import threading import time -from datetime import UTC, datetime +from datetime import UTC, datetime, timedelta import pytest from pypdf import PdfReader @@ -13,6 +13,7 @@ from modules.artifacts.local_image_demand import local_image_demand from modules.artifacts.models import Artifact, ArtifactFileRole from modules.documents.models import DocumentStatus +from modules.llm.model_type import ModelType from shared.config import get_storage_settings from shared.db import create_session_factory from shared.search import retrieve @@ -205,6 +206,31 @@ async def test_binding_a_comic_does_not_start_the_local_image_model( assert local_image_demand(session, datetime.now(UTC)) is None +async def test_a_picture_drawn_beside_a_binding_keeps_the_local_image_model( + tools: ToolEndpoint, + painter: Painter, # noqa: F811 + engine: Engine, +) -> None: + """A binding that started first must not stop sd-server under the picture.""" + workspace_id = await tools.workspace() + first = await _picture(tools, workspace_id, "Cat · page 1") + second = await _picture(tools, workspace_id, "Cat · page 2") + made, _ = await tools.call( + workspace_id, "bind_comic", {"title": "Cat", "pages": [{"image": first}]} + ) + with create_session_factory(engine)() as session: + comic = session.get(Artifact, _artifact_id(made)) + picture = session.get(Artifact, _id(second)) + started = datetime.now(UTC) + comic.document.status = DocumentStatus.PROCESSING + comic.document.updated_at = started - timedelta(seconds=30) + picture.document.status = DocumentStatus.PROCESSING + picture.document.updated_at = started + session.commit() + + assert local_image_demand(session, started) is ModelType.IMAGE_GEN + + async def _listed_ids(tools: ToolEndpoint, workspace_id: int) -> set[int]: listed = await tools.client.get(f"/workspaces/{workspace_id}/artifacts") return {row["id"] for row in listed.json()} diff --git a/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py b/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py index 867f5f2ea3..4b7130b501 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py +++ b/surfsense_local/backend/tests/unit/llm/providers/test_remote_image_edit.py @@ -170,6 +170,28 @@ def answer(request: httpx.Request) -> httpx.Response: assert "aspect_ratio" not in json.loads(asked[2].read()) +async def test_a_refusal_on_references_is_not_resent_unchanged() -> None: + """/images is sent no response_format, so dropping it changes nothing to resend.""" + _routes.clear() + _refused.clear() + asked: list[httpx.Request] = [] + + def answer(request: httpx.Request) -> httpx.Response: + asked.append(request) + if request.url.path.endswith("/images/edits"): + return httpx.Response(404, json={}) + return httpx.Response(400, json={}) + + editor = OpenAICompatibleImageEditor( + "https://gateway.test/v1", "sk-test", transport=httpx.MockTransport(answer) + ) + with pytest.raises(NonRetryableImageError, match="400"): + await editor.edit("m", "page two", (PAGE_ONE,), shape="portrait") + + bodies = [r.read() for r in asked[1:]] + assert len(bodies) == len(set(bodies)) == 2 + + async def test_a_server_error_on_the_edit_route_is_not_tried_elsewhere() -> None: """A 5xx may follow billed work: no second route, no second picture.""" _routes.clear() diff --git a/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py b/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py index 44a4903f45..bafc17418c 100644 --- a/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py +++ b/surfsense_local/backend/tests/unit/llm/test_openai_compatible.py @@ -1,4 +1,5 @@ import base64 +import json import httpx import pytest @@ -372,3 +373,32 @@ def client(**kwargs: object) -> httpx.AsyncClient: assert image.content == PNG assert asked == ["/v1/images/generations", "/v1/images"] + + +async def test_a_shape_on_the_images_route_is_an_aspect_ratio( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """OpenRouter's /images takes `aspect_ratio`, not OpenAI's `size`.""" + _route_cache.clear() + _asks_for_picture.clear() + sent: dict[str, dict] = {} + + def answer(request: httpx.Request) -> httpx.Response: + sent[request.url.path] = json.loads(request.read()) + if request.url.path.endswith("/images/generations"): + return httpx.Response(404, json={}) + return httpx.Response(200, json=PAYLOAD) + + real_client = httpx.AsyncClient + + def client(**kwargs: object) -> httpx.AsyncClient: + return real_client(transport=httpx.MockTransport(answer), **kwargs) + + monkeypatch.setattr( + "modules.llm.providers.openai_compatible.image.httpx.AsyncClient", client + ) + await _provider(8).generate("flux", "page one", shape="portrait") + + assert sent["/v1/images/generations"]["size"] == "1024x1536" + assert sent["/v1/images"]["aspect_ratio"] == "2:3" + assert "size" not in sent["/v1/images"] diff --git a/surfsense_local/backend/tests/unit/worker/test_drawing_prompt.py b/surfsense_local/backend/tests/unit/worker/test_drawing_prompt.py new file mode 100644 index 0000000000..5ccf218462 --- /dev/null +++ b/surfsense_local/backend/tests/unit/worker/test_drawing_prompt.py @@ -0,0 +1,28 @@ +"""What a redraw starts from, for pictures made before their prompt was kept.""" + +import pytest + +from modules.artifacts.models import Artifact +from modules.documents.models import Document +from worker.studio.media.visual.infographic.style import in_infographic_style +from worker.studio.media.visual.redraw.drawing_prompt import drawing_prompt_of + +pytestmark = pytest.mark.unit + + +def _older(fmt: str) -> Artifact: + """No `drawing_prompt` and no recipe: only the body Studio wrote under its title.""" + document = Document(title="Saturn", content="# Saturn\n\nRings: 7 main rings") + return Artifact(format=fmt, artifact_metadata={}, document=document) + + +def test_an_older_infographic_is_redrawn_in_studios_style() -> None: + """Its body is the brief alone; the style it was drawn in is not in it.""" + assert drawing_prompt_of(_older("infographic")) == in_infographic_style( + "Rings: 7 main rings" + ) + + +def test_an_older_image_is_redrawn_from_its_body_as_written() -> None: + """An image's body is the prompt it was drawn from.""" + assert drawing_prompt_of(_older("image")) == "Rings: 7 main rings" diff --git a/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py b/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py index 124c7bd674..9056f23b72 100644 --- a/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py +++ b/surfsense_local/backend/worker/studio/media/visual/redraw/drawing_prompt.py @@ -28,12 +28,15 @@ def drawing_prompt_of(artifact: Artifact) -> str: if isinstance(kept, str) and kept.strip(): return kept drawing = image_drawing_of(meta) - if drawing is not None: - # Drawn as the agent wrote it, in Studio's style for an infographic. - if artifact.format == "infographic": - return in_infographic_style(drawing.prompt) - return drawing.prompt - return _below_title(artifact.document.content or "", artifact.document.title) + written = ( + drawing.prompt + if drawing is not None + else _below_title(artifact.document.content or "", artifact.document.title) + ) + # Neither the agent's prompt nor Studio's body carries the infographic's style. + if artifact.format == "infographic": + return in_infographic_style(written) + return written def _below_title(body: str, title: str) -> str: From 2804607f25793c58ef3db9f4549af854a3b3bac9 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Thu, 12 Nov 2026 13:14:36 +0530 Subject: [PATCH 31/33] fix(local): a waiting render settles on cancel, and a failed settle never redraws Cancelling a queued picture settles what waits on it, a settle that fails at a job's end is logged rather than retried, and the queued picture's result says a document can place it now. --- docs/architecture/agent.md | 4 ++-- .../modules/agent/tool_endpoint/make_image.py | 5 +++-- .../backend/modules/artifacts/service.py | 3 +++ .../integration/agent/test_make_image.py | 3 +++ .../integration/agent/test_waiting_render.py | 21 +++++++++++++++++++ .../tests/integration/worker/test_studio.py | 18 ++++++++++++++++ .../backend/worker/waiting_renders.py | 15 +++++++++++-- 7 files changed, 63 insertions(+), 6 deletions(-) diff --git a/docs/architecture/agent.md b/docs/architecture/agent.md index fd82c947ed..422466aa24 100644 --- a/docs/architecture/agent.md +++ b/docs/architecture/agent.md @@ -165,7 +165,7 @@ Deleting a thread aborts and deletes its session, disposes its opencode instance | `surfsense_render_document` | `title`; `format`, `docx`, `pdf`, `pptx` or `xlsx`; `script`; `artifact_id`, to make the next version of, refused for any version of a document Studio made, which Refine rewrites in Studio, and for a revised copy or a converted PDF, which keep no script and have no Refine, naming `surfsense_revise_document` or the Word version or deck to change and convert again ([`unscripted_base.py`](../../surfsense_local/backend/modules/artifacts/script_documents/unscripted_base.py)); `images`, names from `surfsense_list_images`, of ticked sources only; `template_source_id`, a ticked source's own `.docx` for `docx` or `.pptx` for `pptx`, kept by later versions | runs the script as a new version in Studio and waits for it ([below](#documents-the-agent-makes)) | | `surfsense_save_page` | `path`, the page's file under the thread's `outputs/`; `title`; `artifact_id`, to make a page's next version | keeps the page as a version in Studio and waits for it to be indexed ([web pages the agent makes](#web-pages-the-agent-makes)): the `Rendered artifact <id>, version <n>: <title>` line, the page's size and opening text, the outside resources the viewer will block, up to four screenshots, and the working file to edit next | | `surfsense_bind_comic` | `title`; `pages`, each `{image, lines}` in reading order; `layout`, `pages` or `scroll`; `direction`, `ltr` or `rtl`; `artifact_id`, for a comic's next version | a `comic` in Studio, its pages bound into one PDF ([comics the agent binds](studio.md#comics-the-agent-binds)): the `Rendered artifact <id>, version <n>: <title>` line; or `Queued artifact …` while a page is still being drawn | -| `surfsense_make_image` | `prompt`; `title`, needed for a new picture; `type`, `image` or `infographic`; `comic_page`, for a page of a comic, which Studio does not list; `artifact_id`, to edit a picture this chat made | a picture kept in Studio and waited for ([pictures the agent makes](#pictures-the-agent-makes)): the `Rendered artifact <id>, version <n>: <title>` line, its name `image-<id>`, and the picture itself when the model reads images; or `Queued artifact …` | +| `surfsense_make_image` | `prompt`; `title`, needed for a new picture; `type`, `image` or `infographic`; `comic_page`, for a page of a comic, which Studio does not list; `artifact_id`, to edit a picture this chat made | a picture kept in Studio and waited for ([pictures the agent makes](#pictures-the-agent-makes)): the `Rendered artifact <id>, version <n>: <title>` line, its name `image-<id>`, and the picture itself when the model reads images; or `Queued artifact …` with its name, which a document or comic can place at once, waiting for it on the server, and a web page only once it is drawn | | `surfsense_read_document` | `artifact_id`, of any version, or `document_id`, a workbook source; `sheet`, a workbook's sheet; `offset`, an optional 1-based line, or a workbook's row, to start from | the agent's newest version's number (a lineage it continued from a Studio draft before such renders were refused skips Studio's later versions), artifact id, title, format, status and the images it places, and its script in pages of at most 1,000 lines and 40,000 bytes under "Script, lines a-b of n", ending with the offset of the next page, so a result stays under opencode's truncation limits ([`script_page.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/script_page.py)). A document Studio made is refused, as a render continuing one is: "This document was made in Studio: refine it there, or make a new document." Given the `artifact_id` of a revised copy, it reads the copy's newest ready version as it reads now ([`read_revised_copy.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/read_revised_copy.py)): a Word copy's body with every author's tracked changes marked inline as `<ins author="…">` and `<del author="…">`, `[comment N]` after the text each comment is on, then each comment with its author, whether it is internal, and that text ([`word_marked_text.py`](../../surfsense_local/backend/modules/artifacts/revised_copies/engines/word_marked_text.py)), under its counts of tracked changes and comments; a deck's text slide by slide; a workbook's cells as below. Word and deck text come in pages of lines as a script does. Excel and PowerPoint keep no tracked changes, so their opening counts the edits applied over the copy's versions instead. Every answer ends its opening with the `artifact_id` to revise. Given `document_id`, a ticked `.xlsx` or `.xlsm` source, or the `artifact_id` of a revised copy of one, it returns the workbook's cells instead, read from the file itself (the source's original, or the copy's newest ready version) with openpyxl in read-only mode and formulas as written: each sheet's name and used range, then each non-empty cell as `B7 = 1250.5`, `A1 = "text"` or `C3 = =SUM(C2:C2)`, under the same 40,000 bytes ([`read_workbook.py`](../../surfsense_local/backend/modules/agent/tool_endpoint/read_workbook.py), [`cell_map.py`](../../surfsense_local/backend/modules/artifacts/revised_copies/cell_map.py)). `sheet` reads one sheet and `offset` is then the row to start from; a cut map ends with the row and the call that reads on, or, when one row holds more cells than a map shows, the first cell left out and the next row to read on from. The file is read once the tool's database reads have ended, so a big workbook's seconds of reading never hold SQLite's write lock. Sheet headings take at most a quarter of the map; the sheets past that are only named, up to 4,000 bytes, then counted. The indexed text of a workbook is Docling's tables, without sheet names, addresses or formulas, so the revise tool and the revisions skill send the agent here first. Any other source is refused, pointing at `sources/`. | | `surfsense_list_images` | `source_ids` | each source's figures, one line each: its name, size in pixels, page and caption when known, and its copy under `sources/figures/` to open with `read`; or that the source has none, is not in this workspace, or has its figures being extracted, which queues the figures-only pass ([documents](documents.md#figures)). A source not ticked for the turn is refused | | `surfsense_source_pages` | `document_id`; `pages`, at most four page or slide numbers from 1, the first ones when left out | a PDF, Word or PowerPoint source's own pages as images with the result, in ascending order, with the source's page or slide count; copies at most 1,000 pixels on their long side stay in `sources/pages/<id>-p<n>.png` for a closer look ([below](#source-pages)). Offered only when `opencode.json` declares image input. A source not ticked for the turn, an image, a note or any other kind of file is refused in a sentence saying what to use instead | @@ -228,7 +228,7 @@ The step links the version in the thread as a render's does (`surfsense_save_pag **Prompt length.** As in Studio, nothing limits a prompt per model: the tool's description asks for one self-contained prompt, as short as it says what to draw, and a model's own refusal comes back as a failed picture. -**A slow picture.** A local picture on a processor takes minutes, past a tool call's hard limit, so the call answers `Queued artifact …` with the name it will have, and no tool polls. A render naming it is not refused: it is made pending with `waits_for`, unqueued, and answers at once that it renders once the picture is drawn ([`waiting_renders/`](../../surfsense_local/backend/modules/artifacts/waiting_renders/)). The end of every Studio job queues the renders waiting on that picture once all theirs are ready, or fails one whose picture failed, was cancelled or deleted, with a sentence naming it; one update takes `waits_for` off, so two pictures ending together queue it once. The Studio worker does the same when it starts, after failing interrupted jobs. A web page still refuses a picture being drawn, since it is checked and screenshotted as it is saved. With a local chat model, the agent's next step waits in the model endpoint behind [the image's hold](local-models/admission.md#the-image-jobs-hold), so the picture is usually ready when the agent goes on. +**A slow picture.** A local picture on a processor takes minutes, past a tool call's hard limit, so the call answers `Queued artifact …` with the name it will have, and no tool polls. A render naming it is not refused: it is made pending with `waits_for`, unqueued, and answers at once that it renders once the picture is drawn ([`waiting_renders/`](../../surfsense_local/backend/modules/artifacts/waiting_renders/)). The end of every Studio job queues the renders waiting on that picture once all theirs are ready, or fails one whose picture failed, was cancelled or deleted, with a sentence naming it; one update takes `waits_for` off, so two pictures ending together queue it once. Cancelling a picture settles them too, since a cancelled job still queued is revoked and never reaches its end. A settle that fails at a job's end is logged, not raised, since a raise would have Huey run the job and bill its picture again. The Studio worker settles every waiting render when it starts, after failing interrupted jobs. A web page still refuses a picture being drawn, since it is checked and screenshotted as it is saved. With a local chat model, the agent's next step waits in the model endpoint behind [the image's hold](local-models/admission.md#the-image-jobs-hold), so the picture is usually ready when the agent goes on. **Placing one.** `image-<artifact id>` names one version, so it keeps meaning the same picture as edits are made. It works in a render's `images` and in a page ([`names.py`](../../surfsense_local/backend/modules/artifacts/drawn_images/names.py)), only in the chat that made the picture, as a chart's name does. Finding it reads the database; each use then writes it afresh as a PNG to `outputs/images/<name>.png`, outside any transaction, converted if the model answered in another format and refused past 40 million pixels, since the agent can write to its own outputs and what is placed must be what the model drew. A page's size is worked out before any image is written into it, since a small file can name one picture thousands of times. diff --git a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py index 39c576d90b..47810b8e7a 100644 --- a/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py +++ b/surfsense_local/backend/modules/agent/tool_endpoint/make_image.py @@ -202,8 +202,9 @@ def make( f"{queued_line(made)}\n" f"The picture is still being made after {time.monotonic() - began:.0f} s. Do " "not make it again: it appears in Studio when it is ready. Its name is " - f"{name}, and a document or page can place it once it is ready: if one says " - "it is still being drawn, tell the user so and stop." + f"{name}. A document or comic can place it now and is made once it is " + "drawn. A web page can place it only once it is ready: if one says it is " + "still being drawn, tell the user so and stop." ) diff --git a/surfsense_local/backend/modules/artifacts/service.py b/surfsense_local/backend/modules/artifacts/service.py index 1a9a8ecfa3..d96bcc72b8 100644 --- a/surfsense_local/backend/modules/artifacts/service.py +++ b/surfsense_local/backend/modules/artifacts/service.py @@ -14,6 +14,7 @@ from modules.artifacts.refused import refused from modules.artifacts.schemas import FormatRead, StudioJobCreate from modules.artifacts.studio_documents.recipe import refinement, renders_as_stored +from modules.artifacts.waiting_renders.settle import settle_waiting_renders from modules.documents.models import Document, DocumentStatus, DocumentType from modules.documents.sources import load_selected_sources from modules.llm.model_type import ModelType @@ -145,6 +146,8 @@ def cancel_artifact(session: Session, artifact: Artifact) -> Artifact: """Stop a queued or running generation. The worker exits without writing ready.""" if not cancel_studio_job(session, artifact.document, artifact.id): raise refused(status.HTTP_409_CONFLICT, "nothing is running", "nothing_running") + # A queued job is revoked and never runs its end, so what waits on it settles here. + settle_waiting_renders(session, picture_id=artifact.id) return artifact diff --git a/surfsense_local/backend/tests/integration/agent/test_make_image.py b/surfsense_local/backend/tests/integration/agent/test_make_image.py index f3b7d0941b..07124e3e04 100644 --- a/surfsense_local/backend/tests/integration/agent/test_make_image.py +++ b/surfsense_local/backend/tests/integration/agent/test_make_image.py @@ -350,6 +350,9 @@ async def test_a_slow_picture_answers_queued_with_the_name_it_will_have( assert text.splitlines()[0] == f"Queued artifact {artifact_id}, version 1: Boat" assert f"image-{artifact_id}" in text assert "Do not make it again" in text + # A document or comic waits for it on the server; a web page cannot. + assert "A document or comic can place it now" in text + assert "A web page can place it only once it is ready" in text async def test_regenerating_paints_the_same_prompt_with_no_chat_model( diff --git a/surfsense_local/backend/tests/integration/agent/test_waiting_render.py b/surfsense_local/backend/tests/integration/agent/test_waiting_render.py index 961f330f63..08e07ff2c1 100644 --- a/surfsense_local/backend/tests/integration/agent/test_waiting_render.py +++ b/surfsense_local/backend/tests/integration/agent/test_waiting_render.py @@ -157,3 +157,24 @@ async def test_a_picture_that_ended_while_the_worker_was_down_is_settled_at_star assert error is not None and f"image-{picture} failed" in error assert slow.hold is not None slow.hold.set() + + +async def test_a_picture_cancelled_while_queued_fails_the_render_at_once( + tools: ToolEndpoint, slow: Painter, engine: Engine +) -> None: + """A revoked job never runs, so nothing at its end would settle the render.""" + workspace_id = await tools.workspace() + queued, _ = await tools.call( + workspace_id, "make_image", {"prompt": "A boat", "title": "Boat"} + ) + picture = _artifact_id(queued) + text, _ = await tools.call( + workspace_id, "render_document", _placing(f"image-{picture}") + ) + + cancelled = await tools.client.post(f"/artifacts/{picture}/cancel") + + assert cancelled.status_code == 200, cancelled.text + status, error = _status(engine, _artifact_id(text)) + assert status is DocumentStatus.FAILED + assert error is not None and f"image-{picture} failed" in error diff --git a/surfsense_local/backend/tests/integration/worker/test_studio.py b/surfsense_local/backend/tests/integration/worker/test_studio.py index f7ed807be6..cfbeb7a191 100644 --- a/surfsense_local/backend/tests/integration/worker/test_studio.py +++ b/surfsense_local/backend/tests/integration/worker/test_studio.py @@ -748,6 +748,24 @@ def resolve(job_session: Session) -> ResolvedGeneration: assert api_could_write == [True] +def test_a_settle_that_fails_after_the_job_does_not_ask_for_a_retry( + session: Session, stub_model: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """A raise here would make Huey run the job, and bill its picture, again.""" + + def locked(*_args: object, **_kwargs: object) -> list: + raise RuntimeError("database is locked") + + monkeypatch.setattr("worker.waiting_renders.settle_waiting_renders", locked) + _capture_model(monkeypatch, SUMMARY) + artifact = make_artifact(session) + + run(artifact.id) + + session.expire_all() + assert artifact.document.status is DocumentStatus.READY + + def test_a_generation_failure_leaves_a_reason( session: Session, stub_model: None, monkeypatch: pytest.MonkeyPatch ) -> None: diff --git a/surfsense_local/backend/worker/waiting_renders.py b/surfsense_local/backend/worker/waiting_renders.py index 84c49d7bd5..2be44cd503 100644 --- a/surfsense_local/backend/worker/waiting_renders.py +++ b/surfsense_local/backend/worker/waiting_renders.py @@ -1,14 +1,25 @@ """Settle the documents waiting on pictures from the worker, and tell open windows.""" +import logging + from modules.artifacts.waiting_renders.settle import settle_waiting_renders from shared.config import get_storage_settings from shared.db import create_db_engine, create_session_factory from worker.notify import notify_artifact_updates +logger = logging.getLogger(__name__) + def settle_after_job(artifact_id: int) -> None: - """A job ended, or found its artifact gone: what waited on that picture may go.""" - _settle(picture_id=artifact_id) + """A job ended, or found its artifact gone: what waited on that picture may go. + + Never raises: the job is done, and a raise would have Huey draw it again. + What is left waiting is settled by the sweep at the worker's next start. + """ + try: + _settle(picture_id=artifact_id) + except Exception: + logger.exception("could not settle what waited on picture %s", artifact_id) def settle_every_waiting_render() -> None: From 82820ff44a55eff767861fce76f1f835287737cd Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Thu, 12 Nov 2026 13:14:36 +0530 Subject: [PATCH 32/33] fix(local): a comic is planned in full and lettered in any script A plan short of its pages is asked for again, any page drawn locally moves every line to the strip, and the strip sets Chinese, Japanese and Korean in fonts that have them, wrapping lines with no spaces. --- docs/architecture/studio.md | 6 +-- .../tests/unit/worker/test_comic_binding.py | 52 ++++++++++++++++++- .../tests/unit/worker/test_comic_pipeline.py | 47 +++++++++++++++-- .../studio/media/visual/comic/asking.py | 10 ++-- .../studio/media/visual/comic/pipeline.py | 8 +-- .../worker/studio/media/visual/comic/plan.py | 11 ++-- .../studio/media/visual/comic_binding/bind.py | 33 +++++++++--- .../studio/office/markdown/pdf_fonts.py | 13 +++-- 8 files changed, 152 insertions(+), 28 deletions(-) diff --git a/docs/architecture/studio.md b/docs/architecture/studio.md index 847384b6fb..920f5bbadb 100644 --- a/docs/architecture/studio.md +++ b/docs/architecture/studio.md @@ -65,7 +65,7 @@ Eight formats follow it: - **podcast**: the model outlines the episode from the brief, then drafts it segment by segment. Every segment sends the same one-line system prompt and the sources, and its whole draft prompt after them, because llama-server reuses a prompt only up to its first changed token. Measured through the drafting code on Qwen3 1.7B with about 6,200 tokens of sources: with the draft prompt first, every segment re-read the sources; after them, segments 2 to 4 read about 230 tokens each. The draft prompt goes after them whole: split, with what the segments share left ahead of the sources, the same model wrote half the dialogue, in long lines. Each reply is capped at 12 tokens per word of the segment's target, and never under a planned 250-word segment's worth. Uncapped, Qwen3 1.7B looped on a 225-word segment until its 40,960-token window was full, and the JSON retry, which replays the failed reply, could not fit. The target is the outline's own guess: Qwen3 1.7B once gave a segment 20 words, wrote past them, and a 240-token cap ended both replies mid-JSON. The chosen audio model voices every line through audio.cpp's server, and the transcript is the body. - **image**: the model writes a title and an image prompt, and the image model paints the prompt. - **infographic**: the model writes a factual brief (a title, a summary and up to 8 sections of label, value and detail), and the image model paints a prompt built from it in a fixed sketchnote style. -- **comic**: options `{kind, pages}`, one of ten kinds ([`kinds.py`](../../surfsense_local/backend/modules/artifacts/comic/kinds.py)) and 1 to 12 pictures, checked when the job is created. The model plans a title, a style line, a character sheet and each page's panels and lines, asked for in [`schema.py`](../../surfsense_local/backend/worker/studio/media/visual/comic/schema.py)'s shape; the plan is cut to the count asked for, since each picture is billed or takes minutes. A reply that is not a plan is asked for once more with the reply replayed ([`asking.py`](../../surfsense_local/backend/worker/studio/media/visual/comic/asking.py)), which costs no picture. A second reply with no JSON in it is the model declining, often over the sources themselves, so the job fails with the model's first line rather than a parse error. Page 1 is drawn alone, the rest with page 1 as their reference by [the model that takes a picture in](#which-model-takes-a-picture), side by side on a remote model and one at a time on this computer; with none, each is drawn from its words and the version keeps `drawn_from_words`. A remote model letters each page's lines in bubbles; a local one draws no words, its lettering being unmeasured, and every page's lines are set in a strip under it, as a picture book's and a storyboard's always are. The pages are bound by [`comic_binding/`](../../surfsense_local/backend/worker/studio/media/visual/comic_binding/), which the agent's comics share; the body is every page's panels and lines. A failed comic is never retried by the worker, since every page would be drawn again. No page is kept on its own, so a Studio comic has no Edit: Regenerate draws it again. +- **comic**: options `{kind, pages}`, one of ten kinds ([`kinds.py`](../../surfsense_local/backend/modules/artifacts/comic/kinds.py)) and 1 to 12 pictures, checked when the job is created. The model plans a title, a style line, a character sheet and each page's panels and lines, asked for in [`schema.py`](../../surfsense_local/backend/worker/studio/media/visual/comic/schema.py)'s shape; the plan is cut to the count asked for, since each picture is billed or takes minutes, and one short of it is refused. A reply that is not a plan is asked for once more with the reply replayed ([`asking.py`](../../surfsense_local/backend/worker/studio/media/visual/comic/asking.py)), which costs no picture. A second reply with no JSON in it is the model declining, often over the sources themselves, so the job fails with the model's first line rather than a parse error. Page 1 is drawn alone, the rest with page 1 as their reference by [the model that takes a picture in](#which-model-takes-a-picture), side by side on a remote model and one at a time on this computer; with none, each is drawn from its words and the version keeps `drawn_from_words`. A remote model letters each page's lines in bubbles; when any page is drawn by a local model, the image model or the one drawing pages 2 on, no page carries words, its lettering being unmeasured, and every page's lines are set in a strip under it, as a picture book's and a storyboard's always are. The strip sets letters Helvetica lacks in the Word and PDF builder's fonts ([`pdf_fonts.py`](../../surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py)), and a word too wide for a line, as Chinese or Japanese is, is cut between letters. The pages are bound by [`comic_binding/`](../../surfsense_local/backend/worker/studio/media/visual/comic_binding/), which the agent's comics share; the body is every page's panels and lines. A failed comic is never retried by the worker, since every page would be drawn again. No page is kept on its own, so a Studio comic has no Edit: Regenerate draws it again. Two formats break it. PPTX and XLSX go through `office/`: the tier prompt asks the model to "write one standalone Python script" for the format's library (python-pptx or xlsxwriter), with a `SKILL.md` of authoring guidance beside each format, to the runner's contract: start with a `# title:` comment and save the file at `OUTPUT_PATH`. [`office/pipeline.py`](../../surfsense_local/backend/worker/studio/office/pipeline.py) runs the reply through [the runner](#script-documents), in its own process, never with `exec()` in the worker's. A failing script goes back to the model with its error and the traceback's end, for up to three attempts; one longer than 48,000 characters ran on rather than broke on a line, so it is not shown back but asked for again, shorter, since replayed it overflowed the window ([`script_attempts.py`](../../surfsense_local/backend/worker/studio/office/script_attempts.py)). On a local model each draft stops at 16,384 tokens: a deck's script measured about 3,000, and a 1.7B that looped wrote 130,171 characters for 272 s first. Its failure reason does not start `Script error: `, as for [Word and PDF](#word-and-pdf). The title is the `# title:` comment, else the prompt, the sources or the format; the body is the file's own text, read as a script document's is. No spec is kept: a deck or workbook Studio drafts has no versions and no Refine. @@ -272,7 +272,7 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i | `GET` | `/workspaces/{workspace_id}/studio/podcast/brief` | the podcast brief to review before submitting, the model's `voices` (`null` when they are typed) and the `languages` it may use | | `GET` | `/workspaces/{workspace_id}/artifacts` | the workspace's artifacts, newest first | | `GET` | `/artifacts/{artifact_id}` | detail: the body, the files, quiz or flashcard progress, and for a revised copy `revision: {derived_from_document_id, source_name, counts, applied}` | -| `POST` | `/artifacts/{artifact_id}/regenerate` | run the job again; `202`, or `409` while a run is queued or running, with the code `already_generating` unless the artifact is a revised copy, whose refusal has no code yet. A draft re-resolves a recorded `source_scope` without the artifact itself, else replays the recorded ids that still exist, and gives `409` `sources_gone`, "None of this artifact's sources are left.", when none remain. A refine repeats its rewrite without its sources, and gives `409` `rewrite_too_long` when the rewrite no longer fits the selected model ([Refine](#refine)). An image edit repeats its edit on the same base without its sources, and gives `409` `needs_image_edit` when no edit model can be reached ([editing a picture](#editing-a-picture)). A revised copy applies its stored edits to the same input again and asks no model | +| `POST` | `/artifacts/{artifact_id}/regenerate` | run the job again; `202`, or `409` while a run is queued or running, with the code `already_generating` unless the artifact is a revised copy, whose refusal has no code yet. A draft re-resolves a recorded `source_scope` without the artifact itself, else replays the recorded ids that still exist, and gives `409` `sources_gone`, "None of this artifact's sources are left.", when none remain. A refine repeats its rewrite without its sources, and gives `409` `rewrite_too_long` when the rewrite no longer fits the selected model ([Refine](#refine)). An image edit repeats its edit on the same base without its sources, and a drawn picture paints its prompt again with none; either gives `409` `needs_image` when no image model is chosen or the chosen one is gone, since any image model edits or redraws ([editing a picture](#editing-a-picture)). A revised copy applies its stored edits to the same input again and asks no model | | `POST` | `/artifacts/{artifact_id}/refine` | `{instruction}`; the next version, `pending`; `202`; its `409`s are under [Refine](#refine), six of them with a code: `needs_chat`, as for an unavailable format, and five of a refine's own | | `POST` | `/artifacts/{artifact_id}/image-edit` | `{instruction}`; an image's or infographic's next version, `pending`; `202`; its four `409`s are under [editing a picture](#editing-a-picture) | | `POST` | `/artifacts/{artifact_id}/revisions/{accept-all\|reject-all}` | a Word revised copy's next version from its newest, every tracked change accepted or rejected; `202`, or `409` for a workbook or deck, a version with no changes, or while a version is being made ([revised copies](#revised-copies)) | @@ -330,7 +330,7 @@ Every pipeline returns a `Built`: a `title`, the `markdown` that is always the i - A drawn `image` or `infographic` resolves the `image_gen` selection, and an edit, or a drawing with references, the model that takes a picture in ([below](#which-model-takes-a-picture)). A remote selection loads its connection, which runs the egress check and decrypts the key. An `sdcpp` selection points the same client at the bundled sd-server's loopback URL, with no key and no egress decision, which is what makes an image possible on an offline machine. - sd-server runs only while Studio needs it, because it keeps a model's weights in memory until it exits. The API answers Electron's poll from the jobs ([`local_image_demand.py`](../../surfsense_local/backend/modules/artifacts/local_image_demand.py)): the chosen model of the image type the oldest processing job needs, read from its recipe, so an edit needs `image_edit` though its format is still `image` or `infographic`; else, for 5 minutes after the last such job ended, the same one, so a second image does not reload it; else nothing, and Electron stops it. A cancel ends the window at once, since sd-server cannot stop a generation, and so does any other Studio job running, since a podcast voices in that memory and a text job loads the chat model into it. The job first gives up the text runtime through the API, held for the whole image, because the two share the graphics card and its writing is done by then: the API lets running replies finish, unloads the text models and keeps local text waiting until the image is drawn, and unloads the speech recognition model, starting no recording's window meanwhile ([admission](local-models/admission.md#the-image-jobs-hold)). Then it waits up to 2 minutes for `/sdapi/v1/sd-models` to name its weights before it posts, and fails with a reason if sd-server never comes up. - sd-server is launched with each file of the build on its flag and the model's reviewed `image` defaults as its own (`-W`, `-H`, `--steps`, `--cfg-scale`, `--sampling-method`, `--flow-shift`), since a request that carries none of them starts from the launch settings ([`local-models/catalog.md`](local-models/catalog.md)). -- The client posts `{model, prompt}`, and `size` when a shape is asked, to `/images/generations`, falls back once to `/images` only on `404` or `405`, and remembers the route that worked for the life of the process. A refusal is decided by its status alone, so one whose body is cut off still counts: OpenRouter has hung up partway through the body of a `404`, which failed a comic's pages before ([`image_response.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py)). Only a success must arrive whole. It accepts `b64_json`, a data URL or a URL, caps the sizes, and checks the bytes against the claimed MIME type ([`connections.md`](connections.md)). +- The client posts `{model, prompt}`, and `size` when a shape is asked, to `/images/generations`, falls back once to `/images`, where the shape is `aspect_ratio`, only on `404` or `405`, and remembers the route that worked for the life of the process. A refusal is decided by its status alone, so one whose body is cut off still counts: OpenRouter has hung up partway through the body of a `404`, which failed a comic's pages before ([`image_response.py`](../../surfsense_local/backend/modules/llm/providers/openai_compatible/image_response.py)). Only a success must arrive whole. It accepts `b64_json`, a data URL or a URL, caps the sizes, and checks the bytes against the claimed MIME type ([`connections.md`](connections.md)). - The `text_gen` model writes first in both formats, except a picture the agent asked for ([below](#pictures-the-agent-asks-for)). The markdown body is the image prompt or the brief, so an image can be found by what it shows. - A drawn picture is v1 of its own document: the job records `version` and `recipe: {"kind": "draft"}` as a Word draft's, so its edits group under it. Regenerate on it still draws from the sources and replaces v1 in place. - Every drawing, edit and redraw keeps the exact prompt it sent as `drawing_prompt`, so a later redraw starts from it ([`redraw/`](../../surfsense_local/backend/worker/studio/media/visual/redraw/)). diff --git a/surfsense_local/backend/tests/unit/worker/test_comic_binding.py b/surfsense_local/backend/tests/unit/worker/test_comic_binding.py index 8edba24b35..30447ccb34 100644 --- a/surfsense_local/backend/tests/unit/worker/test_comic_binding.py +++ b/surfsense_local/backend/tests/unit/worker/test_comic_binding.py @@ -6,7 +6,13 @@ from PIL import Image from pypdf import PdfReader -from worker.studio.media.visual.comic_binding.bind import BoundPage, bind +from worker.studio.media.visual.comic_binding.bind import ( + MARGIN, + WIDTH, + BoundPage, + _wrapped, + bind, +) pytestmark = pytest.mark.unit @@ -98,3 +104,47 @@ def test_pages_are_placed_as_jpeg_so_a_book_stays_small() -> None: pdf = bind("Noise", [BoundPage(out.getvalue())], layout="pages", direction="ltr") assert len(pdf) < len(out.getvalue()) + + +def _fonts(pdf: bytes) -> set[str]: + page = PdfReader(io.BytesIO(pdf)).pages[0] + return {str(font["/BaseFont"]) for font in page["/Resources"]["/Font"].values()} + + +def test_a_mangas_japanese_lines_are_set_in_a_font_that_has_them() -> None: + """Helvetica draws a box for every kana and kanji.""" + pdf = bind( + "猫", + [BoundPage(_png(400, 400), ("ノラ: 電球が壊れた!",))], + layout="pages", + direction="rtl", + ) + + assert "/HeiseiKakuGo-W5" in _fonts(pdf) + + +def test_a_line_with_no_spaces_wraps_inside_the_page() -> None: + """Japanese and Chinese have no spaces to break at; the strip grows instead.""" + short = bind( + "猫", [BoundPage(_png(400, 400), ("猫",))], layout="pages", direction="rtl" + ) + long = bind( + "猫", + [BoundPage(_png(400, 400), ("猫" * 200,))], + layout="pages", + direction="rtl", + ) + + assert _sizes(long)[0][1] > _sizes(short)[0][1] + + +def test_a_word_too_wide_is_cut_with_no_spaces_added() -> None: + """Letters of a cut word join as written; a real space between words stays.""" + line = "電球" * 60 + " 猫" + + wrapped = _wrapped(line, WIDTH - 2 * MARGIN) + + assert len(wrapped) > 1 + # No space is added inside the cut word; the break takes the real one's place. + assert " " not in "".join(wrapped[:-1]) + assert "".join(wrapped).replace(" ", "") == line.replace(" ", "") diff --git a/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py b/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py index 6729cdead4..869d233a3e 100644 --- a/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py +++ b/surfsense_local/backend/tests/unit/worker/test_comic_pipeline.py @@ -95,10 +95,8 @@ def _painter( return ResolvedImageGeneration(selection, artist) -def _keeper(artist: Artist) -> PictureModel: - selection = type( - "Selection", (), {"name": "flux", "provider": "openai_compatible"} - )() +def _keeper(artist: Artist, provider: str = "openai_compatible") -> PictureModel: + selection = type("Selection", (), {"name": "flux", "provider": provider})() return PictureModel("image_model", selection, artist) @@ -180,6 +178,26 @@ def test_a_local_model_draws_no_words_and_every_line_goes_in_the_strip( assert "MILO: The light is out!" in _text(built.primary) +def test_later_pages_drawn_on_this_computer_leave_every_line_to_the_strip( + asked: list[str], +) -> None: + """A local edit model draws pages 2 on; one comic is lettered one way throughout.""" + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 3}, + keeper=_keeper(artist, provider="sdcpp"), + ) + + prompts = artist.drawn + [prompt for prompt, _ in artist.with_reference] + assert all("no text, no lettering, no speech bubbles" in p for p in prompts) + assert "MILO: The light is out!" in _text(built.primary) + + def test_a_manga_reads_right_to_left_in_its_pages_and_its_book( asked: list[str], ) -> None: @@ -293,6 +311,27 @@ def test_a_plan_that_is_not_json_is_asked_for_once_more( assert built.title == "The Lighthouse Cat" +def test_a_plan_short_of_the_pages_asked_for_is_asked_for_again( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Four pages asked for are four drawn, not a one-page comic marked ready.""" + four = {**PLAN, "pages": [*PLAN["pages"], PLAN["pages"][0]]} + calls = _replying(monkeypatch, json.dumps(PLAN), json.dumps(four)) + artist = Artist() + + built = comic.render( + _painter(artist), + _writer(), + [Source(1, "Lighthouse", "A cat keeps the light.")], + None, + {"kind": "western", "pages": 4}, + keeper=_keeper(artist), + ) + + assert "3 of the 4 pages" in calls[1]["repair"].instruction + assert len(PdfReader(io.BytesIO(built.primary)).pages) == 4 + + def test_a_model_that_declines_twice_fails_with_what_it_said( monkeypatch: pytest.MonkeyPatch, ) -> None: diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/asking.py b/surfsense_local/backend/worker/studio/media/visual/comic/asking.py index 486eac671d..bf8f65ec76 100644 --- a/surfsense_local/backend/worker/studio/media/visual/comic/asking.py +++ b/surfsense_local/backend/worker/studio/media/visual/comic/asking.py @@ -11,9 +11,6 @@ logger = logging.getLogger(__name__) -_JSON_NUDGE = ( - "Your previous reply was not the plan as JSON. Return only the JSON object." -) # Enough of a reply to tell a refusal from broken JSON in the log. _LOGGED_CHARS = 500 @@ -32,6 +29,7 @@ def ask_for_plan( try: return planning.parse(reply, untitled, count) except ValueError as first: + first_error = first logger.warning( "studio: comic plan unreadable (%s); asking again. It began: %r", first, @@ -42,7 +40,11 @@ def ask_for_plan( system, sources, json_schema=REPLY, - repair=generate.Repair(reply, _JSON_NUDGE), + repair=generate.Repair( + reply, + f"Your previous reply could not be used: {first_error}. Return only " + f"the plan as one JSON object, with exactly {count} pages.", + ), ) try: return planning.parse(retry, untitled, count) diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py b/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py index 1373b6c9a3..8ebde04868 100644 --- a/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py +++ b/surfsense_local/backend/worker/studio/media/visual/comic/pipeline.py @@ -35,11 +35,13 @@ def render( options["pages"], fallback_title(user_prompt, sources, "Comic"), ) - if not plan.pages: - raise ValueError("the writer planned no pages") # Studio cannot look at a page to check its lettering; a local model's is not # measured at all, so its pages are drawn without words and lettered below. - lettered = kind.bubbles and painter.selection.provider != sdcpp.PROVIDER + # Any page drawn locally letters them all so, keeping one comic one way. + drawers = [painter.selection.provider] + if keeper is not None and len(plan.pages) > 1: + drawers.append(keeper.selection.provider) + lettered = kind.bubbles and sdcpp.PROVIDER not in drawers prompts = [ page_prompt( kind, diff --git a/surfsense_local/backend/worker/studio/media/visual/comic/plan.py b/surfsense_local/backend/worker/studio/media/visual/comic/plan.py index d7b51e2246..35f6f0e7e6 100644 --- a/surfsense_local/backend/worker/studio/media/visual/comic/plan.py +++ b/surfsense_local/backend/worker/studio/media/visual/comic/plan.py @@ -47,13 +47,18 @@ def prompt(tier: Tier, user_prompt: str | None, kind: Kind, count: int) -> str: def parse(raw: str, untitled: str, count: int) -> Plan: - """The plan, cut to the pages asked for: each one drawn is billed or takes minutes.""" + """The plan, cut to the pages asked for: each one drawn is billed or takes minutes. + + One short of them is refused, so it is asked for again before any is drawn. + """ spec = parse_json(raw) pages = tuple( panels - for page in as_list(spec.get("pages"))[:count] + for page in as_list(spec.get("pages")) if isinstance(page, dict) and (panels := _panels(page)) - ) + )[:count] + if len(pages) < count: + raise ValueError(f"the plan has {len(pages)} of the {count} pages asked for") return Plan( title=as_text(spec.get("title")) or untitled, style=as_text(spec.get("style")), diff --git a/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py index 1122ec211e..c229d264d5 100644 --- a/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py +++ b/surfsense_local/backend/worker/studio/media/visual/comic_binding/bind.py @@ -10,6 +10,7 @@ from reportlab.pdfgen import canvas from modules.artifacts.comic.reading import Direction, Layout +from worker.studio.office.markdown.pdf_fonts import font_runs # Comic-book trim width; each page is as tall as its picture. WIDTH = 6.625 * inch @@ -72,16 +73,37 @@ def _block(page: BoundPage) -> _Block: def _wrapped(text: str, width: float) -> list[str]: lines: list[str] = [] line = "" - for word in text.split(): - trial = f"{line} {word}".strip() - if line and stringWidth(trial, FONT, SIZE) > width: + for piece, glued in _breakable(text, width): + trial = f"{line}{'' if glued else ' '}{piece}" if line else piece + if line and _width(trial) > width: lines.append(line) - line = word + line = piece else: line = trial return [*lines, line] if line else lines +def _breakable(text: str, width: float) -> list[tuple[str, bool]]: + """Words, with one too wide for a line, as Chinese or Japanese is, cut into + letters; each piece says whether it joins the one before with no space.""" + pieces: list[tuple[str, bool]] = [] + for word in text.split(): + cut = list(word) if _width(word) > width else [word] + pieces += [(piece, n > 0) for n, piece in enumerate(cut)] + return pieces + + +def _width(text: str) -> float: + return sum(stringWidth(run, font or FONT, SIZE) for font, run in font_runs(text)) + + +def _set_line(pdf: canvas.Canvas, x: float, y: float, line: str) -> None: + for font, run in font_runs(line): + pdf.setFont(font or FONT, SIZE) + pdf.drawString(x, y, run) + x += stringWidth(run, font or FONT, SIZE) + + def _draw(pdf: canvas.Canvas, blocks: list[_Block], gutter: float) -> None: """Top to bottom on one page, each picture with its strip under it.""" height = sum(b.height + b.strip for b in blocks) + gutter * (len(blocks) - 1) @@ -89,10 +111,9 @@ def _draw(pdf: canvas.Canvas, blocks: list[_Block], gutter: float) -> None: top = height for block in blocks: pdf.drawImage(block.image, 0, top - block.height, WIDTH, block.height) - pdf.setFont(FONT, SIZE) y = top - block.height - MARGIN - SIZE for line in block.lines: - pdf.drawString(MARGIN, y, line) + _set_line(pdf, MARGIN, y, line) y -= LEADING top -= block.height + block.strip + gutter pdf.showPage() diff --git a/surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py b/surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py index 07c4246978..67b96c0019 100644 --- a/surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py +++ b/surfsense_local/backend/worker/studio/office/markdown/pdf_fonts.py @@ -44,21 +44,26 @@ def register_fonts() -> None: def font_markup(text: str) -> str: """The text escaped for a Paragraph, each run Helvetica cannot set in a font that can.""" + return "".join(_wrapped(run, font) for font, run in font_runs(text)) + + +def font_runs(text: str) -> list[tuple[str | None, str]]: + """The text cut where its font changes; None is the default font's run.""" register_fonts() han = _han_font(text) - parts: list[str] = [] + runs: list[tuple[str | None, str]] = [] run: list[str] = [] current: str | None = None for character in text: font = _font_for(character, han) if font != current and run: - parts.append(_wrapped("".join(run), current)) + runs.append((current, "".join(run))) run = [] current = font run.append(character) if run: - parts.append(_wrapped("".join(run), current)) - return "".join(parts) + runs.append((current, "".join(run))) + return runs def _wrapped(text: str, font: str | None) -> str: From f1e7dffd37e22f5aacd96fdbc1b1fbb2aa4d9ca0 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Thu, 12 Nov 2026 13:20:59 +0530 Subject: [PATCH 33/33] refactor(local): drop the needs_image_edit error nothing sends any more An edit's Retry now refuses only with needs_image, when no image model is chosen. --- .../frontend/src/features/studio/studio-error-text.test.ts | 1 - .../frontend/src/features/studio/studio-error-text.ts | 5 ----- surfsense_local/frontend/translations/de.json | 1 - surfsense_local/frontend/translations/en.json | 1 - surfsense_local/frontend/translations/es.json | 1 - surfsense_local/frontend/translations/fr.json | 1 - surfsense_local/frontend/translations/hi.json | 1 - surfsense_local/frontend/translations/ja.json | 1 - surfsense_local/frontend/translations/ko.json | 1 - surfsense_local/frontend/translations/pt-BR.json | 1 - surfsense_local/frontend/translations/ru.json | 1 - surfsense_local/frontend/translations/zh-CN.json | 1 - 12 files changed, 16 deletions(-) diff --git a/surfsense_local/frontend/src/features/studio/studio-error-text.test.ts b/surfsense_local/frontend/src/features/studio/studio-error-text.test.ts index e82152769c..b5614eea56 100644 --- a/surfsense_local/frontend/src/features/studio/studio-error-text.test.ts +++ b/surfsense_local/frontend/src/features/studio/studio-error-text.test.ts @@ -34,7 +34,6 @@ describe("a refused Studio job", () => { "image_version_unfinished", "This version did not finish: retry it, or edit another version.", ], - ["needs_image_edit", "Needs an image editing model."], [ "edited_from", "A later version is edited from this one: delete the whole document.", diff --git a/surfsense_local/frontend/src/features/studio/studio-error-text.ts b/surfsense_local/frontend/src/features/studio/studio-error-text.ts index b33ba69199..85e93e5a18 100644 --- a/surfsense_local/frontend/src/features/studio/studio-error-text.ts +++ b/surfsense_local/frontend/src/features/studio/studio-error-text.ts @@ -110,11 +110,6 @@ const studioErrorText: Record<string, (detail: Detail) => string | null> = { id: "studio_error_needs_image", defaultMessage: "Needs an image model.", }), - needs_image_edit: () => - intl.formatMessage({ - id: "studio_error_needs_image_edit", - defaultMessage: "Needs an image editing model.", - }), needs_audio: () => intl.formatMessage({ id: "studio_error_needs_audio", diff --git a/surfsense_local/frontend/translations/de.json b/surfsense_local/frontend/translations/de.json index becd05a3d1..69e4d64761 100644 --- a/surfsense_local/frontend/translations/de.json +++ b/surfsense_local/frontend/translations/de.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "Ein Chatmodell und ein Audiomodell sind erforderlich.", "studio_error_needs_chat_image": "Ein Chatmodell und ein Bildmodell sind erforderlich.", "studio_error_needs_image": "Ein Bildmodell ist erforderlich.", - "studio_error_needs_image_edit": "Ein Modell für Bildbearbeitung ist erforderlich.", "studio_error_no_sources": "Wähle mindestens eine Quelle aus.", "studio_error_not_editable": "Nur ein in Studio erstelltes Bild oder eine Infografik kann bearbeitet werden.", "studio_error_not_refinable": "Nur ein in Studio erstelltes Word-Dokument, PDF oder eine Webseite kann überarbeitet werden.", diff --git a/surfsense_local/frontend/translations/en.json b/surfsense_local/frontend/translations/en.json index 03143ab211..d40e961f00 100644 --- a/surfsense_local/frontend/translations/en.json +++ b/surfsense_local/frontend/translations/en.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "Needs a chat model and an audio model.", "studio_error_needs_chat_image": "Needs a chat model and an image model.", "studio_error_needs_image": "Needs an image model.", - "studio_error_needs_image_edit": "Needs an image editing model.", "studio_error_no_sources": "Pick at least one source.", "studio_error_not_editable": "Only an image or infographic made in Studio can be edited.", "studio_error_not_refinable": "Only a Word document, PDF or web page made in Studio can be refined.", diff --git a/surfsense_local/frontend/translations/es.json b/surfsense_local/frontend/translations/es.json index f37fbdf2a7..2d3b98863d 100644 --- a/surfsense_local/frontend/translations/es.json +++ b/surfsense_local/frontend/translations/es.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "Se necesitan un modelo de chat y uno de audio.", "studio_error_needs_chat_image": "Se necesitan un modelo de chat y uno de imagen.", "studio_error_needs_image": "Se necesita un modelo de imagen.", - "studio_error_needs_image_edit": "Se necesita un modelo de edición de imágenes.", "studio_error_no_sources": "Elige al menos una fuente.", "studio_error_not_editable": "Solo se puede editar una imagen o una infografía creadas en Studio.", "studio_error_not_refinable": "Solo se puede refinar un documento de Word, un PDF o una página web creados en Studio.", diff --git a/surfsense_local/frontend/translations/fr.json b/surfsense_local/frontend/translations/fr.json index faabef039e..6d8cbcfda7 100644 --- a/surfsense_local/frontend/translations/fr.json +++ b/surfsense_local/frontend/translations/fr.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "Un modèle de chat et un modèle audio sont requis.", "studio_error_needs_chat_image": "Un modèle de chat et un modèle d’image sont requis.", "studio_error_needs_image": "Un modèle d’image est requis.", - "studio_error_needs_image_edit": "Un modèle de modification d’images est requis.", "studio_error_no_sources": "Choisissez au moins une source.", "studio_error_not_editable": "Seule une image ou une infographie créée dans Studio peut être modifiée.", "studio_error_not_refinable": "Seul un document Word, un PDF ou une page web créé dans Studio peut être affiné.", diff --git a/surfsense_local/frontend/translations/hi.json b/surfsense_local/frontend/translations/hi.json index 80fdb509c7..791e44ac52 100644 --- a/surfsense_local/frontend/translations/hi.json +++ b/surfsense_local/frontend/translations/hi.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "एक चैट मॉडल और एक ऑडियो मॉडल चाहिए।", "studio_error_needs_chat_image": "एक चैट मॉडल और एक इमेज मॉडल चाहिए।", "studio_error_needs_image": "एक इमेज मॉडल चाहिए।", - "studio_error_needs_image_edit": "एक इमेज एडिटिंग मॉडल चाहिए।", "studio_error_no_sources": "कम से कम एक स्रोत चुनें।", "studio_error_not_editable": "केवल Studio में बनी इमेज या इन्फ़ोग्राफ़िक ही एडिट की जा सकती है।", "studio_error_not_refinable": "केवल Studio में बना Word दस्तावेज़, PDF या वेब पेज ही सुधारा जा सकता है।", diff --git a/surfsense_local/frontend/translations/ja.json b/surfsense_local/frontend/translations/ja.json index 880a3cd5bd..6fff2c9f4c 100644 --- a/surfsense_local/frontend/translations/ja.json +++ b/surfsense_local/frontend/translations/ja.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "チャットモデルと音声モデルが必要です。", "studio_error_needs_chat_image": "チャットモデルと画像モデルが必要です。", "studio_error_needs_image": "画像モデルが必要です。", - "studio_error_needs_image_edit": "画像編集モデルが必要です。", "studio_error_no_sources": "ソースを1つ以上選んでください。", "studio_error_not_editable": "編集できるのは、Studioで作成した画像とインフォグラフィックだけです。", "studio_error_not_refinable": "改訂できるのは、Studioで作成したWordドキュメント、PDF、Webページだけです。", diff --git a/surfsense_local/frontend/translations/ko.json b/surfsense_local/frontend/translations/ko.json index 205a8104d6..1bbf147437 100644 --- a/surfsense_local/frontend/translations/ko.json +++ b/surfsense_local/frontend/translations/ko.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "채팅 모델과 음성 모델이 필요합니다.", "studio_error_needs_chat_image": "채팅 모델과 이미지 모델이 필요합니다.", "studio_error_needs_image": "이미지 모델이 필요합니다.", - "studio_error_needs_image_edit": "이미지 편집 모델이 필요합니다.", "studio_error_no_sources": "소스를 하나 이상 선택하세요.", "studio_error_not_editable": "Studio에서 만든 이미지와 인포그래픽만 편집할 수 있습니다.", "studio_error_not_refinable": "Studio에서 만든 Word 문서, PDF, 웹 페이지만 다듬을 수 있습니다.", diff --git a/surfsense_local/frontend/translations/pt-BR.json b/surfsense_local/frontend/translations/pt-BR.json index dd4960937e..51227e7f40 100644 --- a/surfsense_local/frontend/translations/pt-BR.json +++ b/surfsense_local/frontend/translations/pt-BR.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "São necessários um modelo de chat e um modelo de áudio.", "studio_error_needs_chat_image": "São necessários um modelo de chat e um modelo de imagem.", "studio_error_needs_image": "É necessário um modelo de imagem.", - "studio_error_needs_image_edit": "É necessário um modelo de edição de imagem.", "studio_error_no_sources": "Escolha pelo menos uma fonte.", "studio_error_not_editable": "Só é possível editar uma imagem ou um infográfico criados no Studio.", "studio_error_not_refinable": "Só é possível refinar um documento do Word, um PDF ou uma página web criados no Studio.", diff --git a/surfsense_local/frontend/translations/ru.json b/surfsense_local/frontend/translations/ru.json index 6ac10892de..2fb16f6976 100644 --- a/surfsense_local/frontend/translations/ru.json +++ b/surfsense_local/frontend/translations/ru.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "Требуются модель для чата и модель для аудио.", "studio_error_needs_chat_image": "Требуются модель для чата и модель для изображений.", "studio_error_needs_image": "Требуется модель для изображений.", - "studio_error_needs_image_edit": "Требуется модель редактирования изображений.", "studio_error_no_sources": "Выберите хотя бы один источник.", "studio_error_not_editable": "Изменить можно только изображение или инфографику, созданные в Studio.", "studio_error_not_refinable": "Доработать можно только документ Word, PDF или веб-страницу, созданные в Studio.", diff --git a/surfsense_local/frontend/translations/zh-CN.json b/surfsense_local/frontend/translations/zh-CN.json index 8ce5fa13a6..376fbfa97a 100644 --- a/surfsense_local/frontend/translations/zh-CN.json +++ b/surfsense_local/frontend/translations/zh-CN.json @@ -1941,7 +1941,6 @@ "studio_error_needs_chat_audio": "需要对话模型和语音模型。", "studio_error_needs_chat_image": "需要对话模型和图像模型。", "studio_error_needs_image": "需要图像模型。", - "studio_error_needs_image_edit": "需要图像编辑模型。", "studio_error_no_sources": "请至少选择一个来源。", "studio_error_not_editable": "只能编辑在 Studio 中创建的图像或信息图。", "studio_error_not_refinable": "只能完善在 Studio 中创建的 Word 文档、PDF 或网页。",