diff --git a/CLAUDE.md b/CLAUDE.md index 4704b0a..58a8454 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,7 +24,7 @@ Docker-based E2E tests live in `tests/e2e/` (human guide: `docs/testing.md`). Ho ```bash tests/e2e/run.sh # gemini | azure | n8n | infomaniak | filters | all -tests/e2e/run.sh --image v0.11.3-slim gemini # A/B another Open WebUI release (default v0.11.4-slim) +tests/e2e/run.sh --image v0.11.4-slim gemini # A/B another Open WebUI release (default v0.12.0-slim) tests/e2e/run.sh --ref # test files from a git ref; --src DIR / --file PATH=FILE tests/e2e/run.sh --keep --only 'azure.rag' azure # keep container; then --reuse --name tests/e2e/run.sh --strict-known all # as in CI: a tagged check that passes while its marker applies is FAIL @@ -33,7 +33,7 @@ tests/e2e/check_owui_api.sh [TAG|latest] # static check of open_webui import - **What to run:** changed `pipelines/google/*` → `gemini`; `pipelines/azure/*` → `azure`; `pipelines/n8n/*` → `n8n`; `pipelines/infomaniak/*` → `infomaniak`; `filters/*` → `filters` (+ `gemini` for `google_search_tool`); `tests/e2e/**` → `all`. New Open WebUI release → `check_owui_api.sh latest`, then `run.sh --image all`. - **Results:** `PASS` / `FAIL` / `KNOWN`. Exit 1 only on FAIL; exit 2 = setup error (Docker/container, leftover `NAME-data` volume without `--force`, an option without value, unknown suite, git ref or `--file` path, `--only` regex matching no group, driver error); 130/143 after Ctrl-C/SIGTERM. KNOWN = fails the way a bug registered in `tests/e2e/harness/known_.py` does (its `evidence` matches the detail, or its log signature shows up with `since=`); a tagged check that fails differently is FAIL. KNOWN does not fail the run; its fixing branch may still be an unmerged/unpublished PR. Markers are **version-gated**: each `KnownIssue` names its `file` and the `fixed_in` version; once the tested file has that version, a failure is FAIL ("regression of known …") and a pass is listed as an obsolete marker ("drop marker"). So when you fix such a bug, bump the file's `version:` and set `fixed_in` to it; drop the markers once the fix is merged. `--only` is a Python regex: `'gemini.(api|image)'`, not `\|`. Output (git-ignored): `tests/e2e/out//` with `driver.txt`, `results.json`, `summary.md`, `server.log`, `mocks.txt`, staged `functions/` (LF); results are written for interrupted or timed-out runs too (partial). -- **Baseline** (v0.11.4-slim, `--strict-known`): `main` → 0 FAIL, 0 KNOWN (no known bug registered), no obsolete markers. Check counts per suite and the scenario list: `docs/testing.md` → "What is tested" (keep the counts there only, not here). The `filters` suite needs network for the tiktoken encodings, edits the container's `/etc/hosts` and trust store, and its `offline` group needs a fresh container (no `--reuse`). +- **Baseline** (v0.12.0-slim, `--strict-known`; v0.11.4-slim and v0.11.3-slim give the same): `main` → 0 FAIL, 0 KNOWN (no known bug registered), no obsolete markers. Check counts per suite and the scenario list: `docs/testing.md` → "What is tested" (keep the counts there only, not here). The `filters` suite needs network for the tiktoken encodings, edits the container's `/etc/hosts` and trust store, and its `offline` group needs a fresh container (no `--reuse`). - **Driver checks:** `.timeout` (`E2E_SUITE_TIMEOUT`, default 900 s; whole run `E2E_TIMEOUT`, default 1800 s), `.crash`, `.no-checks`, `.interrupted`, `run.server-log` (errors or secrets logged outside every suite's scan window; names the `function_`), `run.mocks-log` (errors in `mocks.txt`). Ctrl-C cleans up within seconds; a killed `run.sh` leaves the container, the volume and an orphan driver in the container, which `--reuse` stops. Suites are discovered from `tests/e2e/suites/` (no registration). - **Add coverage** for every behaviour change: a `t.check(...)` in `tests/e2e/suites/.py` (API path `t.owui.chat`, browser path `t.browser().chat`, upstream request via `mock.last()`), mock behaviour in `tests/e2e/mocks/mock_.py`. Keep `tests/e2e` Ruff-clean. Evidence of a new known bug must not match unrelated failures: the CI meta-test (main's files with `E2E_MOCK_FAULT=500`) allows no KNOWN (`REQUEST_SIDE_KNOWN` in `.github/workflows/e2e.yml`, empty today, is only for bugs that can show nothing but the request sent upstream), so the evidence must include proof that the upstream answered (e.g. `HTTP 200` plus the mock's answer). - **Gotchas:** `valves/update` REPLACES all valves (use `update_valves`, which merges); refresh `/api/models?refresh=true` after model/filterIds changes; settings in the data volume override env vars; only the browser path (socket.io + saved chat) exercises event emitters, saved content, usage and sources; background tasks (`/api/v1/tasks/*`) call pipes with `__task__` set and `__event_emitter__=None`; the first Gemini install pip-installs `google-genai` (~30-60 s); in Git Bash prefix your own `docker exec`/`docker cp` with `MSYS_NO_PATHCONV=1` per command (do not export it; `run.sh` unsets it because `git -C /c/...` needs path conversion); never remove Docker images, only your containers/volumes. Open WebUI 0.12 loads functions as `function__<32 hex>` (the harness normalizes log lines to `function_`) and renamed `__metadata__["user_prompt"]` to `base_user_prompt`; `check_owui_api.sh` only checks names, so run all suites against every new Open WebUI release. diff --git a/docs/google-gemini-integration.md b/docs/google-gemini-integration.md index 8fdbee8..6bc2c4a 100644 --- a/docs/google-gemini-integration.md +++ b/docs/google-gemini-integration.md @@ -31,8 +31,8 @@ Open WebUI runs `pip` for this requirement when the function is saved and again Open WebUI skips the automatic install when `ENABLE_PIP_INSTALL_FRONTMATTER_REQUIREMENTS=false` or `OFFLINE_MODE=true` is set. In that case, and generally for production, multi-worker or multi-replica deployments, install the SDK yourself. For Docker, bake it into your image, which also avoids the in-process `websockets` downgrade: ```dockerfile -# Use the same tag you deploy, e.g. v0.11.4, v0.11.4-slim or main -FROM ghcr.io/open-webui/open-webui:v0.11.4 +# Use the same tag you deploy, e.g. v0.12.0, v0.12.0-slim or main +FROM ghcr.io/open-webui/open-webui:v0.12.0 RUN pip install --no-cache-dir "google-genai>=1.68.0,<3" ``` diff --git a/docs/testing.md b/docs/testing.md index 1b7d25b..22c8664 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -25,7 +25,7 @@ Container and volume are removed afterwards. - **Docker** (Docker Desktop on Windows/macOS, Docker Engine on Linux). - **bash**: Linux/macOS shell, or **Git Bash** on Windows. -- Network access for the first image pull (`ghcr.io/open-webui/open-webui:v0.11.4-slim`, +- Network access for the first image pull (`ghcr.io/open-webui/open-webui:v0.12.0-slim`, ~1 GB), for `pip install google-genai` when the Gemini pipe is installed and, in the `filters` suite, for the tiktoken encodings (`cl100k_base`, `o200k_base`) the driver downloads into the container's `TIKTOKEN_CACHE_DIR` (without them the suite fails). @@ -37,19 +37,19 @@ Open WebUI container, whose Python already ships `aiohttp`, `httpx`, ## Quick start ```bash -tests/e2e/run.sh # all suites against ghcr.io/open-webui/open-webui:v0.11.4-slim +tests/e2e/run.sh # all suites against ghcr.io/open-webui/open-webui:v0.12.0-slim tests/e2e/run.sh gemini # one suite (gemini, azure, n8n, infomaniak, filters) tests/e2e/run.sh azure n8n # several suites -tests/e2e/run.sh --image v0.11.3-slim gemini # another Open WebUI version +tests/e2e/run.sh --image v0.11.4-slim gemini # another Open WebUI version pixi run e2e gemini # same via pixi (Linux) ``` Typical output (abridged): ```text -starting owui-e2e-111759-1234 from ghcr.io/open-webui/open-webui:v0.11.4-slim +starting owui-e2e-111759-1234 from ghcr.io/open-webui/open-webui:v0.12.0-slim Open WebUI healthy after 20s (http://localhost:57765) -Open WebUI 0.11.4, suites: gemini, azure, n8n, infomaniak, filters +Open WebUI 0.12.0, suites: gemini, azure, n8n, infomaniak, filters function versions: google_gemini.py 1.20.0, azure_ai_foundry.py 3.0.1, n8n.py 2.3.1, ... === gemini === [PASS ] gemini.load pipelines/google/google_gemini.py loads (create, import, activate) @@ -57,14 +57,14 @@ function versions: google_gemini.py 1.20.0, azure_ai_foundry.py 3.0.1, n8n.py 2. ... [PASS ] gemini.tools.builtin built-in tool, browser stream=True: Open WebUI runs it, ... ... ---- gemini: 415s +--- gemini: 525s ... === infomaniak === ... [PASS ] infomaniak.models.name-prefix NAME_PREFIX valve changes the model names, and changing it back restores them ... -SUMMARY: 565 PASS, 0 FAIL, 0 KNOWN in 1213s -total runtime: 1249s +SUMMARY: 580 PASS, 0 FAIL, 0 KNOWN in 1263s +total runtime: 1299s output: tests/e2e/out/20261009-111759-owui-e2e-111759-1234 ``` @@ -97,7 +97,7 @@ SIGTERM the script exits with `130` / `143`; see [Interrupted runs](#interrupted | Option | Default | Meaning | | --- | --- | --- | | `[suite ...]`, `-s, --suites a,b` | `all` | `gemini`, `azure`, `n8n`, `infomaniak`, `filters` (the modules in `tests/e2e/suites/`), `all` | -| `-i, --image IMAGE` | `$OWUI_IMAGE` or `ghcr.io/open-webui/open-webui:v0.11.4-slim` | image, or just a tag (`v0.11.3-slim`) | +| `-i, --image IMAGE` | `$OWUI_IMAGE` or `ghcr.io/open-webui/open-webui:v0.12.0-slim` | image, or just a tag (`v0.11.4-slim`) | | `--only REGEX` | – | only scenario groups whose `.` matches (Python regex), e.g. `--only gemini.api`; see below | | `--ref REF` | – | test the function files as of a git ref (`git show REF:path`): a branch, tag or commit | | `--src DIR` | – | test the function files from another checkout / worktree (not together with `--ref`) | @@ -796,9 +796,9 @@ Python under `tests/e2e/` follows the repo's Ruff settings: ## Static Open WebUI API check ```bash -tests/e2e/check_owui_api.sh # against v0.11.4 +tests/e2e/check_owui_api.sh # against v0.12.0 tests/e2e/check_owui_api.sh latest # against the latest Open WebUI release -tests/e2e/check_owui_api.sh v0.12.0 # any tag or branch +tests/e2e/check_owui_api.sh v0.11.4 # any tag or branch ``` It reads every `from open_webui... import ...` and `import open_webui...` in @@ -836,7 +836,7 @@ and makes billed requests (about 40 small ones on flash models, with the default ```bash export GOOGLE_API_KEY=... # from the environment only, never as an argument -tests/e2e/smoke/gemini-tools.sh # v0.11.4-slim, gemini-3-flash-preview + gemini-2.5-flash +tests/e2e/smoke/gemini-tools.sh # v0.12.0-slim, gemini-3-flash-preview + gemini-2.5-flash tests/e2e/smoke/gemini-tools.sh --image v0.11.3-slim --only R3,R14 # some risks, another image tests/e2e/smoke/gemini-tools.sh --dry-run-mock # no key: the same run against the e2e Gemini mock ``` diff --git a/tests/e2e/check_owui_api.sh b/tests/e2e/check_owui_api.sh index cb065b3..5ceb1da 100755 --- a/tests/e2e/check_owui_api.sh +++ b/tests/e2e/check_owui_api.sh @@ -2,7 +2,7 @@ # Static compatibility check: does an Open WebUI release still provide every # open_webui.* symbol, method and injected __param__ the functions in this repo use? # -# usage: tests/e2e/check_owui_api.sh [TAG] TAG: v0.11.4 (default) | latest | any git ref +# usage: tests/e2e/check_owui_api.sh [TAG] TAG: v0.12.0 (default) | latest | any git ref # # Reads pipelines/**/*.py and filters/*.py, fetches the matching Open WebUI backend # modules at TAG (via `gh api` when authenticated, otherwise raw.githubusercontent.com @@ -25,7 +25,7 @@ set -euo pipefail HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) REPO=$(cd "$HERE/../.." && pwd) -TAG=${1:-v0.11.4} +TAG=${1:-v0.12.0} GH_REPO=open-webui/open-webui CACHE=$(mktemp -d) trap 'rm -rf "$CACHE"' EXIT diff --git a/tests/e2e/run.sh b/tests/e2e/run.sh index 5b88449..0e63512 100755 --- a/tests/e2e/run.sh +++ b/tests/e2e/run.sh @@ -25,7 +25,7 @@ HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) REPO=$(cd "$HERE/../.." && pwd) # Default Open WebUI image (.github/workflows/e2e.yml reads this line). -DEFAULT_IMAGE=ghcr.io/open-webui/open-webui:v0.11.4-slim +DEFAULT_IMAGE=ghcr.io/open-webui/open-webui:v0.12.0-slim FUNCTION_FILES=( pipelines/google/google_gemini.py diff --git a/tests/e2e/smoke/gemini-tools.sh b/tests/e2e/smoke/gemini-tools.sh index 34cb3a9..1f95565 100755 --- a/tests/e2e/smoke/gemini-tools.sh +++ b/tests/e2e/smoke/gemini-tools.sh @@ -4,7 +4,7 @@ # harness cannot confirm. Manual and opt-in: it needs a Gemini API key and # makes billed requests (about 40 small ones). # -# GOOGLE_API_KEY=... tests/e2e/smoke/gemini-tools.sh [--image v0.11.4-slim] \ +# GOOGLE_API_KEY=... tests/e2e/smoke/gemini-tools.sh [--image v0.12.0-slim] \ # [--model gemini-3-flash-preview] [--model-25 gemini-2.5-flash] # tests/e2e/smoke/gemini-tools.sh --dry-run-mock # no key: against the e2e mock #