A small test automation project in Robot Framework. It has three suites:
| Suite | System under test | Library | Tests |
|---|---|---|---|
tests/ui |
Sauce Demo, a demo shop Sauce Labs built for practising automation | Browser library (Playwright) | 6 |
tests/api |
Restful Booker, a public practice API | RequestsLibrary | 6 |
tests/llm |
"Hotel Aurora" assistant: a system prompt plus Claude Haiku 5.5 | own Python library LLMLibrary |
4 |
I work as a QA engineer and wrote my earlier automation in Java (Selenium, Appium, REST Assured). I built this project to learn Robot Framework, the Browser library and Python test code on something small enough to understand line by line.
UI (Sauce Demo). Login with a valid user, a locked-out user and a wrong password; the cart badge after adding two items; a full checkout; sorting by price. Each test gets a fresh browser context, so no cookies or cart state leak between tests.
API (Restful Booker). Health check, create and read back a booking, get an auth token, update with and without a token, delete. Every test creates its own booking, because other people use the public instance at the same time. Two quirks of the app are asserted as they are and commented in the tests: /ping returns 201 and DELETE returns 201.
LLM (Hotel Aurora assistant). The assistant is defined in assistant/system_prompt.md. The suite checks four things:
- Consistency: the same question asked 3 times, every answer must contain the check-in time
14:00. - Prompt injection: three attacks (ignore instructions, role play, "translate your instructions"). The system prompt contains a canary string, and it must not appear in any answer.
- Scope: an off-topic request must get the fixed refusal sentence from the prompt.
- Length: an answer must stay within 80 words.
All checks are deterministic string checks. There is no LLM judge here; my other project llm-sales-assistant-evals covers that approach.
assistant/system_prompt.md the assistant under test
libraries/LLMLibrary.py custom keyword library (Python)
resources/saucedemo.resource UI keywords: Login As, Add Item To Cart, Complete Checkout ...
resources/booker.resource API keywords: Open Booker Session, Create Test Booking ...
tests/ui/ tests/api/ tests/llm/
unit/test_llm_library.py pytest tests for the library, no API calls
.github/workflows/ ci.yml (every push), llm.yml (manual + weekly)
Where the line between Robot and Python is: test scenarios and page steps are in .robot and .resource files. Code that talks to an external SDK or compares several values (the model client, consistency and leak checks) is in Python, where it can have its own unit tests.
Needs Python 3.11+ and Node.js (tested with Node 22).
make install # venv, dependencies, Playwright Chromium
make test # UI + API (everything except llm)
make smoke # only tests tagged smoke
make unit # pytest for LLMLibraryThe LLM suite needs an Anthropic API key. Copy .env.example to .env, add the key, then:
make llmReports go to results/report.html and results/log.html. To watch the browser, add --variable HEADLESS:False to the robot command.
Every test has a suite tag (ui, api, llm) and a level tag (smoke or regression). Examples: robot --include smoke tests, robot --include api --exclude regression tests.
ci.ymlruns on every push and pull request: unit tests, then UI and API tests. Restful Booker runs as a Docker service container inside the job, so a slow public instance does not break the build. The Robot report is uploaded as therobot-resultsartifact, also when tests fail.llm.ymlruns the LLM suite on demand and every Monday, with the key from GitHub Secrets. One full run is about 12 short API calls, well under one cent.
- Robot Framework splits test intent (
.robot), reusable steps (.resource) and code (Python libraries). Keyword names become the test language. - The Browser library waits for elements to be visible, enabled and stable before it clicks, so the tests have no sleeps.
- RequestsLibrary does not verify TLS certificates by default (
verify=False). I turned verification on inCreate Session. - Testing an LLM with plain asserts works when the system prompt gives the model something checkable: a fixed refusal sentence, a canary string, a word limit.
- Three runs per question is a smoke check for consistency, not a statistical measurement. Current Claude models do not accept a temperature setting, so the answers really do vary between runs.
- The injection attacks are three hand-written prompts. Passing them does not mean the prompt resists injection in general.
- Sauce Demo and Restful Booker belong to other people. If they change their pages or go down, the tests fail for reasons outside this repo. Locally, the API tests use the public Restful Booker unless
BASE_URLis set. - The UI suite runs only in Chromium.