An autonomous webpage monitoring agent.
Argus is an intelligent web browsing and data extraction system that leverages an autonomous browsing agent and LLM-powered data extraction to collect structured information (records) from websites. It processes URLs asynchronously via a task queue, stores results in Redis, and notifies users via Telegram.
- Autonomous Web Browsing: Agent navigates websites based on natural language instructions
- Intelligent Data Extraction: LLM-driven extraction of records
- Dynamic Schema: Infers data fields from user prompts (no predefined schemas needed)
- Async Task Queue: Celery-based task processing
- Retrieval: Redis-backed data store for record retrieval
- Telegram Notifications: Real-time results delivered via Telegram
User
↓
submit.sh
↓
Celery task submission
↓
Redis Broker (Celery queue)
↓
Worker Process
↓
Argus Agent
↓
browse ──→ Browser Agent ──→ Web Content
↓ ↓
extract ──→ Gemini AI ─────→ Records
↓ ↓
upsert ────────────────────→ Redis Storage
↓
Compare record IDs
↓
IDs changed?
╱ ╲
Yes No
↓ ↓
Telegram No notification
(optional)
- Python 3.14+
- Docker
- Google Generative AI API key
- Telegram Bot token and chat ID (optional, for notifications)
- Clone the repository:
git clone https://github.com/m-maresch/argus
cd argus- Create a virtual environment:
python -m venv .
source bin/activate- Install dependencies:
pip install -r requirements.txt- Start Redis (using Docker):
docker-compose up -dCreate a .env file in the project root:
# Google Generative AI
GOOGLE_API_KEY=your_google_api_key_here
# Telegram (optional)
TELEGRAM_BOT_TOKEN=your_telegram_bot_token
TELEGRAM_CHAT_ID=your_telegram_chat_id./start-argus.shThis starts the Celery worker that processes queued tasks.
Copy the prompt template to prompt.md, then update prompt.md with the instructions for your data collection:
cp prompt-template.md prompt.mdFor example:
1. Search for rental apartments in: Brooklyn, NY with rent under $3000
2. Collect the title, price, bedrooms, bathrooms, and location of up to 5 apartments matching the criteria. If no matching apartments are found, then stop here.
3. Based on the collected apartments, extract 1 record per apartment consisting of title, price, bedrooms, bathrooms, and location./submit.sh "https://example-rental-listings.com/brooklyn"This queues the URL for processing by the worker.
./list-records.sh "https://example-rental-listings.com/brooklyn"Example output:
{
"apt_001": {
"id": "apt_001",
"title": "Modern 2BR in Williamsburg",
"price": "$2,500/month",
"bedrooms": "2",
"bathrooms": "1.5",
"location": "Williamsburg, Brooklyn"
},
"apt_002": {
"id": "apt_002",
"title": "Spacious 3BR Loft",
"price": "$2,950/month",
"bedrooms": "3",
"bathrooms": "2",
"location": "Park Slope, Brooklyn"
}
}Argus is designed to run on a server with scheduled job submissions via cron. Once deployed, the workflow is:
- Schedule
submit.shwith cron to periodically submit URLs for processing - Worker runs continuously listening for jobs on the queue
- Get results via Telegram notifications or by querying with
list-records.sh
- User submits URL via
submit.sh→ Celery enqueues task - Worker picks up task
- Agent executes:
- Browse: Browser agent navigates the URL using natural language prompt
- Extract: LLM parses browser results and extracts records
- Store: Extracted records are saved to Redis with URL-based keys. Records are identified by their
idfield; IDs no longer present in a later result are removed. - Notify: Telegram is notified only when the set of record IDs for a URL changes. Changes to other fields on an existing record do not trigger a notification.
- User can retrieve data via
list-records.sh
Argus compares the IDs of the newly extracted records with the IDs already stored for the URL:
- A newly discovered ID triggers a Telegram notification.
- A previously stored ID that disappears triggers a Telegram notification and is removed from Redis.
- Changes to fields such as price, title, or location do not trigger a Telegram notification.
| Variable | Description | Required |
|---|---|---|
GOOGLE_API_KEY |
Google Generative AI API key | Yes |
TELEGRAM_BOT_TOKEN |
Telegram bot token for notifications | No |
TELEGRAM_CHAT_ID |
Telegram chat ID for notifications | No |
This project is licensed under the MIT License. See the LICENSE file for details. Third-party library notices are documented in THIRD-PARTY-NOTICES.txt.
This project was developed with the assistance of Google Gemini 3 and GitHub Copilot.
Thanks to everyone contributing to any of the following projects:
- browser-use
- langchain
- langchain-google-genai
- Celery
- redis-py
- python-dotenv
- Pydantic
- Requests