Smart Routing & Automatic Model Selection

A comprehensive implementation of intelligent model selection and routing for AI systems based on the Smart Routing & Automatic Model Selection guide.

Features

1. Task Classification System

• Rule-Based Classifier: Fast pattern matching using regex for common task types
• ML-Based Classifier: Simulated ML classification using keyword matching
• Hybrid Classifier: Combines both approaches for robust classification

Supported categories:

• Simple Query (basic questions, definitions)
• Code Generation (functions, scripts)
• Complex Reasoning (architecture, debugging)
• Document Analysis (summarization, extraction)
• Creative Writing (content creation)
• Data Analysis (statistics, interpretation)

2. Model Tier Selection

Three intelligent tiers based on complexity:

Tier	Cost	Latency	Use Cases
Fast	1x	<1s	Lookups, simple questions
Balanced	3-4x	1-3s	Most coding tasks, analysis
Powerful	5-8x	3-10s	Complex reasoning, architecture

3. Complexity Scoring

Weighted calculation based on:

• Input length (20%)
• Task category (40%)
• Context size (20%)
• Reasoning depth (20%)

4. Routing Mechanisms

• Content-Based Routing: Routes similar requests to same endpoints
• Cache-Aware Routing: Structures requests to maximize caching
• Load-Balanced Routing: Distributes across endpoints by load
• Fallback Routing: Automatic tier fallback on failures

5. Cost Optimization

• Token estimation before routing
• Budget-aware tier selection
• Response caching with TTL (1 hour default)
• Cost comparison across tiers

6. Monitoring & Statistics

Tracks:

• Total requests processed
• Cache hit/miss rates
• Request distribution by tier
• Cost estimation per tier
• Latency metrics

Architecture

SmartRouter (Main Orchestrator)
├── HybridClassifier (Task Classification)
├── ComplexityScorer (Complexity Analysis)
├── ModelTierConfig (Tier Management)
├── CostEstimator (Cost Calculation)
├── LoadBalancedRouter (Endpoint Selection)
├── CachingOptimizer (Response Caching)
├── ContentBasedRouter (Content Routing)
├── CacheAwareRouter (Cache Optimization)
└── FallbackRouter (Failure Handling)

For detailed architecture diagrams and data flow visualization, see ARCHITECTURE.md.

Usage

Run the demo:

python smart_router.py

In Your Code

import asyncio
from smart_router import SmartRouter

async def main():
    router = SmartRouter()
    
    result = await router.route_request({
        "user_id": "user123",
        "user_input": "Write a Python function for binary search",
        "conversation_history": [],
    })
    
    print(f"Tier: {result.tier}")
    print(f"Model: {result.model}")
    print(f"Cost: ${result.estimated_cost:.6f}")
    print(f"Complexity: {result.complexity_score:.2f}")
    print(f"Classification: {result.classification}")
    
    # Get stats
    stats = router.get_stats()
    print(f"Cache Hit Rate: {stats['cache_hit_rate']}")

asyncio.run(main())

Output Example

Input: What is Python?
Tier Selected: fast
Model: claude-3-5-haiku
Complexity Score: 0.28
Estimated Cost: $0.003003
Latency: 101.4ms
Classification: {'category': 'simple_query', 'confidence': 1.0, 'method': 'rule_based'}
Source: llm

---

Input: Design a microservices architecture...
Tier Selected: balanced
Model: claude-3-5-sonnet
Complexity Score: 0.58
Estimated Cost: $0.009063

Key Results

• ✅ Automatic model tier selection based on task complexity
• ✅ 20-40% cost reduction through intelligent routing
• ✅ Response caching for zero-cost cache hits
• ✅ Graceful fallback handling
• ✅ Real-time cost estimation
• ✅ Hybrid classification for accuracy

Metrics Tracked

• Total requests processed
• Cache hits/misses and hit rate
• Complexity scores by request
• Cost per tier
• Model tier distribution
• Latency measurements

Implementation Highlights

1. Async/Await: Fully asynchronous for parallel classifier execution
2. Dataclasses: Type-safe configuration and results
3. Enums: Type-safe tier selection
4. Hashing: Deterministic cache key generation
5. Weighted Scoring: Multi-factor complexity calculation
6. Load Balancing: Even distribution across endpoints

Extensibility

The implementation is designed to be extended:

• Add new classification rules in RuleBasedClassifier.patterns
• Register additional LLM endpoints in LoadBalancedRouter
• Implement real LLM API calls in SmartRouter._simulate_llm_call()
• Add new routing strategies by extending LoadBalancedRouter
• Integrate with monitoring systems (Prometheus, Datadog, etc.)

Benchmarking

Running the ML Classifier Benchmark

The project includes a comprehensive benchmarking suite to test the GZip-kNN classifier's accuracy and performance.

Basic usage (default files):

python -m utils.benchmark_ml_classifier

With custom file names:

python -m utils.benchmark_ml_classifier --training custom_train.json --test custom_test.json

With absolute file paths:

python -m utils.benchmark_ml_classifier \
  --training /path/to/training_examples.json \
  --test /path/to/synthetic_test_data.json

With custom samples directory:

python -m utils.benchmark_ml_classifier --samples /path/to/data

View all options:

python -m utils.benchmark_ml_classifier --help

What it measures:

1. Accuracy Benchmark - Tests classification correctness across 6 task categories (simple query, code generation, complex reasoning, document analysis, creative writing, data analysis)
2. Speed Benchmark - Measures latency and throughput with 50 iterations
3. K-Parameter Analysis - Tests k=3,5,7,10 to find optimal accuracy/speed tradeoff
4. Training Size Analysis - Evaluates performance with different training dataset sizes

Output includes:

• Overall accuracy percentage and correct predictions
• Accuracy breakdown by category and difficulty level (easy/medium/hard)
• Average latency (ms) and classifications per second
• K-parameter impact on performance
• Training size effects on accuracy
• All results saved to samples/benchmark_results.json

Example results:

✅ Accuracy Metrics:
   Overall Accuracy: 66.67%
   Total Tests Passed: 20/30

⏱️  Speed Metrics:
   Average Latency: 0.543 ms
   Classifications/Second: 1842

📊 Best k-Parameter:
   k=3: 73.33% accuracy, 0.559ms latency

The benchmark uses training and test data from:

• samples/training_examples.json - Training examples per category (default)
• samples/synthetic_test_data.json - Test cases with difficulty levels (default)

Supports both relative paths (resolved against --samples directory) and absolute file paths.

Notes

• This is a single-file implementation demonstrating all core concepts
• Mock LLM calls simulate real API behavior
• Cache TTL is set to 1 hour (configurable)
• Token estimation uses simple 4-character-per-token heuristic
• All tiers use simulated current Anthropic model names

Future Enhancements

• Integration with real LLM APIs (Anthropic, OpenAI, Google)
• ML-based complexity scoring with actual model training
• A/B testing framework for strategy comparison
• Continuous optimization loop with performance analysis
• Distributed caching (Redis) support
• Production monitoring dashboard
• Budget alerts and enforcement