ScamShield NG is a multilingual scam and phishing message analysis system designed to explore how traditional machine-learning classification can be combined with local-language semantic analysis for Nigerian communication contexts.
The project began as a conventional SMS spam classifier and has evolved into a security-focused prototype combining:
- Traditional machine learning
- TF-IDF text representation
- Logistic Regression
- FastAPI
- N-ATLaS multilingual semantic analysis
- Local GGUF model inference through llama.cpp
- Structured JSON model outputs
- Controlled multilingual evaluation
- Reproducible evaluation artifacts
Important: ScamShield NG is a research and portfolio prototype. Its evaluation results are not production-level security guarantees.
Traditional SMS spam classifiers can perform well on datasets similar to their training data but may struggle with:
- Nigerian English expressions
- Yoruba messages
- Hausa messages
- Igbo messages
- Context-dependent scam language
- Credential and OTP theft attempts
- Financial impersonation
- Fake rewards and prize messages
- Social-engineering language
ScamShield NG therefore explores a two-layer approach:
USER MESSAGE
|
v
+---------------+
| FastAPI API |
+-------+-------+
|
+-----------+-----------+
| |
v v
/predict /analyze
| |
v v
ML Baseline N-ATLaS
| Semantic
| Analysis
v |
Spam Probability v
| Structured Result
| |
+-----------+-----------+
|
v
Security Analysis
The original classifier uses a supervised text-classification pipeline:
Raw Message
|
v
TF-IDF Vectorization
|
v
Logistic Regression
|
v
SPAM / LEGITIMATE
|
v
Spam Probability
The saved model is:
models/final_phishing_classifier.joblib
The pipeline uses:
TfidfVectorizer- Unigram and bigram features
LogisticRegression- Balanced class weighting
The baseline remains important because it provides a measurable reference point for evaluating the multilingual semantic layer.
ScamShield NG integrates N-ATLaS, a multilingual Llama-3-based model supporting:
- English
- Hausa
- Igbo
- Yoruba
The model is served locally using:
llama.cpp
|
v
GGUF Q4_K_M
|
v
N-ATLaS
The quantized model makes local CPU-based experimentation possible on hardware where loading the full-precision 8B model would be impractical.
The N-ATLaS integration analyzes:
- Phishing
- Fraud
- Financial scams
- OTP or credential theft
- Impersonation
- Malicious links
- Fake prizes or rewards
- Loan scams
- Account takeover attempts
- Coercive financial requests
- Other social-engineering patterns
The semantic analysis endpoint uses a JSON schema to enforce a predictable response structure.
Example structure:
{
"verdict": "SCAM",
"confidence": 0.9,
"scam_type": "credential or OTP theft",
"explanation": "The message requests an OTP in connection with a claimed financial reward.",
"user_language": "Yoruba"
}The schema requires:
verdict
confidence
scam_type
explanation
user_language
The schema controls the format and allowed value types. It does not guarantee that the model's semantic judgment is correct.
Returns service information and the available analysis services.
Runs the traditional machine-learning classifier.
Request:
{
"message": "Congratulations! You have won a free prize!"
}Example response:
{
"prediction": "SPAM",
"spam_probability": 0.94
}Runs the N-ATLaS multilingual semantic analysis.
Request:
{
"message": "E ku oriire! O ti gba owo eye. Fi OTP re ranse lati gba owo naa."
}The endpoint returns a structured security analysis containing the verdict, confidence, scam type, explanation, and detected language.
Interactive API documentation is available through the FastAPI /docs endpoint when the application is running.
A controlled evaluation was created to compare the original ML baseline against ScamShield NG.
The evaluation contains:
- English
- Nigerian English
- Yoruba
- Hausa
- Igbo
Each language contains scam and legitimate control examples.
| System | Correct | Accuracy |
|---|---|---|
| Original ML baseline | 7 / 10 | 70% |
| ScamShield NG | 9 / 10 | 90% |
This represents a:
20 percentage-point improvement
on this particular controlled evaluation.
An especially important observation was that the original baseline missed all three non-English scam examples in the controlled set, while ScamShield NG detected all three.
These results should not be interpreted as production accuracy because the evaluation set is intentionally small and controlled.
Detailed evidence is available in:
data/evaluation/SCAMSHIELD_CONTROLLED_EVALUATION.md
Results:
data/evaluation/baseline_controlled_results.csv
data/evaluation/scamshield_controlled_results.csv
The project also includes an independent evaluation using the ExAIS SMS Spam Dataset.
The evaluation process includes:
External Dataset
|
v
Normalization
|
v
Baseline Inference
|
v
Performance Analysis
|
v
Error Analysis
The normalized evaluation data and benchmark results are stored under:
data/evaluation/
The original downloaded archive and extracted raw source files are intentionally excluded from Git.
This separation helps keep the repository lightweight and distinguishes external source material from project-generated evaluation artifacts.
ScamShield NG uses the term SCAM for messages exhibiting meaningful evidence of deceptive, fraudulent, malicious, or coercive intent.
The original ML dataset primarily represents SPAM versus HAM.
These concepts overlap but are not identical.
For example:
SPAM
|
+-- promotional messages
+-- unsolicited advertising
+-- bulk messages
+-- suspicious marketing
SCAM
|
+-- credential theft
+-- OTP theft
+-- impersonation
+-- financial fraud
+-- fake rewards
+-- malicious social engineering
Therefore, benchmark results must be interpreted according to the dataset and task being evaluated.
The current prototype has several known limitations.
A legitimate Nigerian English financial-transfer message was incorrectly classified as SCAM during the controlled evaluation.
One legitimate Igbo message received the correct LEGITIMATE verdict but was incorrectly identified as Yoruba.
This demonstrates an important distinction between:
Correct security verdict
and:
Correct language identification
Both need to be evaluated independently.
The controlled multilingual benchmark contains only 10 messages.
It is useful for demonstrating system behavior but is not large enough to establish production performance.
The current N-ATLaS integration focuses on:
- English
- Nigerian English
- Yoruba
- Hausa
- Igbo
Other Nigerian languages are not currently covered by the semantic layer.
LLM-based analysis can produce incorrect interpretations, especially when messages are ambiguous, abbreviated, deliberately obfuscated, or outside the model's strongest linguistic capabilities.
The system does not currently perform comprehensive:
- URL reputation analysis
- Domain analysis
- Sender identity verification
- Attachment analysis
- Device intelligence
- Threat-intelligence correlation
- Real-time fraud intelligence
ScamShield NG is intentionally being developed around measurable evidence rather than a single accuracy number.
The evaluation strategy is:
Baseline
|
v
Controlled Multilingual Tests
|
v
Independent Dataset Evaluation
|
v
Error Analysis
|
v
Identify Weaknesses
|
v
Improve Dataset / Architecture
|
v
Re-evaluate
This makes it possible to distinguish between:
- Model improvement
- Dataset effects
- Language limitations
- False positives
- False negatives
- Semantic failures
- Language-identification failures
scamshield-ng/
|-- app/
| `-- main.py
|
|-- data/
| |-- raw/
| |-- processed/
| `-- evaluation/
| |-- README.md
| |-- SCAMSHIELD_ARCHITECTURE.md
| |-- SCAMSHIELD_CONTROLLED_EVALUATION.md
| |-- scamshield_schema.json
| |-- baseline_controlled_results.csv
| |-- scamshield_controlled_results.csv
| |-- exais_normalized.csv
| |-- exais_baseline_results.json
| |-- parse_exais.py
| |-- run_exais_baseline.py
| `-- analyze_exais_errors.py
|
|-- frontend/
|
|-- models/
| `-- final_phishing_classifier.joblib
|
|-- notebooks/
|
|-- src/
| |-- predict.py
| `-- natlas/
| `-- client.py
|
|-- tests/
|
|-- baseline_check.py
|-- requirements.txt
|-- .gitignore
`-- README.md
python -m venv .venvActivate it:
.\.venv\Scripts\Activate.ps1Install the project dependencies:
pip install -r requirements.txtRun the tests:
python -m pytestStart the FastAPI application:
python -m uvicorn app.main:app --reloadThe API will be available at:
http://127.0.0.1:8000
FastAPI documentation:
http://127.0.0.1:8000/docs
ScamShield NG's semantic analysis requires the N-ATLaS model server.
The local architecture is:
ScamShield FastAPI
|
| HTTP
v
llama.cpp server
|
v
N-ATLaS GGUF model
The model server runs independently from the FastAPI application.
Example:
127.0.0.1:8080
The ScamShield API then exposes the semantic analysis through:
POST /analyze
The N-ATLaS model files are not stored in this repository.
- Python
- scikit-learn
- NumPy
- Pandas
- TF-IDF
- Logistic Regression
- N-ATLaS
- Llama-3 architecture
- GGUF
- llama.cpp
- FastAPI
- Uvicorn
- Pydantic
- Requests
- HTML5
- CSS3
- JavaScript
- Pytest
- Controlled multilingual evaluation
- Independent dataset evaluation
- Error analysis
- Git
- GitHub
- Jupyter Notebook
- PowerShell
The original application is deployed using separate frontend and FastAPI backend services.
The project evolved through several stages:
Traditional Rule-Based Detection
|
v
SMS Spam Machine Learning
|
v
FastAPI Deployment
|
v
Independent Evaluation
|
v
Nigerian Multilingual Evaluation
|
v
N-ATLaS Integration
|
v
ScamShield NG
The goal is not simply to replace the original classifier with an LLM.
Instead, the original model is retained as a measurable baseline while the multilingual semantic layer is evaluated against it.
Planned improvements include:
- Build a larger verified Nigerian multilingual evaluation dataset.
- Add a dedicated language-identification component.
- Improve Nigerian English handling.
- Develop hybrid decision logic between the ML baseline and semantic model.
- Evaluate additional Nigerian languages.
- Introduce URL and domain intelligence.
- Add explainable security indicators.
- Add model and evaluation versioning.
- Add monitoring and audit logging.
- Evaluate LoRA or QLoRA fine-tuning after sufficient high-quality multilingual data becomes available.
- Add larger independent test sets.
- Evaluate precision, recall, F1, confusion matrices, and per-language performance.
ScamShield NG is intended for:
- Research
- Education
- Cybersecurity experimentation
- Multilingual AI evaluation
- Portfolio demonstration
- Community-oriented security research
It should not be treated as an autonomous authority for determining whether a communication is safe.
Security decisions should consider additional evidence and, where appropriate, human review.
Ogunlade Faith Kayode
AI - Cybersecurity - Machine Learning - Digital Solutions
GitHub: smartexploit
This project is an experimental AI and cybersecurity system.
The reported evaluation results come from specific datasets and controlled tests. They do not represent guaranteed real-world detection performance.
The system may produce false positives and false negatives and should not be used as the sole mechanism for making high-impact security or financial decisions.