This project investigates how Domain Shift (Covariate and Concept Shift) affects the performance of NLP models when moving from a source domain (SMS) to target domains (Enron Corporate Email and Phishing).
The models were trained on the SMS Spam Collection and evaluated against:
- Enron Dataset: Corporate email communication.
- Phishing Dataset: Malicious emails with deceptive intent.
- Exp A: SMS Test Set (Base Performance)
- Exp B: Enron Dataset (Domain Shift)
- Exp C: Phishing Dataset (Domain & Concept Shift)
- Exp D: SMS 60/40 Split (Sanity Check)
Provides statistical proof of data degradation:
- Enron: JS Divergence (Ham: 0.2155, Spam: 0.2610) and high PAD (1.65). Results in covariate shift in both classes with a massive drop in spam recall (-0.7456).
- Phishing: JS Divergence (Ham: 0.3801, Spam: 0.3442). Results in mixed shift: covariate shift plus strong concept shift, evidenced by "confidently wrong" predictions (HCE spam ≈ 0.99).
- Observations: Performance collapsed on Phishing due to longer text and formal language not present in SMS rules.
- Enron: Remained relatively stable as underlying spam patterns (keywords) were more similar despite length differences.
Uses statistical vectorization to measure the robustness of word frequencies.
- Key Confusion Matrix Results (Exp B & C):
- Enron (Exp B): Failed significantly on spam detection (only 383 correctly identified vs 1944 missed).
- Phishing (Exp C): High false negative rate (7005 spam missed) indicating the statistical model could not generalize to phishing vocabulary.
Fine-tuning bert-base-uncased to track performance "Drift" (Original Metric - Target Metric).
- Quantifies the exact loss in Accuracy, Precision, Recall, and F1-Score when the transformer is exposed to shifted data.
A specialized experiment using Frozen BERT embeddings with a trainable classifier head.
- Purpose: This isolates spam-specific adaptation. It tests if the pre-trained knowledge of BERT is enough to overcome drift without deep weight updates.
- Covariate Shift is present in both target domains due to changes in message length and vocabulary.
- Concept Shift is dominant in Phishing data; the model retains high confidence while making incorrect predictions because the "nature" of the spam has changed.
- SMS-trained models are highly unreliable for Phishing detection without domain-specific re-training.
pip install -r requirements.txt
Run Drift_Check.ipynb first for the drift diagnostics, then the model notebooks
(RB.ipynb, TF_IDF.ipynb, BERT.ipynb, BERT_metal.ipynb) in any order.