Skip to content
uscoolPublic

About

How domain shift breaks spam detectors: SMS-trained rule-based, TF-IDF, and BERT models tested on Enron and phishing email, with drift measured via JS divergence and PAD.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

Impact of Domain Shift on Spam & Phishing Detection

This project investigates how Domain Shift (Covariate and Concept Shift) affects the performance of NLP models when moving from a source domain (SMS) to target domains (Enron Corporate Email and Phishing).

Experimental Setup

The models were trained on the SMS Spam Collection and evaluated against:

  • Enron Dataset: Corporate email communication.
  • Phishing Dataset: Malicious emails with deceptive intent.

Experiments:

  • Exp A: SMS Test Set (Base Performance)
  • Exp B: Enron Dataset (Domain Shift)
  • Exp C: Phishing Dataset (Domain & Concept Shift)
  • Exp D: SMS 60/40 Split (Sanity Check)

Notebook Overview

1. Drift Diagnostics (Drift_Check.ipynb)

Provides statistical proof of data degradation:

  • Enron: JS Divergence (Ham: 0.2155, Spam: 0.2610) and high PAD (1.65). Results in covariate shift in both classes with a massive drop in spam recall (-0.7456).
  • Phishing: JS Divergence (Ham: 0.3801, Spam: 0.3442). Results in mixed shift: covariate shift plus strong concept shift, evidenced by "confidently wrong" predictions (HCE spam ≈ 0.99).

2. Rule-Based Baseline (RB.ipynb)

  • Observations: Performance collapsed on Phishing due to longer text and formal language not present in SMS rules.
  • Enron: Remained relatively stable as underlying spam patterns (keywords) were more similar despite length differences.

3. TF-IDF Analysis (TF_IDF.ipynb)

Uses statistical vectorization to measure the robustness of word frequencies.

  • Key Confusion Matrix Results (Exp B & C):
    • Enron (Exp B): Failed significantly on spam detection (only 383 correctly identified vs 1944 missed).
    • Phishing (Exp C): High false negative rate (7005 spam missed) indicating the statistical model could not generalize to phishing vocabulary.

4. BERT Transformer (BERT.ipynb)

Fine-tuning bert-base-uncased to track performance "Drift" (Original Metric - Target Metric).

  • Quantifies the exact loss in Accuracy, Precision, Recall, and F1-Score when the transformer is exposed to shifted data.

5. Frozen Embeddings (BERT_metal.ipynb)

A specialized experiment using Frozen BERT embeddings with a trainable classifier head.

  • Purpose: This isolates spam-specific adaptation. It tests if the pre-trained knowledge of BERT is enough to overcome drift without deep weight updates.

Key Findings

  1. Covariate Shift is present in both target domains due to changes in message length and vocabulary.
  2. Concept Shift is dominant in Phishing data; the model retains high confidence while making incorrect predictions because the "nature" of the spam has changed.
  3. SMS-trained models are highly unreliable for Phishing detection without domain-specific re-training.

Usage

pip install -r requirements.txt

Run Drift_Check.ipynb first for the drift diagnostics, then the model notebooks (RB.ipynb, TF_IDF.ipynb, BERT.ipynb, BERT_metal.ipynb) in any order.

About

How domain shift breaks spam detectors: SMS-trained rule-based, TF-IDF, and BERT models tested on Enron and phishing email, with drift measured via JS divergence and PAD.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages