Semantic Concept-guided Attention Distillation for Vision-Language Models

Semantic Concept-guided Attention Distillation for Vision-Language Models
Xingye Chen¹, Yuanjie Shao^1,*, Xinge You¹, Changxin Gao², and Nong Sang^2
1National Anti-Counterfeit Engineering Research Center, School of Electronic Information and Communication, Huazhong University of Science and Technology, Wuhan, China
²School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China

---

Abstract

Prompt learning has emerged as an efficient paradigm for adapting vision-language models (VLMs), e.g., CLIP, to downstream tasks without extensive fine-tuning. However, existing methods still suffer from insufficient alignment between textual and visual representations. To address this limitation, we propose PromptCAD (Concept-guided Attention Distillation framework for prompting), which systematically integrates semantic knowledge from Large Language Models (LLMs) to enhance cross-modal alignment. Our framework operates through three key phases: 1) Concept Generation & Refinement: We first leverage LLMs to generate comprehensive category descriptions, from which representative semantic concepts are selected via clustering while eliminating irrelevant information. 2) Concept-guided Attention Distillation: A novel distillation mechanism aligns concept-guided attention maps (derived from LLM-generated semantics) with prompt-induced attention maps, guiding the adaptation process and improving text-visual modality alignment. 3) Multi-perspective Inference: During inference, a self-attention pooling strategy dynamically aggregates semantically relevant features from diverse local regions, enabling effective concept-guided reasoning.

Framework

Figure 1. Overview of our proposed PromptCAD framework. (a) Training pipeline employing concept-aware description clustering with LLM-generated descriptions, followed by concept-guided attention distillation to align visual and textual representations. (b) Inference pipeline utilizing multi-perspective feature extraction with self-attention mechanism for adaptive feature fusion.

Highlights

(1). A novel concept-guided attention distillation framework for Vision-Language Models that bridges the semantic-visual gap.

(2). We leverage LLM-generated descriptions through semantic clustering to extract concepts and implement concept-guided attention distillation for precise knowledge transfer.

(3). Multi-perspective inference strategy with self-attention pooling for robust visual-semantic reasoning.

Running

Preliminary

1. Create the environment and install Dassl.pytorch library. Please follow the instructions detailed in INSTALL.md.

2. Download the original ViT-B/16 CLIP model weights from the official OpenAI website. Then place these models in the ./clip folder.
   [ViT-B/16 CLIP]

3. Prepare the dataset. Please follow the instructions detailed in DATASETS.md.

Running PromptCAD

Base-to-Novel Experiments

1. Change the dataset path in scripts/promptcad/base2new.sh line 4 to your current path.

2. Run the commands below to train PromptCAD on the specified dataset.

For example:

nohup bash scripts/promptcad/base2new.sh -l 0.1 eurosat fgvc_aircraft ucf101 > base2new_0.1.log 2>&1 &

Cross-dataset Experiments

1. Change the dataset path in scripts/promptcad/crossdataset.sh line 4 to your current path.

2. Run the commands below to train PromptCAD on the specified dataset.

For example:

nohup bash scripts/promptcad/crossdataset.sh -l 0.1 > crossdataset_0.1.log 2>&1 &

Few-shot Learning

1. Change the dataset path in scripts/promptcad/few_shot.sh line 4 to your current path.

2. Run the commands below to train PromptCAD on the specified dataset.

nohup bash scripts/promptcad/few_shot.sh -l 0.1 eurosat fgvc_aircraft ucf101  > few_shot_0.1.log 2>&1 &