SHARP: Structured Hierarchical Attention Rank Projection for Efficient Language Model Distillation

Abstract

Knowledge distillation has emerged as a crucial technique for compressing large language models into more deployable versions. While existing approaches focus on transferring knowledge at length-based linguistic granularities such as tokens, phrases, and sequences, they often fail to capture the intrinsic hierarchical attention mechanisms of modern transformer architectures. Furthermore, when multiple granularities share a single representation space, they compete for the same capacity, causing complementary information to be lost. We term this phenomenon representation collapse. To address this limitation, we propose SHARP (Structured Hierarchical Attention Rank Projection), a novel distillation framework that decomposes attention patterns into token-level, head-level, and layer-level representations and projects each into a dedicated orthogonal subspace prior to distillation. This mathematical decomposition provides each granularity with independent representational capacity, preventing interference while enabling parallel optimization pathways. Through extensive experiments on both natural language generation (NLG) and natural language understanding (NLU) tasks with teacher models ranging from 350M to 6.7B parameters distilled to a 125M parameter student, SHARP consistently outperforms existing distillation methods. It achieves an average of 8.0% perplexity reduction on NLG tasks relative to the strongest baseline, with consistent improvements of 7 to 9% across all teacher scales. On NLU tasks, SHARP shows consistent accuracy improvements of 1.3 to 1.7% across different model scales. These results confirm that orthogonal capacity allocation is an effective principle for multi-granularity knowledge transfer in large language model compression.