https://blog.baifan.site/en/tags/neural-networks/ Recent content in Neural Networks on ByF (BaiFan) · Daily Deep Think Hugo -- gohugo.io en blog@baifan.site (ByF (BaiFan)) blog@baifan.site (ByF (BaiFan)) CC BY-NC 4.0 Tue, 05 Aug 2025 14:00:00 +0800 https://blog.baifan.site/en/transformer-attention-mechanism-deep-dive/ Tue, 05 Aug 2025 14:00:00 +0800 blog@baifan.site (ByF (BaiFan)) https://blog.baifan.site/en/transformer-attention-mechanism-deep-dive/ <p>The physical barrier modern large language models confront when scaling context length is rarely raw floating-point throughput (FLOPs). It is the memory-access wall between high-bandwidth memory (HBM) and on-chip static random-access memory (SRAM). The standard Scaled Dot-Product Attention operator established by Vaswani et al. (2017) incurs an $O(N^2)$ memory-traffic overhead as sequence length $N$ grows, reducing self-attention primarily to a memory-bound workload.</p> <!-- more --> <p>Understanding the Transformer requires stripping away anthropomorphic cognitive metaphors and grounding the discussion in the geometry of matrix projections alongside the hardware constraints of modern GPUs.</p>