Transformer Architecture Deconstructed: Attention, Geometric Scaling, and Memory IO

blog@baifan.site (ByF (BaiFan)) — Tue, 05 Aug 2025 14:00:00 +0800

The physical barrier modern large language models confront when scaling context length is rarely raw floating-point throughput (FLOPs). It is the memory-access wall between high-bandwidth memory (HBM) and on-chip static random-access memory (SRAM). The standard Scaled Dot-Product Attention operator established by Vaswani et al. (2017) incurs an $O(N^2)$ memory-traffic overhead as sequence length $N$ grows, reducing self-attention primarily to a memory-bound workload.

Understanding the Transformer requires stripping away anthropomorphic cognitive metaphors and grounding the discussion in the geometry of matrix projections alongside the hardware constraints of modern GPUs.

A Complete Guide to LLMs: Tokens and Vectors in Depth

blog@baifan.site (ByF (BaiFan)) — Tue, 22 Jul 2025 14:00:00 +0800

AI Tutorial: A Guide to AI LLMs, from Basics to Depth
#

This article takes you deep into the core concepts of large language models — from basic principles to vector representation — building a complete knowledge system step by step.

1. Foundations of AI Application Development
#

1.1 Basic Principles and Concepts
#

In Plain Words
#

Core mechanism: predict the next word from the previous one — like a word-chain game
How it works: output is generated token by token

A Deeper Look
#

A large AI model involves two key stages:

Transformer on ByF (BaiFan) · Daily Deep Think

Transformer Architecture Deconstructed: Attention, Geometric Scaling, and Memory IO

A Complete Guide to LLMs: Tokens and Vectors in Depth

AI Tutorial: A Guide to AI LLMs, from Basics to Depth #

1. Foundations of AI Application Development #

1.1 Basic Principles and Concepts #

In Plain Words #

A Deeper Look #

AI Tutorial: A Guide to AI LLMs, from Basics to Depth
#

1. Foundations of AI Application Development
#

1.1 Basic Principles and Concepts
#

In Plain Words
#

A Deeper Look
#