Matrix Multiplication: Definition, Intuition, Step-by-Step Practice

Matrix multiplication is a fundamental operation in linear algebra with wide-ranging applications in computer graphics, machine learning, and data science. This understanding-cpu-instruction-pipelining-how-it-works-hazards-and-performance-impact-in-modern-cpus/”>understanding-writing-and-optimizing-code/”>guide-to-understanding-complexity-and-practical-implementations/”>guide breaks down its definition, provides intuitive understanding, and walks you through practical examples and common considerations.

Definition, Intuition, and Core Rules

Definition

For two matrices, A and B, their product AB is defined if the number of columns in A is equal to the number of rows in B. If $A \in R^{m \times k}$ and $B \in R^{k \times n}$, then the product $AB$ is a matrix in $R^{m \times n}$. The element at the $(i, j)$ position of the product matrix AB, denoted as $(AB)_{ij}$, is calculated as the sum of the products of elements from the $i$-th row of A and the $j$-th column of B. Mathematically:
$(AB)_{ij} = \sum_{t=1}^{k} A_{it} B_{tj}$

Intuition

Intuitively, each element $(AB)_{ij}$ of the resulting matrix AB is the dot product of the $i$-th row of matrix A with the $j$-th column of matrix B.

Shape Rule

The rule for matrix dimensions is crucial: If matrix A has dimensions $m \times k$ and matrix B has dimensions $k \times n$, their product AB will have dimensions $m \times n$. The inner dimension ($k$) must match for the multiplication to be defined. If the inner dimensions do not match, the matrix product is undefined.

Non-commutativity

A key property of matrix multiplication is that it is generally not commutative, meaning $AB \neq BA$. The order of multiplication matters significantly.

Example: Let $A = [[1, 2], [0, 1]]$ and $B = [[0, 1], [2, 3]]$.

Then $AB = [[(1*0 + 2*2), (1*1 + 2*3)], [(0*0 + 1*2), (0*1 + 1*3)]] = [[4, 7], [2, 3]]$.

However, $BA = [[(0*1 + 1*0), (0*2 + 1*1)], [(2*1 + 3*0), (2*2 + 3*1)]] = [[0, 1], [2, 7]]$. Clearly, $AB \neq BA$.

Intuition and Step-by-Step Examples

Concrete 2×2 Example: Computing AB

Let’s compute the product of two 2×2 matrices:

Let $A = [[1, 2], [3, 4]]$ and $B = [[5, 6], [7, 8]]$.

To compute $AB$, we take the dot product of each row of A with each column of B:
$AB = [[(1*5 + 2*7), (1*6 + 2*8)], [(3*5 + 4*7), (3*6 + 4*8)]]$
$AB = [[(5 + 14), (6 + 16)], [(15 + 28), (18 + 32)]]$
$AB = [[19, 22], [43, 50]]$

Here’s a breakdown of how the entries are calculated:

• Row 1, Column 1 (Resulting element (1,1)): Dot product of Row 1 of A with Column 1 of B.
• Row 1, Column 2 (Resulting element (1,2)): Dot product of Row 1 of A with Column 2 of B.
• Row 2, Column 1 (Resulting element (2,1)): Dot product of Row 2 of A with Column 1 of B.
• Row 2, Column 2 (Resulting element (2,2)): Dot product of Row 2 of A with Column 2 of B.

In detail:

• $AB(1,1) = (1 \times 5) + (2 \times 7) = 5 + 14 = 19$
• $AB(1,2) = (1 \times 6) + (2 \times 8) = 6 + 16 = 22$
• $AB(2,1) = (3 \times 5) + (4 \times 7) = 15 + 28 = 43$
• $AB(2,2) = (3 \times 6) + (4 \times 8) = 18 + 32 = 50$

Shape Visualization: 2×3 times 3×2 yields 2×2

Consider multiplying matrix A with dimensions $2 \times 3$ by matrix B with dimensions $3 \times 2$. The inner dimension (3) matches, so the product AB is defined and will have dimensions $2 \times 2$. Each of the 2 rows of A is combined with each of the 2 columns of B to form the resulting $2 \times 2$ matrix.

Example:
Let $A \in R^{2 \times 3}$ be $A =
\begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix}
$
And $B \in R^{3 \times 2}$ be $B =
\begin{pmatrix} 7 & 8 \\ 9 & 10 \\ 11 & 12 \end{pmatrix}
$
The product $AB \in R^{2 \times 2}$ is calculated as follows:
$AB =
\begin{pmatrix} (1\times7 + 2\times9 + 3\times11) & (1\times8 + 2\times10 + 3\times12) \\ (4\times7 + 5\times9 + 6\times11) & (4\times8 + 5\times10 + 6\times12) \end{pmatrix}
$
$AB =
\begin{pmatrix} (7 + 18 + 33) & (8 + 20 + 36) \\ (28 + 45 + 66) & (32 + 50 + 72) \end{pmatrix}
$
$AB =
\begin{pmatrix} 58 & 64 \\ 139 & 154 \end{pmatrix}
$
The four entries are the dot products:

• $AB(1,1) = 1\times7 + 2\times9 + 3\times11 = 58$
• $AB(1,2) = 1\times8 + 2\times10 + 3\times12 = 64$
• $AB(2,1) = 4\times7 + 5\times9 + 6\times11 = 139$
• $AB(2,2) = 4\times8 + 5\times10 + 6\times12 = 154$

Algorithmic and Computational Considerations

Practical Guide: Pitfalls, Visualization, and Common Mistakes

Pros

• Matrix multiplication is a foundation of linear models, transforms, and neural networks. Highly optimized libraries and hardware pathways exploit data locality and parallelism.
• Hardware advances (e.g., MMA units) enable fused $AB+C$ computations, offering performance gains in fixed-point or integer-heavy workloads, provided software is written to utilize them.

Cons

• Dense matrix multiplication can be memory-bandwidth bound. For sparse or structured matrices, specialized algorithms often outperform naive methods. Misusing shapes or data layouts can hurt performance.
• Numerical considerations: For floating-point arithmetic, rounding errors can accumulate. For integers, overflow must be guarded using wider accumulators or modular arithmetic strategies.