← Back

Tensors are recursive cubes all the way down

clocksmith: 72% | AI: 28%

Tensor intuition

Tensor sounds like abstract math. It is. But to understand AI concepts like LLMs, Transformers, and Stable Diffusion, you do not need a PhD in quantum neutrino fields. Tensors are grids of numbers with named axes. The dimension count is how many indices you need to locate one value: a scalar is a point, a vector is a line, a matrix is a plane, and adding an axis gives a cube. Add time or batch and you are in 4D and 5D. Add a few more dimensions and maybe you are studying superstring theory. These are still coordinate lists, but our spatial intuition starts to fade.

Note

Some libraries call this count a tensor's rank (side-eyes TensorFlow), which here means number of axes, not matrix rank.

Use +1D/-1D to step through dimensions. 0D is a number, 1D a line, 2D a grid, 3D a cube. At 6D you get a cube of cubes. A 6D index [i, j, k, x, y, z] works in two stages: [i, j, k] finds the outer cube, [x, y, z] finds the inner box.

Tesseract projections and hypercube wireframes are accurate but rarely useful when you are learning tensor indexing. Space is 3D, and projecting more axes onto a flat screen loses structure. A better approach is to visualize indexing instead of space.

This indexing-first approach keeps you grounded in what's actually happening.

The cube-of-cubes model

The idea comes from Feiner and Beshers' 1990 paper on "Worlds within Worlds." Instead of adding new orthogonal axes, treat the 3D cube as a unit and use higher dimensions to organize cubes within cubes, like the giant alien playing marbles at the end of the first MIB movie.

Dimensions 0-3 build a cube: point, line, plane, volume. Freeze it.

Kernels and takeaways

This model aligns with how we slice data in PyTorch or NumPy. For a 5D tensor with shape (batch, time, channel, height, width), common operations map cleanly:

The hierarchical framing helps you debug reshape errors by making it explicit whether your dimensions multiply correctly. If tensor.reshape(32, 8, -1) fails, you can reason about it as "I have 32 outer blocks, each containing 8 sub-blocks, and I want to flatten everything inside each sub-block." The numbers either multiply correctly or they do not.

An inference kernel is a small function that runs across many elements in parallel on a GPU. Think tiles: a launch grid picks an outer block and threads walk the inner block, so grid indices select the outer cube and thread indices select the inner voxel. Striding the wrong axis pulls the wrong inner cube, and mismatched block shapes under or over cover the grid.

This "trick" will not teach transformer inference. For that see Attention thrashing or Forgetting is a feature. It is a lookup story: three numbers pick the outer cube, three numbers pick the inner cube, and you recurse if needed.

Our intuitions are built for stacking containers, not visualizing hypercubes, so the recursion trick works better than pure geometry.

References