New Action 3

Jul 14, 2026

Nemotron Kernels From Inference To Backpropagation

A visual, source-linked derivation of Nemotron inference, training forward, backpropagation, and quantization across Mamba2, attention, ReLU2, and routed experts.

Read post →

Jun 12, 2026

dL/d(LLM): The Full Backward Pass

A capstone walkthrough of the full LLM backward pass: loss to LM head, final norm, decoder layers, attention, FFN, residual splits, embedding scatter-add, training loop, AdamW, and C-Kernel-...

Read post →

Jun 11, 2026

Residual Connections: The Gradient Highway

Lab note Previously: Attention: The Core Of The Transformer. The previous post showed how attention routes information sideways across a sequence. This post is about the architectural trick ...

Read post →

Jun 10, 2026

Attention: The Core Of The Transformer

Attention is the core transformer mechanism: Q/K/V projections, head splitting, RoPE, scaled dot-products, masking, softmax, weighted value sums, GQA, Flash Attention, and the full backward ...

Read post →

Jun 6, 2026

Activation Functions: The Nonlinearity Inside Neural Networks

Lab note Companion post to the Activation Functions carousel. Previously: Softmax: The Probability Engine. The previous Softmax post ended with a strange contrast. Softmax was beautifully sm...

Read post →

Jun 5, 2026

Softmax: The Probability Engine

Lab note Companion post to the Softmax carousel. Previously: Matrix Wx+b: From Scalars To Transformers. The previous post was about the moment a scalar weighted sum grew into a matrix multip...

Read post →

Jun 4, 2026

Matrix Wx+b: From Scalars To Transformers

A step-by-step explanation of how scalar wx+b becomes matrix Wx+b, how shape rules govern transformer layers, and why GEMM is the physical kernel underneath modern models.

Read post →

Jun 3, 2026

Backpropagation From wx+b: The Smallest Training Loop

Lab note This is the companion post to today's backprop-wxplusb carousel. I wanted a version with more room for the pictures, the algebra, and the engineering intuition. Previous post in thi...

Read post →

Jun 2, 2026

Derivatives And The Chain Rule: The Math Of Tiny Changes

Lab note This post walks through the derivative and chain rule from first principles, with visualizations. It is the companion piece to my carousel post on the same topic. I needed this topi...

Read post →

backpropagation

Nemotron Kernels From Inference To Backpropagation

dL/d(LLM): The Full Backward Pass

Residual Connections: The Gradient Highway

Attention: The Core Of The Transformer

Activation Functions: The Nonlinearity Inside Neural Networks

Softmax: The Probability Engine

Matrix Wx+b: From Scalars To Transformers

Backpropagation From wx+b: The Smallest Training Loop

Derivatives And The Chain Rule: The Math Of Tiny Changes

Need an intelligent system to work on real hardware?

Embedded systems · Robotics · Constrained AI · CPU and HPC · Accelerators · Distributed systems

ShivasNotes

Read

Support

Subscribe

Subscribe to emails from Anthony

Need an intelligent system to work on real hardware?

Embedded systems · Robotics · Constrained AI · CPU and HPC · Accelerators · Distributed systems

ShivasNotes

Read

Support