The base weights are frozen. The loss changes. The adapter learns. What actually travels between those three statements?

That is the part I want to make visible. In LoRA & QLoRA: Inside the Update, we placed the adapter inside attention and the MLP. Here we follow the gradients back through it, first with ordinary floating-point weights, then with a quantized base.

We will keep one small example open throughout: one rank-one projection, two outputs and a known target. Then we will put that local calculation back inside a transformer block and a training pipeline. The way I approach this in runtime work is to check a small forward result, check its derivatives, and only then worry about packing weights or fusing kernels.

1. What are we training against?

We do not give LoRA a target matrix and ask it to copy those weights. In supervised language-model fine-tuning, we give the model examples of the behaviour we want: a prompt and an answer, or a conversation. The objective usually asks it to predict the correct next token at selected positions.

{
  "messages": [
    {
      "role": "user",
      "content": "Return the issue category for: motor overheats."
    },
    {
      "role": "assistant",
      "content": "thermal"
    }
  ]
}

The word “thermal” may become one token or several; that depends on the tokenizer. The chat template adds the model’s expected role and boundary tokens. After batching, we commonly have input_ids [batch, length], an attention mask, and labels of the same sequence shape. Token IDs are integer indices, not the continuous hidden vectors that enter a LoRA projection.

For causal language modelling, the logit at position t predicts the token at t+1. Some model/trainer combinations do the shift internally. Check that contract; manually shifting again can silently train the wrong alignment. The TRL SFT documentation describes dataset formats and completion-only or assistant-only supervision. Inspect a tokenized batch rather than assuming the configured mask is correct.

For N supervised positions, the usual averaged loss is L = −(1/N) Σ log p(correct token). The labels and masks determine what “correct” means. If padding is scored accidentally, or answers are truncated, the gradients can be numerically correct while teaching the wrong task. Reject batches with no supervised tokens instead of allowing an undefined average.

2. Start with an unquantized projection

Take one selected linear projection inside one transformer block. A block here means attention, an MLP, normalization and residual connections. It is not the same thing as one GEMM. We will zoom into the attention output projection; the local derivatives also apply to other adapted linear projections.

Use row-vector notation and omit bias and dropout for now. X contains n token-position vectors. The base W is frozen. A and B are trainable. The scalar s is fixed by the adapter configuration; for ordinary LoRA, s = α/r.

X : n × d       W : d × m
A : d × r       B : r × m
Z = XA          # n × r
Y = XW + sZB    # n × m
D = ∂L/∂Y       # n × m, arriving from downstream

Think of D as a message from the rest of the model: how would a small change to each entry of Y affect the loss? D is not generally the language-model probability error at this internal projection. It has already passed backward through whatever lies between Y and the output loss.

First, the gradient for B

Holding Z fixed, each output column is a weighted combination of its r columns. The matrix multiplication’s backward rule gives ∇B = sZᵀD. Check the dimensions: [r × n][n × m] → [r × m]. That is exactly B’s shape. Summing over n accumulates contributions from all local token positions.

Then back through B into A

The gradient reaching Z is dZ = sDBᵀ, with shape n-by-r. Since Z = XA, the gradient for A is ∇A = XᵀdZ. Substituting gives ∇A = sXᵀDBᵀ. We multiply by the current B from this forward/backward pass. Updating B before finishing the backward calculation would mix two parameter states.

Finally, the gradient for X has two paths

X feeds both the base and the adapter. Their contributions add:

∇B = s Zᵀ D             # r × m
 dZ = s D Bᵀ             # n × r
∇A = Xᵀ dZ              # d × r
∇X = D Wᵀ + dZ Aᵀ       # n × d

No optimizer update to W.
No need to materialize ∇W when W is frozen.

In framework terms, freezing W’s parameter gradient is different from running its entire computation without gradient tracking. A blanket detach can also prevent upstream adapters from learning. The LoRA paper defines the frozen-base, trainable-factor parameterization; these equations spell out its local matrix calculus in our row-vector convention.

3. Walk one rank-one example backward

To keep the numbers small, the interactive example takes one projection and treats its two outputs directly as classification logits. This is a deliberate teaching simplification, not an execution of the whole transformer block. The target is the second output, class index 1.

The initial values are X = [1, 2], A = [0.1, −0.2]ᵀ, B = [0, 0], and W = [[0.63, −0.47], [0.28, 0.91]]. Rank and alpha are both one, so s = 1. The base is “unquantized” here in the sense that it has no low-bit encoding; the browser uses double-precision numbers for readable arithmetic, not a simulated BF16 kernel.

The first forward pass gives Z = −0.3, Y = [1.19, 1.35], and probabilities approximately [0.460085, 0.539915]. The target is the second output, so D ≈ [0.460085, −0.460085]. Consequently ∇B ≈ [−0.138026, 0.138026]. A gradient-descent step pushes B’s first entry positive and its second entry negative. Since Z is negative, that lowers the first output and raises the second.

“Why is A’s gradient zero at first? Is it broken?”

No. With B initialized to zero, dZ = DBᵀ = 0, so the first loss gradient for A is zero. B’s gradient can still be nonzero. After B changes, the next pass can produce a nonzero gradient for A. The PEFT initialization documentation describes a common nonzero-A, zero-B initialization. Other initializers exist.

Zeroing both factors would make both loss gradients zero in this plain setup. That is why “start the correction at zero” does not mean “set every adapter parameter to zero.” Also distinguish loss gradients from weight decay: an optimizer can change a parameter through decay even when its loss gradient is zero. The walkthrough sets decay to zero so that distinction cannot obscure the first step.

The curves above repeat one example. They demonstrate the arithmetic, not language-model quality. The quantized toy starts with different logits and happens to start with a lower loss on this target. That is not evidence that quantization improves a real model.

optimizer.zero_grad(set_to_none=True)
X.grad = None
Z = X @ A
logits = X @ W + Z @ B
loss = F.cross_entropy(logits, target)
loss.backward()       # fills A.grad, B.grad and X.grad
print(A.grad, B.grad, X.grad)
optimizer.step()     # updates only A and B

PyTorch 2.15.0+cpu | CPU | rank 1 | alpha 1

Step 1: loss before = 0.616344
dA: [0.0, 0.0]
dB: [-0.138026, 0.138026]
dX: [0.506094, -0.289854]
A after: [0.1, -0.2]
B after: [0.01, -0.01]
loss after = 0.613588; W unchanged = True

Step 2: loss before = 0.613588
dA: [0.009172, 0.018344]
dB: [-0.137579, 0.137579]
dX: [0.505372, -0.290749]
A after: [0.092559, -0.207441]
B after: [0.019999, -0.019999]
loss after = 0.610433; W unchanged = True

PASS: factor/input gradients, first-step behavior, frozen base

python3 -m venv .venv
. .venv/bin/activate
python -m pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/cpu
python lora_torch_demo.py

"""CPU reference: one rank-one projection, not a whole language model."""
import torch
import torch.nn.functional as F

torch.set_printoptions(precision=6, sci_mode=False)
dtype = torch.float64  # readable reference arithmetic, not a BF16 benchmark
# Shapes: X [tokens=1, width=2], W [2,2], A [2,rank=1], B [1,2].
# Row-vector convention; torch.nn.Linear stores its weight transposed.
X = torch.tensor([[1.0, 2.0]], dtype=dtype, requires_grad=True)
# W defaults to requires_grad=False: frozen parameters, live input path.
W = torch.tensor([[0.63, -0.47], [0.28, 0.91]], dtype=dtype)
A = torch.nn.Parameter(torch.tensor([[0.1], [-0.2]], dtype=dtype))
# Nonzero A + zero B starts with zero correction, but lets B learn.
B = torch.nn.Parameter(torch.zeros((1, 2), dtype=dtype))
target = torch.tensor([1])  # the second output is the correct class
frozen_copy = W.clone()
# Only adapter factors belong to the optimizer. No decay in this demo.
optimizer = torch.optim.AdamW([A, B], lr=0.01, weight_decay=0.0,
                             betas=(0.9, 0.999), eps=1e-8)

def values(t):
    return [round(v, 6) for v in t.detach().flatten().tolist()]

print("PyTorch", torch.__version__, "| CPU | rank 1 | alpha 1")
for step in range(1, 3):
    # Gradients accumulate by default; start this step with fresh buffers.
    optimizer.zero_grad(set_to_none=True)
    X.grad = None  # observe input gradients; X is not optimized
    # Rank-one bottleneck, then expand and add to the frozen-base result.
    Z = X @ A
    logits = X @ W + Z @ B
    # Cross entropy takes raw logits, not precomputed probabilities.
    loss = F.cross_entropy(logits, target)
    loss.backward()

    # Check autograd against the local matrix equations before updating.
    with torch.no_grad():
        D = logits.softmax(dim=-1) - F.one_hot(target, 2).to(dtype)
        dZ = D @ B.T
        torch.testing.assert_close(B.grad, Z.T @ D)
        torch.testing.assert_close(A.grad, X.T @ dZ)
        torch.testing.assert_close(X.grad, D @ W.T + dZ @ A.T)
    assert W.grad is None
    if step == 1:
        assert torch.count_nonzero(A.grad) == 0
        assert torch.count_nonzero(B.grad) > 0
    else:
        assert torch.count_nonzero(A.grad) > 0

    print(f"\nStep {step}: loss before = {loss.item():.6f}")
    print("dA:", values(A.grad))
    print("dB:", values(B.grad))
    print("dX:", values(X.grad))
    # All derivatives above use the old factors; update only afterwards.
    optimizer.step()
    with torch.no_grad():
        after = F.cross_entropy(X @ W + (X @ A) @ B, target)
    assert torch.equal(W, frozen_copy)
    print("A after:", values(A))
    print("B after:", values(B))
    print(f"loss after = {after.item():.6f}; W unchanged = True")
print("\nPASS: factor/input gradients, first-step behavior, frozen base")

# Append after the main demo: A and B now contain two optimizer updates.
# Recompute gradients at these current values; the old .grad is stale.
optimizer.zero_grad(set_to_none=True)
loss = F.cross_entropy(X @ W + (X @ A) @ B, target)
loss.backward()
analytic = B.grad[0, 0].item()
epsilon = 1e-6  # suited to this float64 toy, not a universal tolerance

# Perturb only a COPY of B: leave model parameters and optimizer untouched.
with torch.no_grad():
    plus, minus = B.clone(), B.clone()
    plus[0, 0] += epsilon
    minus[0, 0] -= epsilon
    loss_plus = F.cross_entropy(X @ W + (X @ A) @ plus, target)
    loss_minus = F.cross_entropy(X @ W + (X @ A) @ minus, target)
    numeric = ((loss_plus - loss_minus) / (2 * epsilon)).item()

print(f"autograd={analytic:.9f}, finite difference={numeric:.9f}")
assert abs(analytic - numeric) < 1e-8
print("PASS: one B entry agrees with an independent loss perturbation")

# Append after the main demo. This is an inference adapter, not a resume checkpoint.
import io

# Serialize actual tensors, using memory so this example overwrites no files.
# Replace the BytesIO buffer with a filename to keep your own checkpoint.
checkpoint = {
    "base_id": "article-toy-w-v1", "rank": 1, "alpha": 1,
    "A": A.detach().clone(), "B": B.detach().clone(),
}
buffer = io.BytesIO()
torch.save(checkpoint, buffer)
buffer.seek(0)
loaded = torch.load(buffer, map_location="cpu", weights_only=True)
assert loaded["base_id"] == "article-toy-w-v1"
assert loaded["rank"] == 1 and loaded["alpha"] == 1
assert loaded["A"].shape == (2, 1) and loaded["B"].shape == (1, 2)

with torch.no_grad():
    expected = X @ W + (X @ A) @ B
    restored = X @ W + (X @ loaded["A"]) @ loaded["B"]
torch.testing.assert_close(restored, expected, rtol=0, atol=0)
print("PASS: reloaded adapter reproduces the same CPU logits exactly")

4. How does this reconnect to a whole transformer block?

For the attention output projection, the local input X is the concatenated attention-head output. Its ∇X is reshaped back into the head layout. Backpropagation then traverses attention and the Q/K/V projections. Any LoRA factors attached to those projections receive their own local D and use the same factor-gradient equations.

For a single attention head, write S = QKᵀ / √dₕ + mask, P = softmax(S) and C = PV. Given dC, the useful local rules are:

dV = Pᵀ dC
dP = dC Vᵀ
dS = P ⊙ (dP − row_sum(dP ⊙ P))
dQ = dS K / √dₕ
dK = dSᵀ Q / √dₕ

The row sum is broadcast across each row. Masked positions contribute zero probability and the corresponding softmax gradient is zero. Head rearrangement, positional transforms and normalization each have their own backward rules. The adapter is not bypassing any of them.

In a gated MLP, H = SiLU(G) ⊙ U. The down projection first returns dH. Then dG = dH ⊙ U ⊙ SiLU′(G) and dU = dH ⊙ SiLU(G). Those feed the gate and up projections, where their respective adapters receive factor gradients. The two MLP input-gradient branches add. At a residual addition, the incoming gradient is passed to both operands; contributions reunite where they share an earlier tensor.

This is why I start from the computational graph. “Train only the adapter” says which parameters move. It does not say that only adapter operations execute during backward. With multiple adapted blocks, later frozen operations can sit directly on the route to earlier trainable factors.

5. Can AdamW or Muon update the adapter?

Backpropagation computes gradients. The optimizer chooses the parameter update. A and B can be optimized without changing the derivative rules above. The optimizer must receive the intended trainable parameters, not the frozen base.

AdamW: a straightforward baseline

AdamW maintains first and second moment estimates for each trainable entry, uses bias correction, and applies decoupled weight decay when configured. Its learning rate, epsilon, moment coefficients and decay affect the update; the gradient alone does not specify the step. The PyTorch AdamW reference gives the update equations.

Our interactive AdamW mode uses learning rate 0.01, betas (0.9, 0.999), epsilon 10⁻⁸ and zero decay. The second-step option preserves the first step’s moments. Its loss curve should not be compared with SGD as an optimizer benchmark: two updates on one artificial example, with different learning rates, cannot establish that.

Muon: possible, but ask what matrix you are optimizing

A and B are two-dimensional matrices, so they can satisfy the shape requirement of a matrix optimizer. Muon forms a momentum-based matrix update and approximately orthogonalizes it, with implementation-specific scaling. The PyTorch Muon documentation describes its intended use on hidden-layer matrices. That establishes an optimizer contract, not that vanilla Muon on LoRA factors is the best adaptation recipe.

Applying Muon separately to A and B is not the same as applying it to a dense ΔW. If the optimizer changes the factors by δA and δB, the effective correction changes by:

δ(ΔW) = s[(A + δA)(B + δB) − AB]
       = s(δA B + A δB + δA δB)

There is also a parameterization issue: (cA)(B/c) = AB for nonzero c, but the factor gradients and optimizer states need not behave identically under that rescaling. Rank-one factors are extremely rectangular, not square weight matrices. Evaluate the actual factor optimizer, its zero-gradient handling, scaling and task metrics rather than transferring a dense-model result by name.

I would use a documented AdamW adapter recipe as the baseline, then compare a Muon-factor experiment under a controlled budget. Keep unsupported parameter classes on a suitable optimizer. Do not give the two optimizers overlapping parameter ownership. The walkthrough deliberately does not label an ad-hoc orthogonalization routine as a faithful Muon implementation. For the broader matrix-update intuition, see Muon: from wx+b to matrix updates and SGD to AdamW.

6. What changes when W is quantized?

Replace W with a reconstructed operator Ŵ = dequantize(codes, metadata). For standard frozen-base QLoRA, those codes and their quantization representation do not receive optimizer updates. A and B remain higher-precision trainable factors.

Y  = XŴ + s(XA)B
∇B = s(XA)ᵀD
∇A = sXᵀDBᵀ
∇X = DŴᵀ + sDBᵀAᵀ

The factor formulas have the same structure, but their numerical values can change. Quantization changes the forward activations, which changes the prediction, loss and D. It can also change X when earlier projections are quantized. Saying “the gradient formula is unchanged” does not mean that the training run is identical.

“Does backward need to differentiate rounding?”

Not to train A and B against a fixed quantized base. We are differentiating the activation path through a fixed reconstructed linear map. We are not asking for a derivative of the frozen code-selection decision. The kernel can reconstruct tiles when needed; it does not necessarily materialize a full dense model.

“Which low-bit formats are normally used?”

The QLoRA paper combines NF4, double quantization and paged optimizers. Double quantization compresses quantization constants; it is not a gradient trick. Paged optimizers address memory management, not whether derivatives exist. The PEFT quantization guide documents training integrations, while the bitsandbytes linear-layer reference distinguishes four-bit storage and compute settings.

The interactive INT4 switch is deliberately simpler: code = round(W/0.2), then Ŵ = 0.2 × code. All example codes fit a signed four-bit range. This is not NF4, Q4_K, MXFP4 or NVFP4. It lets us isolate the effect of a changed frozen operator without hiding it behind a real format’s block metadata.

7. Can GGUF models have backpropagation?

GGUF is a container, not an autograd engine and not one quantization format. The GGUF specification describes model metadata and tensors, including tensor types. A file can store floating-point tensors or various quantized encodings. Its extension does not make gradients mathematically impossible—or provide them automatically.

Route one: load and dequantize into a training framework

A compatible loader can reconstruct tensors and expose a model through ordinary differentiable operations. LoRA can then be applied if the architecture and modules are supported. But fully dequantizing the base gives up much of the low-bit resident-memory advantage. The reconstructed weights also retain the information loss from quantization; loading does not restore the original checkpoint exactly.

The Transformers GGUF documentation describes loader paths that may retain packed weights on supported combinations or fall back to dequantization. This is version-, architecture- and device-dependent. A successful forward load, including a packed path, is not proof that the required backward kernels exist.

Route two: keep the base packed and implement the backward contract

For our frozen linear operator, we need DŴᵀ when an input gradient is required, along with the ordinary adapter derivatives. That can be implemented with appropriate reconstruction and multiplication. The challenge is engineering: transposed access to packed blocks, metadata interpretation, numerical checks, saved activations, dtype rules and framework integration. An inference kernel tuned for XŴ does not automatically implement the transposed backward operation.

For a full model, every operation on a route to a trainable parameter needs the relevant backward support: attention, normalization, activation functions, residual connections and the linear operators. Special-case demonstrations are not a blanket guarantee for every GGUF model or quantization type.

“Then why not always train LoRA from GGUF?”

Teams often choose the checkpoint and backend already supported by their training stack: target-module discovery, dataset tooling, optimizer state, checkpointing and evaluated low-bit backward kernels. GGUF is frequently the deployment artifact at the end of that pipeline. That is a practical convention, not a theorem forbidding training from GGUF.

A sensible route is to train against a pinned base in a supported training representation, export the adapter, and validate it with the exact serving base and runtime. If serving uses another quantization, measure the resulting quality and output differences. If merging and requantizing, validate that converted artifact too. “Same architecture” is weaker than “the same compatible base.”

8. The data structures behind one training step

Gradient accumulation can process several microbatches before one optimizer update. Normalize consistently: if microbatches have different numbers of supervised tokens, averaging their already-averaged losses equally can weight short examples more heavily than intended. Decide whether the objective is token-weighted or example-weighted and implement that deliberately.

Activation checkpointing trades additional forward computation for lower activation storage. It does not remove the need for backward. Mixed precision adds another contract: parameter dtype, multiplication dtype, accumulation dtype and optimizer-state dtype are not necessarily identical. Loss scaling, where used, must be undone before interpreting or clipping gradients.

A practical sequence for the first real run

1. Define a task and a baseline. Keep held-out examples that the training loop will not consume. Record current model quality and resource use.
2. Version the data. Track source, permissions, transformations, splits and near-duplicate handling. Render a few conversations after templating and inspect which tokens are scored.
3. Pin the base and representation. Save the model revision, tokenizer, quantization configuration and library/runtime versions. Do not rely on a mutable model name alone.
4. Choose an explicit adapter recipe. Target modules, ranks, alpha, dropout, initialization, trainable auxiliary parameters, optimizer, learning rate and schedule belong in one reproducible configuration.
5. Run a small preflight. Check trainable counts, shapes, finite losses, gradients and unchanged frozen weights. Overfit a tiny batch to find pipeline mistakes; do not mistake that for generalization.
6. Train within a resource budget. Record tokens processed, loss, evaluation metrics, memory, throughput and checkpoint identity. A process exiting successfully is not a model-quality result.
7. Export and reload. Adapter-only export is not a resumable training checkpoint. Preserve optimizer/scheduler state, random-number state and data progress when resume matters.
8. Evaluate the serving artifact. Test task quality, regressions and latency after conversion, merging or quantization. Retain a known-good version and a rollback route.

9. How I would check the kernel implementation

The small example gives us a useful test boundary. For each A or B entry, perturb it by ±ε and recompute the loss. Compare the central difference (L(θ+ε) − L(θ−ε))/(2ε) with the analytic gradient. Do the same for X while keeping the reconstructed quantized base fixed. This checks derivatives of the frozen-operator computation, not derivatives of the code-selection procedure.

Check zero-B initialization and a nonzero-B case. Otherwise, a broken ∇A implementation could appear correct simply because the expected first-step gradient is zero. Also check multiple tokens, rectangular input/output widths, non-unit s, and both base/input gradient branches. A rank-one example is the starting point, not a complete kernel certification.

For the numerical examples here, an independent NumPy calculation checks A, B and X gradients against central differences for both the dense and toy-quantized base. The browser implementation is checked against those numerical fixtures. These checks validate the teaching arithmetic; they do not certify any CKE, CUDA or GGUF training backend.

For a real runtime, I would then compare forward values, gradients and optimizer updates against a trusted implementation using dtype-appropriate tolerances. Verify that frozen codes and scales do not change. Save and reload the adapter; compare the result again. Then test the intended end-to-end task.

Keep following the computation

LoRA’s trainable state is small, but the learning signal still travels through a model. QLoRA changes how the frozen base is represented; it does not remove the activation-gradient path. Once the tensors, masks and optimizer ownership are explicit, it becomes much easier to see what a training runtime must actually implement.

• LoRA & QLoRA: where the adapters attach
• Muon: from wx+b to matrix updates
• Four-bit formats: Q4_K, MXFP4 and NVFP4

If this is the kind of detail you want to work through, subscribe to ShivasNotes. I write about the math, kernels and runtime decisions behind models, starting with examples we can inspect.

Primary references

• Hu et al.: LoRA.
• Dettmers et al.: QLoRA.
• PEFT LoRA configuration and initialization and quantized training integrations.
• PyTorch AdamW and Muon.
• bitsandbytes four-bit linear layers.
• GGUF specification and Transformers GGUF loading.
• TRL supervised fine-tuning and loss masking.