Amdahl's Law is the fastest way to stop pretending that more cores automatically mean more speed. Theory of Constraints is the fastest way to stop pretending that optimizing random code automatically makes the system faster.

These two ideas are different, but they belong together. Amdahl's Law is the computing math: the speedup of a system is capped by the fraction that cannot be improved or parallelized. Theory of Constraints is the operations principle: the throughput of a system is controlled by its current bottleneck. If I improve something that is not the bottleneck, the whole system may barely move.

This is exactly how I need to think about C-Kernel-Engine. CKE is not a single kernel. It is a runtime path: tokenization, model loading, weight mapping, quantization, prefill, decode, attention, MLP, KV cache, memory planning, thread scheduling, generated C, and profiling. A fast kernel is useful. A faster whole path is the real target. The project reference points are the C-Kernel-Engine GitHub repository and the C-Kernel-Engine documentation site.

So the optimization question is not "how many cores do I have?" and it is not "did one kernel get faster?" The better question is:

What is the current constraint in the measured CKE pipeline?

The Formula

Amdahl's Law is usually written as:

\(S \;=\; \dfrac{1}{(1-P) + \dfrac{P}{N}}\)

Where S is total speedup, P is the parallel fraction, 1-P is the sequential fraction, and N is the number of processors, cores, workers, or parallel execution lanes.

The ceiling is:

\(S_{\max} \;=\; \dfrac{1}{1-P}\)

If 10% of the work is sequential, the theoretical maximum speedup is 10x no matter how many cores I add. If 5% is sequential, the ceiling is 20x. If 1% is sequential, the ceiling is 100x. The serial fraction keeps winning because it never disappears from the denominator.

The painful part is that real systems often have more than one "serial-looking" drag:

• token-by-token decode order
• layer-by-layer dependency order
• sampling and control flow
• KV-cache updates
• thread synchronization
• memory allocation and copies
• cache misses and TLB misses
• format conversion between quantized and floating-point views

In other words, the sequential fraction is not only "one function that cannot be parallelized." It is all the runtime glue, memory movement, ordering, and synchronization that remains after the obvious math kernels are optimized.

The Theory Of Constraints Version

Theory of Constraints says a system's throughput is limited by the constraint. In a factory, this might be one machine, one station, one queue, one supplier, or one inspection step. Making a non-bottleneck station faster does not increase total throughput if the bottleneck still controls the flow.

The same idea applies to CKE. If prefill is dominated by quantized GEMM, then making sampling faster will not move prefill throughput. If decode is dominated by KV-cache memory traffic, then making a small activation function 2x faster may not change tokens/sec. If the runtime is waiting on memory bandwidth, adding more threads can make things worse.

This is the engineering loop:

1. Measure the whole path.
2. Find the current constraint.
3. Improve that constraint.
4. Remeasure the whole path.
5. Accept that the bottleneck may move.
6. Repeat.

That loop sounds obvious, but it is easy to violate. It is emotionally satisfying to optimize code I understand. It is more useful to optimize the thing the profiler says is actually limiting the run.

Transformer Inference Has Parallel Work And Ordered Work

Transformer inference is a mix of parallel work and ordered work.

The parallel work is real:

• matrix multiplies inside attention projections
• matrix multiplies inside MLP gate/up/down paths
• vector operations across hidden dimensions
• attention score work across heads and positions
• batching across sequences or requests
• SIMD/vectorized dot products inside quantized kernels

But the ordered work is also real:

• token n + 1 depends on token n during autoregressive decode
• layer i + 1 depends on layer i
• KV cache must be updated in the right order
• sampling happens after logits exist
• the next token cannot be embedded until it is selected

That is the entire reason Amdahl's Law matters for CPU AI. I cannot reason from one kernel alone. I need to reason from the whole pipeline.

A Concrete 90/10 Example

Suppose 90% of the CKE path can benefit from parallelism and 10% remains serial or unimproved. On 16 cores:

\(S \;=\; \dfrac{1}{0.10 + \dfrac{0.90}{16}} \;\approx\; 6.4\)

That is much better than 1x, but nowhere near 16x.

Now suppose I work extremely hard and improve only part of the parallel region, but the effective sequential fraction stays 10%. The ceiling is still 10x. The system may improve, but it cannot cross the ceiling unless I reduce the serial fraction or change the workload shape.

This is why CKE optimization must separate three different questions:

The third question is the one that matters most. The first two are useful only when they move the end-to-end path or explain why they did not.

Where CKE Bottlenecks Move

The bottleneck in CKE is not fixed. It moves as the runtime improves.

Early in a path, the constraint may be correctness. If the template is wrong, the fastest kernel in the world is irrelevant. If the model map binds a tensor incorrectly, speed is irrelevant. If the generated C reads the wrong buffer, speed is irrelevant.

Once correctness is stable, the constraint may become conversion or memory layout. Once memory layout improves, the constraint may become quantized GEMM. Once GEMM improves, attention or KV-cache traversal may dominate. Once decode improves, sampling, tokenizer overhead, or synchronization may show up.

Why Core Count Has To Match The Bottleneck

Amdahl's Law already says more cores have diminishing returns when part of the path is serial. That does not mean "more cores are bad." It means core count has to match the active bottleneck. If the active constraint is compute, more cores can help. If the active constraint is memory bandwidth, cache capacity, TLB reach, synchronization, or NUMA placement, adding cores without changing the memory path may only add pressure.

The important ratio is not only cores. It is cores to memory bandwidth, cores to cache, cores to working-set size, and cores to useful independent work. A CPU with more cores but similar memory bandwidth can have lower bandwidth per core. A CPU with many small cores and limited shared cache can make a quantized decode path fight over the same KV-cache and weight stream. A server CPU with more memory channels can make more cores useful because the memory system can actually feed them. The same thread count can be good on one CPU and wrong on another.

This is workload dependent. Large prefill usually has enough matrix work to feed many cores if the tiling and packing are sane. Decode is often smaller, more latency-sensitive, and more KV-cache heavy, so using every core can be wasteful if the runtime spends more time waking workers, crossing caches, or waiting on memory than doing FMA. If work migrates between P-cores and E-cores without care, latency can become noisy. If NUMA placement is wrong, a server CPU can spend time pulling model weights or KV-cache data from the wrong socket.

This is why CKE cannot blindly say "use all cores." It also cannot blindly say "use fewer cores." The right policy might be:

• use many cores for large prefill work
• use fewer pinned cores for small decode work
• size thread count to memory bandwidth and cache capacity
• keep sampling and control tasks on lighter cores
• prefer contiguous cache-friendly KV ownership for each user and layer
• avoid moving one user's KV cache across NUMA domains
• separate batch throughput tuning from single-user latency tuning

The right answer is not ideological. The profiler has to decide. A well-sized CPU node is not simply the CPU with the highest core count. It is the CPU whose cores, cache, memory channels, vector units, and Linux scheduling policy match the CKE workload being hardened.

How This Connects To The CPU Lab

This is also why I am building and using small physical CPU nodes instead of only guessing from cloud numbers. A laptop, a ThinkStation Tiny, a Xeon server, and eventually AMD or newer Intel nodes will not behave identically. They have different cache sizes, memory bandwidth, vector units, thermal limits, P-core/E-core behavior, and BIOS/Linux tuning surfaces.

CKE has to learn from that diversity. A kernel that looks good on one machine may not be the constraint on another. A batch size that helps one CPU may hurt another. AVX-VNNI, AVX-512, AMX, NEON, and ordinary scalar cleanup all matter only if the measured runtime path benefits.

That is the practical value of Amdahl's Law here. It gives me a way to stay honest when hardware gets tempting. More cores, more cache, more RAM, and newer instructions are useful only if they reduce the active constraint.

Amdahl And Constraints Are The CKE North Star

Jensen Huang explains the same law from the large-scale accelerator side: if computation is only half of the total problem, then making computation infinitely fast only gives about a 2x system speedup. After that, the unsolved work becomes networking, switching, placement, pipeline sharding, scheduling, memory movement, and orchestration across machines. The math does not care whether the machine is a GPU cluster, a CPU cluster, a laptop, or a small pile of physical CKE nodes. If part of the path remains unimproved, that part becomes the ceiling.

This is why Amdahl's Law and Theory of Constraints fit together so cleanly. Amdahl's Law says how much speedup is mathematically available. Theory of Constraints says where to spend engineering effort next. One gives the ceiling. The other gives the work order.

For a single CKE node, the active constraint may be quantized GEMM, KV-cache traversal, memory bandwidth, core pinning, or layout conversion. For multiple CKE nodes, the constraint may move again: activation transfer, pipeline bubbles, network latency, stage imbalance, request scheduling, or the cost of keeping KV state close to the worker that needs it. More machines only help if CKE can keep the expensive bytes local, move the unavoidable bytes deliberately, and keep every stage busy enough that the network does not become the new serial fraction.

That is also why the CKE work cannot be only "write a faster dot product." The dot product matters. The memory planner matters. The threadpool matters. The IR matters. The model contract matters. The distributed plan matters. The constraint may start as math, move to memory, move to synchronization, move to networking, and then move back to math after the pipeline gets cleaner. That moving bottleneck is not failure. That is the sign that the system is being hardened.

CKE's north star is not one heroic optimization. It is reducing the active constraint in the measured runtime path, then following the bottleneck when it moves.

A Small C Runtime Sketch

The simplest way to think about this in a runtime is: parallelize the kernels, but do not pretend the dependency chain vanished.

for (size_t t = 0; t < max_new_tokens; ++t) {
    // Ordered: token t depends on previous tokens and KV state.
    for (size_t layer = 0; layer < n_layers; ++layer) {
        // Parallel regions live inside the layer kernels.
        ck_rmsnorm(ctx, layer);
        ck_attention_decode(ctx, layer);
        ck_mlp(ctx, layer);
        ck_residual_update(ctx, layer);
    }

    ck_lm_head(ctx);
    token = ck_sample_next(ctx);
    ck_update_kv_cache(ctx, token);
}

The loop is ordered. The kernels inside the loop can be parallel. CKE's job is to make those kernels correct, make their memory access sane, and make their contribution to the whole runtime visible.

The Practical CKE Rule

The practical rule is:

Do not optimize what is easy. Optimize what is constraining the measured path.

That means a CKE performance claim should always try to say:

• which model was tested
• which quantization was used
• which CPU and thread policy ran
• whether the measured region was prefill, decode, or end-to-end
• what the previous bottleneck was
• what changed
• whether the bottleneck moved

This is the difference between performance theater and performance engineering.

Related Notes

• What Is the C-Kernel-Engine?
• C-Kernel-Engine GitHub Repository
• C-Kernel-Engine Documentation
• CKE Kernel Tuning Methodology
• CKE v7 Inference And Training Runbook
• CKE v8 Runbook
• v8 IR Pipeline Codegen: How CKE Hardens Pure C Inference
• KV Cache Memory: The Hidden State That Makes LLM Decode Work
• CPU Performance Engineering For AI
• Linux System Programming For AI Kernels
• Distributed CPU AI: MPI, RDMA, NUMA, And CKE
• Pipeline vs Tensor Parallelism: How CKE Splits AI Across CPU Nodes
• Threadpools And Memory Pools
• K-Quants Deep Dive
• Amdahl's Law on Wikipedia
• Theory of Constraints on Wikipedia

Closing Thought

Amdahl's Law tells me the ceiling. Theory of Constraints tells me where to work next. CKE needs both because CPU AI performance is not won by one heroic kernel. It is won by finding the current constraint, improving it, and repeating the loop until the whole runtime path gets faster.