CPU measurement hardware

A profiler does not infer retired instructions, cache misses, branch mispredictions, or memory-controller traffic from elapsed time. Event logic distributed through the processor emits signals. Performance-monitoring hardware selects some of those signals, increments counters, and optionally interrupts the CPU so Linux can record where the event occurred.

PMU means Performance Monitoring Unit. It is not one universal sensor hovering over the entire processor. A modern CPU contains multiple monitoring facilities placed beside the hardware they observe: core PMUs near execution pipelines, and uncore PMUs near shared cache agents, memory controllers, interconnects, I/O blocks, and power-management units.

Tools such as Linux perf, Intel VTune, and Intel Advisor do not manufacture these observations. They ask the operating system to program the available PMUs, run a workload, collect counts or samples, and convert raw event encodings into attributed metrics. The Intel 64 and IA-32 Software Developer's Manual documents architectural performance monitoring, model-specific registers, counter overflow, PEBS, and the RDPMC instruction. Linux exposes a portable control interface through perf_event_open.

This matters because optimization is an attempt to close a measured gap. A processor may advertise a vector or matrix throughput ceiling, a memory-bandwidth ceiling, and a cache hierarchy capable of feeding those units. The PMU helps explain why a real kernel reaches only some fraction of the relevant ceiling. It can expose frontend starvation, poor vectorization, execution-port pressure, cache misses, DRAM saturation, NUMA traffic, branch behavior, accelerator inactivity, or frequency limits.

Theoretical peak is not one number

A decode GEMV, a batched prefill GEMM, a tensor repack, and a distributed cache transfer have different ceilings. The useful limit may be operations per cycle, bytes per second, cache residency, instruction issue, or communication latency. Performance engineering first identifies the applicable ceiling, then uses PMU evidence to explain the distance from it.

Simplified CPU PMU hardware path from event-generating logic through qualifiers and an event selector to a counter, overflow detector, interrupt, Linux ring buffer, and profiler report
The implementation is distributed hardware, not a software estimate. Vendors publish architectural registers and event semantics, not the complete proprietary transistor-level netlist.

What Physically Exists Inside The CPU?

Consider a branch instruction reaching retirement. The retirement logic already knows that an instruction completed. The branch unit already knows whether prediction was correct. The cache controller already knows whether a lookup hit or missed. The TLB already knows whether an address translation was present. These facts are required for normal execution before profiling exists.

Performance monitoring adds observation logic around selected facts. At a conceptual digital-logic level, an event path contains:

  1. An event source: control logic produces a pulse or condition when an event occurs, such as an instruction retiring or an L1 miss being allocated.
  2. Qualifiers: comparators and Boolean conditions restrict the event by privilege level, thread, edge, threshold, transaction state, address, or event-specific unit mask.
  3. A selection network: multiplexers route one selected event class toward a programmable counter.
  4. An increment path: an adder updates the counter register by one or by an event-defined amount.
  5. An overflow detector: carry/threshold logic marks overflow when the counter wraps or reaches a programmed sampling boundary.
  6. A notification path: status logic can request a performance-monitoring interrupt, while facilities such as PEBS can write a more precise hardware record.

The exact gates, wires, pipeline stages, and buffering are processor-specific implementation details. A useful mental model is still ordinary synchronous digital logic: registered event conditions, AND/OR qualification, multiplexing, addition, state registers, and interrupt generation. The PMU can count at processor speed because it is integrated into the same clocked hardware, not because a background software thread polls every instruction.

An event is not always one physical pulse

Some events count retired architectural operations. Others count speculative micro-operations, cycles satisfying a condition, cache-line transfers, occupancy, or model-specific approximations. The event definition determines what an increment means. Similar names across processors do not guarantee identical semantics.

Fixed Counters, Programmable Counters, And Event Selectors

Intel core PMUs expose two broad counter classes:

  • Fixed-function counters are dedicated to common architectural events. Typical Intel examples include instructions retired, unhalted core cycles, and unhalted reference cycles.
  • General-purpose programmable counters can count one of many supported events selected through an associated control register.

On x86, software discovers architectural PMU capabilities with CPUID leaf 0xA. It reports information such as the architectural PMU version, number and width of general-purpose counters, and fixed-counter capabilities. The operating system then programs model-specific registers such as:

Register familyPurpose
IA32_PERFEVTSELxSelects an event and unit mask, privilege filters, edge/threshold behavior, interrupt enable, and counter enable for a programmable counter.
IA32_PMCxStores a general-purpose event count.
IA32_FIXED_CTR_CTRLControls privilege filtering, PMI, and related behavior for fixed counters.
IA32_FIXED_CTRxStores fixed-event counts.
IA32_PERF_GLOBAL_CTRLEnables groups of core counters.
IA32_PERF_GLOBAL_STATUSReports overflow and monitoring status.

Event encodings combine an event-select value with a unit mask and optional modifiers. Intel's official PerfMon Events database publishes the supported encodings and restrictions by microarchitecture. For example, one event may be legal only on counters 0–3, another may support PEBS, and another may be speculative rather than retirement-based.

Which Instructions Control Or Read The PMU?

Ordinary applications should usually use perf_event_open, PAPI, VTune, Advisor, or another operating-system-managed interface. Direct PMU programming is privileged and must coordinate with scheduling, virtualization, security, and other profiler users.

MechanismRoleNormal owner
CPUID leaf 0xADiscovers architectural PMU capabilities.Kernel, profiler, diagnostic tool
WRMSRWrites PMU selector, counter, and global-control MSRs.Kernel or privileged driver
RDMSRReads PMU MSRs.Kernel or privileged driver
RDPMCReads a selected performance counter with low overhead.Kernel; user space only when explicitly enabled and coordinated
perf_event_openLinux system call that requests an event, scope, filters, sampling policy, and output record format.Application or profiler through the kernel
ioctl on a perf file descriptorEnables, disables, resets, refreshes, or groups configured events.Application or profiler through the kernel

RDPMC reads; it does not generally configure the PMU. Linux may expose low-latency user-space reads through the perf memory-mapped metadata page when policy and event state permit. The metadata is needed because Linux may multiplex counters, move a task, or rescale a value. Reading a raw register without coordinating with the kernel can produce the wrong event, wrong CPU, or wrong time interval.

Software control path from perf or VTune through perf_event_open into the Linux PMU driver, which programs event-select and counter registers; RDPMC or read returns counts and overflow samples enter a ring buffer
User tools request a measurement contract. Linux owns the hardware schedule and programs the privileged registers.

What Linux Kernel Code Actually Programs The PMU?

A user-space profiler begins by filling a struct perf_event_attr. It specifies the event type and encoding, whether to count or sample, the sampling period or frequency, user/kernel filters, the target process or CPU, and which fields should appear in each sample. The profiler passes that structure to perf_event_open().

The syscall enters kernel/events/core.c at SYSCALL_DEFINE5(perf_event_open, ...). The generic perf subsystem then:

  1. copies and validates the requested attributes;
  2. checks security policy and perf_event_paranoid-related permissions;
  3. selects the PMU registered for the requested event type;
  4. allocates a struct perf_event through perf_event_alloc();
  5. asks the PMU driver's event_init callback to validate and translate the event;
  6. attaches the event to a task context or per-CPU context;
  7. returns a file descriptor representing the event to user space.

This generic code does not know the exact register encoding for every Intel, AMD, Arm, or uncore PMU. Linux PMU drivers register a struct pmu containing callbacks such as event_init, add, start, stop, read, pmu_enable, and pmu_disable. The generic scheduler invokes those callbacks when an event must be placed on hardware.

Linux source areaRepresentative codeResponsibility
include/uapi/linux/perf_event.hperf_event_attr, record formats, ioctl constantsStable user/kernel measurement contract
kernel/events/core.csys_perf_event_open, perf_event_alloc, context installation, scheduling and readsGeneric ownership, permissions, grouping, multiplexing, task/CPU contexts, and file descriptors
include/linux/perf_event.hstruct pmu, struct perf_eventInternal callback and event abstractions used by architecture drivers
arch/x86/events/core.cx86_pmu_add, x86_pmu_start, x86_pmu_stop, x86_perf_event_updateCommon x86 counter assignment, start/stop, period management, and count updates
arch/x86/events/perf_event.h__x86_pmu_enable_eventWrites model-specific event configuration through wrmsrq()
arch/x86/events/intel/ and arch/x86/events/amd/Vendor and generation-specific PMU setupEvent constraints, precise sampling, model quirks, uncore topology, and vendor-specific capabilities
kernel/events/ring_buffer.cperf_output_begin, perf_output_copy, perf_output_endTransfers sample records from the kernel into the perf mmap ring buffer

Linux perf data path from a user tool and perf_event_attr through perf_event_open, generic perf core, x86 vendor PMU callbacks and MSR programming, autonomous hardware counting, overflow PMI handling, mmap ring buffer, and user-space reports
Linux configures and schedules the measurement. The processor counts autonomously. User-space tools turn returned counts and sample records into reports.

When Does Linux Write The Registers?

Opening an event creates the kernel object, but hardware programming happens when the event is scheduled onto an eligible counter and enabled. That can occur immediately for an enabled per-CPU event, when a monitored task is scheduled onto a CPU, after an explicit PERF_EVENT_IOC_ENABLE, or when multiplexing rotates this event onto limited hardware counters.

On x86, x86_pmu_add() assigns an available physical counter under the processor's event constraints. x86_pmu_start() loads a sampling period when needed and enables the event. The helper __x86_pmu_enable_event() uses wrmsrq() to write extra configuration registers and the event-select register. For sampling, x86_perf_event_set_period() writes a negative starting value into the counter so overflow occurs after the requested number of events.

Linux does not increment the counter

After configuration, the PMU's digital logic increments the hardware counter while the qualifying event occurs. There is no kernel function call for every cache miss or instruction. Linux becomes involved again when it reads or reschedules the counter, when a task moves, when multiplexing changes the active event set, or when overflow raises a performance-monitoring interrupt.

How Do Counts Return To User Space?

Counting and sampling use different return paths:

  • Counting: a user tool calls read() on the event file descriptor. The kernel invokes the PMU read callback. On current x86 code, x86_perf_event_update() reads the physical counter with RDPMC, computes the delta from the previous value, and accumulates it into the generic event count. The result can also include time enabled and time running so user space can identify multiplexing and scale carefully.
  • Sampling: counter overflow generates a PMI, commonly routed as an NMI on x86. perf_event_nmi_handler() dispatches to the active x86 PMU interrupt handler. The kernel records requested fields such as instruction pointer, process/thread IDs, time, CPU, call chain, branch stack, data address, registers, or PEBS/IBS data. The record is written into the mmap perf ring buffer.

User space maps that ring buffer. It reads records between the kernel-updated data_head and the consumer-managed data_tail, processes complete records, and advances the tail. This avoids one syscall for every sample.

How Do perf, LIKWID, VTune, And Advisor Produce Reports?

The kernel returns measurements, not interpretations such as “memory bound” or a finished roofline chart.

  • perf stat: opens counting events, reads final values, applies time-running scaling where appropriate, derives ratios such as IPC, and prints text or delimiter-separated output.
  • perf record: consumes sampled records from the mmap ring buffer and writes perf.data. perf report and perf script later resolve instruction addresses against processes, binaries, symbols, source, call chains, and timestamps.
  • LIKWID: can use the Linux perf-event interface, direct MSR/PCI access, or its privileged access daemon. Its performance-group files define event sets and formulas; the tool turns collected counters into derived metrics and can serialize CSV output.
  • VTune: commonly uses Linux Perf driverless hardware-event sampling on Linux, or Intel's sampling drivers for supported capabilities. Its analysis definitions combine event counts, samples, call stacks, topology, and event formulas into viewpoints such as Microarchitecture Exploration and Memory Access.
  • Advisor: combines measured execution data with loop/function structure, operation or trip-count analysis, memory behavior, and calibrated or modeled roofs. A roofline chart is therefore a user-space analytical product built from several measurements, not a structure emitted by the PMU.

CSV, JSON, HTML, flame graphs, timelines, and roofline plots are serialization and analysis layers above the kernel. Two tools may read the same PMU event but present different metrics because they use different formulas, grouping, attribution, calibration, and assumptions. The raw event definition and collection scope remain the evidence boundary.

Next source-level investigation

A follow-up article can trace one real event end to end: construct perf_event_attr, enter sys_perf_event_open, inspect struct pmu dispatch, follow Intel and AMD event initialization, observe wrmsrq programming, trigger a PMI, decode one ring-buffer sample, and reproduce the final count with a small C program.

Is The PMU Per Core Or Universal?

The precise answer is both, but in different hardware blocks.

Core PMUs

Core PMUs observe events near one execution core: instructions, micro-operations, branches, L1 behavior, many L2 events, pipeline stalls, execution ports, and offcore requests initiated by that core. Architecturally, counters are associated with a logical processor's execution context. On an SMT core, both logical processors share physical execution machinery, so event attribution and counter-sharing details can be model-specific. A sibling thread can also change the resource contention experienced by the measured thread.

A core PMU does not reach across the package and inspect another core's private L1 cache. To collect system-wide core events, Linux configures compatible events on each requested logical CPU and aggregates the results. To follow one process, Linux schedules the process's perf context with it: counters are enabled when the task runs and disabled/read when it is switched out or migrated.

Uncore PMUs

“Uncore” refers to processor components outside the individual execution cores. On Xeon platforms these can include caching/home agents, LLC slices, integrated memory controllers, UPI links, mesh interconnect, I/O units, and power-control blocks. Intel's Ice Lake Xeon Uncore Performance Monitoring Reference Manual describes PMUs distributed across these components.

An integrated memory-controller PMU counts traffic at that controller. A CHA PMU observes requests handled by that cache/home agent. A UPI PMU observes its link. These units monitor shared traffic from multiple cores and devices because that traffic physically passes through the monitored block. They still are not one omniscient package-wide observer.

Processor topology with local core PMUs beside private L1 and L2 caches, and separate uncore PMUs beside distributed LLC slices, mesh, memory controllers, and inter-socket links
Observation points follow dataflow. Private-core activity is measured near the core; shared traffic is measured where it crosses cache, fabric, memory, or I/O agents.

The Method Applies Across Intel Xeon And AMD EPYC Generations

This article is not limited to Xeon 600. CKE is targeting that workstation family as a practical future laboratory, but the profiling method applies to older and newer Intel Xeon systems and to AMD EPYC systems. The questions remain stable while the available instructions, cache topology, memory channels, accelerators, event encodings, and vendor tools change.

Processor generationRelevant compute evolutionPMU investigation emphasis
Intel Xeon Scalable 1st GenAVX-512 foundationVector use, frequency behavior, frontend/backend pressure, LLC and memory traffic
Intel Xeon Scalable 2nd GenAVX-512 VNNI adds a stronger INT8 dot-product pathWhether quantized kernels dispatch VNNI and whether memory or packing hides its benefit
Intel Xeon Scalable 3rd GenGeneration and SKU differences include AVX-512 BF16 on Cooper Lake; Ice Lake expands the server platform differentlyExact CPU identity, supported ISA, socket/NUMA topology, uncore traffic, and native event definitions
Intel Xeon Scalable 4th GenAMX INT8/BF16 and integrated accelerators including DSA arrive on Sapphire RapidsAVX-512 versus AMX dispatch, tile reuse, accelerator activity, DDR5 bandwidth, mesh and NUMA placement
Intel Xeon Scalable 5th GenEmerald Rapids retains AMX and expands cache/platform capabilityWhether larger cache and platform changes reduce memory pressure for the exact model and context
Intel Xeon 6 and Xeon 600 workstationP-core products retain AVX-512/AMX and add newer precision and platform capabilities; exact features remain SKU-specificPer-core matrix/vector use, distributed LLC/fabric, memory-channel utilization, accelerator offload, and multi-node behavior
AMD EPYC Zen 2 / Zen 3Strong AVX2-era server execution with chiplet, shared-L3, Data Fabric, and memory topology considerationsCore PMCs, IBS, per-L3 PMCs, Data Fabric, memory channels, xGMI, and NUMA locality
AMD EPYC Zen 4AVX-512 support is introduced using the generation's execution implementationActual vector width, instruction mix, IPC, cache/data-fabric pressure, frequency, and bandwidth
AMD EPYC Zen 5Turin provides a full 512-bit data path and newer platform capabilityRetired 512-bit work, frontend/backend utilization, L3/DF/UMC traffic, and whether wider execution improves the workload outcome

The table is a navigation map, not a substitute for checking the installed SKU. Even processors sold under the same generation can differ in core count, cache, memory channels, accelerator availability, topology, firmware, and enabled events. Always begin with lscpu, CPUID or ISA discovery, perf list --details, PMU sysfs enumeration, and the vendor's processor-specific event documentation.

Using PMUs To Approach The Workload's Practical Limit

  1. Define useful work: exact output, tokens, matrix dimensions, bytes transformed, training steps, or completed requests.
  2. Identify the likely ceiling: vector/matrix compute, cache capacity, DRAM bandwidth, branch/front-end throughput, synchronization, or network movement.
  3. Calculate a bound: use supported instructions, active cores, frequency, memory channels, measured STREAM-like bandwidth, and the operation/byte count for this shape.
  4. Measure the baseline: record wall time, throughput, correctness, cycles, instructions, cache behavior, memory traffic, frequency, and relevant accelerator events.
  5. Attribute the gap: determine whether the unused headroom comes from software dispatch, packing, scalar tails, poor tile reuse, cache misses, NUMA placement, bandwidth, throttling, or waiting.
  6. Change one contract: adjust the kernel, data layout, precision, thread placement, memory policy, batching, or offload boundary.
  7. Repeat with identical evidence: accept the change only when correctness remains valid and the workload-level outcome improves.

Two ratios are especially useful:

compute_efficiency = measured_useful_operation_rate / applicable_compute_ceiling
bandwidth_efficiency = measured_useful_byte_rate / measured_sustainable_bandwidth

Each ratio must compare like units. Useful operations per second must use the same operation definition and precision as the compute ceiling. Useful bytes per second must represent bytes required by the algorithm rather than every incidental cache transfer.

Performance optimization feedback loop from defining useful work and a hardware ceiling through PMU measurement and bottleneck attribution to changing one kernel contract and revalidating correctness
Optimization is a closed experimental loop. A faster inner instruction is irrelevant if the end-to-end workload remains limited somewhere else.

Neither ratio should use marketing peak blindly. If decode is constrained by reading quantized weights from memory, AMX's theoretical multiply rate may be irrelevant. If prefill reuses large packed matrices across a batch, the matrix engine may become the applicable ceiling. PMUs, profiler timelines, disassembly, and numerical tests determine which story matches the execution.

The PMU is the feedback loop, not the optimization

Hardware counters do not make code fast. They prevent blind tuning. The engineer still has to change the generated kernel, circuit schedule, packing format, thread topology, memory placement, or communication plan and then demonstrate that the measured bottleneck moved.

Xeon 600 As A Planned CKE Laboratory Example

Server processors do not replace the core-PMU model with one giant counter. They repeat and extend the distributed model across larger packages. Each logical CPU still has core counters, while shared components expose additional uncore PMUs. Linux enumerates what is physically present and enabled rather than what a motherboard family could theoretically support.

Intel's 2026 Xeon 600 processors for workstation use Redwood Cove+ cores and the W890 workstation platform. The family scales to 86 cores, up to 128 PCIe 5.0 lanes, Intel AMX with FP16 support, and as many as eight DDR5 memory channels on qualifying SKUs. The processor choice matters: Intel's Xeon 600 workstation product brief lists the Xeon 636, 638, and 634 with four memory channels, while higher models such as the 654 and 656 expose eight.

The board does not create missing channels

An eight-channel-capable W890 workstation can accept different Xeon 600 SKUs, but the installed CPU determines how many memory channels and related hardware blocks are active. PMU enumeration and memory-controller counters therefore follow the installed processor, firmware, kernel driver, and actual DIMM population.

Xeon 600 workstation observability map showing per-core AMX and execution counters, distributed LLC and mesh counters, four or eight memory-controller channels depending on SKU, and PCIe or network I/O counters
A Xeon workstation PMU investigation moves from each core toward the shared fabric. The useful result is a joined explanation, not one isolated counter.

1. Did The Generated Kernel Use The Intended Compute Unit?

Core events can establish whether instructions retired, whether execution was frontend- or backend-bound, and whether vector or matrix units were active. Intel's public Granite Rapids core-event database documents EXE.AMX_BUSY, which counts cycles where the AMX unit is busy, alongside architectural and top-down events.

That does not make EXE.AMX_BUSY a universal command for every Xeon 600 workstation. Before using it, inspect the installed system:

perf list --details | grep -i -E 'amx|tmul|topdown|fp_arith'
find /sys/bus/event_source/devices -maxdepth 1 -mindepth 1 -printf '%f\n' | sort
lscpu

AMX-busy cycles answer whether the matrix engine was occupied. They do not prove that the kernel used the best tile shape, avoided conversion overhead, produced correct INT8/BF16/FP16 results, or improved end-to-end tokens per second. Join them with wall time, instructions, cycles, top-down slots, cache traffic, and numerical gates.

2. Can The PMU Distinguish AVX-512, AMX, And DSA?

Yes, but not through one universal “AI acceleration” counter. These are three different execution contracts. AVX-512 instructions execute in a core's vector pipelines. AMX tile instructions execute through the per-core tile-matrix facility. Intel Data Streaming Accelerator, when present and enabled, is a separately configured accelerator reached through work queues. Each path therefore exposes different evidence.

A precision correction matters here. Intel AMX is not a general FP32 matrix engine. Sapphire Rapids-class AMX introduced INT8 dot products with INT32 accumulation and BF16 dot products with FP32 accumulation. Xeon 6 P-core products add FP16 support. FP32 vector arithmetic remains an AVX-512 concern. For quantized inference, both AVX-512 VNNI and AMX INT8 can matter.

AVX-512 VNNI versus AMX INT8 is a measured choice

AVX-512 VNNI is often practical for irregular quantized kernels, matrix-vector decode, small batches, tails, mixed operators, and paths where repacking into AMX tiles would cost too much. AMX INT8 can dominate sufficiently large, reusable matrix tiles and batched matrix-multiply work. Shape, reuse, packing, conversion, thread placement, and memory bandwidth decide the winner. The ISA name does not.

Three PMU evidence paths showing AVX-512 and VNNI in core vector pipelines, AMX INT8 BF16 and FP16 in a per-core tile engine, and DSA as a separate work-queue accelerator connected to memory
AVX-512, AMX, and DSA are not interchangeable counters. The PMU evidence must follow the hardware block that performed the work.

PathWhat executesUseful evidenceWhat the evidence does not prove
AVX-512 / VNNIVector instructions in core execution pipelinesRetired instruction classes where supported, cycles, IPC, top-down slots, port pressure, cache and frequency evidenceThat a specific quantized kernel is optimal or numerically correct
AMXTile loads, stores, configuration, and tile matrix multiply in each supporting coreEXE.AMX_BUSY or documented model-specific events, cycles, cache traffic, tile setup and wall timeUseful operations per busy cycle, good tile reuse, or lower end-to-end latency
DSADescriptors submitted to an accelerator work queue for supported data-movement operationsDSA completion and operation statistics, Intel PCM pcm-accel, IMC bandwidth, core cycles released, queue and wall-time evidenceThat normal memcpy, tensor repacking, quantization, or dequantization was automatically offloaded

PMU support for AVX-512 is useful but model-specific. Some processors expose retired floating-point arithmetic by vector width; integer dot-product and VNNI attribution can require different events. No portable event named avx512_used proves the complete instruction mix. Confirm generated assembly or disassembly first, then use PMU counters to establish how that code behaved at runtime.

objdump -d -M intel generated_kernel.so | \
  grep -i -E 'vpdp|vpmadd|zmm|tdpb|tile(load|store|zero|config)'

perf list --details | \
  grep -i -E 'avx|vnni|fp_arith|amx|tmul|matrix'

DSA has an even harder boundary: it does not watch ordinary loads and decide to help. Software, a library, or a transparent-offload layer must submit supported operations to a configured work queue. Offload has descriptor, queue, completion, and synchronization overhead, so it is normally investigated for sufficiently large copies or asynchronous movement that can overlap compute. A CKE tensor-layout transformation is not a pure copy merely because it moves many bytes; repacking, transposition, quantization, and dequantization still require compute unless the accelerator explicitly implements that transformation.

A defensible DSA experiment therefore compares the original CPU path with an explicit offload path, reports bytes and transfer sizes, verifies output bytes, measures accelerator operations and completions, measures memory-controller traffic, and records whether useful core time was actually released. Intel's open-source Performance Counter Monitor includes pcm-accel for DSA, IAA, and QAT monitoring.

3. Are The Cores Starved By Shared Cache Or Mesh Traffic?

Larger Xeon packages distribute the last-level cache and home/coherence agents across the package fabric. Core events can describe requests originating from a core. Uncore cache/home-agent events observe requests where they reach a shared slice or coherence agent. Mesh and die-to-die paths can become relevant as threads and memory pages spread across a high-core-count package.

The Xeon 600 platform also advertises per-CDIE and per-core performance-limit reporting and per-CDIE ring/mesh tuning. Those controls are not themselves proof of a bottleneck. PMU and frequency evidence should establish whether the run is limited by core execution, shared cache, fabric traversal, power, or thermal policy before tuning.

4. Are Four Or Eight Memory Channels Actually Delivering Bandwidth?

Integrated-memory-controller PMUs count traffic at the controllers. This is the decisive observation point for a memory-bound model because core cache-miss events alone do not reveal whether DRAM channels are balanced or saturated.

For a Xeon 636, the question is how effectively four supported channels are populated and used. For an eight-channel Xeon 600 SKU, the question becomes whether all eight channels are populated symmetrically and receive traffic. Buying an eight-channel CPU but installing too few DIMMs can leave theoretical bandwidth unavailable. Conversely, a four-channel 636 can still be a useful compute and software-development node; it simply tests a different memory-bandwidth envelope.

On a supported Linux system, begin by discovering, not guessing:

numactl --hardware
dmidecode --type memory
find /sys/bus/event_source/devices -maxdepth 1 \
  -iname '*imc*' -o -iname '*uncore*'
perf list --details | grep -i -E 'imc|cas_count|dram|memory.*read|memory.*write'

Exact DRAM bandwidth metrics depend on the event definitions and transfer units for that processor. VTune Memory Access, Intel PCM, and raw perf uncore events can all be useful, but they must agree on socket, controller, channel, direction, duration, and byte conversion.

5. What Changes In A Two-Node Xeon Experiment?

A single-socket workstation PMU cannot directly count activity inside the second workstation. Each node records its own core, cache, memory-controller, PCIe, and network observations. Distributed evidence then aligns those independent timelines with MPI, oneCCL, socket, or RDMA instrumentation.

  • Core PMUs show whether communication overlaps useful compute.
  • Memory-controller PMUs show tensor packing, staging, and receive-buffer traffic.
  • PCIe/I/O PMUs and NIC counters show bytes, queueing, and link behavior.
  • Software traces show collective calls, synchronization, and message sizes.
  • CKE's throughput unit joins compute, memory, communication, and useful model progress.

No single counter says “distributed scaling is good.” The claim requires one-node and two-node wall time, useful model output, CPU utilization, memory traffic, network traffic, synchronization cost, and repeated-run variance.

A Xeon 600 CKE Evidence Matrix

QuestionCore evidenceUncore/platform evidenceOutcome evidence
Is AMX active?AMX-busy or documented matrix events, cycles, top-downPower/frequency limits where availableCorrect output and kernel latency
Is AVX-512 VNNI the better quantized path?Disassembly, model-specific vector events, IPC, top-down, port pressureLLC and IMC traffic, frequency behaviorLatency by shape, batch, packing policy, and exact numerical gate
Does DSA help movement?Core cycles and CPU work avoidedDSA operations/completions, IMC bytes, queue behaviorVerified destination bytes, overlap, and end-to-end wall time
Is decode memory-bound?Backend/memory-bound slots, LLC request outcomesIMC reads, channel bandwidth, LLC/CHA trafficTokens/s versus model bytes moved
Are channels balanced?Core request pressurePer-controller/channel read-write countsLatency and throughput under thread/NUMA placement
Does more threading help?IPC, stalls, context, SMT interferenceMesh, LLC, and DRAM trafficScaling efficiency and tail latency
Does node two help?Compute/communication overlapMemory, PCIe, NIC, and fabric evidence per nodeEnd-to-end speedup and synchronization cost

Does An AMD EPYC PMU Differ From An Intel Xeon PMU?

Yes. Both vendors implement programmable counters, overflow-based sampling, Linux perf integration, and separate monitoring for core and shared-package activity. That common model makes high-level questions portable. The raw events and several important sampling mechanisms are not portable.

BoundaryIntel XeonAMD EPYC
Core PMUArchitectural and model-specific counters, fixed counters, top-down metrics, PEBS on supported eventsCore PMCs with model-specific events; exact counter count and definitions follow the processor PPR
Precise samplingPEBS records supported precise events with platform-specific capabilitiesIBS fetch and IBS operation sampling associate sampled instruction behavior with execution and memory effects
Shared cacheCHA/LLC and other uncore PMUs vary by Xeon generation and package topologyEach shared L3 complex exposes L3 PMCs on supported Zen processors
Package fabric and memoryIMC, UPI, mesh, IIO, PCIe, power, and other uncore blocks expose generation-specific PMUsData Fabric PMCs expose memory, xGMI, PCIe, DMA, and related socket traffic; UMC metrics cover memory controllers
Primary vendor toolVTune, Advisor, Intel PCM, and Linux perfAMD uProf, AMDuProfPcm, and Linux perf

The portable layer is the question: cycles, instructions, IPC, cache behavior, DRAM bandwidth, NUMA locality, branch behavior, vector utilization, and wall time. The vendor-specific layer is how those questions are encoded and attributed. An Intel raw event copied onto AMD can be meaningless even when both commands use perf stat -e r....

AMD's Instruction-Based Sampling is also not simply an AMD spelling of Intel PEBS. Both improve attribution beyond ordinary interrupt sampling, but their hardware records, event rules, skid behavior, and tool workflows differ. AMD uProf documents separate core PMCs, L3 PMCs, Data Fabric PMCs, and IBS sources. A cross-vendor CKE report should preserve the common metric name while recording the native event and source used on each processor.

AMD's PMU capability also evolves across generations. Zen 2, Zen 3, Zen 4, and Zen 5 do not expose one frozen event contract. For example, current AMD uProf documentation provides generation-specific metric groups, while AMD's Turin guidance shows a native event for distinguishing retired scalar, 128-bit, 256-bit, and 512-bit floating-point work. That makes it possible to verify AVX-512 utilization on newer EPYC processors, but the event code and interpretation belong to that processor family.

# Portable discovery first
perf list --details
find /sys/bus/event_source/devices -maxdepth 1 -mindepth 1 \
  -printf '%f\n' | sort

# Vendor identity and topology
lscpu
grep -m1 -E 'vendor_id|model name' /proc/cpuinfo

Do not compare raw counter names; compare answered questions

A Xeon and EPYC can both answer whether a CKE decode loop is memory-bound. Xeon may use core top-down events plus CHA and IMC counters. EPYC may use core PMCs or IBS plus L3 and Data Fabric/UMC counters. The report must normalize the conclusion and units, not pretend the underlying events are identical.

How Are L1, L2, And L3 Events Measured?

Cache controllers already perform tag lookup, hit/miss classification, replacement, fill-buffer allocation, eviction, and coherence transitions. PMU event sources tap selected state transitions or conditions.

  • L1: core events may count retired loads that hit L1, outstanding L1 misses, replacements, or cycles stalled behind demand loads.
  • L2: core or cluster-local events may count requests, hits, misses, lines in/out, prefetch behavior, and occupancy depending on the architecture.
  • LLC/L3: core events can attribute some last-level-cache outcomes to requests originating from a core. Uncore cache/home-agent PMUs can observe traffic at individual distributed LLC slices and coherence agents.
  • DRAM: integrated memory-controller PMUs count reads, writes, commands, queue behavior, and channel traffic at the memory-controller boundary.

Modern Xeon LLCs are physically distributed. Addresses are hashed across cache/home agents, so one core's request may be serviced by a non-co-located LLC slice. Intel's DDIO performance-monitoring guide describes CHA/LLC, mesh, I/O, and memory-controller observation points and explains that PMCs are distributed throughout the architecture.

“Cache misses” is underspecified

A generic cache-misses event may not mean every miss at every cache level. Ask which request type, which cache, which observation point, speculative or retired, local or remote, demand or prefetch, and which processor definition produced the number.

Counting And Sampling Are Different Circuits And Questions

Counting answers: “How many selected events occurred while this measurement was enabled?” Sampling answers: “Where was execution around every Nth occurrence?”

Counting

  1. Linux selects an event and resets or snapshots a counter.
  2. The selected hardware event increments the counter.
  3. Linux disables and reads the counter at the end.
  4. The profiler reports the total, possibly scaled for multiplexing.

Overflow sampling

  1. Linux preloads a counter so it will overflow after approximately N events.
  2. The event increments the counter until overflow.
  3. Overflow status requests a Performance Monitoring Interrupt through the local interrupt machinery.
  4. The kernel records fields requested by sample_type, such as instruction pointer, process/thread ID, time, CPU, call chain, or data address.
  5. The counter is reloaded and execution continues.

The perf_event_open manual defines sample_period, frequency mode, overflow notification, and the mmap ring buffer used to return records. Sampling introduces overhead and statistical error. The interrupt can arrive after the triggering instruction, a displacement known as skid. Precise Event-Based Sampling (PEBS) reduces attribution error for supported events by letting hardware capture a structured record closer to the event.

Two timelines comparing a monotonically increasing PMU count with periodic counter overflow that triggers sample records containing instruction pointers
One total can establish magnitude. Repeated overflow records provide a statistical distribution over code locations.

What This i7-1260P Actually Exposes

The current development laptop is a 12th-generation Intel Core i7-1260P with four performance cores, eight efficient cores, and 16 logical CPUs. Linux does not expose one generic core PMU directory. It exposes:

/sys/bus/event_source/devices/cpu_core
/sys/bus/event_source/devices/cpu_atom
/sys/bus/event_source/devices/uncore_cbox_0
/sys/bus/event_source/devices/uncore_cbox_1
...
/sys/bus/event_source/devices/uncore_imc_0
/sys/bus/event_source/devices/uncore_imc_1

The P-core and E-core PMUs have different event capabilities and encodings. A generic command can therefore expand into both PMU types:

perf stat \
  -e cycles,instructions,branches,branch-misses,cache-references,cache-misses \
  -- sleep 1

The observed report included:

<not counted> cpu_atom/cycles/       (0.00%)
      1809949 cpu_core/cycles/
      2154982 cpu_core/instructions/  # 1.19 insn per cycle

This does not mean the E-core PMU is broken. The short sleep process happened to execute on a P-core during the measured interval, so the corresponding E-core event group was not active. This is a practical example of why event type, CPU placement, enabled time, and running time must accompany the final number.

Why A CPU Cannot Count Everything At Once

A processor can expose hundreds or thousands of named events while implementing only a small number of programmable counters per logical execution context. The selector network lets those counters observe different events, but it does not create one physical counter per event.

When a request needs more counters than are physically available, Linux can multiplex event groups:

  1. Program event set A for part of the run.
  2. Save its counts.
  3. Program event set B.
  4. Scale each result using time_enabled and time_running.

Scaling assumes the sampled intervals represent the full workload. That can be weak when execution has distinct phases such as model loading, prefill, and decode. Prefer compatible event groups, separate controlled runs, repeated measurements, and phase-specific instrumentation over one enormous event list.

Metrics Are Equations Over Event Contracts

IPC = instructions retired / core cycles

branch miss rate = retired mispredicted branches / retired branches

cache miss rate = defined cache misses / defined cache references

memory bandwidth = counted transferred bytes / measured time

Each metric inherits the semantics, scope, multiplexing, skid, privilege filters, SMT interference, frequency behavior, and errata of its input events. A high IPC is not automatically fast if the instruction count is unnecessarily large. A high cache-miss percentage is not automatically harmful if total misses are small. Bandwidth is not useful without knowing which memory channels, read/write direction, socket, and workload phase were measured.

How Linux Turns PMUs Into A Process Measurement

A profiler fills struct perf_event_attr and calls perf_event_open. The request identifies the PMU type, event configuration, task or CPU scope, user/kernel exclusions, inheritance, grouping, sampling period, and requested sample fields.

Linux then:

  1. checks permissions and perf_event_paranoid policy;
  2. selects the matching PMU driver;
  3. validates the event encoding and counter constraints;
  4. schedules the event onto available physical counters;
  5. programs registers when the target task or CPU becomes active;
  6. accumulates, scales, and returns counts or writes samples to a ring buffer.

Event sources appear under /sys/bus/event_source/devices/. Their type, format, events, and sometimes cpumask files describe how Linux addresses them. Uncore PMUs may be package-scoped and represented by one designated CPU for interrupt handling even though the monitored hardware is shared.

kernel.perf_event_paranoid participates in access control. kernel.kptr_restrict is separate: it controls aspects of kernel-pointer visibility needed for kernel symbolization. The preceding article, What kernel.kptr_restrict=0 Actually Does, explains that boundary.

Why The Same VTune Analysis Can Fail On An Older CPU

A CKE deep VTune command completed locally on Ubuntu with the i7-1260P but failed while configuring events on a contributor's Core i5-5200U Broadwell system. The command requested V8_VTUNE_DEEP=1, which adds Memory Access analysis beyond ordinary hotspot collection.

The qualified conclusion is not “Broadwell has no PMU.” Broadwell has core performance counters. The compatibility boundary includes:

  • the exact processor PMU and event set;
  • the installed VTune release and its retained target support;
  • the selected analysis and generated event configuration;
  • the operating-system kernel and distribution;
  • permissions or sampling driver;
  • the command constructed by CKE.

Disabling deep analysis preserves a useful baseline:

make profile-v8-vtune \
  V8_MODEL="$REPORT_MODEL_DIR" \
  V8_PERF_RUNTIME=cli \
  V8_PREP_WITH_PYTHON=0 \
  V8_WITH_VTUNE=1 \
  V8_VTUNE_DEEP=0

The CKE Profiling Capability Contract

LevelEvidenceTarget requirement
Portable baselineWall time, generated-runtime timing, application symbols, flame graphOrdinary supported Linux execution
Core PMUCycles, instructions, branches, cache/TLB events, sampled hotspotsCompatible core PMU, kernel access, and event mapping
Deep platform analysisMemory controllers, socket interconnect, NUMA, uncore, and roofline evidenceQualified processor, profiler, driver, kernel, permissions, and topology

CKE should detect the available PMU types, show which events are valid, record enabled/running time, distinguish counting from sampling, and degrade cleanly. A contributor should not need a new processor to provide a useful build, compatibility fix, portable hotspot report, or flame graph.

A Reproducible PMU Experiment

lscpu
uname -r
cat /etc/os-release
perf --version
perf list --details
find /sys/bus/event_source/devices -maxdepth 1 -mindepth 1 -printf '%f\n' | sort
sysctl kernel.perf_event_paranoid kernel.kptr_restrict

perf stat -r 5 \
  -e cycles,instructions,branches,branch-misses,cache-references,cache-misses \
  -- ./your_workload

perf record -e cycles:u -c 100000 --call-graph dwarf -- ./your_workload
perf report

Record with the report:

  • CPU model, topology, microcode, and active logical CPUs;
  • kernel, distribution, governor, power mode, and thermal state;
  • PMU type and exact event encoding;
  • user/kernel filters, process/CPU scope, and thread placement;
  • time_enabled, time_running, multiplexing, and repetitions;
  • sampling period/frequency, sample fields, call-graph mode, and precise-event level;
  • model hash, CKE commit, generated runtime hash, command, and raw artifacts.

Current evidence boundary

The i7-1260P core/atom split, PMU-device topology, and basic perf stat collection are directly observed. The Broadwell VTune result remains a qualified compatibility case until its exact VTune version and complete target metadata are recorded. No deep uncore or CKE kernel-performance conclusion should be inferred from the short sleep fixture.

Primary References

Continue Reading And Contributing

  • CKE profiling guide connects these counters to real generated C kernels.
  • CKE V8 runbook contains current model and profiler commands.
  • CKE Discord is the discussion channel for Linux profiling, CPU kernels, numerical parity, and contributions.
  • Work with us on a bounded CPU AI, kernel, Linux, numerical, or embedded performance investigation.

The short answer

A CPU PMU is a set of distributed hardware observation points, selectors, counters, and overflow machinery. Core PMUs measure activity near individual execution contexts. Uncore PMUs measure shared traffic where it passes through cache, memory, interconnect, and I/O blocks. Linux coordinates those scarce counters and turns them into safe process- or CPU-scoped measurements; profilers turn the measurements into engineering evidence.