I Want To Use My Linux Workstation Without Looking At It

I have spent a lot of time building software that lets me run AI models on CPUs. Now I want to use some of that work for a more personal reason: my eyes hurt after too much time on the screen. I still want to work with my coding agents, follow experiments on my CPU nodes, and move between CKE, Antsand, and robotics work. I just do not want to stare at ten terminal windows to do it.

It sounds a little funny to say I want a visionless workstation while I am also working on vision models. But I mean it literally as a user goal: for routine work, I should be able to close my eyes and still know which agent is doing what, ask a question, hear the useful answer, and stop or correct an action. I can always open my eyes when a graph, PCB, video frame, or UI genuinely needs inspection.

The first test was not speech

We have already tested one small but important control path on my AwesomeWM desktop. We identified a live Terminator window running a coding agent, focused it, typed a request with xdotool, pressed Enter, and confirmed that the agent received the request. That proves we can reach a running agent. It does not prove robust speech control, reliable response capture, or safe routing after a window is moved or restarted.

The first design I imagined used window numbers: go to workspace two, then Terminator tab three, then read the agent there. That can help me remember a desktop visually. But if I am not looking at it, those numbers should not be the primary interface. I want to say, "Ask the Parakeet agent for an update," and let a controller resolve the named work session. AwesomeWM can still arrange the windows. A persistent session registry, ideally tied to tmux or PTYs rather than fragile X11 IDs, should hold the actual identity.

This grows out of the LinuxUtilities control layer I have already been building. AwesomeWM gives me a spatial map of screens, workspaces and clients. LinuxUtilities already has ways to inspect that state, manage audio and recover my Sony headset when Bluetooth or PipeWire gets confused. I want those deterministic tools to answer "where am I?" before asking a vision model to guess from a screenshot.

The workstation would have one voice-facing coordinator, or "mother agent," that knows the names and states of the other agents. It should receive speech from my Sony headset, send the request to the right session, capture the answer, and speak a short, useful summary. I should be able to ask for the full text, repeat a paragraph, pause playback, or cancel. A joystick button can provide an immediate physical push-to-talk or stop control, but I do not want a joystick button for every window and tab.

Conceptual path from a headset and game controller through a Linux workstation to three named work sessions, with a spoken response returning to the headset.
Concept illustration of the workflow I want to build, not a screenshot of a working voice controller.

That is also why adding another desktop model app is not the main answer for me. I am trying to stop looking at a UI, not collect more of them. The speech controller needs stable commands and model interfaces; if a model manager cannot actually use CKE as its inference engine, it can be interesting to study without becoming a dependency of this workflow.

One conversation, several agents

In a normal work session I might have one agent working on a CKE audio kernel, another reviewing its tests, and another editing an Antsand page. I do not want to memorize which Terminator tab each one occupies. I want to ask the mother agent, "Which of them needs me?" and hear something like: "The kernel test is still running. The docs agent has a draft ready. The Antsand agent needs approval before publishing." If I ask it to tell the docs agent to explain a result, it should address that named session, not whichever terminal happens to have focus.

The registry behind that conversation needs more than a window title. Each child should have a stable name, repository, machine, tmux or PTY address, current state, last request and last meaningful response. The T14 can remain my headset and control endpoint while P3 or Ryzen run long jobs without a graphical desktop. AwesomeWM still helps me navigate when I want to see the work, but it does not have to be the source of truth for an agent's identity.

I have a wired Logitech controller connected now. Linux currently identifies it as a Logitech Dual Action, even though I have talked about the F310 and F710 as possible controllers. I would start with that actual device. The proposed map is deliberately small: hold a button to talk, ask "where am I?", cycle named agents, read the latest useful response, and stop speech immediately. D-pad movement can announce workspaces; an LB modifier can move through child agents. The final button codes need to be discovered from this controller rather than assumed from the labels on a different model. None of this gamepad navigation is a tested feature yet.

A proposed six-action map for the wired controller: hold to talk, announce location, move between workspaces or named agents, read the latest answer, and stop.
This is the button map I want to try on my wired controller. I still need to check what Linux reports for each button. Open the diagram for a larger view.

What CKE contributes today

CKE already runs Whisper speech recognition on CPUs, and I have used its output in my real YouTube editing workflow. A retained 42:22 recording and five-minute regression fixture make this more than an isolated demo. But transcribing a long file quickly is not the same as low-latency conversation. Push-to-talk, short utterances, audio-device recovery, and the time from button release to actionable text still need their own tests.

That video-transcription experiment also found a CKE compiler bug. It is useful evidence for the speech input side, not evidence that a live headset conversation works. Generated Parakeet and Cohere audio circuits give me more models to evaluate later; Whisper is enough to build the first interactive loop.

For the other direction, CKE is beginning a Kokoro text-to-speech path. The Kokoro bring-up checklist traces the model from phoneme IDs through duration prediction, prosody, text encoding, and waveform generation. Recent PRs added checked runtime extents, a generated-C call path for a bounded audio fixture, an isolated bidirectional LSTM scan provider, and a style-conditioned adaptive LayerNorm kernel. That last operation uses the selected voice to change how the model normalizes its features; it has bounded buffers and independent primitive tests, but is not yet connected into a complete Kokoro circuit.

The model side is getting less hand-wavy too. The pinned weight and voice exporter records where the model tensors came from and checks their imported values. Predictor checkpoints capture intermediate LSTM and adaptive LayerNorm results so a future generated circuit can be compared stage by stage with the reference. The operation checklist separates tested primitives from candidate math and missing graph connections. This is how I want to harden speech generation: verify each kernel, bind the real weights, compare the connected graph, and only then listen to the waveform. Export and checkpoints make that process more reproducible; they are not speech output.

Conceptual phoneme tiles passing through unfinished speech-generation stages toward an incomplete waveform.
Concept illustration. The incomplete waveform is deliberate: CKE has not yet generated and validated a complete Kokoro utterance.

I want to be clear about the present boundary: CKE does not yet generate a complete Kokoro utterance or play it through my headset. A passing duration-expansion or inverse-STFT oracle is a useful kernel result, not a working voice assistant. The first meaningful TTS gate is fixed phoneme IDs and one pinned voice producing a checked FP32 waveform through generated C. Native text normalization and pronunciation come after that. Then come PCM output, playback, cancellation, and latency measurements.

Until that path works, I can use an existing Linux speech engine as a temporary output adapter. The controller should depend on a simple speak(text) contract, not on Kokoro being finished. That lets the accessibility workflow improve now while CKE's TTS math is hardened properly.

The control loop I actually want

  1. Press push-to-talk on the headset or joystick and speak a short request.
  2. CKE Whisper returns text, with an easy way for me to hear or correct what it understood.
  3. The coordinator resolves a named agent session, not a guessed screen coordinate.
  4. It asks the agent, captures the response, and distinguishes "working," "needs approval," and "finished."
  5. A speech queue reads a concise summary; I can pause, repeat, or request detail.
Proposed request and reply path from Sony headset and gamepad through LinuxUtilities, CKE Whisper, a coordinator, named agents, and speech playback. Existing pieces are marked separately from unbuilt integration.
This is how I picture one request making the round trip. Some pieces exist; the complete loop does not yet run. Open the diagram for a larger view.

The machine should read AwesomeWM and terminal metadata before taking screenshots. A vision model is useful for a Kdenlive preview, an unexpected dialog, or a diagram. It should not have to infer which terminal is active from pixels when the window manager already knows. This also matters for speed and reliability on CPU-only hardware.

The audio side has its own plumbing. Holding push-to-talk should make LinuxUtilities verify that the Sony microphone is actually active, record the utterance, and release cleanly even if the controller disconnects. Once CKE Whisper returns text, the coordinator should say what it understood and which agent it intends to address. Safe status questions can go through; publishing, deleting, flashing hardware or sending a command to the wrong repository cannot be a guessed action. The coordinator needs a small set of typed tools for focus, session status, submit, read and cancel, instead of inventing raw xdotool commands.

On the way back, I want the mother agent to read the child agent's useful result, not every line of a build log. A speech queue should let me pause, repeat a sentence and stop from the controller or media keys. An existing Linux TTS engine can serve that role first; CKE Kokoro is a future replacement when the generated waveform is real. The Sony headset may need to switch between microphone/call mode and better-quality playback mode, so the system must announce the active route and recover when it is wrong. A response that cannot be captured should be reported as unavailable, never summarized from a guess.

Here is the first test I want to pass on my own workstation: with one named agent running in a tmux session, I press push-to-talk and ask for its status without looking at the screen. The system announces the recognized request and target, captures the agent's actual response, then reads a short summary through the headset. I can ask it to repeat, pause it, or cancel before it sends a new command. A stale window ID, an ambiguous session name, a disconnected microphone, or an empty response must produce an error I can hear, not a plausible answer invented by the coordinator. I would record each step and its elapsed time so we can distinguish speech-recognition delay from agent work and playback delay. Until that loop passes repeatedly, I would call this a roadmap rather than an accessible workstation.

There must be a safety boundary. Reading status is not the same as sending Enter to a live shell. Publishing a blog, deleting files, flashing hardware, or buying something should require explicit confirmation. The system should say which agent and which repository it is about to affect. If microphone capture fails, Bluetooth switches to the wrong profile, or the agent response cannot be captured, it should fail visibly and audibly rather than silently guessing.

I would build this in layers. First, make the wired controller announce the correct AwesomeWM location and survive reconnects without an AI model. Second, give three child agents stable named sessions and prove that requests and responses cannot cross routes. Third, add short-utterance Whisper and transcript confirmation. Fourth, add controllable speech playback through the existing Linux engine. Only then should CKE-generated Kokoro speech and occasional screenshot/vision fallback enter the live loop. That sequence lets me tell whether a failure belongs to the controller, audio profile, agent routing or model math.

A five-stage build order: one tested terminal-to-agent command path, followed by planned controller navigation, named agent routing, speech input, and spoken replies.
I want to get each step working on my desk before I depend on the next one. So far, only the first command path has been demonstrated. Open the diagram for a larger view.

Headless computers can do the work

The interesting part is that my P3 and Ryzen nodes do not need desktops. They can run CKE jobs, model tests, and child agents as named headless sessions. The T14 remains my audio and control endpoint. I might add another small CPU node for always-available speech and orchestration, but that is an engineering choice to measure, not a prerequisite for the first prototype.

This project joins several things I have been building separately: LinuxUtilities handles the workstation and Sony headset; AwesomeWM provides deterministic workspace state; CKE supplies speech recognition today and is growing toward native speech output; the agents do the coding and research work. The coordinator is the missing piece that makes them feel like one system.

I am not claiming we have an accessibility product yet, and this is not a replacement for a screen reader or AT-SPI. I am describing a workflow I personally want to use and can test every day. The first milestone is simple: speak to one named agent, get a reliable spoken update, and recover cleanly when the audio or terminal path breaks. Then we can add more agents, gamepad controls, and eventually CKE-generated speech. If I can work through a day with less eye strain and still keep my projects moving, that will be a far more useful result than another polished dashboard.

Related reading

Evidence checked against CKE 3307fe56a, September 26, 2026. This post is a roadmap, not a claim that the voice-control workflow or Kokoro waveform has been completed.