# Architecture

Source: https://heyparlour.app/docs/architecture

> How Parlour's ports, providers, sessions and pipeline fit together, and why.

This page shows you how Parlour is put together, so you can change it. One npm
package, `parlour`, holds a pure core, the built-in providers behind its
ports, the integrations, the network server and the CLI. The desktop app in `apps/desktop` drives the CLI and knows nothing else.

```text
packages/parlour/src/
  core/          ports.ts, providers.ts, builtins.ts, plugins.ts, skills.ts, config.ts,
                 migrate.ts, paths.ts, secrets.ts, agent.ts, session.ts, audio.ts, endpoint.ts,
                 text.ts, router.ts, queue.ts, triage.ts, tasks.ts, decision.ts, dispatch.ts,
                 loop.ts, speaker.ts, registry.ts, search.ts, timers.ts, prompt.ts, types.ts,
                 events.ts, logger.ts, process.ts, services.ts, localmodel.ts, version.ts
  providers/     audio/ffmpeg, audio/afplay, wake/openwakeword, stt/whisper-cpp, stt/yap,
                 stt/parakeet-mlx,
                 tts/kokoro, tts/macos-say, llm/openai-compatible, llm/anthropic, llm/ai-sdk,
                 decision/laya-mlx, search/searxng, search/brave, secrets/macos-keychain,
                 secrets/file, service/launchd
  integrations/  home-assistant/, mcp/, connectors/
  server/        the HTTP and WebSocket server, discovery.ts, satellite.ts, web/
  cli/           one file per command
  testing/       fakes for every port, published as parlour/testing
```

Nothing in `core/` imports from `providers/` except `core/builtins.ts`. That
file only registers the built-ins. A program that embeds Parlour with its own
providers can leave it out.

## Ports

`core/ports.ts` declares the seams between core and anything that touches
hardware, a model or the network. Each interface is small enough to fake in
a test and to fill from an npm package.

| Port | What it does | Built in |
| --- | --- | --- |
| `AudioSource` | Yields 16 kHz mono frames of 1280 samples (80 ms). | `ffmpeg` on avfoundation |
| `AudioSink` | Plays a WAV, and can be stopped. | `afplay` |
| `WakeWordEngine` | Loads once, hands out one `WakeWordDetector` per stream. | `openwakeword` |
| `SpeechToText` | WAV in, text out. | `whisper-cpp`, `yap`, `parakeet-mlx` |
| `TextToSpeech` | Text in, WAV out, with a `warm()` to load before the first reply. | `kokoro`, `macos-say` |
| `ChatModel` | Messages and tool specs in, a completion out. | `openai-compatible`, `anthropic`, `ai-sdk` |
| `DecisionModel` | A request and typed questions in, answers with probabilities out. | `laya-mlx` |
| `SearchProvider` | A query in, results out. | `searxng`, `brave` |
| `SecretStore` | Where connector tokens are kept. | `macos-keychain`, `file` |
| `ServiceManager` | Keeps Parlour running at login. | `launchd` |
| `Integration` | A named source of tools, prompt lines and a gate. | `home-assistant`, `mcp`, `connectors` |

Every port except `SecretStore` and `ServiceManager` can also expose
`doctor(): Promise<Check[]>`. `parlour doctor` joins every configured part's
checks together and adds three of core's own: the config parses, the network
is let in, and something keeps Parlour running.

Core composes two things itself rather than leaving them to a provider:

- **Speaking.** `Speaker` is a `TextToSpeech` plus an `AudioSink`, with
  sentence splitting and abort. It renders and plays the reply one sentence
  at a time, so the first words come out while the rest is still being made.
  `FallbackTextToSpeech` wraps two voices. When the first throws, it warns
  once, uses the second, and tries the first again on the next sentence.
  `tts.fallback` in config names the second voice. The default is
  `macos-say`, so Parlour still speaks while Kokoro is downloading.
- **Timers.** This is the one tool that has to keep working when the network
  does not. So it lives in core and announces through the `Speaker`.

## Providers

`core/providers.ts` is a registry keyed by `(kind, name)`. The kind is one
of `audioSource`, `audioSink`, `wake`, `stt`, `tts`, `llm`, `decision`,
`search`, `secrets`, `service` or `integration`. Built-ins call
`registerProvider` when they are imported. You select each slot in config by name:

```json
{ "tts": { "provider": "kokoro", "voice": "bf_emma" } }
```

Core only reads the `provider` key of each slice. The rest (`voice`, `url`,
`model` and so on) passes straight through. The provider's own Zod schema
validates it when the provider is created. Core never needs to know that
Kokoro has a voice or that whisper has a URL.

A name that is not registered is tried as a package with `await import(name)`.
Its default export is accepted when its `kind` matches, and a package of
another kind is reported as exactly that rather than as a typo. That is the
whole extension mechanism, and [Writing a provider](/docs/providers) walks you
through it.

You don't choose `secrets` and `service` in config. Core picks them by
platform. It uses the Keychain on a Mac with `security` on the PATH, and
otherwise a file per key under `connector-secrets/` in the config directory.
It uses launchd on macOS. Elsewhere it throws an error saying that a systemd
provider would be a welcome contribution.

## Assembly

`buildAgent(config, secrets, paths)` in `core/agent.ts` does the wiring and
hands back an `Agent`. It loads the plugins, resolves every provider and
wires up the fallback voice. Then it builds the tool registry: every
integration's tools, the search tool unless `search.provider` is `"none"`,
the timers, and `read_skill` when the house has skills. Finally it builds the
`Router`. The microphone loop, the text REPL, the doctor and the network
server all take one `Agent` rather than building their own. That is why a
timer set from a phone still sounds in the room, and why every client shares
one tool list.

Plugins load first, before any provider is resolved. A plugin
(`core/plugins.ts`) is one package that can bring providers, skills and
integration config at once. A provider it registers has to exist by the time
a config slot names it. Its `integrations` block is merged underneath the one
in `config.json`, key by key, so what you wrote wins.

Skills (`core/skills.ts`) are markdown files under `~/.config/parlour/skills`
and in any directories the plugins add. Only each skill's name and
description go in the system prompt. The model fetches the body with the
`read_skill` tool when it decides the rule applies. A local model with a
small context cannot carry every house rule on every turn.
[Skills, MCP and plugins](/docs/skills) is the guide.

The house can run without a cloud model, decision triage or web search. A
provider that cannot start (nearly always a missing key) is left out, and the
doctor reports it. A name that leads nowhere is a config mistake and throws,
so a typo cannot quietly leave you with the wrong model.

## The session

`core/session.ts` is one state machine over a stream of 80 ms frames. It
doesn't know where the frames come from. This machine's microphone, a
satellite relaying audio, a phone or custom hardware all feed the same object.

```text
idle ---wake word---> listening ---silence---> thinking ---answer---> speaking ---> idle
  ^                       |                        |
  |     nothing said      |     nothing heard      |
  +-----------------------+------------------------+
```

- **idle.** Every frame goes to the wake word detector. When it fires, core
  asks every integration's `gate()`. If any says true (the house is muted),
  the wake ends there. Otherwise anything being said is stopped and the
  session starts listening.
- **listening.** Frames go to the endpointer in `core/endpoint.ts`. It gives
  up after 2.5 s of leading silence, finishes after `audio.silenceMs` (800 ms)
  of quiet once speech has started, and caps a request at
  `audio.maxUtteranceMs` (15 s).
- **thinking.** The frames become a WAV and the WAV becomes text. Whisper's
  hallucinations on silence (`[BLANK_AUDIO]`, a lone "Thank you.") are
  dropped by `core/text.ts`, and the text goes to the router.
- **speaking.** The sink says the answer. Frames that arrive while it speaks
  reset the wake word instead of being listened to, unless `audio.bargeIn`
  is on. With one box in one room, the microphone hears the speaker.

Some clients already know where a request starts and stops, such as a phone
with a button or a satellite that did its own wake word. They call
`utterance(frames)` instead of `push(frame)` and skip the first two states.

Every client has its own session, keyed by a client id, so a follow-up in
the kitchen cannot pick up something asked in the study. The router forgets a
session's history after four minutes of quiet, so "turn it off" cannot mean
a light from an hour ago. The id is also the client's lane in the queue and
its own wake word detector. So it decides both what a client can see and how
much of the house it can hold up.

The id comes off the network, so the server trims it to something short and
printable. When two clients claim the same name at once, the server gives
one of them an id of its own. A satellite that reconnects should pick its
conversation back up. The kitchen and a guest's phone, both called
"parlour", should not share one.

## The pipeline

Every request runs through the same stages in `core/router.ts`, whichever
way it came in. The answer goes back to the client that asked, and to
nobody else.

```text
ask -> queue -> triage -> tasks -> action agent -> one thing to say
```

### The queue

A house has more microphones than GPUs. When two satellites hear their wake
word at once, both want the local model. Letting both in together makes each
answer arrive later than taking turns would. The model is one process, and
its batch is one request deep.

`core/queue.ts` gives every client its own lane, which keeps satellites out
of each other's way. One request per lane runs at a time, so a conversation
stays in order and no client holds more than one slot. Lanes take turns as
they finish, so the study cannot starve the kitchen however fast it asks.
`pipeline.concurrency` caps how many requests run at once across the house.

A lane that backs up past `pipeline.queueDepth` drops its own oldest waiting
request, and nobody else's. Someone who asks twice wants the second answer,
and hearing the first one first is worse than not hearing it. The client is
told there is nothing to say rather than given a stale answer.

### Triage

Speech recognition hands over a plausible sentence, not always a correct
one. People say "it" and "in there" and expect the house to know. One breath
is often two jobs. Give a small model all of that at once, with thirty tools
in front of it, and it does the first job and claims the second.

`core/triage.ts` reads the request before anything acts on it. It repairs the
words and resolves pronouns from the conversation and the room. What is left
becomes a list of at most `pipeline.maxTasks` tasks (4 by default). Each is
written to stand on its own and marked as something to do, something to look
up, or small talk. A question triage is
sure needs the clever one is marked so. It skips the local round trip that
would only have ended in `ask_the_clever_one`.

Triage costs a round trip, and "lights off" does not need one. In the default
`auto` mode, a request that is plainly one short instruction goes straight
through. Anything compound, long or ambiguous gets the full pass. `always`
buys the accuracy on every request, and `never` gives it up. Triage is an
optimisation, never a gate. If the model answers with prose, with rubbish or
not at all, the request stays exactly as it was said.

### The task list

`core/tasks.ts` holds what the request turned out to be. The action agent
takes one task at a time, in the order they were said. Each task sees what
the tasks before it did, so "turn the lamp on and tell me if it worked" has
something to work with. The replies are joined into one thing to say, and
two jobs that both answer "Done." become one "Done." rather than a stutter.
A task that fails abandons the rest, because you are owed one apology, not
three.

### The action agent

`core/dispatch.ts` is local first. The local model gets the system prompt,
the tools and one extra tool, `ask_the_clever_one`. The prompt tells it to
call that tool whenever a question needs real reasoning, current information
or knowledge it is unsure of. When it does, the rewritten question and the
session's history go to the cloud model, which has server-side web search
and no house tools. If the local model throws or times out and
`llm.cloud.onLocalFailure` is true, the cloud model gets the original
question instead. Small talk gets no tools at all. That is faster, and it is
the only reliable way to stop a small model calling one.

With no cloud model configured, three things change together. The escalation
tool is not offered, the persona stops naming it, and failures are final. A
small model told to hand over, with nobody to hand over to, ends up
apologising instead of turning the light off. Small models trained on the
pattern sometimes ask for the clever one anyway. The loop then answers the
call with "there is nobody to hand this to, use the tools you have" rather
than an unknown-tool error, so the next round does the job.

You can also add a decision model. `llm.decision` names an on-device model
that scores each task before the local model takes its turn, and decides
whether it should stay local or go to the cloud. `core/decision.ts` asks the
questions and turns the answers into a plan. The built-in model is
`laya-mlx` (`providers/decision/laya-mlx.ts`). It needs Apple silicon and
`pip install laya-mlx`, and it stays warm in a local Python worker, so the
transcript never leaves your Mac for the decision. The default is
`provider: "none"`.

- **`shadow` mode** (the default) only logs the scores beside whatever the
  local model did.
- **`triage` mode** acts on them, when there is a cloud model to hand to. A
  `needs_cloud` or `needs_web` score at or above `escalateThreshold` (0.85)
  skips the local model and hands the task to the cloud. A house or timer
  intent at or above `localConfidence` (0.75) keeps the escalation tool off,
  so the answer stays on your Mac.

Like triage, it is an optimisation. If the decision model fails, the task
carries on locally.

`core/loop.ts` runs the tool rounds for whichever model is answering, up to
`llm.maxToolRounds`. It stops with an apology rather than looping forever.
Every tool the model asked for in one round runs at once. The model asked for
all of them before seeing any answers, so nothing in the round depends on
anything else in it. Two lights cost the slower one, not the sum.

`pipeline.timeoutMs` is the ceiling on a whole request, checked between
rounds and between tasks. Whatever is left of a request that runs past it is
abandoned, rather than spoken ten seconds after everyone has left the room.

## Configuration, secrets and paths

| What | Where |
| --- | --- |
| `config.json`, `secrets.env`, `connectors.json`, `skills/` | `~/.config/parlour/`, or `$PARLOUR_HOME` |
| Just the config file | `$PARLOUR_CONFIG`, or `parlour --config <file>` |
| Models | `~/Library/Caches/parlour/models/{openwakeword,whisper,llm}` |
| Logs | `~/Library/Logs/parlour/{agent,whisper,llm}.log` |
| LaunchAgent labels | `io.parlour.agent`, `io.parlour.whisper`, `io.parlour.llm` |
| Bonjour service type | `_parlour._tcp` |
| Keychain service for connector tokens | `parlour-connector` |
| The desktop app's own settings | `~/Library/Application Support/io.parlour.desktop/settings.json` |

`config.json` is a Zod schema in which every key has a default. An empty
file is a complete config, and `parlour config show` prints what is in force.
Secrets (`HA_TOKEN`, `ANTHROPIC_API_KEY`, `PARLOUR_TOKEN`, `BRAVE_API_KEY`,
`LOG_LEVEL`) stay in `secrets.env` with mode 600. The process environment
wins over the file, so you can override something for a one-off run without
editing anything.

On Linux, the cache and logs follow `XDG_CACHE_HOME` and `XDG_STATE_HOME`.
Nothing else about Linux is done yet, and we would be glad of your help.

## Events

`parlour start --events` prints one JSON line per state change on stdout:
`ready` (with the tool count and whether a cloud model is up), `state`,
`heard`, `reply` (with `via` local or cloud and the time taken in `ms`),
`muted` and `error`, each with an `at` timestamp. The desktop app reads these to draw its status instead of
scraping log text. Without the flag, stdout is a log for people to read.

## Why each part was chosen

| Stage | Choice | Why |
| --- | --- | --- |
| Wake word | openWakeWord, ONNX, in process | Free and offline. It runs in the same Node process as everything else, which makes barge-in possible. |
| Capture | ffmpeg on avfoundation | Nothing to compile, and the only reliable way to pin one input device on macOS. |
| Speech to text | whisper.cpp `small.en`, kept warm | Loading the model costs more than transcribing a sentence, so it runs as a server. |
| Model | llama.cpp with a Qwen3.5 or Gemma 4 GGUF, kept warm | It comes with Parlour, so the house answers straight after `init`, with no LM Studio to install and no server to remember to start. It speaks the OpenAI API, so LM Studio, Ollama or anything else can take its place. |
| Escalation | Claude with server-side web search | The local model is fast and private but wrong more often. Handing over is cheap. Being wrong out loud is not. |
| Speech | Kokoro 82M, ONNX | Close to a cloud voice, runs in process, and has British voices. `say` is the fallback when it fails. |

Every stage adds to the wait before an answer, so each one is chosen to be
fast rather than best.