Architecture

This page shows you how Parlour is put together, so you can change it. One npm package, parlour, holds a pure core, the built-in providers behind its ports, the integrations, the network server and the CLI. The desktop app in apps/desktop drives the CLI and knows nothing else.

packages/parlour/src/
  core/          ports.ts, providers.ts, builtins.ts, plugins.ts, skills.ts, config.ts,
                 migrate.ts, paths.ts, secrets.ts, agent.ts, session.ts, audio.ts, endpoint.ts,
                 text.ts, router.ts, queue.ts, triage.ts, tasks.ts, decision.ts, dispatch.ts,
                 loop.ts, speaker.ts, registry.ts, search.ts, timers.ts, prompt.ts, types.ts,
                 events.ts, logger.ts, process.ts, services.ts, localmodel.ts, version.ts
  providers/     audio/ffmpeg, audio/afplay, wake/openwakeword, stt/whisper-cpp, stt/yap,
                 stt/parakeet-mlx,
                 tts/kokoro, tts/macos-say, llm/openai-compatible, llm/anthropic, llm/ai-sdk,
                 decision/laya-mlx, search/searxng, search/brave, secrets/macos-keychain,
                 secrets/file, service/launchd
  integrations/  home-assistant/, mcp/, connectors/
  server/        the HTTP and WebSocket server, discovery.ts, satellite.ts, web/
  cli/           one file per command
  testing/       fakes for every port, published as parlour/testing

Nothing in core/ imports from providers/ except core/builtins.ts. That file only registers the built-ins. A program that embeds Parlour with its own providers can leave it out.

Ports

core/ports.ts declares the seams between core and anything that touches hardware, a model or the network. Each interface is small enough to fake in a test and to fill from an npm package.

Every port except SecretStore and ServiceManager can also expose doctor(): Promise<Check[]>. parlour doctor joins every configured part's checks together and adds three of core's own: the config parses, the network is let in, and something keeps Parlour running.

Core composes two things itself rather than leaving them to a provider:

• Speaking. Speaker is a TextToSpeech plus an AudioSink, with sentence splitting and abort. It renders and plays the reply one sentence at a time, so the first words come out while the rest is still being made. FallbackTextToSpeech wraps two voices. When the first throws, it warns once, uses the second, and tries the first again on the next sentence. tts.fallback in config names the second voice. The default is macos-say, so Parlour still speaks while Kokoro is downloading.
• Timers. This is the one tool that has to keep working when the network does not. So it lives in core and announces through the Speaker.

Providers

core/providers.ts is a registry keyed by (kind, name). The kind is one of audioSource, audioSink, wake, stt, tts, llm, decision, search, secrets, service or integration. Built-ins call registerProvider when they are imported. You select each slot in config by name:

{ "tts": { "provider": "kokoro", "voice": "bf_emma" } }

Core only reads the provider key of each slice. The rest (voice, url, model and so on) passes straight through. The provider's own Zod schema validates it when the provider is created. Core never needs to know that Kokoro has a voice or that whisper has a URL.

A name that is not registered is tried as a package with await import(name). Its default export is accepted when its kind matches, and a package of another kind is reported as exactly that rather than as a typo. That is the whole extension mechanism, and Writing a provider walks you through it.

You don't choose secrets and service in config. Core picks them by platform. It uses the Keychain on a Mac with security on the PATH, and otherwise a file per key under connector-secrets/ in the config directory. It uses launchd on macOS. Elsewhere it throws an error saying that a systemd provider would be a welcome contribution.

Assembly

buildAgent(config, secrets, paths) in core/agent.ts does the wiring and hands back an Agent. It loads the plugins, resolves every provider and wires up the fallback voice. Then it builds the tool registry: every integration's tools, the search tool unless search.provider is "none", the timers, and read_skill when the house has skills. Finally it builds the Router. The microphone loop, the text REPL, the doctor and the network server all take one Agent rather than building their own. That is why a timer set from a phone still sounds in the room, and why every client shares one tool list.

Plugins load first, before any provider is resolved. A plugin (core/plugins.ts) is one package that can bring providers, skills and integration config at once. A provider it registers has to exist by the time a config slot names it. Its integrations block is merged underneath the one in config.json, key by key, so what you wrote wins.

Skills (core/skills.ts) are markdown files under ~/.config/parlour/skills and in any directories the plugins add. Only each skill's name and description go in the system prompt. The model fetches the body with the read_skill tool when it decides the rule applies. A local model with a small context cannot carry every house rule on every turn. Skills, MCP and plugins is the guide.

The house can run without a cloud model, decision triage or web search. A provider that cannot start (nearly always a missing key) is left out, and the doctor reports it. A name that leads nowhere is a config mistake and throws, so a typo cannot quietly leave you with the wrong model.

The session

core/session.ts is one state machine over a stream of 80 ms frames. It doesn't know where the frames come from. This machine's microphone, a satellite relaying audio, a phone or custom hardware all feed the same object.

idle ---wake word---> listening ---silence---> thinking ---answer---> speaking ---> idle
  ^                       |                        |
  |     nothing said      |     nothing heard      |
  +-----------------------+------------------------+

• idle. Every frame goes to the wake word detector. When it fires, core asks every integration's gate(). If any says true (the house is muted), the wake ends there. Otherwise anything being said is stopped and the session starts listening.
• listening. Frames go to the endpointer in core/endpoint.ts. It gives up after 2.5 s of leading silence, finishes after audio.silenceMs (800 ms) of quiet once speech has started, and caps a request at audio.maxUtteranceMs (15 s).
• thinking. The frames become a WAV and the WAV becomes text. Whisper's hallucinations on silence ([BLANK_AUDIO], a lone "Thank you.") are dropped by core/text.ts, and the text goes to the router.
• speaking. The sink says the answer. Frames that arrive while it speaks reset the wake word instead of being listened to, unless audio.bargeIn is on. With one box in one room, the microphone hears the speaker.

Some clients already know where a request starts and stops, such as a phone with a button or a satellite that did its own wake word. They call utterance(frames) instead of push(frame) and skip the first two states.

Every client has its own session, keyed by a client id, so a follow-up in the kitchen cannot pick up something asked in the study. The router forgets a session's history after four minutes of quiet, so "turn it off" cannot mean a light from an hour ago. The id is also the client's lane in the queue and its own wake word detector. So it decides both what a client can see and how much of the house it can hold up.

The id comes off the network, so the server trims it to something short and printable. When two clients claim the same name at once, the server gives one of them an id of its own. A satellite that reconnects should pick its conversation back up. The kitchen and a guest's phone, both called "parlour", should not share one.

The pipeline

Every request runs through the same stages in core/router.ts, whichever way it came in. The answer goes back to the client that asked, and to nobody else.

ask -> queue -> triage -> tasks -> action agent -> one thing to say

The queue

A house has more microphones than GPUs. When two satellites hear their wake word at once, both want the local model. Letting both in together makes each answer arrive later than taking turns would. The model is one process, and its batch is one request deep.

core/queue.ts gives every client its own lane, which keeps satellites out of each other's way. One request per lane runs at a time, so a conversation stays in order and no client holds more than one slot. Lanes take turns as they finish, so the study cannot starve the kitchen however fast it asks. pipeline.concurrency caps how many requests run at once across the house.

A lane that backs up past pipeline.queueDepth drops its own oldest waiting request, and nobody else's. Someone who asks twice wants the second answer, and hearing the first one first is worse than not hearing it. The client is told there is nothing to say rather than given a stale answer.

Triage

Speech recognition hands over a plausible sentence, not always a correct one. People say "it" and "in there" and expect the house to know. One breath is often two jobs. Give a small model all of that at once, with thirty tools in front of it, and it does the first job and claims the second.

core/triage.ts reads the request before anything acts on it. It repairs the words and resolves pronouns from the conversation and the room. What is left becomes a list of at most pipeline.maxTasks tasks (4 by default). Each is written to stand on its own and marked as something to do, something to look up, or small talk. A question triage is sure needs the clever one is marked so. It skips the local round trip that would only have ended in ask_the_clever_one.

Triage costs a round trip, and "lights off" does not need one. In the default auto mode, a request that is plainly one short instruction goes straight through. Anything compound, long or ambiguous gets the full pass. always buys the accuracy on every request, and never gives it up. Triage is an optimisation, never a gate. If the model answers with prose, with rubbish or not at all, the request stays exactly as it was said.

The task list

core/tasks.ts holds what the request turned out to be. The action agent takes one task at a time, in the order they were said. Each task sees what the tasks before it did, so "turn the lamp on and tell me if it worked" has something to work with. The replies are joined into one thing to say, and two jobs that both answer "Done." become one "Done." rather than a stutter. A task that fails abandons the rest, because you are owed one apology, not three.

The action agent

core/dispatch.ts is local first. The local model gets the system prompt, the tools and one extra tool, ask_the_clever_one. The prompt tells it to call that tool whenever a question needs real reasoning, current information or knowledge it is unsure of. When it does, the rewritten question and the session's history go to the cloud model, which has server-side web search and no house tools. If the local model throws or times out and llm.cloud.onLocalFailure is true, the cloud model gets the original question instead. Small talk gets no tools at all. That is faster, and it is the only reliable way to stop a small model calling one.

With no cloud model configured, three things change together. The escalation tool is not offered, the persona stops naming it, and failures are final. A small model told to hand over, with nobody to hand over to, ends up apologising instead of turning the light off. Small models trained on the pattern sometimes ask for the clever one anyway. The loop then answers the call with "there is nobody to hand this to, use the tools you have" rather than an unknown-tool error, so the next round does the job.

You can also add a decision model. llm.decision names an on-device model that scores each task before the local model takes its turn, and decides whether it should stay local or go to the cloud. core/decision.ts asks the questions and turns the answers into a plan. The built-in model is laya-mlx (providers/decision/laya-mlx.ts). It needs Apple silicon and pip install laya-mlx, and it stays warm in a local Python worker, so the transcript never leaves your Mac for the decision. The default is provider: "none".

• shadow mode (the default) only logs the scores beside whatever the local model did.
• triage mode acts on them, when there is a cloud model to hand to. A needs_cloud or needs_web score at or above escalateThreshold (0.85) skips the local model and hands the task to the cloud. A house or timer intent at or above localConfidence (0.75) keeps the escalation tool off, so the answer stays on your Mac.

Like triage, it is an optimisation. If the decision model fails, the task carries on locally.

core/loop.ts runs the tool rounds for whichever model is answering, up to llm.maxToolRounds. It stops with an apology rather than looping forever. Every tool the model asked for in one round runs at once. The model asked for all of them before seeing any answers, so nothing in the round depends on anything else in it. Two lights cost the slower one, not the sum.

pipeline.timeoutMs is the ceiling on a whole request, checked between rounds and between tasks. Whatever is left of a request that runs past it is abandoned, rather than spoken ten seconds after everyone has left the room.

Configuration, secrets and paths

config.json is a Zod schema in which every key has a default. An empty file is a complete config, and parlour config show prints what is in force. Secrets (HA_TOKEN, ANTHROPIC_API_KEY, PARLOUR_TOKEN, BRAVE_API_KEY, LOG_LEVEL) stay in secrets.env with mode 600. The process environment wins over the file, so you can override something for a one-off run without editing anything.

On Linux, the cache and logs follow XDG_CACHE_HOME and XDG_STATE_HOME. Nothing else about Linux is done yet, and we would be glad of your help.

Events

parlour start --events prints one JSON line per state change on stdout: ready (with the tool count and whether a cloud model is up), state, heard, reply (with via local or cloud and the time taken in ms), muted and error, each with an at timestamp. The desktop app reads these to draw its status instead of scraping log text. Without the flag, stdout is a log for people to read.

Why each part was chosen

Every stage adds to the wait before an answer, so each one is chosen to be fast rather than best.