A companion that listens, remembers, and speaks.

01Problem

Companion apps promise presence and deliver a chat box. Text is too slow for intimacy, and off-the-shelf voice stacks add three to five seconds of dead air per turn — which reads as indifference. The product question wasn't "can an LLM chat" but "can a machine hold a conversation that feels alive."

02Discovery

Prototyping showed the emotional ceiling was set almost entirely by latency and voice quality — not model intelligence. Users forgave a shallow answer delivered instantly with warmth; they abandoned brilliant answers that arrived late in a flat voice. That inverted the roadmap: the voice pipeline became the product, the model became a component.

03Architecture

04Engineering

• Speak-while-thinking. The LLM streams tokens into a sentence segmenter; each sentence is synthesized and played while the next is still being generated. First audio lands before the model finishes its thought.
• Custom TTS/STT. Off-the-shelf voices broke the character illusion. A tuned voice stack gave each character a consistent identity — and cut per-minute cost enough to make long sessions viable.
• Live2D expression channel. Sentiment and phoneme events ride a side-channel to the renderer, so the face reacts during speech, not after it.
• Interruption handling. Barge-in detection cancels synthesis mid-sentence and re-plans the turn — the difference between a recording and a conversation.

# the core loop: audio out begins before the model finishes
async def speak(turn):
    async for text in llm.stream(turn.context):
      for sentence in segmenter.push(text):
          audio = tts.synthesize(sentence)   # ~120ms to first chunk
          await ws.send(audio)               # play while thinking
          emit_expression(sentence.sentiment)

05Impact

06Lessons

Emotional products are latency products. Every architectural decision — segmentation, chunked synthesis, barge-in — existed to protect a feeling, not a metric. When the engineering worked, nobody mentioned the engineering. That was the goal.