Skip to content
Free shipping on orders over $2,000.

Mode C — full-duplex agent

The full-duplex conversational transport: the hello and ready handshake, PCM in both directions, the mandatory mic-mute rule, button barge-in, and using TCP backpressure as your pacing mechanism.

13 min read

The richest mode: a real two-way conversation with an AI agent, PCM in both directions over one socket. This is the mode behind the table-side ordering terminal, and the default for the Table Top.

Transport

textEndpoint
wss://<host>/ws
  • TLS 1.2 minimum, 1.3 preferred. Validate the server certificate. Ship a CA bundle in flash; pin the CA if you pin anything, never a leaf, or certificate rotation bricks your fleet.
  • SNI required.
  • Two frame types on one socket, distinguished by WebSocket opcode. Binary 0x2 carries raw PCM with no header and no envelope — the WebSocket frame is the envelope. Text 0x1 carries UTF-8 JSON control messages.
  • Do not add your own length prefix or sequence number. Do not batch audio frames.

Audio

Uplink (device → server)Downlink (server → device)
Encodings16le monos16le mono
Rate16 000 Hz exactly16 000 Hz
Frame20 ms = 320 samples = 640 B (10–40 ms acceptable)40 ms = 640 samples = 1280 B

One WebSocket binary frame per audio frame. Batching adds latency directly onto turn detection, which is the part the user feels.

The server writes speech audio to the socket as fast as the socket accepts it. Text-to-speech generates far faster than real time, so a ten-second reply is roughly 320 KB arriving in a burst.

Two consequences worth designing around: your buffer occupancy is your pacing mechanism, and your playback buffer is the only jitter absorption anywhere in the system. Report its high-water mark in telemetry so the buffer can be sized on evidence rather than on a guess.

Handshake

The first message after the upgrade must be a text frame. Declare audio honestly — it is what you will actually send, not what you wish you sent. Same for capabilities.

jsonDevice → server: hello
{
  "type": "hello",
  "protocol": 1,
  "mac": "AA:BB:CC:DD:EE:FF",
  "session": "b7f3a1c2",
  "property_id": 4,
  "table": 5,
  "firmware_version": "2.1.4",
  "ota_version": "2.1.4",
  "audio":        { "encoding": "pcm_s16le", "sample_rate": 16000,
                    "channels": 1, "frame_ms": 20 },
  "capabilities": { "aec": false, "mic_mute_on_playback": true,
                    "button_interrupt": true, "playback_buffer_ms": 400 }
}

Send no audio until ready arrives.

jsonServer → device: ready
{
  "type": "ready",
  "session_id": "srv-9f2c",
  "property_id": 4,
  "table": 5,
  "currency": "PKR",
  "tts": { "encoding": "pcm_s16le", "sample_rate": 16000,
           "channels": 1, "frame_ms": 40 },
  "tail_ms": 300,
  "heartbeat_interval_s": 20
}

Or a failure, after which the socket closes.

CodeMeaningDevice must
unknown_deviceMAC not provisionedStop. Fault indicator. No retry loop
unknown_propertyproperty_id invalidStop. Provisioning fault
protocol_unsupportedVersion mismatchStop. Needs an update
menu_unavailableBackend unreachableRetry with backoff
busyServer at capacityRetry with backoff

Control messages

Device → serverWhenPayload
helloFirst messageSee above
interruptButton pressed while the agent speaks{"type":"interrupt"}
byeClean end{"type":"bye","reason":"user_ended"}
errorDevice fault worth reporting{"type":"error","code":"…","message":"…"}
Server → deviceMeaningDevice action
readyHandshake acceptedStart streaming mic audio
speaking_startAgent audio followsMute the microphone
speaking_endAgent audio finishedUnmute after tail_ms
order_updateCurrent transaction stateOptional: drive a display
errorFaultPer the code table above
closeServer ending the sessionClose cleanly

The mic mute rule — mandatory, not tunable

The microphone must be muted whenever the speaker is producing sound, plus a tail of tail_ms (default 300 ms) afterwards.

Muted means stop sending uplink audio frames entirely. Do not send silence and do not send attenuated audio — anything you send keeps the recognition socket open and burns billable audio seconds. tail_ms arrives in ready so it can be tuned without an update; treat the server's value as authoritative and default to 300 ms if absent.

Button semantics

StatePressAction
IDLEshortStart session: connect, handshake
SPEAKINGshortSend interrupt, stop playback immediately, flush the playback buffer, unmute
LISTENINGshortReserved — no action in v1
anylong (≥ 2 s)Send bye, close the socket, return to IDLE

On interrupt, stop audio locally without waiting for the server. The person pressed the button because they want the talking to stop now; a round trip is not an acceptable delay. Send interrupt in parallel so the server abandons the rest of the turn. Debounce 20 to 50 ms — a bouncing button sends a burst of interrupts.

Heartbeat and liveness

Use WebSocket protocol-level ping and pong (RFC 6455 opcodes 0x9 and 0xA), not application JSON.

ParameterValue
Server → device pingEvery 20 s (heartbeat_interval_s in ready)
Device pong deadline10 s
Declare deadTwo consecutive missed pongs → reconnect
  • TCP does not notice a device that lost power or left the Wi-Fi. The socket stays open server-side until TCP keepalive fires, which on a Linux default is two hours — and until then the server holds a pipeline and a recognition socket for a terminal with nobody at it.
  • Load balancers kill idle connections. Nginx, ALB and Azure Application Gateway commonly default to 60 seconds. The design deliberately goes silent when idle, so without heartbeat traffic the proxy drops a perfectly healthy connection.

A reference client

pythonagent_client.py — handshake, backpressure-aware playback, barge-in
import asyncio, json, websockets

URL  = "wss://api.yourcompany.com/ws"
HELLO = {
    "type": "hello", "protocol": 1,
    "mac": "AA:BB:CC:DD:EE:FF", "session": "b7f3a1c2",
    "property_id": 4, "table": 5,
    "firmware_version": "2.1.4", "ota_version": "2.1.4",
    "audio": {"encoding": "pcm_s16le", "sample_rate": 16000,
              "channels": 1, "frame_ms": 20},
    "capabilities": {"aec": False, "mic_mute_on_playback": True,
                     "button_interrupt": True, "playback_buffer_ms": 400},
}
PERMANENT = {"unknown_device", "unknown_property", "protocol_unsupported"}

async def session(mic, speaker, button):
    async with websockets.connect(URL, ping_interval=None,      # server pings us
                                  max_queue=8) as ws:           # bounded: backpressure
        await ws.send(json.dumps(HELLO))
        ready = json.loads(await ws.recv())
        if ready["type"] == "error":
            if ready["code"] in PERMANENT:
                raise SystemExit(f"permanent: {ready['code']}")  # never retry
            raise ConnectionError(ready["code"])                 # retry w/ backoff

        tail_ms = ready.get("tail_ms", 300)
        muted   = asyncio.Event()

        async def uplink():
            async for frame in mic:                 # 640-byte s16le buffers
                if muted.is_set():
                    continue                        # drop, do NOT send silence
                await ws.send(frame)

        async def downlink():
            async for msg in ws:
                if isinstance(msg, bytes):
                    # Awaiting playback is the pacing mechanism: the receive
                    # window closes behind us and TCP throttles the server.
                    await speaker.play(msg)
                    continue
                evt = json.loads(msg)
                if evt["type"] == "speaking_start":
                    muted.set()
                elif evt["type"] == "speaking_end":
                    await asyncio.sleep(tail_ms / 1000)
                    muted.clear()
                elif evt["type"] == "close":
                    return

        async def barge_in():
            async for press in button:
                if press.long:
                    await ws.send(json.dumps({"type": "bye", "reason": "user_ended"}))
                    return
                if speaker.active:
                    speaker.stop(); speaker.flush()   # locally, first
                    muted.clear()
                    await ws.send(json.dumps({"type": "interrupt"}))

        await asyncio.gather(uplink(), downlink(), barge_in())
javascriptReconnect with jitter — mandatory, not cosmetic
// A site's access point reboots and thirty devices reconnect in the same
// second. Each reconnect costs a TLS handshake, a device lookup, a context
// fetch and a recognition socket. Thirty at once can wedge a process.
let attempt = 0;

function nextDelay() {
  const base = Math.min(2 ** attempt, 60) * 1000;   // 1,2,4,8,16,32,60 s cap
  attempt += 1;
  return base * (0.75 + Math.random() * 0.5);       // ±25 % jitter
}

function onConnected() {
  // Reset only after the connection has survived a while — a socket that
  // opens and dies immediately is still a failing connection.
  setTimeout(() => { attempt = 0; }, 60_000);
}

Something wrong or missing on this page? Tell us.