LLM chat streaming

Three conversations with a mock LLM. Each reply streams in token by token. All three can stream at the same time, but only the chat you’re looking at is on screen.

Things to try:

Open a chat and watch the reply stream in. Later tokens update in place. The Suspense fallback only ever shows before the first token.
Switch away mid-stream, wait a moment, and switch back. The first chat kept streaming while hidden. It reveals instantly, fully caught up.
Hover a chat you haven’t opened before clicking it. The reply starts streaming in the background, so opening it skips the fallback entirely.

import {
  Activity,
  Suspense,
  use,
  useState,
} from 'react'
import {
  preloadObservablePromise,
  useObservablePromise,
} from 'react-rx'

import {
  CHATS,
  conversation$,
  type Chat,
} from './chat'

function ChatView({chat}: {chat: Chat}) {
  // Suspends until the first token, then streams in place. Later emissions
  // never re-trigger the Suspense fallback.
  const messages = use(
    useObservablePromise(conversation$(chat)),
  )

  return (
    <div>
      {messages.map((message) =>
        message.role === 'user' ? (
          <p key="user">
            <strong>You:</strong>{' '}
            {message.content}
          </p>
        ) : (
          <blockquote key="assistant">
            {message.content}
          </blockquote>
        ),
      )}
    </div>
  )
}

export default function App() {
  const [activeId, setActiveId] = useState(
    CHATS[0].id,
  )
  const [visitedIds, setVisitedIds] = useState([
    CHATS[0].id,
  ])

  const open = (chat: Chat) => {
    setActiveId(chat.id)
    setVisitedIds((ids) =>
      ids.includes(chat.id)
        ? ids
        : [...ids, chat.id],
    )
  }

  return (
    <>
      <div role="group">
        {CHATS.map((chat) => (
          <button
            key={chat.id}
            type="button"
            // Hovering a chat you haven't opened yet starts its reply
            // streaming in the background, so opening it skips the fallback.
            onMouseEnter={() =>
              preloadObservablePromise(
                conversation$(chat),
                {ttl: 30_000},
              )
            }
            onClick={() => open(chat)}
            aria-current={
              chat.id === activeId || undefined
            }
            style={{
              fontWeight:
                chat.id === activeId ? 700 : 400,
            }}
          >
            {chat.title}
          </button>
        ))}
      </div>

      <Suspense
        fallback={
          <p aria-busy="true">
            Waiting for the first token…
          </p>
        }
      >
        {CHATS.filter((chat) =>
          visitedIds.includes(chat.id),
        ).map((chat) => (
          // Visited chats stay mounted but hidden: they keep their state and
          // reveal instantly, including every token that streamed while you
          // were looking at another chat.
          <Activity
            key={chat.id}
            mode={
              chat.id === activeId
                ? 'visible'
                : 'hidden'
            }
          >
            <ChatView chat={chat} />
          </Activity>
        ))}
      </Suspense>
    </>
  )
}

Open Sandbox

The mock vs. your code

llm.ts is the only mock in this demo. It stands in for a real streaming LLM API and emits token deltas as an observable. In a real app it would wrap a fetch ReadableStream or an SSE connection. Everything else (chat.ts and App.tsx) is what your own code would look like.

The userland recipe is small:

scan folds tokens into the reply. The conversation stream emits the whole message list on every token. Components just render the latest value.
shareReplay({bufferSize: 1, refCount: false}) makes the stream independent of who’s watching. The reply keeps streaming while its chat is hidden or unmounted. Any subscriber, new or returning, immediately gets the latest state. The source completes when the reply ends, so nothing leaks.

Why the React side “just works”

useObservablePromise + use() suspend until the first emission, then update in place. Streaming tokens never re-trigger the fallback.
<Activity mode="hidden"> keeps visited chats mounted with their state intact. Hiding a chat tears down its live subscription (like an effect). The shared conversation stream keeps running. On reveal, the hook synchronously reads the current snapshot. No flash, no refetch, and all the tokens that arrived meanwhile are there.
preloadObservablePromise warms the same cache the hook reads from, outside of render. Hover-to-preload is one line on the button.

See Activity and preload for a side-by-side comparison of prefetch strategies. See Suspense data fetching for the promise semantics on their own.