Client tools

A client tool is a function your web page or mobile app runs when the agent asks. The agent picks the tool, calls your code over the session, and speaks whatever you return. Nothing touches your server. Use them when the tool needs the screen, like showing the product the user just asked about. They only exist on web sessions.

How a call reaches your code

Your app and the agent share a LiveKit room. The agent calls your tool as an RPC on that room, with the tool name as the method: Three things to get right, and the rest is your code:

Write one handler per tool. The SDK registers it under the tool’s name.
Parse data.payload. It is a JSON string. arguments holds what the model filled in, shaped by your parameters schema.
Return a JSON string. The agent parses it and hands it to the model as the tool result.

Declare the tool

A client tool is any tool with no execution field. Put it on the assistant next to the prompt, so the model reads the two together:

{
  "name": "Shop assistant",
  "config": {
    "agent": {
      "instructions": "You help shoppers on our site. When they ask to see something, show it on screen and describe it in one line.",
      "tools": [
        {
          "name": "show_product",
          "description": "Show a product on screen. Call this when the user asks to see a product or picks one you offered.",
          "parameters": {
            "type": "object",
            "properties": {
              "productId": { "type": "string", "description": "The product id from the catalog" }
            },
            "required": ["productId"]
          },
          "timeout": 5
        }
      ]
    },
    "stt": { "default": { "provider": "soniox", "model": "stt-rt-v4", "language": "ur" } },
    "tts": { "default": { "provider": "upliftai", "voiceId": "v_meklc281", "outputFormat": "MP3_22050_32" } },
    "llm": { "default": { "provider": "google", "model": "gemini-2.5-flash" } }
  }
}

That is a create assistant body. timeout is how many seconds the agent waits for your handler. The default is 10. Your app carries the definition again, with the handler attached. When the SDK connects with a tools prop, that list replaces the assistant’s for the session, so keep the two the same. An empty or missing prop leaves the assistant’s list alone.

Handle it with the React SDK

@upliftai/assistants-react is a thin wrapper around the LiveKit room. It registers a handler per tool, sends the definitions to the agent on connect, and gives you hooks to change them later.

npm install @upliftai/assistants-react livekit-client @livekit/components-react

import { UpliftAIRoom } from '@upliftai/assistants-react'

const showProduct = {
  name: 'show_product',
  description: 'Show a product on screen. Call this when the user asks to see a product or picks one you offered.',
  parameters: {
    type: 'object',
    properties: { productId: { type: 'string', description: 'The product id from the catalog' } },
    required: ['productId'],
  },
  timeout: 5,
  handler: async (data) => {
    const { arguments: args } = JSON.parse(data.payload)
    setProduct(args.productId)
    return JSON.stringify({
      result: { shown: args.productId },
      presentationInstructions: 'It is on screen now. Describe it in one line.',
    })
  },
}

function App({ token, wsUrl }) {
  return (
    <UpliftAIRoom token={token} serverUrl={wsUrl} audio={true} tools={[showProduct]}>
      <YourUI />
    </UpliftAIRoom>
  )
}

token and wsUrl come from your server minting a session token. A public assistant can mint one straight from the browser. The full component and hook reference is on the React SDK page. Another client? Underneath, this is LiveKit RPC. The LiveKit SDK on every platform can register a method, so a mobile app can do the same today. We haven’t wrapped it for iOS, Android, Flutter or React Native yet. Tell us which one you need at founders@upliftai.org.

Change tools mid-session

The useUpliftAIRoom hook gives you addTool, removeTool, upsertTools and updateInstruction. All four take effect on the live session, once the agent has joined:

const { upsertTools, updateInstruction } = useUpliftAIRoom()

// the user reached checkout: swap the whole tool set
await upsertTools([fillAddress, applyCoupon, placeOrder])
await updateInstruction('The user is at checkout. Confirm the address, offer to apply a coupon, then place the order.')

upsertTools replaces the whole set with what you send. That is the easy way to move through a flow: one list for browsing, another for checkout. addTool and removeTool change one at a time. All four ride on two RPCs to the agent, update_tools and update_instructions. The agent only learns the definitions. The handlers stay in your app.

What to return

The SDK convention is two fields, and the model reads both:

{ "result": { "shown": "blue-42" }, "presentationInstructions": "It is on screen now. Describe it in one line." }

result is the data. presentationInstructions is how to say it. The model treats that as a hint, not a script. On failure return { "error": "...", "presentationInstructions": "..." } so the model can tell the user what went wrong. If the handler throws, or takes longer than timeout, the model gets an error result and keeps talking.

Before you ship

Keep client tools on web assistants. On a phone call there is no client to call. If you need a tool mid-call on the phone, use a webhook tool.
Only one app gets the RPC. The agent sends it to the first other participant in the room.
Debug from the session detail. It lists every tool call afterwards, arguments and result included.

Getting started

Concepts

Assistants

Tools

Personalization

Calling

Campaigns

Webhooks

Testing

API Reference

How a call reaches your code

Declare the tool

Handle it with the React SDK

Change tools mid-session

What to return

Before you ship

​How a call reaches your code

​Declare the tool

​Handle it with the React SDK

​Change tools mid-session

​What to return

​Before you ship

How a call reaches your code

Declare the tool

Handle it with the React SDK

Change tools mid-session

What to return

Before you ship