> ## Documentation Index
> Fetch the complete documentation index at: https://docs.upliftai.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Core Concepts

> Understanding the building blocks of Realtime Assistants

<Note>
  The Realtime Assistants API is currently in **beta**. We're actively improving performance and adding features based on user feedback.
</Note>

## How Realtime Assistants Work

Realtime Assistants combine several advanced technologies to create seamless voice interactions:

1. **WebRTC Connection** - Low-latency audio/video streaming
2. **Speech Processing** - Real-time STT and TTS
3. **AI Agent** - Intelligent conversation management
4. **Tool Execution** - Dynamic function calling via RPC

## Key Components

### Sessions

A session represents a single conversation between a user and an assistant. Each session:

* Has a unique room identifier
* Supports multiple participants (user + agent)
* Maintains conversation context
* Can be configured with custom settings
* Has a configurable TTL (time-to-live)

```json theme={null}
{
  "token": "eyJ0eXAiOiJKV1...",
  "wsUrl": "wss://upliftai-livekit-url...",
  "roomName": "assistant-room-abc123"
}
```

### Agents

The AI agent is the brain of your assistant. It:

* Processes user speech in real-time
* Manages conversation flow
* Handles interruptions gracefully
* Executes tools when needed
* Maintains context throughout the session

### Configuration

Each assistant can be configured with:

<Tabs>
  <Tab title="Agent Settings">
    ```json theme={null}
    {
      "agent": {
        "instructions": "You are a helpful assistant",
        "initialGreeting": true,
        "greetingInstructions": "Say hello and ask how you can help",
        "tools": []
      }
    }
    ```
  </Tab>

  <Tab title="STT Settings">
    ```json theme={null}
    {
      "stt": {
        "default": {
          "provider": "groq",
          "model": "whisper-large-v3",
          "language": "en"
        }
      }
    }
    ```
  </Tab>

  <Tab title="TTS Settings">
    ```json theme={null}
    {
      "tts": {
        "default": {
          "provider": "upliftai",
          "voiceId": "john",
          "outputFormat": "MP3_22050_32"
        }
      }
    }
    ```
  </Tab>

  <Tab title="LLM Settings">
    ```json theme={null}
    {
      "llm": {
        "default": {
          "provider": "groq",
          "model": "openai/gpt-oss-120b"
        }
      }
    }
    ```
  </Tab>
</Tabs>

## Tools and Functions

Tools extend your assistant's capabilities. These tools can be executed through RPC communication from the agent to your user device, or external API calls directly from the agent (coming soon.). Your frontend can then execute:

**Custom Tools**:

* Access external APIs
* Perform calculations
* Query databases
* Execute custom business logic

The agent will also support **MCP (Model Context Protocol) Tools**:

* Connect to MCP servers
* Bridge to enterprise systems and common places like Shopify.

### Tool Definition

```javascript theme={null}
{
  name: "get_weather",
  description: "Get current weather for a location",
  parameters: {
    type: "object",
    properties: {
      location: {
        type: "string",
        description: "City and state"
      }
    },
    required: ["location"]
  },
  timeout: 10
}
```

### Tool Execution Flow

```mermaid theme={null}
sequenceDiagram
    participant User
    participant UserDevice
    participant Agent
    participant ExternalAPI
    
    User->>UserDevice: Speak "What's the weather?"
    UserDevice->>Agent: Audio
    Agent->>UserDevice: Tool Request (get_weather)
    alt Custom Tool
        UserDevice->>ExternalAPI: Fetch Weather Data
        ExternalAPI->>UserDevice: Weather Response
    end
    UserDevice->>Agent: Tool Response
    Agent->>UserDevice: Audio of "It's 72°F and sunny"
    UserDevice->>User: "It's 72°F and sunny"

```

## Dynamic Configuration

Assistants can be updated in real-time without disrupting active sessions. The follow show using React example with [`@upliftai/assistants-react`](https://www.npmjs.com/package/@upliftai/assistants-react) package:

### Update Instructions

Change the assistant's behavior on the fly

```javascript theme={null}
await updateInstruction("You are now a pirate. Speak like one.");
```

### Manage Tools

Add or remove tools dynamically:

```javascript theme={null}
// Add a new tool
await addTool(calculatorTool);

// Remove a tool
await removeTool("calculator");

// Update existing tools
await upsertTools([tool1, tool2]);
```

## Connection Lifecycle

<Steps>
  <Step title="Session Creation">
    Client requests a session token from your backend
  </Step>

  <Step title="WebRTC Negotiation">
    Client connects to LiveKit server using the token
  </Step>

  <Step title="Agent Join">
    AI agent joins the room and initializes
  </Step>

  <Step title="Conversation">
    Real-time audio streaming and processing
  </Step>

  <Step title="Tool Execution">
    Agent requests tool execution via RPC when needed
  </Step>

  <Step title="Session End">
    Client disconnects or session expires
  </Step>
</Steps>

## Public vs Private Assistants

For agents that are created and `public` flag is enabled, then a session can be created for them without an API key. This is recommended for public facing agents like on websites etc if user authentication isn't required.

You can use the upliftai widget for public agents. See [npm](https://www.npmjs.com/package/@upliftai/assistant-widget) for more options.

```
<script src="https://cdn.jsdelivr.net/npm/@upliftai/assistant-widget@0.0.1"></script>
<upliftai-assistant 
  assistant-id="YOUR_ASSISTANT_ID">
</upliftai-assistant>
```

Or if you want more control on the UI, you can create the session in the frontend without API key like:

```javascript theme={null}
// Public assistant session
const response = await fetch(
  `https://api.upliftai.org/v1/realtime-assistants/${id}/createPublicSession`,
  {
    method: 'POST',
    body: JSON.stringify({ participantName: 'User' })
  }
);
```

## Performance Optimization

### Latency Reduction

* **Geographic Distribution**: Agents *AND* **MODELS** deployed close to Pakistan
* **Provider Selection**: Choose fastest providers for your use case
* **Connection Pooling**: Pre-warmed connections to providers

## Security Considerations

<Warning>
  Always validate and sanitize tool inputs and outputs. Never expose sensitive API keys or credentials in client-side code. Tokens given by UpliftAI using createSession are safe to use.
</Warning>

### Best Practices

1. **Authentication**: Always use backend session creation for production
2. **Tool Validation**: Implement strict input validation in tool handlers. **ALWAYS** make sure inputs are in the format you expect.

## Next Steps

<CardGroup cols={2}>
  <Card title="Quick Start Tutorial" icon="rocket" href="/assistants/tutorial">
    Build your first assistant in minutes
  </Card>

  <Card title="API Reference" icon="book-open" href="/assistants/api-reference/create-assistant">
    Explore all API endpoints
  </Card>

  <Card title="Tools Guide" icon="wrench" href="/assistants/tools">
    Learn how to build custom tools
  </Card>

  <Card title="SDK Documentation" icon="code" href="/assistants/sdk/react">
    Integrate with React, JS, and mobile
  </Card>
</CardGroup>