This cookbook serves as a practical guide to help AI Engineers maximize the effectiveness of OpenAI's Realtime API, specifically when dealing with data-intensive function calls. We'll focus on scenarios common in speech-to-speech agents, where vast amounts of data must be handled smoothly and efficiently.
This post won't cover the basics of setting up a Realtime API solution. Instead, you'll gain clear insights and actionable strategies to enhance the performance and reliability of your real-time conversational agents. It addresses specific challenges unique to handling large amounts of data in real-time conversational contexts.
Before we dive in, let’s quickly recap the API for those who are new. The OpenAI Realtime API is a recent offering that supports low-latency, multimodal interactions—such as speech-to-speech conversations and live transcription. Picture scenarios like real-time voice-based customer support or live movie transcriptions.
Agents need access to tools and relevant data to perform their tasks. For instance, a financial analyst agent might pull real-time market data. In many cases, services already exist in your environment that expose this information through APIs.
Historically, APIs weren’t designed with agents in mind and often return large volumes of data, depending on the service. As engineers, we frequently wrap these APIs with function calls to accelerate agent development—which makes perfect sense. Why reinvent what already exists?
If not carefully optimized, these data-intensive function calls can quickly overwhelm the Realtime API—leading to slow responses or even failures to process user requests.
Our example centers on an NBA Scouting Agent that calls multiple functions to deliver in-depth analysis of upcoming draft prospects. To demonstrate practical guidelines for Realtime API interactions, we use large, realistic payloads inspired by NBA draft prospects. Below, you’ll find a monolithic searchDraftProspects function defined in the Realtime session to set the stage.
// "Hey, pull up point guards projected in the top 10 in the 2025 draft"{"type": "session.update","session": {"tools": [ {"type": "function","name": "searchDraftProspects","description": "Search draft prospects for a given year e.g., Point Guard","parameters": {"type": "object","properties": {"sign": {"type": "string","description": "The player position","enum": ["Point Guard","Shooting Guard","Small Forward","Power Forward","Center","Any" ] },year: { type: "number", description: "Draft year e.g., 2025" },mockDraftRanking: { type: "number", description: "Predicted Draft Ranking" }, },"required": ["position", "year"] } } ],"tool_choice": "auto", }}
The searchDraftProspects function call returns a hefty payload. The example’s structure and size are drawn from real-world scenarios we’ve encountered.
It almost goes without saying—when building function calls, your top priority is to design clear, well-defined functions. This makes it easy to trim response sizes and avoid overwhelming the model. Each function call should be straightforward to explain, sharply scoped, and return only the information needed for its purpose. Overlapping responsibilities between functions inevitably invites confusion.
For example, we can limit the searchDraftProspects function call to return only general details—such as player stats—for each prospect, dramatically reducing the response size. If more information is needed, the new getProspectDetails function call provides expanded details. There’s no universal solution; the right approach depends on your use case and data model.
{"tools": [ {"type": "function","name": "searchDraftProspects","description": "Search NBA draft prospects by position, draft year, and projected ranking, returning only general statistics to optimize response size.","parameters": {"type": "object","properties": {"position": {"type": "string","description": "The player's basketball position.","enum": ["Point Guard","Shooting Guard","Small Forward","Power Forward","Center","Any" ] },"year": {"type": "number","description": "Draft year, e.g., 2025" },"maxMockDraftRanking": {"type": "number","description": "Maximum predicted draft ranking (e.g., top 10)" } },"required": ["position", "year"] } }, {"type": "function","name": "getProspectDetails","description": "Fetch detailed information for a specific NBA prospect, including comprehensive stats, agent details, and scouting reports.","parameters": {"type": "object","properties": {"playerName": {"type": "string","description": "Full name of the prospect (e.g., Jalen Storm)" },"year": {"type": "number","description": "Draft year, e.g., 2025" },"includeAgentInfo": {"type": "boolean","description": "Include agent information" },"includeStats": {"type": "boolean","description": "Include detailed player statistics" },"includeScoutingReport": {"type": "boolean","description": "Include scouting report details" } },"required": ["playerName", "year"] } } ],"tool_choice": "auto"}
Realtime conversations allow for generous 30-minute sessions—but the rolling context window only supports ~16,000 tokens (depending on the model snapshot, context window limitations are improving). As a result, you may notice performance gradually decline during extended exchanges. As conversations progress and more function calls are made, the conversation state can expand quickly with both important information and unnecessary noise—so it’s important to focus on keeping the most relevant details. This approach helps maintain strong performance and reduces cost.
i) Periodically summarize the conversation state
Periodically summarizing the conversation as it unfolds is an excellent way to reduce context size—cutting both cost and latency.
See @Minhajul's' epic guide on implementing automatic summarization in Realtime conversations (link).
ii) Periodically remind the the model of its role and responsibilities
Data-heavy payloads can quickly fill the context window. If you notice the model losing track of instructions or available tools, periodically remind it of its system prompt and tools by calling session.update—this keeps it focused on its role and responsibilities.
i) Use filtering in your function calls to trim data-heavy responses down to only the essential fields needed to answer the question
Generally, fewer tokens returned by function calls lead to better quality responses. Common pitfalls occur when function calls return excessively large payloads spanning thousands of tokens. Focus on applying filters in each function call, either at the data-level or function-level, to minimize response sizes.
ii) Flatten hierarchical payloads—without losing key information
Hierarchical payloads from API calls can sometimes include repeated level titles—like "ProspectInfo" or "Stats"—which may add extra noise and make things harder for the model to process. As you explore ways to make your data more efficient, you might try flattening these structures by trimming away some of the unnecessary labels. This can help improve performance, but consider what information is important to keep for your particular use case.
The way you structure your data has a direct impact on how well the model processes and summarizes API responses. In our experience, clear, key-based formats like JSON or YAML help the model interpret data more accurately than tabular formats such as Markdown. Large tables, especially, tend to overwhelm the model—resulting in less fluent and less accurate outputs. Still, it’s worth experimenting with different formats to find what works best for your use case.
Underlying models often struggle to transition smoothly from data-heavy responses to accurate answers. To improve fluency and accuracy when working with complex data, provide a function call hint immediately after the function call. These hints guide the model on the specific task—teaching it how to interpret key fields and domain-specific values.
The following example illustrates an effective hint prompt.
// Function call hintlet prospectSearchPrompt =`Parse NBA prospect data and provide a concise, engaging response.General Guidelines- Act as an NBA scouting expert.- Highlight key strengths and notable attributes.- Use conversational language.- Mention identical attributes once.- Ignore IDs and URLs.Player Details- State height conversationally ("six-foot-eight").- Round weights to nearest 5 lbs.Stats & Draft Info- Round stats to nearest whole number.- Use general terms for draft ranking ("top-five pick").Experience- Refer to players as freshman, sophomore, etc., or mention professional experience.- Location & TeamMention hometown city and state/country.- Describe teams conversationally.Skip (unless asked explicitly)- Exact birth dates- IDs- Agent/contact details- URLsExamples- "Jalen Storm, a dynamic six-foot-four point guard from Springfield, Illinois, averages 21 points per game."- "Known for his clutch shooting, he's projected as a top-five pick."Important: Respond based strictly on provided data, without inventing details.`;
In practice, we first append the function call result to the conversation. Then, we emit a response from the Realtime API with the hint prompt. Voilà—the model gracefully handles all the information.
// Add new conversation item for the modelconstconversationItem= { type: 'conversation.item.create', previous_item_id: output.id, item: { call_id: output.call_id, type: 'function_call_output', output: `Draft Prospect Search Results: ${result}` }};dataChannel.send(JSON.stringify(conversationItem));// Emit a response from the model including the hint promptconstevent= { type: 'response.create', conversation: "none", response: { instructions: prospectSearchPrompt # functioncallhint }};dataChannel.send(JSON.stringify(event));
Building effective agents with the Realtime API is an ongoing process of exploration and adaptation.
Summary of Key Recommendations
Filter data: Only include fields and details that are directly relevant to the user’s request or the model’s next step. Trim the rest.
Flatten and simplify structures: Reduce deeply nested or redundant data. Present information in a way that’s easy for both models and humans to scan.
Prefer clear, structured formats: Use JSON (or YAML) with consistent field names and minimal noise. Avoid large tables or markdown for data-heavy responses.
Guide the model with hint prompts: After returning lots of data, follow up with a targeted prompt that explains exactly what the model should extract or summarize.
Remember—experimentation is essential. Realtime models keep improving, and we’ll continue sharing tips to help you get the most out of the Realtime API.