v1.7.1 — Converse Stream API + Streaming UX Improvements

This release introduces the Bedrock Converse Stream API for multi-route synthesis, applies a typing effect to AgentCore Gateway responses, and adds time range filters to monitoring queries to resolve pg pool exhaustion issues.

Key Changes

Converse Stream API · Typing Effect (50char/15ms) · Chat Bubble Width Stabilization · 3 Monitoring Query Time Filters

Real-time Synthesis with Converse Stream API

Multi-route response synthesis has transitioned from fake chunking to real token streaming.

Before / After

Item	Before (v1.7.0)	After (v1.7.1)
Synthesis API	InvokeModelCommand	ConverseStreamCommand
Synthesis Function	synthesizeResponses()	synthesizeResponsesStreaming()
Streaming Method	Full response generated → simulateStreaming()	Token sent immediately via chunk event on generation
First Token Latency	Wait for full response (5-15s)	~200ms (first token immediate)
User Experience	Long wait then fast output	Starts immediately, natural flow

Code Changes

Before — synthesizeResponses() + simulateStreaming():

// src/app/api/ai/route.ts (v1.7.0)
const synthesized = await synthesizeResponses(
  lastMsg, successful, modelKey, clientLang
);
// Split completed text into 50-char chunks for fake streaming
await simulateStreaming(synthesized, send);

After — synthesizeResponsesStreaming() + ConverseStreamCommand:

// src/app/api/ai/route.ts (v1.7.1)
const synthesized = await synthesizeResponsesStreaming(
  lastMsg, successful, send, modelKey, clientLang
);
// send() callback transmits chunk immediately on each contentBlockDelta

synthesizeResponsesStreaming() core logic:

const response = await bedrockClient.send(new ConverseStreamCommand({
  modelId,
  system: [{ text: systemPrompt }],
  messages: [{ role: 'user', content: [{ text: synthesisPrompt }] }],
  inferenceConfig: { maxTokens: 4096 },
}));

for await (const event of response.stream) {
  if (event.contentBlockDelta?.delta?.text) {
    const text = event.contentBlockDelta.delta.text;
    fullContent += text;
    send('chunk', { delta: text });  // Immediate token delivery
  }
}

Typing Effect (simulateStreaming)

Since AgentCore Gateway returns completed text all at once, a typing effect is simulated.

// src/app/api/ai/route.ts
const CHUNK_SIZE = 50;     // 50 chars per chunk
const CHUNK_DELAY_MS = 15; // 15ms delay between chunks

async function simulateStreaming(
  text: string,
  send: (event: string, data: any) => void,
): Promise<void> {
  for (let i = 0; i < text.length; i += CHUNK_SIZE) {
    const chunk = text.slice(i, i + CHUNK_SIZE);
    send('chunk', { delta: chunk });
    if (i + CHUNK_SIZE < text.length) {
      await new Promise(r => setTimeout(r, CHUNK_DELAY_MS));
    }
  }
}

Applied to (3 locations):

1. Single AgentCore Gateway response — Typing effect on completed text
2. Multi-route synthesis (v1.7.0) — Applied to synthesizeResponses() result (replaced by Converse Stream in v1.7.1)
3. Multi-route single success — When only 1 Gateway succeeds

Why is simulation needed?

Direct Bedrock calls can use InvokeModelWithResponseStreamCommand for real streaming, but AgentCore Gateway calls tools multiple times internally and returns only the final text. The 50-char/15ms typing effect is applied to provide a consistent streaming experience to users.

Chat Bubble Width Stabilization (PR #8)

Fixed an issue where chat bubble width would suddenly change during SSE streaming.

Item	Before	After
CSS	max-w-5xl	w-full max-w-5xl
Behavior	Width jumps as content grows	Full width fixed from first chunk
File	src/app/ai/page.tsx	Same

Before — Width jump:

<div class="max-w-5xl rounded-lg px-4 py-3 ...">
  <!-- Narrow when content is short, suddenly widens when longer -->
</div>

After — Stable width:

<div class="w-full max-w-5xl rounded-lg px-4 py-3 ...">
  <!-- w-full ensures max width from start, max-w-5xl caps the upper limit -->
</div>

CSS Pattern

For containers where content is progressively added (like SSE streaming), use the w-full max-w-{size} combination. Using only max-w-{size} causes width to vary with content amount.

Monitoring Query Time Range Filters

Fixed an issue where monitoring queries without time range limits were fetching full historical data via the Steampipe API, occupying all 5 pg pool connections.

Query	File	Before	After
ec2CpuHourly	queries/metrics.ts	No time limit	WHERE timestamp >= NOW() - INTERVAL '24 hours'
ebsIopsHourly	queries/metrics.ts	WHERE r.timestamp IS NOT NULL	AND r.timestamp >= NOW() - INTERVAL '24 hours'
rdsCpuDaily	queries/metrics.ts	No time limit	WHERE timestamp >= NOW() - INTERVAL '30 days'

pg Pool Exhaustion Symptoms

In environments with 46+ EC2 instances + many EBS volumes, queries without time limits occupy connections for several minutes. When all max: 5 pg pool connections are in use, the dashboard and all pages wait until statement_timeout: 120s and then fail.

Summary of 3 Streaming Modes

In v1.7.1, AWSops AI responses use 3 streaming modes depending on the situation.

Mode	When Used	Implementation	First Token Latency
Real Streaming	Direct Bedrock calls	InvokeModelWithResponseStreamCommand	~200ms
Converse Stream	Multi-route synthesis	ConverseStreamCommand + contentBlockDelta	~200ms
Simulated Streaming	AgentCore Gateway responses	simulateStreaming() (50char/15ms)	Immediate after response completes

Unified Client Experience

All 3 modes use the same SSE chunk event, so the client (src/app/ai/page.tsx) does not distinguish between modes — it simply accumulates deltas in the streamingContent state and renders in real time with ReactMarkdown.