For Fire TV · Amazon Developer Hackathon 2026
Ask your TV for the moment you remember.
Semantic Playback is an AI layer for Fire TV. Say show me his goal
, who is she?
or catch me up
. The TV finds the exact moment, plays it and remembers you asked.
Real recordings of the Fire OS app on the Android TV emulator. Every step on this page comes from a recorded trace.
- 1.0 s
- from “show me his goal” to the build-up, with no AI model call
- 26
- MCP tools for the player, moments, people, memory, clips and live
- 10
- kinds of requests, recorded on a native Fire OS app
- 3
- open-source contributions so Fire TV apps respect captions
- Fire OS · Media3
- Amazon Rekognition
- Amazon Polly
- Amazon IVS
- Amazon DynamoDB
- Bedrock AgentCore Memory
- MCP 2025-11-25
The problem
Players understand timestamps. People remember moments.
You never think “go to 2:12”. You think “his goal”, “the line where she says two minutes left”, “what did I miss?”. Today the remote can only skip ten seconds, so you hold rewind, overshoot and try again.
With a remote
Rewind 10 s. Too far. Forward 10 s. Missed it. Forty seconds later you find the build-up, and the moment has passed.
With Semantic Playback
The TV knows “his” is the player you just asked about, finds his goal in the match index and starts on the build-up. It took 1.0 seconds in our recording.
Every moment of the match, indexed
Soccer Aid for UNICEF 2023 highlights, 13:29. Built at ingest with Amazon Rekognition shots, faces and on-screen text, plus captions. Hover a mark, or search it the way you'd say it.
This search box runs a simple keyword match in your browser over the real index. The product uses hybrid search (BM25, embeddings and people) with spoiler limits.
Try it · no Fire TV needed
Pick something to say. Watch the TV do it.
Each request plays the real recording from the Fire OS app. On the right, the steps replay at their recorded timing: what our server decided, which AWS service it called and what came back.
Long AI model waits are shown with their real duration. Bedrock was blocked on our new AWS account for the whole hackathon, so the agent ran the same code on OpenRouter (Nemotron 120B), and every answer says which engine ran.
How it works
Understand, ground, act, remember, continue.
Follow one request through the loop. The AI model is used where meaning is ambiguous. Everything after the moment is found is exact and deterministic.
-
1
Understand
The Context Envelope tells the agent what is on screen, what you just asked about and how far you've watched.
“his” → Usain Bolt
spoilers up to 2:27 -
2
Ground
Hybrid search over the index built with Amazon Rekognition picks the moment, then starts it on the build-up.
moments.search
→ Usain Bolt scores first goal @ 2:12 -
3
Act
A precise command reaches the Fire OS app over our player bridge, and ExoPlayer executes it natively.
player.seek 2:12 → play
243 ms -
4
Remember
The moment is saved to your Viewing Memory in Amazon DynamoDB and mirrored to Bedrock AgentCore Memory.
semantic_seek · Usain Bolt -
5
Continue
Follow-ups are relative to the grounded moment, so “five seconds earlier in slow motion” is exact maths.
seek −5 s · speed 0.5×
772 ms, no model
Rules plus hybrid search, no AI model call
Direct requests go straight to tools: who's this, his goal, slow motion, clip it, spoilers off, not that one, catch me up.
- “No, not that one, the one before half-time”0.6 s
- “Go back five seconds in slow motion”0.8 s
- “Show me his goal”1.0 s
An LLM agent chooses among 26 MCP tools
Open questions go to the agent with the Context Envelope and the tool list. A late answer is never executed, and the spoiler limit is enforced in code.
- “Where have I seen her before?”entities.appearances
- “When do they let the dough rest?”moments.search
- “Go back to the part about schools teaching software”transcript.window
What it does
It behaves like someone who watched with you.
Hover a card to preview the recording. Click to open it in the demo with its trace.
Architecture
Select a component to see what it does.
The self-hosted MCP server is the exact surface an Alexa+ add-on connects to. Alexa+ developer tools are partner-only during the hackathon, so our own agent and the Fire OS app call the same server today.
At ingest, every title becomes an index
- Video
- Amazon S3
- Rekognition Video shots
- Rekognition faces, text, labels
- Captions
- Descriptions + embeddings
- Hybrid index
Accessibility · open source
We went to fix captions in our app and found them broken for every React Native Fire TV app.
People who rely on captions, or on Audio Description (a narrator describing the picture for blind and low-vision viewers), turn it on once in Fire TV settings. Apps are supposed to follow it. Our native app did. Amazon's React Native starter for TV apps didn't, and a bug underneath meant subtitle files never loaded on Android at all.
-
1
A new open-source module
Gives any React Native TV app the viewer's Captions and Audio Description settings, and updates live when they change.
expo-tv-accessibility-settings -
2
A pull request to Amazon's TV sample
Captions appear in the chosen language when turned on. The described version plays while Audio Description is on, and switches back mid-video without losing your place.
react-native-multi-tv-app-sample #144 -
3
The root-cause fix in react-native-video
Since 6.16, side-loaded subtitle files never loaded on Android. The open issue had no cause. The fix loads each track the way Media3 expects.
react-native-video #5178
Built honestly
What runs where, including what didn't.
Our new AWS account stayed in verification for the whole hackathon. Instead of hiding it, every feature degrades visibly and tells you which engine answered.
FAQ
Questions people ask first.
Is the demo real?
Yes. Every clip is a screen recording of our Fire OS app (Kotlin, Media3 ExoPlayer) on the Android TV emulator, API 30, which is the Android base of Fire OS 8. The steps next to each clip come from the trace our server saved for that request. Requests were sent by a script through the same endpoint typed or spoken input uses.
Is this Alexa+?
No. Alexa+ is the voice front end we designed for. We built the part it connects to: a self-hosted MCP server (spec 2025-11-25, Streamable HTTP, 26 tools). Alexa+ add-on tools are partner-only, so our own agent calls the same server today.
Does it spoil things?
No. The spoiler boundary moves only with real playback. Search, catch-up and open questions never use moments after it until you say “spoilers off”, and that rule lives in code rather than in a prompt.
Which videos does it work with?
Any video that has been indexed. The demo uses Creative Commons titles: Soccer Aid for UNICEF 2023 highlights, Tears of Steel, a cooking video and a TEDx talk, plus a live Amazon IVS channel. Each title's index is built once, at ingest.
What does it use AWS for?
Amazon Rekognition for shots, faces, on-screen text and film characters. Amazon Polly for every spoken answer. Amazon DynamoDB and Bedrock AgentCore Memory for Viewing Memory. Amazon IVS for the live channel and its goal markers. Amazon S3 for media. Bedrock is integrated as the agent's model and switches on when the account is enabled.
Can I run it?
Yes, without an AWS account. The repository ships compact copies of the six titles, and features that need AWS say so when it's missing. The source is shared with the hackathon judges.