Loading
Paste a YouTube URL. Screensay reads every frame and the audio and returns one JSON document plus a screenplay: what was on screen, what was said, how it was said and what each “this” pointed at, timestamped to the tenth of a second.
Free. No API key. The first 3 minutes of any public YouTube video, 3 runs per IP per hour.
Set up for agentswatch_video and ask_video as MCP tools for Claude Code, Claude Desktop and Cursor.
Measured on one 43:15 video, 394 of 2595 frames read at 1280 px. What is in the document
Captions carry the words and drop the screen, the pointing, the stress and the timing. Three moments from the sponsor read in A proper guide to Fable 5, 03:05.2–04:04.0: the caption on the left, the document's rows on the right.
seensaiddeliverypointed at
transcript, 03:13…not only do they have the best off platform that is the easiest to integrate…
The captions heard “off”. The heading on screen reads “More than authentication,”, so the document writes “auth”, keeps the caption's reading beside it, and lists the fix under utterances[].fixes with the text row as evidence.
transcript, 03:29What's way more exciting to me is the subscription platform that's now built in.
The referring phrase is a row in points[]: the words, the on-screen heading it resolved to, and how (screen match). The beat records that he was looking at the screen, not the camera.
transcript, 03:32I can confidently say Clerk is the easiest and has the best abstraction.
Rate, loudness, peak and pause fraction are measured from the audio for every utterance and compared with the speaker's own baseline (3.82 words/s). “slows down” is a threshold, not an opinion; “confident” is the model's reading, with a confidence of 0.75.
A text-only model answered 31 verified questions about one 8-minute podcast interview, twice: once with the YouTube transcript, once with the Screensay document.
One video so far. 6 more, one per genre (vlog, unboxing, conference talk, cooking, gaming comedy, screen-share tutorial), have verified questions written and are queued to run. The reference answers were written by a model that watched the video and independently verified by a second model before any question was kept.
| Category | n | Transcript only | Screensay document | Delta |
|---|---|---|---|---|
| Literal on-screen text button labels, a CSS line read from a code-editor inset | 2 | 0.0% | 100.0% | +100.0 pts |
| Cross-modal what was on screen while something was said | 3 | 0.0% | 100.0% | +100.0 pts |
| Production and editing a picture-in-picture inset, a search-result overlay | 2 | 0.0% | 100.0% | +100.0 pts |
| Humour and subtlety the transcript wins: one whispered callback was split across two caption cues and the document quoted half of it | 3 | 83.3% | 66.7% | -16.6 pts |
Accuracy is the mean judge score out of 2 (wrong, partial, right). Screensay document means the standard-detail screenplay plus the JSON document, 47,269 tokens; the transcript was 2,319.
The unit of the document: what was on screen, what was said, what the presenter did and where they looked, joined by id. The screenplay is one paragraph per beat, and each beat says whether the screen was the subject, an illustration, a contradiction or wallpaper.
Every heading, button, menu row and price is stored once with the seconds it was visible, never paraphrased. Menus, cards and forms keep their rows.
Words per second, loudness, peak and pause fraction are measured from the audio for each utterance against the speaker's own baseline. A model then adds voice tags, the stressed words, and one hedged reading such as “exasperated” or “mock complaint” with a confidence that never reaches 0.9.
A points table records each referring phrase, what it resolved to on screen and how (cursor, hand, gaze, screen match), and when the thing named was not on screen at all.
Shot changes are detected from frame hashes, not guessed. Screen events carry an actor, so a page's own demo animation is never reported as something the presenter clicked. Caption errors are corrected from the screen and the original reading is kept.
What a viewer would remember, where a setup pays off, when a sponsor was announced, named, shown and disclosed, and every entity with its mentions. Names and prices inside demo UIs are marked as placeholders.
JSON for code. A screenplay of about 11 to 12 thousand tokens per 15 minutes for prompts. A summary card of under 600 tokens for routing. All three are rendered from the same rows.
POST a question to a finished job and get an answer with timestamped evidence. Monitor a channel and receive each new upload at your endpoint with an HMAC-SHA256 signature. watch_video and ask_video as tools for agents.
Four stages, each cached under the video id. What the document holds and how to call it.
Metadata, captions and the 1080p video are fetched in parallel. Rolling auto-captions are flattened to word times and regrouped into clean segments. A video with no captions is transcribed from its audio with Gemini instead.
One frame a second, scaled to 1280 px. Near-duplicates are dropped by a perceptual hash of the left 75% of the frame, so a webcam picture-in-picture does not defeat deduplication; a frame is kept when it differs enough from the last kept one, and at least every 30 seconds. On the test video: 394 of 2595.
Kept frames go to the model in windows of up to 12, each with the transcript for those seconds, at high media resolution and temperature 0. Windows run concurrently and are cached, so a rerun skips what is already on disk.
One text-only pass over every window and the full transcript produces the tables. The screen wins when it conflicts with the captions, and each correction is recorded with the on-screen row as evidence. The JSON is validated before it is returned.
Start a job, poll it, read the result as JSON, as a screenplay, or as the summary card. The same URL at the same detail level returns the cached document at no charge. Pass a webhook URL to skip polling.
# Start a job
curl -X POST https://screensay-api.fly.dev/v1/analyze \
-H "Authorization: Bearer $SCREENSAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://www.youtube.com/watch?v=8GRmLR__OGQ", "detail": "standard"}'
# 202 {"job_id": "job_…", "status": "queued", "estimated_seconds": 210}
# Poll until done
curl https://screensay-api.fly.dev/v1/jobs/job_… -H "Authorization: Bearer $SCREENSAY_API_KEY"
# {"status": "done", "progress": {"stage": "done", "pct": 100}, "billed_minutes": 43.3, …}
# Fetch the document as JSON, or as a screenplay for a prompt
curl "https://screensay-api.fly.dev/v1/jobs/job_…/result" -H "Authorization: Bearer …"
curl "https://screensay-api.fly.dev/v1/jobs/job_…/result?format=screenplay" -H "Authorization: Bearer …"An agent hits a YouTube link mid-task. With the MCP server it calls watch_video, gets the summary card and the screenplay, and carries on knowing what was on screen while each sentence was said.
MCP server setupIndex text rows and beats instead of caption fragments. A query for a version string hits the settings pane it appeared in, with the seconds it was visible and the sentence spoken over it.
The screenplay renderingEach sponsor with the time it was announced, named, shown and disclosed, its URL, and how the read was delivered. Entities with every mention and whether the name came from speech or the screen.
Sponsors and entities in the document