Multi-Modal Prompting with Gemini (Image, Video, Audio In)

TL;DR: "Gemini" is at least four different surfaces: a consumer app, AI Studio, the Gemini API, and Vertex, and they take different files, enforce different limits, and sometimes use different field names for the same setting. Google's own docs disagree with themselves on where to put a text prompt relative to an image, video, or document, and on how many languages its Live API covers. Below: what each surface actually accepts, and the generation-side conventions that changed most recently.

Which "Gemini" Are You Actually Prompting?

Every tutorial that says "in Gemini, do X" is skipping a step, because the answer to "what does Gemini accept" depends on which of at least four surfaces you mean.

The consumer app at gemini.google.com is plan-gated: file limits, context window, and even model access change between the free tier, AI Plus, AI Pro, and AI Ultra. AI Studio is a developer playground that talks to the same underlying models as the API, for free, with a UI in front of it. The Gemini API is what a server-side integration actually calls, now built around a unified endpoint (POST https://generativelanguage.googleapis.com/v1beta/interactions) that Google calls the Interactions API, alongside the older generateContent endpoint that still works and still gets documented separately. Vertex AI is the enterprise surface on Google Cloud, with its own model cards and, per Google's own pages, a wider aspect-ratio list than the consumer-facing API guide for the same image models.

If you want copy-paste prompt templates organized by what you're feeding in, image, PDF, audio, video, Gemini Prompt Templates (Multi-Modal Ready) already covers that ground in depth. This post is the mechanics underneath: why a template that works on one surface can silently misbehave on another, and what changed most recently on the generation side.

Where Do You Put the Text? Google's Own Guides Disagree

This sounds like a small detail until you've built a pipeline around the wrong assumption. Google's own documentation gives three different, surface-specific answers, verified directly against the current pages:

• Images: "When using a single image with text, place the text prompt before the image in the input array."
• Video: "If combining text and a single video, place the text prompt after the video part in the input array."
• Documents: "If using a single page, place the text prompt after the page."

Three input types, three different placements, all from the same vendor's own current guides. There's no unifying rule to memorize here, and no indication one order is a hard requirement versus a soft recommendation. The practical takeaway: don't port a prompt-ordering habit from one modality to another inside Gemini, and if a specific request keeps producing worse results than expected, try flipping the order before you touch the wording.

Screenshot-and-Ask: The Workflow Gemini Is Actually Good At

If Gemini has one genuinely strong, differentiated habit, it's this: paste in a screenshot, a photo, or a diagram, and just ask about it, in the same turn, with no separate upload step or file-management ceremony. That's the core of what "multi-modal prompting" means in practice, and it's worth grounding in the vendor's own numbers rather than vibes. The equivalent workflow inside ChatGPT is covered separately in How to Prompt with Images in ChatGPT; the mechanics below are specific to Gemini's own surfaces and limits.

On the API side, Gemini accepts up to 3,600 image files in a single request, with inline image data capped at 20MB total per request (larger files go through the Files API instead). Google's own best-practice tips for this are worth following literally: "Verify that images are correctly rotated," "Use clear, non-blurry images," and for a single image plus text, put the text first, per the placement rule above.

Underneath that, the image is tokenized in a specific, published way worth knowing before you send a batch of screenshots and wonder why the bill looks the way it does: an image at 384 pixels or smaller in both dimensions costs 258 tokens, and anything larger gets tiled into 768x768 blocks, each costing 258 tokens on its own. A single high-resolution screenshot can quietly cost several times what a compressed one does, for the same question.

The screenshot-and-ask pattern extends further than static images. Gemini's video understanding defaults to static frame sampling at 1 FPS, but the current Flash-family models also support what Google calls agentic video understanding, where the model "dynamically explores the video timeline, selectively inspecting transcripts and adaptively adjusting frame rates and resolution on the fly based on the prompt." In practice, that means you can hand Gemini a screen recording instead of a single screenshot and ask a question that needs the model to find the right moment on its own, referencing timestamps in MM:SS format if you need to point it at a specific point.

Audio extends the same habit to sound the model wasn't explicitly told to listen for. Google's own documentation for audio understanding notes that "Gemini understands non-speech sounds (birdsong, sirens, etc.)," not just transcribable speech, which is a genuinely different capability from a transcription service: you can ask it to flag a specific background noise in a recording, not only what was said.

Which Image Model Are You Even Using?

"Nano Banana" gets used as a single name in casual conversation, and Google's own documentation is explicit that it isn't one: "Nano Banana refers to four distinct models available in the Gemini API," each with its own id and its own resolution ceiling. Which one you're actually calling changes what the output can do, independent of anything in your prompt.

Nano Banana 2 Lite (gemini-3.1-flash-lite-image) is described as the fastest and cheapest option. Google's own docs add that it is "Not optimized for multiple reference inputs or multi-turn sequential editing," and it's capped at 1K resolution. Nano Banana 2 (gemini-3.1-flash-image) is the generalist workhorse, with "state-of-the-art 4K generation" and the strongest multi-reference consistency of the non-Pro tier; it's also the one model in the family that adds a 512-pixel (0.5K) resolution option below 1K. Nano Banana Pro (gemini-3-pro-image) is positioned for the most complex visual tasks, with the highest world knowledge and localization support, and is the model Gemini's own docs recommend for precision creative control. The legacy Nano Banana (gemini-2.5-flash-image) still works, but Google's own guidance is to move off it, noting that "we strongly recommend that customers transition to Nano Banana 2 Lite to experience enhanced quality, faster generation speeds, and lower API pricing."

None of that resolution ceiling matters if you're prompting from the consumer app rather than the API. Google's own Gemini Apps help documentation is explicit that download resolution in the app is capped at 2K on a paid Google AI plan and at 1K without one, regardless of which Nano Banana model actually drew the picture. A 4K generation is real and native on the API, AI Studio, or Vertex; it is not something the consumer app will hand you as a file, on any plan, today.

Why Does Image Generation Use Two Different Field Names?

This is the part that trips up anyone copying a code sample from two different pages of Google's own documentation, and it's the most load-bearing fact in this post.

Both conventions are genuinely live at the same time. The Interactions API's own documented example for image output reads exactly: { "type": "image", "aspect_ratio": "16:9", "image_size": "1K", "mime_type": "image/jpeg" }. The generateContent API reference, meanwhile, documents an ImageConfig object whose fields are aspectRatio and imageSize, camelCase, and lists a wider set of supported ratios than the newer guide, including 1:4, 4:1, 1:8, and 8:1. Neither is a typo, and neither has been quietly retired. If you copy a snake_case example into generateContent, or a camelCase example into an Interactions API call, it will simply not work.

Copy-paste, side by side, both taken directly from Google's own current reference examples:

// Interactions API (current): snake_case, nested under response_format
{
  "type": "image",
  "aspect_ratio": "16:9",
  "image_size": "1K",
  "mime_type": "image/jpeg"
}

// generateContent (still documented): camelCase, its own ImageConfig object
// nested under generationConfig.imageConfig
{
  "generationConfig": {
    "imageConfig": {
      "aspectRatio": "16:9",
      "imageSize": "1K"
    }
  }
}

Case sensitivity adds a second layer of the same problem, and it doesn't even split cleanly along the endpoint boundary. Image generation wants an uppercase "4K"; Google's own generation guide states this outright, warning that lowercase values like 1k are rejected. Video is the opposite: Veo's own resolution parameter, in every code sample Google publishes, is lowercase "4k". That's not one convention per endpoint, either. Inside the Interactions API's own response_format object, its ImageResponseFormat.image_size field takes uppercase "4K" while its own VideoResponseFormat.resolution field, defined a few lines later in the same reference, takes lowercase "4k". Google is not one API, and in this specific case it isn't even one convention inside one API.

What Happened to responseSchema?

If you built a Gemini integration around structured output any time before this year, the field you used is now marked deprecated. Google's generateContent API reference lists both responseSchema and the alternate _responseJsonSchema, and each one now carries the same flag, verbatim: "This item is deprecated!"

The documented replacement lives on the Interactions API, in the same response_format object already covered above, using a type, a mime_type, and a schema for text output shaped as JSON. Existing code built on responseSchema still functions; it's the guidance you'd write today that needs to change, not necessarily production code you already shipped.

The bigger surprise sits one layer down: the Interactions API doesn't just rename its sampling controls, it drops them. A direct check of Google's own Interactions API reference turns up zero occurrences of temperature, top-p, or top-k as request parameters; the only top_k field anywhere on the page belongs to an unrelated file-search tool's chunk-retrieval setting, not to sampling. Compare that to generateContent, whose GenerationConfig still documents temperature, topP, topK, seed, and maxOutputTokens in full, plus frequencyPenalty and presencePenalty with no published numeric range or default for either. If your workflow depends on temperature or top-p tuning, generateContent is still where that lives; the Interactions API currently has nowhere to put it. Temperature, Top-P, Top-K: AI Sampling Parameters Explained covers what each of those actually does if you need the mechanics rather than just the field names. For the naming mess around token ceilings specifically, Max Tokens vs Max Output Tokens covers Gemini's maxOutputTokens alongside the other providers' equivalents in one place. And if the structured-output move above means you're building a schema-validated prompt for the first time, Free JSON Prompt Generator turns a plain description into the schema shape either endpoint expects.

What Can You Actually Feed In?

On the Gemini API, per Google's own current documentation: images support up to 3,600 files per request with a 20MB inline cap; PDFs go up to 50MB or 1,000 pages, at roughly 258 tokens per page; audio runs up to 9.5 hours at 32 tokens per second and understands non-speech sound, not just transcribable words; video accepts up to 10 files per request on Gemini 2.5 and later, sampled at 1 FPS by default, with public YouTube URLs accepted directly (private and unlisted videos are not).

On the consumer app, per Google's own support pages: up to 10 files per prompt, each capped at 100MB (2GB for video specifically), with total video length capped at 5 minutes and total audio at 10 minutes on a free account, extending to 1 hour of video and 3 hours of audio on AI Pro or AI Ultra. The context window is plan-gated on its own separate schedule: 32k tokens with no AI plan, 128k on AI Plus, and 1 million tokens on AI Pro and AI Ultra.

Those two sets of numbers describe the same underlying models answering to two different sets of rules. A workflow you tested against the API's 3,600-image ceiling tells you nothing about what a teammate can actually upload in the consumer app that afternoon.

Is "Gemini Live" the Same Thing as the Live API?

No, and the shared name is doing real damage to anyone searching for help on either one. Gemini Live is a consumer feature inside the app: a real-time voice conversation with memory across sessions and direct integration with Gmail, Calendar, and Maps. The Live API is a separate developer product, a WebSocket-based real-time interface, currently in Preview, with its own configurable voice-activity detection and thinking-level settings, and none of Gemini Live's consumer integrations. Google's own feature list for the Live API is aimed squarely at builders, not end users: barge-in support, described as "Users can interrupt the model at any time for responsive interactions," and tool use, described as one that "Integrates tools like function calling and Google Search for dynamic interactions." If you're prompting a voice agent you're building yourself, you want the Live API's docs; if you're troubleshooting the assistant inside the app, those docs won't have your answer.

Even Google's own documentation for the Live API disagrees with itself about scope: the product overview page advertises "Converse in 70 supported languages," while the capabilities guide, on the same product, lists a full BCP-47 language table that runs to 97 entries when counted directly. Both numbers are currently live on Google's own site, for the same API. That's a small, almost funny detail, and also the cleanest illustration in this entire post of the rule worth taking away: Gemini is not one product with one set of facts. Name the surface, check the date, and verify the specific page before you promise a client or a teammate a specific number.