Skip to main content

​
What is a vision language model?

Time: 9:30 AM to 9:55 AM
Yesterday, your LLM took text in and generated text out. Today, it learns to see. A vision language model (VLM) takes both images and text as input, so you can ask questions about what the robot’s camera captures.

​
From text to vision

On Day 3, your robot could hear and speak, but it was blind. Today you give it eyes. A VLM is not a separate model — it is the same LLM (GPT-4o-mini) with a vision encoder bolted on. The vision encoder converts images into tokens the language model can understand.

​
How vision encoding works

When you send an image to the API, it goes through a multi-step pipeline before the language model ever sees it:
1

Image capture

The robot’s camera captures a frame as a JPEG image (640x480 pixels).
2

Base64 encoding

The JPEG is converted to a base64 text string so it can be included in the API request.
3

Vision encoder

On OpenAI’s servers, a vision encoder (like a CLIP model) converts the image into a sequence of visual tokens — a compressed numerical representation of the image.
4

Combined input

These visual tokens are concatenated with your text prompt tokens. The language model now sees both.
5

Response generation

The model generates text tokens conditioned on both the image and your question, producing a response that references what it “sees.”
The model does not see like a human. It receives a compressed mathematical representation of the image and predicts tokens based on patterns learned during training. It can be fooled by unusual angles, low lighting, or objects it has never seen in training data.

​
What this enables

With a VLM, you can ask your robot:

​
What VLMs struggle with

VLMs are impressive but have clear limits:
VLMs hallucinate just like text-only LLMs. The model may confidently describe an object that does not exist in the image. Always verify important observations.

​
Model and cost

Today’s exercises use GPT-4o-mini via the OpenAI API. Each vision API call costs roughly 300-800 tokens depending on image resolution. We use detail: "low" to keep costs down while still getting useful descriptions.

​
From blind to sighted: the week so far

By the end of today, your robot will have all five layers: movement, sensing, speech, language intelligence, and vision. That is the full stack of embodied AI.