Skip to main content

​
Tool calls: how an LLM controls real things

Time: 10:05 AM to 10:25 AM
You just learned that an LLM is a brain in a jar — it can only generate text. So how do you make it drive a robot, read a sensor, or speak out loud? The answer is tool calling (also called function calling). Instead of just writing words, the model can trigger real actions in your code.

​
The problem

Without tool calls, here is what happens when you tell an LLM to move your robot:
You: “Drive forward for 3 seconds.” LLM: “Okay, I’m driving forward for 3 seconds!”
The robot does not move. The model generated the words “I’m driving forward” but has no way to actually control the motors. It is like telling someone to turn on the lights over a phone call — they can say “done!” but the lights stay off. Tool calling fixes this by giving the model a menu of real functions it can ask your code to run.

​
How tool calling works

1

You define available tools

You give the LLM a list of functions it can call, along with descriptions of what each function does and what arguments it accepts.
2

The model decides to use a tool

When the model determines it needs to take an action, it outputs structured JSON specifying which function to call and what arguments to pass. It does not run the function itself.
3

Your code runs the function

Your Python program reads the JSON, calls the actual function (which controls the robot’s motors, servos, or speaker), and gets a result.
4

The result goes back to the model

The function result is sent back to the model. The model incorporates the result and generates its final spoken response.

​
What a tool definition looks like

When you send a message to GPT, you also send a list of tools. Each tool is a JSON object that describes a function. Here is one from the program you will run later today:
The model reads the name, description, and parameters to understand what the function does and what arguments to pass. It never sees your Python code — only this JSON description.
The quality of your descriptions matters. If you write a vague description like “does a thing”, the model will not know when to use it. Clear, specific descriptions lead to better tool use.

​
Your robot’s tool menu

Your robot has 10 tools the LLM can call. Here is the full menu: The model can call multiple tools in one response. If you say “make a square”, it might call move_forward then turn_right four times in sequence.

​
The brain and body analogy

Think of it this way: the LLM is the brain, and tool calls are the motor nerves connecting it to a body. Without tool calls, the brain can think but cannot act. With tool calls, thoughts become physical motion.The results that come back are like sensory nerves — they tell the brain what happened so it can decide what to do next.

​
Why this matters beyond robots

Tool calling is not just for robots. It is the same pattern behind:
  • ChatGPT plugins — the model calls web search, code execution, or image generation
  • AI assistants — Siri and Alexa use function calling to set timers, play music, and control smart home devices
  • AI agents — autonomous programs that browse the web, write code, and manage tasks
Every time an AI “does something” instead of just “says something”, tool calling is what makes it happen. You are learning the exact same architecture that powers the most advanced AI systems in the world.
After this section, take a 10-minute break (10:25 AM to 10:35 AM).