The Map: what robots are, and why they’re hard
Before history or frontier models, get the vocabulary into your hands: what a robot is, the four jobs robots do, what “manipulation” means, and the seven facts that make robotics hard.
What is a robot?
A robot is a machine that senses the physical world, decides what to do, and acts on it, over and over. The loop is the whole point: each action changes the world, and that changes what the robot senses next.
Pick a robot and tap a stage. Notice how different the parts are, while the loop stays the same.
| LLM agent | Robot | |
|---|---|---|
| Observes | text, screenshots, tool output | camera pixels, depth, joint angles, forces |
| Acts | a tool call every few seconds | motor commands, 10–1000× per second |
| While it thinks | the world waits | the cup keeps sliding |
| Undo | often possible | a dropped glass stays broken |
| Training data | the internet | whatever humans demonstrate on real robots |
Which came first: a computer that beat the world chess champion, or a robot that can reliably fold any T-shirt you hand it?
Chess fell in 1997 (IBM’s Deep Blue beat Garry Kasparov). Folding arbitrary laundry is still a showcase demo for the best robot labs in 2025–2026. This is Moravec’s paradox, the first of the seven hard truths in Module 4: skills that evolution spent hundreds of millions of years perfecting, like seeing, grasping and balancing, are the hardest to engineer.
Which is the best definition of a robot?
Arms, legs and human shapes are optional: a Roomba and a Waymo car have none of them. What makes a robot is the closed loop with physics. An LLM agent closes a loop too, but with software, which waits for it and often forgives mistakes.
The four jobs robots do
Manipulation: changing the world with a hand
- What it is
- The robot uses an end-effector (a gripper, suction cup or hand) to make contact with objects and change their state: move them (pick-and-place), reshape them (fold a towel), or operate them (open a drawer, turn a valve, insert a plug).
- Real examples
- A car-factory arm welding bodies; a warehouse arm picking one item out of a jumbled bin; a surgical robot suturing (driven by a surgeon); a research robot folding laundry.
- Why it’s hard
- Contact. The instant a finger touches a mug, tiny differences in friction and geometry decide whether it is grasped or knocked over. Objects are endlessly varied: floppy, shiny, transparent.
Locomotion: moving your own body
- What it is
- Moving the robot itself with legs, wheels or both, without falling over.
- Real examples
- Boston Dynamics’ Spot climbing stairs; humanoids walking; quadrupeds hiking rough trails.
- Why it’s hard
- Balance and dynamics. A legged robot is always about to fall, and control has to react within milliseconds.
Navigation: getting from A to B
- What it is
- Knowing where you are (localization), knowing what the world looks like (mapping), and finding a safe path (planning).
- Real examples
- Roomba; Waymo’s driverless cars; warehouse robots that carry shelves; Mars rovers.
- Why it’s hard
- Uncertainty. Sensors are noisy, maps are imperfect, and people move.
Mobile manipulation: all of the above
- What it is
- A robot that walks or drives somewhere, then manipulates: “go to the kitchen and load the dishwasher.”
- Real examples
- Home-robot research such as Physical Intelligence’s π0.5 cleaning kitchens in homes it had never seen (2025); humanoids in factories; Spot with its arm opening doors.
- Why it’s hard
- Errors in each part multiply, and tasks are long (many minutes, many steps).
Sort the robots
A robot slides a key into a lock and turns it. Which job is that?
It changes the state of objects through contact. Insertion (keys, plugs, pegs in holes) is a classic hard manipulation task, because millimetre errors decide jam vs. success.
Anatomy of a robot arm: what manipulation is made of
Every joint has an encoder that measures its angle. That stream of joint angles is the robot’s proprioception, its sense of its own body. Papers call it the robot’s state.
Joints, links and degrees of freedom
- An arm is a chain: rigid links connected by joints. Most joints rotate (like your elbow); some slide (like a drawer).
- Each joint adds one degree of freedom (DoF): one independent way to move.
- To put a hand anywhere in 3D and point it any way, you need 6 DoF: 3 for position (x, y, z) and 3 for orientation (roll, pitch, yaw). Many research arms have 7, one spare. So does yours: shoulder 3, elbow 1, wrist 3.
Play: forward and inverse kinematics
Forward kinematics (FK): joint angles → where the hand is. Easy; there is exactly one answer.
Inverse kinematics (IK): where I want the hand → joint angles. Hard; there may be zero, one, two or infinitely many answers.
- Move the joints: watch the hand trace a path. That is FK.
- Drag the hand: the arm solves IK to follow your finger.
- Drag beyond the dotted circle. Why can’t it get there?
- Hold a spot and slide hand angle: the hand stays put, the arm reshapes (redundancy). Flip elbow up / down: same pose, different arm.
This flat arm has 3 joints. If its hand must stay on one point, how many arm shapes can do that?
A point on a flat page takes 2 numbers (x, y), and the arm has 3 joints, so 1 DoF is spare. Every hand angle gives a new shape (the slider you played with), and each comes in elbow-up and elbow-down versions. Spare DoF = a whole continuum of solutions.
Senses: what a robot can perceive
- Cameras: on the wrist, plus one or more watching the scene. Most VLAs see the world this way.
- Depth sensors: per-pixel distance. They fail on glass and shiny metal.
- Joint encoders: proprioception, as above.
- Force/torque and touch sensors: how hard am I pushing? Is it slipping?
Action space: what a robot policy actually outputs
| Action space | Means | Used by, for example |
|---|---|---|
| Joint positions | “put joint 3 at 41°”: a target for every joint | ALOHA / ACT: 14 numbers (two 6-joint arms + 2 grippers), 50× per second |
| End-effector deltas | “move the hand 1 cm left, tilt 5°, close the gripper” | RT-1, RT-2, OpenVLA: Δx Δy Δz, Δroll Δpitch Δyaw, gripper, each binned into 256 levels |
| Velocities or torques | “spin this joint this fast / push this hard” | low-level motor controllers, and some learned walking policies (many output joint targets instead) |
Why this matters for the frontier: if one robot “speaks” joint angles and another speaks hand moves, a single model trained on both has to translate. Choosing a shared action space is part of the open cross-embodiment problem (Day 6).
Show me the math (optional)
For a 2-link arm with link lengths L₁, L₂ and joint angles θ₁, θ₂, FK just adds up the links:
x = L₁·cos θ₁ + L₂·cos(θ₁+θ₂) y = L₁·sin θ₁ + L₂·sin(θ₁+θ₂)
IK runs this backwards with the law of cosines. The ± in the solution is literally elbow-up vs. elbow-down. In 3D, every joint becomes a 4×4 matrix (“rotate, then shift”), and FK multiplies the chain of matrices.
The Jacobian answers: “if I wiggle each joint a tiny bit, how does the hand move?” When the arm is fully stretched, some hand directions become impossible no matter how you wiggle. That is a singularity.
OpenVLA outputs 7 numbers at every step. What are they?
End-effector deltas: 3 for position, 3 for orientation, 1 for the gripper. Each is discretized into 256 bins, so a VLA can “say” an action as 7 tokens, just like words. More on that on Day 5.
Why robotics is hard: the 7 hard truths
1Moravec’s paradoxWhat’s easy for us is hard for machines.›
Example: Deep Blue beat Kasparov in 1997, yet folding laundry is still a frontier demo.
Why: evolution spent hundreds of millions of years tuning perception and motor control, while abstract reasoning like chess is, evolutionarily, brand new. We badly underestimate how much computation “easy” skills take.
The field’s response: stop hand-coding skills and learn them from data (Day 4).
2You never see the whole worldPartial observability and noise.›
Example: the mug is behind the box; depth sensors fail on glass; lidar gets noisy in rain.
Why: sensors are partial and noisy, so the robot must act on beliefs (probabilities), not facts.
The field’s response: probabilistic robotics: Kalman and particle filters, SLAM (Day 3).
3Contact is hardPhysics breaks at the moment of touch.›
Example: a 1 mm offset decides whether a peg slides in or jams; friction changes with dust and humidity.
Why: in free air, motion is smooth and predictable. At contact it becomes discontinuous: stick or slip, touch or not.
The field’s response: feedback and compliance (Day 2), learning from demonstrations (Day 4).
4Errors compoundSmall mistakes snowball in closed loop.›
Example: CMU’s ALVINN (1989) learned to steer a van by imitating a human driver. Its builders had to synthesize “off-centre” training views so it could learn to recover from drift.
Why: each small error puts the robot somewhere slightly unfamiliar, where it errs more. LLM folks know this as exposure bias, except here it’s physical.
The field’s response: DAgger, feedback, action chunking (Day 4).
5Data is scarceThere’s no internet of robot actions.›
Example: Llama 3 trained on ~15 trillion text tokens. Open X-Embodiment (2023), the big pooled robot dataset, has ~1 million episodes, mostly collected by humans steering real robots.
Why: text and video show what happened, not which motor commands made it happen.
The field’s response: data engines, cross-robot pooling, simulation, human video (Day 6).
6Sim ≠ realSimulators are cheap but subtly wrong.›
Example: a policy that is flawless in simulation fails on the real robot because friction, cloth, cables, lighting and motor delays differ slightly.
Why: learning algorithms exploit every flaw in the simulator.
The field’s response: domain randomization and sim-to-real transfer; OpenAI’s Rubik’s-cube hand (Day 4).
7The world doesn’t waitReal time: big brains are slow, bodies need reflexes.›
Example: a walking robot corrects its balance hundreds of times per second, while a 7-billion-parameter vision-language model in a 2025 humanoid runs at only 7–9 decisions per second.
Why: physics keeps going while the model thinks.
The field’s response: two-speed “System 1 / System 2” architectures, action chunking, asynchronous inference. This is the systems frontier (Days 2, 5, 6).
An imitation-learned policy picks the right action 99% of the time at each step. A task takes 200 steps, and any mistake ruins it. Roughly how often does it succeed?
0.99²⁰⁰ ≈ 0.13. That’s hard truth #4, and it’s optimistic: in practice early mistakes make later ones more likely. This one number explains a lot of robot-learning research.
A VLA needs 150 ms per decision, but the arm wants a new command every 20 ms (50 Hz). What do real systems do?
Two standard tricks: action chunking (π0 predicts 50 future actions per call) and two-speed designs (Figure’s Helix: a 7–9 Hz “thinker” plus a 200 Hz “doer”). You’ll see the 1990s ancestor of the second idea tomorrow.
The 60-year story on one screen
1961 · Day 2Program itUnimate at General Motors
✓ Solved: tireless, precise repetition of a taught motion in dangerous jobs.
✗ Broke: zero perception. Shift the part and it grabs air.
…so give robots senses and the ability to plan →
1966 · Day 2Plan itShakey at SRI: sense, model, plan, act
✓ Solved: reasoning about goals (“push the box off the platform”).
✗ Broke: slow and fragile; its world model went stale whenever something moved.
…so skip the model and react directly →
1986 · Day 2ReactBrooks’ subsumption → Genghis → Roomba
✓ Solved: fast, robust behavior in a messy world, with cheap sensors.
✗ Broke: no goals or memory; it gets trapped in dead ends.
…so put a planner on top of the reflexes →
1990s · Day 2Layer itThree-layer architectures; NASA’s Remote Agent
✓ Solved: goals and reflexes in one system.
✗ Broke: every layer was hand-engineered, and the whole stack still needed accurate models of the world.
…so model the uncertainty and the physics properly →
1995–2015 · Day 3Model the worldMinerva, Stanley, BigDog
✓ Solved: noisy sensors (probability) and agile balance (physics models).
✗ Broke: you can’t hand-model the open world (every object, every fabric). The DARPA Robotics Challenge robots famously fell over.
…so learn the models and policies from data →
2012–2022 · Day 4Learn itVisuomotor policies, sim-to-real RL, imitation
✓ Solved: perception and control learned from data; RL conquered legged locomotion.
✗ Broke: each policy was narrow: one task, one robot, lots of data.
…so borrow the internet’s knowledge →
2022– · Day 5Foundation modelsRT-2, π0, Gemini Robotics, GR00T
✓ Solved: generalization to new objects, instructions and scenes, using web-pretrained vision-language models plus pooled robot data.
✗ Breaks: slow vs. real time, data-starved, hard to evaluate, weak at physics.
…which defines today’s frontier →
2025– · Days 5–7The frontierWorld models · agentic robots · RL from experience · systems
Learned simulators of the world, LLM-style orchestrators that call robot skills as tools, robots that keep learning after deployment, and the systems engineering (latency, compute, data, evaluation) that holds it all together. This is where you’ll do research.
Why did reactive, behavior-based robots (Brooks, 1986) appear right after the Shakey era?
Each era answers the last one’s failure. Shakey could reason but couldn’t keep up with the world; Brooks gave up on reasoning to keep up. Tomorrow you’ll race the two against each other.
Glossary for today (tap to open)
- Manipulation
- Changing objects’ state through contact: moving, reshaping, operating them.
- Locomotion / navigation
- Moving your own body / getting from A to B.
- End-effector
- The tool at the end of the arm: gripper, suction cup, dexterous hand.
- Degree of freedom (DoF)
- One independent way to move, usually one joint.
- Forward / inverse kinematics
- Joints → hand pose / hand pose → joints.
- Proprioception (“state”)
- The robot’s sense of its own body: joint angles and velocities.
- Policy
- The function that maps observations to actions: a controller, or a neural network.
- Action space
- What the policy outputs: joint targets, hand moves, velocities, torques.
- Teleoperation
- A human driving the robot remotely; how most demonstration data is collected.
- Embodiment
- A specific robot body (its joints, sensors, shape). “Cross-embodiment” = across different robots.
Go deeper (optional)
Long lectures from top courses. Phone tip: play them at 1.5× and stop after the first 20 minutes.
- Free textbook: Lynch & Park, Modern Robotics (Northwestern), chapters 1–2 on configuration space and degrees of freedom.
- Free textbook: Tedrake, Robotic Manipulation (MIT), chapter 1.
Sources for today’s facts
- Open X-Embodiment: 1M+ trajectories, 22 robot embodiments: arXiv 2310.08864
- Llama 3 pre-training on ~15T tokens: arXiv 2407.21783
- Figure Helix (7B S2 at 7–9 Hz, 80M S1 at 200 Hz): figure.ai/news/helix
- π0 (action chunks of 50 steps): arXiv 2410.24164 · π0.5 in unseen homes: pi.website/blog/pi05
- OpenVLA (7-D end-effector actions, 256 bins): arXiv 2406.09246 · RT-1: arXiv 2212.06817 · ALOHA / ACT: arXiv 2304.13705
- ALVINN: Pomerleau, “ALVINN: An Autonomous Land Vehicle in a Neural Network”, NeurIPS 1988 · Moravec, Mind Children (1988)