Robot manipulation is two to three orders of magnitude harder than motion control, and Silicon Valley has bet the wrong way
Sudo believes the mainstream Silicon Valley end-to-end imitation learning route has a fundamental flaw in open-world manipulation, and that a hierarchical top-down/bottom-up structure will return to the mainstream in the second half of this year.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Video training cannot reach sub-millimeter precision
Han Zheng explains why you cannot directly train robot manipulation on internet video. 2D images and video do contain three-dimensional information, but it is "not quite complete, and not quite accurate." Autonomous driving navigation only needs precision of around 10 centimeters, but robot manipulation demands sub-millimeter precision — for example, inserting a cable into the corresponding hole. Only datasets built from 3D modeling can achieve that precision. The simulator requires the robot's understanding of geometric precision to reach millimeter or even sub-millimeter level.
— Han ZhengZero-shot general grasping hits 98% single-attempt success
In internal tests in November and December last year, Sudo R1 operated over 100 objects for a continuous hour, 240 pick-and-place cycles, and none of these objects had ever appeared in the training data. Single-attempt pick-and-place success rate was close to 98%, and if the first attempt failed, the robot would use closed-loop control to rapidly adjust its strategy and reach 100%. Han Zheng calls this the first time the Sim2Real route has closed the gap. Afterwards they had nearly 1,000 people test it with random objects on site at ICRA and CVPR, something no one had done at academic conferences before.
— Han ZhengShort-skill generalization matters more than chaining long-horizon tasks
Han Zheng argues that zero-shot stable short skills for manipulating basic objects in the open world are more core than reproducing long-horizon tasks. If short skills are stable and reliable enough, with sufficiently high generalization, they can be assembled into all kinds of long-horizon operations. This is also the core idea of the ManiSkill framework. Around the same time the Sudo team proposed chain of thought in 2023, there was a piece of work called Chain of Thought Predictive Control, specifically addressing the chaining of short tasks into long-horizon tasks.
— Han ZhengManipulating objects is two to three orders of magnitude harder than motion control
Han Zheng points out that reinforcement learning in simulation and Sim2Real have already been fully validated in motion control for Boston Dynamics and Unitree humanoid robots and quadruped dogs, and part of the reinforcement learning engine inside Nvidia Isaac was contributed early on by members of the Sudo team. But when a robot manipulates objects, the difficulty is two to three orders of magnitude higher. Because motion control only needs to focus on its own actions, it does not need to understand object shape or material, nor does it need to know how to move near an object and make adjustments.
— Han ZhengTesla cannot be replicated; cold start must rely on simulation
Han Zheng explains why robots cannot collect data by selling cars the way Tesla did. Tesla started selling cars over a decade ago, millions of drivers drive on the road every day, and data accumulates naturally. But it is very hard for a robot to sell 5 million units into offices and homes before it has any function, with a worker standing next to it remote-controlling it. Domestic companies might reach a scale of a hundred or even a thousand units, but that is far from 5 million or 10 million units entering every household. So the cold start must rely on simulation, requiring millions or even tens of millions of full data collected in open environments.
— Han ZhengThe hierarchical structure Silicon Valley abandoned is about to return to the mainstream
Han Zheng reveals that when Physical Intelligence was still at Google Robotics before it was founded, everyone tacitly assumed that once the low-level model had stable manipulation capability, the high level would handle environment understanding and task decomposition. But when they founded their startups in 2023 and 2024, everyone chose the end-to-end playbook they were good at and familiar with, and stopped mentioning hierarchy. Han Zheng's hot take is: starting in the second half of this year, the hierarchical top-down/bottom-up structure will return to the mainstream, and may be the final commercialization solution. After exchanges between academia and industry in recent weeks, some people have already realized this could be the new direction.
— Han ZhengMost bullish on DeepMind plus Boston Dynamics Atlas
Han Zheng considers the biggest competitor to be DeepMind plus the electric version of Boston Dynamics Atlas. Last year's Demis Hassabis interviews and documentary showed that Google's bet on physical-world AI is the most resolute among all the big companies. They have already switched their hardware platform from Apptronik to the electric version of Atlas, and poached core people from Boston Dynamics' hardware division to DeepMind. Google also acquired the MuJoCo team, has its own world model Genie 3, and has almost unlimited GPUs and the best talent. Han Zheng believes this is simply not on the same order of magnitude as other companies.
— Han ZhengAmazon has many scenarios but integration is difficult
Han Zheng believes Amazon has always been the big company investing the most in robotics, with warehousing and logistics AGVs, Kiva, various robotic arms and humanoid robot companies such as Agility and Covariant. It will provide scenarios, but in the end it will either acquire these companies or do it itself. Covariant at the time also counted Amazon as one of its core customers. But integrating multiple robot systems is difficult.
— Han ZhengIn their own words · checked verbatim
2D images and video do contain three-dimensional information, but it's not quite complete, and not quite accurate.
2D的图片和视频里边 三维的信息有 但是不太全 也不太准
Han Zheng13:21
The single-attempt success rate for pick-and-place on this object is close to 98%, but if it fails the first time, the robot will still use a closed-loop control method to rapidly adjust its strategy again, and then it can reach 100%.
单次对于这个物体 做抓放的成功率是将近98% 但是如果它第一次失败之后 这个机器人 还会用一个闭环控制的方法 迅速地再去调整它的策略 就可以做到100%
Han Zheng28:05
If a robot is going to manipulate objects, the difficulty is two to three orders of magnitude higher.
机器人 如果是要对于物体去做操作 它这个难度要高两到三个数量级
Han Zheng44:51
It's very hard for a robot — the same model of robot, before it has any function — for me to sell 5 million robots into offices, homes, or factories, with a worker standing next to it remote-controlling it to do all kinds of operations.
机器人你是很难 同一款机器 在没有任何功能之前 我卖500万台机器人 到各个办公室 家庭 或者是工厂里边去 并且旁边还有一个工人在遥控它 去做各种各样的操作
Han Zheng53:27
We have a hot take: possibly starting in the second half of this year, this hierarchical top-down/bottom-up structure will return to the mainstream, and the solution everyone chooses to push into commercialization afterwards may be the final method.
我们有一个暴论 可能从今年的下半年开始 上下分层的这种结构 会重新再回到主流里边来 而且可能在之后 大家选择的方案推到商业化里边去 可能是最终的一个方法
Han Zheng55:38
He may be the strongest in the future, because he has both hardware and software — I think this is simply not on the same order of magnitude as other companies.
他可能是未来最 因为他硬件也有 软件也有 我觉得这个跟其他公司 完全不是在一个量级上的
Han Zheng1:05:40
Figures
| Sudo R1 zero-shot grasping success rate | 98% single attempt, 100% after closed-loop adjustment | 28:05 |
| Sudo R1 test scale | 1 continuous hour, over 100 objects, 240 pick-and-place cycles | 28:05 |
| Number of people in on-site ICRA testing | nearly 1,000 | 32:11 |
| Manipulation is harder than motion control by | 2 to 3 orders of magnitude | 44:51 |
| Unitree 2025 shipments | over 5,000 units | 59:08 |
| Tesla Figure shipments | about 150 units | 59:08 |
| Skild AI latest valuation | $14 billion to $15 billion | 1:00:34 |
| Skild AI funding raised | $1.4 billion | 1:00:34 |
| Physical Intelligence valuation | $5.6 billion | 1:00:34 |
| Hyundai Group's stake in Boston Dynamics | SoftBank retains 10% to 20% | 1:08:58 |
Glossary
- Sim2Real
- Training robots in a virtual simulation environment, then transferring the learned capabilities to the real physical world.
- Zero-Shot
- A model directly completing a task on objects or environments it has never seen in training, without additional fine-tuning.
- VLA
- An end-to-end model primarily imitating human actions, jointly modeling vision, language and action.
- URDF
- A standardized file format used to describe a robot's geometry and dynamics constraints.
- teleops
- Humans remotely controlling a robot via a remote-control device to collect manipulation data.
How to listen
Founders, investors and robotics engineers following the divergence in embodied AI technical routes, especially those who want to understand Sim2Real feasibility, data cold-start strategies and the Silicon Valley competitive landscape.
The discussion from 1:09:01 to 1:10:58 about Amazon and vertical-scenario commercialization, which is low in information density.