Robot manipulation is two to three orders of magnitude harder than motion control, but Silicon Valley is betting the wrong way
Sudo Technology argues that Silicon Valley's mainstream end-to-end imitation learning approach has a fundamental flaw in open-world manipulation, and that a hierarchical structure will return to the mainstream in the second half of this year.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
Video training cannot achieve sub-millimeter precision
Han Zheng explains why internet video cannot be used directly to train robot manipulation. Although 2D images and videos contain three-dimensional information, it is "not quite complete, and not quite accurate." Autonomous driving navigation only needs about 10 centimeters of precision, but robot manipulation requires sub-millimeter precision—for example, inserting a cable into the corresponding hole. Only datasets built from 3D modeling can achieve this precision. Simulators require the robot's understanding of geometric precision to reach millimeter or even sub-millimeter level.
— Han ZhengZero-shot general grasping achieves 98% single-attempt success rate
In internal tests in November and December last year, Sudo R1 operated over 100 objects for one continuous hour, performing 240 pick-and-place actions, none of which had ever appeared in the training data. The single-attempt success rate was close to 98%, and if the first attempt failed, the robot would use closed-loop control to quickly adjust its strategy and achieve 100%. Han Zheng calls this the first time the Sim2Real route has closed the gap. Later, they had nearly 1,000 people test with random objects at ICRA and CVPR, something no one had done at academic conferences before.
— Han ZhengShort-skill generalization is more critical than long-horizon task chaining
Han Zheng believes that Zero-Shot stable short skills for manipulating basic objects in the open world are more core than reproducing long-horizon tasks. If short skills are stable, reliable, and general enough, they can be assembled into various long-horizon operations. This is also the core idea of the ManiSkill framework. Around the same time the Sudo team proposed Chain of Thought in 2023, there was a work called Chain of Thought Predictive Control, specifically addressing the chaining of short tasks into long-horizon tasks.
— Han ZhengManipulating objects is two to three orders of magnitude harder than motion control
Han Zheng points out that reinforcement learning in simulation and Sim2Real have already been fully validated in motion control for Boston Dynamics and Unitree's humanoid robots and quadruped dogs, and some of the reinforcement learning engines in NVIDIA Isaac were early contributions from Sudo team members. But robot manipulation of objects is two to three orders of magnitude harder. Because motion control only needs to focus on its own actions, it does not need to understand object shape, material, or know how to move near the object and make adjustments.
— Han ZhengCannot replicate Tesla, must rely on simulation for cold start
Han Zheng explains why robots cannot collect data by selling cars like Tesla. Tesla started selling cars over a decade ago, with millions of drivers driving on roads every day, and data naturally accumulates. But it is very difficult for a robot to sell 5 million units to various offices and homes before it has any functionality, and there would need to be a worker nearby to teleoperate it. Domestic companies might achieve a scale of a hundred or even a thousand units, but that is far from 5 million or 10 million units entering every home. So the cold start must rely on simulation, requiring millions or even tens of millions of full data collected in open environments.
— Han ZhengThe hierarchical structure abandoned by Silicon Valley is about to return to the mainstream
Han Zheng reveals that when Physical Intelligence was founded, while working at Google Robotics, everyone tacitly agreed that after the low-level model had stable manipulation capabilities, the high level would handle environment understanding and task decomposition. But when they started their own companies in 2023 and 2024, everyone chose the end-to-end approach they were good at and familiar with, and no longer mentioned hierarchy. Han Zheng's bold claim is: starting in the second half of this year, the hierarchical structure will return to the mainstream, and may be the final commercialization solution. After recent exchanges between academia and industry in the past few weeks, some people have realized this might be a new direction.
— Han ZhengMost bullish on DeepMind plus Boston Dynamics Atlas
Han Zheng considers the biggest competitor to be DeepMind plus Boston Dynamics' electric Atlas. Last year's interview and documentary with Demis Hassabis showed that Google's bet on physical world AI is the most resolute among all big tech companies. They have already switched their hardware platform from Apptronik to the electric Atlas, and poached core personnel from Boston Dynamics' hardware department to DeepMind. Google also acquired the MuJoCo team, has its own world model Genie 3, and has almost unlimited GPUs and the best talent. Han Zheng believes this is completely on a different order of magnitude from other companies.
— Han ZhengAmazon has many scenarios but integration is difficult
Han Zheng believes Amazon has always been the big company that invests the most in robotics, with AGVs for warehousing and logistics, Kiva, various robotic arms, and humanoid robot companies like Agility and Covariant. It will provide scenarios, but ultimately it will either acquire these companies or do it itself. Covariant also had Amazon as one of its core customers at the time. But integrating multiple robot systems is difficult.
— Han ZhengIn their own words · checked verbatim
2D images and videos do contain 3D information, but it's not quite complete, and not quite accurate.
2D的图片和视频里边 三维的信息有 但是不太全 也不太准
Han Zheng13:21
The success rate for a single pick-and-place of this object is nearly 98%, but if it fails the first time, the robot will use a closed-loop control method to quickly adjust its strategy and achieve 100%.
单次对于这个物体 做抓放的成功率是将近98% 但是如果它第一次失败之后 这个机器人 还会用一个闭环控制的方法 迅速地再去调整它的策略 就可以做到100%
Han Zheng28:05
If a robot is to manipulate objects, the difficulty is two to three orders of magnitude higher.
机器人 如果是要对于物体去做操作 它这个难度要高两到三个数量级
Han Zheng44:51
It's very difficult for a robot—the same model of robot—before it has any functionality, to sell 5 million units to various offices, homes, or factories, and have a worker nearby teleoperating it to do all kinds of operations.
机器人你是很难 同一款机器 在没有任何功能之前 我卖500万台机器人 到各个办公室 家庭 或者是工厂里边去 并且旁边还有一个工人在遥控它 去做各种各样的操作
Han Zheng53:27
We have a bold claim: possibly starting in the second half of this year, the hierarchical structure will return to the mainstream, and the solution everyone chooses to push into commercialization may be the final method.
我们有一个暴论 可能从今年的下半年开始 上下分层的这种结构 会重新再回到主流里边来 而且可能在之后 大家选择的方案推到商业化里边去 可能是最终的一个方法
Han Zheng55:38
He might be the most—because he has both hardware and software—I think this is completely on a different order of magnitude from other companies.
他可能是未来最 因为他硬件也有 软件也有 我觉得这个跟其他公司 完全不是在一个量级上的
Han Zheng1:05:40
Figures
| Sudo R1 zero-shot grasping success rate | 98% single attempt, 100% after closed-loop adjustment | 28:05 |
| Sudo R1 test scale | 1 continuous hour, over 100 objects, 240 pick-and-place actions | 28:05 |
| Number of people tested on-site at ICRA | nearly 1,000 | 32:11 |
| Manipulation is harder than motion control by | 2 to 3 orders of magnitude | 44:51 |
| Unitree 2025 shipment volume | over 5,000 units | 59:08 |
| Tesla Figure shipment volume | about 150 units | 59:08 |
| Skild AI latest valuation | $14 billion to $15 billion | 1:00:34 |
| Skild AI funding amount | $1.4 billion | 1:00:34 |
| Physical Intelligence valuation | $5.6 billion | 1:00:34 |
| Hyundai Group's stake in Boston Dynamics | SoftBank retains 10% to 20% | 1:08:58 |
Glossary
- Sim2Real
- Training robots in virtual simulation environments and then transferring the learned capabilities to the real physical world.
- Zero-Shot
- A model directly completes a task on objects or environments it has never seen during training, without additional fine-tuning.
- VLA
- An end-to-end model primarily based on imitating human actions, unifying vision, language, and action modeling.
- URDF
- A standardized file format for describing a robot's geometry and dynamic constraints.
- teleops
- Humans remotely controlling a robot via a remote control device to collect manipulation data.
How to listen
Founders, investors, and robotics engineers focused on the divergence in embodied AI technical routes, especially those who want to understand Sim2Real feasibility, data cold-start strategies, and the Silicon Valley competitive landscape.
The discussion from 1:09:01 to 1:10:58 about Amazon and vertical scenario commercialization is low in information density.