In this instalment of AI pioneers, we speak to award-winning roboticist, filmmaker, and artist, Ken Goldberg. We discuss the culture clash within robotics, what art and science have to learn from each other, and how AI will shape the future of art.
Welcome, Ken Goldberg. Thank you very much for coming onto AI Pioneers. You’ve been a leader in the field of robotics and automation and just elected into the US National Academy of Engineering. You’re also an artist in your own right. First off, I really enjoyed your plenary at ICRA, where you proposed a new direction in robotics and highlighted the tension between what you call good old-fashioned engineering and vision language models. Can you tell me a bit more about that?
Sure! I’ve been thinking a lot about a gap between the two communities within robotics: the traditional roboticists who have been working for 50-plus years on control methods, model-based methods, and the new wave – the younger generation that has embraced learning and, in many cases, rejects the older methods.
I perceive an almost dogmatic difference between the two. Could there be an opportunity for synergy in putting them together? I’ve been struggling with that for a long time – how can we get the best of both worlds?
The advantages of the “good old-fashioned engineering” model-based methods are primarily interpretability, because you get to understand what’s going on. And you also have generalizability. You know that if something works in one part of the state space, you can transform it to a different part of the state space and it’ll work there as well. You don’t need to get more data in the new environment – the new environment is just like the old environment, but transformed. The same model, using the same mathematics, applies there as well.
That gives a real advantage, because you get reliability and you don’t have to keep collecting data. This is coming to a head, because model-free methods have now matured to a point where there’s a lot of expectation that it’s ready for deployment into real environments.
Another division is between the generalists and the specialists. The specialists are the old school, trying to solve one problem at a time and then develop an understanding that will grow over time and accumulate into more general systems. But the generalists go the other way. They go top-down, assuming a fully general system training with as much diversity as possible. That is problematic because full generality is enormously dimensional – so many different things can happen in any environment. The generalists look at the example of ChatGPT and language models and say, well, those are fully general.
The analogy with robotics is to start with data collected from everywhere. I think that’s a problem, because we have a massive data gap. The amount of data we have so far for robotics is only on the order of 10 to 100 years. Compare that to the amount of data on the internet available for training large models, which is on the order of 100,000 years. Robotics has nowhere near the internet-scale data that language models have.
I’ve been looking for a way to bring these two camps together, because industry is getting impatient waiting for these robots to generate revenue. That requires systems that perform extremely fast and reliably. The throughput is the combination of a success rate and cycle time, and you have to get both of those to a pretty high level. And then every little increment is very difficult, because as you get closer and closer to 100% success rate, every improvement gets harder and harder to achieve. The same applies for time – you can squeeze out some time, but then squeezing more time gets harder and harder. These are nonlinear scales, and it’s increasingly difficult to push up throughput.
Why do you think there’s a reluctance of the two sides to work together on this – is it because of the way they were trained and the way they’ve always done things?
That’s a great point. There is a wonderful essay by C. P. Snow from Great Britain, called The Two Cultures. It’s short, maybe 20 pages, from an address he gave at Cambridge in 1959. He said that there are two cultures at the university: there’s the scientists and the humanists. They coexist but they don’t talk to each other. I think that’s still true to a large degree on many campuses.
We have something similar happening within robotics: I see two cultures. The older generation has spent many years developing classical model-based methods. The young generation tends to be very enamored with the new model-free methods.
This is very common in history, where you have a generational clash. It’s dogmatic to say, “let’s reject that old stuff.” And there’s dogma on the other side that says, “hey, you guys don’t understand history, you don’t understand the realities of robotics, so it’s not going to work.” But they both have huge value. The key is how do you bring them together? When I perceive two very different subcommunities like that, I start thinking about why they’re different, how they’re different, and how they can come together.
That’s where the really interesting things tend to lie, right? Because the intersection is where the new material is.
Yes, I’ve done this to some degree with my career in terms of art and science – trying to understand both and to bridge the gap. In this new context, I’ve been thinking about this gap and talking about it for a few years, and then something new came along in the late fall of 2025. As usual, it was my students who brought it to my attention.
Max Fu, who was a PhD in my lab, had been revisiting this idea of code as policy. That is, where you have a large language model, which is very good at writing code, write code for robots. That idea’s been around for a while. Actually, it was another one of my students who wrote the first paper on this, Jacky Liang. He had finished in my lab and was at Google. He led an effort to do this back in 2022, four or five years ago. But at that time the large models weren’t that great at programming, so it didn’t really take off.
Max and his colleagues at NVIDIA, Jim Fan, Yuke Zhu, and others noticed that these new models were getting really good. Simultaneously, the OpenClaw breakthrough happened. Suddenly, agents collaborating with other agents really took off. This opened up new terrain – it was exactly the missing link I’d been looking for. Because large foundation models were trained on lots and lots of code, they’re very good at coding.
So they can write procedures based on good old fashioned engineering and model-based methods and call in trained models for various specific subtasks, like segmenting an image. Segment Anything 3D is an excellent learned model.
I want to emphasize I’m not against learned models in any way – I’m convinced they will take over the world at some point. But useful systems could take many years. Having them used in specific sub-areas – coming back to the specialist/generalist division – is a really effective way of getting them to be useful. Using agents to combine them is extremely exciting for me, because I was struggling to see how these two worlds could work together. I didn’t have an answer, and then this dropped in – you can start combining them with large language models for code, and you get systems that have elements of both.
You’re essentially harnessing – and by the way, I appreciate “harness” for this. That’s what’s needed to make this work, a harness for the large language models. It’s the set of prompts and constraints you give to the large language model to get it to do what you want. “Harness” is exactly the right term, because agents are like wild horses – very powerful and capable, but they don’t always follow the rules. If you ask them to do something, they’ll do it mostly, but then they’ll occasionally bolt and do something really different and unexpected. This is where hallucinations come in, but also in general, if you give them too many rules and constraints, they’ll stop following them.
But the complexity of the coding increased. If you want to write something as simple as “put these grocery items into a basket,” that turns out to require quite a bit of code to get it to work. And that was getting to the point where it was causing the large models to get confused and start thrashing – they’d fix something and something else would break. Sometimes the context window isn’t big enough to manage all that.
When we saw that problem, one of my other students, Eric Chen, suggested a graphical structure so we can have each agent focus on one node in the graph. That way each agent could be focused on one node to try to improve the performance of this node. For example, the node which might be “locate the object” – a perception node. Another one could be “move the robot to this location” – a motion planning node. And then another one for “grasp this object” – a grasp planning node. As such, these nodes can independently be fine-tuned. That’s the graph-as-policy idea I presented at ICRA.
That’s a nice way of both abstracting the problem and harnessing the agents. Bringing back in the specialists and generalists idea – could you have several specialists on each node, and then a generalist agent overseeing all the nodes?
Yes, you can think of it that way. We use the term “orchestrator” – an orchestrator is trying to manage everything, but the complexity gets balanced with a hierarchical structure.
You tested this policy on some leading LLMs, and they performed differently to each other. Why do you think Gemini did better than, say, Claude or ChatGPT?
That’s a great question. First of all, it’s changed since we wrote the paper in May, and it’s changing all the time as new models and versions of models are coming out. The difference is what they’re trained on. I think Gemini has been emphasizing 3D perception in its models for several years now. I don’t think that’s been a priority for Claude or ChatGPT, but now they are taking an interest in it. They’re increasing their training set to include more examples of this.
Another postdoc – Shuangyu Xie – has been harvesting data from robotics paper for understanding the three-dimensional world – or from a robotics perspective, the six-dimensional world, with position and orientation. You have coordinate frames moving around, and you need to understand how they relate to each other. In the early days of ChatGPT, you could say “put block A on top of block B and then put block C,” and it would do it in the wrong order. But now it’s very, very good at this kind of spatial understanding. Gemini, I think, has had a head start in that area, because Google has fed it with a lot of images and presumably 3D models.
Given that we’re going to have an increasing need for more data to train robotics models, do you think industrial robots will become very important in terms of providing training data?
I think they can be a great source of data – if you get a robot working in some production environment, it collects data as it works. That’s very reliable data, because it’s working in a real environment with real variations of real products – you can’t simulate that and you can’t get that in a lab.
But getting a system out there in the real world is a chicken-and-egg problem. You have to have some level of performance that’s good enough for people to want to pay for it. That’s again where agentic robotics comes in, because it can provide a springboard to getting there – it gets you up to a point, and then maybe there’s even some human fine-tuning involved. The agentic system could be combined with human intuition and feedback, which is something I’ve been thinking about too.
The company I co-founded, Ambi Robotics, is using large models with human fine-tuning. We have 20 to 30 years of elapsed data now, from many machines working around the clock. Now that we have that real, high-quality data, we can train on that data and get better results.
But Ambi is not trying to solve the fully general problem – in our case, we’re trying to solve the package-sorting problem. One of the things we trained our original system on was 3D CAD models that we found on the internet. What we encounter in reality is lots of bags. They’re really complicated to manage, because they deform in ways that are very hard to model. We didn’t train on that, but now we have the data and our system has learned how to pick up bags very effectively with suction cups. Agentic robotics can bootstrap a robotic system to get it out into production, and then it can collect data and get better and better over time.
I wanted to ask you a little bit about your art career. I think it’s valuable for scientists to have an interest in art, because it’s a mindset where you question the meaning of things. With robots, I think it’s important to consider what it means to have robots in society, and keep this in mind when we should design them. I sometimes get the sense that people in the humanities aren’t so aware of the progress being made in robotics, so we don’t have an influx of their perspectives just yet.
Actually, there are people in the humanities who’ve been thinking about robots for a long time. The concept of the “uncanny valley”. It turns out to have its roots in Sigmund Freud’s essay from 1919, The Uncanny. He defines this idea and links it to robots, actually automatons, because it was published one year before the word “robot” was coined. I believe Freud’s essay influenced the term “robot.”
The uncanny is a huge subject in the humanities, and people have analyzed a lot of literature in the context of Freud’s theory. The uncanny valley is where it comes back to robots. This has also been a very big topic in films and books and the humanities, stories that go back to the Greek myth of Pygmalion, to Frankenstein, and right up to the present day with Ex Machina and all these films on this question of how humans react to robots. I think this connects to theories about “the other” – how people react to something in an unfamiliar culture. This connects to contemporary theory about colonialism and orientalism.
These are major themes in the humanities, in art and literature. It seems the way people think about robots is like immigrants. There’s someone foreign that’s coming for their jobs, and people have a very fearful, often negative reaction to it without understanding it. Then they tend to exaggerate its capabilities – that’s actually one of the things about orientalism, that people also glamorize things they don’t understand. They think these things can do all kinds of amazing things, and they project their fears, but also project their fantasies.
You see this in how people are thinking about robots today. People are inspired by science fiction, and by CEOs – like Elon Musk. He’s saying we’re going to have humanoid robots by the end of the year, and many people accept that and believe it’s going to happen, including many investors. This is something I worry about, because I don’t think it’s going to happen that soon – it’s going to be a while. I worry about people being disillusioned, disappointed, and the backlash that could come from that.
Generating unrealistic expectations and fear in order to garner more investment is not a very responsible way of introducing a new technology to the public – we’ve seen the same thing happen with AI and AGI. Then these companies end up wielding undue influence, because they’ve scared the politicians.
Yes, because politicians, the press, and companies – they all want to get attention. One way to do that is to create emotional scenarios, and they know how to use that to get attention. It’s very human, and I’m not blaming anyone for doing it, but it leads to exaggeration and a herd mentality. It’s one area I really do know.
And we’ve built systems that tend to reward these tendencies of people to exaggerate and make big claims which are not necessarily justified.
Yes, there’s a lot of exaggeration going on. On the one hand, I love it, because it’s fuelled a mass excitement about robots. I’ve never seen robotics more energized in my whole life. It’s bringing in the best minds – great students are gravitating toward robotics right now. It’s an incredible period for robotics – the technology is evolving very quickly and there’s lots of industry interest.
And who knows, someone could make a breakthrough and suddenly everything starts working – that could happen. Everything I’m saying doesn’t preclude a magical moment when somebody solves all the challenges. But we can’t bet on that. We need to have a backup plan.
Again, I don’t think robots will steal jobs – they’ll shift jobs, and people will be able to spend more time interacting with other people. This comes back to what we were talking about with the two cultures. I think a lot of technologists don’t always appreciate the value of human contact. It’s very subtle, but everything in the humanities points to the value of human-to-human interaction and how important that is, and that’s not going to go away. No matter how good robots get, you’ll never have a robot that’s going to be able to understand you and understand your stress. So you want a person. And there are plenty of jobs for people: we don’t have enough teachers, nurses, caretakers, people at the store who are willing to sit down and talk to you. I’m a believer that there’s no shortage of jobs, and I don’t see it as a threat.
I like that vision, because it would be great to have robots displace the hard labor and menial work – say, a care robot that can do heavy lifting and the tasks that exhaust human carers, so the human carer has energy to give emotionally.
Exactly. When I get older, if I need someone to dress me in the morning, I’ll take the robot over a stranger.
How does this relate to your artwork?
My artwork is trying to connect with that very tangible, material human world: the natural world. I want to reconnect people to that natural world in the context of robots. That’s why I created the Telegarden in 1995. It was a real garden with a robot in the middle of it, and people could remotely operate the robot to plant seeds and reconnect with nature.
All my work is at the intersection between the digital world and the natural world. The key with art is you don’t want to sound dogmatic and patronizing. If you want to send a message or convey something, you have to do it artistically, in a very indirect way that disarms people and gets them thinking in new ways, without telling them something outright.
As a means for people to reflect.
Yes, you want to get people to connect emotionally, not in the way a lecture connects with people’s minds. That’s one thing. Artwork is a different way of communicating.
Did you always feel drawn to make art about the systems you were building, or to make art generally?
As a kid I built a lot of model ships, cars, and rockets. I think that evolved into an interest in painting and sculpture and art more generally, such as visual art, dance, sculpture, installations. As an engineer, it’s very different from writing a paper – I like it because it challenges me, gives me different constraints, and stretches my mind in different directions.
And I suppose it provides a different sort of space for you to play with the questions that emerge from the engineering side of things as well.
Absolutely, sometimes artwork leads to interesting engineering questions. The Telegarden that I mentioned earlier was also an engineering project, and it led to almost three decades of research on what’s now called cloud robotics.
After the Telegarden, I started thinking about other applications for controlling devices over the internet, and that led to what we called networked robotics. We started an IEEE technical committee on networked robotics, wrote a book about that, and developed a number of research projects. Then, when the name “cloud robotics” came out in 2015, we adopted that and started writing papers about it, and that’s grown into our work in fog robotics, as a way of connecting Robot Operating System (ROS) nodes to the cloud for computation.
So it works both ways – art can influence science and engineering, and new ideas in engineering can often be applied to create interesting artworks. And maybe this comes back to where we started, talking about the two cultures at a university. The arts and the sciences often don’t communicate, but they actually have a lot to say to each other. If they would communicate, they could inspire each other in interesting ways.
How do you think AI will impact the future of art? While there is a lot of concern about AI replacing artists, humans seem infinitely adaptable. We’ve seen this with every major technological change. For example, the camera may have replaced painting as the dominant medium, but then came film and everything else as a result of the camera, which nobody could have anticipated.
Yes! Your example is exactly right – the camera was a mechanical piece of technology that had a profound effect on art. Art stopped being just about realism and became much more abstract, then conceptual. I think AI is another technology that’s changing art, in ways that are very subtle.
And it seems like we’re in the AI-slop period right now, but perhaps that’s what happens with a new technology or medium. We’re experimenting, working out what works and what doesn’t.
Throughout history we’ve seen many big technologies – writing, agriculture, electricity, computers – have massive effects on humans. They’ve all been shocking and amazing when they first happened, but basically humans are still humans. We haven’t changed that much in the last 100,000 years. So we’re going to be the same humans we were, and I think AI is just the newest thing, but it’s not going to change us in our fundamental ways. It’s going to change the world – there’s still going to be jobs and people are still going to want to hang out with other people.
My final question is: which upcoming things are you excited about?
I’m really excited about agentic robotics. All my students are working on it, and we’re having so much fun and collaborating with NVIDIA, Bosch, Siemens, and Ambi Robotics. To me, agentic robotics is the biggest thing since the internet. Many new research questions have just opened up, and I’m super excited to work on them.
About Ken Goldberg
Ken Goldberg is William S. Floyd Distinguished Professor of Engineering at UC Berkeley and an award-winning roboticist, filmmaker, artist and popular public speaker on AI and robotics. At UC Berkeley, Ken trains the next generation of researchers and entrepreneurs. He has published over 450 papers, 3 books, and holds 10 US Patents. Ken’s artwork has been featured in 70 art exhibits including the Whitney Biennial. He is a pioneer in technology and artistic visual expression, bridging the “two cultures” of art and science. With unique skills in communication and creative problem solving, invention, and thinking on the edge, Ken has presented over 600 invited lectures at events around the world.