“The day I hear a song of the quality of the Beatles, I will say: ‘Okay, we are done’. And I’ve
never heard anything like that. Never.”
Dr François Pachet is an AI researcher and musician, and one of the most influential figures in AI and music. His innovative contributions have defined the field over the past decades through creative systems such as the Continuator, and Flow Machines, among others. After leading the Spotify Creator Technology Research Lab and the Sony Computer Science Lab, he went on to create his own companies: Imagine All The People and Ynosound.
In the context of IJCAI2025, he spoke about what deep learning changed, and what still remains wide open. He explains why tools like Suno and Udio—ChatGPT-like platforms for music generation—can produce astonishing results that still feel unsatisfying; why the next step for music generation requires combining sampling with search; and why the most important problems in artistic domains are, by nature, ill-defined—because there is no loss function to determine what is “good”. Above all, he defends the importance of researcher autonomy: work on the questions that genuinely fascinate you, even when they fall outside prevailing trends—perhaps especially then.
Liliane-Caroline Demers: Hello, Dr Pachet. Thank you for joining me for this interview. Could you begin by telling us when was your first IJCAI and a memory related to it?
François Pachet: I think the first IJCAI I attended was in Montreal in ‘95. I was there for a couple of workshops, one of them was about music and AI, and the other one I think was on advisor systems, something like that. And I remember there was a French colleague who was there also, at the time he was doing his PhD. And there was this researcher called Herbert Simon, who is a Nobel Prize pioneer of AI. I remember having chatted a little bit with this French guy who was very bold, and he just went up to Simon, said “Hey,” and he started a conversation with him. And I was very impressed by the fact that you could meet those kinds of guys informally in a corridor or something at this conference.
That’s my oldest memory of IJCAI. Oh, there are lots of other memories, but Simon was one of the big pioneers of the very old AI. But a very interesting guy who co-built a system called GPS, General Problem Solver, in the fifties. It was something very bold for that time.
Liliane-Caroline: Music clearly holds a central place in your life. But listening, performing, or even composing music is one thing; deciding to generate it is another. What originally drew you to apply AI to music?
François: Yes, there is something I want to say about that.
Nowadays, AI is mostly seen as a way to solve problems—and I would even say, at an industrial scale. ChatGPT, diffusion models, all these things are about providing new services or new applications to people. It makes sense and it’s incredible.
But during the eighties, the primary goal of AI was not that at all—maybe only because AI did not work as well as it does today. But so many worked on AI then because it was a tool. A tool to try to understand things dealing with intelligence, the brain, human behavior, humans in general. A tool for investigation, rather than a way to make money.
So, that’s what brought me to try to solve musical problems. Not necessarily to solve them, but because I wanted to understand more about the problems themselves. For instance, how do people improvise in jazz? How do people compose? I was—and still am, actually—fascinated by some composers, classical and pop, etc. How do they invent all these melodies or songs? One way to understand it is to try to do it yourself—which is very hard. Another way is to try to build sequences that try to do that. And when you do this, you understand more. Though not everything. Honestly, today I understand a little bit more about it, but the problem is not completely solved.
So, AI was a tool for investigation for problems that I found fascinating, more than trying to solve them, as we have today.
Liliane-Caroline: And how has that goal evolved over the course of your career?
François: In the last ten to fifteen years, of course, deep learning has changed everything, and its importance is huge. But concerning the problem of how people compose songs, the mystery remains totally open.
We know more now about how to imitate a style than we did 20 years ago. We know what it is, how you differentiate between styles, how to separate form from texture, and what is style by opposition to the rest. We know how to combine different styles.
But when it comes to creating something new—not from scratch, but from things you have learned before? It’s really not clear, and in some sense, the problem remains open.
It’s becoming very interesting because you have all these tools now to deal with this problem. For instance, transformers on the one hand, and diffusion models on the other. They are very good at generating things, including audio, and music. But they still don’t generate what I would like to generate—music with the kind of surprises, or catchy harmonies you find in songs composed by people.
So, while technology has evolved incredibly, the problem is still quite open. How do you create something that’s interesting?
I have used some of these tools, like Suno and Udio, and I’m still not satisfied. There is still a lot to do.
This is interesting because, as you well know, AI is often criticized for solving many problems that humans solve—often better than humans in many areas. And that’s true, it’s a problem for society. However, in the case of creation, like music, it’s still not completely clear that AI is able to do that.
I don’t think it is—which is good news, because there is still a lot to understand about not only how to generate music, but how do we, as humans, listen to music? What do we like in music? Honestly, no one knows. Why do you like this song and not another? There are theories to explain it, but no definitive rule.
Liliane-Caroline: You mentioned platforms like Suno and Udio. While their results can be impressive, they can sometimes feel directionless. In your opinion, what is fundamentally missing in their generative process?
François: There are some videos on the web where I start by playing a few songs in the Greek style generated by Suno, and it’s very impressive.
But my position is very simple. These systems, they are called end-to-end.
You start by a prompt, which is a text, and you end up with a completely fully-fledged mixed multi-track, even master. So, you have the complete end product, from end to end.
And obviously what’s missing is the way humans create something—whether it is a music, a book or a house. There are always two phases in creation. There is the design, and the execution. For instance, if you are making a house as an architect, you first draw the plans for the house, and then, with your pencil and eraser, you do backtracks. You do trial and error, and then it’s done. Then you give it to a contractor, and they build a house according to the plan.
When you compose music, it’s the same. Like McCartney composing Yesterday, he woke up and he had a song in his head, he played it on the guitar for a while, finished composing it, and then he went to the studio and they recorded it.
So, in music, you have the composition and the production. It has always been like that. And it’s like that in all creative domains.
Suno and Udio are systems that get rid of the composition stage. They go directly to the production. And it’s very impressive, credible, but there is no plan. That’s why I think that there is a lot of room for research to add that stage. I’m not saying that we should imitate humans—because we don’t need to—but what’s lacking in the process is a plan that has been honed. Like what we call search in AI. When you search in a space, you try something, realize it didn’t work, you backtrack, and you take another path. And all of this process has disappeared—replaced by sampling. Or you train a model and the sampling is very simple. You just draw dice with some probabilities.
I am convinced that if you want to create art, you have to do some search in some space because, it’s not going to be a one click thing. And so, the limitation of today’s systems is that there is no search. We are building houses without plans, so it cannot work. It’s like starting to paint the walls before you have finished the rest of the house.
I hope that the next generation—and I’m working on this myself—we create solutions that have both search and the production as we have today.
I understand why people do this, it’s because technology allows it to happen like this. So, it does, and it has to. So, I’m not criticizing, but I think that the next generation has to combine these systems.
In his book Thinking, Fast and Slow, Daniel Kahneman talks about system one and system two. Basically, the brain has two modes. System one, which is, you go very fast, you have a very large knowledge of the world, and so you are very good, but you don’t really think. And then when you have a hard problem, you’re using system two: you slow down to solve the problem.
And so today you have system one, which is ChatGPT, and system two, which is, let’s say solvers, which can solve very hard problems. And no one is really able to do both in interesting ways. And that’s a big research area. I think music is exactly the main problem where we are not able to combine search and sampling in a way that’s interesting.
Liliane-Caroline: What would that kind of search look like in musical composition?
François: Search works like this, for instance, you start with something like ‘Hey Jude, hey Jude.’ It’s a bit boring. So, you introduce something surprising, like a G7 chord, and you are now in an unstable place. So, what do you do? You have created this sequence, and now you have to fix it. And the ‘fixing’ is the search, because you have to look at many different solutions, and you possibly have to backtrack, because maybe it wasn’t a good idea to do this big interval. So, this becomes a problem to solve with a search.
I call it search, in the sense that you are searching in a huge space, trying to find a good solution. Sampling never does this. Of course, models like ChatGPT give you the impression that they are thinking, but in fact it’s not true, what they do is more like chain of thought, they see patterns of reasoning. So, they never really search, they don’t have any search algorithm.
So, to me the future lies there. There are lots of things to do, but especially bringing end-to-end systems to be able to do this would be a huge improvement, I think.
But is it for tomorrow, next year, or never? I don’t know.
Liliane-Caroline: One of the challenges in music generation is ensuring long-term structure. In your opinion, what are the most promising ways of capturing meaningful repetition, or long-term dependencies in generative models?
François: This idea of long-term structure—or long-term correlations in statistical physics, and in large text—is still a bit of a mystery. I’m not very sure that it is that important in music.
Pop songs, for instance, are very short. And today’s pop songs are even worse, they’re very repetitive—it’s always the same thing. So, for me the problem is not so much long-term structure, it’s more about what we were talking about previously: the fact that there is no search, models spit out something, and that’s it.
For instance, the Beatles is an interesting example because everything is there. There are some songs with, like, four bars—and what is going on in terms of the interplay between the melody and harmony is so smart. You always have these two lines: the melody, and the harmony behind it. So, you have two sequences, each with their own logic, plus, of course, the interaction between the harmony and melody. And I think this is not yet well modeled.
Very often, you have the melody that’s going in one direction and then the harmony going in another, but it still fits. And that’s what creates a result that is really pleasing. The Beatles do this all the time. And that mechanism of dealing with two sequences is done extremely poorly today.
If you look at the transformer, for instance, it’s very hard to consider this. A song has one sequence. You can do it in ten, twenty different ways—but it’s never really satisfying because you don’t really capture the interesting interplay that happens between subsequences of melody and subsequences of harmony. But modelling two sequences at the same time is a very hard problem. People try to do this by doing a linearization of the music into one sequence of tokens that kind of represents everything. But to me, this is the weak point. This is where it doesn’t work because it doesn’t allow the system to create a truly interesting interface between melody and harmony.
Diffusion models are another story. They are very interesting, but totally uncontrollable. The idea that you start by a totally random thing, and then you progressively end up somewhere—it has an incredible domain, and the results are incredible. The day I hear something of the quality of the Beatles, I will say: ‘okay, we are done’. But I’ve never heard anything like that. Never.
And I don’t think it’s a problem of structure. I know many people are focusing on that because it’s a relatively well-defined problem. You can test, you can evaluate it. But I would say I’m not convinced that long-term structure is the solution, at least not for pop songs, and definitely not for jazz, since jazz is about very fast stuff occurring on a very small-time scale.
Liliane-Caroline: Do you mean that since pop songs are already repetitive, long-term structure may not be where the most interesting aspects of composition lie?
François: Yes, to me, the interesting part of interesting songs is never the structure. Of course, there are some songs where structure is very important. But what interests me most in composition are strange yet satisfying surprises—Paul McCartney especially is a genius at that.
My point is that unexpected and smart melodies or notes that happen within a song is the result of some very hard search in some space. Like what McCartney does typically is having something surprising or dissonant happen, followed by something that is going to resolve this in very satisfying ways.
We don’t model this at all. Sequence models, diffusion models, they don’t have any idea if something is surprising.
I think that is a crucial thing to achieve when you compose: to create those kinds of surprises, which you resolve or not. And the current systems which compose don’t have any explicit representation of these things.
Liliane-Caroline: So, you’re saying that long-term structure may not be the right angle for music generation. Still, music typically feels coherent and directed because it exhibits some form of structure, while creativity provides engagement. When you design a generative system, how do you think these two aspects should be balanced?
François: I’m doing experiments. Before composition, I was interested in freeform improvisation. I did a system called Ator—which I reimplemented recently actually because I still don’t think it’s doing what I want it to.
The way I work is: I have an idea, I implement it, I play with it. I have some people play with it, too. Then I listen and try to make sense of what it doesn’t do.
It’s not a very structured process. It’s like having fun with systems, really. I had the chance to be able to do this and being paid for it. I’m not studying like a physicist, with well defined data that I want to replicate or understand. It’s more like a quest—trying to discover what to do.
Over time, I’ve reduced my ambitions a lot. Today, I would like to compose four bars—only four bars. A monophonic melody, with simple rhythms, and two chords per bar that have some interesting properties.
Already, this alone is very hard to achieve. Because satisfying musical rules is easy. The hard part is: how do you make something that’s interesting?
So, I’m not answering your question because I don’t know how to balance structure and engagement. What I’m trying to do is to accomplish this goal.
And I’ve been criticized in the past while directing research teams, actually, and especially at Spotify, because my goals were not clear enough. That is actually a problem in AI, I think, because everyone—especially young people—they want everything to be clear. And sure, sometimes it’s a good thing because it’s a very strong engine. But my goal is not very well defined. I want to generate something with my system that I find interesting.
Okay—if you want to have a career in AI, maybe it’s not a very good goal, but I think it’s more interesting to work on how to imitate a style, or how to replicate a property, than trying to solve an existing, well-defined problem, because by doing that, you understand more about music.
I want to mention someone whom I admired a lot: Douglas Lenat. This guy was a genius. He received the IJCAI Computers and Thought Award in the seventies. He created this project called Cyc—it’s an encyclopedia of common sense. 30 years later, it’s still going on. He started with his PhD at MIT at what is now the media lab with Marvin Minsky, and he built a completely crazy, very strange program called AM, Automatic Mathematician. He wrote this program in Lisp, a very strange program, which had no input, only output. So, you can run a problem and just look at the screen. You cannot interact or do anything. And the program is just trying to invent mathematical conjectures and find new theorems by itself like a mathematician. His program was very impressive because he was able to reinvent a lot of theorems in elementary mathematics, like set theory, and prime numbers.
And that problem is still something that no one is doing today. Can you imagine a PhD student doing a program with no input, only output? Doesn’t make any sense. But the guy didn’t care. He did this because he thought it was interesting, and he became one of the most famous AI researchers. And Minsky also used to do that kind of stuff.
So I think today, we should have more of these people, obsessed with an idea, doing whatever they want, even if it’s stupid. And now, all the papers are all at the same caliber, the same structure, database, architecture, evaluation, and, okay. So it’s important I think to not only solve existing problems, but to invent new problems. I think that’s even more powerful.
Liliane-Caroline: I would like to discuss the evaluation of results in the context of music generation, which is notoriously difficult. We can survey listeners, we can try to analyze for patterns and repetitions, but none of these techniques truly capture creativity. So, I was wondering, in your opinion, what would be a meaningful way to evaluate success in generated music?
François: The first thing to observe is that the quality of a song is not related to its popularity. So if you try to evaluate results by releasing songs and counting the streams you get, it’s not a good measure because this depends on too many factors that does not have much to do with intrinsic quality.
I can tell you an anecdote about when we did this album called Hello World. We worked with various musicians, and one of them was Stromae—who is a big name in Europe. We were working on various things, but we knew that at the end we had to make a real song, meaning that it had to be credited with musicians, and composers, all sorts of people. And while doing this, it became obvious that Stromae, or any good musician, will never put his name on the song if he does not think that it is good. And that, to me, is the best evaluation you can have.
Personally, I don’t care about my musical reputation so I’m completely okay with putting my name on songs which I don’t think are very good. But if you are a professional musician with a reputation, you will be very careful. You want to have your name on that song because you think it’s so good. So, that’s a very important criteria: how willing are the composers to be credited for the composition, the orchestration of a song? Because they are putting their reputation on the line.
The second criteria for evaluation is the producer, or at least the distributor. When you release music, someone pays for it. There are different levels of distribution, from high-level to low-cost ones. If you have a high-level distributor, it means that they think it is very good because it’ll be successful.
And when you compose, especially pop music, you usually are not alone, but with two, three, or more people in the studio. And when a song is good—even if you don’t know exactly why—everyone agrees right away.
In science you can’t use this kind of information, but I think it’s more important than surveying a panel of people. It’s hard to pinpoint the intrinsic qualities of music like this because people are so different, with different experiences.
So, to me, that is the evaluation. There is this paper that I quote each time I do a talk, which everyone should read, called ‘Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market’ by Salganik, Dodds, and Watts. They made an incredible experiment where they showed that the intrinsic qualities of songs is very poorly related to popularity. And, in fact, popularity data is chaotic.
Basically, if you go back in time, you go back eight times—which they did in some way—you have eight times different results. So, it means that you cannot count on the collective subjective judgement. That’s why I gave up doing that kind of stuff, because it’s a little arbitrary.
Liliane-Caroline: What do you think people misunderstand the most about AI and music—or AI and creativity—even within the research community?
François: That’s a big question. First, the word ‘creativity’ is very large, and the idea of what counts as creative debatable.
But my definition of the difference between an artistic artifact—like a piece of music, a painting, a story—and something not creative is very simple. In artistic artifacts, you don’t have an evaluation function. You don’t have a loss function. You can’t say, this is good, and this isn’t.
If you perform an automatic analysis of the brain or tumors, you know exactly that this one is a tumor, this one is not. So, you have a groundtruth.
When you don’t have a groundtruth, you are in a domain where you don’t have any evaluation function. And this is a problem, because you still use these algorithms trying to minimize loss, and stuff like that. But we are not able to exactly define a loss function in this domain.
Margaret Boden wrote books to define creativity. But I think it’s a word we should avoid. What I come back to instead is: do I find this interesting? This is well defined, at least for me, because I’m able to answer the question. And among my expert musician friends, usually we agree on what is interesting. So, this is a way to define the problem. But it’s a strange definition of ‘interestingness’ because it’s based on the opinion of a very precise group of people—it’s not universal.
I think that AI has solved mostly all well-defined problems. If the problem is well defined and you have data, any kind of architecture can solve it better than people. AI should focus on situations where problems are badly defined because the rest is honestly solved.
There are a lot of scientific or engineering contexts where AI used to be about solving ill-defined problems. For instance, when researchers worked on automatic translation, they didn’t know precisely what constitutes language. It is very difficult to define the English, or French, language. What determines a good translation? But now it has become solvable because of the critical mass of data we have. So, we can evaluate results.
In the artistic domain, we should invent new types of ill-defined problems. It’s not random, but there is no category for that kind of problem. This is what I think AI will focus on in the future: problems for which we don’t have a good definition, or an evaluation function, but for which we have some competency.
Liliane-Caroline: But there’s a lot of public backlash regarding AI in the arts, and how it’s replacing human artists. How do you react to people who think research efforts should be put elsewhere?
François: I try to ignore this. I think as a researcher it’s important to work on what genuinely interests you. I heard a very interesting interview by Geoffrey Hinton. He said, if you are a young researcher, do exactly what you want, don’t read too many papers. And if someone tells you, ‘what you’re trying to do is stupid,’ do it anyway. If they tell you ‘it has already been done,’ do it anyway. And the more people you have telling you that your idea is not good, the more you should do it.
I think it’s completely right. Because even if you fail, you will have learned so much. You learn more by doing than by trying to chase problems that your supervisor or other people, told you to do, that you don’t care about.
François Pachet concluded with a reflection on how working on music generation with AI has changed the way he listens to music:
I listen to songs differently now—especially ones I already know. I try to find moments where I can almost see a system in action. I have an idea of a system; then, when I listen to music, I ask myself: would the system I have in mind be able to do this? And sometimes I realize—oh, no, it couldn’t. It’s the way, I think, a researcher would listen to music.
About François Pachet
François Pachet got his Ph.D. and Habilitation from Université Pierre et Marie Curie (UPMC), after an engineer’s degree from Ecole des Ponts et Chaussées. He has been assistant professor in artificial intelligence at UPMC until 1997 when he joined SONY Computer Science Laboratory Paris to conduct research on interactive music listening, composition and performance. He became its director in 2015. In 2017, he joined Spotify to create the Content Technology Research Lab. He left in 2023 to start Ynosound, specialized in AI-assisted music composition and Imagine All The People, dedicated to AI-assisted decision making.
During his research career, he developed several award winning technologies such as constraint-based spatialisation, intelligent music scheduling using metadata and systems: MusicSpace, Continuator for interactive music improvisation, and Flow Composer which paved the way for AI-based score composition.
François Pachet has published intensively in artificial intelligence and computer music. He was elected EurAI Fellow in 2014 and doctor Honoris Causa of the University of Pernambuco (Brazil) in 2017.
He was Principal Investigator of the Flow Machines ERC-funded project, which produced (with the musician SKYGGE) Daddy’s Car, a song in the style of the Beatles, and Hello World, the first mainstream music album composed with AI.
He is also a musician and has published two music albums (jazz and pop) as composer and performer, as well as an augmented book about the ontogenesis of a musical ear, Histoire d’une Oreille.