Experimental Composer Holly Herndon Built An Ai Voice Clone That Anyone Can Use

This musician built an AI clone of her sound truthful anyone tin singing arsenic her

Experimental composer Holly Herndon says this exertion isn’t present to switch artists—and that nan early of productivity belongs to corporate intelligence

By Deni Ellis Béchard edited by Eric Sullivan

Holly Herndon astatine nan Serpentine North Gallery successful London, October 2024.

Matthew Chattle/Future Publishing via Getty Images

Holly Herndon hears nan early of music successful data. Herndon came to physics euphony aft singing successful religion and choirs successful East Tennessee. She earned a master’s grade astatine Mills College and a doctorate astatine Stanford University’s Center for Computer Research successful Music and Acoustics.

When she began experimenting pinch machine learning successful 2015, nan outputs sounded “scratchy,” but she recalls seeing “the gem successful nan rough.” Today those experiments person evolved into custom models that let anyone to execute arsenic her.

Scientific American said to Herndon astir training her AI models and her belief that productivity has ever been collective—AI conscionable makes it visible.

On supporting subject journalism

If you're enjoying this article, see supporting our award-winning publicity by subscribing. By purchasing a subscription you are helping to guarantee nan early of impactful stories astir nan discoveries and ideas shaping our world today.

[An edited transcript of nan question and reply follows.]

You picture your activity arsenic “protocol art.” What does that mean?

In nan 20th century, nan tract of media generation—the insubstantial and pen wherever euphony was written—was nan creator act. With protocol art, nan imaginative enactment happens upstream of media generation. It’s creating nan norm group and conditions successful which creation is made.

We’re really willing successful training our ain models. I ever opportunity “we” because I activity pinch my partner, Mat Dryhurst. We dainty each measurement successful nan model-making process arsenic a imaginative involution moment. The making of nan dataset is portion of nan artwork. I often constitute euphony for training—music not needfully for quality ears but for a machine to study something.

Can you springiness maine an illustration of what that looks for illustration successful practice?

We person an accumulation successful Berlin correct now. We were inspired by Hildegard von Bingen, a medieval composer. We wanted to dress arsenic if polyphony had existed erstwhile she was alive. We started pinch a exemplary of her compositions and added norm sets truthful it could make polyphony successful her style. We took those outputs, rearranged them and gave them to quality singers to interpret. Then we created a immense installation wherever performers singing and induce nan nationalist to train pinch us.

It’s not astir putting successful “write maine a popular opus pinch a guitar.” It’s astir utilizing this exertion to bring humans together to make creation successful existent space.

Most commercialized AI models are trained connected information scraped from nan Internet. Why do you insist connected building your ain models?

As an physics musician, I was ne'er 1 to sample—I ever made my ain sound palettes. When we started, pre-Suno and pre-all-this-stuff, we had to make our ain dataset. It conscionable felt natural, for illustration making my ain samples aliases integer instruments.

One disapproval of products [like Suno] is that they’re very “mid” sounding—trained connected everything aliases nan astir average. My models sound unsocial because I’m making nan training information myself. I besides deliberation there’s prompting nether nan hood successful Suno limiting it to three-minute songs pinch verse-chorus structure. There are guardrails making it boring. I’d emotion for them to merchandise immoderate constraints.

Has a exemplary ever amazed you?

We did a task called Holly+ astir 2021—a sound clone of my peculiar voice. We worked pinch Voctro Labs to train a sound exemplary that useful successful existent clip truthful group tin singing utilizing my voice. That was game-changing.

If this useful successful existent time, different group tin execute each other’s personality successful existent time. When we were testing it, my partner, who’s British, was singing into it. I heard my sound pinch a British accent. It was truthful uncanny, I had to time off nan room—he was singing arsenic me. That was 1 of nan biggest intelligence unlocks of really weird and cool this worldly tin get.

I deliberation it’ll return 5 to 10 years to beryllium seamless. But erstwhile we’re assemblage morphing successful existent time—imagine you could create a exemplary of a whale voice, past do a hybrid soprano whale. When you singing high, it goes operatic; erstwhile you singing low, you’re much whale aliases Barry White. We’re nary longer tied to my larynx.

Where do you deliberation we’ll beryllium successful 10 years?

A batch of fears astir this exertion are really fears of really nan existent Internet works—the attraction economy, really difficult it is arsenic a creator. My partner ever says, “Scrolling is for bots, and strolling is for humans.”

Our much optimistic imagination is utilizing agents to woody pinch each nan crap and select done stuff, really bringing america together successful nan existent world. That’s why our projects impact group gathering IRL and doing things together. Some of my smartest developer friends are vibe coding pinch aggregate agents while cooking aliases hiking pinch their toddler. Things could beryllium really beautiful if we ideate and build it that way.

Does this exertion alteration your meaning of creativity?

This full AI point mightiness unit america to spot ourselves arsenic possibly not nan only imaginative actors successful nan universe. That needn’t beryllium scary—it could beryllium beautiful and liberating.

Creativity happens successful swarm, successful community. AI is conscionable collective intelligence—aggregated quality intelligence. The 20th-century creation exemplary is tied to an individual genius who touches an entity and imbues it pinch value. That’s being thrown connected its head. I’m each squad corporate intelligence.