Credit: Google

On Thursday, Google announced that “commercially motivated” actors have attempted to clone knowledge from its Gemini AI chatbot by simply prompting it. One adversarial session reportedly prompted the model more than 100,000 times across various non-English languages, collecting responses ostensibly to train a cheaper copycat.

Google published the findings in what amounts to a quarterly self-assessment of threats to its own products that frames the company as the victim and the hero, which is not unusual in these self-authored assessments. Google calls the illicit activity “model extraction” and considers it intellectual property theft, which is a somewhat loaded position, given that Google’s LLM was built from materials scraped from the Internet without permission.

Google is also no stranger to the copycat practice. In 2023, The Information reported that Google’s Bard team had been accused of using ChatGPT outputs from ShareGPT, a public site where users share chatbot conversations, to help train its own chatbot. Senior Google AI researcher Jacob Devlin, who created the influential BERT language model, warned leadership that this violated OpenAI’s terms of service, then resigned and joined OpenAI. Google denied the claim but reportedly stopped using the data.

Even so, Google’s terms of service forbid people from extracting data from its AI models this way, and the report is a window into the world of somewhat shady AI model-cloning tactics. The company believes the culprits are mostly private companies and researchers looking for a competitive edge, and said the attacks have come from around the world. Google declined to name suspects.

The deal with distillation

Typically, the industry calls this practice of training a new model on a previous model’s outputs “distillation,” and it works like this: If you want to build your own large language model (LLM) but lack the billions of dollars and years of work that Google spent training Gemini, you can use a previously trained LLM as a shortcut.

To do so, you need to feed the existing AI model thousands of carefully chosen prompts, collect all the responses, and then use those input-output pairs to train a smaller, cheaper model. The result will closely mimic the parent model’s output behavior but will typically be smaller overall. It’s not perfect, but it can be a far more efficient training technique than hoping to build a useful model on random Internet data that includes a lot of noise.

The copycat model never sees Gemini’s code or training data, but by studying enough of its outputs, it can learn to replicate many of its capabilities. You can think of it as reverse-engineering a chef’s recipes by ordering every dish on the menu and working backward from taste and appearance alone.

In the report published by Google, its threat intelligence group describes a growing wave of these distillation attacks against Gemini. Many of the campaigns specifically targeted the algorithms that help the model perform simulated reasoning tasks, or decide how to process information step by step.

Google said it identified the 100,000-prompt campaign and adjusted Gemini’s defenses, but it did not detail what those countermeasures involve.

A clone of a clone

Google is not the only company worried about distillation. OpenAI accused Chinese rival DeepSeek last year of using distillation to improve its own models, and the technique has since spread across the industry as a standard for building cheaper, smaller AI models from larger ones.

The line between standard distillation and theft depends on whose model you’re distilling and whether you have permission, a distinction that tech companies have spent billions of dollars trying to protect but that no court has tested.

Competitors have been using distillation to clone AI language model capabilities since at least the GPT-3 era, with ChatGPT a popular target after its launch.

In March 2023, shortly after Meta’s LLaMA model weights leaked online, Stanford University researchers built a model called Alpaca by fine-tuning LLaMA on 52,000 outputs generated by OpenAI’s GPT-3.5. The total cost was about $600. The result behaved so much like ChatGPT that it raised immediate questions about whether any AI model’s capabilities could be protected once it was accessible through an API.

Later that year, Elon Musk’s xAI launched its Grok chatbot, which promptly cited “OpenAI’s use case policy” when refusing certain requests. An xAI engineer blamed accidental ingestion of ChatGPT outputs during web scraping, but the specificity of the behavior, down to ChatGPT’s characteristic refusal phrasing and habit of wrapping responses with “Overall…” summaries, left many in the AI community unconvinced.

As long as an LLM is accessible to the public, no foolproof technical barrier prevents a determined actor from doing the same thing to someone else’s model over time (though rate-limiting helps), which is exactly what Google says happened to Gemini.

Distillation also occurs within companies, and it’s frequently used to create smaller, faster-to-run versions of older, larger AI models. OpenAI created GPT-4o Mini as a distillation of GPT-4o, for example, and Microsoft built its compact Phi-3 model family using carefully filtered synthetic data generated by larger models.

DeepSeek has also officially published six distilled versions of its R1 reasoning model, the smallest of which can run on a laptop.