GALMET (“Genetic Algorithm using Language Models for Evolving Text”) generates text by putting a large language model made for predictions inside a genetic algorithm. It was an early example, arguably one of the first methods to combine a genetic algorithm with a large language model to generate text, predating the later wave of evolutionary LLM approaches. We used it to evolve real news headlines into more satirical ones.
Why a genetic algorithm?
BERT-style models such as RoBERTa are excellent at analysing text, but they were never built to write it: their masked language model head can only fill in a blank in an existing sentence. A genetic algorithm, however, needs exactly the two things these models are good at: a way to make a small change to a candidate (mutation) and a way to score it (fitness). GALMET plugs a fine-tuned RoBERTa model into both.
How it works
GALMET starts from a single real headline, copies it 50 times into an initial population, and then repeats the classic genetic algorithm loop of scoring, selecting, crossing over and mutating:
- Mutation: a masked language model, fine-tuned on real and satirical headlines, masks a token (a word or part of one) and fills the gap with a new one, or inserts an extra token. A third operator simply deletes a random token.
- Crossover: two candidates are cut at a word they share and swap halves, combining the edits of both parents.
- Fitness: a regression head, fine-tuned to tell real headlines from satire, scores how satirical each candidate reads, from 0 (real news) to 1 (satire). The edit distance to the original headline keeps the result recognisable.
Along the way, GALMET remembers the most satirical candidate for every number of edits, and it stops as soon as a headline scores above 0.99 with at most seven edits. Here is one run, with the best headline it found for each number of edits:
| Edits | Satire score | Headline |
|---|---|---|
| 0 | 0.01 | Most Americans Want Congress To Investigate Michael Flynn |
| 1 | 0.02 | Why Americans Want Congress To Investigate Michael Flynn |
| 2 | 0.07 | Most Americans Want To Investigate Michael Jordan |
| 3 | 0.46 | 224 Americans Asked To Investigate Michael Flynn |
| 4 | 1.00 | 224 Americans Asked To Investigate Michael Jordan |
Both models were trained on a mix of public datasets of real, satirical and humorously edited headlines, including Humicroedit, FunLines, Unfun and The Onion or Not.
Results
To evaluate it, we took 408 real headlines from the Humicroedit dataset, which also contains a version of each headline that a human edited to be funnier. Eighteen volunteers then judged 1,498 pairs of headlines, each time picking the funnier one.
- Against the human edits, GALMET’s headline won 26% of the time. Humans clearly still wrote better satire, but that is a decent score for a machine.
- Against its own mutation model, simply making seven edits without any evolution, GALMET narrowly lost: judges preferred the plain fine-tuned masked language model 53% of the time.
The culprit was mostly readability. Because GALMET also inserts and removes word fragments due tokenization putting subwords in separate tokens, it sometimes coined words like “Vomoted” or “Haveared”. Judges found such broken headlines hard to rate, and the satire scorer, which had only ever seen real words, scored them almost at random.
So the results were modest, and that is fine: GALMET was meant as a proof of concept. It showed that you can steer a masked language model with a separate scoring model rather than letting it generate on its own. The same recipe could evolve text towards other domains that are hard for computers to grasp, such as poetry, or power a co-creative tool that suggests how to nudge your own writing towards a particular style.
Code
The code, the two fine-tuned RoBERTa models and an example notebook are available on GitHub.


