Genomic Language Models Rewrite Biology

[SOFIA] Okay, so here's a thing that still slightly blows my mind. We built machines that read text — they finished your sentences, they write your emails. And somewhere along the way people asked: what if we point the exact same machinery at DNA? At protein? Just treat a genome like a language and let a model learn the grammar.

[DANIEL] And "language" there is not just a cute metaphor. A protein is a string over a twenty-letter alphabet. DNA is a string over four. Language models are, at bottom, prediction engines over sequences — mask out a token, predict what belongs there. Whether the token is a word or an amino acid, the math doesn't care.

[SOFIA] Right, and the payoff is huge if it works. Because most of biology's sequence data is unlabeled. We have oceans of genomes and we don't know what most of the genes do. If a model can learn structure and function just from the raw sequence, no annotation required —

[DANIEL] — then you've turned your biggest problem, all that dark unannotated data, into your training set. That's the dream. Now, before we get to the language models, I want to plant a flag on where this actually starts, because it's older and weirder than people think.

[SOFIA] The 2015 mitochondrial paper.

[DANIEL] The 2015 mitochondrial paper. And on its face it has nothing to do with AI. It's a cancer genomics study — 527 tumors from the TCGA, fourteen cancer types. They call somatic mutations in mitochondrial DNA, and here's the clever part: they had both the DNA and the RNA sequenced for the same tumors. Mitochondria give you insanely deep coverage, over five thousmore-fold on average, so the mutation calls are rock solid.

[SOFIA] And DNA and RNA agree almost perfectly, right? The allele frequencies track at r of 0.91.

[DANIEL] Almost. There are fifteen outliers where the RNA doesn't match the DNA. And twelve of those fifteen — eighty percent — are mutations in mitochondrial tRNAs. Every single one lands in a stem of the cloverleaf structure. They break the fold. And what happens downstream is the beautiful part: the mis-folded tRNA doesn't get processed, and the unprocessed precursor piles up, up to a hundred-fold.

[SOFIA] So the enzyme that cuts the transcript apart isn't reading the sequence. It's reading the shape.

[DANIEL] It's reading the structure. That's the whole point. The processing signal is secondary structure, not primary sequence. Two tRNAs with totally different letters but the same fold get cut the same way. And I bring this up because it's the earliest clean demonstration in this arc that the biological signal lives in a layer above the letters.

[SOFIA] Which is exactly the thing a language model is supposed to catch. The letters are the surface. The grammar — the folding, the interactions — that's what you actually want the model to learn. In 2015 they found it by hand, one tRNA at a time.

[DANIEL] By hand and with a hypothesis. Fast forward to 2022, and DeepLoc 2.0 is where you see the machine start doing that discovery on its own.

[SOFIA] Oh, I love this one. So DeepLoc predicts where a protein goes in the cell — nucleus, mitochondria, membrane, ten locations, and it's multi-label, meaning a protein can be sent to more than one place. But the part that gets me is how they do it. They take protein language model embeddings — ESM1b, ProtT5, models trained on hundreds of millions of protein sequences — and they pool them with an attention layer.

[DANIEL] And attention is the mechanism that tells you where the model is looking. So you can ask it: which residues drove your prediction?

[SOFIA] And the attention peaks land right on the sorting signals. The little address labels in the sequence that actually target the protein. Nobody told the model where those were.

[DANIEL] Which is the 2015 finding, but automated. In 2015 humans found the structural signal buried in the sequence. In 2022 the model finds the functional signal and points at it. Same idea — meaning lives above the letters — different engine.

[SOFIA] Okay, this is the good stuff, because 2024 is where it really breaks open. The OMG dataset and gLM2. Three-point-one trillion base pairs of metagenomic sequence — that's DNA pulled straight from environmental samples, soil, ocean, guts, mostly organisms nobody's ever cultured.

[DANIEL] And the trick with gLM2 is it's mixed-modality. Most models pick a lane — you're a protein model or you're a nucleotide model. gLM2 reads both at once, along a whole stretch of genome. Amino acids for the genes, nucleotides for the stuff between them.

[SOFIA] Which matters because in a real genome, genes sit next to each other for a reason. Bacteria run them in operons, co-regulated, and proteins that physically touch tend to co-evolve. A protein-only model that sees one gene at a time can't see any of that. gLM2 learns the coevolutionary interface signals — which residues on two different proteins are talking to each other — and it picks up regulatory syntax in the DNA between genes.

[DANIEL] The word I'd underline is context. It's reading the neighborhood, not the isolated word.

[SOFIA] And 2025 pushes context all the way up. WGRL — whole-genome representation learning. Instead of one gene, or one operon, it models ordered conserved elements across the entire genome and builds one embedding for the whole organism.

[DANIEL] And they tested it honestly, which I appreciate. Head-to-head against the standard workhorse — protein domain presence/absence, basically "which functional parts does this genome have." Twenty-five bacterial phenotypes. WGRL wins on twenty-three.

[SOFIA] Twenty-three of twenty-five. That's not a squeaker.

[DANIEL] It's not. Though I'll flag — this is one paper, one benchmark suite, and the interesting tension is that WGRL is partly a critique of the genomic-foundation-model idea it comes out of. It's saying the order and context of elements carries phenotype signal that the big general-purpose models weren't capturing. So the field's arguing with itself a little.

[SOFIA] Which is healthy! And then two more from 2025 that show the two directions this all splits into. CodonTransformer — trained on a million gene pairs across 164 species — learns each organism's codon grammar. And here's the emergent bit: when it generates a gene, it avoids negative cis-regulatory elements on its own. Nobody penalized it for those.

[DANIEL] Which for you is the whole game, right? That's a design tool.

[SOFIA] That's the dream for anyone doing expression in a non-model organism! You want to move a gene into some weird bacterium and have it actually translate well — you need that organism's codon dialect. A model that just knows it, and dodges the sequences that'd shut you down — yes, sign me up.

[DANIEL] And the enzyme annotation review is the sober bookend. It traces the same march we just walked — hand-crafted features, then deep learning pulling features automatically, and now generative models. But it makes the sharp point: the frontier isn't re-labeling enzymes we already know. It's proposing functions we've never seen.

[SOFIA] Which lands us right back where we started. In 2015 you found one structural signal by staring at tRNAs. Now the question is whether these models can hand us biology that isn't in any database yet.

[DANIEL] Reading, then writing, then maybe discovering. That's the arc. I just want the benchmarks to stay this honest as it scales.

[SOFIA] On that we agree. That's the through-line — from a cloverleaf fold in a tumor to trillion-base-pair models writing genes. We'll pick up the writing half next time. Stick around.