Genomic Grammar Unlocked By Machine Learning

[SOFIA] Okay, so here's a question that sounds almost philosophical until you realize people are actually doing it: can you teach a machine to *read* DNA? Not sequence it — we've been sequencing for decades — I mean read it the way you'd read a sentence. Figure out what it means.

[DANIEL] Which is a much harder problem than it sounds, because DNA isn't written like a sentence. There's no punctuation, the "words" overlap, and the same letters mean completely different things depending on context.

[SOFIA] Right, and this is the whole subject today — the arc of how we got from "DNA is a string of A, C, G, T" to models that supposedly understand genomic grammar. And it starts somewhere that has nothing to do with AI at all.

[DANIEL] Let me set up the problem first, because I think the stakes get lost. We can sequence a genome from an organism nobody's ever cultured — some microbe scraped off a hot spring — and get millions of base pairs. And then... we mostly can't tell you what any of it does. Huge fractions of genes in environmental samples have no annotation. "Hypothetical protein." That's the dark matter of biology.

[SOFIA] And for anyone coming from outside the field — the dream here is borrowed straight from language models. The same idea behind the chatbots. You train a model on enormous amounts of text, no labels, just "predict the next word," and it learns grammar and meaning as a side effect. Swap text for DNA or protein sequence, and the bet is the model learns biology the same way.

[DANIEL] That's the "foundation model" idea. And a key term for the newcomers: a *language model* here just means a model trained to predict masked or missing pieces of a sequence. When it's trained on protein sequences, we call the output an *embedding* — a vector, a list of numbers, that captures what the model learned about that protein.

[SOFIA] And embeddings are the magic trick, honestly. Because once a protein is a point in some high-dimensional space, proteins that behave similarly land near each other — even if their sequences look nothing alike.

[DANIEL] So where do you want to start the arc? Because the oldest paper on our list isn't a language model at all.

[SOFIA] No, and that's exactly why I love it as the root. 2015 — the mitochondrial DNA cancer study. 527 tumors from TCGA, and they call somatic mtDNA mutations from both the DNA and the RNA at the same time. Over 5000x coverage, which is bananas.

[DANIEL] And the headline result is a correlation. DNA allele frequency versus RNA allele frequency — r of 0.91. For most mutations, what's in the genome is what shows up in the transcript. Clean.

[SOFIA] But the fun is in the fifteen that *break* the correlation. Twelve of them are tRNA mutations, and every single one sits in a stem of the cloverleaf fold.

[DANIEL] Which tells you something mechanistic without any machine learning at all. The mutation disrupts the predicted secondary structure, and these mis-folded tRNAs pile up as unprocessed precursors — up to a hundredfold. So the cell's processing machinery isn't reading the sequence. It's reading the *shape*.

[SOFIA] That's the through-line for me. Structure is the signal, not letters. A mis-folded tRNA even blocks maturation of its neighbors, and the clusters get processed 3-prime to 5-prime. The information was never just in the string of bases.

[DANIEL] And that's the thing a sequence-only model would completely miss. Hold that thought, because it comes back.

[SOFIA] Okay, jump to 2022 — DeepLoc 2.0. This is where the language-model era actually shows up in our story. They take protein embeddings from these big pretrained models, ESM1b and ProtT5, and predict where a protein goes in the cell. Ten subcellular locations, and it's multi-label, because a protein can live in two places.

[DANIEL] And the part that earns my respect — they use attention pooling, and the attention peaks land right on the sorting signals. The actual physical signal sequences that tell a protein where to go.

[SOFIA] Which is the first "oh, it's really learning biology" moment. Nobody told it where the signal peptides were. It found them because they mattered for the prediction.

[DANIEL] A sanity check that the embedding isn't just memorizing. The model's attention agrees with decades of cell biology. That's the kind of thing that makes me trust it a little more.

[SOFIA] Then 2024, and this is the good stuff — the OMG dataset and gLM2. 3.1 trillion base pairs of open metagenomic sequence, and they build the first *mixed-modality* genomic language model.

[DANIEL] Define that, because it's the whole point.

[SOFIA] So every model before this picked a lane — you're either a protein model reading amino acids, or a DNA model reading nucleotides. gLM2 reads both, across a whole stretch of genome with multiple genes in order. Protein-coding regions as amino acids, the intergenic stuff as nucleotides.

[DANIEL] And why does that matter? Because in bacteria, genes that work together sit next to each other. Operons. Regulatory sequence between genes. A protein-only model throws all of that away — it never sees the spacing, the order, the regulatory syntax.

[SOFIA] And they claim gLM2 picks up coevolutionary signals at protein interfaces — basically which residues on two different proteins are evolving together because they physically touch. You can only see that if you're looking across genes at once.

[DANIEL] Now here's where the arc gets interesting, because 2025 gives us a genuine tension. WGRL — whole-genome representation learning. They do self-supervised language modeling on ordered conserved elements and get whole-genome embeddings. And those embeddings beat protein domain presence/absence on 23 of 25 bacterial phenotypes.

[SOFIA] Which sounds like a clean win for the foundation-model crowd — except it's tagged as being *in tension* with genomic foundation models. Why?

[DANIEL] Because it's a pointed result. Presence/absence of protein domains is the old, interpretable, workhorse method — just a checklist of what functional parts a genome has. WGRL says: a learned representation of gene *order and context* beats the checklist on almost everything. It's a rebuke to the idea that you can understand a genome as a bag of parts.

[SOFIA] It's the 2015 lesson again, scaled up. Context and arrangement carry information the parts list doesn't.

[DANIEL] That's a fair reading. Although I'd want to see those 25 phenotypes broken down — 23 of 25 is great, but which two failed, and were the easy ones easy for both methods? The brief doesn't say.

[SOFIA] Fair. And then CodonTransformer, also 2025 — this one flips from reading to *writing*. A BigBird transformer trained on a million gene pairs across 164 species, learning organism-specific codon grammar.

[DANIEL] This is the engineer's dream, right up your alley. If you want to express a gene in a non-model host, you have to match that organism's codon preferences or expression tanks.

[SOFIA] And it generates sequences matching the natural codon distribution for a given organism — but the part that got me, it *emergently* avoids negative cis-regulatory elements. Nobody penalized it for those. It just learned that natural sequences dodge them.

[DANIEL] Same flavor of result as DeepLoc's attention finding the signal peptides. The model learns the rule without being handed the rule.

[SOFIA] And the review paper — Kim and colleagues on enzyme annotation — basically narrates this whole arc. Hand-crafted features, to deep learning, and now the frontier is generative: not just re-labeling known enzymes but proposing genuinely new functions.

[DANIEL] Which is the honest statement of where we are. Reading is maturing. Writing — proposing new biology — is the open edge. And it still needs the thing I always want: experimental validation in a cell, not just a confident embedding.

[SOFIA] So the arc runs from a cancer genome telling us shape beats sequence, all the way to models that read context across whole genomes and write codons for organisms they've never been engineered into. The through-line never changed — meaning lives in the arrangement, not the letters.

[DANIEL] And the machines are finally reading the arrangement. We'll see if they can write it.

[SOFIA] That's the arc. Stick around — more after this.