Genomic Language Models Decoding Biology

[SOFIA] So here's a question I've been chewing on. We spent decades learning to read DNA — sequencing got cheap, we have billions of genomes sitting in databases. But reading the letters and understanding the letters are two very different things. And lately people have started training the same kind of AI that autocompletes your text messages on genomes instead. Teaching machines to read DNA the way they learned to read English.

[DANIEL] Hm. And the analogy is doing a lot of work there, so let's be careful with it. A language model, in the ChatGPT sense, is trained to predict the next token — the next word — from the words before it. Do that over enough text and the model builds up an internal representation of grammar, meaning, context, without anyone labeling any of it.

[SOFIA] Right, nobody hand-annotates what a noun is. It just falls out.

[DANIEL] It falls out of prediction. So the bet in genomics is: swap words for nucleotides, or for amino acids, train on enough sequence, and the model should absorb the grammar of biology. Which promoters go with which genes, which residues touch each other in a folded protein. The self-supervised part matters — you're not telling it the answers, you're letting the sequence teach itself.

[SOFIA] And this matters because the bottleneck in biology was never data anymore. It's interpretation. We have way more sequence than we have function. Most of a metagenome is genes nobody's ever characterized.

[DANIEL] That's the stakes, yes. Ninety-plus percent of what you pull out of soil or the ocean is unannotated. If a model can predict function from raw sequence, you skip a lot of wet-lab work. But — and this is the whole reason I care about the arc — you have to prove the model learned biology and not some artifact of the training set.

[SOFIA] Which is a great place to start the story, because the roots of this go back before the language models. Okay, this is the good stuff — the 2015 cancer mitochondrial paper. Because that one wasn't AI at all.

[DANIEL] No, it's classical, and I love it as the origin point precisely because it's careful. They took 527 tumors from TCGA, paired whole-genome sequencing with polyA RNA-seq, and called somatic mitochondrial mutations at over 5000-fold coverage. Six hundred sixteen high-confidence calls.

[SOFIA] And the beautiful bit — DNA allele frequency and RNA allele frequency agree at r of 0.91. So mostly, what's in the DNA shows up in the RNA at the same dose. Clean.

[DANIEL] Except for fifteen outliers. And that's where it gets interesting, because twelve of those fifteen — eighty percent — are mitochondrial tRNA mutations. Every one of them sits in a stem of the folded tRNA. They disrupt the predicted secondary structure, and you see the unprocessed precursor pile up, up to a hundred-fold.

[SOFIA] So the machine here isn't a neural net — it's the cell's own RNA processing machinery. And the lesson is that it's reading structure, not sequence. The fold is the signal. A misfolded tRNA jams the maturation of its neighbors.

[DANIEL] They even work out that clustered tRNAs get processed three-prime to five-prime. Which is a mechanistic claim, grounded in the data they had. I bring it up because it plants a flag: biological function lives in structure and context, not in the letters alone. Hold onto that — it becomes the whole tension later.

[SOFIA] So jump forward almost a decade and the machines get literal. The protein side matured first — AlphaFold, RoseTTAFold, models that learned to fold proteins from coevolution. That 2024 pathogens paper is the scaled-up version.

[DANIEL] RoseTTAFold2-Lite. A surrogate model, twenty times faster than the full thing, and — this is the design choice I respect — trained on both interacting and non-interacting protein pairs. You need the negatives. A model that's only ever seen things that touch will say everything touches.

[SOFIA] And with the speed they screened 78 million candidate interactions across 19 human bacterial pathogens. Landed 3,613 high-confidence complexes, and 37% of those nobody had seen before. First structural models of tRNA modification machinery, urease for virulence, outer-membrane biogenesis.

[DANIEL] So coevolution — residues that mutate in a correlated way because they're physically in contact — that's the signal these models read. It's the protein-language version of the 2015 lesson: structure and interaction, learned from statistics across many genomes.

[SOFIA] But proteins are only half the genome. And this is the turning point I love — people started asking, why throw away the DNA? The gLM2 paper, the OMG dataset. 3.1 trillion base pairs, JGI and MGnify metagenomes stitched together.

[DANIEL] And deduplicated in embedding space, which is a nice trick — you remove near-duplicate sequences by how the model sees them, not by exact string match. gLM2 is the first model that reads mixed modality: amino acids and nucleotides, across multiple genes in a locus, together.

[SOFIA] Which means it can see the intergenic stuff. The regulatory syntax between genes. A protein-only model literally cannot look there.

[DANIEL] Right, and they show it picks up coevolutionary interface signals a protein-only model can't access. So the field's argument becomes: context matters, read the whole locus.

[SOFIA] And then two 2024-into-2025 papers push on whether these DNA models actually learned anything real. The nucleotide dependency one is clever — you take a trained DNA language model and probe it by asking, if I change base A, how does that shift the model's prediction at base B?

[DANIEL] Pairwise substitution dependencies. And where the model says two positions depend on each other, you find functional elements — transcription factor binding sites, RNA structures. They validated four novel E. coli RNA structures that way. As accurately as models trained directly on experiments.

[SOFIA] Which is the 2015 lesson coming full circle! The dependencies the model learned are basically base-pairing — structure. Nobody told it about folds. It read them out of raw sequence.

[DANIEL] It's a genuinely satisfying convergence. Though the same paper flags architecture blind spots — some model designs just can't see certain dependencies. Which brings me to the one that's in tension.

[SOFIA] WGRL. The whole-genome one.

[DANIEL] Self-supervised, but not over raw nucleotides — over ordered conserved elements across the genome. Whole-genome embeddings. And it beat protein-domain presence/absence on 23 of 25 bacterial phenotypes.

[SOFIA] But it's listed as in tension with the genomic foundation model idea. Why the clash?

[DANIEL] Because it's a quiet challenge to the "just train a giant model on raw sequence" orthodoxy. WGRL says the representation — how you encode the genome before the model sees it, ordered conserved blocks — might matter more than raw scale. That's a real methodological disagreement, not a headline one, and I think it's healthy.

[SOFIA] And the newest, CodonTransformer, is almost the flip side — generation instead of reading. Trained on a million gene pairs across 164 species, it learns each organism's codon grammar and writes sequences that match natural distributions.

[DANIEL] And emergently avoids negative cis-regulatory elements. Nobody penalized those in training — the model just learned to steer around them. Which is exactly the promise: learn the grammar deeply enough and good design falls out.

[SOFIA] Which for someone who spent a postdoc trying to get DNA into weird bacteria — designing a synthetic gene that reads as native to your chassis is huge.

[DANIEL] The through-line for me: 2015 said function lives in structure and context. Ten years of models keep rediscovering that, from different angles. The open question is whether scale or smart encoding gets us there.

[SOFIA] Read the whole locus, respect the fold. That's the arc. More next segment — stick with us.