Onpode
Cover art for Why transformers replaced RNNs—the attention mechanism's parallelization advantage

Why transformers replaced RNNs—the attention mechanism's parallelization advantage

October 14, 2026 · 13 min

Alex Mercer & Jordan Hale

Transformers replaced RNNs because self-attention eliminates sequential hidden-state dependencies, allowing all tokens to be processed simultaneously during training — the breakthrough that enabled BERT and GPT to scale to billions of parameters. But at inference, transformer-based language models still generate one token at a time, autoregressive and sequential, making inference latency an unsolved engineering problem.

Transformer models represent a foundational architectural shift in machine learning, particularly for natural language processing. Traditional recurrent neural networks (RNNs), developed in the 1980s–90s and refined through variants like Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs), process sequential data one token at a time.

0:0012:40
Get the next episode on Artificial Intelligence →

Follow it free — new episodes land in your feed.

Or make your own — any topic, in minutes

More Onpode episodes on Artificial Intelligence →

About this episode

Most explanations of why transformers replaced RNNs skip the most important clause. The parallelization breakthrough — the one that made BERT, GPT, and billions of parameters possible — happened at training time. When you're actually using one of these models and watching it generate a response, that's still sequential. One token, conditioned on all the previous ones. The inference wall is real, and it's an active engineering problem. This episode starts with that gap and works backward. It explains why RNNs couldn't just be fixed — the sequential dependency wasn't a setting, it was load-bearing — and why LSTMs and GRUs, for all their genuine improvements, were working inside a constraint they couldn't structurally remove. It then traces how self-attention replaced recurrence entirely, why that architecture mapped so cleanly onto GPU hardware that already existed, and what that alignment actually unlocked at scale. But the episode doesn't stop at the triumph. It sits with a specific number — a TCN-Attention hybrid outperforming a pure transformer on streamflow prediction — and asks what that means for the 'transformers solved sequence modeling' story. The honest answer is that the revolution was contingent: real and large, but bounded by conditions that don't hold across every domain or every hardware era. Worth your time if you want the fuller picture.

Frequently asked

Why can't RNNs be parallelized during training?

RNNs cannot be parallelized during training because each hidden state h_t depends mathematically on the previous hidden state h_{t-1}. That sequential dependency is built into the core formula — it is not a tuning parameter. Step 47 literally cannot be computed until step 46 is complete, no matter how powerful the hardware.

How does the transformer attention mechanism enable parallelization?

Transformer self-attention, introduced in the 2017 Google paper 'Attention Is All You Need,' removes recurrence entirely. Every token queries every other token simultaneously, computing relevance weights across all positions at once. This means training a transformer maps directly onto GPU and TPU parallel matrix operations, with no sequential waiting between steps.

Do transformers generate text sequentially or in parallel?

Transformer-based language models like GPT generate text sequentially at inference. Each output token is conditioned on all previous tokens — autoregressive generation — meaning users experience one-token-at-a-time output. The parallelism transformers are famous for applies during training, not during the text generation users actually see.

What problem did LSTM solve that RNNs had, and why wasn't it enough?

LSTMs, developed in the 1990s, solved the vanishing gradient problem by adding gating mechanisms that allow gradients to survive more time steps during backpropagation. However, LSTMs did not remove the sequential hidden-state dependency. Training was still one step at a time, so LSTMs could not scale to billions of parameters the way transformers eventually did.

Are transformers always better than RNNs and hybrid models?

Transformers are not universally superior. A TCN-Attention hybrid model outperformed a pure transformer on streamflow prediction, scoring an NSE of 0.2016 versus 0.1738 for the pure transformer. Transformers dominate under conditions of massive datasets and GPU alignment, but on scarce time-series data, recurrent and convolutional hybrid architectures can win.

Grounded in 12 sources
Hardware Acceleration for Neural Networks: A Comprehensive Survey ↗ · arxiv.org
Will LLMs Scaling Hit the Wall? Breaking Barriers via Distributed Resources on Massive Edge Devices ↗ · arxiv.org
Forwardformer: Efficient Transformer With Multi-Scale Forward Self-Attention for Day-Ahead Load Forecasting ↗ · doi.org
GRAPH-ENHANCED TRANSFORMER NETWORK FOR FRAUD DETECTION IN DIGITAL BANKING: INTEGRATING GNN AND SELF-ATTENTION FOR END-TO-END TRANSACTION ANALYSIS ↗ · doi.org
Enhancing session-based E-commerce recommendations with self-attention: a transformer-based approach ↗ · doi.org
Research of improvement of multilingual scientific translation model based on neural network attention mechanism ↗ · doi.org
TCN-attention: a temporal convolutional network with self-attention mechanism for streamflow prediction in Chinese River Basins ↗ · doi.org
Transformers as the Foundation of Large Language Models ↗ · researchgate.net
Intro to the Transformer Architecture ↗ · medium.com
Exploring Attention Mechanisms in Transformers: Self-Attention, Multi-Head Attention, and… ↗ · medium.com
Why RNNs Fail at Sequential Data — And What Finally Fixed It | by Himanshu Kumar | Medium ↗ · medium.com
Efficient Implementation of Large Language Models (LLMs) | by Jung-Hua Liu | Medium ↗ · medium.com
Read transcript

Jordan Hale: Alex, hey — I need to confess something to you before we start.

Alex Mercer: This is a new format. Go ahead.

Jordan Hale: I've been telling people — for like two years — that transformers solved sequential processing. Like, at dinner, to my family, to anyone who asked why AI got so good so fast. 'They got rid of the sequential bottleneck.' I said that with total confidence. And I think I was, no wait — I was wrong. Or at least, like, I was telling a story that's missing its most important clause.

Alex Mercer: What's the missing clause?

Jordan Hale: During training. Transformers solved sequential processing during training. The 2017 'Attention Is All You Need' paper — Google researchers, self-attention replaces recurrence — the breakthrough is that every token in a sequence can now be processed simultaneously when you're training the model. But GPT, when it's generating output, is still going one token at a time. Autoregressive. Conditioned on everything before it. That's the same sequential shape we said we escaped.

Alex Mercer: Okay — I want to push on whether 'wrong' is exactly the right word, though. Because the thing transformers actually removed was the hidden-state dependency that made Recurrent Neural Networks impossible to parallelize during training. That was the constraint that made LSTMs and GRUs hit a ceiling on sequence length and model size. Self-attention replaced that with direct token-to-token relationships, all computed simultaneously.

Jordan Hale: Which is real and important, I'm not saying it isn't—

Alex Mercer: But the inference behavior, the thing users actually see, is still sequential. That part you're right about.

Jordan Hale: And that gap is what let BERT and GPT scale to billions of parameters — the training unlocked that — but the users watching it type are watching something that looks a lot like what we had before, just faster hardware.

Alex Mercer: I think the more precise way to frame it: transformers won the training-time war. Whether that's the whole revolution or just the part that mattered for scaling — that's actually the thing worth unpacking.

Jordan Hale: Because if it's 'just' training, does that change what we think transformers actually are? Like, are we looking at a fundamental architectural revolution, or something that was perfectly shaped for a specific era of hardware and data?

Alex Mercer: That's basically the question. And I don't think the answer is obvious.

Jordan Hale: But to even get to that training-time question, I feel like we have to go back one step — because I don't think people understand why RNNs couldn't just, you know, be fixed. Like, why wasn't it a tuning problem?

Alex Mercer: Okay — here's the plain version. Imagine you're reading a novel, but you're only allowed to remember the previous sentence. Every time you finish a sentence, you compress everything you know into one sticky note, throw the rest away, and carry that note forward. That's a Recurrent Neural Network. The sticky note is the hidden state — h_t. The technical name for it.

Jordan Hale: And if the answer to a question is 300 sentences back—

Alex Mercer: It has to survive 300 compressions. Each one lossy. That's what people mean by long-range dependencies — information degrades across every intermediate state.

Jordan Hale: But wait — the part that I keep tripping over is that it's not just that the sticky notes get blurry. You can't even write the next sticky note until you've finished reading the current sentence. You literally cannot start note 47 until note 46 exists. That's — that's why you can't parallelize it.

Alex Mercer: That's exactly the mechanism. h_t equals f of h_{t-1} and x_t. Every step depends on the previous hidden state. It's baked into the formula — not a setting you can turn off.

Jordan Hale: So when LSTM showed up — what was that actually solving?

Alex Mercer: The vanishing gradient problem, mostly. During backpropagation, gradients shrink exponentially through many time steps — so the model basically stops learning from anything far back. LSTM added gating mechanisms in the 1990s to let some gradients pass through more cleanly. GRU did something similar, simpler gating. Both real improvements.

Jordan Hale: But neither of them touched the sequential constraint itself.

Alex Mercer: That's the thing. You could make the sticky notes better — stickier, more organized — but you still couldn't write two of them at the same time. The sequential dependency wasn't an implementation choice. It was load-bearing.

Jordan Hale: Which means, like — no matter how clever your gating gets, you're still fundamentally one-step-at-a-time. And that ceiling is what stopped RNNs from scaling the way transformers eventually did on GPUs.

Alex Mercer: Right — and I think that's the distinction worth holding onto. LSTM and GRU were meaningful. They extended what was possible. But they were working inside a constraint that, structurally, they couldn't remove. It wasn't a tuning problem. The architecture itself was the ceiling.

Jordan Hale: So the question isn't why LSTM failed — it's why nobody could see that the constraint was the problem, not the implementation. And that's actually where 2017 gets interesting, because what 'Attention Is All You Need' did was just... not have a hidden state at all.

Alex Mercer: Not just no hidden state — no sequential dependency at all. That's the architectural replacement. 'Attention Is All You Need' didn't optimize recurrence. It removed it entirely. Every token just... queries every other token directly.

Jordan Hale: Wait, like, all at once? Every token to every other token, simultaneously?

Alex Mercer: All at once. During training. No intermediate states required.

Jordan Hale: Imagine — instead of one librarian passing notes down a row of students one by one, you just pin every piece of information on a wall, and every student reads the whole wall at the same time. No chain. No waiting for the student in front of you to finish. That's what self-attention actually is, right? Each token is basically asking every other token: how relevant are you to me, right now?

Alex Mercer: That's the mechanism, yeah. And the weight each token assigns to every other one — that's what gets learned. The model figures out which parts of the wall actually matter.

Jordan Hale: And then — wait, there's the multi-head thing. Because Google didn't just do one set of those queries, they ran several in parallel. Multiple attention heads, each one learning different relational patterns, and then you concatenate the outputs. So it's not one librarian reading the whole wall, it's like — several, each looking for something different.

Alex Mercer: Multi-head self-attention. And that's richer than any single attention function could be — one head might track syntax, another might be doing something more semantic. The concatenation gives you all of that at once.

Jordan Hale: Which is where the GPU thing clicks for me, because — those parallel matrix operations, that's exactly what GPUs and TPUs are built for. RNNs made those chips sit and wait. Every step had to finish before the next one could start. You're paying for massively parallel hardware and using maybe a fraction of it.

Alex Mercer: That's the alignment story. Transformer self-attention maps directly onto the hardware paradigm that already existed. You didn't need new chips — you needed an architecture that could actually use the chips you had.

Jordan Hale: And BERT and GPT are basically the proof. I mean, those models couldn't have existed — not at their parameter counts — under RNN constraints. The sequential dependency would have made training them impossible at that scale.

Alex Mercer: To some extent, that's the whole point. BERT and GPT aren't just big — they're only possible because the parallelism unlocked the compute. Billions of parameters trained across all positions simultaneously. An LSTM version of GPT-3 is basically not a thing you could build.

Jordan Hale: But all of this parallelism is at training time — and this is the part that I think is going to complicate everything we just said. There's a version of this story where, when you actually deploy the model and start generating tokens, the picture looks pretty different from the wall-of-information version we just described.

Alex Mercer: I think that's exactly right, and it gets more complicated from there — because it's not just inference where the story gets messy.

Jordan Hale: Right — but the part that doesn't fit is, like, when you actually use ChatGPT and watch it generate a response, that's not a wall of information anymore. Every token it outputs is conditioned on all the previous ones. One at a time. That's autoregressive. That's sequential.

Alex Mercer: And the latency is real. Inference on long outputs hits a hard wall because of exactly that.

Jordan Hale: Wait — so we celebrated transformers for escaping the sequential constraint, and then... we built GPT, and GPT generates sequentially?

Alex Mercer: At inference, yes. The parallelism that changed everything — the thing that let GPT and BERT scale to billions of parameters — that happened during training. The generation process users actually experience is still one token conditioned on all previous tokens. The sequential shape is back.

Jordan Hale: That's — I mean, that's a genuinely uncomfortable thing to sit with. Because we didn't solve sequential generation, we just, like — we moved the bottleneck. Training got fast. Inference didn't.

Alex Mercer: Which is why inference latency is still an active engineering problem. The 2017 paper didn't close that question.

Jordan Hale: And there's this other piece that I think is even weirder — because if transformers are just architecturally superior, everywhere, universally, then why does a TCN-Attention hybrid beat a pure transformer on streamflow prediction? Like, the numbers are specific: NSE of 0.2016 for the hybrid versus 0.1738 for the pure transformer.

Alex Mercer: The NSE gap — 0.2016 versus 0.1738 — that's not noise. That's a domain-specific architecture, Temporal Convolutional Networks combined with self-attention, outperforming the pure transformer on a task where the data doesn't match the conditions transformers were built for.

Jordan Hale: Which is — no, wait, that's the thing — it's not just that the hybrid won. It's what that means for the whole 'transformers solved sequence modeling' story. If they solved it, why are recurrent and convolutional approaches still winning at the edges?

Alex Mercer: Because the conditions that make transformers dominant — massive datasets, hardware alignment, training at scale — those don't hold equally across all domains. Streamflow prediction doesn't give you billions of tokens. It gives you long time-series with relatively scarce data. Different regime entirely.

Jordan Hale: And that's actually why the state-space model thing makes sense to me now — the field is actively building linear-complexity alternatives because quadratic attention cost, at very long contexts, just hits a ceiling. That's not incrementalism, you know, that's the field acknowledging a structural limit.

Alex Mercer: Basically, yes. Quadratic attention means cost scales with sequence length squared. That's fine at moderate lengths. At very long contexts it becomes the constraint. So the retreat toward linear-complexity models isn't an accident.

Jordan Hale: So the deepest version of this is — transformers solved a specific bottleneck, under specific conditions. Big data, GPU alignment, training scale. That's not universal. That's contingent.

Alex Mercer: And 'contingent' doesn't mean unimportant. It means the revolution was real — but its domain has edges. Which the streamflow number makes pretty concrete.

Jordan Hale: And that's the thing I genuinely can't settle. Like, somewhere right now, a researcher is watching a pure transformer lose to some hybrid on a narrow benchmark — maybe hydrology, maybe something else entirely — and they're deciding: is this a footnote, or is this the first data point of something? And I don't know how you'd even tell the difference from inside it.

Alex Mercer: I think that's actually the most honest place the episode took us. The training-time parallelism was real — it unlocked BERT, GPT, billions of parameters. But the claim that transformers solved sequence modeling universally? That doesn't survive the streamflow number. Or the inference wall.

Jordan Hale: The inference wall is what keeps turning in my head. Because if the hardware changes — you know, quantum, neuromorphic, something that doesn't privilege those parallel matrix operations the way GPUs and TPUs do — does the whole architectural calculation flip? Like, is the transformer the right answer, or the right answer for this particular hardware era?

Alex Mercer: Genuinely open question. I don't think anyone can answer that honestly right now.

Jordan Hale: Which is — I mean, we spent this whole time unpacking one word. 'Parallel.' And it turned out to mean something completely different at training time versus when you're actually using the thing. That's not a small semantic error. That's the whole story.

Alex Mercer: To some extent, yeah. The revolution was real and contingent at the same time. Those aren't in contradiction — they're just both true, and most of the discourse picks one and ignores the other.

Jordan Hale: I'm glad we didn't pick one. Good talk — genuinely.