"Agentic AI" Is a Bonfire of the Tokens While Fab Capacity, Power Grids, and P&Ls Are the brakes: (NOT THE) READ OF THE DAY

Treść źródłowa

We scaled “attention is all you need” into an industrial‑scale stochastic parrot farm, then bolted on agents and tools until it started to look somewhat more like thought. Now the engineering reality—fabs, power, and eye‑watering token bills—is asking whether what we are doing is worthwhile. And general‑purpose LLMs start in‑breeding on their own output, unlike game AIs that thrive on tightly constrained, adversarial synthetic data. Are we trapping ourselves in a slop-filled sub-hyperplane of the potential reasoning space?Start with attention is all you need <https://arxiv.org/abs/1706.03762>, and scale. And the results are, as Cosma Shalizi noted lo these three years ago:ShareGive a gift subscriptionCosma Shalizi: “Attention”, “Transformers”, in Neural Network “Large Language Models” <https://bactra.org/notebooks/nn-attention-and-transformers.html>: ‘[an] incredibly impressive engineering accomplishment of [actually] making the blessed thing work. A large, able and confident group of people pushed kernel-based methods for years in machine learning, and nobody achieved anything like the feats which modern large language models have demonstrated. The reason I put effort into understanding these machines and papers is precisely because the results are impressive!…Again: finite-order Markov models…. Lots of people have played around with them, including tricks like variable context length, various kinds of partial pooling, etc. Nobody, so far as I know, has achieved results anywhere close to what contemporary LLMs can do. This is impressive enough that (as I said at the beginn…