A lot of top researchers have already moved on from transformers.

Yann LeCun, Meta’s longtime chief AI scientist, quit and said LLMs are a “dead end” because scaling text-only models can’t produce real intelligence, and he’s not the only one who thinks so. Lots of engineers understand the limitations of LLMs.