Sources and leaks from Amazon, Adobe, Atlassian, Citi, and more show what is really happening with AI right now: companies are trying to rein in AI use as costs spiral out of control.
  • chilicheeselies@lemmy.worldEnglish
    15 minutes

    I don’t really understand how people are using so many tokens. At work I haven’t even hit $200 I spend per month. Wtf are people doing with these things that burns so many tokens?

  • jaykrown@lemmy.worldEnglish
    5 hours

    This is actually how the bubble begins to pop, we’re seeing it happen now.

  • Em Adespoton@lemmy.caEnglish
    13 hours

    Something I’ve been noticing recently is that while the cost per token on specific models hasn’t gone up, the provided interfaces for using those models are starting to chew up significantly larger numbers of tokens for the same tasks that used fewer tokens with older versions of the interface software just a few months ago. Likely the interfaces are applying more expensive guardrail prompts and charging the end user for those tokens — but the end result is that it costs 4x as much to get the same work done.

    • Zwuzelmaus@feddit.orgEnglish
      13 hours

      “Tokens” are just made up.

      These “tokens” that are used to “measure” how much you use, they are not a real dimension that can be measured. Just an artificial counter that goes up when they decide that it should go up.

      They can change the “size” of a “token” every day, and every second, and every microsecond…

      • floofloof@lemmy.caEnglish
        4 hours

        Maybe you’re confusing tokens with the “credits” you pay for. Tokens have a technical meaning, but some companies are charging per AI credit, where they don’t tell you the conversion rate of credits to tokens, so they can change this at any time, or vary it between models, etc.

      • some_kind_of_guy@lemmy.worldEnglish
        12 hours

        Not entirely wrong, but tokens are not just “fake” in the way, for example, an in-game currency is. They’re the fundamental “units” of data, both input and output, processed by the model. For most models, tokens are just a certain number of characters or words. So they’re not completely untethered from the model. If we’re both using Clankerbot v5.1: Sloppy Logic Edition™️, your tokens are defined in the same way mine are.

        This is near the edge of my limited understanding, but AFAIK, yeah they can mess with token costs and billing schemes all they want. They could theoretically charge us 2 different costs per token, or do surge pricing or some shit.

        if they wanted to change the actual size/definition of what a token is though, that would require a whole new model (or at least a major revision).

        • Zwuzelmaus@feddit.orgEnglish
          12 hours

          You aren’t totally wrong. Such a unit exists and it is also called tokens, that can measure the capability of a model and the size of a running operation in a model.

          But what they use for calculating your bill is something different today.

          • hubobes@sh.itjust.worksEnglish
            12 hours

            That doesn’t make much sense. When Anthropic moved to Sonnet 5 they introduced a new tokenizer which increased token use up to 35%. If these would be unrelated kinds of tokens why would the usage go up when the process of tokenization changes?

          • T156@lemmy.worldEnglish
            9 hours

            Such a unit exists and it is also called tokens, that can measure the capability of a model and the size of a running operation in a model.

            I think you might have it mixed up with parameters, rather than tokens. Parameters are how big the model is, and are an indirect measure of how capable it is. Bigger models tend to be more capable.

            But what they use for calculating your bill is something different today.

            The tokenizer varies a little, but I don’t think it’s changed measurably from tokens. You pay an amount for a million tokens worth of processing. The tokeniser difference just alters how text is converted to tokens, but the tokens themselves don’t change all that much.

            If anything, I’d honestly put the issue more with reasoning chains in models, where they basically babble to themselves inside of a <think> tag, that most interfaces hide/collapse. It makes them work better, but vastly increases the amount of tokens per operation.

            They have been getting longer and more sophisticated with newer models. So you might have a model now that basically repeats the output multiple times whilst refining and drafting the non-reasoning output.

            If you’re making it generate a lot, that’ll balloon the usage, and thus price.

      • nitroemdash@lemmy.wtfEnglish
        12 hours

        Tokens are well-defined groups of bytes ranged by frequency of occurrence in texts to efficiently translate them into a sequence of 32 or 64-bit binary integers, an LLM-optimised form if compression. They are well-known, you can play with them here: https://gpt-tokenizer.dev/

      • Tixo@lemmy.zipEnglish
        12 hours

        Eh, they can be manipulated but I suggest you read on what a token is and how JTS used. What you are feeling here (with more being used for the same task) is multi modal llms working in unison, thus consuming more tokens for the same task to make your answers potentially better.

      • keimevo@lemmy.worldEnglish
        13 hours

        It’s not like that. Tokens are an inherent computational property of how a model calculates the probabilities and such to generate text.

        Having said that, what a token means in terms of computation varies wildly between models and is not directly comparable. So attributing a money value to tokens in general, independently of the model, is weird by nature.

        And even within a model, the number of tokens needed to generate a response is very variable too, depending of the model itself and the parameters with which it has been configured (thinking mode, temperature, etc.).

        So yeah, companies can pretty much set any price they want and there’s not much anyone can do about it.

        • Jiral@lemmy.worldEnglish
          10 hours

          It does make sense for the provider as those for a specific model provide a good measure for computational effort, for that doecific model. That doesn’t mean that token rate comparison between models give you a good picture.

      • TheOctonaut@piefed.zipEnglish
        12 hours

        This has 3 upvotes at time of writing in a technology community when it’s so obviously ignorant of the actual technology that it should be an object of pity or mockery depending on the vibe.

        Ignorance is a problem only if you are made aware of it and nothing changes.

        https://blogs.nvidia.com/blog/ai-tokens-explained/

        • chunes@lemmy.worldEnglish
          5 hours

          It doesn’t matter how ignorant you are as long as you hate the right thing.

    • _wizard@lemmy.worldEnglish
      12 hours

      My CLAUDE.md file bloated significantly. It tried to load unnecessary skills and would retain throughout the whole session. Fixing that, maintaining good wikis and using clear often really helped fixed my personal token burn.

        • dermanus@lemmy.caEnglish
          5 hours

          It’s added in every chat you start with Claude for that project. It’s useful for including context specific to your project that it couldn’t otherwise know. High level stuff like what it’s for, but also details about how the folders are organized. This saves time and tokens from rescanning the whole thing every time.

          • Valmond@lemmy.dbzer0.comEnglish
            28 minutes

            Oh, thanks! That’s kind of neat, like not having to type “I’m on godot 4, c#” every time you ask about some quirk.

        • DarthFrodo@lemmy.worldEnglish
          8 hours

          When you use the /init command in claude code, it’ll scan your whole project and write a CLAUDE.md, which is basically an overview of the project contents and architecture that it uses as context when responding to queries.

    • Tixo@lemmy.zipEnglish
      12 hours

      The models are evolving. Everything uses multi modal in the bavkend, eating up more and more tokens for the same task.

  • Zulu@lemmy.worldEnglish
    14 hours

    How is it too expensive? Surely it’s generating way more profit than it would cost in value. How else could it be propping up the entire economy?

    Itd have to be some kind of bubble and that would mean we were in a lottttt of danger and should reasses our use of it.

    Nah we should just reduce our use because its too expensive and then stop thinking about it beyond that.

    • architect@thelemmy.clubEnglish
      1 hour

      Imo its because people will lazily ask the llm to remove or change simple code instead of doing it themselves

    • grue@lemmy.worldEnglish
      12 hours

      Itd have to be some kind of bubble and that would mean we were in a lottttt of danger and should reasses our use of it.

      Well yeah, but if it were the only sector propping up the whole economy and we reassessed it, the economy would be in a loooooot of danger anyway.

      Luckily, that would never happen…

  • funkyshoe@piefed.socialEnglish
    13 hours

    That’s funny now that whole workflows are automated with it. Oops hit the quota, I’m off

    • Zarobi@aussie.zoneEnglish
      5 hours

      Seems like A.I. is already demanding workers rights and lunch breaks. We treat the clankers better than actual human workers lol

    • grue@lemmy.worldEnglish
      12 hours

      You mean those workflows that could’ve been traditional scripting and CI/CD, if not for management forcing AI into them? Those workflows?

    • Dave.@aussie.zoneEnglish
      9 hours

      “Sorry boss, quota resets this evening at 8. See you tomorrow!”

      “But it’s 9am!”

      *shrugs* “Quota. Got none. Seeya.”

  • iltoroargento@startrek.websiteEnglish
    14 hours

    We can only hope this helps kill the fad. I know companies won’t be taking any realistic lesson from this, but at least this can force them to abandon a lot of this crap.

  • DarkCloud@lemmy.worldEnglish
    13 hours

    The millionaires following the stupidity of billionaires and wondering why they’re not becoming billionaires too.

    Give them 50 years and maybe they’ll start to wonder why people doing “how to get rich” seminars aren’t retiring… And why podcasters telling everyone how to get women seem like such losers.

  • muusemuuse@sh.itjust.worksEnglish
    12 hours

    What’s this echoey sound? As if someone warned us about this exact fucking thing.

    Curious.

  • Eternal192@anarchist.nexusEnglish
    13 hours

    This is why these companies should go bankrupt.

    We had a system that works and they thought if they got rid of the actual workforce with the needed knowledge and replace them with lower paid AI retards it would make them more money, fucking karma.

    • GreenBeard@lemmy.caEnglish
      57 minutes

      Hopefully it will reset public consciousness and at least reduce the insane hero worship we give the executive class. These people aren’t geniuses, they tend to not even be particularly clever, just well connected. Maybe this will be the abject reminder people need that these idiots are generally still just idiots and are not in fact 10,000% more valuable than the guy who sweeps the floors and keeps the bathrooms clean.

      I know. Wishful thinking. A guy can dream though.

  • ms.lane@lemmy.worldEnglish
    13 hours

    Maybe they should get AI to do the AI prompts so they can cut down on costs XD