Once you’ve trained your large language model on the entire written output of humanity, where do you go? Here’s Ilya Sutskever, ex-OpenAI, admitting to Reuters that they’ve plateaued: [Reuters] The…

I’m usually the one saying “AI is already as good as it’s gonna get, for a long while.”

This article, in contrast, is quotes from folks making the next AI generation - saying the same.

    • Korne127@lemmy.worldEnglish
      11·
      2 days ago

      LLMs are one version of AI. It’s just one tiny part of AIs that are used every day, from chess bots to voice transcription, but they also are AI.

  • cron@feddit.orgEnglish
    0·
    3 days ago

    It’s absurd that some of the larger LLMs now use hundreds of billions of parameters (e.g. llama3.1 with 405B).

    This doesn’t really seem like a smart usage of ressources if you need several of the largest GPUs available to even run one conversation.

    • 31337@sh.itjust.worksEnglish
      0·
      1 day ago

      Larger models train faster (need less compute), for reasons not fully understood. These large models can then be used as teachers to train smaller models more efficiently. I’ve used Qwen 14B (14 billion parameters, quantized to 6-bit integers), and it’s not too much worse than these very large models.

      Lately, I’ve been thinking of LLMs as lossy text/idea compression with content-addressable memory. And 10.5GB is pretty good compression for all the “knowledge” they seem to retain.

      • brucethemoose@lemmy.worldEnglish
        0·
        17 hours ago

        I don’t think Qwen was trained with distillation, was it?

        It would be awesome if it was.

        Also you should try Supernova Medius, which is Qwen 14B with some “distillation” from some other models.

        • 31337@sh.itjust.worksEnglish
          1·
          20 minutes ago

          Hmm. I just assumed 14B was distilled from 72B, because that’s what I thought llama was doing, and that would just make sense. On further research it’s not clear if llama did the traditional teacher method or just trained the smaller models on synthetic data generated from a large model. I suppose training smaller models on a larger amount of data generated by larger models is similar though. It does seem like Qwen was also trained on synthetic data, because it sometimes thinks it’s Claude, lol.

          Thanks for the tip on Medius. Just tried it out, and it does seem better than Qwen 14B.