• QuadratureSurfer@lemmy.world
        link
        fedilink
        English
        arrow-up
        15
        ·
        8 months ago

        I’ve got it running with a 3090 and 32GB of RAM.

        There are some models that let you run with hybrid system RAM and VRAM (it will just be slower than running it exclusively with VRAM).

        • Deceptichum@kbin.social
          link
          fedilink
          arrow-up
          16
          ·
          8 months ago

          Yeah but damn does it get slow.

          I always find it interesting how text is so much slower than image generation. I can do a 1024x1024 in probably 20s, but I get like 1 word a second with text.

      • DarkThoughts@fedia.io
        link
        fedilink
        arrow-up
        2
        arrow-down
        1
        ·
        8 months ago

        Hopefully we see more specific hardware for this. Like extension cards with pretty much just tensor cores and their own ram.

    • mesamune@lemmy.world
      link
      fedilink
      English
      arrow-up
      10
      ·
      8 months ago

      Nice! Thats a cool project, ill have to give it a try. I love the idea of self hosting local LLMs. Ive been playing around with: https://lmstudio.ai/ and it directly downloads from hugging face.

    • DarkThoughts@fedia.io
      link
      fedilink
      arrow-up
      1
      ·
      8 months ago

      I tried llamafile for text gen too but I couldn’t get ROCm to properly work with it to run it through my GPU without having to build it myself, which I’m really not into. And CPU text gen is waaaaaay too slow for anything. Mixtral response was like ~250 seconds or so for ~1k context tokens, I think Mistral was about 52 seconds or something around that number.

      https://github.com/Mozilla-Ocho/llamafile Mixtral is definitely beefy, Mistral is quite a bit faster and there’s a few even smaller prebuilt ones. But the smaller you go the less complex the responses will be. I think llamafile is a good step in the right direction though, but it’s still not a good out of the box experience yet. At least I got farther with it than with oobabooga, which is the recommendation for SillyTavern, which would just crash whenever it generated anything without even giving me an error.

        • DarkThoughts@fedia.io
          link
          fedilink
          arrow-up
          1
          arrow-down
          1
          ·
          8 months ago

          Have you missed the first part where I explained that I couldn’t get it to run through my GPU? I would only have a 6650 XT anyway but even that would be significantly faster than my CPU. How far I can’t say exactly without experiencing it though, but I suspect with longer chats and consequently larger context sizes it would still be too slow to be really usable. Unless you’re okay waiting for ages for a response.

          • Flumpkin@slrpnk.net
            link
            fedilink
            English
            arrow-up
            1
            ·
            8 months ago

            Sorry, I’m just curious in general how fast these local LLMs are. Maybe someone else can give some rough info.