Local AI Just Got Easy (and Cheap)

42 Comments

1. why secure boot on linux computer?

2. Local AI is the future.

3. Please, specify a moment, when I can see something "fast" on that video.

4. usb 3 is supposed to be 5 gb/s not 650mb im pretty sure

5. Want this on a slot for my Framework 16 instead of the graphics card.

6. nice work, binge-watching your stuff! what's the screen recording software you're using with the rounded-borders face cam bottom right?

7. does intel also sell neural compute stick for the same purpose?

8. What is the tiny monitor that we see early in the video clip?

9. Suddenly, someone needed to sell a bunch of slow 3 year old inference chips with very limited use. This is their story.

   An RTX4090 is more energy efficient than an equivalent number of these things assuming no sparsity @ 660 INT8 TOPS for ~380W (ignoring the dual Epyc or PCIe switch cards needed to run 160 of them in the same machine and still come in below the base clock int8 performance). I suppose you could attempt to run that many off USB somehow on a 16 lane consumer processor but it won't be able to cope with the interrupt management and USB overhead since it's a garbage protocol. That number is wrong since the 4090 will be running quite a bit faster than base clock at 380W draw but it doesn't matter. 0.576W / TOP vs 0.8125W / TOP. Sparsity potentially halves that number. Of course most of the models that can be quantized to INT8 will run in INT4 as well so you could be running everything at ~1.3 INT4 PetaOPS.

   Asus makes the only PCIe cards that hold 16 of them per card and according to this incredibly fat gay black trans-man you might as well cut your junk off and join scientology before expecting any of their products to work correctly. They're either going to require a 1/1/1/1/1… etc 1×16 way bifurcation nothing supports, or Asus is going to be charging you 500-600 for a 2-generation outdated PCIe switch that they've probably programmed wrong if their redrivers on Threadripper boards are any indication. The fact that they need a 2-slot card to dissipate 54W on these is a good indication. Their new boards with USB 4 state that the cables shouldn't be removed or swapped until all power to the motherboard is killed. You know, USB-C connectors, those famously stable things. Their last threadripper board managed to have slots that stopped working if you used other slots on a platform with 128 lanes available, and lacked enough 8-pin PCI-e power to the motherboard to run more than 2 GPUs… and it sounded like less of a disaster than their old x99 boards.

   TL;DR if you're going to buy outdated tech junk, instead of feeding money to whatever goofdouche here is promoting, hop on ebay and treat your home network to an upgrade to P2P 56Gb Ethernet / Infiniband dual port cards and some SR4 transceivers so you can quit being insulted by things like 2.5Gb ethernet in brand new machines, or any of the FPGAs that are available that I'm just gonna go ahead and bet will destroy the performance of these things (plus people might actually hire you for "Languages known: VHDL / Verilog" on a resume, they won't for "I SET UP PYTHON AND DETECTED FACES!!!!!!11").

10. i guess you already know, but you should use compose files, otherwise maintenance of your containers will become a hell..

11. I'm curious if it's possible to use the pcie version via USB with a simple adapter. I understand that it might affect speeds but could it at least hypothetically function? I'm trying to research this ATM. Also the USB version is about $30 more than the USB version so… ya know. Pinching pennies where I can and all that,

12. Person detection. Training the AI to hunt us more efficiently. lol
    This is an amazing little thing. I do have an issue with the AI online. Having a local device that can be use to monitor and do what it has to do on an offline local network is amazing.

13. Phenomenal

14. 4:13 /dev is devices (and not drivers). There was no apex_0 because the device wasn't plugged in yet.

15. Stop using that warp terminal please

16. gains sentience as you plugged in the TPU made me lol

17. Pretty sure you do not need to add root to any groups 😉

18. I'm curious whether the tensor flow models are updatable. Ie, could an enterprising person add a model which detects an approaching person (vs departing)… same with a vehicle.

19. "easy", just follow this simple 30 step process

20. Would love to see this in the pi 5 since it has pcie to see what it’s capable of

21. I love that you relate Frigate to golden eye, I do the same thing with relating facilities to bathrooms because of that game.

22. how much will all of this cost me

23. The usb one isn't only slower because the interface, it's also underpowered. The one you got isn't the greatest either, you should have got the dual edge which is by far the best price:performance ratio

24. So….slower than without it?

25. It will be interesting to know if it can run stable diffusion and how fast.

26. I was out when you said built by google

27. I cannot find an adapter so I can plug this PCIe Mini Card into a PCIe slot

28. that zooming in and out of the screen capture is actually making me barf

29. AI is the antichrist well done humans your understand the second coming of christ is near.

30. Can you confirm the Goog-Le hardware doesnt 'phone home' and share things, that it is completely self-contained and works offline?

31. using ffmpeg just like that is …. dumb. You need serious reencoding performance AND you need to crank the settings as high as possible – so that the image quality is decent so that the inference can make actual f*cking guesses and doesnt miss all the time because you're sending a pixelated 320×240-mess. OR : (hear me out) you could use a proper webcam or surveillance cam which pumps the stream out itself.

32. Maybe I missed it, but what card did you plug your TPU into? to be able to plug it into a PCIE slot?

33. Can it run llama 7b?

34. >by google
    no thanks

35. is the voice AI generated? its kinda choppy

36. Is there a discord or something to join

37. the annoying thing with google tensorflow is how often stuff is broken with new releases. Over the last 20 years, I've lost count of how many times Google decided to break backward compatibility in their projects. Google's idea of open source kinda blows.

38. My goal is to make a llm one day but i dont have a good enough graphics card to handle llm of higher sizes.

    Can you give me an idea on what you would doo for strictly an LLM but one that can handle a 70b llm

    Ideas for small but powerfull sets or soemthing even specific for that would be cool. Was ready to buy a zima blade and board and coral thing but i want the best without buying an a100

39. Given the keying of your M2 module is should have a max transfer rate of 250MB/s or maybe 500MB/s if using both PCIE channels.

40. PS there is nothing wrong with using an older version of python. Use conda or venv to properly version your pip deps

41. I don't know how you do your video editing, but I really like the zoom and other effects thrown in here. Nice walkthrough.

42. What software did you use to screen record while talking at 4:06. That flow is was so smooth