Seth for AI

Hello, everybody. Welcome back to Freedom Tech Friday, a live and interactive show taking place taking place every Friday at 9AM eastern and 2PM UK time across the ungovernable feed. Each week, we dig into the latest in FreedomTech, including Bitcoin, Monero, encrypted messengers, and privacy tools, and, of course, everything in between.

Essentially, if there's a news item, tool, or topic that can help you take back some control in today's digital Panopticon, then we wanna talk about it. This show and the topics we covered are powered by freedom dot Tech, a daily news desk that uses AI

to monitor hundreds of sources so we can continue to bring you the signal each and every week. My name is Q and A and I'm head of customer experience at Foundation.

And as always, I am joined by my good friend Max, the head honcho of the Ungovernable Empire, and Seth who is COO at K Quality. Freedom Tech Friday is live and interactive, so you can help steer the conversation by commenting and

sharing it with friends. Top support for the last show covering our incredible new website, more on that shortly, at ungovernable.network, for those of you that are new to it, It comes from Ape Mithrandeer who sent 0101743045926

XMR and said, awesome new website. I really hope this means my XMR boosts get read out now. Well, Ape, there's living proof for you. Thank you for your support. And if you have no idea what Ape's talking about, head to ungovernable.network/v4v

and you can see the boost wall. You can get involved and if you do send in some sats or some Moneros, just like Nostagang just has. Look at that. Nostagang just did it as I said it. Oh, wow. You get a special feature at the top of the page. Look at that, Nostagang. Thank you very much. Was almost

biblical timing. I'm very, very happy about that, and I promise, Nostigang, I didn't

prime them to do that. So thank you very much. Although it did disappear pretty quickly, so while you guys are chatting, I'm gonna go make sure it stays up a little bit longer. Ape, thank you for your support. Nostagang, thank you also for your support. That was a perfect time. I'm really happy about that.

Let's let's dive into the show. Matt, Seth, how's how's it going? You had a good week? It's been a good week, for sure. Been an absolute blast across all fronts. Good time cake stuff. Good time AI cake stuff, which I'm excited to dive into today. Yeah. It's been good.

Good here as well, except we've been eating lettuce for the last week and a half and nothing more because I'm really worried that what Seth is gonna tell us is gonna make me spend money that I shouldn't be spending on some hardware.

So other than being malnourished, everything is good. Yes. I'm also very fearful. This show is gonna have the exact same outcome for myself. So Yes. I might need to give the wife the credit card to keep on the lock and key somewhere before I should've done it before the show, really, because it seems Keep it in cold storage.

Yeah. Absolutely. Okay. Well, no guests this week, as you can see. And it's just gonna be the three of us chatting, Seth leading the way. Every week we talk about taking back control of your money, your messages, your identity, you name it. You know, take back some control on today's digital Panopticon. It's literally in in the intro.

And today we're gonna add something else to that list, your intelligence. Quite literally, the AI that you use. It will come as no surprise to anybody listening or watching that we've all gone down the AI rabbit hole in varying degrees of severity in in the past six months.

And I think Seth very much takes the crown for chief clanker at the moment at the ungovernable empire, and we're going to find out why very shortly. But Seth, what's what's been going on? What have you been buying?

We're gonna do a little bit of show and tell. But I guess, let's just kind of start at the very beginning. Like, you've been on a journey, a very steep rabbit hole

in the the course of the last, probably what? Maybe two to three months, if probably less than that maybe. It's not been long. Yeah. It's not been long. He Seth was the laggard, the AI laggard, and he I hope he's okay with me saying that. And he's quickly leapfrogged

me and all my clankers to be like the the well, I I was gonna suggest say suggest saving

oh my god. I need him a coffee. He's the a he's the AI addict, I think. I was going to suggest changing his name to Seth for AI because, like, he's been bitten hard here. It's getting close. It's getting close to that. Yeah. Yeah. Absolutely. So so walk us through it, Seth.

Like, I guess, set the scene, and then let's get into what you've been up to lately. Yeah. It's been quite the quite the rabbit hole. This is this is my nature, though. I feel like I I find something interesting and that has, like,

the ability to be something that's really important for freedom or just really fun to play around with, especially if that those two things overlap. And I just and I I go in deep. But like you said, it's it's really weird because I was not the early adopter of AI.

Like, in general, absolutely not. Even among our more cautious circles, like you said, like, I feel like the three of us were relatively late adopters too, but Q was first, started driving on that, then Max, and I I fell fell much later.

But it was really interesting because I I I very much thought it was just not good enough, not good enough, not good enough, kept trying it. It was never matching what I felt like it needed to be to be useful. But Vic, Foundercake, my boss,

he's been pushing hard on AI stuff for a long time. Like, he was very, very early on the Open Claw train specifically, and that's what kinda opened up his eyes. And he started running his own models on Mac Studios that he had. He already had them for work, he so started with that, and then bought more before the the AI boom obliterated

the price of RAM, unfortunately. So he got them at a reasonable price, at least in today's terms. But he got started really on this super early. So he was running things locally, LM Studio, Olama, testing these things out with OpenClaw really early on. And that was the the start of our kind of local AI push at Cake, but was

long before I actually got into this. I think we've talked a little bit about, like, I first really started diving into AI probably three months ago, if that, I would say, in general, was was when I started doing it. And that was just the normal Claude code Claude co work, like, the the the normal route.

But about a month ago now, there was a big shift in the open weight model space, which is models that you can actually run at home, some of which are what we would call open source. Some of them have, like, commercially restricted licenses, which we would call open weight in the AI space,

kind of source viewable if you want in the the broader open source software space. And there was a huge, huge shift, and the models that were available went from, oh, these are pretty good, to, oh, this is actually competitive with some of the models that state of the art firms like

Anthropic and OpenAI are serving. Not quite frontier models. Like, we didn't have anything Fable level on the open weight side, but they were finally to the place where you could actually host a model that was fantastically effective.

And that, specifically, the model that launched the that made us go, okay. We need to start doing this locally is DeepSeek v four Flash. And so Vic and I had been thinking about this and chatting about it for a bit, decided to take the dive and get two NVIDIA DGX Sparks

and start with a cluster of two, because that was the the minimum size you needed to run DeepSeek v four Flash in a a performant way, at least at that point. You can run it on one quite well now, but at that point, that was that was what you needed.

And so that was the the real start of this journey. And and for those who don't know, like, you can, I think, often think of self hosted AI, and you're like, oh god? It's gonna be this, like, data center in my in my living room or something. How am I actually gonna host this?

But the the sparks and we'll get into other hardware GPUs, that sort of thing later. But the the sparks themselves are actually, like, they're, like, Mac mini sized. They're absolutely tiny.

They are beautifully built. Like, all all aluminum chassis, really, really nice front grill. Like, they're just really, really well thought out and built. And each one has a 128 gigs of unified RAM. So the same kind of idea as modern Macs, where you have a a unified pool of RAM for the CPU and the GPU.

Same idea here, where there's a a simple CPU and then a massive GPU in these, and then a bunch of unified RAM for them to share. And then I think with the Spark specifically is that if you want more than a 128 gigs of RAM, you can actually cluster them together over 200 gig

network ports that they have in the back. And so you can get two, you can just connect them together with the the one cable, and you suddenly have 256 gigs of VRAM

with a specific hardware set up that's optimized for running AI, which is the the way that we kick this off, and it's something that is very, very, very quiet and runs cool and can easily be run-in your house, which is a nice way to start for us, for sure.

So I wanna dive in a couple of questions there. Why the DGX Spark, and how much does one of these run you? Yeah. So the the Spark itself, MSRP is 4 k for each one, which I know that sounds like an insane amount, and it it is. Like, I think something that I'm really excited for is probably in the next four or five years,

AI specific hardware to come down in price. Right now, it's going up in price. It's a little bit of a painful time. Like, I think we spent 4,600 each for these, which is the kind of going rate right now.

But they're, yeah, about 4 to 5 k each depending on where you are in the world. And the reason why these specifically, because you can run AI models on anything, and you can run them on

a CPU with a little bit of RAM. If you have eight gigs of RAM, there are some small models that you could run that way. It's not gonna be fast, but you can run it that way. You can run it on a regular gaming GPU, especially if you have a higher end one with more VRAM.

If you have one with 16 gigs of VRAM or more, you can run quite performant models. But if you wanna run something larger and you wanna run something specifically that's going to be more intelligent and it's going to do a better job on long horizon tasks, like a broad

project that you're working on or a large code base that you need to scan, which is specifically why we needed this, you need a lot more RAM. And so the Sparks offer for for four k, five k, they offer a really good amount of VRAM and a a very much AI AI optimized

hardware setup that works really well for this. It's much more performant for our specific use case than Macs as well. I know a lot of people have been running, like, Mac Studios, like, Vixstarter with those. But these work better for that.

Because if you want to run the bigger models, you have to have enough VRAM or RAM generally to fit the entire model in, or else you're going to have very, very terrible performance.

So, like, DeepSeek is just the right size that I can easily fit DeepSeek v four Flash into the VRAM on the two d j x part cluster and have enough k v cache, like, basically memory or context.

You can think of it that way, where I can do very large, long horizon tasks on it without running out of VRAM. So that was the reason for the sparks over, like, a GPU rig, but there's definitely pros and cons to each that we can we can dive into a little bit as well. So do these

come preinstalled with, like, an operating system? Like, presumably, you're running all this headless. Like, talk us through that side of things. Yeah. So under the hood, they they run basically a custom NVIDIA fork of Ubuntu.

So they are they are at their core, basically, a Linux box that's specifically built around this idea. So the the OS has everything you need to get started when you power them on. They they basically they launch a Wi Fi hotspot that you connect to from your phone or your laptop,

and it guides you through the initial setup process. It downloads updates and does the initial reboot, and then you're you're off to the races. They have, just like a mini PC, they have an HDMI port on the back. They have Ethernet. They have USB c. So you can,

like, run it on your desk and plug it into your monitor and be able to actually manage and set it up that way, like, with a display.

But since I like, I'm gonna run them headless anyways, they're they're gonna be plugged into a switch in another room. I didn't want to bother with that, so I just set them up with this Wi Fi hotspots, and then they're plugged in via Ethernet. So then I can just SSH in and do everything that I need via SSH.

Although, in this in this day and time, most of the stuff I'm doing, I'm not doing manually. I'm I'm having AI set up and manage my AI for me, which is a a big part of what made this possible much, much faster. So you quickly moved to two. What was the

what what helped make that decision? Was it that you had your eye on a specific model and one machine just wasn't gonna cut it? Wasn't it two to four? Yeah. It was one cluster to two clusters. So, yeah, we started with two sparks right at the beginning, clustered together.

And then we just added, literally this week, we added the second cluster. So there's a lot of different ways you can approach it. For us, specifically, with our the security scanning that we're doing, we benefit a lot by being able to run multiple jobs at once. Like,

the core of what we do on the security side is we're doing nightly scans of all open PRs at their latest state, so we can catch security issues before they're actually been introduced into the the code base and tell our devs about them, give them specific remediation steps,

basically catching things before they would ever even be able to be released to the public. And then we also do weekly scans that are entire repo scans with no context to try and discover new or missed vulnerabilities

in the broader code base, things that have been pushed into dev that are are potentially going to end up in the next release. We obviously wanna catch those early as well.

And then on top of that, we do scan specifically against the tagged branch before we do a release so that we can have a third layer of verification to make sure that we're not introducing something unexpected from a security standpoint or that no outside contributor has stuck something in, all of the normal security stuff that we need.

And that that's a lot of things to be scanning and doing at once. So, really, we needed better concurrency.

DeepSeek v four Flash is actually really, really, really good at security scanning if you do multiple passes as well. So every time we're scanning anything, we're scanning it three times with no shared context between the three scans and then deducing and combining the scan results of each.

And so when you think about that, we're doing scanning every PR three times a night. We're scanning the whole repo, which our repos are very large and upstream repos as well, at least once a week in every tagged release. That's just a lot of jobs that we have to run.

So the the scale up from one Spark cluster of two sparks to two spark clusters of four sparks total allowed us to double how many jobs we could run at once. Rather than running a bigger model, which we we will do probably more when we get to eight sparks, I would guess, we could do now, but would take a big performance hit.

Right now, we're running two replica clusters. So, basically, it's two separate clusters that we we tie together with a router so that whenever you go to hit it, you you get automatically routed to whatever cluster's least used, and that let us double how many jobs you can run at once. So right now, we're

doing eight different tasks at once with full 1,000,000 context and really, really good speed. I you've kind of, in a roundabout way, already just answered it, and and I already know the answer to this, I think. But I wanna, like, dive into the economics of this a little bit.

So you're what? Like, the the better part of $20,000 in. Is that a fair quick math? Yeah. That is one hundred months of Claude Max subscription. So where do you see the that that's the $200 a month one, by the way, which is a lot of usage. Where do you see the the ROI coming in for for this?

And and did did that factor in? Or is it kind of a bit more of a a winged decision and it turns out, because I've obviously seen some of your tweets, that this is a very good decision for for some of the reasons that I'm sure you're about to go into.

Yeah. Let me see real quick if I can share my screen without doxing something. Hold on. Let me try this.

Because I think this would be helpful. Yeah. Go ahead. While you're doing that, I was just thinking as well outside of just the maths, isn't there also an element of you're running all of this locally and therefore you're not feeding Claude with it and that might be a better thing for your security?

So, like, there there is obviously, like, the raw number of it's four times this thing and that compared to the monthly, but but that must be a massive win for cake. Yeah. That's there's really, like there's three key ingredients that go into making this decision.

Woah. Should be Alright. Tony Stark. Fucking hell. Check this guy out. Again, I did not make this. So this this specifically, before I get answer your question, this specifically is a a Grafana dashboard that's using stats that we pull from our live cluster.

This dashboard itself was actually built by another community member in the local AI space who's been doing really, really cool work on the monitoring side. So I I took and adapted his.

But I had set up all the Grafana and Prometheus side for actually collecting this data before that. But to answer the broader question, so, like, where's the breakeven point? I I think there's there's three key things that often get overlooked when thinking about, like, oh, you could just just buy the subscription, you'll save money.

One, and I think something that people are finally starting to understand, those subscriptions are massively subsidized. And the amount of tokens, the amount of inference you're getting for your subscription is way more than the amount that you're spending. That is not going to last forever, and that is already quickly, quickly changing.

So there's been a lot of people noticing that subscription specifically to Cloud and OpenAI, their limits haven't they haven't said that they've changed, but they feel like they're getting a lot less work done before they actually hit limit caps. People are begging on Twitter all the time for limit resets and

trying to trying to get more out of their subscriptions. And that's because as as these companies scale up, as they aim to actually become profitable, they're going to need to get a much better amount of profit per token that they're they're sending out instead of doing this at a loss subsidized subscription model.

So the amount that you get for a subscription is going to continue to go down, not up, from everything we can tell, at least among the The US frontier labs. The second part of that answer is there are just huge advantages to running this yourself on prem from a privacy and sovereignty perspective.

Like, specifically for us, we're able to to do things like build, execute, and validate proof of concept executions against vulnerabilities that I just I wouldn't do if I was trusting Claude or ChatGPT with this. I wouldn't want to give them sensitive info on findings that we have or

even give it actual seed phrases with money to play with to actually do tests or do QA testing of features before we release them. Like, there are are privacy benefits to this. The second is that a lot of the the best security scanning that's available today is not available through ChatGPT or Quad.

And accessing these models would normally require going to, like, using DeepSea's API, which is an API observed out of China and obviously not something from a a privacy or freedom perspective that would be would be ideal for us to be sending everything out to. But the third thing as well is that this allows

us as a a company to do much longer running and extensive tasks than we would ever do if we were paying API prices for this. Like, none of the stuff we're doing can you can do through a subscription,

at least not easily or well. Because you're not allowed to use your subscription via API through something like Py or through a lot of the other harnesses. You have to use Cloud Code, or you have to use Cloud ChatGPT codecs.

And this allows us to build things that we were just never able to do, like a Slack bot that we have in our Slack, where any engineer, after they build something, can just type slash QA, a PR number, and ask a bot to actually run automated QA through every step of whatever flow they're testing, send screenshots, send logcat,

give info on every step that it that it tried, let them know if it failed or succeeded, things that actually running through, Claude's version of that, which is Claude Tag, are horrendously expensive, like hundreds of dollars per run. We can now do

for free. We're just paying energy. You can see that since we bought this cluster, since I set it up and started collecting, we spent $6 in electricity. Yes. We spent 20 k on the initial,

cost of hardware. But if we had been running these scans with Opus five, which is the minimum we would be running with if we are using one of our existing subscriptions, we would have already spent 3 k on

AI usage so far in one week. One week. And that is only gonna scale up because we've only just gotten our security harness really optimized for this specific setup and been able to move off of other subscriptions and using OpenRouter, that sort of thing.

So it it is going to be a an easy breakeven, I think, but that is only gonna become even more clear as the subscriptions that we do have access to become more and more neutered as these Frontier Labs need to find profit somewhere to satisfy investors.

Wow. I had no idea it was gonna be that quick of a turnaround, you know, if you extrapolate that out to where you would meet, you know, the 20 k outlay and, you know, a couple of bits of change for some electricity.

It's not gonna take long at all. Right? Yeah. And it like, it it will differ per person. Like, obviously, this is a very different workload than maybe you would be doing if you're just kind of vibe coding stuff here and there and managing your self hosted infrastructure and that sort of thing. Like, you're not gonna want

20 k worth of hardware for that. That that wouldn't actually make any sense for you. Yeah. But even, like, OneSpark can run this model and run it quite well, and that's something that you could use to serve yourself, your friends and family, and uncle Jim

AI in a way that's perfectly privacy preserving, and you you can never hit token limits. As long as it's online, you can serve tokens twenty four seven. No limits. No no issues with that, and do it in a way where you can ask things that you wouldn't ask otherwise. You can feed it data, like personal

photos and information about your family to to do something cool for your wife, for instance, that you would never do to Claude. You would never give it private photos of your kids, but you can do it because it's running on prem and it has no access to the outside world. So there's there's a lot of other things that get opened up by it.

But, yeah, it's definitely a different scale. Like, we're obviously gonna be using way more tokens, as a company than

than we would you would ever be using on your own. So maybe the breakeven looks a little bit different there, but maybe that's where we can get into, like, the the different levels of hardware that's available too because you don't have to go straight to a Spark.

There are other models you could run on lower end hardware that are gonna be good enough for the the regular day to day usage for a lot of people as well. Before we get into the different types of hardware, I'm just interested for our own purposes, ungovernable.

Obviously, what you're doing is more like pen testing and, like, shadowy supercoder stuff. For someone like me who needs to do a lot of design work, image gen, editing, audio stuff, visual stuff, do you know what the open weight models are like or have any insight into whether those run as well as the kind of

pay monthly subscription types. Because I'd be very interested in doing this, a, from, like, a privacy perspective because it all gets a bit complicated, but, b, just being able to have these things running overnight and dial things in to how we work a little bit more.

Yeah. That side of things. So I will caveat with the fact that I don't do any of that side of things with AI generally. So it's not an area of expertise, but from the the circles that I run-in now that I'm a an AI influencer,

I have heard really, really good things about specifically running a model that just dropped last week called MiniMax m three, which is I've heard MiniMax is good. Mhmm. Yeah. It's I've seen really, really, really good content coming out of it, and it's people generating that content on Spark specifically.

I think probably, like, a two times like, a a a two Spark cluster would be ideal for really good performance. I think I saw that you can run it on one and run it pretty well. And, again, if you're not time constrained, like, you can just kick off jobs and let them run while you do other stuff,

you probably don't need a a larger cluster. But that model specifically, I've been hearing really, really good things about for image and video gen, like long form videos as well.

A ton of really good visual content creation stuff there. So that is definitely a side that is is coming along at the same time, and it seems like at the same pace. Like, even mini max, I've heard, are are coming out with their next model in 3.1 or something

very soon as well, which is gonna be even an improvement over that. And as far as I know, remain at the same size and be something that's very self hostable on on something something like like a a Spark. Spark.

Interesting. Yeah. It's it's pretty crazy. Any even like smaller models like Quinn 3.827 b, which the 27 b refers to how many parameters there are, it needs, like, 17 ish gigs of of vRAM or RAM to be able to to run pretty well.

That's something like you could you could run on a consumer level GPU. I mean, you'd need a higher end one, something like a forty ninety or a fifty ninety. So quite expensive these days, but that would be really, really fast and be good at that kind of thing too.

But it it sounds like from what I've heard that that mini max m three is the really the state of the art in the the open weight image and video generation side and something that I've I've seen people run, and I've seen people run it. I know there's a there's a recipe for running it on two Sparks alongside DeepSeek,

so you can still get the best of the the coding and general AI usage from DeepSeek and then FireOx jobs via MiniMax m three at the same time. That's exactly what I was just about to ask was, you know,

most of my work is is, like, visual audio stuff, but then I do need to do normal stuff. So that would be ideal then if you could run both. Yeah. I think you definitely can. I'll have to dig it up, and I'll I'll try to send it to you because one of the people I follow has been working on that. But it's it's pretty crazy.

Okay. I'm glad I've been eating salad then. Great. Just my Is salad like the British version of of eating ramen when you're poor? Or, like, what's the never Well, salad's even cheaper. Salad's even cheaper because ramen, you have to ramen, you have to actually buy or shoplift.

And if you want salad, you can just go and pick it or grow it in your garden. So it's the cheapest of cheap. That's true. I just I I picture Max, like, just going out and grazing. It's just the Just like a fuck. Way he tortures.

Big cow in a field.

Not far off, mate. I wanna, like, go in into the sets a little bit more. Like, you mentioned it's quite simple. You just plug it in, give it some Internet and some power, connect to the hotspot, and then you you you kind of a way choose your model, obviously, and then you're away to go.

Obviously, you're using this in a power user setting, and you're not the only user of the cluster that lives at your house. Right? I presume the whole cake team are using that. So walk us through that. Like, is is that, you know, in a in like a private network? Or how how is everybody else in cake benefiting from this cluster as well?

Yeah. For sure. So the the first thing that I should have mentioned a little bit before too is, like, after you do that initial setup, obviously, then you need to actually deploy the bottle. There there's a lot that goes into that. You need a router that's going to serve it, something like the LLM.

There's a lot of other things that go on top, but the the really nice thing, and this is like, if you've been in the self hosting space or, you know, I'm sure this kind of stuff happens in the the Mesh Jadel and and the Bitcoin space, like, people who do this stuff for fun and explore, they publish the work that they do,

and they open source it. And so I had already found a really good recipe, which is basically just like a a runbook of how to deploy the specific model I wanted on the specific setup that I had by someone named Maya AI, who's a a Bitcoiner, M and A person, and a Cakewell user, which is super cool.

She's she's popping up in the the local AI space like crazy right now, but she had already made this recipe and published it. So I had just set up a plan using Claude because I was the AI head at the time. I had set up a plan for cutover day one. I'd already downloaded the the model weights. I had everything ready locally.

And so I literally deployed DeepSeek v four Flash on these two sparks while I made dinner, ate dinner, and got the kids down for bed. And then when I was done getting the kids down for bed, I had DeepSeek v four Flash serving at home.

Like, I I didn't really have to do anything. I think I had to answer answer Claude's questions twice, and then I had the entire cluster spun up. And that's one of the really beautiful things here is, like, it the the actual barrier to entry of deploying this and managing it is quite low because of how AI helps to improve it. Like, even

I'm running constant self improvement jobs on the cluster that work on fixing itself, and that I I use it to deploy monitoring to improve it. I used it to deploy all of this Grafana dashboard that you see by just telling it, hey. This repo is awesome. I really want this dashboard.

I already have these data sources. You're gonna need to add a few more, figure it out. And it was able to deploy it all and get it set up and working properly.

So that that is a huge part of this, is the actual setup itself is is getting much, much, much, much easier, and the community around this is awesome. Like, it reminds me a lot of the the early days back when I was doing custom Android ROMs a long time ago now. It seems like a lifetime ago.

But it's the same kind of feel where it's like a community of people who just want to publish and talk about the fun things that they've been doing, and it makes it way easier for everybody else to be able to get started. As far as, like, how this is actually served, so,

yeah, I mean, it's really straightforward. So, like, I'm already I already have been self hosting stuff for many, many, many years. So I already have a good self hosting setup. So I I'm just exposing the the VLLM router, via traffic, which is a a Docker reverse proxy, essentially.

And any of the team can hit it, but it's API key restricted. So I just create API keys as teammates need it, and they can hit it directly. You could also obviously, you could set this up over something like Tailscale if you wanted to.

I didn't mind my teammates being able to to hit the stuff I'm hosting at home because the cluster's at home. So I didn't have a problem with that. And if you do tail scale, specifically in a work environment where you're often also using VPNs, it can be a little bit of a mess on some OSs.

So I didn't want to introduce that headache. So, yeah, quite simple. Just an API key that a person has. Again, you don't have to necessarily know exactly how to configure this. You can just say to to Claude, hey. Add this

endpoint to my py installer to open code or whatever. Give it the API key, and you can plug it in. So that that part is actually quite straightforward and not complex, because it's just served in an open API an OpenAI compatible API format. So any literally any harness

can connect to this and use it. It's not anything super customer specific. That makes sense. Alright. Well, the the next part of the the minefield for me, generally speaking, if I was to walk down this route would be, you know, obviously hardware is one of the main gating

factors of what you can run-in terms of models and sizes and capabilities and stuff. Obviously, that's the the the biggest kind of primary factor essentially. But let's assume that you have some beefy hardware and it's, you know, one spark or a cluster of sparks.

How do you then go and decide which model you're gonna run, which set of parameters does it have? Is it gonna be quantized? Like, there's so many different you know, you talked about KV cache and things like that. Like, how did you navigate that journey? Because that's where I always become unstuck.

Yeah. I mean, the first part is I'm a weirdo, and I really like diving into rabbit holes like this and learning completely new things. So, like, I've just been absolutely loving that side, Like, getting into the technical details, following the new model releases, learning all about what all of these different terms mean.

So I'm I'm going to have gone deeper than most people who would self host this would or should go. Like, there's not a lot of reason to learn exactly what a KBCache is and why it why it works and why it's necessary for the average person.

But the like, the I think the thing that also is way easier than it seems at first is picking a model. Because picking a model is as simple as if you already have hardware and you're just trying to run something on your hardware, you just need to figure out how much VRAM or RAM do I have,

and that's going to limit what you can actually run. Like, for instance, if you have let's say you have a a gaming rig. You already have a gaming rig with a 59 d and an NVIDIA 59 d. You can easily run any of the 27 b, so 27,000,000,000 parameter models, things like Quinn 3.827 b. You can run that easily

in VRAM with a good sized KBCache and have a really good performant experience.

If you have something like a Spark, I mean, you you have way more options, because I can run anything from a very small very small model to a much, much larger model because the amount of VRAM that I have. But that's a side where it's also getting much easier as the local AI space takes off.

Like, it seems like really in the last three to six months, this space has absolutely exploded and gotten much, much more complete. So, like, even there's a new a new site that I've been playing around with and and got into early called local.ai.

And the whole goal of it is that you tell it what hardware you have, and it tells you what models are are possible to run and how they compare in terms of intelligence, speed, content generation, like what what each thing is good at.

And they do their own benchmarks, and other people can can recommend models as well. You can see what the people that you follow or keep up with are running. Like, you can go to my profile there. You get to see that we have four Sparks running DeepSeque v four Flash. It it'll be, I think, a really good hub for that.

So that's something that I'm excited to to see as well. They just launched and are are expanding expanding that side. But the other one is just like when you got into Bitcoin, just like when you got into self hosting, you you find a good community, and you ask questions.

And that's a fantastic way to learn.

And you'll you'll meet people who are running specific models, and they can give you feedback on why you should or should not run that, and what works for this use case, and, oh, should you buy new hardware, or can you get by with what you already have? Like, that that side, a a real part of it is a community. And I've been really, really

impressed and enjoying the Local AI community lately, because it feels like a lot of the best parts of the Freedom Tech space and the cryptocurrency space and the

the self hosting space kind of merging into one in a really fascinating way. So that it's it's something we're just getting plugged in, I think, will be the the best fit for figuring out what exactly is right for you or just asking somebody.

Like, if you have hardware and you're curious, like, just shoot me a message or post on Twitter and tag me or, like, that there there are lots of people I'm sure would be willing to to answer those questions and help out.

We've got sovereign logos in the comment on the x saying, what local AI community is recommended? I think you mentioned the domain, but let's just go back over it. Or if there's multiple, let us know.

Yeah. So I don't have, like, a I'm not, like, in a Telegram group or Discord or something. I just I don't have time for group chats to just kinda deal with sorting through them and finding what's useful for me.

So most of the community I found is on on X, and then that local.ai site is a really good resource as well. So I first, like, got kind of connected into this space because there was a there's a guy who goes by Serro, zero zero x, s e r o, on Twitter.

And he did a podcast actually with Alex Gladstein from the HRF,

and he did some stuff at Presidio Bitcoin. He's he's been kind of starting to get connected into the Bitcoin space. And so he was kind of, like, my gateway, and then I just started following a bunch of people that he talked about or that he recommended, and that kind of started the the network of local AI stuff for me on X.

That's a good idea, though. I should probably post some good follows that I found after this and and help people find find good community because there's a lot of really, really good people doing work in there. Alright. So I wanna

in a second, I'm gonna take us back to basics so that people can get some ideas on how they can test this out in a bite sized way. But before we do that, you guys, you've got your cluster up and running. You you know, you've jumped through all the hurdles, you've done all the network.

The cake while the team can cannot access it even though it's at your house, which is great. And you seem to have coalesced on a model that you're happy with. How do you compare that to in terms of benchmarking and performance compared to the latest frontier models? Like, are you getting Fable equivalent outputs?

Surely, I'm guessing you must have done some comparisons there. What's that been like? Yeah.

I mean, it's gonna heavily depend on the model you're running. So I'll just specifically talk about what we're running, which is DeepSeek v four Flash, which is a a recent model. They just they had it in preview for a few months, and then they released the the final version.

I think it was July 31 they released it. So, again, very, very recent, and there's been massive shifts in the model space even since then. And it looks like there's about to be another massive shift as of some new stuff that just came out last night. So this space moves really, really quickly.

But where we're at right now with DeepSeek, I would say the the the raw performance, like, in terms of intelligence, is somewhere between a a Sonnet five in Claude and an Opus five, leaning a little bit more towards the Opus five, but not quite as good.

So it's definitely not fable level, but it is more than good enough to accomplish the vast majority of tasks a regular person would have. And what I have found is the the absolute best way to use this is if you have a very large task that you need to do or you have a large project or you have a large code base you need to work with,

the the best pairing by far, right now, at least, until the models you can run at home become Fable level and greater, which they absolutely will. Like, we're already getting very close to that, like GLM 5.3,

which you can run on four sparks, which I know is still a lot today, but is going to keep coming down. That model is is legitimately Fable level, in many ways. With DeepSeek,

the best approach that I've found, and this is something I've been playing around with and just published a plug in for the the harness I use on my Pi, yesterday, actually, for this, is to actually pair a state of the art large model, something like Kimi k three or GLM 5.3 that you have API

access to. So that could be via OpenRouter. That could be via Venice. That could be via NanoGPT. Pair that just as an orchestrator and planner with your local AI model as the actual worker that does all of the grunt work, that actually implements everything, that does all of the research,

and that just provides output back to the orchestrator to to manage and to handle. And that pairing of combining a, like, Fable level orchestrator with a locally run and thus free, you're not paying per token, worker gives you

essentially all of the performance of a Fable level for everything plan because Fable is actually finding and telling the workers to fix any issues that they make, and you'd spend drastically

less. So you can spend just a little bit on the API subscription that you're using for the the the large model, k m a k three, h l m 5.3, and then do 95% of the work with your local worker. And I think that's the model that that people are going to use in the short term until we can self host,

even more easily these very large, models that are good at long horizon tasks. But even that, you only need that for, like, very advanced stuff. Like, I'm never touching that for any of my self hosting, for any of the,

show planning I'm doing, for a lot of the work that I'm doing. I'm just using DeepSeek directly. I I don't need even the orchestration piece for that, and it handles those jobs really beautifully.

But when you need that tier up, you have something like that, and you can still make the most of your local models at the same time. So that's kind of where I see it fitting in. But like I said, this space is changing so frequently that even, like, deep seek is gonna become kind of outdated soon. That's this is a this is 21 old model.

And rapidly, the space is changing and better options are coming out, and are only accelerating at this point as well. Nice. Alright. Well, let let's take a take a step back, and let's look at how people can get a taste of this without shelling out, you know, $4,812.16, $20,000.

And, obviously, they're not gonna get the same level of performance that you and your cluster are gonna have, And it also depends heavily on what they wanna do with AI.

So there's a lot I know there's a lot of variables, but let's say somebody wants to you know, they like the sound of this and they wanna dip their toes in before they make maybe a big investment into some proper hardware. Like, where would you advise they start?

You know, what are some reasonably priced hardware, assuming that they don't already have something that they could repurpose to to kind of get their toes wet and see what this is all about?

Yeah. That's the really nice thing here is, like like I mentioned earlier in the show, you can you can run local AI on pretty much anything. Obviously, the performance is where it's going to differ greatly.

I would say if you if you already have a gaming computer at home, you already have everything you need to at least try out local AI and get a good feel for what it can be like. If you have, especially, a higher end gaming PC, like a an NVIDIA thirty ninety and up, like a thirty ninety,

forty eighty, forty ninety, fifty eighty, fifty ninety, you're gonna be able to run really, really good models in a very performant way. Because you can run, like, that that latest Quinn 3.827 b model, which is fantastic.

Like, it'll it'll do all of the kind of Hermes OpenCLOS style stuff, all of the the smaller coding tasks that you would have at home,

all of the the automation, it's gonna knock all of that out of the park. And it's actually going to be very, very fast. If you have a good GPU that you can fit the entire model into VRAM with, it's going to be actually more performant than the Sparks.

I think that's something that also people at, like, in the local AI space, people are always fighting about, like, do you buy a GPU or do you buy a Spark? They're very different things. Like, you you buy a Spark when you want to run very large MOE called mixture of experts models.

So things like DeepSeek v four Flash, things like GLM 5.3, things where you need a large amount of VRAM. Like, for even for DeepSeek, which I would say is the smallest, like, highly intelligent model, you need 256

gigs of of VRAM. Like, you you need a lot. You would never be able to get that by stacking GPUs unless you're gonna spend a massive amount of money. Like, you'd have to buy four NVIDIA RTX Pro six thousands, which would be, like, $60,000

right now to be able to run those in a, like, a a GPU style setup. And that would be much, much faster, but that's a lot more money. That's three, four times the money. So I think that the best starting for starting point for people is, obviously, if they have a gaming PC, great.

If you don't, you could build one and get something like a used RTX thirty ninety, which is gonna be, like, 600 to $800. And that will serve those really good small models, like the Quinn Quinn 3.81 that I mentioned,

in a really good fast enough way for home, like, in a in a speed that would be enjoyable and out of quality that would be enjoyable. And just like the rest of the space, those models are getting drastically better, it feels like, almost every day at this point. And there's a there's a big focus on small models becoming more performant as

AI labs are trying to get these to work well on phones, on laptops at smaller scale, which a GPU can serve much, much, much quicker. So that's probably the best starting point, something like an NVIDIA thirty ninety to start.

The best kind of bang for your buck is if you can do it, getting a fifty ninety building a simple rig with a fifty ninety GPU.

That's gonna give you plenty of VRAM for all of the small models, and it's gonna be, like, wicked fast. Very, very, very fast. It's gonna feel probably faster than any API usage you would be doing. And, again, we'll serve models that are plenty good for for home use.

So there's kind of, like you can, like, ladder up and pick what you want. Obviously, start small and start with whatever fits within your means. But it can also be something like I wouldn't be surprised if there's a world where

friends and family pool together to build a rig that they can all use, like, in the like, maybe not right now. I don't have friends and family who care that much about local AI yet, at least. But I could see that being a world where you can like, this is a a really, really easy thing to uncle Jim. You can expose this easily, do API keys,

and share the model, and you're getting much, much better kind of return on investment the more you're using it. Because you're only paying for electricity. You're not paying for token.

And so I could see a a world where people are, like, pooling funds to have someone host the AI for them, and then none of them need a Clawd or ChatGPT subscription anymore, and they can all all use that local setup and have better privacy, have no usage limits,

and have really self sovereignty. That's something I've been wondering about a lot. I think I've brought it up in the last couple of episodes actually is I wonder whether, like, a Meshtadel or a mining mafia or or both of them merge together sharing a couple of, like, self hosted models could work

and, like, how that would work in terms of privacy because you have a group of people who probably gonna wanna use this kind of stuff. We can get access to cheap power. We could pool in together potentially, but would it be a privacy risk as an example if someone's like

you know, if you're running this at home, then you're okay to put family photos on or projects that you might not share outside of your family or your self custody with Bitcoin or Monero or whatever else you're doing. If you're then feeding that into a shared piece of hardware, is it then gonna leak out and cause problems

because other people are using it? Does that make sense? Because it's like it's a trusted network Mhmm. Of people, a few 100 people, maybe 500 people who might use it. You trust them to some extent. You're close closer to some than others, but you're not gonna trust them as much as your wife and kids. So how does that work? Does it leak?

Yeah. I mean, you'd basically be in the, like, the zero data retention approach, which is pretty common in the the AI space if you're paying for API. Like, if you're using something like NanoGPT or Venice, a lot of the models you can get access to are not end to end encrypted,

where they they could see what you're doing, what prompts you're putting in, but they have a promise of zero data retention. You'd be in basically that same position. I I mean, I I definitely see a future where the hardware we can run this on also has trusted execution environments, and we can do perfectly into an encrypted

multi user setups. That's not where we're at right now. So you are still trusting the operator to not log prompts and log that sort of thing. But you you also have to think it's a little different. It's not like you're

it's not like you're using someone else's Nextcloud, for instance, and you're just dumping all of your family photos into it.

And they can just look at it whenever they want. It would be much more complex because essentially, you're streaming prompts and info to it, but the the model's not storing that. None of that's going on disk in the same way. And by default, none of that would be logged. But, yes, a malicious operator obviously could could see that.

So you would have to make sure that it's someone who's who's trusted and doesn't have a reason to do that, and also is good enough from a cybersecurity perspective that they're not gonna get pwned and someone else is monitoring that for some reason. Yeah. But I I think in the vast majority of these cases, that's a totally fine trust model

for this specific use case and even better than most of the other Uncle Jim stuff where you would be, like, blatantly trusting them with your data at rest. Right now, you're really just trusting them with your data in transit,

and you can limit what exactly you're revealing to it as well, if you are really concerned about that. But, yeah, you basically get that kind of, like, I'm trusting you not to log VPN style promises.

Infinity divided by 21, isn't the problem that you can only have one user at a time? No. So it it definitely depends on the hardware that you're running it on. If you're running it, like, the bare bones base level, like you're running a

an old Mac laptop with just enough unified RAM to be able to run the model, like, yeah, you're probably not gonna wanna have 20 users running at a time. But, like, for instance, our Spark cluster, we can at our specific usage,

we can serve eight users at a time with no performance loss. And you actually the way that these things work, you actually technically gain performance as you add users because of the way that they share cache and everything.

Like, when I'm I'm using it as one user, I'm getting something like 70 to 90 tokens a second. When I'm maxing it out and using all eight tasks to do a large job, I'm getting close to 200 tokens per second total output.

So these things actually scale quite well. And for the more kind of common AI usage in our cluster specifically, you could easily do 16 at a time, if not more. But one of the really nice things about if you're doing this via GPUs as well is you can actually usually do a lot higher concurrency.

So, like, if you're running a a 27 b model and a fifty ninety, for instance, I think I've seen people serving, like, 32 jobs at once out of that.

Okay. Don't quote me on that, but you can get quite high concurrency. And if you do, like, obviously, the bigger the hardware, the more users you can serve. Yeah. You're not serving you're not running one Spark and serving 500 people at once. That's not how it works. But these, like, these these systems also queue well. So if you have

let's say you have something that can serve eight tasks at a time and you have 15 people who you've given access to, they're not all using it at the same time 100 of the time. That's not how any regular person at least uses AI.

They're doing little bursts where they ask a little question, and it does a search job, and it gives them back feedback.

And those jobs can queue up. So if someone else is using it and there's no no slot available for you yet, your messages just get queued, you'd wait for a few seconds until the other person's job finished. That sort of thing. It's it's just like any other kind of server. It'll handle the wait.

If it can't handle that load at the time, it'll queue you up, and you won't just get dropped or denied or something like that. Sovereign Logos is back in the x chat saying, what's your take on the m five max laptops, the ones that have got a 128 gigs of unified memory?

They're just so freaking expensive for the performance that you get out of them. If your main goal is hosting AI with them, they're they're not really the best tool for the job. They can work. So, like, if you already have one or, like, your work will buy you a a nuts maxed out

MacBook or Mac Studio or something, yeah, you'll be able to run the models, and you'll run them well.

I think the main thing that I've seen and noticed in hosting these models on Macs versus on something like a Spark or a proper GPU rig is that they're they do pretty well on the the the decode speed side, so the side of actually processing info and giving you output.

They are much, much, much, much, much slower when it comes to the prefill side. So if you have, like, a really large job or large code bases or large tasks that you're having it run, you'll be waiting for a while before the task actually starts compared to running it on a Spark.

But I think for if you're needing to run large models, I don't really know why Like, I I would not prefer the the Mac laptop specifically. The Mac Studios are okay, but they're also you can't get them with high unified memory anymore, like five twelve gigs plus.

Theoretically, they're gonna come out with a new one that will have one terabyte plus of unified memory, which that actually I don't think they have the new chips in them either. I think only the laptops got the new m five chips, and the and the the studios haven't yet.

Yeah. You're right. Yeah. I think the new studio is supposed to have that one, but, yeah, they've they've been lagging a little bit behind and are getting hit by this this NAND shortage as well.

I have an interesting question. So, like, somebody wants to scale into this, and they get themselves one spark, And then they go away, and they they quickly hit the the the capabilities of it. They wanna run larger models or they need more tokens per second or something, and they wanna add

another machine to their cluster. But by the time they get there, the the industry, the hardware industry moved on or all the sparks are sold out and you can't get anymore. Mhmm. Good one. Can they get an equivalent

machine or the next gen machine or whatever that is and still chain it to the Spark and get something like the output that you guys are getting a cake?

So it will depend a lot. I mean, the the short answer is if you can build a very fast networking backbone between whatever the two devices are, the answer is yes. There's some actually, really really good experiments by, I'm black I'm blinking on his last name, but he's he's another cofounder of Local dot a I.

Alex is his first name. But he's actually been working on exactly this problem and idea of, basically, if you have one Spark, can you expand its abilities by connecting it to a Mac that you also have?

And it's that exact idea of, like, how do we how do we allow merging different architectures into one shared pool of VRAM for hosting models? And so that that's actually what he's been working on, and that's very

nascent, I would say. Like, I I don't wanna guarantee that that'll be something that'll be really performant, but it definitely is doable. And there's actually already really good tooling for just pooling AI resources in general across a network. I think it's called no. Was gonna say Exo, EXO.

I think that's the name of a different AI company right now. But there's a a tool I'll have to dig up that basically allows you to do this, where you you add each different unique device to a cluster, and you can serve models across all of them, at the same time. But it's

you'll always be limited by how much bandwidth you have between the devices. So, like, the reason the Sparks work so well when they're clustered is that they have they have 200 gig QSFP ports, like a special

Ethernet port on the back, and you use a 200 gig cable between them. And that's a massive amount of bandwidth and is is enough essentially for their memory bandwidth to be pulled and not not take a hit.

Technically, you would take a small hit because their memory bandwidth is, like, 270 gigs per second, but it's it's a non noticeable hit in real real world usage. So that 200 gig connection is enough for that. But if you're if you're doing something like the Mac to

Spark setup, the Mac has at most a 10 gig Ethernet connection. I think the Mac Studios have 10 gig.

And so you're you are gonna take some loss to performance, but it technically could work. I think the thing that, like, I'm definitely hopeful for with the Spark is if they do come out with, like, a a gen two that has 256 gigs of VRAM, as long as they have that same QSFP port, you can just connect them together.

There's no like, I'm sure that that will be extensible. But if you're going, like, totally different architecture, like, oh, I have a gaming PC and a Spark, right now, the the ways that you connect those things are a little bit limited,

but are getting better and better as we go. And there's also some cool work by a company called Tiny Labs, where they've made, like, PCIe stands, basically, where you can connect a GPU to some other system.

So, like, you can actually have a Spark, and you can get a GPU, like a fifty ninety, and connect it to the Spark and pool the resources over a very fast connection. So this is definitely it's it's definitely a developing space. It's not something that's, like, guaranteed right now, but the Spark specifically have really good

stock right now from everything I've seen are not sold out and have not been taking the the same price increases as GPUs in recent recent times at least. Awesome. Well, this is fast. We are at time, but I'm gonna do many more shows like this. And I'm sure your

experience is gonna change over time as we get new models and additional hardware and stuff like that. So definitely something that we intend to kind of regularly check-in with you on. But I appreciate you sharing the insights as always, mister for AI.

And, yeah, that brings us to the close of the show. So I'm gonna wrap this up, and I hope everybody has a wonderful weekend, and we'll be back for more FreedomTech Friday goodness at the same time next week.

Thank you for listening to Freedom Tech Friday. To everyone who boosted, asked questions, and participated in the show, we appreciate you all. Make sure to join us next week on Friday at 9AM EST and 2PM London.

Thanks to Seth, Max, and Q for keeping it ungovernable. And thank you to Cake Wallet Foundation and my Nim Box for keeping the Ungovernable Misfits going. Make sure to check out ungovernablemisfits.com to see mister Crown's incredible skills and artwork.

Listen to the other shows in the feed to hear Kareem's world class editing skills. Thanks to expatriotic for keeping us up to date with Boost's XMR chats and sending in topics. John, great name and great guy, never change and never stop keeping us up to date with mining news or

continuing to grow the Mestadel. Finally, a big thanks to the unsung hero, our Canadian overlord, Jordan, for trying to keep the ungovernable in check and for the endless work he puts in behind the scenes.

We love you all. Stay ungovernable.