What can you DO with 1TB of RAM?

[music] Hey, I'm Dave. Welcome to my shop. I started out with 4K of RAM, a cassette deck, and a rotary dial modem. And now I'm staring at a server with a full terabyte of memory, about $25,000 worth of are you sure, and 64 cores waiting like the starting grid of Leong. Why? Because somewhere inside this machine is the answer to a question that's haunted pretty much every other upgrade I've ever done. What happens when memory stops being a constraint and starts being a resource? Well, today we're going to find out by force-feeding a server a thousand GB in order to learn the basics of virtualization. Imagine your home lab is a Swiss Army knife that finally learned a new magic trick. Your one box suddenly runs Windows for your Blue Iris cams, Ubuntu for your home assistant nags, a playground Windows Insider build for tinkering, and a true NAS head to feed Plex like a fire hose, and even a quarantine lab for that flaky driver that you don't quite trust yet. all in the same box, same power bill and same fans. Well, the trick virtualization. And once you internalize what the word really means and how it differs from hypervisor and docker, you stop buying random little boxes and start just adding superpowers to the iron that you already own. And you certainly won't need a terabyte to get started. Our story starts with a very honest piece of iron, a 45 drive store Q30 that they supplied with a dozen discs and then it grew as home labs do by gradually filling every bay. Eventually, it held 30 14 TBTE drives, two HBAs, eight SSDs for metadata and hot tiers, and 384 gigs of RAM backing a Zeon Silver 4216. It spent its early life as a storage workhorse, saturating the network and being gloriously boring while ZFS soaked up RAM like a sponge. But then the GitHub Primes project came along where we test up to a 100 prime civ implementations across nearly as many languages every night. And that meant real compute work, clean isolation, and the ability to spin up, tear down, and repeat without contaminating the base OS. Storage only just wasn't enough for us anymore. We needed a brain transplant. 45 drives obliged with a surgical upgrade. Same store chassis, same HBAS and drive sleds and everything, but a new motherboard with a 64 core AMD Epic and a terabyte of memory. It's the same pickup truck we love, but now it towes the whole house. For home labers, that's the value proposition. You keep your know and good storage story and bolt on a virtualization platform that turns idle watts into useful services. But before we install Proxmox and start carving it up, let's make sure we're all speaking the same language clearly. Virtualization is the idea that one physical computer pretends to be many simultaneously. Each pretend computer is a virtual machine or a VM and it boots its own operating system, has its own discs and network cards and does not know that it's sharing anything. The conductor of this orchestra is the hypervisor. That's software that runs right on and talks directly to the hardware, schedules the CPUs, meters out the RAM, and presents virtual devices to each guest. When the hypervisor is the kind that installs directly on the metal, it's a type one hypervisor. Proxmox and Windows HyperV are good examples. When it rides on top of a host OS, think type two. That's more Parallels or Virtual Box. Now, in either case, the hypervisor is the ring master that convinces a dozen operating systems that they each have the tent to themselves. Now Docker is not that Docker is containers. A container shares the host kernel and isolates at the process and namespace level. Think fence and resource governors, not the fake CPU and chipsets. Your hardware is on the bottom layer and then on top of that sits the hypervisor. And on top of the hypervisor sit all of the various operating systems that you will install. And inside one of those, the kernel itself will be virtualized for multiple clients. And that is Docker. Docker's magic is that you can package up an app plus its libraries and configuration into a single image and then run that image anywhere that can run Docker. Startup is faster, overhead is lighter, and density is higher because you're not booting a whole OS for each service. But the kernel is shared. So if you need a different kernel or a different OS altogether, containers can't give you that. But VMs can. So the short version is that VMs virtualize the whole computer and Docker just virtualizes the kernel. You can use both and you often will on the same host and the trick is knowing which knob to turn for which job. So with the vocabulary settled, let's upgrade the storeator like we mean it. First, Proxmox. If you're coming from a storage only life, Proxmox will be a breath of fresh air. It's a type 1 hypervisor built on Debbie and Linux with a clean web UI, sensible defaults, and just enough opinionated tooling to keep you out of the weeds. On RQ30, the path is straightforward. Back up the Proxmox config, update to the current stable release. Proxmox sits at the base layer. Your VMs and containers live on top of that. Each guest sees only the slices you hand it. The cores, the RAM, and the discs that you specify, while the hypervisor does the time sharing. If you've never seen it, the feeling you get when you watch a Windows installer's progress in a window inside of a browser tab is actually pretty cool. And the first heavy lift is going to be storage. Now, we keep Trunaz scale as a VM because ZFS likes its own discs and because Trun has guardrails that I like. PCI pass through is a clean way to make that happen. Rather than handling 30 discs and passing them through the VM one by one, you pass through the disc controllers and then True NAV sees its HBAS and its 30 drives exactly as if it were on the bare metal. That buys your native performance and keeps your ZFS pools blissfully unaware of the hypervisor's existence. Give Trunz the RAM it deserves on our box. That's a fixed 256 GB reserved with no balloon. And now you've got your storage head in a VM with the full power of snapshots and backups at the VM boundary. It's the best of both worlds. Robust storage with the trivial roll back the entire VM safety. Then come the workh horses. We have a Linux VM for the services that want a full OS and a couple of Windows VMs for the few things that still insist on living in Redmond. A Mac OS VM for the odd Xcode iPhone app deployment subject to Apple's licensing which may restrict Mac OS virtualization to Apple hardware. So if you're not running on Apple Silicon or a Mac host, treat Mac OS as a learning exercise and legal gray zone, not a production dependency. On a CPU layout, resist the urge to hand out whole sockets like their party favors. Cores are time sliced wonderfully well. Assign virtual CPU counts to reflect the real work you need done, not vanity numbers. Now, Proxmox will let you overcommit the CPU safely. And for home lab workloads like web apps, CI runners, lowduty services, that buys you density. Memory is where you want to be more conservative. And yes, you can overprovision with ballooning limits. And yes, it works, but nothing ruins an evening like swapping on a host that you thought had actual memory to give out. You can also thin provision your VM discs, and you'll feel like a magician until you learn the other lesson. If you overallocate things aggressively, set monitoring so you don't wake up to a paused VM because the host ran out of real bites behind the scenes. Ask me how I know. Networking is your next lever. If your nick supports Srio, then carve VFS for the chatty guests and enjoy near bare metal throughput with almost no CPU tax. If not, Linux bridges are fine workh horses. Keep management on its own bridge and storage replication on another and client traffic on a third. Proxmox makes VLAN tagging pretty painless. So, this is a perfect time to finally separate IoT, cameras, and everything else if you have not done so already. Now for the nightly science experiment, the GitHub primes test farm. Why do we need a virtualization at all if Docker is so good at isolation? Well, because we do both. Every language tool chain lives in its own container image so it can't fight with its neighbors. The orchestration that schedules, times, and records results, runs in a VM, so it's insulated from the host, and gets clean rollback semantics. Containers are perfect for packaging a Go tool chain today and a Rust nightly tomorrow. A virtual machine is perfect for pinning the orchestration operating system and its tuning so that we can reproduce results over months without any surprises. A Docker images start fast, share common base layers to save space, and are easy to update on mass. Whereas the VM boundary gives us a crisp snapshot test and go back loop. On a good night, the farm lights up a dozen cores across 20 containers, slurps in the day's pull requests, and compiles and runs them in parallel, then drops a clean scoreboard before we wake up. It's all been fully automated by a couple of fellas named Ruter and Tutor without whom it couldn't have happened at all. If you've never used containers, here's what virtualize the runtime and not the machine feels like in practice. You take an application, say a small ASP.NET service or a Python fast API endpoint, and you codify its dependencies in Docker file and then you build an image. That image is layers of file systems. Each layer representing a step, the base OS, then your packages and then your app. And Docker caches those layers so that when you rebuild after a small change, it's pretty instant. When you run the container, Docker gives its own process name space, its own view of the file system, and its own croup and force CPU and memory quotas. All the containers share the host kernel, which is why they're so light. It's also why they can't run a different kernel. You can't run Mac OS or Windows in a Linux Docker container. For that, you have to go back and do a full VM. Now, if it helps, think of containers as shipping an app in a self-contained suitcase and VMs as shipping the entire apartment fuse box and all. Let's bring back that mental model to our Epic Storeinator. At the bottom layer is our new hardware. Above that is Proxmox, our type one hypervisor. On top of that, two strata full VMs for operating systems and containers for services. We use VMs when we need different kernels, strict blast radius boundaries, or the ability to pause, snapshot, and roll back an entire operating system. We use containers when we want 10 microservices to spring to life in under a second and share base layers and scale horizontally without any guilt. And because Proxmox supports both KVM virtual machines and LXC containers, you can mix them pretty freely. For services that don't need a full VM's isolation, but do benefit from Linux native containment, the LXC method is wonderfully light. For anything that expects a Docker ecosystem, install Docker inside a small Linux VM and drive it from there. We actually run it in our main Windows system because it's just easier to administer one big system, but that's just how we do it. You'll end up with a tidy split usually osshaped things in VMs and appshaped things more in containers. And here's what life looks like after our big upgrade. Our true NASVM still has and owns the HBAS and all 30 discs. It gets that 256 GB of memory locked from the host, and it keeps ZFS Arc fat and happy. A Windows VM handles desktop class odds and ends from the occasional firmware flasher to that single vendor utility that only ships as an MSI. The GitHub Primes orchestrator lives in a VM with pinned virtual CPUs so that nightly runs finish on time. And because their containers they rise from the cool to start serving traffic in the time that it takes me to sip coffee. Now, because the orchestration VM is a VM, I can snapshot the whole environment, test the change, and revert it if I don't like it or if it misbehaves. It's a clean separation of concerns enforced by the hypervisor and made nimble by containers. Now, a few practical notes that home labers learn along the way that aren't fun if nobody tells you. First, time is important. Jeff Gerling will tell you this. Pin NTP carefully. Let Proxmox be your stratum inside the rack and don't let the guests fight it out over who's the boss of the actual wall clock time. Second, back up like you're a pessimist. Proxmox makes scheduled snapshots and offhost backups trivial, so use them. test restores. Nothing wins an outage day like a one-click VM restore that just works. It's hard to overstress how much easier it is to just roll back a VM than it is to restore Windows from a backup or even have to reinstall it. What used to be an afternoon collapses to mere seconds. Third, decide where your state's going to live. For us, that's true. And everything else is cattle, not pets. If a container's config matters, it's in Git. If a VM's disk matter, it's on a ZFS data set with snapshots and it's replicated. Fourth, be intentional about Numa on the big epic boxes. Keep a virtual machine's virtual CPUs and RAM on the same Numa node if you can, or you'll pay a latency tax that you didn't expect. And finally, the very human part. This upgrade didn't replace tinkering. It enabled and focused it. Instead of stutterstepping around a single OS that sort of does everything, I now give each workload exactly what it needs and let theuler figure out the rest. On some days, the store nater loafs along at a few hundred watts and all the VMs are pretty much idle while the NAS serves plex. At other times, like when the nightly prime saves run, CPU load spikes, a dozen containers light up, and the box earns its keep. The point is that nothing else in the rack changed. Same chassis, same discs, new brain, and a new habit of thinking in layers. So, what's the difference practically between a virtualization, hypervisor, and Docker? Well, the hypervisor is the referee that slices the hardware up and hands those slices to the guests. Virtualization is the trick that lets each guest pretend those slices are a whole cake. Docker is a toolkit that lets you ship your machine to the customer, basically, including your config, while still sharing your kernel with his neighbors. Put them together and you get a home lab that looks and feels like you've multiplied boxes in the night, but without multiplying the clutter or the number of little power bricks. If you're staring at your own storage box and wondering, could I do this? The answer is probably yes. Despite the name, 45 Drives makes boxes as small as four bays and as large as 60. So, you can be pretty much assured that they make something that will suit your needs. You can also customize the internal buildout as we did in order to perfectly fit your situation. If it is a 45 unit, then the upgrade path is downright civilized. You keep your sleds and your HBAS and you swap the mainboard for something with more lanes and more cores. You feed up memory until the dims beg for mercy and you let Proxmox do what it was born to do. The first time you pause a Windows VM to free up cores for Linux build farm or snapshot an entire service VM before you apt upgrade, it'll feel a bit like cheating, but it's not. It's what modern hardware was designed to do. And if you want to just dip a toe in the waters, you do not need a rack and a store to start. Installing Proxmox on some old retired desktop will teach you more about real world virtualization in a weekend than a month of white papers. Spin up one Linux VM, one Windows VM, and an Lex container and give them little jobs to do and watch how they share. then add Docker for the apps that fit that model best and feel the difference. If you fall in love with it, well, you'll know exactly what to do with your Storinator when you get one. Thanks for joining me out here in the shop today. Here's a link to one of my other episodes that the algorithm predicts you'll enjoy. And if this episode helped you make sense of virtualization, then the upgrade did more than just speed up our prime sibs. It gave you a new way to think about your own hardware, which is the most valuable upgrade of all. Please consider subscribing to the channel if you enjoy this kind of thing. In the meantime and in between time, hope to see you next time right here in Dave's Garage. >> Do it. Do it. Do it.

Dave installs 1TB of RAM in our 45Drives Storinator and uses it to explain how virtualization works, and what the difference between hypervisors, virtual machines, and docker containers are using real world examples. 45Drives Servers: https://www.45drives.com/products/hardware-storage-and-virtualization-servers/