Edit2: OK Per feedback I am going to have a dedicated external NAS and a separate homeserver. The NAS will probably run TrueNAS. The homeserver will use an immutable os like fedora silverblue. I am doing a dedicated NAS because it can be good at doing one thing - serving files and making backups. Then my homeserver can be good at doing whatever I want it to do without accidentally torching my data.
I haven’t found any good information on which distro to use for the NAS I am building. Sure, there are a few out there. But as far as I can tell, none are immutable and that seems to be the new thing for long term durability.
Edit: One requirement is it will run a media server with hardware transcoding. I’m not quite sure if I can containerize jellyfin and still easily hardware transcode without a more expensive processor that supports hyper-v.
As of my understanding, immutable systems are useful for Devices that are more bound to change, like a Desktop you actually use to install programs try out things and so on.
I do not see much benefit here for a stable server system. If you are worried about stability and uptime, a testing system does a better job here, IMHO.
Yeah that part confuses me too. It’s a NAS. Install something simple and whatever services you need and I can’t imagine it breaking any time soon. Shit as long as someone else has tested the software I’d be more than happy to install something complex… Which I have and has been running for almost 10 years now with no issues. FreeNAS has been rock solid for me and it sure as hell ain’t minimal.
Immutable systems are useful for separating the system and application layers and to enable clean and easy rollbacks. On servers the applications are often already separated anyway through the use of container technologies. So having atomic system updates could enable faster and less risky security patching without changing anything about how applications are handled.
This logic is kind of backwards, or rather incomplete. Immutable typically means that the core system doesn’t change outside of upgrades. I would prioritize putting an immutable OS on a server over a desktop if I was forced to pick one or the other (nothing wrong with immutable on both), simply because I don’t want the server OS to change outside of very controlled and specific circumstances. An immutable server OS helps ensure that stability you speak of, not to mention it can thwart some malware. The consequences of losing a server is typically higher than losing a desktop, hence me prioritizing the server.
In a perfect world, you’re right, the server remains stable and doesn’t need immutablitiy…but then so does the desktop.
Virtual machines also exist. I once got bit by a proxmox upgrade, so I built a proxmox vm on that proxmox host, mirroring my physical setup, that ran a debian vm inside of the paravirtualized proxmox instance. They were set to canary upgrade a day before my bare-metal host. If the canary debian vm didn’t ping back to my update script, the script would exit and email me letting me know that something was about to break in the real upgrade process. Since then, even though I’m no longer using proxmox, basically all my infrastructure mirrors the same philosophy. All of my containers/pods/workflows canary build and test themselves before upgrading the real ones I use in my homelab “production”. You don’t always need a second physical copy of hardware to have an appropriate testing/canary system.
I really like this strategy. I currently use proxmox for my home server needs, but I am curious what you use now instead?
I have condensed almost all of my workflows into pure bash scripts that will run on anything from bare metal to a vm to a docker container (to set up and/or run an environment). My dockerfiles mostly just run bash scripts to set up environments, and then run functions within the same bash scripts to do whatever things they need to do. That process is automated by the bash scripts that built my main host. For the very few workflows I have that aren’t quite as appropriate for straight docker (wireguard for example) I use libvirt to automate building and running virtual machines as if they were ephemeral containers. Once the abstraction between container and vm is standardized in bash, the automation doesn’t really need to care which is which, it just calls start/stop functions that change based on what the underlying tech is. Because of that, I can have the canary system build and run containers/vms in a sandbox, run unit tests, and return whether or not they passed. It does that via cron once a week and then supplants all the running containers with the canary versions once unit tests pass.
Basically I got sick of reinventing the wheel every time a new technology came out and eventually boiled everything down into bash so that it’ll run on anything it needs to. Maybe podman in userland becomes the new hotness next year, or maybe I run a full fat k8s like I do at work. Pure bash lets me have control over everything, see how everything goes together, and make minor modifications to accommodate anything I need it to.
It sounds more complicated than it really is, It took me like a week of evenings to write and it’s worked flawlessly for almost a year now. I also really really really hate clicking things by hand lol, so I automate anything I can. Since switching off proxmox, this is the first environment that I have entirely automated from bare-metal to fully running in a single command.
I’m incredibly lazy; it’s one of my best qualities.
That sounds pretty slick. I envy your scripting prowess. You really have to know your system top to bottom to be able to boil it all down like that.
I’m just beginning my journey into this whole space, and it’s really interesting how many different ways people have to deal with the same basic things.
I’m also incredibly lazy, so maybe more scripting is in my future! Thanks for taking the time to write such a detailed reply!
I certainly wasn’t just born good at this. Unironically if you want to learn how something works, try to automate it. By the time it’s automated you’ll understand basically every part of it at at least a basic high-level.
That is incredibly true. I try to automate everything I can. That’s where laziness is a superpower.
my other server is a cloud tho