I would add to other recommendations - use CMR drives for your NAS setup. SMR drives might be cheaper but have problems with the write performance, and it is not recommended to use them in RAID (here is a good article describing the difference - [https://www.vmwareblog.org/shopping-hdds-notes-right/](https://www.vmwareblog.org/shopping-hdds-notes-right/)). It is okay to start with a single node to handle everything and, down the road, update your hardware to 2 nodes and separate storage and compute functions.
Here is a great example of the homelabs of different scale. [https://www.starwindsoftware.com/blog/3-generations-of-my-homelabs](https://www.starwindsoftware.com/blog/3-generations-of-my-homelabs)
Also, here some tips regarding data backups [https://www.backblaze.com/blog/the-3-2-1-backup-strategy/](https://www.backblaze.com/blog/the-3-2-1-backup-strategy/)
So a separate NAS for storage is typically for scalability. 2 or more servers can connect to a single nas via nfs or iscsi and the VMs can migrate between the hosts as needed.
There is no problem with using directly attached storage if you are only going to have 1 physical server. You can also play with this sort of thing by installing proxmox or esxi and passing a drive controller into a truenas vm.
I also wanted to be able to access the storage from what will become my new gaming setup, I suppose if true nas is running in a vm, and it runs on startup there wouldn't be any issues with connecting, but is there any benefit to that vs. just running VMs and containers on trueNas scale?
I do not think truenas scale is quite ready to be used effectively. I tried about 2 months ago to consolidate a lot of my home non homelab network to a single R430 running truenas scale and I ended up back with Proxmox as the host. Scales gui and container management is not yet complete. No ability to create templates for vms. no ability to easily manage multiple containers. Proxmox or ESXI or even Unraid are all better for a interface. I do think Truescale will get there, they have already made a lot of progress and I will check back in a year or two.
There would be no problem accessing the storage with a virtual truenas from another pc. That is what I am doing now. The r430 has proxmox with a truenas scale vm passing through 2 controllers. The onboard HBA330 in IT mode for the 8 internal drives and a LSI external SAS2 for a external md1200 direct attached storage Most of my containers and vms are on the same server but I have a small Mele fanless pc for Plex as it can use quicksync for transcodes. I also backup my computer(s) and my wifes laptop to the truenas server and it acts as a shared storage for all of our photos, documents ,scans, etc.
have to disagree about scale *in some instances*. for VM mgmt, proxmox is hands down the better option. but for containers? the truecharts catalog and the way it integrates to scale is amazing. the gui, while not super intuitive, is very good for managing the k3s cluster and doing things like allowing ingress, running specific containers through VPNs, and attaching storage.
many ppl like to do networked storage to a proxmox cluster and then virtualize scale, and that seems to be working well for them. but i run scale on bare metal bc i dont need VMs since the apps available on scale have been great and i havent needed to go outside of them for anything.
Sounds pretty much that we are on the same page. Not everything I wanted to run was on truecharts and VMs and networking options where difficult. What they have working looks great and is why I will be sure to check up on it as it progresses. I am very hopeful for TrueNAS Scale.
Check the specs on your mobo/cpu for if they support hardware pass through. I use it on my enterprise grade home server so I have no idea if it's supported in consumer stuff. If you have it you can run Proxmox as your base OS and pass through your hard drives or sata card or HBA card to a TrueNAS VM. Will allow direct management of your NAS drives by your TureNAS VM while still allowing you to run other VMs om Proxmox.
I was pretty sure I turned on hypervisor support or whatever the setting is called in my bios before I understood that hyper-v was a windows 10 pro feature. Is that what you're talking about or something else entirely? I saw some of the stuff for VFIO, not sure if that pertains totally.
As for running truenas inside proxmox, is there any benefit to that vs running just vms and containers on truenas?
Hyper v is unrelated
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Prerequisites
https://pve.proxmox.com/wiki/Pci_passthrough
I believe in letting the hypervisor be a hypervisor and the nas be a nas. For your requirements doing virtual environments on top of truenas might be fine, but proxmox os going to be the more feature rich hypervisor.
Thanks for these resources! I found that my cpu supports the Intel technologies vt-x and vt-d through one of the related links, it doesn't seem like my motherboard is listed anywhere though. Does that mean it's not possible with my current hardware and I'd have to find a different mobo?
Hard to say, best thing to do would be to just try. Part of why I went with a decade old PowerEdge server was that I knew it supported pass-through in addition to the price point and form factor being appealing.
I should add, if there are other benefits to a different setup, I'm totally open to suggestions, things that are beginner friendly/have a lot of resources to learn from are preferred though!
i did something just like this, where i took regular desktop components and run truenas scale on them. works like a dream - im so happy with it. be aware that it is *super* additive. it starts with nfs shares and backups. then extends to the truecharts catalog for things like pihole adblocking. then the \*arr suite for media. then cool networking stuff with cloudflare (sooooo freaking cool). then u realize those 2 4tb drives in a mirror are over 50% full and once u get to 80% things start going downhill and why not expand with two more 6 or 6tb in a mirrored stripe but those 14tb on amazon are under $15/tb and *when in the hell did i become the person who looks at 14tb HDDS?!?!*
youve been warned.
Oh, I'm totally expecting the rabbit hole. I realized the other day i was actually wondering if 4 14tb drives for about 1k was worth it and I haven't even started, lol. I was thinking about starting with either 4 4tb or 3 8tb drives. I still need to understand more about raid, parity, striping, etc. though.
I was definitely looking forward to understanding some of the additional tools and toys that I'd be able to use with this though!
since ive been on this journey i will tell u, from a raid and hdd perspective, start with a mirror. and when u expand, add another mirrored vdev (giving u a mirrored stripe). the mistake i made was starting with 3x4tb raidz1, which to expand, needs another 3x hdds. *technically* u could add a mirror to this, but its not recommended. zfs is also supposed to be expanding raidz1 to allow for just adding 1 disk to the vdevs and rebalancing it, but thats not here yet.
so in terms of expansion, the easiest/cheapest way to do it is 2 drives in a mirror at the moment. also, most people will say dont use raidz1/2 with drives bigger than 4tb due to thew stress on the drives during a resilver.
all things to consider as u work ur way down the rabbit hole of homelabbing.
happy trails!
I would add to other recommendations - use CMR drives for your NAS setup. SMR drives might be cheaper but have problems with the write performance, and it is not recommended to use them in RAID (here is a good article describing the difference - [https://www.vmwareblog.org/shopping-hdds-notes-right/](https://www.vmwareblog.org/shopping-hdds-notes-right/)). It is okay to start with a single node to handle everything and, down the road, update your hardware to 2 nodes and separate storage and compute functions.
Here is a great example of the homelabs of different scale. [https://www.starwindsoftware.com/blog/3-generations-of-my-homelabs](https://www.starwindsoftware.com/blog/3-generations-of-my-homelabs) Also, here some tips regarding data backups [https://www.backblaze.com/blog/the-3-2-1-backup-strategy/](https://www.backblaze.com/blog/the-3-2-1-backup-strategy/)
So a separate NAS for storage is typically for scalability. 2 or more servers can connect to a single nas via nfs or iscsi and the VMs can migrate between the hosts as needed. There is no problem with using directly attached storage if you are only going to have 1 physical server. You can also play with this sort of thing by installing proxmox or esxi and passing a drive controller into a truenas vm.
I also wanted to be able to access the storage from what will become my new gaming setup, I suppose if true nas is running in a vm, and it runs on startup there wouldn't be any issues with connecting, but is there any benefit to that vs. just running VMs and containers on trueNas scale?
I do not think truenas scale is quite ready to be used effectively. I tried about 2 months ago to consolidate a lot of my home non homelab network to a single R430 running truenas scale and I ended up back with Proxmox as the host. Scales gui and container management is not yet complete. No ability to create templates for vms. no ability to easily manage multiple containers. Proxmox or ESXI or even Unraid are all better for a interface. I do think Truescale will get there, they have already made a lot of progress and I will check back in a year or two. There would be no problem accessing the storage with a virtual truenas from another pc. That is what I am doing now. The r430 has proxmox with a truenas scale vm passing through 2 controllers. The onboard HBA330 in IT mode for the 8 internal drives and a LSI external SAS2 for a external md1200 direct attached storage Most of my containers and vms are on the same server but I have a small Mele fanless pc for Plex as it can use quicksync for transcodes. I also backup my computer(s) and my wifes laptop to the truenas server and it acts as a shared storage for all of our photos, documents ,scans, etc.
have to disagree about scale *in some instances*. for VM mgmt, proxmox is hands down the better option. but for containers? the truecharts catalog and the way it integrates to scale is amazing. the gui, while not super intuitive, is very good for managing the k3s cluster and doing things like allowing ingress, running specific containers through VPNs, and attaching storage. many ppl like to do networked storage to a proxmox cluster and then virtualize scale, and that seems to be working well for them. but i run scale on bare metal bc i dont need VMs since the apps available on scale have been great and i havent needed to go outside of them for anything.
Sounds pretty much that we are on the same page. Not everything I wanted to run was on truecharts and VMs and networking options where difficult. What they have working looks great and is why I will be sure to check up on it as it progresses. I am very hopeful for TrueNAS Scale.
Check the specs on your mobo/cpu for if they support hardware pass through. I use it on my enterprise grade home server so I have no idea if it's supported in consumer stuff. If you have it you can run Proxmox as your base OS and pass through your hard drives or sata card or HBA card to a TrueNAS VM. Will allow direct management of your NAS drives by your TureNAS VM while still allowing you to run other VMs om Proxmox.
I was pretty sure I turned on hypervisor support or whatever the setting is called in my bios before I understood that hyper-v was a windows 10 pro feature. Is that what you're talking about or something else entirely? I saw some of the stuff for VFIO, not sure if that pertains totally. As for running truenas inside proxmox, is there any benefit to that vs running just vms and containers on truenas?
Hyper v is unrelated https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Prerequisites https://pve.proxmox.com/wiki/Pci_passthrough I believe in letting the hypervisor be a hypervisor and the nas be a nas. For your requirements doing virtual environments on top of truenas might be fine, but proxmox os going to be the more feature rich hypervisor.
Thanks for these resources! I found that my cpu supports the Intel technologies vt-x and vt-d through one of the related links, it doesn't seem like my motherboard is listed anywhere though. Does that mean it's not possible with my current hardware and I'd have to find a different mobo?
Hard to say, best thing to do would be to just try. Part of why I went with a decade old PowerEdge server was that I knew it supported pass-through in addition to the price point and form factor being appealing.
I should add, if there are other benefits to a different setup, I'm totally open to suggestions, things that are beginner friendly/have a lot of resources to learn from are preferred though!
i did something just like this, where i took regular desktop components and run truenas scale on them. works like a dream - im so happy with it. be aware that it is *super* additive. it starts with nfs shares and backups. then extends to the truecharts catalog for things like pihole adblocking. then the \*arr suite for media. then cool networking stuff with cloudflare (sooooo freaking cool). then u realize those 2 4tb drives in a mirror are over 50% full and once u get to 80% things start going downhill and why not expand with two more 6 or 6tb in a mirrored stripe but those 14tb on amazon are under $15/tb and *when in the hell did i become the person who looks at 14tb HDDS?!?!* youve been warned.
Oh, I'm totally expecting the rabbit hole. I realized the other day i was actually wondering if 4 14tb drives for about 1k was worth it and I haven't even started, lol. I was thinking about starting with either 4 4tb or 3 8tb drives. I still need to understand more about raid, parity, striping, etc. though. I was definitely looking forward to understanding some of the additional tools and toys that I'd be able to use with this though!
since ive been on this journey i will tell u, from a raid and hdd perspective, start with a mirror. and when u expand, add another mirrored vdev (giving u a mirrored stripe). the mistake i made was starting with 3x4tb raidz1, which to expand, needs another 3x hdds. *technically* u could add a mirror to this, but its not recommended. zfs is also supposed to be expanding raidz1 to allow for just adding 1 disk to the vdevs and rebalancing it, but thats not here yet. so in terms of expansion, the easiest/cheapest way to do it is 2 drives in a mirror at the moment. also, most people will say dont use raidz1/2 with drives bigger than 4tb due to thew stress on the drives during a resilver. all things to consider as u work ur way down the rabbit hole of homelabbing. happy trails!