Purpose of this post
To show off and explain my current set up. I don't get to talk about my home lab much.
Considerations
There are a few things that dictate what goes into my set up, and what I am comfortable using in "production"
- Must be truly self hosted, and cannot rely on external servers to function. I don't like the idea of plex requiring external authentication, or using Googles OAuth in front of my services.
- Security. Only I and my partner use my set up. The only acccess to my lab/set up is via a VPN. Everthing else is blocked by default.
- If it doesn't need access to the internet, it doesn't get access to the internet. For example, why does my printer need to phone home? Why does my CCTV system need to talk to the manufacturers servers? These things are blocked by default and cannot talk to anything on the LAN.
- Before I run a container, I will go through the dockerfile to try and understand what is going on. I dislike blindly trusting things. While I'm no pro, I can get a pretty good idea of what's happening.
Hardware
ESXI Host
Dell Precision T5600
Dual Intel Xeon E5-2667 Processors
112GB ECC RAM
1x1TB SSB for VM Storage
2x3TB HDD's, RAID1
1 built in gigabit ethernet port plus a intel quad gigabit card
NVidia Geforce Quadro P4000 Graphics Card, but will be replaced by a AMD R9 280X because AMD don't pull stupid manouvers with their graphics card drivers
Freenas Host
HP Proliant Microserver Gen8
Intel Xeon E3-1260L
16GB ECC RAM
2x3TB HDD
2x12TB HDD
Total Usable is about 13.6TB
pfSense Host
Noname Mini PC from Aliexpress
6 intel gigabit ports
4GB RAM
32GB SSD
Switch TP-Link 16 port gigabit managed switch
Access Point (Add in later), running OpenWRT
Software / VM's / Containers
Heimdall (Container) A dashboard for all my internal services and hardware. I use this instead of bookmarks as I just need to update one location. It's not dependant on a browser which is ideal for my use case.
Portainer (Container) Docker front end. Works well. Is connected to my DC's for SSO.
Leantime (Container) Project managements
Snipe IT (Container) IT Asset Management
phpIPAM (Container) IP Address Management - Some of you have asked for more information about how I'm using this in my set up. The short version is that I use this to track what IP addresses are currently assigned, and what one's are available. This is handy because if I want to set up a new cluster, I prefer them to all have sequential addresses. Not necessary, but this order pleases me. I used to note all the in use IP addresses in a page on my bookstack, but when I learnt of IPAM, I decided to give it a go. Currently it's only tracking IP addresses used, but there are way more features such as network scanning, rack diagrams, vlan documenting etc.
Guacamole (Container) Remote RDP client running in a browser. Super handy to have.
Gitea (Container) GIT version control. Mainly for storing my ansible playbooks and configuration files.
Transmission (Container) Yar Har Fiddle De Dee
pyLoad (Container) Download manager written in python. Pretty handy for downloads that could take a while.
Syncthing (Freenas Jail) Initially I was using Nextcloud, but for my purposes, it just felt super bloated. I only needed a way to synchronise my keepass database.
Bookstack (Container) Internal wiki.
Jellyfin (Container) Media player.
Ansible / VS Code(Dedicated VM) Dedicated VM which I use as an Ansible Master. I've installed VS Code Server on here so I can edit any playbooks and do things remotely if I feel the need without having to set up a new environment. Is also pretty useful for managing a kubernetes clusters as I have kubectl and rke set up on here. Saves having to set up multiple workstations and sync things across.
YoutubeDl (Container) For the downloading of youtubes and things.
NordVPN (Container) Some *cough* websites for the downloadings are not strictly allowed where I'm from, so this container allows me to bypass that to a degree. Transmission, pyload and youtubedl all go through this container. Works pretty well.
Pwndrop (Container) Has way more functionality than I'm currently using. I'm mainly using it to share files between devices/vms. Also, the default 404 page is a rick roll. The dev is my kinda people.
Ubooquity (Container) For the reading of ebooks and comics. I tried calibre, but for my needs, it just felt super bloated. I like dealing with files directly, and not the apple approach which is to organise things for me.
Veeam (Dedicated VM) Amazing software for backups. The only complaint I have is that it requires windows, but you can't have it all. Free version allows the backup of 10 devices, or, if you get the NFR version (also free), you can do up to 20.
WS2019 - DC1 (Dedicated VM) WS2019 - DC2 (Dedicated VM, but under bhyve on Freenas) Domain controllers. For the controllering of my domain.
Docker Host - (Dedicated VM) The majority of my services run on this VM. It has shares from my Freenas box mounted so the containers can access them.
Factorio Server - (Dedicated VM) This will be decomissioned soon pending the successful deployment of a Pterodactyl server.
Borgbackup - (Dedicated VM) Can't reccomend borgbackup enough. Works well for my needs and super easy to set up.
Windows10 Domain Manager - (Dedicated VM) This is a dedicated Windows 10 VM that I use to manage the windows infrastructure. DNS, Active directory and control of the VEEAM server is all done from here. This is connected to my domain so SSO works really well from here. I prefer not to use my primary workstation or laptops for these purposes, so having a vm to rdp into is super handy.
My Infrastructure Explained
Firewall
Let's start with the firewall as it's a good place as any. I have used pfSense ever since I started hosting my own things, and have had no issues with it. Originally, it was virtualised, and this worked great for many years. However, once I started to tinker more, I realised that my entire network going down because I needed to reboot my esxi host was becoming a pain. I purchased a (relatively) cheap device from Aliexpress with 6 gigabit ethernet ports and migrated my pfsense install over this. It has been on pretty much 24/7 since then with no issues whatsoever. I have single sign on enabled on here, linked to my Active Directory (which I will come back to later). I must say, the migration for virtual to physical hardware was a breeze with pfSense's built in back up and restore feature.
Aside from running my firewall, I have OpenVPN set up on here, which is super handy, as I can remote in to this if any issues occur with the rest of my lab and I'm away from home. I also have HAProxy running on here which provides a reverse proxy and ssl offloading for anything that I don't want to provide a self signed certificate for. Additionally, this device also hosts my certificate authority. While I have no issue using Lets Encrypt for any publicly facing servers (of which there are currently none), I prefer using my own CA and signing my own certificates. I have the root certs installed on all my devices so everything is trusted internally.
Ideally, I wanted to add wireless capability to my pfSense machine, but that was an excercise in futility as getting BSD to recognise the 5 different wireless cards I tried was just not happening. May revisit this is in the future if/when pfsense adds in support for more wireless cards. If anyone has any experience here, I would love some feedback. I went through the "supported" list of cards, but just couldn't get anything recognised. The consensus seems to be to not use pfsense for this, but goddammit, that's what I want to do. The system even has 2 mount points for wireless antennas so it's begging for this.
ESXI Host
My current pride and joy. It's a pretty old Dell Precision T5600 which I bought a few months ago. It's an absolute chonker, but was a major upgrade from my HP Microserver G8 which was limited to 16GB RAM. I'm running ESXI 6.7U3 on this, with VCenter installed as a VM. I really do like esxi, even though it's closed source. I did try running Proxmox in lab for a while, but ran into a few issues which caused me to go crawling back to vmware's offerings. I have also tried XCP-NG, but I didn't like the amount of work XOA required to set up (if you wanted to self host). Aside from some issues getting GPU passthrough to work (thanks to NVidia and their stupid error 43), this thing has been rock solid on ESXI 6.7.
Freenas Host
Love it or hate it, Freenas has been super stable for me. Initially it was virtualised, but now it's running on my microserver since the G8 is retired (mostly) from virtualisation duties. This provides the majority of stable storage to my services. Additionally, it runs Syncthing in a jail which I set up from scratch (thanks to Lawrence Systems on Youtube. Those guys are a gold mine of information) which backs up a few documents and my keepass database to all my devices. The only other thing running on this server is my secondary domain controller which is a windows 2019 VM that runs AD and DNS for my domain.
Switch
Nothing overly interesting here, it's a cheapo TP-Link managed switch. It's a switch. It does switch things, and does them well. Aside from the annoyance of dealing the engrish on the web panel, I've had no issues here.
Access Point
Super cheap wireless travel router flashed with OpenWRT. Provides enough coverage for everything as the majority of things in my set up are wired. This is on my list as the next upgrade I would like to do.
Active Directory
I wanted single sign on for all my devices, so first I tried FreeIPA. This worked pretty well, but apparently doesn't play nice with Freenas. Freenas is bae, so FreeIPA had to go. After this I spent ages trying to get Samba AD to work as my primary domain controller. I had it set up on a Raspberry Pi as the primary DC, with a secondary installed as a VM. This worked ok. I even wrote my first bash script to set to set up 2 samba DC's for me as it's a pretty involved process. However, I kept running into a few niggly issues which really bugged me enough to resort to using Windows Server 2019. This has been rock solid since then. The DNS and DC are replicated to a secondary WS2019 machine being virtualised under bhyve on my freenas box. Going from esxi to bhyve was a major downgrade, and I wouldn't reccomend it, but I needed some redundancy here for when I need to reboot my esxi host.
Backups
My backup strategy is very simple at the moment. All important data is stored on the Freenas box. This includes backups from VEEAM which back up all the VM's from my esxi host. This deals with bit rot and other data storage shenanigans. I'm using RAIDZ1 which covers me in the case of a disk failure. The shares I actually care about are mounted to a Borgbackup VM, which then backs up to 2 external hard drives that I rotate from an off-site location weekly. The backup keys are stored in my keepass db, which is synced to all my local devices, as well as on to the external hdd's. This should be enough to cover me in most scenarios.
Documentation
This has been a lifesaver a few times. Any time I try something new, and manage to get it to work successfully, I document it. Currently using Bookstack, and can highly reccomend it. Lovely to work with. Before this, I was using DokuWiki, which is also great, but lacked some of the fancier features Bookstack has built in.
Monitoring
So, I'm only recently getting into this, and need to spend much more time on this aspect. However, I have a working TIG (Telegraf, InfluxDB, Grafana) stack up and running, and a few graphs which others have made. I do plan on building my own super fancy dashboard eventually. I've also just managed to set up an ELK (Elasicsearch, Logstash, Kibana) stack, but it's not doing a whole lot just yet.
Automation
Ever have that issue where you want to spin up some services but don't want to go through your documentation and copy paste all your commands into a terminal like some sort of pleb? Automate it. I started with Puppet, and while it was very fancy and worked pretty well, Ansible is the real bae here. Super easy to get started with. I've set up an Ansible Server, with VS Code installed so I can do all the things from a web browser. This set up feels super fancy and is probably one of my favourite new toys.
VM's and their "quirks"
The primary Linux OS I use as the base for all my VM's in Debian. While I have iso's for other distros that I use if I must, Debian 10 is the standard base for everything. To simplify rolling out new services that require a dedicated vm, I have created a "Golden Image". This is a base install of Debian 10, with my self signed certs installed, domain name set up, and a few common bits of software I will use everywhere such as open-vm-tools. If I need to roll out a new vm based on this, I can clone it in VCenter, and use a template to assign it a static ip and hostname. I've also created an ansible playbook that can set up a new golden image on another distro if I need to. This has been a major time saver.
Container Management
Docker has simplified so much when it comes to deploying software. Being able to run a multitude of services without building a dedicated VM for each one is brilliant. I'm currently running quite a few services on 1 VM with 4vCPU's and 8GB RAM. Portainer is the current frontend. While this set up works great and I've had no issues with it, I'm currently working on migrating to singled node "HA" k8s cluster with Rancher. I have this mostly working, but I'm stuck on getting rancher to work with Active Directory, so if anyone has any tips, I'm here for that.
Current Projects
Rancher / Kubernetes - Complete overkill, but I'd like to migrate my containers to Rancher. I don't want to use the docker version of Rancher as I would like to be able to move to a full HA if I want to. I currently have a single node, HA Rancher install working. Next steps are to set up another single node kubernetes cluster, add it to rancher and convert my docker-compose files to work with rancher.
Pterodactyl io - More containers! But for games. Set up seems pretty easy to do, but I'd like to create a playbook so I don't need to manually do this if I fancy a rebuild.
Documentation - While I have documented part of my set up, I realised I wanted it to be much more in depth. This post is the result of that. Worst case scenario, I add another page to my bookstack.