Contents Link to heading

The Incident Link to heading

A few nights ago, my kids wanted to watch an earned episode of their show, but the Jellyfin TV app wouldn’t let me log in and the error message wasn’t really clear. I self-host some applications in my lab (see lab setup) but this specific situation was odd. My first step was to see if it was the TV, the app or something else, so I pulled out my phone to see if it was at least a client or server issue. You can take a kid out of TAC but troubleshooting is something you can’t just forget. So as the app opened the issue was clear, “Unable to reach server” which meant something on my physical server or virtual machine was down.

This has happened but usually after a power outage or an update or something I had done, but I didn’t recall anything. So then I logged into ESXi to see what was going on from the console. Up front I didn’t see anything, I did see some process failing on one of the tasks during the reboot command I had sent about 15 minutes prior but I had seen that before so I forced a power down. It did strike me for a second as I never used to get those errors. As I screenshotted and collaborated with Claude, the response was not good. I knew from the beginning it would be a disk failure. It was destined. I had missed non-selfinduced outages and it had been stable…but too stable. I got to the ESXi logs and found this message:

No connection for heartbeat on volume 'WD-HDD'

And that was confirmation enough before I called it a night.

The Wrong Drive Link to heading

The next day I was at peace with the drive failure and just needed to go spin up a new VM on different storage. Thankfully I did not have actual important data loss because the media was on an unaffected drive. All I need to do was stand up a new VM, load the OS and software, connected to the good drive, and perform an initial configuration for users, where to look for files, etc. At worst I would inconvenience one other user with new login credentials and a mystery as to which episode they last watched for a particular series, and for me, it was just burning some extra time.

To build the new VM I went into the console, named the new VM, and selected one of my good drives with the most free storage and there it was…I saw a folder with the old VM name and files inside.

I had 3 disks connected to the original VM - One for OS and system files, One dedicated for media, and one that used to be dedicated to media. I actually anticipated one day that I should store the media and important stuff on separate drives when I made the decision to spin up a Jellyfin server. But that old drive wasn’t just media storage, but also housed some of my core lab servers for my self-hosted on-premise Cisco Contact Center lab. I had virtual machines across all of my drives until I got dedicated storage for media and since during the outage I noticed my DNS server went down and was on the failed drive, I assumed the application VM OS vmdk was on the failed drive.

🖥️ Application VM
  - 💾 Storage — OS (`/dev/sda` → `/boot`)
  - 💾 Storage — Media  (`/dev/sdc1` → `/mnt/media`)
  - 💾 Storage — Media Old (`/dev/sdb1` → `/mnt/media2`)

Two Steps Forward, One Step Back Link to heading

With a major sigh of relief, I quickly navigated out of the new VM setup, went to reconfigure the old VM, manually removed the mapping to the bad storage, hit save, and powered on. With my surprise the VM came up in no time. Once it did, the first thing I did was mounting the storage and seeing if I could load the web app, but nothing. I checked the logs and found that some services were not running at all. Specifically one of the traces showed I was out of OS diskspace so I did some spring cleaning and made space and reloaded the Jellyfin web app and I could navigate and see all of the episode playback and metadata so I logged out. Mission complete.

At work, User Acceptance Testing (UAT) is a defined process with multiple stakeholders ensuring no stone is unturned in testing so that there is nothing out of order as we go into production. At home, UAT happens after going into production.

I was excited to tell the family everything was fixed and show everything was working as it had been days before. So I opened the app and logged in. Smiles around and comments that “it’s working” because we could see the episode listings. But then, I hit play, and hit a playback error. Two steps forward, one step backward.

I looked in the logs and saw:

    [ERR] FFmpeg exited with code 254

This maps to issues reading the file or some permission issue. At this point I realized that when I mounted the drive, the old mount script had me pointing at an empty folder. Sure enough, the right drive was mounted to the wrong folder. I quickly unmounted and remounted to the correct folder and checked playback, and it was working. Finally.

Risk, Cost, and Convenience Link to heading

This whole ordeal has been a reminder of how the technology and the risk tradeoffs have shifted. I have had to rebuild production servers due to storage and hardware failures in the past but have not seen it in some time. The main reason is most of my customers are migrating to the cloud and the leading customer engagement platforms are all cloud based without on-premise footprints beyond voice gateways, if that. This helps with the risk of data loss as personal responsibility and pushes it to the vendor.

Even with cloud storage, the monthly storage fees add up, and if you need to pull that back the egress traffic costs add up. Calculating what it would have cost to have my 1TB of data backed up to Amazon Web Services (AWS) instead, it comes out to ~$35 for year 1, with a total of ~$100 over 5 years. This even considers efficiencies of migrating files from S3 Standard to S3 Glacier after 30 days. Then, when you need to restore, the egress cost for 1 TB will be an additional ~$85. With no additional files being added over the life of this drive, and noting this drive has lasted me almost exactly 6 years, we’re looking at around ~$185 in AWS costs.

Compared to dedicated media storage, the two (2) 4TB Network-Attached Storage (NAS) rated drives cost me ~$175 and gave me a considerable amount of flexibility.

So Why Is Any of This Important? Link to heading

  • It humbled me. Data and hardware related outages are painful, even when you have a backup or nothing was truly lost, because there is time and effort to get things back up and running.
  • This triggered me to think about lifecycle management for my home lab and continuity.
  • It helped me realize what was really important. For example, the DNS server going down wasn’t important to me because it just held some local routes that I could shift to my existing Unifi Dream Machine Pro (UDMP) which can be configured with local addresses.
  • Don’t get overconfident. Just because you’ve fixed one thing doesn’t mean everything is working. One fix may uncover another issue that you didn’t think to test for (app working but video not playing).
  • Cost vs. convenience isn’t always what you’d expect. Cloud backup shifts the risk of hardware failure onto the vendor, but for a large, mostly static library like this one, dedicated NAS drives came out cheaper and gave me roughly 2TB more usable storage in the process. The convenience of “someone else worries about it” in the cloud or “my data when I want it” locally can also drive decisions.

There is always opportunity when things go wrong. I have been putting off taking apart my server and upgrading components to run things like a local LLM and this drive failure gives me a good reason to push that project up on the roadmap.