SeaweedFS

Why SeaweedFS?

2024-11-08T06:00:00Z

I got an email last week asking if I could explain why I set up a Seaweed server. It's been a while since I talked about it, mostly in hints in this post from 2022 and this post from 2024. The request gives me an opportunity to expand on it, and to remind myself why I do it.

Stories

In some regards, there is a bit of trauma in me when it comes to losing out on things. This includes hearing stories that my father told me and realizing that when he died, they were gone forever. These same thoughts came up in my novel Sand and Bone:

The little girl's confessions echoed in Rutejìmo's head in a quiet symphony as he wrote in his book. The hand-bound collections of pages creaked under his hand, the leather thong strained to hold the almost fifty pages of tightly-spaced writing. Over the years, he had added a dozen pages to the collection. It wouldn't be too long before the binding couldn't handle the additional pages, but he thought he had a few more years left before that happened.

Even with his additional pages, he didn't have room to write down all of the stories he had heard over the years. He wanted to detail the joys of the little girl's death, such as the choked story about how she had stolen her brother's toy when he wasn't looking. He had also wanted to write the horrors, like one man's confession for killing his sister. Each one was precious and important. Time would erase their stories and a part of Rutejìmo died every time he forgot one.

— Sand and Bone 8: Alone

This fear is more than just one person, losing stories of things that happened decades ago as told by an old man. This includes my own stories, which will also be lost when I die. There is little chance I will be “done” with Fedran. There is little chance I will get all of the stories out of my head and onto paper. There isn't enough time, I don't have the self-esteem to do it, and I don't have the drive. When I'm gone, so are those stories. But, even going beyond that, there are so many stories of horror and joy that I feel we are losing with the passage of time. Happy stories of people falling in love, horror stories of Auschwitz, and everything in between. It is that breadth of stories, lessons, and happiness that we lose.

But the big question is, will anyone care? Does anyone want to know the first time I got caught shoplifting. Or the first time I finally said “I love you” to Partner?

Probably not.

But, you never know.

I'm never going to record them, but when I was given my father's artwork and decades of effort, I don't want to lose them yet. I don't want to just chuck it aside. So, I need to keep them.

Hoarding

I come from a long line of hoarders. I realized that as I look at my DVD collection or the boxes of books I have. My LEGO collection isn't huge, but it does fill five or six boxes. I once got rid of it, and I felt guilty for years for doing that; which is why I built it up again. Sometimes, I just play.

My mother did it. She had a massive display case of dragons and cats. She has dozens of sets of china in the basement. Entire room with the bones of a bankrupt lumber mill. A massive library (that I inherited).

My father did it. I spent days going through pieces of paper where he kept every calendar, shred of note, and personal letter he sent to his children. I saw research when I asked for help to get through college, mainly to prove that I was going the wrong path and I really should just drive two hours one way to go the college he thought was a better choice for me. I ended up saying “screw you” and went into debt for a couple decades instead.

My father has terabytes of images that he drew. Decades of him struggling to be a “good” artist, self-doubt and pain. But I'm so happy to watch him to do it. Also, he did the artwork for the nuclear reactor project he was on, and the particle accelerator. And birthday cards for all of his children and grandchildren.

For me, there is a good size of digital hoarding going on. I mean, a couple thousand ebooks is one things, but I have PDFs, for game systems that go back years. The original PDFs of HERO System 5 and 6. Every scan of the Dragon magazine. GURPS. Legal documents, funny stories, and the like. And then there are my DVDs. Back when I didn't have children, I was buying 3-5 a week. When Suncoast Video went out of business, I had a huge refund from the IRS. Walked into the store and said “I'm gunna buy everything I can.” Now, there are DVDs I can't buy anymore, I can't find. But they are slowly rotting in my file cabinet (did you know they only have a 20-30 year shelf life?). I don't want to lose them, even if I don't use them every day.

And then there is Partner's photography business. They do large photo shoots but they also need to keep them around for years “just in case” someone loses their wedding photos. She doesn't “have to”, but I don't want to ever have to say “sorry, I don't have them” for a senior photos or someone's puppies. Or family members are lost.

So I want to keep them.

Storage

All this takes space. The nice thing about digital space is that I can make backups. I can take them with me with a few kilograms of hard drives or upload them into the cloud. I can make copies to make it harder to lose (like when I was hit with a ransomware that took out my media server). The goal is 3-2-1 Backup Strategy: three copies, two locations, and at least one offline copy.

Now, dealing with storage is something I've done for quite a long time. I remember being pulled out of sixth grade school for a day to help my mother recover her RAID 5 box. At the time, drives were in the low 100s of megabytes, but I had to learn a lot about how RAID worked to figure it out. Then watching the slow recovery that took an entire day… then having another drive fail about a day after we recovered the first time and doing it all again. I used and watched hardware and software RAID controllers fail.

At the time, the data wasn't static. We were processing millions of records to do analysis. The drives would get corrupted by poor power, heavy usage, and the eventual failure of mechanical drives. We had customers lose data, watched our data centers blow a disk. Like Partner's photos, we had to keep our analysis for years after because of re-evaluation or the occasional lawsuit.

Over time, it became apparent that there was a maximum size where RAID was useful. After a while, it wasn't a matter “if” a drive goes back but “when” a drive goes bad. Backblaze (where I keep my backups, I recommend them) had a quarterly blog post about drive stats. They track with my own experiences. Yeah, 1.7% failure rate doesn't seem like much but when I'm dealing with stuff that needs to be kept for years, it hits a lot faster than you'd think it would. And offline backups fail too.

When I left my mother's company, I went back a little and just threw some drives into a big machine. Didn't bother with RAID because I knew it would fail since I was looking at a decade of hard drive use and started with a manual mirroring between a couple Windows partitions. But as the DVDs got ripped, music got purchased from Amazon, and photos kept being taken, I started to run out. Soon, it wasn't just Q:\, R:\, and S:\ being copies but each one having a portion of everything.

When I got hit with the ransomware attack, I lost my DVD collection (years of ripping) but not most of Partner's photos and the other things. I started to recover them but one of the drives didn't make it.

Then I lost another drive a few months later.

Then we didn't have money to handle the failure indicators, so I was just watching the SMART notices popping up knowing that there was nothing to do but watch the slow-moving train accident.

Then the media server decided to go down for a week.

Not having access to anything was a stark reminder that I had to do “something” if I didn't want to lose everything. And it wasn't just tossing in another drive every once in a while. I had hit the threshold where I needed to spread it out across more than just drives, I needed to spread it across servers.

Ceph

Enter Ceph. I've read about Ceph for many years before that point and it always appealed to me. It seemed to solve a lot of the problems I've experienced over the years. And it didn't have to require same-sized disks to pull of a RAID. And I could add machines if I needed to add more storage.

The biggest thing with Ceph is that it balanced across multiple servers to reduce failure, but also let you mark files as needing one, two, or more copies. Partner's photo library? Two copies. Videos? One is probably good enough. The servers didn't have to be that powerful either, so I could use my older machines to keep the disk when I had to upgrade to a newer one. In theory, I could use a Raspberry Pi to act as a cluster.

The incident when this happened was when the media server died again. I had saved some money from my commissions (earmarked to get a book edited) and used it to buy fresh hard drives. If anything, replacing the 10+ year old drives with something new and bigger would give me some. I also took the opportunity to switch to NixOS instead of Windows, which I dislike anyways. I mean, NixOS had options for Ceph, how hard could it be?

Apparently, it really wasn't ready for prime time. Eventually the wiki popped up to say that.

Took me a week to figure it out. There was a lot missing (and still is) from the core, but I managed to write up some notes for myself on how to make it work. But, after days of trial and errors, of “almost got it” joys only to watch it crash, I finally got something working. And it was glorious. At least until I realized was swapping like mad and Ceph really needs 1 GB per TB of storage. A few frantic purchases later, and I had a couple more cheap Dell machines (one of the ones that actually died this week) and I ended up with a fairly balanced Ceph cluster.

SeaweedFS

I liked Ceph, but I'm not on it anymore. Partially it was selfish reasons, I want to fix the struggles I had bringing on a new drive into the cluster but the community decided to go a different way. But there was also no one who was able to to do that different way while I was down. To test a fix myself, I had to build for twelve hours just to see if something worked (I also work with 10+ year old computers a lot). I was on unstable, so I went weeks being unable to build because I couldn't downgrade to stable. I also learned that nixpkgs had some patterns that were really hard to get into, like redefining lua at the package level to mean a specific version of lua and no one on the Matrix or Discourse forum was able to tell me that until I happened to find one person who explained it and said I should have just “known” about that mapping.

And then, the same day I realized my efforts to get Ceph working were linked on the NixOS wiki for Ceph, I also saw a little line:

Another distributed filesystem alternative you may evaluate is SeaweedFS.

I vaguely remembered looking at SeaweedFS before, but I was focused on getting Ceph working. And in that moment, when Ceph was not building and I didn't have the resources to fix the problem myself, I decided to try it out.

SeaweedFS does 80% of Ceph. It didn't have all the fancy features, but it had the features I wanted:

Distributed across multiple machines
Ability to add or remove storage on the fly
Variable replication copies
Currently built and could be run
Can be mounted on Linux
It uses 30 GB volumes on standard ext4 partitions instead of a custom one I cannot debug
Single Go executable (don't code Go but it only takes twenty minutes build, not twelve hours)
Could create S3/cloud tier backup (Ceph couldn't do that)

Yeah, it didn't have NixOS options but Google and GitHub gave me a starting point for me to puzzle it out on my own. I learned a new library and started with a little partition to see if it worked. Like Ceph, I had a lots of fits and starts, trials and puzzling through it, but eventually got it working.

And it was more scattered in terms of information but it worked.

It didn't do Ceph's “deep cleaning” to detect bit rot on failing drives.

It blew up when you tried to create too many encoding shards without the space.

It blew up when you tried to replicate without having enough nodes.

Around that time, Ceph started building on NixOS again but I was already enamored by SeaweedFS. It was “good enough” for me. It took me about a week to migrate hunks of the Ceph data to the Seaweed, decommission a Ceph drive and make it a SeaweedFS drive. Then repeat until everything was moved over.

A few weeks ago, I tried to consolidate my dad's hard drives into the cluster and ran out of space. I ended up buying a new minicomputer and throwing 11 TB worth of drives into it to give me room (two copies of everything, even the media files). This week, I lost one of those old Dells so I had to shuffle around the volumes with a few commands and it “just worked”. I'm going to replace the dead computer and I'm confident that it will “just work” then too. And, more importantly, I don't have to spend a week to figure out the commands to make it happen since I can just copy/paste a bunch of Nix code and redeploy my servers.

Thoughts

I could hoard less. It takes time and energy to keep the computers running but it makes sure Partner has their Golden Girls, the kids have their videos, and I have my dad's artwork. Also, there is something peaceful about looking at a 29.9 TB partition and having it working smoothly.

+---------------------------------------------------------------------------------------------------------------+
| 1 fuse device                                                                                                 |
+-------------+-------+-------+-------+-------------------------------+----------------+------------------------+
| MOUNTED ON  |  SIZE |  USED | AVAIL |              USE%             | TYPE           | FILESYSTEM             |
+-------------+-------+-------+-------+-------------------------------+----------------+------------------------+
| /mnt/home   | 29.9T | 20.9T |  9.0T | [#############.......]  69.8% | fuse.seaweedfs | fs.home:8888:/         |
+-------------+-------+-------+-------+-------------------------------+----------------+------------------------+

Eventually this is all going to go away. When I die, my family isn't going to be able to keep it going. Like Rutejìmo's stories and my dad's artwork, all this will fade. But I'm going to keep it going as long as I can. And try to find more pages for my book.

Recovering SeaweedFS

2024-10-25T05:00:00Z

Lately, I've been quite fond of SeaweedFS. It isn't as powerful as Ceph but it considerably easier to maintain and manage. There are some tradeoffs, such as finding bit rotting (when the disks start to fail), but I find it not quite as “fragile” when it comes to using a random collection of Linux machines.

One of the features I want to play with SeaweedFS is the ability to upload a directory transparently to a S3 bucket (not AWS though, they are too big). I'm thinking about that for later, when I want to make an extra, off-site back up critical files including Partner's photo shoots.

Overfilling

Last week, I worked on one of the tasks I've been stalling on: archiving my dad's artwork. He had a lot of copies of nearly identical files and I didn't have the working storage on my laptop. I figured I had this huge (22 TB, though mostly full) cluster, I could use that.

Yeah… not the best of ideas.

I didn't realize I had made a mistake until everything started to fail because all of the nodes were at 98% or more full and the system couldn't replicate even the replication logs. I didn't even realize that until Partner said Plex was down.

Well, with replication down, I couldn't even use the weed shell to remove a file. When I did that, it just hung for hours.

$ weed-shell
> rm -rf in/dad-pictures

Nix Shell Scripts

Above, I use weed-shell. This is a custom script I generate with NixOS that is installed in any server that can talk to my SeaweedFS.

inputs:
let
  shellScript = (
    pkgs.writeShellScriptBin "weed-shell" ''
      weed shell -filer fs.local:8888 -master fs.local:9333 "$@"
    ''
  );
in
{
  environment.systemPackages = [ shellScript ];
}

This lets me handle common functions I use when maintaining things. In this case, I don't have to enter the common parameters needed to talk to my SeaweedFS cluster.

Cleaning Up

I tried a bunch of things, such as trying to force a more extreme of vacuuming (cleaning deleted files):

> volume.vacuum --help
Usage of volume.vacuum:
  -collection string
    	vacuum this collection
  -garbageThreshold float
    	vacuum when garbage is more than this limit (default 0.3)
  -volumeId uint
    	the volume id
> volume.vacuum -garbageThreshold 0.1

This didn't help as much as I hoped, but it did allow some replication and some commands to go through. I needed to clear up a lot more space so I could remove files properly and do a wholesale rm -rf to blow away father's files and try again later once I get some more space.

Replication

I have my volumes set to 010 replication. These are three numbers as data center, rack, and host.

Data Center: Make X copies to replicate across multiple data centers. I don't have multiple data centers, so this is always 0 for me.
Rack: This is to replicate across multiple racks. My setup is that each computer is a “rack” so my 1 replication means make an extra copy on a different machine. I treat this as a rack because I also have a DeskPi Super6C which is 6 Raspberry Pi CM4 (compute modules) in a single case, so I treat all six as a “rack” but with separate hosts.
Host: The same machine. I don't have much use for two copies on the same machine, so I always set this to 0.

If I ever got a friend where I could set up a local server, I would consider setting up a second “data center” to have an off-site backup. That probably would require Tailscale but that's beyond my current scope.

Volumes

SeaweedFS basically creates multiple 30 GB files which act as a blob with multiple files inside it. That way, the problems with thousands of small files aren't an issue since everything is done on the 30 GB files called “volumes”.

Replication is done at the volume level, which means I was able to turn off replication for a series of volumes.

> lock
> volume.configure.replication -replication 000 -volumeId 1
> volume.configure.replication -replication 000 -volumeId 2
> volume.fix.replication
> volume.balance -force
> unlock

The lock and unlock are important when making changes like this, they prevent some critical operations from corrupting the cluster. The commands will tell you when it is needed.

volume.configure.replication basically changed those volumes to no replication (risky). Once that is done, volume.fix.replication and volume.balance -force deletes the excessive copies and shuffle things around, giving me some breathing room to get replication running again so I can mass delete files.

When I'm done, I just go and change all the nodes back to -replication 010 to give me the second backup.

Data Hoarding

The problem ultimate is data hoarding. Both my father and I both have multiple copies of files running around. It isn't great, but when you don't have time to clean out a copy of a laptop dying, it is sometimes easier to rsync the entire laptop into a directory of the new machine and then move on.

In this case, I needed to do some trimming of the duplicates from his files. The script is based on the one from a StackOverflow answer:

find . -not -empty -type f -printf "%s\n" \
    | sort -rn \
    | uniq -d \
    | xargs -I{} -n1 find . -type f -size {}c -print0 \
    | xargs -0 sha256sum \
    | sort \
    | uniq -w32 --all-repeated=separate

The output is pretty simple because it only lists duplicates and the paths to find them.

$ echo one > a.txt
$ echo one > b.txt
$ echo two > c.txt
$ echo two > d.txt
$ echo three > e.txt
$ find . -not -empty -type f -printf "%s\n" \
    | sort -rn \
    | uniq -d \
    | xargs -I{} -n1 find . -type f -size {}c -print0 \
    | xargs -0 sha256sum \
    | sort \
    | uniq -w32 --all-repeated=separate
27dd8ed44a83ff94d557f9fd0412ed5a8cbca69ea04922d88c01184a07300a5a  ./c.txt
27dd8ed44a83ff94d557f9fd0412ed5a8cbca69ea04922d88c01184a07300a5a  ./d.txt

2c8b08da5ce60398e1f19af0e5dccc744df274b826abe585eaba68c525434806  ./a.txt
2c8b08da5ce60398e1f19af0e5dccc744df274b826abe585eaba68c525434806  ./b.txt
$

With this, I can find the duplicates in my own system and delete them to clear out a few terabytes worth of data. It takes time, but I haven't done it before so I pointed it at /mnt/seaweed and let it run.

Once that is done, I can turn replication back on, fix replication, rebalance, and I should be good to go.

Forward Steps

I knew I was running out of storage for a while now, so I blew my monthly budget and got the fourth server ordered. This one has three 4 TB NVMe sticks (about 11 TB which will be added to the cluster) and should give me enough room to get my dad's files collected, deduplicated, and then look into uploading them to a cheap S3 storage (Backblaze) for later.

Switching Ceph to SeaweedFS on NixOS

2024-03-21T05:00:00Z

At the end of 2023, I realized that I was running out of space on my home Ceph cluster and it was time to add another node to it. While I had space for one more 3.5" drive in one of my servers, I was feeling a little adventurous and decided to get a DeskPi Super6C, a Raspberry Pi CM4, a large NVMe drive, and try to create a new node that way.

Well, over the following few months, a lot of mistakes were made that are worthy of a dedicated post. But, when most of those problems were resolve, I encountered another series of “adventures” which led me to switching out my home's Ceph cluster for a SeaweedFS one.

Unable to Build Ceph

Around the time I was working on the Pi setup, my NixOS flake was unable to build the ceph packages. Part of this is because I was working off unstable and so the few weeks of being unable to build meant I couldn't get Ceph working on the new hardware. I tried even compiling it myself, which takes about six hours on my laptop and longe because I had to remote build on the Pi itself since I have yet to figure out how to get Colmena to build aarch64 on my laptop.

Also, I was dreading setting up Ceph since I remember how many manual steps I had to do to get the OSDs working on my machines. While researching it, I was surprised to see my blog post on it was on the wiki page, which is kind of cool and a nice egoboo.

There was a PR on Github for using the Ceph-provided OSD setup that would have hopefully alleviated it. That looked promising, so I was watching that PR with interest because I was right at the point of needing it.

Sadly, that PR ended up being abandoned for a “better” approach. Given that it takes me six hours to build Ceph, I couldn't really help with that approach which meant I was stuck waiting unless I was willing to dedicate a month or so trying to figure it all out. Given that the last time I tried to do that, my PR was abandoned for a different reason, I was preparing to keeping my Ceph pinned until the next release and just having my Raspberry Pi setup sit there idle.

I was also being impatient and there was something new to try out.

SeaweedFS

Then I noticed a little thing on top of the NixOS wiki for Ceph:

Another distributed filesystem alternative you may evaluate is SeaweedFS.

I vaguely remember looking at it when I first set up my 22 TB Ceph cluster, but I'd been dreaming about having a Ceph cluster for so long that it was dismissed because I really wanted to do Ceph.

Now, the need was less there so I thought I would give it a try. If anything, I still had a running Ceph cluster and I could run them side-by-side.

A big difference I noticed is that SeaweedFS has a single executable that provides everything. You can run it as a all-in-one process, but the three big services can also be run independently. That includes the master (coordinates everything), the volumes (where things are stored), and filer (make it look like a filesystem).

Also, Ceph likes to work at the block level whereas SeaweedFS wants to be pointed to plain directories. So the plan was to take the 1 TB drive for my Raspberry Pi and turn it into a little cluster to try it out.

SeaweedFS and NixOS

The first thing was that SeaweedFS doesn't have any NixOS options. I couldn't find any flakes for it either. My attempt to create one took me three days with little success. Instead, I ended up cheating and just grabbed the best-looking one I could find and dump it directly into my flake. It isn't even an override.

Yeah I would love to have a flake for this but I'm not skilled enough to create it myself.

Masters

With that, a little fumbling got a master† server up and running. You only need one of these, so pick a stable server and set it up.

# src/nodes/node-0.nix
inputs @ { config
, pkgs
, flakes
, ...
}: {
  imports = [
    ../../services/seaweedfs.nix # the file from dermetfan
  ];

  services.seaweedfs.clusters.default = {
    package = pkgs.seaweedfs;

    masters.main = {
      openFirewall = true;
      ip = "fs.home"; # This is what shows up in the links
      mdir = "/var/lib/seaweedfs/master/main";
      volumePreallocate = true;

      defaultReplication = {
        dataCenter = 0;
        rack = 0;
        server = 0;
      };
    };
  };
}

This is a really basic setup that doesn't really do anything. The master server is pretty much a coordinator. But, what is nice is that that it shows something by starting up a web server at fs.home:9333 that lets you see that it is up (sadly, no dark mode) and running. This site will also let you get to all the other servers through web links.

Another important part is the defaultReplication. I made it explicit, but when messing around, setting all three to 0 means that you don't get hung up the first time you try to write a file and it tries to replicate to a second node that isn't set up. All zeros is basically “treat the cluster as a single large disk.”

Later on, you can change that easily. I ended up setting rack = 1; in the above example because I treat each node as a “rack” since I don't really have a server rack.

† I don't like using “master” and prefer main, but that is the terminology that SeaweedFS uses.

Volumes

Next up was configuring a volume server. I ended up doing one per server (I have four nodes in the cluster now) even though three of them had multiple partitions/directories on different physical drives. In all of these cases, I named the directory /mnt/fs-001 and created an ext4 partition on it. I could have used ZFS, but I know and trust ext4 and had trouble with ZFS years ago. But it doesn't matter, just make a drive.

# src/nodes/node-0.nix
inputs @ { config
, pkgs
, flakes
, ...
}: {
  imports = [
    ../../services/seaweedfs.nix # the file from dermetfan
  ];

  services.seaweedfs.clusters.default = {
    package = pkgs.seaweedfs;

    volumes.${config.networking.hostName} = {
      openFirewall = true;
      dataCenter = "home";
      rack = config.networking.hostName;
      ip = "${config.networking.hostName}.home";
      dir = [ "/mnt/fs-001" ];
      disk = [ "hdd" ]; # Replication gets screwy if these don't match
      max = [ 0 ];
      port = 9334;

      mserver = [
        {
          ip = "fs.home";
          port = 9333;
        }
      ];
    };
  };
}

Once started up, this starts a service on http://node-001.home:9333, connects to the master which will then show a link on that page, and basically say there is plenty of space.

The key parts I found are the disk and max.

Replication is based on dataCenter, rack, and server but also only if the disk types agree. So, hdd will only sync to other hdd even if half of them are ssd or nvme. Because I have a mix of NVMe and HDD, I made them all hdd because it works and I don't really care.

The value of 0 for max means use all the available space. Otherwise, it only grabs a small number of 30 GB blocks and stops. Since I was dedicating the entire drive over to the cluster, I wanted to use everything.

Filers

The final service needed is a filer. This is basically the POSIX layer that lets you mount the drive in Linux and start to do fun things with it. Like the others, it just gets put on the server. I only set up one filer and it seems to work, but others set up multiples. I just don't really understand why.

# src/nodes/node-0.nix
inputs @ { config
, pkgs
, flakes
, ...
}: {
  imports = [
    ../../services/seaweedfs.nix # the file from dermetfan
  ];

  services.seaweedfs.clusters.default = {
    package = pkgs.seaweedfs;

    filers.main = {
      openFirewall = true;
      dataCenter = "home";
      encryptVolumeData = false;
      ip = "fs.home";
      peers = [ ];

      master = [ # this is actually in cluster.masters that I import in the real file
        {
          ip = "fs.home";
          port = 9333;
        }
      ];
    };
  };
}

Like the others, this starts up a web service at fs.home:8888 that lets you browse the file system, upload files, and do fun things. Once this is all deployed (by your system of choice, mine is Colmena), then it should be up and running. Which means you should be able to upload a folder with the port 8888 site.

Debugging

I found the error messages are a little confusing a time, but weren't too much of a trouble to find. I just had to tail journalctl and then try to figure it out.

journalctl -f | grep seaweed

If you have multiple servers, debugging requires doing this to all of them.

Secondary Volumes

Adding more volumes is pretty easy. I just add a Nix expression to each node to include drives.

  services.seaweedfs.clusters.default = {
    package = pkgs.seaweedfs;

    volumes.main = {
      openFirewall = true;
      dataCenter = "main";
      rack = config.networking.hostName;
      mserver = cluster.masters; # I have this expanded out above
      ip = "${config.networking.hostName}.home";
      dir = [ "/mnt/fs-002" "/mnt/fs-007" ]; # These are two 6 TB red drives
      disk = [ "hdd" "hdd" ]; # Replication gets screwy if these don't match
      max = [ 0 0 ];
      port = 9334;
    };
  };

As soon as they deploy, they hook up automatically and increase the size of the cluster.

Mounting

Mounting… this gave me a lot of trouble. Nix does not play well with the auto-mount and SeaweedFS, so I had to jump through a few hoops. In the end, I created a mount.nix file that I include on any node that I have to mount the cluster on, which always goes into /mnt/cluster.

inputs @ { config
, pkgs
, ...
}:
let
  mountDir = "/mnt/cluster";

  # A script to go directly to the shell.
  shellScript = (pkgs.writeShellScriptBin
    "weed-shell"
    ''
      weed shell -filer fs.home:8888 -master fs.home:9333 "$@"
    '');

  # A script to list the volumes.
  volumeListScript = (pkgs.writeShellScriptBin
    "weed-volume-list"
    ''
      echo "volume.list" | weed-shell
    '');

  # A script to allow the file system to be mounted using Nix services.
  mountScript = (pkgs.writeShellScriptBin
    "mount.seaweedfs"
    ''
      if ${pkgs.gnugrep}/bin/grep -q ${mountDir} /proc/self/mountinfo
      then
        echo "already mounted, unmounting"
        exit 0
      fi

      echo "mounting weed: ${pkgs.seaweedfs}/bin/weed" "$@"
      ${pkgs.seaweedfs}/bin/weed "$@"
      status=$?

      for i in 1 1 2 3 4 8 16
      do
        echo "checking if mounted yet: $i"
        if ${pkgs.gnugrep}/bin/grep -q ${mountDir} /proc/self/mountinfo
        then
          echo "mounted"
          exit 0
        fi

        ${pkgs.coreutils-full}/bin/sleep $i
      done

      echo "gave up: status=$status"
      exit $status
    '');
in
{
  imports = [
    ../../seaweedfs.nix
  ];

  # The `weed fuse` returns too fast and systemd doesn't think it has succeeded
  # so we have a little delay put in here to give the file system a chance to
  # finish mounting and populate /proc/self/mountinfo before returning.
  environment.systemPackages = [
    pkgs.seaweedfs
    shellScript
    volumeListScript
    mountScript
  ];

  systemd.mounts = [
    {
      type = "seaweedfs";
      what = "fuse";
      where = "${mountDir}";
      mountConfig = {
        Options = "filer=fs.home:8888";
      };
    }
  ];
}

So, let me break this into the parts. SeaweedFS has a nice little interactive shell where you can query status, change replication, and do lots of little things. However, it requires a few parameters, so the first thing I do is create a shell script called weed-shell that provides those parameters so I don't have to type them.

$ weed-shell

The second thing while doing this is that I wanted to see a list of all the volumes. SeaweedFS creates 30 GB blobs for storage instead of thousands of little files. This makes things more efficient in a lot of way (replication is done on volume blocks).

$ weed-volume-list | head
.> Topology volumeSizeLimit:30000 MB hdd(volume:810/1046 active:808 free:236 remote:0)
  DataCenter main hdd(volume:810/1046 active:808 free:236 remote:0)
    Rack node-0 hdd(volume:276/371 active:275 free:95 remote:0)
      DataNode node-0.home:9334 hdd(volume:276/371 active:275 free:95 remote:0)
        Disk hdd(volume:276/371 active:275 free:95 remote:0)
          volume id:77618  size:31474091232  file_count:16345  replica_placement:10  version:3  modified_at_second:1708137673 
          volume id:77620  size:31501725624  file_count:16342  delete_count:4  deleted_byte_count:7990733  replica_placement:10  version:3  modified_at_second:1708268248 
          volume id:77591  size:31470805832  file_count:15095  replica_placement:10  version:3  modified_at_second:1708104961 
          volume id:77439  size:31489572176  file_count:15067  replica_placement:10  version:3  modified_at_second:1708027468 
          volume id:77480  size:31528095736  file_count:15118  delete_count:1  deleted_byte_count:1133  replica_placement:10  version:3  modified_at_second:1708093312

When doing things manually, that is all I needed to see things working and get the warm and fuzzy feeling that it worked.

Getting it to automatically mount (or even systemctl start mnt-cluser.mount) is that the command to do so is weed fuse /mnt/cluster -o "filer=fs.home:8888".

NixOS doesn't like that.

So my answer was to write a shell script that fakes a mount.seaweedfs and calls the right thing. Unfortunately, it rarely worked and it took me a few days to figure out why. While weed fuse returns right away, I'm guessing network latency means that /proc/self/mountinfo doesn't update for a few seconds later. But systemd had already queried the mountinfo file, saw that it wasn't mounted, and then declared the mount failed.

But, by the time I (as a slow human) looked at it, the mountinfo showed success.

The answer was to delay returning from mount.seaweedfs until we give SeaweedFS a chance to finish so systemd could see it was mounted and didn't fail the unit. Hence the loop, grep, and sleeping inside mount.seaweedfs. Figuring that out required a lot of reading code and puzzling through things to figure out, so hopefully that will help someone else.

After I did, though, it was working pretty smoothly since, including recovering on reboot.

Changing Replication

As I mentioned above, once I was able to migrate the Ceph cluster, I changed replication to rack = 1; to create one extra copy across all four nodes. However, SeaweedFS doesn't automatically rebalance like Ceph does. Instead, you have to go into the shell and run some commands.

$ weed-shell
lock
volume.deleteEmpty -quietFor=24h -force
volume.balance -force
volume.fix.replication
unlock
exit

You can also set it up to do it automatically, I'm not entirely sure I've done that so I'm not going to show my attempt.

Observations

One of the biggest thing I noticed is that Ceph does proactive maintenance on drives. It doesn't sound like much, but I feel more comfortable that Ceph would detect errors. It also means that the hard drives are always running in my basement; just the slow grind of physical hardware as Ceph scrubs and shuffles things around.

SeaweedFS is more passive in that regard. I don't trust that it won't catch a failed hard drive a fast, but it still doesn't have the failures of RAID, lets me spread out data across multiple servers and locations. There is also a feature for uploading to a S3 server if I wanted. I use a Restic service for my S3 uploads.

That passivity also means it hasn't been grinding my drives as much and I don't have to worry about the SSDs burning out too quickly.

Another minor thing is that while there are a lot less options with SeaweedFS, it took me about a third of the time to get the cluster up and running. There were a few error messages that threw me, but for the most part, I understood the errors and what SeaweedFS was looking for. Not always the case with Ceph and I had a few year-long warnings that I never figured out how to fix that I was content to leave as-is.

I do not like the lack of dark mode on SeaweedFS's websites.

Opinions

I continue to like Ceph, but I also like SeaweedFS. I would use either, depending on the expected load. If I was running Docker images or doing coding on the cluster, I would use a Ceph cluster. But, in my case, I'm using it for long-term storage, video files, assets, and photo shoots. Not to mention my dad's backups. So I don't need the interactive of Ceph along its higher level of maintenance.

Also, it is a relatively simple Go project, doesn't take six hours to build, and uses more concepts that I understand (mkfs.ext4) that I'm more comfortable with it.

It was also available at the point I wanted to play (though Ceph is building on NixOS unstable again, so that is moot problem. I was just being impatient and wanted to learn something new.)

At the moment, SeaweedFS works out nice for my use case and decided to switch my entire Ceph cluster over. I don't feel as safe with SeaweedFS, but I feel Safe Enough™.