Teaching NixOS about OpenTofu

2024-01-05T06:00:00Z

In my endless quest to come up with a completely data-driven and reproducible environment, I decided to take a stab at a new automation tool: OpenTofu. I've already gotten a good NixOS setup, but I wanted to also be able to check in the setup for my instances (and to a smaller degree, my bare metal servers in my home lab) to expand on the functionality. It didn't hurt that work had settled on Terraform.

Previously, I had taken a stab at Pulumi (during my wedding anniversary trip in 2022). It was fun and I liked the code but I ended up gutting it later. For some reason, it quickly ended up feeling like a chore to play with. At that point, I figured I would just do things manually. But then I saw the announcement that OpenTofu had forked from Terraform because of enshittification of licenses (develop with an open license, then switch to a more limited one once profits became important). That little thing set me off and I decided to try it out.

Installing

Since all my infrastructure code is in a Nix flake, to get started just required me to add to the shell's packages.

devShell.${system} = pkgs.mkShell {
  buildInputs = [
    pkgs.just
    pkgs.opentofu
    pkgs.openstackclient
  ];
};

I also grabbed the OpenStack client because it made easier to find some of the nasty little identifiers I needed to import.

Configuration Files

The way tofu works, it grabs all the *.tf files in the same directory. So inside my infrastructure flake, I have a src/tofu directory with configuration files that make sense to me:

000-providers.tf
050-secrets.tf (.gitignored)
050-secrets.tf.enc (SOPS encrypted based on my user key)
200-networks.tf
500-instance0.tf
500-instance1.tf

All of them are picked up, merged together, and made into a single set of settings. I also use tofu fmt a lot since I like to normalize my files on every commit.

OpenStack and DreamHost

Fortunately, my hosting provider of choice is DreamHost. They aren't the cheapest or the best, but they appear to be ethetical. Mostly, I stick with them because they went to the court to fight some overreaching gag orders.

(I also tried DigitalOcean at the same time as Pulumi but dropped that also.)

OpenTofu (via Terraform plugins) does a wonderful job of supporting both OpenStack and DreamHost DNS to tie everything together.

terraform {
  required_version = ">= 0.14.0"

  required_providers {
    dreamhost = {
      source  = "adamantal/dreamhost"
      version = "0.3.2"
    }

    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = "~> 1.53.0"
    }
  }
}

I added the adamantal/dreamhost plugin so I could also assign the DNS record directly from my script and have everything working.

Secrets

Even though my infrastructure flake is in a private repository, I still encrypt all my secrets. I use SOPS for this, which means setting up the .sops.yaml file to encrypt and then decrypting/encrypting files using `just``:

decrypt: decrypt-clouds decrypt-secrets

decrypt-clouds:
    if [ ! -f clouds.yaml ];then sops -d clouds.yaml.enc > clouds.yaml;fi

encrypt-clouds:
    cp clouds.yaml clouds.yaml.enc
    sops -i -e clouds.yaml.enc

decrypt-secrets:
    if [ ! -f 050-secrets.tf ];then sops -d 050-secrets.tf.enc > 050-secrets.tf;fi

encrypt-secrets:
    cp 050-secrets.tf 050-secrets.tf.enc
    sops -i -e 050-secrets.tf.enc

I do the same with my .env file so I can get the information I need set up properly.

Creating an Instance

Here is a short segment for creating an instance on DreamHost (or most OpenStack providers).

resource "openstack_compute_instance_v2" "instance1" {
  provider    = openstack.dreamhost
  name        = "instance1" // It really isn't instance1, but just pretend
  key_pair    = "keypair1" // This is my SSH key set up somewhere else
  flavor_name = "gp1.supersonic" // gp1.supersonic means I don't need a swap disk

  user_data = <<-EOT
    #cloud-config
    runcmd:
      - curl https://raw.githubusercontent.com/elitak/nixos-infect/master/nixos-infect | NIX_CHANNEL=nixos-unstable bash 2>&1 | tee /tmp/infect.log
    EOT

  # This sets up the boot device as /
  block_device {
    source_type           = "image"
    uuid                  = "2b2c61c6-324c-47f4-88c1-9ae8a978ddfd" # Ubuntu
    boot_index            = 0
    delete_on_termination = true
    destination_type      = "volume"
    multiattach           = false
    volume_size           = 80
  }

  network { // Also configured somewhere else
    name = openstack_networking_network_v2.public.name
  }
}

resource "dreamhost_dns_record" "instance1" {
  record = "instance1.mfgames.com"
  value  = openstack_compute_instance_v2.instance1.network[0].fixed_ip_v4
  type   = "A"
}

For me, the part is really cool is that I can bake in the NixOS infect script right in. To my surprise, it just ran the first time without errors (thought I had to wait about ten minutes after OpenTofu said it was done).

All I had to do was either show the results:

tofu plan

Or apply the changes:

tofu apply

Importing an Instance

Actually, the first thing I did was import my existing instances into the system. This involves creating a .tf file with the same basic setup at the other instance (some fields can be skipped but I was still learning), then import with the ID from OpenStack. Of course, getting the IDs was the hard part. Fortunately, this is where the OpenStack client comes into play. I can use that to get the list of servers, figure out the ID, then import it into Tofu.

$ openstack --os-cloud dreamhost server list
+--------------------------------------+-----------+--------+------------+--------------------------+----------------+
| ID                                   | Name      | Status | Networks   | Image                    | Flavor         |
+--------------------------------------+-----------+--------+------------+--------------------------+----------------+
| 55f8ee35-31b2-4137-af1d-b7597d348271 | instance0 | ACTIVE | public=*** | N/A (booted from volume) | gp1.supersonic |
| 1a38092b-bbc5-46bd-9092-0df979ca8fe4 | instance1 | ACTIVE | public=*** | N/A (booted from volume) | gp1.supersonic |
$ tofu import openstack_compute_instance_v2.instance0 55f8ee35-31b2-4137-af1d-b7597d348271

NixOS

Now, while this was great for setting up things, I also wanted to pull that data into my Nix infrastructure flake. Fortunately, OpenTofu has a way of exporting the data pulled from the cloud. To do that, I need to add an output stanza at the bottom of my 500-instance1.tf file:

output "instance1_ipv4" {
  value = openstack_compute_instance_v2.instance1.network[0].fixed_ip_v4
}

Since I'm (recently) fond of using Just for automation, I banged up a little stanza that automatically creates a default.nix file inside that directory every time I apply.

apply: decrypt format && export
    tofu apply

format:
    tofu fmt

plan: decrypt format
    tofu plan

export:
    echo "inputs: {" > default.nix
    tofu output | sort | perl -ne 'chomp;s@_@.@g;print "  $_;\n"' >> default.nix
    echo "}" >> default.nix

I had to use _ in the output since dotted notation isn't accepted, but I use perl to convert those underscores into Nix-happy format.

inputs: {
  instance0.ipv4 = "1.2.3.4";
  instance1.ipv4 = "2.3.4.5";
}

I use this to pull into my networking.nix which is used to drive things like configuring AdGuard, services like Maddy (for DeltaChat) and other services.

inputs:
let
  tofu = import ./tofu/default.nix { };
in
{
  instance0 = tofu.instance0;
  instance1 = tofu.instance1;
  instance2.ipv4 = "192.168.0.2";
}

From there, I have a single place to get all my IP addresses:

inputs:
let
  ip = (import ../../../../networks.nix { }).instance0.ipv4;
in
{
}

Conclusion

It isn't the best or most graceful way of doing things, but I'm pretty happy how everything turned out. I made a few mistakes along the way of setting up Gitea Actions and had to drop and rebuild my instance0. That was just a matter of renaming 500-instance0.tf, applying to drop, and then name the file back. Then I had nice clean slate to push out a new closure.

OpenTofu is much nicer than Pulmui. It didn't insist on having a cloud to maintain state, the file is checked into Git instead. It has a declarative language instead of code, and since I really don't need a lot of that logical flow, it just works for me. Plus I was able to inject into my just deploy top-level script that pushes out changes to my home lab and all my instances in a single call.