Introduction
In my last post, I went through some of the interesting (at least, to me) insights I gained through the 1001 Albums process. One thing I didn’t discuss much was the mechanics of listening to an assigned album 5 days a week for 4 years. It turns out it can be tricky to stay the course. Various distractions can happen, such as vacations, business trips, or just really busy workdays full of meetings, that can make keeping up a challenge. It’s worth noting that if you do miss a day, you don’t miss your chance to listen and review the album. However, if you miss a week or so without reviewing, the site will pause your progression. We never actually ran into that, but it warned us it was going to do it at some point.
I didn’t stumble upon the 1001 Albums Generator on my own. I heard about it through the Heavy Blog is Heavy discord, one of my main Internet hangs, where they were going to start a 1001 group. When you have a group, everyone in the group gets the same selections day-to-day. When the 1001 group started, there was about 25 of us. I can’t quite remember how fast people fell off, but it was a lot faster than I expected. I think within 8 months, it was down from the original 25 to a lowly 3. There was me, my good friend Noah, and this one other guy named Aviator. Aviator wasn’t cool, insofar as we often had diffing opinions on reviews. However, Aviator was cool, insofar as he stayed the course (almost… he fell off about 2/3 the way through). I don’t know if I could have done it on my own like he (almost) did. I was lucky to have my friend Noah with me, and we also had a bot.
1001Bot
Noah and I went to engineering school together. Since graduating in 2013, we’ve stayed in contact in various chats (with another 8 or so people). It started as group text, then a google hangout (RIP), then finally, a self-hosted Mattermost. There are various channels in the Mattermost instance, but I spend a lot of time in ~music, and so does Noah.
Seeing how Noah and I (and actually, 1-2 other members of the Mattermost who joined the group, but fell off) were excited about the 1001 list and talking about it in ~music, I figured it’d make sense to have a bot. That way it’d be really easy to see what album we had that day, and maybe have a conversation about it in chat.
The concept behind this “bot” (it’s barely a bot. It’s just an incoming webhook that posts content) was simple enough: At midnight, post in our Mattermost server’s ~music channel with the next album, artist, what album # it is, the album art, and a few links to the album on Wikipedia and Spotify. Noah and I both used Spotify (I recently switched to Tidal) and this is given in the 1001 Albums Generator payload, so having it in the bot post makes it one click to get going in the morning, in a place we’re probably already looking.
The Script
Normally, I’d reach for something like python for such a script, but since this is sorta a one-shot once a day, I decided to just write a shell script.
Breakdown
This first part are a few bc functions used to help figure out what percent of the way done we were. Originally, we used the isnewmod5() function to be informed every 5% of the way done (every 55 albums, or every 11 weeks). Later, I added the anotherpercent() function in order to be informed every 1%. We switched to that in the last 10% or so.
#!/usr/bin/env bash
BCFUNCTIONS=$(mktemp)
cat << EOF > $BCFUNCTIONS
define int(x) {
auto s; s=scale; scale=0; x/=1; scale=s; return x;
}
define mod(x, m) {
auto s; s=scale; scale=0; x = x%m; scale=s; return x;
}
define isnewmod5(x) {
auto a,b,m1,m2
a=int((x/1089)*100)
b=int(((x-1)/1089)*100)
m1=mod(a,5)
m2=mod(b,5)
if (m1 == 0 && m2 == 4) {
return a
}
}
define anotherpercent(x) {
auto a,b
a=int((x/1089)*100)
b=int(((x-1)/1089)*100)
if (a != b) {
return a
}
}
EOF
This bit is just some simple setup, leveraging some envvars that need set in the script environment ($BOT_USERNAME, $WEBHOOK_URL_FILE, $ALBUMGENERATOR_URL). It fetches the 1001 Albums Generator URL with curl and saves the payload as well. There’s an API documented here.
PAYLOAD_FILE=$(mktemp)
curl "$ALBUMGENERATOR_URL" > "$PAYLOAD_FILE"
webhook_url=$(cat $WEBHOOK_URL_FILE)
username="$BOT_USERNAME"
Here’s a big part of the meat of the thing. Just running jq on the payload, separating out parts into envvars. albumNumber needs bumped by 1. Most keys are under .currentAlbum.
Note: late into the game (more than 1000 albums in), we had a day where the bot didn’t post. It turned out to be because the artist was ‘Bonny “Prince” Billy’. I needed to add the line
artist=${artist//\"/\\\"} (magic bash regex to replace " with \") in order to escape any " characters because otherwise they’d mess up my crappy HEREDOC JSON encoding later in the script.
albumNumber=$( cat $PAYLOAD_FILE | jq '.numberOfGeneratedAlbums')
albumNumber=$(($albumNumber + 1))
currentAlbum=$(cat $PAYLOAD_FILE | jq '.currentAlbum')
album=$( echo $currentAlbum | jq -r '.name')
album=${album//\"/\\\"}
albumYear=$( echo $currentAlbum | jq -r '.releaseDate')
artist=$( echo $currentAlbum | jq -r '.artist')
artist=${artist//\"/\\\"}
image=$( echo $currentAlbum | jq -r '.images[0].url')
spotifyId=$( echo $currentAlbum | jq -r '.spotifyId')
wiki=$( echo $currentAlbum | jq -r '.wikipediaUrl')
This bit is just a bit of formatting before we generate the JSON payload, setting up the Mattermost post title, and potentially appending that we’re another 1% (or 5%) done.
title="1001 Albums #$albumNumber"
onemorepercent=$(echo "anotherpercent($albumNumber)" | bc -lq $BCFUNCTIONS)
if [[ $onemorepercent -ne 0 ]]; then
title="$title ($onemorepercent% Complete!)"
fi
Here’s the HEREDOC JSON payload I mentioned. This is based on the Mattermost attachment format. It should be pretty self-explanatory. "short": true means make the field only half-width instead of taking an entire line in the attachment. <url|text> is the format of links.
payload=$(cat <<EOF
{
"username": "$username",
"attachments": [
{
"thumb_url": "$image",
"fields": [
{
"title": "Artist",
"value": "$artist",
"short": true
},
{
"title": "Album",
"value": "$album ($albumYear)",
"short": true
},
{
"title": "Wikipedia",
"value": "<$wiki|Link>",
"short": true
},
{
"title": "Spotify",
"value": "<spotify:album:$spotifyId|App> - <https://open.spotify.com/album/$spotifyId|Web>",
"short": true
}
],
"title": "$title"
}
]
}
EOF
)
And last but not least, print out the payload for debugging purposes (I might have added this with the ‘Bonny “Prince” Billy’ debacle), remove the temporary file with the 1001 payload (probably not necessary), url-encode our new Mattermost JSON attachment and post it.
echo $payload
rm $PAYLOAD_FILE
curl -i -X POST --data-urlencode "payload=$payload" $webhook_url
The nix module
If I were a normal human, I’d add this script to cron and call it a day. Fortunately, I’m not a normal human, so I wanted to add this to my NixOS config. This is effectively my way of making sure I mean what I say and I say what I mean. It also means my system isn’t a pile of hacks. Doing it this way, everything about the script, including how to deploy it (systemd service + timer), and what it’s dependencies are, are captured in code, and (in this case) encapsulated into one file.
Breakdown
This first bit basically just says what arguments our module needs, and says “make an alias cfg as a shortcut to config.services."1001bot".
{
pkgs,
config,
lib,
...
}:
with lib; let
cfg = config.services."1001bot";
in {
Module options determine the interface of the module, i.e. what needs set by the consumer of this module. In our case, we have the username, albumgeneratorUrl, and webhookSecretFile options. username and albumgeneratorUrl are ok to be public, but webhookSecretFile contains a secret. In order to keep the secret out of the nix store, we just pass a path to the secret file, and the script knows how to consume the file (with webhook_url=$(cat $WEBHOOK_URL_FILE)).
options.services."1001bot" = with lib; with types;
{
enable = mkEnableOption "1001bot";
username = mkOption {
type = types.str;
};
albumgeneratorUrl = mkOption {
type = types.str;
};
webhookSecretFile = mkOption {
type = types.str;
};
};
Module config is really the meat of a module. This makes a systemd timer and service. The systemd timer isn’t configurable, as we know we always want it at midnight, monday through friday. The systemd service is of type oneshot, runs as root, and defines the packages needed by the script (and only those packages), as well as mapping the nix module options to the environment variables needed by the script.
config = mkIf cfg.enable {
systemd.timers."1001bot" = {
wantedBy = ["timers.target"];
timerConfig = {
OnCalendar = "Mon..Fri *-*-* 00:00:00";
Persistent = true;
};
};
systemd.services."1001bot" = {
script = ''
<THE SCRIPT CONTENTS WE HAD ABOVE>
'';
serviceConfig = {
Type = "oneshot";
User = "root";
};
unitConfig = {
StartLimitInterval = "200";
StartLimitBurst = "5";
};
path = [
pkgs.bash
pkgs.jq
pkgs.curl
pkgs.bc
];
environment = {
BOT_USERNAME = cfg.username;
ALBUMGENERATOR_URL = cfg.albumgeneratorUrl;
WEBHOOK_URL_FILE = cfg.webhookSecretFile;
};
};
};
}
Consuming the 1001bot nix module
Now that the nix module is made, I needed to consume the module into my main nix config. We are importing the nix module file, and then configuring the module options we defined. As previously mentioned, the webhookSecretFile is secret, and in my nixos-config, I’m leveraging a nix module called sops-nix in order to keep the secrets out of the nix store. Effectively, sops-nix is able to decode the secrets at runtime, and give well-known temporary paths of those secret files to the modules.
{
config,
pkgs,
...
}: {
imports = [
./1001bot.nix
]
services."1001bot" = {
enable = true;
username = "1001-bot";
albumgeneratorUrl = "https://1001albumsgenerator.com/api/v1/groups/heavy-1001-is-heavy";
webhookSecretFile = config.sops.secrets."1001bot/webhook_url".path;
};
Conclusion
The 1001 Albums list was a lot of fun. I have a tendency to turn most problems into a technical problem at some point, and this was no exception. Hopefully, if you’ve read this far, you’ve learned a trick or two about a shell trick or a nixos module or something.
If not… sorry, better luck next time!