TeamSpeak.gg - A Toolkit for Somebody Else's Server

April 19, 2021 Multi-tenancyTypeScriptCloudflarePHPWebRTC

For about three years I ran a public TeamSpeak server called OpossumTS.net, and built an increasingly silly amount of software on top of it: self-service channels that expired when nobody used them, identity verification, rank badges pulled from Riot’s API, a points economy, moderation tooling. That post is the story of the server. This one is about what happened when other people started asking whether they could have the same thing.

The answer was no. Not because I minded, but because none of it was portable. It was a pile of single-purpose PHP scripts hardcoded to one server, sharing a database schema that only made sense to me, with that server’s own IDs baked into the queries. There was nothing to hand over.

So I tried twice to build the version somebody else could run.

The first attempt shipped in late 2019 and got real users: a handful of communities, mostly people I already knew. The second was a full rewrite through the winter of 2020 into spring 2021, built with Himyu, and never launched. The honest description of that second one is a tech tryout. We picked the stack we wanted to play with, and it did not survive contact with what the thing actually cost to run.

The thing you’re given

Worth thirty seconds if you didn’t read the other post. A TeamSpeak server does voice on one port and exposes ServerQuery on another: a plain-text admin protocol you reach over telnet. You connect, log in, and type commands at it. It answers with a line of key-value pairs in TeamSpeak’s own escaping scheme.

No REST, no JSON, and no events pushed to you unless you hold the connection open and subscribe to them. That last bit is the whole architectural problem here, because holding a connection open costs you a process, and a process is a thing somebody has to run.

It gets worse in one specific way. If a query connection talks to a user in chat and then disconnects, the user’s chat window shows a “this user went offline” message. So you can’t connect, say something, and leave. Anything conversational forces you to stay.

Getting a stranger’s server connected

This problem only exists once you have more than one tenant, and it turned out to be the hardest part of v1 by some distance.

To do anything at all I need a ServerQuery login for a machine I don’t own, from a person who has every reason to be suspicious, over a protocol with no concept of scoped tokens. And it has to work first time, because a stranger who hits a generic error on your signup form doesn’t file a bug report. They leave.

So the v1 signup does five things before it’ll accept you.

First it resolves the address the way the client does. People give you the hostname they hand their community, and TeamSpeak servers advertise their real host and port through DNS SRV records. If what you typed isn’t an IP, it goes looking:

$srvResult = dns_get_record("_ts3._udp." . $_POST['queryip'], DNS_SRV);

Then it takes the server’s own UUID as the tenant key. virtualserver_unique_identifier is stable, unique, and something only somebody with access can read. Deduplication comes free: try to add a server that’s already registered and it tells you so, whichever address you reached it by. Both versions of the product used this, and it’s the one design decision that made it through the rewrite untouched.

It also argues with you about credentials. Hand it the serveradmin account without SSH and it refuses, asking for explicit confirmation, because that’s the god account going over a plaintext socket.

TeamSpeak’s own errors get turned into something you can act on. My favourite is the ban case. TeamSpeak blocks a query IP after a few failed logins, which is a real hazard when you’re the one machine connecting to everybody, so the code digs the duration out of the error string:

if (strpos($string, 'you are banned') !== false) {
    $banTime = preg_replace('/[^0-9]/', '', $string);
    return "Zu viele falsche Logins, bitte warte " . $banTime . " Sekunden";
}

And then the part that earned its keep: it checks thirteen named permissions and reports back exactly which ones you’re missing. ServerQuery permissions are numeric, so the code carries the mapping and runs a permoverview against each:

$permissionNames['25']  = "b_virtualserver_info_view";
$permissionNames['29']  = "b_virtualserver_client_list";
$permissionNames['33']  = "b_virtualserver_client_dbinfo";
$permissionNames['190'] = "b_client_permissionoverview_view";
$permissionNames['209'] = "b_client_ban_list";

The form paints the answer straight onto the list, green for what you have and red for what you don’t:

The alternative is “could not connect”, which is true and useless.

Look closely, though, and that list is fifteen items long for thirteen checks. b_virtualserver_info_view appears twice, and b_virtualserver_client_dblist is in the form but not in the code, so it can never come back red no matter what your query group actually has. Neither did any harm, and both sat there for the whole life of the product.

Once you’re through all that, the credentials go in encrypted, AES-256-CBC under a random 32-character key generated per row and stored beside it. Not a strong scheme, and I’ll come back to it, but it beat what I’d been doing before, which was nothing.

The rewrite threw all of this away. v2’s daemon has a POST /servers/add that takes an address and a UUID, connects, and hands back a token. No SRV lookup, no permission preflight, no error translation. Everything I’d learned about how a stranger’s server actually fails to connect lived in v1 and didn’t make the crossing, because we were building the interesting parts first and onboarding isn’t the interesting part.

It’s the clearest regression between the two versions, and a fairly ordinary way for a rewrite to go wrong.

Poll, or listen

Once you’re connected to somebody’s server, you need to know what happens on it. There are exactly two ways, and the two versions picked one each.

v1 polled. Five PHP scripts on cron, each selecting the servers with its own feature flag set, connecting to each in turn, doing its work, disconnecting: an analytics crawler, the AFK mover, two Twitch crawlers, a version tracker. Tenants chose a scan interval of 1, 5, or 15 minutes, and the 1-minute option was disabled for the bottom tier in the <option> tag itself.

The analytics crawler is the one that mattered, and it does something I’d still defend. When it can’t connect, it doesn’t skip the server. It writes a row anyway, with the failure classified:

$a['connection failed, you are banned'] = "banned";
$a['Connection timed out']              = "timeout";
$a['Connection refused']                = "refused";

Downtime gets recorded rather than going missing. That one decision is what makes the uptime percentage on the public server list mean anything, because it’s computed as avg(status) * 100 over every scan ever taken. Skip the failures and the number becomes a lie that gets prettier the worse a server is doing.

That’s the v1 panel, showing a tenant with 3,981 tracked identities. Under the usage graph it breaks down client versions and operating systems, which no server admin otherwise gets to see. The operating-system percentages are visibly wrong, incidentally. They sum to about five percent, because they’re computed against the wrong denominator. Nobody ever reported it.

v2 listened. One daemon in TypeScript on ts3-nodejs-library, holding a persistent connection per server and subscribing to nine event types:

server.instance?.on("clientconnect",  (e) => handler(e, uuid, "clientconnect"))
server.instance?.on("clientdisconnect", (e) => handler(e, uuid, "clientdisconnect"))
server.instance?.on("textmessage",    (e) => handler(e, uuid, "textmessage"))
server.instance?.on("clientmoved",    (e) => handler(e, uuid, "clientmoved"))
// channelcreate, channeledit, channelmoved, channeldelete, serveredit

Every event gets POSTed straight out to a Cloudflare Worker at https://api.teamspeak.gg/webhooks/<eventType>, carrying the event, the tenant UUID, and the daemon’s own address. The daemon runs no business logic at all. It’s a transport.

The return path is the half I like. That same daemon exposes a REST API over the telnet protocol, one route per thing you might want to do:

POST /:uuid/clients/move       /:uuid/clients/message   /:uuid/clients/poke
POST /:uuid/clients/kick       /:uuid/clients/ban
POST /:uuid/clients/servergroups/set | add | remove
GET  /:uuid/clients/list       /servers/list

A Worker wakes up on an event, does whatever it needs to in the database, and calls back into the daemon to act:

return await fetch(`https://${daemonURI}/${uuid}/clients/message`, {
    method: 'post',
    body: JSON.stringify({ clients: targets, message: msg }),
    headers: {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer ' + await ACCESS_TOKENS.get(uuid)
    },
})

That’s the shape I was after from the start. Everything except the socket itself scales to zero, and the one stateful component is deliberately as stupid as I could make it. Business logic lives in Workers that exist for a few milliseconds per event, and the thing that has to stay running holds no state worth losing.

The instinct stuck, even though the product didn’t. A daemon that’s always up is always costing money and always able to break at three in the morning, and that bothered me the entire time. What I hadn’t worked out yet was that moving the logic off a server doesn’t move the cost anywhere. It just moves where the meter is.

Where a tenant’s data lives

v1 took the obvious route, the one almost everybody writes first. One MySQL database, and every table carries a serverid column. Tenancy is a WHERE clause. It works, it’s easy to query across, and every bug in it is a data leak, because the only thing between one community’s user list and another’s is that you remembered the predicate.

v2 gave each tenant their own database. Fauna secrets are per-database, so the Worker pulls the tenant’s secret out of a Cloudflare KV namespace, keyed by that same server UUID, and builds a client scoped to exactly one tenant:

const client = new Client({
    secret: await SECRET_TOKENS.get(body.uuid),
    fetch: fetch.bind(globalThis)
})

After that line there’s no way to address another tenant’s data, because the credential you’re holding can’t see it. Isolation becomes a property of the connection instead of a property of every query you write, so it holds even when you’re careless. And I was going to be careless eventually.

Database-per-tenant is still the pattern I’d want here. Picking Fauna to do it with is the part I got wrong, and I’ll come back to that at the end.

Three KV namespaces in total: SECRET_TOKENS for the per-tenant database credential, ACCESS_TOKENS for the daemon bearer tokens, and API_CACHE for settings, so a lookup on every single connect event doesn’t turn into a database round trip:

const cacheKey = body.uuid + "_greetingSettings";
let greetingSettings = await API_CACHE.get(cacheKey, "json")

Both versions kept Redis alongside the main store for leaderboards, the one component that genuinely wanted a different shape of database.

The same feature, twice

The AFK mover is where the two versions line up most neatly, because it’s a small feature both of them implemented and neither cheated on.

v1 ran on cron. It reads the tenant’s idle threshold, warns at 80% of it, moves you at 100%. The part I’m still quietly pleased with is the cooldown on the warning, because there was no obvious place to put per-user state and I didn’t want another table:

$filename = "/var/www/api.opossum.media/cache/afkmove_" . $serverid . "_" . $client['client_database_id'];
if (file_exists($filename)) {
    if (time() - filemtime($filename) > 600) {
        $client->message("Du bist jetzt seit ... Minuten inaktiv");
        touch($filename);
    }
}

The modification time of an empty file is the entire nagging schedule. Over a thousand of those files are still sitting on the drive I pulled this off.

v2 made it configurable, and the configuration shows how much the ambition had grown. v1’s idle check was “how long have you been idle”. v2’s is a type:

export interface IdleChecker {
  readonly onlyInactiv?: boolean
  readonly onlyFullMuted?: boolean
  readonly idleTime?: number
  readonly excludedGroups?: number[]
  readonly includedGroups?: number[]
  readonly excludedChannels?: number[]
  readonly includedChannels?: number[]
}

onlyFullMuted came straight out of running a real server. Somebody sitting in a channel with their microphone and speakers both off is idle in a way that somebody just not talking isn’t. You can’t see that distinction until a few hundred people have complained at you about being moved out of a channel they were listening in.

The v2 implementation is a Worker that fetches the client list from the daemon, filters it, posts the move back. Same feature, except the thing on a schedule is now stateless and the thing holding a connection doesn’t know what an AFK channel is.

What v2 was going to be

The types package is the most honest artifact in the whole project, because it’s the product spec with none of the marketing. Fifteen settings interfaces, each a feature the panel was going to expose:

AntiVPN, ChannelBot, Complaints, CustomChannel, CustomGroup, Filter, IdleChecker, IdleMove, LevelanPoint, LoL, Logging, Verification, VideoChat, Website, ChannelNotification.

Read down that list and you’re reading a bill of materials for OpossumTS, generalised. CustomChannel is the self-service channel system with an expires field and purchasable upgrades. LoL is the Riot rank badges, with the tier and division thresholds turned into a mapping any community could configure instead of the one I’d hardcoded. Verification is the identity check. Every feature I’d built once for myself became a struct.

The moderation filter had grown the most:

export interface Filter {
  active: boolean
  nickname?: ClientPunishment
  clientDescription?: ClientPunishment
  avatar?: ClientPunishment
  channelnames?: ChannelPunishment
  channelDescription?: ChannelPunishment
  badWords?: boolean
  capslock?: boolean
  links?: boolean
  linkType?: LinkType
  advertisment?: boolean
  zalgo?: boolean
  // allowed / forbidden word and link lists
}

Every one of those toggles is a specific thing somebody did to my server. The zalgo flag is there because combining diacritics render vertically in the TeamSpeak client, so one person can make the entire channel list unreadable. linkType separates blocking all links from blocking only shady ones, which is where you land after banning links outright and finding out that people mostly paste links at each other for good reasons.

And then there’s Website, which gives away that this had stopped being a hobby:

export interface Website {
  communityName: string
  primaryColor: string
  domain: string
}

White-labelling. Every tenant gets their own domain and colour.

Ranking

Both versions used a Redis sorted set for the leaderboard. Users gathered XP for activity, and to encourage competition we wanted everyone’s rank shown. A sorted set is almost the definition of the right structure here: elements sit in order and get reinserted at the correct position on update, so adding, removing or changing somebody’s XP is O(log(N)) and reading a leaderboard page is a range query.

Resetting them is the interesting part. Wiping a leaderboard is destructive and racy, so we never did. Instead the key itself carries a CycleKey, and a new period just starts writing to a key that doesn’t exist yet. Yesterday’s leaderboard stays exactly where it was, permanently readable, and expiry becomes somebody else’s problem.

Only the weekly one needs explaining. Unix epoch fell on a Thursday, so dividing by a week’s worth of milliseconds gives you weeks starting on Thursday. The 345600000 is four days of offset to drag the boundary back to Monday.

Video chat

This was v1, and the biggest selling point we had, because in 2019 the thing every one of these communities wanted was video and TeamSpeak didn’t have it.

The integration is what made it work. You opened your server’s video chat URL in a browser, authenticated, and shared camera or screen with everyone in the same voice channel. Leave that voice channel and you’re dropped from the matching video room and joined to the one for wherever you went. The video call follows the voice channel, so nobody manages rooms and nobody shares links.

That let us lean entirely on TeamSpeak’s own access control. Whoever can hear you can see you, by construction, and the permission model had already been configured by an admin who understood their own community. We ran a fork of jitsi-meet so we could drive participants and permissions from server events, and so we could turn off some P2P settings that would otherwise leak participants’ IP addresses to each other.

Each participant generated a random key and encrypted their media with it using WebRTC Insertable Streams. The keys got distributed over an end-to-end encrypted channel established through TeamSpeak’s existing chat, which is a slightly absurd sentence and also my favourite thing we built. The voice server people already trusted became the key exchange for the video calls it couldn’t host.

The part people saw

Not everything sat behind a login. There was a public surface, and it existed to bring server owners in.

The server list ranked registered servers by votes, and each row carried an uptime percentage from the analytics history plus a freshness traffic light. That last one worked by tracking TeamSpeak’s own release feed into a versions table and asking how many releases behind each server was:

(SELECT (SELECT max(id) FROM versions) - id FROM versions WHERE version = d.version) AS versiondiff

One release behind is green, up to four is amber, past that is red. So the public list was quietly telling every visitor which communities were running an unpatched voice server, and telling every admin, without a support ticket, that they were one of them.

The icon generator at icons.teamspeak.gg was over six hundred FontAwesome glyphs you could recolour and download as a TeamSpeak icon pack, no registration. Same engine as IconGenerator.eu, rebranded, tracking views and downloads per icon in a JSON file. Almost certainly the most-used thing on the domain, and it had nothing to do with the product.

The user map rendered a country breakdown of a tenant’s members on a world map, built from TeamSpeak’s own client_country field, at a public URL a community could embed. The avatar service solved something smaller: TeamSpeak avatars only exist as files inside the server, so there’s no URL for a member’s picture. That service opens a query connection per request and streams the image out, with the server and user identifiers encrypted in the URL so it doesn’t leak tenant slugs. v2 grew a banner-bot alongside it, generating server status banners as images.

Making it a business

v1’s pricing was three lines of PHP. Tiers were an integer on the user row, and the tier decided how many servers you could add:

$limits = array("1" => 1, "2" => 5, "3" => 100);

That’s the entire billing system. There was no billing system. The scan interval gating was similarly direct: the 1-minute option in the settings dropdown carried a disabled attribute if you were on the bottom tier.

v2 modelled it properly, which in hindsight is a tell. There’s a servicePlans type with a name, price, limits and an available-features list. There’s a paymentDetails type with card and PayPal branches and a full billing address. Users have roles per server, sessions, notifications, a twoFAToken. We were building the accounts and billing layer for a product whose entire userbase was communities we ran ourselves.

What I got wrong

Reading this back years later, three things.

The daemon’s auth is broken in a way I find funny. When you register a server, the daemon signs you a JWT. The signing secret it uses is the server’s UUID:

newServer.token = jwt.sign({ /* ... */ }, req.body.uuid)

And verification pulls that same UUID out of the request path:

const uuid = req.params.uuid
jwt.verify(bearerToken, uuid, (err) => { /* ... */ })

The signing secret is the public identifier. It’s in every URL. Anyone who knows a tenant’s server UUID, meaning anyone who has ever looked at a request, can mint themselves a valid token for that tenant. I reached for JWT because that’s what you reach for, never asked what the secret was supposed to be, and passed in the only string that happened to be in scope.

The multi-tenant rewrite ran with one password for every tenant. The daemon connects like this:

const username = "serveradmin"
const password = process.env.TS_QUERY_PW

One environment variable, the god account, every server. v1’s careful per-row encrypted credentials didn’t survive either. None of this ever broke anything, and that’s the point: every server it connected to was one of ours. The credential handling was never tested by reality because reality never arrived.

The one deployed Worker had another community’s migration hardcoded inside it. The live webhook handler contains thirty-two server group IDs and forty TeamSpeak identities in an array, under a comment reading hardcoded shit for transition period from SpotOn to ManaZone, plus a branch that backfills a specific XP value to those forty people. It reads less like a bug than a confession. The multi-tenant platform’s only production traffic was one of our own communities merging into another, with the merge pinned into the source of the platform meant to serve strangers.

The v1 credential encryption is worth one more line. It generates a fresh random key per row, then uses the same hardcoded initialisation vector for all of them. The per-row key does most of the work and the fixed IV throws some of it away. In 2019 I knew enough to know key reuse was bad, and not enough to know the IV was doing a job.

Why it stopped

We finished v2. The daemon works, the event pipeline works, both Svelte apps work. Himyu wrote most of it: seventy-six of the daemon’s ninety commits, all of the shared types package, both frontends, the theme, the banner bot. We never released any of it.

The usual story here is that nobody wanted it, and that isn’t what happened. There was interest. Not spectacular, but enough that shipping would have been reasonable. Two other things killed it.

The first was the bill. Fauna charged by operation, and the design I was so pleased with a few sections ago generates operations constantly. Every connect and every disconnect runs a match, a read and a write. The XP path adds three more queries on top. A community with a few hundred people cycling in and out of voice chat over an evening produces a silly number of events, and none of that cost amortises across tenants, because each tenant is a separate database with its own indexes and its own collections. The property that makes the security model hold without me being careful is the same one that makes the bill scale with the number of communities. We weren’t charging them anything.

Fauna was a bad fit and I picked it anyway, because I wanted to try it. It wanted you to write FQL, which nobody else on earth writes, so everything I learned building on it was worth exactly one project. It priced an event-driven workload in the least forgiving way available. And it’s dead now, shut down entirely, which answers the question better than I can.

The second was that TeamSpeak was going. Discord had taken the category and by 2021 the question a community asked wasn’t which TeamSpeak tools to run, it was whether to keep the TeamSpeak server at all. Fixing the cost problem meant either finishing the billing layer and becoming a company, or ripping the storage back out to something shared. Both were real work, neither was the fun part, and the platform underneath had fewer servers on it every month. There was no moment where we decided any of this. The commits just stop in April 2021.

v1 taught me how a stranger’s server fails to connect, which you can’t learn any other way and which I then threw away. The v2 lesson is smaller and more practical: don’t experiment with your database.

Picking up unfamiliar tech is most of why side projects are worth doing, and I’d still do it with the router, the frontend framework, the deployment target, any of the parts you can rip out on a wet afternoon. A database isn’t one of those. It sets your cost model, it shapes every query you write, and by the time you know whether it was a good idea, everything else is already written against it. I chose Fauna to find out what it was like, and what it was like turned out to be a load-bearing wall.