↓Skip to main content
Fedihosting Foundation

Toot Community migration

·601 words·3 mins

Toot.Community
#

History
#

Last May, we took over management of toot.community from Jorijn.
He had set this instance up in November 2022, but no longer had the time to maintain it. He contacted the Fedihosting Foundation asking if we could take over this Mastodon instance.

He left us a well-maintained infrastructure, running on Kubernetes.

Reason for migration
#

Although the instance was running on a very decent and state-of-the-art technology stack, we immediately saw the need for migrating it to ‘more simple’ infrastructure. There were a few reasons for this:

  • The technology stack consists of a few tools we have no (or not enough) experience with. Things like Talos, fluxcd etcetera haven’t been used by us. We need at least a few in our team to understand all the tools used so we can support it when stuff breaks.
  • The costs for running the instance were close to 500 EUR/month, while donations were under 300 EUR/month. This is not sustainable in the long run.
  • The hardware was under-used. It was sized to handle the load we had when the user influx happened (end of 2022) but usage has gone down since.
  • Toot used Fastly (a CDN), which cost EUR 70,- per month. FHF has a free Fastly account using their Fast Forward program, so we wanted to move toot in that account to save money.

The new “hardware”
#

The kubernetes setup was using 4 physical servers and 3 VMs at Hetzner. Our new environment will consist of VMs, also at Hetzner.

The migration
#

We had already prepared the migration by:

  • Creating the Terraform scripts for the new servers and application
  • Creating the APP, DB and Loadbalancer VMs
  • Replicating the database to a new VM
  • Making a plan for the migration
  • Announcing the migration on Mastodon

On Nov 10th, 19:00 UTC we stopped the Mastodon software, and cutover the database to our replica.
We removed the old Fastly config, and tried recreating the Fastly configuration in our FHF account. There we ran into an issue: Fastly gave an error, telling us the domain was already used in another account. We even removed the old account entirely but the error remained. We decided to go ahead without Fastly for now, and add that later.

We started the Mastodon application on our new app server, and quickly noticed the CPU was 100% used immediately. We created a second APP server, and moved the sidekiq part to that.

The site didn’t load yet. We found an issue with volumes in our docker environment and with the Redis (thanks to @renchap@oisaur.com for pointing us in the right direction) and fixed that.

The site was then loading, but the images didn’t display correctly. We needed to add a domain for the static images, which fixed that.

The search index (Elasticsearch) then had to be re-created, which is still running at this moment (it’s indexing 320 Million items).

Lessons learned
#

Of course we learned some things we hope we’ll remember next time ;-)

  • Communication: We had announced the migration using an Announcement in Mastodon. But apparently not everyone sees them. We will need to come up with a better way. (The @support@toot.community account has under 500 followers, where the instance has over 30.000 accounts, so posting there also doesn’t reach everyone)
  • Fastly: We now know that Fastly needs time to remove a domain before it can be re-used.

Conclusion
#

We still need to fix the Fastly config, but other than that all seems running fine now.
The migration was mainly done by Jelloeater, Ruud and MrKaplan, so a big thank you to them!!