blog.thms.uk

Pull missing posts from new followers, as well as recently followed accounts into Mastodon

Apart from Mastodon not pulling in replies to posts that might be on my timeline, another thing that tends to annoy people (particularly on single user instances) is that when they follow a new account, or a new account follows them, those users' past posts aren't imported into their own instance, making them appear empty.

Like the missing replies, this problem has been brought up on github, but never been addressed.

With the release of v3 of FediFetcher, you can now also address this issue.

What's new in FediFetcher v3?

  1. In addition to getting missing replies, FediFetcher can now also backfill posts from accounts that you have recently followed, as well as from accounts that have recently followed you.
  2. Additionally I've tidied up some internal workings, to ensure that all requests to mastodon instances now fully honour rate limits, without hammering any of the servers.

How does FediFetcher backfill posts?

The script will get a list of your most recently followings and followers. For each of the people on the list, it will then request the most recent 40 posts, and add them to your own server.

Obviously, it'll remember any account it has already looked at, so it won't request posts from the same account over and over again. It's also fault tolerant, so if it fails to add post from a user in one run, it'll simply pick up any leftovers the next time it runs.

Why 40 most recent posts per account? Because that's the maximum you can get through the Mastodon API without pagination, and pagination adds a lot of load on the other server. We want to be mindful of their resources.

How can I configure the backfilling of recent followings and followers?

If you wish to make use of this, just follow these steps:

  1. Update your fork - You must have v3.0.1 or later.
  2. Provide three additional environment variables:
    1. USER This is the user whose most recently followed accounts you wish to backfill. You should only supply the 'local' part of their name. So, if their full name is @michael@thms.uk, simply supply michael.
    2. MAX_FOLLOWINGS How many of their most recently followed accounts do you wish to backfill? Provide an integer greater than 0. I suggest a value no greater than 80, to avoid pagination.
    3. MAX_FOLLOWERS How many of their most recent followers do you wish to backfill? Provide an integer greater than 0. I suggest a value no greater than 80, to avoid pagination.

That's it - the script should now automatically backfill past posts from your recently followed accounts, as well as from recent followers.

How can I disable the backfilling of recent followings and followers?

If you are currently running on v3.0.0 or below, you don't need to do anything: If you don't supply both the MAX_FOLLOWINGS and USER environment variables, this part will automatically be disabled. However, you can set MAX_FOLLOWINGS to 0, if you want to be really sure.

Where can I find out more?

If you haven't yet done so, have a look at the FediFetcher for Mastodon page, and at the GitHub repository FediFetcher