FediFetcher for Mastodon

FediFetcher is a tool for Mastodon that automatically fetches missing replies and posts from other fediverse instances, and adds them to your own Mastodon instance.

Get FediFetcher for Mastodon

The problem

When running a very small (or single user) Mastodon instance, one of Mastodon’s short comings becomes particularly painful:

A lot of posts in your home timeline, will be from other instances. Very often, replies to those posts will not make it to your instance, and that’s a shame. It means you might miss out on interesting parts of a conversation.

Additionally, if someone new from another instance follows you, their profile will often be completely empty. This is also a problem, because you have no idea who the person that’s just followed you is. This is possibly even more acute if you require approval for new followers, as you have no data to base your approval or rejection on.

A solution

FediFetcher is a simple Python script that can pull missing posts into Mastodon using the Mastodon API. It can be run as either a GitHub Action, as a scheduled cron job, or a pre-packaged container.

Here is what FediFetcher can do:

It can pull missing remote replies to posts that are already on your server into your server. FediFetcher can be configured to

• fetch missing replies to posts in your home timeline,
• fetch missing replies to your bookmarks,
• fetch missing replies to your favourites,
• fetch missing replies to posts in your lists.

It can also backfill profiles on your instance. FediFetcher can be configured to

• fetch missing posts from users that appear in your notifications,
• fetch missing posts from users that you have recently followed,
• fetch missing posts form users that have recently followed you,
• fetch missing posts form users that have recently sent you a follow request,
• fetch missing posts from users that have recently been added to your lists.

Each part of this script is fully configurable, and you can completely disable parts that you are not interested in.

Who is FediFetcher for

FediFetcher is almost certainly most suitable for users of single user and other very small mastodon instances. Whilst it could be used elsewhere, the need on larger instances will be far less, as these naturally pull in far more posts from remote instances.

How do I get an Access Token

If you are an admin on your instance, you can get an Access Token from your Mastodon Preferences.

If you are a normal user, you can use GetAuth for Mastodon to generate an Access Token.

How to configure and use FediFetcher

You can run FediFetcher as a GitHub Action, as a scheduled cron job, a pre-packaged container, a systemd timer, or as a Scheduled Task on Windows. Full instructions are available on the FediFetcher Wiki.

Supported servers

FediFetcher makes use of the Mastodon API. It’ll run against any instance implementing this API, and whilst it was built for Mastodon, it’s been confirmed working against Pleroma as well.

FediFetcher will pull in posts from any server that implements the Mastodon API, including Mastodon, Pleroma, Akkoma, Pixelfed, and probably others.

Is FediFetcher respectful of existing norms on the fediverse?

There is a lot of reluctance on the fediverse against scraping. I believe FediFetcher is not at odds with this at all: It simply offers very targetted help with the federation of smaller (especially single user) instances.

FediFetcher does not scrape. Despite its name it doesn’t actually fetch any of the content being fed into the user’s instance. Rather, the user’s Mastodon instance fetches any posts itself, directly from the original authors’ instances.

You can read more about how Mastodon works in this Mastodon post.

Alternatively, if you don’t want to take my word for it, I invite you to read Ben’s very thorough independent write-up about FediFetcher. Why should you take Ben’s words over mine? Well, because Ben is a person who is very sceptical of bots, and runs ‘Scraper Snitch’: a whole project dedicated to blocking bots from the Fediverse.

Can FediFetcher be used to circumvent bans?

No. The way FediFetcher fetches content is by using the API key owner’s instance’s search functionality. It’s functionally identical to them pasting the URL of the posts to be added into their own instance’s search bar. Therefore, if an instance or user is blocked, FediFetcher will not be able to fetch content.

Can I, as a user, opt out of having my content pushed into other instances using FediFetcher?

Firstly, please note that FediFetcher does not itself fetch any content from you. It is the users’ Mastodon instances that fetch content directly from your own Fediverse instance. You can read more about how Mastodon works in this Mastodon post.

FediFetcher also respects blocks in the robots.txt, and this is the most efficient way if you want to block FediFetcher from your instance (see below).

However, if you are still uncomfortable with this, and your instance doesn’t block FediFetcher using robots.txt, you may opt out of having your profile backfilled through FediFetcher. You can do so through any of these mechanisms:

• Turn off ‘Include public posts in search results’ in your profile.
• Turn off ‘Feature profile and posts in discovery algorithms’ in your profile.
• Add the #NoBot tag to your profile.

If you do at least one of these, your profile won’t be backfilled by FediFetcher.

Does FediFetcher respect robots.txt?

Yes, since version 7.0.1 FediFetcher respects the Allow and Disallow directives in the robots.txt.

We encourage you to Allow FediFetcher access to your Mastodon instance, even if you Disallow other crawlers, as FediFetcher is not a crawler, but simply a tool to help with the federation of smaller Mastodon instances.

What User Agent does FediFetcher use?

FediFetcher uses a User Agent of the form FediFetcher/{VERSION}; +{SERVER} (https://go.thms.uk/ff), where {VERSION} is the currently running version of FediFetcher, and {SERVER} is the instance of the user that’s making use of FediFetcher.

Until June 2024, FediFetcher has used the simpler User Agent FediFetcher (https://go.thms.uk/mgr)

I found a reference to a FediFetcher User Agent with my own domain name in my logs. Why is that?

The reference you are seeing is because a user on your instance has configured FediFetcher for their own use. See the answer to the previous question to understand the structure of the FediFetcher User Agent.

FediFetcher is hammering my server. What can I do?

You can block FediFetcher using either the robots.txt or your firewall based on the User Agent (see above for FediFetcher’s User Agent). Since every user is running their own instance of FediFetcher IP blocks will not be effective.

However, before blocking FediFetcher, please consider that FediFetcher is extremely important for users on small instances. FediFetcher includes the name of the instance that it is being run against in the User Agent string, and it would be great if - rather than blocking all FediFetcher - you were able to block only the ‘worst offender’ FediFetcher instances. This will allow users on Single User Mastodon instances to continue to use FediFetcher, whilst instance that have lots of users, and thus better federation in the first instance, can be blocked.

Any questions?

Feel free to reach out on Mastodon at @michael@thms.uk, or open an issue at nanos/FediFetcher.