simono41/FediFetcher
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

documentation updates

Browse source
This commit is contained in:
nanos 2023-06-15 09:11:45 +01:00
parent 9edbee7285
commit 9fec312b38
1 changed files with 27 additions and 15 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

42
README.md
View file

@ -35,17 +35,25 @@ You can run FediFetcher either as a GitHub Action, as a scheduled cron job on yo
Regardless of how you want to run FediFetcher, you must first get an access token: Regardless of how you want to run FediFetcher, you must first get an access token:
#### If you are an Admin on your instance
1. In Mastodon go to Preferences > Development > New Application 1. In Mastodon go to Preferences > Development > New Application
1. Give it a nice name 1. Give it a nice name
2. Enable the required scopes for your options. You could tick `read` and `admin:read:accounts`, or see below for a list of which scopes are required for which options. 2. Enable the required scopes for your options. You could tick `read` and `admin:read:accounts`, or see below for a list of which scopes are required for which options.
3. Save 3. Save
4. Copy the value of `Your access token` 4. Copy the value of `Your access token`
If you are not a server admin, you do not have access to Preferences > Development. You can use [GetAuth for Mastodon](https://getauth.thms.uk) to generate an Access Token instead. #### If you are not an Admin on your Instance
### 2.1) Configure and run the GitHub Action 1. Go to [GetAuth for Mastodon](https://getauth.thms.uk?scopes=read&client_name=FediFetcher)
2. Type in your Mastodon instance's domain
3. Copy the token.
To run FediFetcher as a GitHub Action: ### 2) Configure and run FediFetcher
Run FediFetcher as a GitHub Action, a cron job, or a container:
#### To run FediFetcher as a GitHub Action:
1. Fork this repository 1. Fork this repository
2. Add your access token: 2. Add your access token:
@ -55,35 +63,39 @@ To run FediFetcher as a GitHub Action:
3. Create a file called `config.json` with your [configuration options](#configuration-options) in the repository root. **Do NOT include the Access Token in your `config.json`!** 3. Create a file called `config.json` with your [configuration options](#configuration-options) in the repository root. **Do NOT include the Access Token in your `config.json`!**
4. Finally go to the Actions tab and enable the action. The action should now automatically run approximately once every 10 min. 4. Finally go to the Actions tab and enable the action. The action should now automatically run approximately once every 10 min.
Keep in mind that [the schedule event can be delayed during periods of high loads of GitHub Actions workflow runs](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule). > **Note**
>
> Keep in mind that [the schedule event can be delayed during periods of high loads of GitHub Actions workflow runs](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule).
### 2.2) Run FediFetcher locally as a cron job #### To run FediFetcher as a cron job:
If you want to, you can of course also run FediFetcher locally as a cron job: 1. Clone this repository.
1. To get started, clone this repository.
2. Install requirements: `pip install -r requirements.txt` 2. Install requirements: `pip install -r requirements.txt`
3. Create a `json` file with [your configuration options](#configuration-options). You may wish to store this in the `./artifacts` directory, as that directory is `.gitignore`d 3. Create a `json` file with [your configuration options](#configuration-options). You may wish to store this in the `./artifacts` directory, as that directory is `.gitignore`d
4. Then simply run this script like so: `python find_posts.py -c=./artifacts/config.json`. (Read below to get a list of all options.) 4. Then simply run this script like so: `python find_posts.py -c=./artifacts/config.json`.
If desired, all configuration options can be provided as command line flags, instead of through a JSON file. An [example script](./examples/FediFetcher.sh) can be found in the `examples` folder. If desired, all configuration options can be provided as command line flags, instead of through a JSON file. An [example script](./examples/FediFetcher.sh) can be found in the `examples` folder.
When using a cronjob, we are using file based locking to avoid multiple overlapping executions of the script. The timeout period for the lock can be configured using `lock-hours`. When using a cronjob, we are using file based locking to avoid multiple overlapping executions of the script. The timeout period for the lock can be configured using `lock-hours`.
If you are running FediFetcher locally, my recommendation is to run it manually once, before turning on the cron job: The first run will be significantly slower than subsequent runs, and that will help you prevent overlapping during that first run. > **Note**
>
> If you are running FediFetcher locally, my recommendation is to run it manually once, before turning on the cron job: The first run will be significantly slower than subsequent runs, and that will help you prevent overlapping during that first run.
> **Note** > **Note**
> >
> If you wish to run FediFetcher using Windows Task Scheduler, you can rename the script to the `.pyw` extension instead of `.py`, and it will run silently, without opening a console window. > If you wish to run FediFetcher using Windows Task Scheduler, you can rename the script to the `.pyw` extension instead of `.py`, and it will run silently, without opening a console window.
### 2.3) Run FediFetcher from a container #### To run FediFetcher from a container:
FediFetcher is also available in a pre-packaged container, [FediFetcher](https://github.com/nanos/FediFetcher/pkgs/container/fedifetcher) - Thank you [@nikdoof](https://github.com/nikdoof). FediFetcher is also available in a pre-packaged container, [FediFetcher](https://github.com/nanos/FediFetcher/pkgs/container/fedifetcher) - Thank you [@nikdoof](https://github.com/nikdoof).
1. Pull the container from `ghcr.io`, using Docker or your container tool of choice: `docker pull ghcr.io/nanos/fedifetcher:latest` 1. Pull the container from `ghcr.io`, using Docker or your container tool of choice: `docker pull ghcr.io/nanos/fedifetcher:latest`
2. Run the container, passing the command line arguments like running the script directly: `docker run -it ghcr.io/nanos/fedifetcher:latest --access-token=<TOKEN> --server=<SERVER>` 2. Run the container, passing the configurations options as command line arguments: `docker run -it ghcr.io/nanos/fedifetcher:latest --access-token=<TOKEN> --server=<SERVER>`
The same rules for running this as a cron job apply to running the container: don't overlap any executions. > **Note**
>
> The same rules for running this as a cron job apply to running the container: don't overlap any executions.
Persistent files are stored in `/app/artifacts` within the container, so you may want to map this to a local folder on your system. Persistent files are stored in `/app/artifacts` within the container, so you may want to map this to a local folder on your system.
@ -136,13 +148,13 @@ Option | Required? | Notes |
| `on-done` | No | Optionally provide a callback URL that will be called when processing is finished. A query parameter `rid={uuid}` will automatically be appended to uniquely identify each execution. This can be used to monitor your script using a service such as healthchecks.io. | `on-done` | No | Optionally provide a callback URL that will be called when processing is finished. A query parameter `rid={uuid}` will automatically be appended to uniquely identify each execution. This can be used to monitor your script using a service such as healthchecks.io.
| `on-fail` | No | Optionally provide a callback URL that will be called when processing has failed. A query parameter `rid={uuid}` will automatically be appended to uniquely identify each execution. This can be used to monitor your script using a service such as healthchecks.io. | `on-fail` | No | Optionally provide a callback URL that will be called when processing has failed. A query parameter `rid={uuid}` will automatically be appended to uniquely identify each execution. This can be used to monitor your script using a service such as healthchecks.io.
#### Multi User support ### Multi User support
If you wish to [run FediFetcher for multiple users on your instance](https://blog.thms.uk/2023/04/muli-user-support-for-fedifetcher?utm_source=github), you can supply the `access-token` as an array, with different access tokens for different users. That will allow you to fetch replies and/or backfill profiles for multiple users on your account. If you wish to [run FediFetcher for multiple users on your instance](https://blog.thms.uk/2023/04/muli-user-support-for-fedifetcher?utm_source=github), you can supply the `access-token` as an array, with different access tokens for different users. That will allow you to fetch replies and/or backfill profiles for multiple users on your account.
This is only supported when running FediFetcher as cron job, or container. Multi-user support is not available when running FediFetcher as GitHub Action. This is only supported when running FediFetcher as cron job, or container. Multi-user support is not available when running FediFetcher as GitHub Action.
#### Required Access Token Scopes ### Required Access Token Scopes
- For all actions, your access token must include these scopes: - For all actions, your access token must include these scopes:
- `read:search` - `read:search`