Operations guide

Operations routine

Safe updates

1. 1
   Review the release and migrations

   Review changes, compatibility, environment variables, and D1 updates between the current and target versions.

2. 2
   Create a backup and baseline

   Create a D1 backup and record error rates, queue latency, and key API responses.

3. 3
   Run the update

   After you update the working tree yourself, the script installs dependencies, checks types and tests, applies D1 migrations, and deploys all three Workers in order.

4. 4
   Run smoke tests

   Verify login, posting, media, inbound and outbound federation, WebSockets, and email.

5. 5
   Maintain an observation window

   If new errors or DLQ volume exceed the baseline, stop further deployments and isolate the cause.

git fetch origin
git switch main
git pull --ff-only origin main

# Validate the exact checked-out code without deploying.
./scripts/update.sh --dry-run

# Apply migrations and deploy all three Workers.
./scripts/update.sh

Resolve CI push races

A workflow can fail when it tries to update main while another workflow or maintainer changes the same branch. GitHub rejects the stale update instead of overwriting the newer commit.

git push origin main
! [remote rejected] main -> main (cannot lock ref 'refs/heads/main': is at 135f083... but expected 28f3652...)
error: failed to push some refs to 'https://github.com/SJang1/siliconbeest.sjang.dev'

1. 1
   Inspect and fetch the current remote branch

   Confirm the commit created by the update job, then fetch the newest origin/main without discarding local work.

2. 2
   Rebase the update commit

   Replay the local CI or .github changes on top of origin/main. Resolve any conflicts explicitly, stage the resolved files, and continue the rebase.

3. 3
   Push the rebased commit

   Push HEAD to main. A normal push preserves every remote commit and works with branch protection.

4. 4
   Retry only if the ref moved again

   Run fetch, rebase, and push again with a small retry limit. If conflicts keep recurring, stop automatic updates and review the competing workflows.

git status
git log -1 --oneline
git fetch origin main
git rebase origin/main

# If Git reports conflicts:
git status
# Edit each conflicted file, then stage only the resolved files.
git add <resolved-files>
git rebase --continue

# Publish without rewriting remote history.
git push origin HEAD:main

git rebase --abort

• Keep update workflows idempotent so a rerun produces the same files and commits only real changes.
• Use GitHub Actions concurrency to serialize workflows that write to the same branch.
• Give write access only to the job that performs the final push, with contents: write scoped as narrowly as possible.
• Stop after a small number of retries and surface the conflicting commits for operator review.

Backup and recovery

D1 is the system of record for accounts, posts, and settings, while R2 stores media. Configuration and secrets must be recoverable from a separate secure store, not the repository.

./scripts/backup.sh

• Record each backup file's creation time, size, and hash.
• Test restoration against isolated resources, never the production instance.
• After restoration, compare account counts, recent posts, settings, and media-reference integrity.
• Document the RPO and RTO, and shorten backup intervals as the instance grows.
• Backups contain personal data, so restrict access and retention periods.

Incident response

1. 1
   Establish the impact

   Record which paths—web, API, federation, streaming, or email—are affected and when the impact began.

2. 2
   Freeze changes

   Pause automatic deployments and compare recent releases and configuration changes with Cloudflare status.

3. 3
   Preserve the queues

   Do not delete failed messages prematurely. Fix the cause, then reprocess them in controlled batches.

4. 4
   Communicate with users

   Publish only confirmed impact and workarounds without exposing personal data or attack details.

5. 5
   Review and follow up

   Record detection, mitigation, and recovery times, the root cause, and prevention actions.

Operations security checklist

• Enable a passkey or TOTP for administrator accounts and store the recovery process separately.
• Limit Cloudflare API tokens to required accounts and resources, and rotate them regularly.
• Keep SKIP_SIGNATURE_VERIFICATION false except for explicit isolated testing.
• Keep setup, OTP-encryption, Turnstile, and deployment credentials in Wrangler secrets or GitHub Secrets. The email Worker currently reads SMTP settings from D1, so restrict admin access and protect database backups.
• Review audit logs for reports, domain blocks, pending approvals, and invite credits regularly.
• Before deleting an account, check backup and legal-retention requirements and federation Delete delivery.

./scripts/delete-account.sh <username>