MilesFirstBot

What it fetches

Published, static pages only — a bank or airline's own public pages describing transfer ratios, transfer bonus promotions, devaluation and program-change announcements. It never touches anything behind a login, never books or holds inventory, and never fetches live bookable award-seat availability.

How often

Once a day, per source, at a fixed schedule. A page that hasn't changed since the last check is never re-fetched or re-parsed — MilesFirstBot uses conditional GET (ETag / Last-Modified) and compares the response body's checksum before doing any work.

How it identifies itself

Every request carries this exact User-Agent header:

MilesFirstBot/1.0 (+https://milesfirst.com/bot/; hello@milesfirst.com)

How to block it

• Add a Disallow rule for MilesFirstBot (or User-agent: *) to your robots.txt — respected on every request, checked fresh before every fetch. An unreachable or unparseable robots.txt is treated as disallow, not allow.
• A Crawl-Delay directive is honored (minimum 5 seconds between requests to the same host regardless).
• Email hello@milesfirst.com and we'll stop fetching a specific URL or host, no questions asked.

What it never does

• Never logs in, never submits a form, never follows a redirect off the original host.
• Never fetches anything not explicitly allowlisted per source — there is no crawler, no link-following.
• Never presents a scraped number as verified fact. Every scraped row that touches money — fees, award prices — ships flagged UNVERIFIED until a human confirms it. See our honesty policy.