Turn a website into clean text you can search.

crawl follows a site's links, pulls the readable text out of every page, splits it into passages and saves files any knowledge base can import.

Buy crawl for $49

One payment, every future update. Delivered as a private GitHub repository you're invited to.

python -m webcrawl https://tools.dominares.org --out ./kb --max-pages 8

Run summary (stdout)

{
  "ok": true,
  "pages": 7,
  "chunks": 75,
  "errors": 0,
  "out_dir": "./kb"
}

kb/chunks.jsonl, first 3 of 75

{"id": "1f324a36398cd288-f2c4d55f09bbf2a3-0",
 "url": "https://tools.dominares.org",
 "title": "crawl: turn a website into clean, searchable text | Dominares Tools",
 "text": "Turn a website into clean text you can search."}{"id": "72bf6cf11d19a068-8c47b69035fa6579-0",
 "url": "https://tools.dominares.org/access",
 "title": "Get access | Dominares Tools",
 "text": "Already bought crawl? Enter the GitHub username you bought it for. If that account has a purchase, it gets access or a f…"}{"id": "782fce1805b02e43-03a3b0978b0c1ad3-0",
 "url": "https://tools.dominares.org/terms",
 "title": "Terms of sale | Dominares Tools",
 "text": "Version 2026-09-13. These terms cover buying crawl at tools.dominares.org."}

Real output from crawl 1.0.0, run against tools.dominares.org on 13 September 2026. Passages shortened to fit.

What it does

Crawls politely: Follows links within a site, obeys robots.txt and its crawl delay, can space out requests to each host, and can start from sitemap.xml.
Keeps the words: Pulls each page's title and readable text out of the HTML. PDFs and JavaScript-rendered pages work with optional extras.
Re-reads only changes: --resume skips pages whose content hasn't changed and asks servers whether a page is newer before downloading it.
Splits for retrieval: Breaks text into passages sized for search and AI retrieval, each with an id built from its page and content.
Searches two ways: Keyword ranking is built in. Search by meaning with a local index that needs no API key, or with OpenAI embeddings.
Sends it anywhere: Portable JSON files, a local JSON or SQLite vector store, pgvector, Pinecone, or your own webhook.
Runs as a service: An HTTP job API with API-key auth and per-client rate limits that refuses private, loopback and cloud-metadata addresses.

What you get

crawl 1.0.0: the Python library, the webcrawl command and the HTTP API, with 219 automated tests. Python 3.10 or newer; the base install needs only httpx.

A commercial license. Use it and change it in your own projects and your clients' projects. Don't share or resell the source. Read the license.

Every update. New versions land in the same repository; git pull to get them.

A bonus pack of 11 small tools, in the same repository:

Tool	What it does
`ad-gallery-generator`	Turn a folder of PNG images into one self-contained, sectioned gallery.html.
`apple-notes-exporter`	Export a macOS Apple Notes folder to normalized JSON.
`bare-catch-linter`	Find empty JavaScript catch {} blocks that silently hide failures.
`branding-leak-guard`	Fail a build when forbidden brand or identity words appear in what you ship.
`email-groups-copier`	Split a big email list into copyable groups, offline in the browser.
`env-drift-check`	Detect Python environment drift before tests and gates run.
`gate-runner`	Run a repo's gate commands in order and stop at the first failure.
`git-secret-scan`	Scan a git repository's history, staged files or working tree for committed secrets.
`html-to-png-renderer`	Screenshot every HTML file in a folder to a crisp 2x PNG.
`live-homepage-snapshot`	Save the exact HTML a public URL returns right now to a timestamped file.
`update-bundle-kit`	Build and apply self-contained software update bundles with rollback.

How buying works

Enter your GitHub username. You'll see the account before you pay, so you can check it's yours.
Pay $49 on Stripe's checkout page.
Accept the invitation GitHub emails you. It's also waiting at github.com/dominares-tools/crawl/invitations. Invitations expire after 7 days; you can get a new one any time.

Clone and install:

git clone https://github.com/dominares-tools/crawl.git
pip install ./crawl

Questions

Do I need GitHub?: Yes. Access is a read-only invitation to a private repository, sent to the GitHub account you choose.
Can I get a refund?: Yes, within 14 days, no questions asked. Access ends when the refund goes through. Refund policy.
Who takes the payment?: Stripe. Your purchase is sold through Link, Stripe's checkout service, which also works out sales tax and sends your receipt.
Something wrong?: Open an issue in the repository or email josephlinares02@gmail.com.