Sweepy developer reference
Sweepy runs as a small Python project with a flat engine. This page maps each pipeline stage to the module that runs it, so you know where to make a change.
Audience
Read this page if you plan to change Sweepy's code or add a job source. Reviewers can use it to orient a pull request. It covers internals only: modules, dependencies, and Python versions.
tip
Skip this page if you want to run sweeps. Start with Installation.
Pipeline stages and modules
swee.py runs five stages in order. The terminal prints each stage as [1/5] through [5/5].
| Stage | Module | What it does |
|---|---|---|
| 1. Loading profile | engine/config.py | Loads and validates your YAML profile. The wizard in the same module writes one. |
| 2. Discovering roles | engine/discovery.py | Queries aggregator feeds through python-jobspy and classifies each platform. |
| 2. Discovering roles | engine/sources/ | Fetches Greenhouse, Ashby, Lever, Workable, and Built In NYC boards from public endpoints. |
| 2. Deduplicating | swee.py | Merges aggregator and source results into one card per job link. |
| 3. Validating URLs | engine/validator.py | Checks every link with an HTTP HEAD request and a GET fallback. It prunes dead postings. |
| 4. Scoring against resume | engine/scorer.py | Extracts posting and resume terms, then scores overlap on requirements and keywords. It tags postings that demand written answers. |
| 5. Routing resume tracks | engine/enrich.py | Picks the track with the highest term overlap and assigns the matching badge color. |
| 5. Enriching cards | engine/enrich.py | Builds card summaries, compensation lines, and tailored tip text. |
| 5. Default card PDF | engine/report.py | Renders the card layout, six cards per page by default, and returns the page count. |
| 5. Legacy table PDF | engine/pdf_builder.py | Renders the table layout when your profile sets report.style: legacy. |
| 5. Data exports | engine/exporter.py | Writes CSV and JSON files. It rasterizes PNG page images on request. |
| 5. Job tracking | engine/tracking.py | Adds the tracking columns and reads earlier exports so the values you enter come back. |
engine/session.py holds the card store for the stateful Model Context Protocol (MCP) flow. It keeps cards in a thread-safe map indexed by a stable hash of each job link.
Where to make a change
| You want to change | Start in |
|---|---|
| A job source or its endpoint | engine/sources/, plus a seed list in data/sources/ |
| How matches score | engine/scorer.py |
| What a card shows | engine/enrich.py, then engine/report.py |
| How links get checked | engine/validator.py |
| A command-line flag | swee.py |
| An MCP tool | server.py |
tip
server.py imports the same engine modules as the command-line tool. Test the MCP path after any engine change, because one fix affects both entry points.
Module reference
Entry points
| Module | Purpose |
|---|---|
swee.py | Primary orchestrator and argparse command-line entry point for the five-stage pipeline. |
server.py | Model Context Protocol (stdio) server that exposes engine tools to AI assistants. |
Engine modules
| Module | Purpose |
|---|---|
engine/config.py | Interactive wizard plus YAML loading and validation. |
engine/discovery.py | Scraper orchestration for python-jobspy and the deterministic sources, Applicant Tracking System (ATS) detection, and deduplication. |
engine/validator.py | Link uptime checks with soft-404 detection. Unverifiable links stay in the run. |
engine/scorer.py | Resume term extraction, percentage matching, tagging, and per-track scoring. |
engine/enrich.py | Card enrichment for summaries, compensation, track routing, and tailored tips. |
engine/session.py | Thread-safe card store with stable IDs for the stateful MCP flow. |
engine/report.py | Default card-layout PDF. |
engine/pdf_builder.py | Legacy table-layout PDF for report.style: legacy. |
engine/exporter.py | CSV and JSON exports, plus PNG rasterization of report PDFs. |
engine/tracking.py | Tracking columns and the carry-over read of earlier exports. |
engine/docx.py | DOCX to PDF conversion with python-docx and reportlab, plus a Microsoft Word fallback. |
engine/documents.py | Document dispatch for --convert and the MCP converter, covering DOCX, DOC, text, Markdown, and PDF. |
engine/logging.py | Opt-in debug logs under logs/. |
engine/cache.py | SQLite scrape cache keyed by query and engine set, with a one-hour time to live. |
engine/circuit_breaker.py | Counts consecutive source errors and pauses calls during a cooldown. |
engine/engine_defaults.py | Shared default site lists and engine settings, kept import-free to avoid cycles. |
engine/engines.py | Adapter registry and installer for the optional hirehunt, openroles, apify, and jobdrop engines. |
engine/history.py | Persistent record of postings already reported. A daily sweep reports only new roles. |
engine/normalize.py | Normalizes raw engine rows into the shared posting columns. |
engine/salary.py | Pay extraction from posting text when a board omits structured pay. |
engine/sweep_config.py | Applies the shared engine defaults to profile settings. |
engine/updater.py | Git-based update check and dependency reinstall for the checkout. |
Deterministic sources
| Module | Purpose |
|---|---|
engine/sources/__init__.py | Fetcher exports and seed-list loading from data/sources/. |
engine/sources/_common.py | Shared helpers for HTTP headers, title pre-filtering, and HTML-to-text conversion. |
engine/sources/greenhouse.py | Greenhouse job board fetcher. |
engine/sources/ashby.py | Ashby job board fetcher. |
engine/sources/lever.py | Lever postings fetcher. |
engine/sources/workable.py | Workable widget API fetcher. |
engine/sources/builtinnyc.py | Built In NYC category scraper. |
Engine adapters
| Module | Purpose |
|---|---|
engine/adapters/__init__.py | Registry that maps each engine name to its adapter. |
engine/adapters/base.py | ScraperAdapter base class, shared User-Agent header, and result-shape helpers. |
engine/adapters/jobspy_adapter.py | python-jobspy aggregator adapter for LinkedIn, Indeed, Glassdoor, and ZipRecruiter. |
engine/adapters/ats_adapter.py | Adapter over the deterministic ATS fetchers in engine/sources/. |
engine/adapters/hirehunt_adapter.py | Optional hirehunt engine adapter. |
engine/adapters/openroles_adapter.py | Optional openroles engine adapter. |
engine/adapters/apify_adapter.py | Optional Apify actor adapter. |
engine/adapters/jobdrop_adapter.py | Optional jobdrop engine adapter. |
Data, scripts, and tests
| Path | Purpose |
|---|---|
data/sources/*.txt | Seed lists for boards, companies, organizations, and categories. |
scripts/make_sample.py | Builder for the sample report stored in docs/. |
tests/ | pytest suite with a file per area: adapters, config, documents, docx, engines, enrich, history, logging, and report. Further files cover salary, scorer, server, session, sources, swee, tracking, updater, and validator. |
note
Add a dedicated test file when you change discovery.py, exporter.py, or pdf_builder.py. Put the file in tests/.
Dependencies
Runtime packages come from requirements.txt. Versions below match that file.
| Package | Version | What it does for Sweepy |
|---|---|---|
reportlab | 5.0.1 | Builds the styled PDF report with clickable links. |
pymupdf | 1.28.2 | Extracts text from resume PDFs and rasterizes PNG exports. |
python-docx | 1.2.0 | Reads .docx resumes and rebuilds DOCX text as a PDF without Word. |
requests | 2.32.5 | Reaches the ATS endpoints and checks each job link for a live response. |
beautifulsoup4 | 4.15.0 | Parses HTML from the Greenhouse, Lever, Workable, and Built In NYC sources. |
pyyaml | 6.0.3 | Reads and writes YAML profiles. |
questionary | 2.1.1 | Drives the interactive wizard questions. |
prompt-toolkit | 3.0.53 | Drives the wizard's interactive text input. |
python-jobspy | 1.2.0 | Scrapes LinkedIn, Indeed, Glassdoor, and ZipRecruiter with no API credentials. |
pandas | 2.2.3 | Structures postings and builds the CSV and JSON exports. |
mcp | 1.30.0 | Runs the Model Context Protocol server. |
anyio | 4.12.1 | Runs the asynchronous MCP transport. |
requirements.txt lists one optional package too. Uncomment docx2pdf when Word should build the PDF.
docx2pdf(0.5.2, optional) renders DOCX to PDF through Microsoft Word on Windows. Sweepy calls it whenever Microsoft Word exists on the machine.
note
mcp stays on 1.x on purpose. The 2.x SDK serves only the 2026-07-28 protocol era. A client that probes server/discover and then falls back to the initialize handshake gets a -32022 error. VS Code Copilot follows that fallback.
tip
pandas stays on 2.x on purpose. Pandas 3.x requires Python 3.11 or newer, and the CI matrix still tests Python 3.10.
Development packages (requirements-dev.txt)
| Package | Version | What it does |
|---|---|---|
pytest | 9.1.1 | Runs the test suite in tests/. |
pytest-cov | 7.1.0 | Reports coverage for swee.py, server.py, and engine/scorer.py. |
ruff | 0.17.0 | Lints the code and sorts imports. |
tomli | 2.2.1 | Supplies tomllib for the Python 3.10 test runs. |
Terminal
pip install -r requirements.txt
pip install -r requirements-dev.txt
ruff check .
pytest -q
Requirements and supported Python versions
| Item | Detail |
|---|---|
| Python floor | Sweepy requires Python 3.10 or newer and targets 3.10 syntax in every module. |
| Tested range | The CI workflow in .github/workflows/ci.yml runs Python 3.10, 3.11, and 3.12 on Ubuntu. |
| Lint target | pyproject.toml sets ruff target-version = "py310". A newer-only construct fails the lint job. |
| Line length | Ruff enforces a 100-character line limit and ignores E501. |
| CI steps | The job installs both requirements files and runs ruff check . before pytest. |
note
pyproject.toml declares requires-python = ">=3.10", so pip refuses an older interpreter at install time. The CI matrix and the ruff target define the tested range. Python 3.13 works in practice but stays outside that range.
Python 3.13
The pymupdf package may compile from source when no pre-built wheel exists. Install the Microsoft C++ Build Tools and select the "Desktop development with C++" workload before you install dependencies.
Repository layout
Repository layout
swee.py CLI entry point and sweep orchestrator
server.py MCP server over stdio
engine/ Pipeline modules
engine/sources/ Deterministic ATS fetchers
engine/adapters/ Aggregator and optional engine adapters
data/sources/ Seed lists for each fetcher
profiles/ YAML search profiles
results/ Generated PDF, CSV, JSON, and PNG files
logs/ Opt-in debug logs
styles/ Vale style rules for the docs
tests/ pytest suite
When you change a tool signature in server.py, update the MCP tool reference in the same commit.