Scraping sources and platform support

Sweepy finds jobs in two ways. It queries four public ATS boards and one New York job board directly. It can also read aggregator feeds through python-jobspy. This page lists every source and shows how to point Sweepy at your field.

Default sources

Out of the box, Sweepy sweeps five deterministic sources plus one aggregator path. Deterministic sources use public endpoints that anyone can open. They don't need an account, a login, or a paid service.

Source	Config key	Seed file	Endpoint
Greenhouse	sources.greenhouse.boards	data/sources/greenhouse_boards.txt	boards-api.greenhouse.io/v1/boards/{board}/jobs?content=true
Ashby	sources.ashby.orgs	data/sources/ashby_orgs.txt	api.ashbyhq.com/posting-api/job-board/{org}
Lever	sources.lever.companies	data/sources/lever_companies.txt	api.lever.co/v0/postings/{company}?mode=json
Workable	sources.workable.slugs	data/sources/workable_slugs.txt	apply.workable.com/api/v1/widget/accounts/{slug}
Built In NYC	sources.builtinnyc.categories	data/sources/builtin_categories.txt	builtinnyc.com/{category}
Glassdoor	sources.jobspy.glassdoor	None	via python-jobspy

Gating rules

Sweepy runs a deterministic source when both settings below hold:

1. You set sources.<name>.enabled to true.
2. You set platforms.<name> to true where a matching setting exists.

Both keys default to true.

Keep these exceptions in mind:

• Built In NYC has no platforms.builtinnyc setting. The engine honors sources.builtinnyc.enabled alone. Each card includes company, workplace, location, and pay from the listing page. Set sources.builtinnyc.detail_pages: true to read full posting text too. Built In challenges those pages after a few requests, so Sweepy stops at the first challenge.
• Glassdoor runs through python-jobspy and honors sources.jobspy.glassdoor. No platforms.glassdoor key exists, so that sources key alone switches it off.

To switch a source off, set one of its keys to false.

If your profile predates v2.2.0

warning

Greenhouse and Lever now run by default. Sweepy changed this behavior in v2.2.0. Existing profiles discover the new postings unless you turn those sources off. No flag default changed.

The flags platforms.ashby: false, platforms.workable: false, platforms.greenhouse: false, and platforms.lever: false now stop those deterministic seed sources. Before v2.2.0, they filtered only python-jobspy results.

Turn off a source with sources.<name>.enabled: false or its platforms.<name>: false.

How seed files work

Sweepy uses seed files in data/sources/ as fallbacks when your profile leaves source lists empty.

• An empty list, such as boards: [], falls back to the curated seed file.
• An explicit list overrides the seeds entirely. Your list replaces the seeds, so the engine won't merge the two.

Your list wins. You pick the boards and the job categories.

Examples for your field

Add the ATS boards, orgs, companies, slugs, and categories you care about under the sources section of your profile.

Legal

profiles/paralegal_nyc.yaml

sources:
  ashby:
    orgs:
      - "crosby"
      - "normlaw"
      - "hargrove-firm"
      - "lawhive"
  workable:
    slugs:
      - "shegerianand-associates-1"
      - "harper-james"
      - "legalvision"
  builtinnyc:
    enabled: true
    categories:
      - "jobs/legal"
      - "jobs/legal/search/paralegal"

Technical writing

profiles/tech_writer.yaml

sources:
  greenhouse:
    enabled: true
    boards:
      - "gitlab"
      - "cloudflare"
  ashby:
    orgs:
      - "modal"
      - "perplexity"
      - "replit"
  lever:
    enabled: true
    companies:
      - "spotify"
  workable:
    slugs:
      - "soda-data-nv"
  builtinnyc:
    enabled: true
    categories:
      - "jobs/technical-writer"

Healthcare/biotech

profiles/healthcare.yaml

sources:
  workable:
    slugs:
      - "bask-health-1"
      - "atria-health"
  builtinnyc:
    enabled: true
    categories:
      - "jobs/healthcare"

tip

Copy the block closest to your field, then swap in the companies you already follow. Board tokens come from the company's public careers address.

Custom platforms

Register custom ATS domains under platforms.custom. The engine reads every job link it discovers and compares it with your domains. Sweepy routes matches into their own report directory.

profiles/paralegal_nyc.yaml

platforms:
  custom:
    - name: "LawCrossing"
      domain: "lawcrossing.com"
      enabled: true
    - name: "Indeed Legal"
      domain: "indeed.com/legal"
      enabled: true

Each entry takes three fields:

Field	Required	What it does
name	Yes	The label Sweepy shows for this platform in reports.
domain	Yes	The web domain or path the engine compares with each job link.
enabled	No	Set to false to keep the entry but skip the match. Defaults to true.

Optional engines

Sweepy includes the ats and jobspy engines by default. Optional engines add boards when you download them.

Engine	Boards	Before you install
hirehunt	LinkedIn, Indeed, Naukri, Shine, Internshala, Unstop, Google Careers, Amazon, Meta, Apple, Netflix, Microsoft	None
openroles	Greenhouse, Lever, Ashby, and Workday boards you name	companies slugs
jobdrop	33 boards, including Google, Wellfound, Workday, iCIMS, Dice, and USAJobs	Pass --allow-engine-conflicts alongside JobSpy
apify	LinkedIn through a managed actor	APIFY_TOKEN and Apify billing

Install every compatible engine at once:

Terminal

sweepy --install-engines all

The all set skips JobDrop because it pins curl-cffi<0.8 while JobSpy needs curl_cffi>=0.16.2.

To install JobDrop on its own:

Terminal

sweepy --install-engines jobdrop --allow-engine-conflicts

The setup wizard's Download job engines option runs the same installs. Then list the engine under sources.enabled and set its board options:

profiles/paralegal_nyc.yaml

sources:
  enabled: [jobspy, ats, openroles]
  openroles:
    companies:
      - "figma"
      - "stripe"

Bandwidth trade-off

Greenhouse boards ask for full content in one request per board by appending ?content=true. Descriptions, departments, and pay metadata then arrive ready for scoring. Some boards return hundreds of postings, so a sweep can pull low tens of megabytes across all seeded boards.

Sweepy keeps that cost bounded in three ways:

• It pre-filters postings by your profile titles while normalizing each record.
• It sends one request per seed source at a time, never in parallel.
• It pauses for the SWEEPY_SOURCE_DELAY interval between requests, which defaults to 0.2 seconds.

Smaller seed lists finish faster. If a sweep feels slow, trim your seed lists first.

Platform compatibility

Because Sweepy scrapes plain HTML and runs without a headless browser, some platform types return less data than others.

Platform type	Examples	Compatibility	Caveats
Public ATS	Greenhouse, Ashby, Lever, Workable, SmartRecruiters, Breezy HR	Fully supported	Sweepy parses standard server-rendered job pages and checks each link for a live response.
Client-rendered SPAs	Workday on myworkdayjobs.com, Oracle HCM	Limited	These platforms render job pages with client-side JavaScript. Without a headless browser, initial HTML payloads may lack descriptions.
Auth-walled portals	Handshake, corporate intranets	Unsupported	These portals require session cookies or SSO credentials.
Anti-bot aggregators	Boards with Cloudflare Turnstile or PerimeterX	Rate-limited	Automated HTTP requests may trigger HTTP 403 blocks without proxy rotation.

note

You may still see Workday postings in reports when an aggregator feed includes them. Sweepy validates the link and scores the role from whatever text the payload contains.