This guide shows how to find a company's board, pull every open job from Greenhouse and Lever, get descriptions and salaries, filter, export to CSV and detect new postings between runs. All the code was run on 2 November 2026 (NZ time) against Airbnb (Greenhouse) and Zoox (Lever).

Step 1: Find the company's board name (the "slug")

Every request needs the company's board name. You can usually read it straight from a job link:

Companies with their own branded careers site (Airbnb's is careers.airbnb.com) often still run on Greenhouse behind the scenes. The gh_jid parameter in a job URL is the giveaway.

Step 2: Pull every job from Greenhouse

The Greenhouse Job Board API says it plainly: "Job Board data is publicly available, so authentication is not required for any GET endpoints."

GET https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true

One request returns the whole board, with no pagination. Airbnb's had 202 jobs, and meta.total matched. Without content=true you get titles, locations, dates and URLs (155 KB for Airbnb). With it you also get departments, offices and the full description (2.9 MB), so turn it on only if you need those fields.

Two details to know:

• The description in content is HTML-escaped HTML (<p> rather than <p>). Unescape it before you strip tags or convert it.
• There are two dates: first_published (when the post went live) and updated_at (any edit). Use first_published for "posted in the last N days". For Airbnb's engineering roles, updated_at made 34 look recent, against 20 by first_published.

Step 3: Pull every job from Lever

Lever's postings API is just as open:

GET https://api.lever.co/v0/postings/{slug}?mode=json

With no limit, it returned all 227 of Zoox's jobs in one response. It also supports skip and limit for paging, and server-side filters for location, commitment, team and department. A slug that doesn't exist returns 404. A board that exists but has nothing open returns an empty list [], which is worth telling apart in your logs.

Differences from Greenhouse:

• createdAt is a Unix timestamp in milliseconds.
• Department, team, location and commitment sit inside categories. categories.allLocations lists every location for multi-location roles.
• workplaceType is onsite, hybrid or remote.
• Descriptions come in both HTML (description) and plain text (descriptionPlain).
• salaryRange (min, max, currency, interval) appears when the company publishes pay. 211 of Zoox's 227 jobs had it. Spotify's board had none.

Step 4: One script for both

Here's a small script that normalises both feeds into the same row format. It needs only requests:

# pip install requests
import html, re
from datetime import datetime, timedelta, timezone
import requests

S = requests.Session()
S.headers["User-Agent"] = "jobs-research-script (you@example.com)"

def strip_html(s):
    return re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", html.unescape(s or ""))).strip()

def greenhouse_jobs(slug, with_description=False):
    # content=true adds departments, offices and the description (about 20x bigger response)
    r = S.get(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs", params={"content": "true"}, timeout=30)
    r.raise_for_status()  # 404 = no public board under that slug
    for j in r.json()["jobs"]:
        yield {
            "ats": "greenhouse", "company": j.get("company_name") or slug, "id": str(j["id"]),
            "title": j["title"],
            "department": (j.get("departments") or [{}])[0].get("name"),
            "location": (j.get("location") or {}).get("name"),
            "posted": datetime.fromisoformat(j["first_published"]) if j.get("first_published") else None,
            "url": j["absolute_url"],
            "description": strip_html(j.get("content")) if with_description else None,
        }

def lever_jobs(slug, with_description=False):
    r = S.get(f"https://api.lever.co/v0/postings/{slug}", params={"mode": "json"}, timeout=30)
    r.raise_for_status()  # 404 = unknown slug; [] = board exists but nothing open
    for j in r.json():
        cat, pay = j.get("categories") or {}, j.get("salaryRange") or {}
        yield {
            "ats": "lever", "company": slug, "id": j["id"],
            "title": j["text"],
            "department": cat.get("department"),
            "location": cat.get("location"),
            "workplace": j.get("workplaceType"),
            "posted": datetime.fromtimestamp(j["createdAt"] / 1000, tz=timezone.utc),
            "url": j["hostedUrl"],
            "salary": (pay.get("min"), pay.get("max"), pay.get("currency"), pay.get("interval")) if pay else None,
            "description": j.get("descriptionPlain") if with_description else None,
        }

cutoff = datetime.now(timezone.utc) - timedelta(days=30)
jobs = [*greenhouse_jobs("airbnb"), *lever_jobs("zoox")]
recent_eng = [j for j in jobs if "engineer" in j["title"].lower() and j["posted"] and j["posted"] >= cutoff]
print(len(jobs), "open roles;", len(recent_eng), "engineering roles posted in the last 30 days")

Output: 429 open roles; 49 engineering roles posted in the last 30 days (20 at Airbnb, 29 at Zoox). Greenhouse includes a company_name field. Lever doesn't, so the slug stands in for the name.

Step 5: Get salaries from Greenhouse

Greenhouse doesn't put pay in the job list. It's on the single-job endpoint, behind a pay_transparency flag:

def greenhouse_pay(slug, job_id):
    r = S.get(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs/{job_id}",
              params={"pay_transparency": "true"}, timeout=30)
    r.raise_for_status()
    return [(p["min_cents"] / 100, p["max_cents"] / 100, p["currency_type"], p["title"])
            for p in r.json().get("pay_input_ranges") or []]

print(greenhouse_pay("airbnb", 8316130))  # [(212000.0, 265000.0, 'USD', 'Pay Range')]

Amounts are in cents, and a job can have several ranges (for example one per country). It's one request per job, so only call it for the jobs you keep. 13 of the 20 recent Airbnb engineering roles had a published range.

Step 6: Export to CSV and spot new jobs

The usual next step is "tell me what's new since yesterday". Store the job IDs from each run and compare:

import csv, json, pathlib

STATE = pathlib.Path("seen_jobs.json")
seen = json.loads(STATE.read_text()) if STATE.exists() else {}

current = {f'{j["ats"]}:{j["company"]}:{j["id"]}': j for j in jobs}
new = [j for key, j in current.items() if key not in seen]
removed = [seen[key] for key in seen if key not in current]
print(len(new), "new,", len(removed), "removed since last run")

STATE.write_text(json.dumps({k: {"title": j["title"], "url": j["url"]} for k, j in current.items()}))

with open("jobs.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["ats", "company", "title", "department", "location", "posted", "url"],
                       extrasaction="ignore")
    w.writeheader()
    w.writerows(jobs)

The first run reports everything as new (429 here), so treat it as a baseline. Run it daily with cron or a GitHub Actions schedule, and send new to Slack or email.

Be a good citizen

These feeds exist so companies can show their jobs on their own sites and in job aggregators, and reading them is the intended use. Still:

• Keep the rate low. One request per board per run is all you need for the job list. Add a short pause between boards if you watch hundreds.
• Set a real User-Agent with a contact address, as in the script.
• Cache descriptions. They rarely change. Fetch them once per job, not on every run.
• Don't collect personal data. Some descriptions include a recruiter's name or email. Strip them if you republish.

Where the DIY script runs out

The script above is enough for a handful of companies on Greenhouse and Lever. It gets harder when:

• your list includes Ashby, Workable, SmartRecruiters, Recruitee or Personio, each with its own format;
• you have company names, not slugs, and need to work out which ATS each one uses;
• you want per-company summaries (open roles by department, location and remote/hybrid/onsite) and new/removed tracking without running a server.

We built ATS Jobs & Hiring Signals on Apify for that. It reads the same public feeds for seven ATSs (including Lever's EU data centre) and returns one row format plus a free summary row per company. Here's the same query as Step 4:

# pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("flyingkiwi/ats-jobs-hiring-signals").call(run_input={
    "companies": ["https://boards.greenhouse.io/airbnb", "https://jobs.lever.co/zoox"],
    "titleKeywords": ["engineer"],
    "postedWithinDays": 30,
})
rows = list(client.dataset(run.default_dataset_id).iterate_items())
jobs = [r for r in rows if r["type"] == "job"]
summaries = [r for r in rows if r["type"] == "company"]

(That's apify-client 3.x, where call() returns a Run object. With 1.x or 2.x, use run["defaultDatasetId"].)

The run took about 4.5 seconds and returned the same 49 jobs as the DIY script (20 Airbnb, 29 Zoox), plus two summary rows. Airbnb's summary counted 202 open roles, 92 posted in the last 30 days, and 155 remote, 37 hybrid and 10 onsite. Zoox's counted 227 roles, 100 of them in Software. Set "monitorMode": true and schedule it to get only new jobs on each run.

Honest gaps:

• Greenhouse salaries aren't included yet. The Actor reads Lever's salaryRange (27 of the 29 Zoox rows had pay) but doesn't make the per-job pay_transparency call from Step 5, so the Airbnb rows had none. If you need Greenhouse pay, use that snippet.
• Workday, iCIMS, LinkedIn and Indeed aren't covered. Many large enterprises use Workday.
• Lever names come from the slug (zoox becomes "Zoox"). Set companyNames if a slug doesn't match the real name.

It costs $0.005 per run plus $0.001 per job returned, and summary rows are free, so this run was about $0.054. The DIY script is free. Pick whichever costs you less time.

We (flyingkiwi) built and maintain ATS Jobs & Hiring Signals. The numbers in this post come from runs on 2 November 2026 (NZ time).

Tool: https://apify.com/flyingkiwi/ats-jobs-hiring-signals

This guide converts a real PDF to Markdown, splits it into chunks that keep their heading and page number, cleans up the usual PDF debris and checks for missing pages. Everything uses free, open-source tools. At the end there's a hosted option for when you'd rather call an API.

The test file throughout is the original RAG paper, Lewis et al. 2020 (19 pages, arXiv PDF).

Why Markdown, and not plain text?

Plain-text extraction gives you one long string. Markdown keeps the structure that makes chunking and retrieval work:

• Headings (## 2 Methods, ### 2.1 Models) give you natural chunk boundaries, and a heading is useful metadata to embed alongside the chunk.
• Tables stay as rows and columns instead of a jumble of numbers.
• Lists stay as lists, so a step-by-step procedure doesn't merge into one paragraph.

LLMs also read Markdown well, so the same text works for embedding and for the context you pass to the model.

Step 1: Pick a converter

There's no single best PDF-to-Markdown tool. These are the ones worth trying, all free to run yourself:

Start with the lightest one that handles your documents, and test it on your own PDFs, not someone else's benchmark. A tool that's great on arXiv papers can struggle with scanned invoices or two-column brochures.

We'll use PyMuPDF4LLM because it's quick and returns one result per page, which makes page numbers easy.

Step 2: Convert the PDF page by page

# pip install "pymupdf4llm<2" langchain-text-splitters
import pymupdf4llm

pages = pymupdf4llm.to_markdown("rag.pdf", page_chunks=True)
print(len(pages))                          # 19
print(pages[0]["metadata"]["page_number"]) # 1
print(pages[0]["text"][:300])

A note on that version pin: at the time of writing, pip install pymupdf4llm installs version 2.0, which is an empty placeholder package. The PyMuPDF team moved the code into PyMuPDF 2.x itself. Pinning <2 gets you the 1.x release with the API used here.

On our test machine (a small cloud VM, CPU only), the whole 19-page paper took about 2.7 seconds. page_chunks=True returns a list of dictionaries, one per page, each with the page's Markdown in text and its 1-based page_number in metadata. Headings came out as ### **2 Methods** and #### **2.1 Models**. PDFs don't actually store headings, so every tool infers them from font size and weight. Spot-check them on your own files.

Step 3: Chunk by heading, keep the page number

Splitting every 1,000 characters regardless of structure works, but it cuts sections in half and loses the heading. A better default is to split on headings first, then split any section that's still too long:

import re
import pymupdf4llm
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

pages = pymupdf4llm.to_markdown("rag.pdf", page_chunks=True)

header_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3"), ("####", "h4")],
    strip_headers=False,
)
size_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)

chunks, current = [], {}
for page in pages:
    page_no = page["metadata"]["page_number"]  # 1-based in pymupdf4llm 1.x
    for section in header_splitter.split_text(page["text"]):
        # a page that starts mid-section keeps the previous heading
        current = section.metadata or current
        heading = next((current[k] for k in ("h4", "h3", "h2", "h1") if k in current), None)
        for piece in size_splitter.split_text(section.page_content):
            chunks.append({
                "text": piece,
                "page": page_no,
                "heading": re.sub(r"[*_]", "", heading).strip() if heading else None,
                "source": "rag.pdf",
            })

print(len(chunks), "chunks")

This gave 110 chunks. Each one carries page, heading and source. Store those as metadata in your vector database so an answer can say "Lewis et al., page 3, section 2 Methods" instead of just quoting text.

Splitting page by page means a section that runs across a page break becomes two chunks. For citations that's usually what you want. If you'd rather have whole sections, join the pages first and insert a marker such as <!-- page 3 --> at each boundary, so you can still work out which page a chunk starts on.

Step 4: Clean up the PDF debris

Look at a few chunks before you embed anything. In ours, the chunk for "2 Methods" ended with a footnote and a lone 2, which is the page number from the footer. Common debris in PDF text:

• Page numbers, running headers and footers repeated on every page.
• Ligatures: many PDFs store "fi" and "fl" as single characters (fi, fl). The text looks fine on screen, but "fine-tuned" won't match a search for "fine-tuned", and embeddings treat them differently.
• Words broken across lines with a hyphen, such as knowl- + edge.

PyMuPDF4LLM already joined the broken words and expanded the ligatures in our paper. Not every extractor does: pdfminer-based tools often keep both. If yours doesn't, a few lines fix most of it:

import re
import unicodedata

def clean(text: str) -> str:
    text = unicodedata.normalize("NFKC", text)       # "fi" -> "fi"
    text = re.sub(r"(\w)-\n(\w)", r"\1\2", text)      # "knowl-\nedge" -> "knowledge"
    text = re.sub(r"\n\s*\d{1,3}\s*$", "", text)      # lone page number at the very end
    return text

The hyphen rule is a trade-off. In this paper it would also glue genuine compounds that happen to break at the line end, like knowledge- + intensive and cross- + encoder. For retrieval that's a smaller problem than half-words, but don't use this rule on text you'll show to users as-is.

Step 5: Check for pages that came out empty

This is the step most pipelines skip. A scanned page, or a page where the text was converted to vector shapes, has no text layer. Every converter returns nothing for it, and nothing raises an error. Check each page's text length and whether the page contains images:

import pymupdf  # installed with pymupdf4llm

def suspicious_pages(path, min_chars=50):
    doc = pymupdf.open(path)
    flagged = []
    for page in doc:
        if len(page.get_text().strip()) < min_chars:
            reason = "image only (scanned?)" if page.get_images() else "blank or near-blank"
            flagged.append((page.number + 1, reason))
    return flagged

print(suspicious_pages("rag.pdf"))  # [] for this paper

Our paper came back clean. For a pile of real-world documents, route any flagged file to OCR (Docling and Marker both have it, as does Tesseract via ocrmypdf) or to a human, instead of embedding a document with holes in it. A word count per page that's far below the rest of the document is another cheap warning sign.

Step 6: Embed and store

From here it's standard RAG: embed text, keep page, heading and source as metadata, and pass the metadata through to your prompt so the model can cite. Re-run Steps 4 and 5 whenever you add documents, because a new batch often brings a new kind of mess.

The hosted option: an API instead of a conversion server

If you'd rather not run and maintain this (or you need to convert Word, PowerPoint and Excel files and shared Google Drive, Dropbox or SharePoint folders, not just PDFs), we built Document to Markdown on Apify. You send a URL, it returns Markdown, chunks with page and heading, and a per-document quality report covering empty and scanned pages. OCR for scanned pages is optional.

We ran it on the same paper:

# pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("flyingkiwi/document-to-markdown").call(run_input={
    "urls": ["https://arxiv.org/pdf/2005.11401"],
    "chunkSize": 1000,
    "chunkOverlap": 100,
})

for doc in client.dataset(run.default_dataset_id).iterate_items():
    print(doc["title"], doc["pageCount"], len(doc["chunks"]), doc["quality"]["possibleContentLoss"])

(That's apify-client 3.x, where call() returns a Run object. With the older 1.x and 2.x clients, use run["defaultDatasetId"].)

Or with plain HTTP, which waits for the run and returns the results in one request (fine for small jobs; long runs should use the asynchronous endpoints):

curl -X POST "https://api.apify.com/v2/acts/flyingkiwi~document-to-markdown/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://arxiv.org/pdf/2005.11401"], "chunkSize": 1000, "chunkOverlap": 100}'

The run took about 7 seconds. It returned the title "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", 19 pages, 9,692 words and 96 chunks averaging about 730 characters. Every chunk had a heading and a page number:

{
  "index": 17,
  "page": 4,
  "heading": "3 Experiments",
  "charCount": 753,
  "text": "### 3 Experiments\n\nWe experiment with RAG in a wide range of knowledge-intensive tasks. For all experiments, we use\na single Wikipedia dump for our non-parametric knowledge source. ..."
}

The quality report showed 19 of 19 pages with text, no scanned pages and possibleContentLoss: false. The cost was $0.006: $0.003 to start the run and $0.003 for the document (up to 20 pages, then $0.001 per extra page).

To be straight about how it compares with the DIY route on this paper: PyMuPDF4LLM's output was cleaner on the text itself. Document to Markdown left the ligatures (fi) and the line-end hyphens in place, so run the clean() function from Step 4 on its output too. Where it helps is everything around the conversion: one call for mixed file types and whole shared folders, chunks and page numbers built in, the empty-page check on every document, and nothing to install or keep updated.

Which route should you take?

• A few hundred PDFs, and you're comfortable with Python: go DIY. PyMuPDF4LLM (mind the AGPL licence) or Docling, plus the chunking and checks above, will do the job for free.
• Papers full of equations, or heavily laid-out PDFs: try Marker or MinerU and compare them on your own files.
• Mixed Office files and shared folders, called from an app or an agent, without running a server: a hosted converter saves the maintenance. Check whichever one you pick for empty pages anyway.

We (flyingkiwi) built and maintain Document to Markdown. The numbers in this post come from runs on 2 November 2026 (NZ time): PyMuPDF4LLM 1.28.2 and langchain-text-splitters 1.1.3 locally, and Document to Markdown on Apify.

Tool: https://apify.com/flyingkiwi/document-to-markdown

This post shows how to do that loop with Apple's free public data, first with a few lines of Python and then with a hosted tool that adds complaint counting and an AI idea report. It uses a real run and includes the weak ideas.

Step 1: Get the raw data yourself

Apple publishes the raw material for free:

• public RSS feeds for the top charts in every country and category;
• a public "most recent reviews" feed for each app;
• the iTunes Lookup API for app details.

There's no login and no scraping of web pages. This script works as written:

import requests

# Top 50 free Health & Fitness apps in the US (genre 6013)
chart = requests.get("https://itunes.apple.com/us/rss/topfreeapplications/limit=50/genre=6013/json", timeout=30).json()
apps = [(e["im:name"]["label"], e["id"]["attributes"]["im:id"]) for e in chart["feed"]["entry"]]
print(apps[:3])

# Most recent reviews for the first app
name, app_id = apps[0]
feed = requests.get(f"https://itunes.apple.com/us/rss/customerreviews/page=1/id={app_id}/sortby=mostrecent/json", timeout=30).json()
reviews = [(int(e["im:rating"]["label"]), e["title"]["label"]) for e in feed["feed"].get("entry", []) if "im:rating" in e]
low = [r for r in reviews if r[0] <= 2]
print(name, len(reviews), "reviews,", len(low), "rated 1-2 stars")

On 28 October 2026 (NZ time) the top three were Strava, Flo and Planet Fitness. Page 1 of Strava's review feed had 50 reviews, 21 of them rated 1–2 stars. From here, the work is:

• paging through reviews (Apple returns at most 500 recent reviews per app and country);
• counting which words and phrases recur in the negative reviews;
• tracking rank changes between runs;
• turning all of that into something you can decide on.

Step 2: The hosted version

We built App Store Trends & AI App-Idea Report on Apify to do that whole loop in one run. It:

1. pulls a chart;
2. picks apps (rising or top);
3. counts recurring complaint terms in their recent 1–2 star reviews;
4. optionally asks an AI model to suggest ideas based on those complaints.

This is the exact input we used:

{
  "country": "us",
  "chart": "top-free",
  "genre": "health-fitness",
  "chartLimit": 50,
  "appsToAnalyse": 5,
  "pickAppsBy": "rising",
  "reviewsPerApp": 100,
  "reportFocus": "ideas a solo developer could build and launch in about a month"
}

reportFocus is free text that steers the AI section. The report is on by default. Set "generateReport": false if you only want the chart and review data, with no AI.

From code:

# pip install -U "apify-client>=3"
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("flyingkiwi/app-store-trends-app-ideas").call(run_input={
    "country": "us", "chart": "top-free", "genre": "health-fitness",
    "chartLimit": 50, "appsToAnalyse": 5,
})
report = client.key_value_store(run.default_key_value_store_id).get_record("REPORT")
print(report["value"])

The run took about 27 seconds. The dataset holds the chart rows and per-app review stats, and the Markdown report is saved under the REPORT key. The excerpts below are copied from a run on 28 October 2026 (NZ time), shortened but not otherwise edited.

Step 3: Start with the part that isn't AI

The most useful section is computed directly from the reviews. For each app it gives the recent average rating, the share of negative reviews and how many 1–2 star reviews mention each term:

• Planet Fitness (#3, recent avg 1.72★ from 100 reviews, 79% negative): update (24), check (24), new (21), back (12), membership (11).
• Flo Cycle & Period Tracker (#2, recent avg 2.73★ from 100 reviews, 54% negative): pay (18), free (10), money (9), subscription (8), tracking (8).
• Strava: Run, Bike, Walk (#1, recent avg 3.1★ from 100 reviews, 43% negative): subscription (8), update (8), new (7), annoying (6), everything (5).

Single words like "check" and "back" don't mean much on their own, so each app also gets an anonymised example review. For Planet Fitness, the example was about a recent update making check-in confusing for older members. The terms tell you where to look; you still have to read the reviews.

The "What's rising" list flagged Rythm Health at #35, released 20 days before the run, as the strongest mover. On a first run, "rising" is based only on chart position and app age. Real rank changes appear once you've run the same chart more than once.

Step 4: Read the AI ideas sceptically

The report then gives AI-written ideas, five in this run. Each comes with who it's for, the evidence it's based on, must-haves, a money model and a risk. The best one from this run:

"Cycle Sync Companion": A free, privacy-focused period tracker that offers core tracking without a subscription.

• Evidence: Flo Cycle & Period Tracker: pay (18 reviews), free (10 reviews), money (9 reviews), subscription (8 reviews)...
• Risk: Competing with established, feature-rich (but often paid) apps.

The weakest ones:

• "Gym Check-in Pro", a nicer check-in app for Planet Fitness members. The report itself lists the problem as the risk: it depends on one gym chain's check-in technology, so an outside developer can't really build it.
• "No-Frills Health Log", which was based on a single review mentioning crashes.

Two more issues showed up in that run:

• The summary printed an unrounded rating ("1.84845") for one app. Ratings are now rounded to 2 decimals.
• A note at the bottom said two quoted phrases couldn't be matched to any collected review, so treat those as paraphrase. Quotes are checked against the real reviews, and unmatched ones are listed in the report row as unverifiedQuotes.

That's the honest picture. The computed complaint data is reliable. The ideas are a starting list to argue with, not a validated plan. Whichever route you take, the useful habit is the same: start from counted complaints with real example reviews, and treat any generated idea as a hypothesis.

Honest limits

• No downloads or revenue. Apple publishes chart positions, not market size.
• Reviews skew recent and negative. The feed returns the most recent public reviews, which over-represent the latest update and unhappy users.
• Rank changes need repeat runs. Schedule it daily or weekly on the same chart.
• The AI can be wrong. It runs through Apify's OpenRouter proxy (google/gemini-2.5-flash by default), only sees the data collected in the run, and is told not to invent numbers.
• App Store only. No Google Play.
• You're not the only one doing this. Several tools work on the same idea. Compare before you commit.

What it costs

The DIY script costs nothing. The hosted version charges:

• $0.005 per run start;
• $0.001 per chart row;
• $0.02 per analysed app;
• $0.50 per AI report, charged only if the report is produced.

The AI model usage is billed separately by Apify's proxy. With the default model that was about $0.01 per report on paid Apify plans in our tests, and roughly $0.08 on the free plan. The run above comes to about $0.66 plus that AI usage. Without the report it would be about $0.16.

We (flyingkiwi) built and maintain App Store Trends & AI App-Idea Report. Apple and App Store are trademarks of Apple Inc.; the tool isn't affiliated with Apple.

Tool: https://apify.com/flyingkiwi/app-store-trends-app-ideas

This post shows how to pull open roles straight from the public job-board feeds of Greenhouse, Lever and Ashby, how to normalise them into one format, and how to turn the result into hiring signals.

Step 1: Use the public job-board feeds (no scraping needed)

Greenhouse, Lever, Ashby and several other ATSs publish public job-board feeds. These are the same endpoints their own careers widgets use, so there's no login, no headless browser and no LinkedIn scraping. This script works as written:

import requests

def greenhouse(slug):
    jobs = requests.get(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs", timeout=30).json()["jobs"]
    return [{"title": j["title"], "location": j["location"]["name"], "url": j["absolute_url"]} for j in jobs]

def lever(slug):
    jobs = requests.get(f"https://api.lever.co/v0/postings/{slug}?mode=json", timeout=30).json()
    return [{"title": j["text"], "location": j["categories"].get("location"), "url": j["hostedUrl"]} for j in jobs]

def ashby(slug):
    jobs = requests.get(f"https://api.ashbyhq.com/posting-api/job-board/{slug}", timeout=30).json()["jobs"]
    return [{"title": j["title"], "location": j["location"], "url": j["jobUrl"]} for j in jobs]

for name, rows in [("airbnb", greenhouse("airbnb")), ("spotify", lever("spotify")), ("openai", ashby("openai"))]:
    print(name, len(rows), rows[0])

On 28 October 2026 (NZ time) it printed 195 Airbnb roles, 89 Spotify roles and 801 OpenAI roles. The slug is the last part of the careers URL: boards.greenhouse.io/airbnb, jobs.lever.co/spotify and jobs.ashbyhq.com/openai.

If you only care about a handful of companies on one or two ATSs, this is enough, and you should just write it. The work grows once you need:

• more ATSs, each with its own field names, date formats and salary formats;
• department and workplace-type (remote, hybrid, onsite) fields that are named differently everywhere;
• per-company counts, and a diff against yesterday's run to spot new and removed roles.

Step 2: The hosted version (if you'd rather not maintain it)

We built ATS Jobs & Hiring Signals on Apify to do those next steps. It covers seven ATSs (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee and Personio), returns one consistent row format, adds a per-company summary and can flag roles that are new or removed since the last run.

The input is just the careers-page URLs:

{
  "companies": [
    "https://boards.greenhouse.io/airbnb",
    "https://jobs.lever.co/spotify",
    "https://jobs.ashbyhq.com/openai"
  ],
  "maxJobsPerCompany": 15
}

There are optional filters for title, location and department keywords, remote-only and "posted within N days". Filters are applied before anything is charged. From Python:

# pip install -U "apify-client>=3"
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("flyingkiwi/ats-jobs-hiring-signals").call(run_input={
    "companies": ["https://jobs.lever.co/spotify"],
    "titleKeywords": ["engineer"],
})
rows = list(client.dataset(run.default_dataset_id).iterate_items())
jobs = [r for r in rows if r["type"] == "job"]
summaries = [r for r in rows if r["type"] != "job"]

Our three-company run finished in about 3.4 seconds and returned 45 job rows (15 per company) plus three company summary rows.

Step 3: Read the job rows

Every job has the same fields, whichever ATS it came from. Here's a real Greenhouse row, trimmed:

{
  "type": "job",
  "ats": "greenhouse",
  "company": "Airbnb",
  "title": "Associate Legal Counsel, Japan",
  "department": "Legal",
  "location": "Tokyo, Japan",
  "workplaceType": "hybrid",
  "postedAt": "2026-04-13T08:49:05+00:00",
  "url": "https://careers.airbnb.com/positions/7789554?gh_jid=7789554",
  "salaryMin": null
}

Salary is filled in only when the company publishes it. In this run, all 15 OpenAI (Ashby) rows had salary ranges, for example "Research Engineer, $250K - $445K". None of the Airbnb or Spotify rows did.

Step 4: Use the company summaries as hiring signals

The summary row counts all of a company's open roles, not just the 15 we asked for. On 28 October 2026 (NZ time):

The summary also splits roles by workplace type. For Airbnb, 148 of 195 roles were tagged remote. For Spotify it was 48 hybrid, 28 remote and 13 onsite. Numbers like these help you prioritise accounts, or spot a team that's being built out, such as a new go-to-market push or a new office.

A few ways to use them:

• Sales prospecting: sort target accounts by postedLast30Days and look at which departments are growing.
• Recruiting: watch competitors' engineering openings by location.
• Investing and research: track headcount intent over time, which needs repeat runs (next step).

Step 5: Turn on monitor mode for alerts

Set "monitorMode": true and schedule the run, for example daily. Each company summary then lists newJobsSinceLastRun and removedJobsSinceLastRun. With "only new jobs" on, a run with nothing new costs only the start fee. You can send the output to Slack, a sheet or your CRM with an Apify integration or a webhook. If you're building this yourself, the equivalent is storing yesterday's job IDs per company and diffing them.

Honest limits

These apply to the approach in general, not just to our tool:

• Seven ATSs only. Workday, iCIMS, Taleo, BambooHR, LinkedIn and Indeed aren't supported. Many large enterprises use Workday, so check where your targets host their jobs first.
• Salary is often missing. It depends on what the company publishes. In our sample, only the Ashby company had it.
• Lever and Ashby don't publish a company display name. The name is built from the board slug, so openai becomes "OpenAI". A slug that doesn't match the real name (palantir for Palantir Technologies) needs the optional companyNames input.
• Location clean-up is light. "Remote - USA", "Remote, USA" and "Remote-USA" are merged into "Remote - United States" (14 Airbnb roles), and USA/UK are spelled out. Cities are left as written, so "San Francisco" and "San Francisco, CA" still count separately.
• "Posted" dates come from different fields on each ATS. Depending on the ATS, postedAt is a first-published, published or created date, and a re-posted role can look new. Treat "posted in last 30 days" as a rough signal.
• The first run is a baseline. New and removed roles only show up from the second monitored run.

What it costs

The hosted version costs $0.005 per run start plus $0.001 per job returned, and summary rows are free. The run above (45 jobs) costs $0.05. If you only want counts, set a low maxJobsPerCompany and read the summaries. The DIY script above costs nothing but your time.

We (flyingkiwi) built and maintain ATS Jobs & Hiring Signals. The numbers in this post come from real runs on 28 October 2026 (NZ time).

Tool: https://apify.com/flyingkiwi/ats-jobs-hiring-signals

This post walks through that preprocessing step on real files: what good output looks like, what to check before you trust it, and where conversion still goes wrong.

Option A: do it yourself with open-source tools

If you're happy running Python, start here. MarkItDown (Microsoft, MIT licence) handles Office files and HTML well:

# pip install 'markitdown[all]'
from markitdown import MarkItDown

md = MarkItDown()
print(md.convert("quarterly-report.docx").text_content)

For PDFs, look at Docling, Marker or MinerU as well. They do layout analysis, which matters for multi-column papers, equations and complex tables. Test them on your own documents: PDF quality varies a lot by tool and by file. For example, when we ran MarkItDown 0.1.9's built-in PDF converter on the "Attention Is All You Need" paper, it returned plain text with no headings or page markers, and some lines lost the spaces between words.

The rest of this post uses a hosted option, for when you'd rather call an API (or schedule a job) than maintain a conversion box. The checks in the "What to check" section apply whichever tool you pick.

Option B: a hosted converter

We built Document to Markdown on Apify. It uses MarkItDown for Office and HTML formats, plus pdfminer.six and pdfplumber for PDFs (page markers, heading detection from font size and weight, and tables drawn with lines). It also has optional OCR for scanned pages, and a quality report on every document.

Step 1: Give it a URL, a folder link or a file

Inputs can be:

• one url or a list of urls (PDF, DOCX, PPTX, XLSX/XLS, HTML, EPUB, CSV, Jupyter notebooks);
• Google Drive, Dropbox, OneDrive or SharePoint share links for single files;
• a whole shared folder from Google Drive, Dropbox or SharePoint / OneDrive for Business, with subfolders included;
• files uploaded directly in the Apify Console.

Here is the exact input from one of our test runs on the arXiv paper:

{
  "urls": ["https://arxiv.org/pdf/1706.03762"],
  "chunkSize": 1500,
  "chunkOverlap": 150
}

chunkSize is in characters. Leave it at 0 if you only want the full Markdown. PDF table detection (pdfTables) and saving a .md file per document (saveMarkdownFiles) are both on by default.

Step 2: Run it from code (or the Console)

# pip install -U "apify-client>=3"
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("flyingkiwi/document-to-markdown").call(run_input={
    "urls": ["https://arxiv.org/pdf/1706.03762"],
    "chunkSize": 1500,
    "chunkOverlap": 150,
})

for doc in client.dataset(run.default_dataset_id).iterate_items():
    print(doc["fileName"], doc["pageCount"], doc["wordCount"], len(doc["chunks"]))
    print(doc["quality"]["possibleContentLoss"], doc["markdownFileUrl"])

Step 3: Look at what comes back

The run took about 8 seconds. It returned one dataset item for the 15-page PDF, titled "Attention Is All You Need": 5,830 words, 38,426 characters and 44 chunks. 43 of the chunks carry the section heading they belong to, and every chunk carries the page it starts on:

{
  "index": 10,
  "page": 3,
  "heading": "3.2 Attention",
  "charCount": 542,
  "text": "### 3.2 Attention\n\nAn attention function can be described as mapping a query and a set of key-value pairs to an output,\nwhere the query, keys, values, and output are all vectors. The output is computed as a weighted sum\n3\n\nScaled Dot-Product Attention\nMulti-Head Attention\nFigure 2: (left) Scaled Dot-Product Attention. (right) ..."
}

Notice the stray 3 (a page number) and the figure labels in that chunk. Real PDFs are messy, and it's worth knowing what your chunks actually contain before you tune retrieval.

The Markdown marks each page with <!-- page N -->. PDFs don't store headings, so they're inferred from font size and bold text. On this paper that gave # Attention Is All You Need, ## Abstract, ## 1 Introduction, ### 3.1 Encoder and Decoder Stacks, ### 3.2 Attention and so on. Each document is also saved as a .md file in the run's key-value store, and markdownFileUrl links to it.

Next, we converted three non-PDF files in one run: a Python docs tutorial page (HTML), Microsoft's Financial Sample workbook (XLSX) and a test deck from python-pptx (PPTX).

• HTML: the page starts at the content (# 3. An Informal Introduction to Python), with the site navigation and footer stripped and the real headings kept (## 3.1. Using Python as a Calculator, ### 3.1.1. Numbers).
• XLSX: each sheet becomes a ## Sheet name heading followed by a Markdown table. The 700-row workbook came out as one 16-column table: first row as the header, empty rows and columns dropped, and no NaN or Unnamed cells.
• PPTX: the slide came out as:

<!-- Slide number: 1 -->
# Presentation Title Text
Subtitle Text

Step 4: Convert a whole shared folder

Paste a folder link instead of a file link. We used a public Dropbox folder of rule documents (3 PDFs and 3 DOCX files):

{ "url": "https://www.dropbox.com/scl/fo/<folder-id>?rlkey=<key>&dl=0", "chunkSize": 1500 }

All 6 files converted, one dataset row each, with relativePath showing where the file sits in the folder. Files the converter doesn't support (images, videos and so on) are skipped and listed in the run's status message and a FOLDER_SUMMARY record. The folder has to be shared as "anyone with the link". Private folders get a clear error and aren't charged. maxFiles (default 500) and includeSubfolders (default on) keep big folders under control.

What to check before you embed anything

Silent content loss is the most common RAG preprocessing bug. A scanned page or an image-only slide produces no text, and nothing tells you. Whatever tool you use, check every document for:

• pages with no text at all (scans, or text drawn as shapes);
• a word count that's far too low for the page count;
• tables that came out as jumbled lines.

Document to Markdown puts these checks in a quality object on every row. For a PDF with a scanned page and OCR off, it looks like this:

{
  "tablesFound": 0, "pagesWithText": 1, "textCoverage": 0.5,
  "emptyPages": [3], "scannedPages": [2], "ocrPages": [],
  "possibleContentLoss": true,
  "warnings": ["1 page(s) look scanned (an image, or text drawn as shapes) and have no text: 2. Turn on OCR to read them."]
}

A simple pipeline rule is: if possibleContentLoss is true, re-run that document with "ocr": true or send it for review rather than embedding it. OCR is opt-in. Only pages without a text layer are sent to a vision model (Google Gemini 2.5 Flash by default, through Apify's AI proxy). Text pages are never OCR'd.

Step 5: Feed the chunks into your vector store

From here it's ordinary RAG plumbing. Loop over chunks, embed text, and store page, heading and the source url (or relativePath for folder runs) as metadata. If you'd rather use your own splitter (LangChain, LlamaIndex or something else), set chunkSize to 0 and split the markdown field yourself.

Honest limits

• OCR is AI-based and slower. Our 3-page scanned test read every page correctly, but the run took a little over a minute. Poor scans and handwriting can be misread.
• PDF headings are a best guess. They're inferred from font size and bold text. That worked on the arXiv paper, but PDFs that style headings like body text get few or none, and a stray bold line can become a heading. DOCX, PPTX and HTML headings come straight from the file.
• PDF tables work best with ruling lines. Borderless tables stay as plain text. The arXiv paper's tables have no ruling lines, so they came through as text (tablesFound: 0).
• Multi-column PDFs are read in the order the text appears in the file, which is usually, but not always, reading order.
• HTML clean-up is rule-based. An unusual page template may keep some boilerplate, and JavaScript-rendered pages can come out empty.
• Spreadsheets are flattened to one table per sheet. Charts and merged-cell layout don't carry over.
• Images are dropped. You get text and tables only.
• Folder limits: personal OneDrive folders (1drv.ms, onedrive.live.com) don't work, because Microsoft requires a sign-in. Dropbox shares a folder as one ZIP, so the whole folder is downloaded, including any videos. Google Drive shortcuts are skipped.

What it costs

Pricing is per event:

• $0.003 per run start;
• $0.003 per converted document (up to 20 pages);
• $0.001 per extra page;
• $0.008 per scanned page read, only if you turn OCR on. The AI model usage for OCR is billed separately by Apify at cost, about $0.001 a page in our tests.

At those prices, the 15-page PDF above costs $0.006 in total. The 6-file Dropbox folder came to $0.035: the start fee, 6 documents and 14 extra pages. You can set a maximum cost per run. The Actor checks it before each document and stops cleanly instead of going over.

When to use something else

If you need equations, layout-aware reading order for complex multi-column papers, or heavy OCR workloads, run Docling, Marker or MinerU yourself and compare them on your documents. If what you need is "turn this pile of mixed office files, or this shared folder, into Markdown chunks from an API, with a check for missing pages", that's the gap we built this for.

We (flyingkiwi) built and maintain Document to Markdown. Every number in this post comes from real runs on 28 October 2026 (NZ time).

Tool: https://apify.com/flyingkiwi/document-to-markdown