Case study

A daily pipeline for every open job at 1,500 US tech companies

Every open job at 1,500 US tech companies, collected each morning from their public job boards and delivered as one clean table. Built and run in public by the founder of SteadyData to show the same method on a much larger set of sites.

  • Portfolio project · public data · not client work
  • By Siddharth Kumar, Founder of SteadyData
  • Published

What problem does it solve?

Open tech jobs are spread across 1,500 separate company job boards. Analysts need them as one table they can filter and chart.

The pipeline reads the public job board of every tracked company: 779 on Greenhouse, 669 on Ashby and 52 on Lever.

How does it work?

  • CollectOne request per company per day from the official public job-board feeds, with a short pause between requests and no logins.
  • CheckTitles become role types, locations become city and state, and salaries become yearly US dollars. Every run must pass the automatic data checks before anything is published.
  • StoreEach day's raw data is kept, so the history of when a job opened, closed or changed its salary can always be rebuilt.
  • DeliverThe clean table, a live dashboard and a monthly download, refreshed every day.

What are the real numbers?

The run of 20 November 2026 listed 64,204 open jobs, read 1,493 of 1,500 job boards and passed all 91 automatic data checks.

  • 64,204open jobs
  • 1,493 / 1,500company job boards read
  • 30 / 30second-source spot checks matched
  • 91automatic data checks passed
  • 97doubtful salary figures held back, not guessed
  • 13malformed job ads set aside
  • 1,531requests in the run
  • 9.2 minfor the full run

From 4 to 20 November 2026: 17 days running, 17 days published, 0 missed and 0 held. The first day needed a failed and a held attempt before it published, and the public run log keeps every attempt.

What broke and how was it fixed?

Real problems from building and running it. Each one is written up in the public incident log.

  1. 4 Nov 2026 · first real run

    An odd salary format stopped the first run

    One job ad wrote its salary as "140.000.00". The salary reader did not expect that format, and the whole first run stopped after 58 seconds.

    Fix: The number reader now handles 140.000.00, 140.000,00 and 120,000.00, and salary reading can no longer stop a run: one odd ad only skips itself.

  2. 4 Nov 2026

    Locations made only of dashes stopped the data model

    A few job ads gave their location as just a dash. The step that reads locations did not expect that, so the run was held and nothing wrong was published.

    Fix: Empty locations are handled, a test covers them, and the same day's data was re-processed and published.

  3. 4 Nov 2026

    A saved copy of the database was missing a table

    The nightly saved copy was taken a moment too early, so it could miss the newest changes. The next run would have built on it and lost history without noticing.

    Fix: Every copy is completed and checked before it is saved and again after it is restored. A bad copy is never used; the data is rebuilt from the raw daily files instead.

Can SteadyData run this for my competitors?

Yes. The same method runs as a SteadyData monitoring plan for the competitor stores you choose, daily or weekly. Send 1-3 competitor links → get up to 1,000 rows + a sample change report within 48 hours.

What else do readers ask?

Is this client work?

No. It is a portfolio project on public data, built and run in public by the founder of SteadyData. No client or client data is involved.

Why is a jobs pipeline on an e-commerce site?

It shows the same method, polite collection, checks on every row and doubtful values held back, working across 1,500 sites. SteadyData's service is e-commerce price and stock monitoring.

Why are some salaries empty?

Either the company did not publish one, or the figure looked wrong once converted to a yearly amount and was held back instead of guessed: 97 in the run of 20 November 2026.