DeepSearchJobs

DeepSearchJobs is a core module of the Play2Path ecosystem.
Its purpose is to find and extract unique job opportunities from sources that are often overlooked and not available on major platforms such as LinkedIn, Indeed, Glassdoor, Monster etc.

Unlike traditional job boards, DeepSearchJobs focuses on hidden, high-value, niche, or hard-to-access opportunities, offering users unique insights into the job market.

This advanced job-discovery engine by combining:

• Smart scraping
• Asynchronous job ingestion
• Worker-based processing
• Data enrichment and cleanup

We are continuously improving the module, and you are welcome to contribute!
👉 See the Contributing Guide to get started.

If you find this project useful or appreciate the work, please consider leaving a ⭐ star on the repository, it helps a lot!

---

❓ Why DeepSearchJobs?

From personal experience and from watching friends, relatives, and colleagues struggle, it has become clear that finding a job is becoming harder every year. Traditional job platforms are saturated, and the competition is extremely high. Everyone applies to the same listings…

To succeed in such a crowded environment, it’s essential to increase the variance — to look where others aren’t looking.

Across the web, thousands of unique job opportunities exist that are not pushed on the main platforms, either because:

• They are published on niche websites
• They appear on small company pages
• They are hidden in specialized communities
• They are poorly indexed or never aggregated

DeepSearchJobs exists to reveal these hidden opportunities.

It focuses specifically on discovering low-competition, high-value job postings that most job seekers never see, giving DeepSearchJobs users a genuine edge in their job search.

On top of that, we also fetch direct contacts whenever available — hiring managers, HR emails, founders, or team leads. Having the right contact dramatically increases your chances of being seen, considered, and fast-tracked.

Together, this creates a powerful advantage: 👉 More unique job opportunities 👉 Less competition 👉 Direct ways to reach the right people

---

▶️ How to Use DeepSearchJobs

You can use DeepSearchJobs in two ways:

1. Use it Online

The simplest option is to use the full platform directly:

👉 Visit : https://www.deepsearchjobs.com

There, DeepSearchJobs runs automatically and retrieves unique job opportunities for you.

---

2. Run DeepSearchJobs Locally with Your Own Data

If you want to run the module yourself, you can provide your own data sources using a simple .xlsx file or using the data provided (feel free to contribute to this list).

A reference file is provided here:

• test.xlsx

You can create your own .xlsx file by following the same structure.

Minimum required information

• Company Name (mandatory)

Optional (recommended for faster & more accurate scraping)

If you have additional data, DeepSearchJobs will process companies much more quickly:

• Company website
• Internal career page
• External career page

The more info you provide, the better the scraping performance.

Configure the environment

Create the local environment files from the provided templates:

cp .env.example .env
cp backend/db/.env.example backend/db/.env
cp worker/.env.example worker/.env
cp frontend/.env.example frontend/.env.production

Review the generated files and replace every change-me value. The root .env is the source of truth for production infrastructure credentials; make the matching database values in the other environment files consistent when running development tools directly on the host. Keep all .env files private and do not commit them.

The worker needs an LLM for structured job extraction. Choose one of the following options.

Option A: External AI provider

Configure the primary OpenAI-compatible provider in worker/.env:

LLM_API_KEY=your-provider-api-key
LLM_MODEL=your-provider-model
LLM_BASE_URL=https://your-provider.example/v1

Any provider exposing an OpenAI-compatible Chat Completions API can be used.

Option B: Local vLLM (no external AI provider)

The optional vLLM service requires an NVIDIA GPU available to Docker. It serves Qwen/Qwen3.5-4B by default and keeps the downloaded model in a Docker volume.

Configure worker/.env as follows:

LLM_API_KEY=local-vllm
LLM_MODEL=qwen-local
LLM_BASE_URL=http://vllm:8000/v1

For a smaller model, replace Qwen/Qwen3.5-4B with Qwen/Qwen3.5-2B in docker-compose.yml.

You may also configure an optional fallback provider. It is used only after all attempts with the primary provider fail:

LLM_BACKUP_API_KEY=your-backup-api-key
LLM_BACKUP_MODEL=your-backup-model
LLM_BACKUP_BASE_URL=https://your-backup-provider.example/v1

Leave the three backup values empty to disable failover.

▶️ Run the Module

With an external AI provider, run:

make prod

To run the complete stack with the optional local vLLM server, run instead:

make prod-vllm

When the process finishes, open:

👉 http://localhost:3000

You're all set! DeepSearchJobs will be running locally!

---

🎥 Video Tutorial

A full video tutorial is available here:

👉 Watch the tutorial

---

Potential Improvements

DeepSearchJobs is an evolving module, and there are several areas where contributors can help push it even further. Here are some ideas currently on the roadmap:

• Faster "job still open" check Speed up availability detection by leveraging the hash_job_description_page column in the all_jobs table — use Hamming distance comparisons on page content hashes to efficiently determine whether a job listing is still active.

• Improve data reliability Enhance validation, detection of broken sources, deduplication logic, and consistency across ingested data.

• Support multiple contexts running inside a single worker Better parallelism, smarter resource usage, and more scalable ingestion pipelines.

• Generate resumes tailored to each job offer Already available on DeepSearchJobs — the goal is to integrate this capability directly into the local module.

• Improve scraping speed Through optimized concurrency, caching, batching requests, or smarter prioritization.

• Improve stealthiness of the scraping Techniques to reduce blocking (already implemented: header rotation, proxy rotation, user behavior simulation)

---

📄 License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

This means:

• You are free to use, modify, and redistribute the software.
• If you modify the code and make it available to users over a network, you must also make your modified source code available under the same license.
• Any derivative work must also remain open-source under AGPL-3.0.
• The original copyright and license notices must be preserved.

👉 For full legal details, see the official license text: GNU AGPL v3.0

Bundled third-party components remain under their respective licenses. See THIRD_PARTY_NOTICES.md for details.