How do you structure vacancy data for use in your own job board?

To structure vacancy data for use in your own job board, you need to define a consistent schema that captures all relevant fields, normalize that data across sources, and store it in a format your search engine and front end can query efficiently. The right structure makes the difference between a job board that delivers fast, relevant results and one that frustrates both candidates and employers. The questions below walk through every layer of that structure, from raw fields to live indexing.

What fields does vacancy data typically include?

Vacancy data typically includes a core set of fields: job title, job description, company name, location, employment type, salary range, publication date, expiration date, and an application URL or contact method. These fields are the minimum needed to display a meaningful listing and allow candidates to filter and apply.

Beyond the basics, well-structured job listing data often extends to additional fields that improve searchability and filtering:

• Job category or function (e.g. marketing, engineering, logistics)
• Required education level (e.g. MBO, HBO, university)
• Years of experience required
• Working hours (full-time, part-time, number of hours per week)
• Remote or on-site indicator
• Contract type (permanent, temporary, freelance)
• Industry or sector
• Skills or competencies
• Unique vacancy ID (critical for deduplication across sources)

The more complete and granular your field set, the better your filtering and recommendation logic will perform. Even if some fields are optional for display, capturing them in your data model from the start saves significant rework later.

What data format works best for job board integration?

JSON is the most practical data format for job board integration. It is lightweight, human-readable, natively supported by most modern APIs and databases, and easy to transform. For structured data job board implementations that also need to communicate with search engines like Google for Jobs, JSON-LD is the standard format used to embed schema.org markup.

XML remains common in vacancy feeds from larger job aggregators and ATS platforms, so your integration layer should be able to parse both. If you are pulling data from multiple sources, a unified internal JSON schema acts as your canonical format, with transformations applied at the point of ingestion.

For search indexing specifically, formats like JSON or NDJSON (newline-delimited JSON) work well when pushing large batches of vacancies into engines like Apache Solr or Elasticsearch, both of which accept JSON documents natively and allow you to define field types, analyzers, and boost values per field.

How do you collect vacancy data from multiple sources?

You collect vacancy data from multiple sources through a combination of vacancy feeds, APIs, and web crawling. Most larger job boards and ATS providers offer XML or JSON feeds that update on a schedule. Smaller employers or niche boards often require crawling their websites directly to extract listing data.

Using vacancy feeds and APIs

Vacancy feeds are the most reliable collection method when available. They are structured, regularly updated, and require minimal transformation. Many recruitment software platforms expose REST APIs that return job data in a predictable schema, making integration straightforward. Always check the feed’s update frequency and whether it signals deletions, not just additions, so your board stays accurate.

Web crawling for job data

When no feed or API exists, web crawling is the alternative. A crawler visits the employer’s careers page or job listing pages, extracts the relevant HTML elements, and maps them to your internal data schema. This requires more maintenance because page structures change, but it gives you access to sources that do not publish structured feeds. Tools like Apache Nutch are well suited to large-scale crawling operations, and pairing a crawler with a robust parser allows you to handle dozens of source formats simultaneously.

How should vacancy data be normalized across different sources?

Vacancy data should be normalized by mapping every source’s fields to a single canonical schema, standardizing values for categorical fields, and deduplicating listings that appear across multiple sources. Without normalization, your job board will display inconsistent results and filtering will break because the same employment type might be labeled “fulltime,” “full-time,” or “40 hours” depending on the source.

Normalization steps to apply at ingestion include:

1. Field mapping: Map source-specific field names to your internal schema (e.g. “vacatureomschrijving” maps to “description”).
2. Value standardization: Create controlled vocabularies for fields like employment type, education level, and contract type, then map incoming values to those controlled terms.
3. Location normalization: Parse free-text location strings into structured components (city, region, country) and optionally enrich with coordinates for proximity search.
4. Date normalization: Convert all date formats to ISO 8601 (YYYY-MM-DD) for consistent sorting and expiry logic.
5. Deduplication: Use a combination of the source URL, vacancy ID, job title, company name, and location to detect and merge duplicates before indexing.

A well-designed normalization pipeline is one of the highest-value investments in job board development. It directly affects the quality of search results and the reliability of your filters.

What’s the best way to index vacancies for fast search results?

The best way to index vacancies for fast search results is to use a dedicated search engine like Apache Solr or Elasticsearch, define field types carefully, and apply relevance tuning so that job title matches rank higher than description matches. A relational database alone is not designed for full-text search at scale and will slow down as your vacancy volume grows.

Key indexing decisions that affect search performance include:

• Field types: Use text fields with language-appropriate analyzers for job title and description. Use keyword fields for category, location, and employment type so filters are exact matches.
• Boosting: Assign higher relevance weight to matches in the job title field compared to the description field.
• Faceting: Index categorical fields as facet-enabled so your front end can display filter counts dynamically without separate queries.
• Geo-search: If location is a primary filter, store coordinates as a geo-point field and use distance-based queries for “vacancies near me” functionality.
• Expiry handling: Filter out expired vacancies at query time using a date range filter rather than deleting them immediately, which preserves your data for analytics.

How do you keep vacancy data up to date automatically?

You keep vacancy data up to date automatically by scheduling regular recrawls or feed fetches, implementing delta updates that only process changed records, and setting expiry logic that removes or hides listings past their end date. Manual updates do not scale once you are managing thousands of listings from dozens of sources.

A practical update strategy involves three layers. First, set fetch intervals based on each source’s update frequency: high-volume sources may need hourly checks, while smaller employer pages may only need daily crawls. Second, use checksums or last-modified timestamps to detect changes so you only reprocess and reindex records that have actually changed, which keeps your pipeline efficient. Third, build explicit deletion handling: when a vacancy disappears from its source, your system should mark it as inactive and remove it from search results within a predictable time window.

Monitoring is equally important. Automated alerts for feed failures, crawl errors, or sudden drops in vacancy volume help you catch data quality issues before they affect users. A job board that shows outdated or already filled positions loses trust quickly.

How Openindex helps with vacancy data and job board development

We at Openindex specialize in exactly the kind of data infrastructure that powers a reliable, scalable job board. Whether you need to collect vacancy data from dozens of sources, normalize it into a consistent schema, or build a fast search layer on top of it, we have the expertise and tooling to deliver it. Our work covers:

• Custom crawling and vacancy feed integration across multiple sources and formats
• Data normalization pipelines that standardize fields, values, and locations at ingestion
• Search engine setup and tuning using Apache Solr or Elasticsearch for fast, relevant job search
• Crawling as a Service, where we handle the entire data collection process and deliver clean, structured vacancy data directly into your system
• API development to connect your job board front end to a well-indexed, always-current data backend

If you are building or improving a job board and want a data infrastructure that scales without the maintenance burden, get in touch with us and we will help you design the right solution from the ground up.