Deduplication of Job Postings (Temporal)

Job postings are deduplicated at a point in time using the normal deduplication criteria–textual similarity, identical locations, identical companies, etc. Job postings are also deduplicated over time to account for new postings appearing for the same vacancy after the other postings for the vacancy were taken down. A vacancy here refers to a job opening with any number of duplicate postings associated with it.

A vacancy is considered expired or closed when there are 0 active postings for it among all of its duplicate postings. For instance, a vacancy with three total postings is considered expired when all three associated postings are no longer active. However, there are cases in which a vacancy can expire and another posting will appear for it after its expiration. In cases like these, if the new posting appears within six weeks of the vacancy’s expiration, we revive the vacancy and count the new posting as another duplicate.

Here’s an example:

Day 1: A unique posting for a Nurse job in Winnipeg appears.
Day 3: Two duplicates of the posting appear on different job boards. The vacancy now has one unique posting and three total postings.
Day 6: The original posting and one of the newer postings are taken down. The vacancy still has one unique posting and three total postings.
Day 19: The last posting is taken down. The vacancy as a whole is now considered closed since all of its associated postings have been taken down.
Day 60: Another duplicate posting for the Winnipeg Nurse job opening appears. Today is the 41st day after the last posting expired (Day 19). 41 days is just shy of 6 weeks, so this posting is considered a true duplicate, and the vacancy is revived. The vacancy now has one unique posting and four total postings.
Day 65: The new posting is taken down and the vacancy is expired again, since there are 0 active postings associated with it.
Day 107: This is the 6-week cutoff date for the Winnipeg Nurse vacancy. No more postings have appeared for this vacancy, so the vacancy is retired permanently. Going forward, any postings that could have been considered duplicates of this vacancy will instead be counted to a new vacancy.
Day 250: Another “duplicate” posting for the Winnipeg Nurse job opening appears. However, since this posting appears after the 6-week cutoff, it is considered the first posting for a brand new vacancy and does not revive the prior vacancy.

See also Deduplication, Total Job Postings, Unique Job Postings