Hey, I have a question about using Apify for long-term lead generation. I'm planning to use Apify to fetch around 1,000 leads per day based on specific locations and job positions. I expect to run this process continuously for about 4–5 months to build and enrich my database. My main concern is duplicate profiles across different runs. For example, let's say I've already collected 10,000 profiles. The next day, I request another 1,000 profiles matching the same criteria. How does Apify ensure that those 1,000 results haven't already been returned in previous runs? If 900 of those profiles are duplicates and only 100 are new, I would effectively be paying for 1,000 results while only adding 100 new leads to my database. Is there a built-in mechanism to prevent previously retrieved profiles from being fetched again across multiple runs? Ideally, I would like to maintain a list of already retrieved profile IDs/URLs and have Apify exclude them before scraping or charging for the results. Does Apify support something like this, or would the deduplication need to be handled on my side?

message has been deleted

Check whether the specific lead-gen Actor you're using has a "skip already-scraped items" or "existing IDs" input field — many do, since this exact problem is common. If it doesn't, maintain your own master list (a named Key-Value Store or an external DB) of seen profile URLs/IDs, and pass that as an exclude-list input if the Actor supports it. If the Actor has no way to accept an exclude-list, dedup will only be possible post-scrape — meaning you should budget for the fact that you'll be charged for re-fetched duplicates, and plan your query rotation (varying location/title combos more granularly, or narrowing as your database saturates a niche) to naturally reduce overlap over the 4–5 months rather than relying on Apify to filter it for you. Given you're running this for months at real cost, it might be worth asking Apify support directly whether the specific Actor you're eyeing supports an exclude-list input — that's the one thing that actually saves you money, versus dedup tooling that only saves you database mess.