Why not use one large model for everything?

Cost and reliability. Cleaning and de-duplication are high-volume, low-judgement tasks that a small fast model handles well; titles, summaries and categories benefit from a stronger model on a much smaller number of items. Splitting them cuts cost and makes each step testable.

Does the AI decide what ranks first?

No. Rank is computed from interaction counts with a fixed formula. The models only write titles and summaries and assign categories. If a summary is wrong, the item's position is unaffected.

How do you detect a repost across platforms?

By content, not by URL. Each item's caption is normalised (case, whitespace, links, hashtags) and hashed together with the author. Two items with the same hash are one item, whichever platform each came from.

What happens to the raw data?

Every scrape is stored verbatim before any agent touches it, so summaries can be traced back to the source. Raw data is deleted after 48 hours along with the feed items it produced.

Two agents, one clean feed: how Prowler de-duplicates reposts across five platforms

A brand posts a launch video to TikTok, a cut of it to Instagram Reels, a still to Facebook and a link to X, all within twenty minutes. A naive monitoring tool shows you four items. A feed you can trust shows you one, with the platform badges attached and the engagement summed. Getting from four to one, reliably, on hundreds of items an hour, is most of the engineering in Prowler's pipeline. This is how it works.

The pipeline in one picture

Sources (main URL → up to 4 backups)
   → Scrapers (per-platform)
   → Raw store (verbatim)
   → Agent 1: clean · normalise · de-duplicate
   → Agent 2: title · summary · category
   → Rank (deterministic) → Feed
   → Trending (time-decayed, platform-normalised)

Two decisions shape everything: the work is split between two agents with different jobs and different models, and the language models never touch the ordering.

Why two agents

One prompt that says "here are 400 posts, clean them, remove duplicates, write summaries and rank them" is tempting and wrong. It is expensive, because every item goes through the strongest model; it is fragile, because one bad output corrupts several fields at once; and it is untestable, because you cannot say which sub-task failed.

Splitting it:

Agent	Job	Model class	Volume
Agent 1	Strip junk, normalise fields across platforms, collapse duplicates	Small and fast (Claude Haiku)	Every scraped item
Agent 2	Write a title and one-line summary, assign one of 12 categories	Stronger (Claude Sonnet)	Only the survivors

Agent 1 sees everything and makes cheap, mechanical decisions. Agent 2 sees a fraction of the volume and makes the judgement calls people actually read. If Agent 2 is unavailable, a keyword fallback still assigns categories and the feed still ships; the pipeline degrades, it does not stop.

Step 1: normalise five platforms into one shape

Each platform returns a different object. Instagram has likes and comments; TikTok has plays, likes, shares and saves; X has reposts and quotes; a website has none of these. Before anything else, every item is mapped onto one record: author, text, timestamp, media, link, and a single interactions number that sums whatever engagement the platform exposes.

The mapping is defensive on purpose. Platforms change their markup and their field names without notice, so each field has a fallback path and a missing field becomes zero rather than a crash.

Step 2: de-duplicate by content, not by link

Reposts do not share URLs, so URL matching is useless. Instead:

Normalise the caption: lower-case, collapse whitespace, strip links, strip trailing hashtags, drop emoji variants.
Hash the normalised caption together with the author identity.
Items with the same hash are the same item. Keep the first, attach the other platforms' badges, sum the interactions.

This catches the launch-video case above, and the subtler one where a caption is reposted with a different link shortener. It deliberately does not catch paraphrases; two genuinely different captions about the same launch are two items, which is what an editor would want.

Re-scrapes of the same source are handled separately by a unique index on the raw record, so scanning a source hourly never produces duplicate feed items.

Step 3: enrich, briefly

Agent 2 writes a title of a few words and a summary of one sentence for each surviving item, and picks exactly one category from a fixed list: Business, Funding, Geopolitics, Technology, AI, Markets & Finance, Sports, Entertainment, Health, Science, Culture, Other.

The fixed list is a design choice. Free-text tags drift ("AI", "artificial intelligence", "GenAI") and make filtering useless within a week. Twelve categories are enough to filter a feed and few enough to stay consistent.

Step 4: rank without the model

Rank is arithmetic. Inside each scan, items get a 0–100 score from their interactions relative to the batch; across the feed, Trending re-scores every item over a seven-day window as:

trend = interactions ÷ platform_baseline × exp(−ln2 × age / 24h)

The platform baseline is what makes a TikTok view and an Instagram like comparable, and the exponential term halves an item's weight every 24 hours. None of this involves a language model, which means the order of the feed cannot be hallucinated, cannot drift between runs, and can be explained to a user in one sentence. The full reasoning is in why a TikTok like isn't an X like.

Step 5: keep the receipts, then forget

Every scrape is stored verbatim before Agent 1 runs, so any summary can be traced back to the original post. Then both the raw records and the feed items are deleted after 48 hours. Saved items are the exception: they are copied, image included, into permanent storage. The reasoning behind that asymmetry is its own post, why your feed should forget.

In Prowler: each scan produces up to twelve feed items after de-duplication, and only a scan that returned content is charged (0.4 credits for X and websites, 0.6 for Instagram and Facebook, 0.8 for TikTok). Hide and Junk buttons on each item feed back into what the pipeline shows you next.

What we got wrong first

The first version ranked with the model. It was fluent and inconsistent: the same batch produced a different order on each run, and users rightly asked why. Moving ranking to a formula and confining the model to text was the single change that made the feed trustworthy. The general lesson: let models write, and let arithmetic decide.