Twin Campaigns (in situ)

Total traffic is intact. Campaign identity is not.

Across the 12-hour post-truncate window, linkrunner-backend and hazelnut agree on 98.9% of attributed installs in aggregate — the pipeline is healthy, Kafka is flowing, the consumer is not lagging. But at the campaign-id level the two services diverge violently: on Project 89 they share only 2 of 21 Meta campaigns, while still reporting nearly identical install totals. The same install is being attributed to two parallel Campaign rows that differ only in which Meta identifier they were keyed on.

One sentence: hazelnut's Meta-ID normalizer prefers the modern campaign_id field when adset_id is also present (§05), while linkrunner-backend unconditionally uses campaign_group_id. Those are different numeric IDs on a non-trivial share of payloads. The findExistingCampaign chain in hazelnut (file cites below) then misses every existing legacy row and falls through to INSERT, minting a parallel Campaign. Legacy keeps picking its row; hazelnut keeps picking its own. Neither side ever sees the twin.

Pathologist · the finding is structural, not flaky. Bifurcation reproduces deterministically on every sampled install_instance_id (5/5). Traces on 10.1.0.33 show attribution-consumer p99 at 228 ms; the retry consumer p99 at ~40 s is within design. This is not a lag story.

Two slides. Same tissue. Different stain.

The Meta IDs differ by six in the trailing digits. …960758 is the Meta campaign_group (top level); …960752 is the Meta campaign / ad-set (middle level). Same campaign, two identifiers. The services chose different levels of the Meta hierarchy to key on — see §05.

Five installs, same night. Each one split, the same way.

Sampled from the post-truncate window. For each install_instance_id the query ran once against Postgres ("Install") and once against ClickHouse (hazelnut.installs_denormalized FINAL). Column campaign_id · PG is what the legacy service attached; campaign_id · CH is what hazelnut attached. Every specimen in the sample bifurcates 25703326 → 26096423. Deterministic, not racy.

A campaign-level dashboard does GROUP BY campaign_id. When half the traffic lands on 25703326 and half on 26096423, each dashboard shows what its underlying store saw — and the numbers will never match, even though every single install is accounted for.

The mutation sits at one branch, in one function.

Hazelnut normalizes the decrypted Meta payload in one place before any Campaign lookup. The branch below decides which Meta identifier becomes campaignID. When Meta sends both adset_id and campaign_id — the modern field set — hazelnut picks d.CampaignID. Legacy linkrunner-backend, when it can read the payload at all, picks campaignGroupId. Those are different Meta entities.

// Meta legacy mapping (matching TS normalizeMetaIds):
//   adgroup_id        → ad_id (creative)
//   campaign_id       → adset_id
//   campaign_group_id → campaign_id
// When modern fields (adset_id + campaign_id) are both present, prefer those directly.
func NormalizeMetaIDs(d *MetaDecryptedData) NormalizedMetaIDs {
    ...
    campaignID := ""
    if d.AdSetID != "" && d.CampaignID != "" {
        // Modern: both present, campaign_id is the real campaign.
        campaignID = d.CampaignID          // ← picks …960752
    } else {
        campaignID = d.CampaignGroupID     //   would pick …960758
    }
    ...
}

That campaignID flows through the Meta strategy to the repository's find-or-create. The lookup chain is literal — four SELECTs, LIMIT 1, no ORDER BY, no cross-schema fallback. Each miss lands on the next. A total miss falls through to INSERT, which mints a fresh Campaign row.

// findExistingCampaign tries each lookup strategy in priority order:
// google_campaign_id → meta_campaign_id → display_id → natural key.
if campaign.MetaCampaignID != "" {
    lookups = append(lookups, func() (*Campaign, error) {
        return r.findCampaignByMetaCampaignID(ctx, campaign.ProjectID, campaign.MetaCampaignID)
        // WHERE project_id=$1 AND meta_campaign_id=$2 AND NOT deleted  — misses, value is …752
    })
}
if campaign.AdNetworkID > 0 || campaign.AdNetworkCampaignID != "" {
    lookups = append(lookups, func() (*Campaign, error) {
        return r.findCampaignByNaturalKey(ctx, campaign.ProjectID, campaign.AdNetworkID, campaign.AdNetworkCampaignID)
        // WHERE project_id=$1 AND ad_network_id=$2 AND ad_network_campaign_id=$3  — first run = miss.
    })
}
// no ORDER BY, no meta=TRUE bias, no ad_network_id IS NOT NULL preference.
// two equal candidates? implementation-defined heap order.

The legacy service's mirror of this code is shorter and less forgiving. It will only find a row keyed on meta_campaign_id, and will never look at ad_network_campaign_id. Which means: once hazelnut has created its new-schema row, legacy is structurally incapable of noticing it.

let campaign = await prismaClient.campaign.findFirst({
    where: {
        meta_campaign_id: metaAdsData.campaignGroupId.toString(),  // …960758 — the legacy top-level ID
        project_id: projectId,
        deleted: false,
    },
    // no orderBy. Prisma returns the row Postgres hands back first —
    // with two rows matching, typically insertion order, so the older row wins.
});

And one more piece — the empirical one: of 1,521 active ad_network_id=3 rows in Postgres right now, 1,392 (91.5%) have meta=FALSE and meta_campaign_id=NULL. Whatever code path minted those specific rows never wrote the legacy columns. Those rows are, by construction, invisible to every legacy WHERE meta_campaign_id=? query in linkrunner-backend. That's what makes the bifurcation permanent rather than self-healing.

Systemic, not incidental.

Hazelnut has been minting new-schema Meta rows continuously since mid-March, not just as a one-time migration blast. Two conspicuous burst days (Apr 8 · 398, Apr 9 · 423) likely coincide with deploys or catch-up traffic — worth an independent look, but not the cause of the divergence itself.

Ruled out, so we don't chase ghosts.

Three treatments, ranked. One is cheap and stops the bleed.

Recommendation: ship Option 01 as a hotfix this week to stop new twins. File a follow-up for Option 02 once the campaign_group_id coverage question is answered against a week of live Meta payloads. Treat Option 03 as the planned retirement — worth scoping, not worth rushing.