If you syndicate the same posts to dev.to, Hashnode, Zenn or Qiita, sooner or later you wonder whether search engines read the copies as duplicate content. I got far enough into that worry to write a remediation plan: unpublish the live copies, set canonical everywhere, start over.
Then I counted the actual overlap. Across 11 articles and 22 files, three prose lines matched between origin and copy — and two of them were --- rules. The plan was unnecessary. What replaced it was adding a canonical URL on the platforms that accept one.
Three steps: strip frontmatter normalize whitespace, drop blank lines tag each line by whether it sits inside a code fence
Step 3 is the one that decides the answer. Without it you get a single percentage and no way to interpret it.
/.test(line)) { inFence = !inFence; out.push([line, "code"]); continue; } if (!line) continue; if (inFence) { out.push([line, "code"]); continue; } if (/^#{1,6}\s/.test(line)) { out.push([line, "heading"]); continue; } if (/^\|/.test(line)) { out.push([line, "table"]); continue; } out.push([line, "prose"]); } return out; }
// Put the origin's lines in a set, walk the syndicated copy const origin = new Set(classify(bodyOf(originFile)).map(([l]) => l)); const ext = classify(bodyOf(copyFile)); const matched = ext.filter(([l]) => origin.has(l));
const count = (k) => matched.filter(([, t]) => t === k).length; console.log({ ext: ext.length, matched: matched.length, code: count("code"), table: count("table"), heading: count("heading"), prose: count("prose"), }); js body: JSON.stringify({ article: { title, bodymarkdown: body, published: false, tags, canonicalurl: "https://example.com/blog//", }, }), js const input = publish ? { title, contentMarkdown, publicationId, tags, originalArticleURL: canonical } : { title, contentMarkdown, publicationId, tags }; // no field for it on a draft ``
dev.to fixes the value at post time too, so already-published articles need PUT /articles/{id}` or a manual edit in the editor.
Without a canonical field, the way back from unpublishing is posting again: new URL, views reset, inbound links broken. It looks reversible and is not.
Which is the practical reason to measure first. Thirty lines of script, one run, and the answer may well delete the entire task list behind it.
The full story — how I ended up conflating an AdSense decision with search-side duplicate handling, the per-channel traffic numbers behind the reversal, and the per-article overlap breakdown — is on Aulvem → Aulvem | Is cross-posting duplicate content? I measured it before deleting anything