I build devdigest, a daily tech digest. It pulls from RSS feeds, has a model read each excerpt to categorise and score it, writes an original summary, and emails subscribers a list of titles, summaries and links back to the original articles. Full article text is never reproduced or stored beyond a short-lived per-run cache. To read anything, you click through to the publisher.
Before charging money for that, I wanted to know whether any of my sources explicitly banned it. So I read the actual Terms of Service for all of them. Not the privacy policy, not a summary, the real current terms, quoting the clause or explicitly recording "no ToS found."
I expected the answer to track company size. Big corporate sites run their own RSS feeds, press APIs and developer-relations teams, so I assumed they would be relaxed. Small scrappy publications felt like the ones who would care.
The split The most restrictive terms came from VC-data companies and large corporate infrastructure blogs. Verbatim:
Crunchbase News bans using content "to train models (including generative artificial intelligence technologies)," and separately bans anything that "'Crawls,' 'scrapes,' or 'spiders' any page, data, or portion of" the content.
HPCwire/AIwire bans any "robot, spider, or other automatic device" without prior written permission, plus "any form of data extraction or data mining, or other commercial exploitation of any kind." The most specifically targeted restriction I found anywhere.
TechRadar (Future plc): "You shall not conduct, facilitate, authorise or permit any text or data mining or web scraping... for any purpose, including the development, training, fine-tuning or validation of AI systems or models."
Cloudflare's blog bars automated bots from "scrape, data mine, copy, or use the materials or content on this website for developing, training, fine-tuning, or otherwise contributing to or improving a machine learning model or artificial intelligence (AI) system."
Sifted (§7.6), the European startup publication: "You must not conduct, facilitate, authorise or permit any text or data mining or web scraping in relation to our Platform... for any purpose, including the development, training, fine-tuning or validation of AI systems or models."
InfoQ: "We permit the posting of a summary and then a link back to the InfoQ landing page." That is my exact pipeline, described approvingly, in their terms.
MIT News: its Terms of Use state it "offers RSS feeds for syndication purposes."
GitHub: "these Terms do not restrict lawful access to or use of the contents of public repositories by third parties."
arXiv: the API Terms of Use explicitly names RSS-based discovery and notification tools as a permitted use case, in those words.
TechCrunch maintains dedicated RSS terms, separate from its general ToS, explicitly permitting display of feed content with attribution and a link to the full article.
Engineering blogs, university news offices and syndication-native sites were mostly fine with it. The well-lawyered commercial sites were not.
My best guess at why: the restrictive ones have their own data-licensing business to protect, or expect to. A clause banning AI training is an asset if you intend to sell that access later. The permissive ones live on distribution. Being read and linked to is the product, so the feed is a front door rather than a leak.
The part that actually matters: three ways this goes wrong The size heuristic being backwards was interesting. What was useful was discovering that the obvious check is wrong in three different directions. The ToS is clean and robots.txt bans you.
I originally cleared Towards Data Science. It is Medium-hosted, and Medium's Terms of Service contain no RSS, scraping, commercial-use or AI-training restriction. Clean read.