Scraping the Internet for AI Training Data: Fair Dealing, Implied Consent, or Mass Copyright Violation?

Ashish Pratap Singh (IPR RESEARCH SERIES 2026)

Introduction

Every time someone asks a large language model to draft a legal brief or summarise a news article, the intelligence behind that response was built on a foundation of human-created text scraped and processed at a scale copyright law has never before confronted. OpenAI's GPT-4, Google's Gemini, and Anthropic's Claude were each trained on datasets comprising hundreds of billions of words, the overwhelming majority of which were harvested from the publicly accessible internet without the authors' knowledge or consent.

The central legal question this raises is not a narrow technical one. It strikes at the architecture of intellectual property law itself: does mass automated copying of copyrighted works for the purpose of training a commercial AI system constitute permissible use, or does it amount to what critics have called the largest act of copyright infringement in history?

The Mechanics of Web Scraping and Why It Matters Legally

Web scraping is the automated extraction of data from websites using software bots or crawlers. AI companies have relied heavily on datasets such as Common Crawl, an archive of over 250 billion web pages, alongside proprietary corpora like WebText and licensed publisher data. The legal problem begins the moment a crawler stores a copy of a webpage's HTML: that act is, prima facie, a reproduction of copyrighted material under virtually every copyright regime, including Section 14 of India's Copyright Act, 1957, Section 106 of the US Copyright Act of 1976, and the Berne Convention's universal obligation to protect original literary and artistic works against unauthorised reproduction. The scale is difficult to overstate: a 2024 study examining the CommonPool dataset, a 12.8-billion-text-image-pair corpus, found that 60% of samples from the top 50 domains came from websites whose Terms of Service explicitly prohibit scraping, and that an estimated 122 million samples bear copyright notices.[1] Since protection attaches automatically upon creation, AI companies that scraped this content without authorisation were simultaneously infringing millions of copyrights.

The Fair Use Defence: The American Battleground

The dominant legal theory advanced by AI companies is fair use, codified under Section 107 of the US Copyright Act. Fair use decisions turn on how a court views these factors. First up, why did someone use the material matters, was it transformed or just copied? The type of work involved also plays a role; some creations receive stronger protection. How much was taken enters the picture, not just quantity, but the importance of what was lifted. Last thing considered? Whether the original stands to lose value as a result.

Some artificial intelligence firms claim their learning methods qualify as transformative, citing the Campbell v. Acuff-Rose Music Inc.[2] case. Because the systems learn trends in the data rather than copying visible words. These businesses refer to another decision, Authors Guild versus Google - where scanning entire libraries for search purposes counted as acceptable. The judges said finding info this way did not replace buying books. Their working versions hold no source material you could actually pull back out. Authors Guild Inc. v. HathiTrust[3] similarly upheld digitisation for search and accessibility as transformative.

These analogies, however, have been sharply contested. The critical disanalogy is that Google Books directed users back to the original work; the value proposition of an LLM is the opposite: it provides answers and synthesis that may eliminate the need to consult the original at all. When a model reproduces a news article verbatim or summarises subscriber-only content in detail, the fourth factor, market harm, is plainly implicated.[4]

The landmark ruling on this question came in Thomson Reuters Enterprise Centre GmbH v. Ross Intelligence Inc. Ross had built an AI-powered legal research tool trained on Westlaw's proprietary headnotes, concise editorial summaries authored by Thomson Reuters editors. A judge ruled in favour of Thomson Reuters when he found that Ross’s artificial intelligence system performed tasks too similar to those already handled by Westlaw. Although the new tool was positioned as an innovation, it occupied an existing niche, offering virtually no functionality different from existing solutions. Profit motives clearly played a role by tipping the scales toward unacceptable borrowing. There was no need to duplicate existing materials to create something new. Competition between the services created measurable damage, making it harder to justify reuse under legal protection. The outcome turned on how closely the systems overlapped in practice, not just on design intent. The case has been appealed to the Third Circuit (No. 25-8018, 3d Cir. Jun. 17, 2025), making it the first federal appellate court to address fair use principles in AI-generated commercial products.[5]

In the New York Times Co. v. OpenAI Inc.[6] case, which was preserved from a partial motion to dismiss in March 2025, the Times says ChatGPT can reproduce the publisher’s articles word-for-word, including material available only to paying subscribers, in response to specific queries. The US Copyright Office's May 2025 pre-publication report on Copyright and Artificial Intelligence, Part 3: Generative AI Training, took an unequivocal position: using copyrighted works to train AI models "may constitute prima facie infringement of the right to reproduce such works," and certain uses cannot be defended as fair use, particularly where AI output directly competes with or substitutes for the original content market.[7]

Implied Consent and the Myth of the Open Internet

A less formally articulated defence is implied consent, the theory that a creator who publishes freely on the internet has implicitly consented to automated collection in the same way they consent to being crawled by a search engine. This argument rests on the Robots Exclusion Protocol (robots.txt), treating the absence of a disallow instruction as consent to scraping.

Legal scholars have increasingly challenged this framing. The analogy between search engine crawling and AI training breaks down fundamentally because their purposes and the benefits to content creators are asymmetric.[8] A creator who allows Google to crawl their site receives traffic referrals; an AI training crawler takes content but provides no referral in return. As Tech Policy Press observed in January 2025, the "symbiotic relationship" that made the open web function has been irrevocably broken.[9]

As one article in the Oklahoma Law Review (2025) argued, the foundational basis for early large-scale web crawling was not fair use but an implied licence, a contractual theory of consent premised on a quid pro quo. AI companies, by copying content for entirely different commercial purposes that confer no reciprocal benefit, cannot legitimately invoke the same implied licence.[10] A Northwestern University Law Review article from 2025 further identified the "leaky" nature of robots.txt: it is a voluntary protocol with no legal force, it only blocks direct scraping (leaving indirect acquisition via intermediaries like Common Crawl unaddressed), and treating the absence of a disallow instruction as publisher indifference is a presumption courts may not share.[11]

The US Copyright Office has declined to treat robots.txt compliance as a sufficient indicator of consent for copyright purposes. Deliberately bypassing a robots.txt restriction may also constitute a violation of the Computer Fraud and Abuse Act, as reinforced by hiQ Labs, Inc. v. LinkedIn Corp.[12], though that case involved public profile data rather than copyrighted content.

The European Union: TDM Exceptions and the Opt-Out Regime

The EU's Directive on Copyright in the Digital Single Market (DSM Directive) 2019/790 introduced two TDM exceptions. Article 3 creates a mandatory exception for TDM conducted by research organisations for scientific research, a narrow carve-out that does not benefit commercial AI companies. What Article 4 allows is text and data mining even when profit is involved - yet creators retain a way to exclude their work, provided they signal it clearly, perhaps through automated tags. Referenced directly in the EU AI Act's fifty-third clause, this exit option forms part of broader regulatory alignment.

Two German court decisions illustrate the framework's limits. In Kneschke v. LAION[13] (Hamburg Regional Court, September 2024), the court held that creating AI training datasets qualifies as TDM under Article 3, though it left open whether an effective opt-out had been made. In GEMA v. OpenAI[14] (Munich Regional Court, November 2025), the court confirmed that the Article 4 TDM exception covers AI training but found that where a model memorises and reproduces training content ("regurgitation"), that use falls outside the exception's protection.[15]

Still, rights holder groups claim tossing Article 4 into AI training clashes with the Berne Convention’s three-part check. That rule insists limits must fit narrow situations, avoid messing with usual uses, plus skip unfair harm to writers’ real stakes. Opting out? Seen as shaky by artist circles, it makes makers dodge inclusion rather than forcing tech firms to ask first.

India: Fair Dealing Under Section 52 and the ANI v. OpenAI Battleground

India's engagement with these questions is no longer theoretical. In late 2024, Asian News International (ANI) Media Pvt. Ltd. filed a legal complaint with the Delhi High Court targeting OpenAI. It claimed automated systems pulled ANI’s protected reports - some behind paywalls - into ChatGPT’s learning pool absent permission or payment. Voices joined later: DNPA stood up, IMI stepped forward, publishers grouped together. This courtroom clash now marks Asia’s biggest dispute where artificial intelligence meets ownership rights.[16]

The core legal question is whether OpenAI's use qualifies as fair dealing under Section 52 of the Copyright Act, 1957. India's fair dealing framework is considerably narrower than the US fair use; it is a closed, exhaustive list of enumerated exceptions. One ruling from Delhi’s top court made a clear point: in Super Cassettes v. Chintamani Rao[17], it said that Section 52 lays out exactly what does not count as copyright infringement. The list there stands complete on its own, built with precision. Judges can’t stretch those exceptions wider unless lawmakers step in directly. What fits outside stays outside, by design.

The permissible purposes under Section 52(1)(a), private study, research, criticism or review, and reporting of current events, do not, as currently interpreted, comfortably encompass commercial-scale ingestion of news content to train a commercial AI product. The DNPA has argued that "unauthorised scraping of copyrighted content and storing it for AI training does not qualify as fair dealing" and that even temporary storage during pre-training constitutes infringement under Section 51.[18]

OpenAI's defence rests on the argument that training extracts "non-expressive" features, statistical patterns rather than expression, and that the trained model contains no verbatim copy of OpenAI's articles. The Delhi High Court appointed two independent copyright experts whose opinions differ: one holds that copying and storing text for training models may fall under the permissible "storage" exception in Article 52(1)(a), provided the expressive content is not made publicly available; the other argues that unlicensed commercial copying for training AI falls squarely outside the scope of fair use.[19]

Before handing down its ruling in April 2026, judges at the Delhi High Court were hesitant about a decision that could change how intellectual property is treated in courts across India. There’s no specific law that allows text and data mining under local copyright law, putting the country at odds with places like Japan, where such uses are allowed for profit-making AI training. Tokyo has cleared the way for machine-learning research, but New Delhi has been hesitant, leaving innovators in a grey zone. The ruling could tip the scales one way or the other. The government's Commerce Ministry has established a panel of eight experts to assess whether legislative amendment is required, acknowledging that the existing framework may be inadequate.[20]

 

Market Harm and the Licensing Alternative

The fourth fair use/dealing factor, market harm, may be the most legally consequential dimension of these disputes. In Campbell v. Acuff-Rose, the Supreme Court called this "the most important element of fair use." If AI training systematically undermines the market for original works by enabling users to extract the value of journalism and creative writing without paying for it, the case for fair use collapses regardless of how transformative the training process itself may be.

Professor Paul Goldstein, in his treatise Copyright (3rd ed., Wolters Kluwer, 2012), argues that the market harm analysis must account not only for lost direct sales but also for the destruction of licensing markets.[21] Content creators have advanced exactly this argument: even if readers still subscribe to the New York Times, OpenAI's refusal to pay licensing fees has destroyed the market for AI training licences, a market that would otherwise generate substantial revenue for creators. The USCO's May 2025 report noted that "the market for licensing content for AI training is emerging and evolving rapidly" and that reliance on fair use may unjustifiably foreclose a market rightsholders would otherwise exploit.

That some AI companies are willing to pay for licences while others invoke fair use suggests the latter position is partly a commercial strategy rather than a settled legal analysis. The Associated Press, News Corp, and Axel Springer have already struck licensing deals with OpenAI. Most tellingly, in September 2025, Anthropic agreed to pay $1.5 billion to settle claims by authors whose books were used to train Claude approximately $3,000 per copyrighted book across an estimated 500,000 titles. The settlement, while not an admission of liability, is powerful evidence that AI companies privately recognise the copyright exposure their public fair use arguments resist acknowledging.[22]

The Way Forward: Three Possible Frameworks

Three broad frameworks are emerging globally:

The first, Compulsory Licensing with Compensation, would require AI companies to pay standardised licence fees into a collective fund administered by copyright societies and distributed to rights holders in proportion to the presence of their works in training data, acknowledging both the public benefit of AI development and creators' legitimate economic interests.

The second, Opt-Out with Transparent Disclosure, would formalise the robots.txt mechanism into binding law and require AI companies to maintain publicly auditable training data logs; this is essentially the EU's Article 4 model at a global scale, though critics rightly note it burdens creators with exclusion rather than obliging AI companies to seek consent.

The third, a Legislative TDM Exception with Safeguards, would create explicit statutory exceptions for text and data mining modelled on Japan's permissive approach but conditioned on prohibitions against verbatim reproduction and limited to contexts where a licensing market does not already exist, a path India's Commerce Ministry expert panel appears to be actively evaluating.

Conclusion

The question of whether scraping the internet for AI training constitutes fair dealing, implied consent, or mass copyright infringement has no single, clear answer and that ambiguity is itself the legal crisis of our moment. What is clear is that the creative and journalistic ecosystem that produced this training data has not been compensated for its contribution.

Thomson Reuters v. Ross Intelligence, the ANI v. OpenAI proceedings in Delhi, and the USCO's 2025 report collectively signal that the free-rider era is drawing to a close: public accessibility is not the same as permission, and the scale of reproduction demands legal accountability proportional to the economic value extracted.

For India, ANI v. OpenAI is an inflexion point. Another solution could be a more permissive reading of Article 52, under which the machines do their nondescript adaptive thing unless money is involved, in which case payment is required. The law has to get this right.

[1] Chung Peng Lee et al., How Do Data Owners Say No? A Case Study of Data Consent Mechanisms in Web-Scraped Vision-Language AI Training Datasets, arXiv:2511.08637 (2025), https://arxiv.org/abs/2511.08637.

[2] Campbell v. Acuff-Rose Music, Inc., 510 U.S. 569, 578–79 (1994), https://supreme.justia.com/cases/federal/us/510/569/.

[3] Authors Guild, Inc. v. HathiTrust, 755 F.3d 87, 97–102 (2d Cir. 2014), https://www.copyright.gov/fair-use/summaries/authorsguild-hathitrust-2dcir2014.pdf.

[4] 17 U.S.C. § 107(4); Authors Guild v. Google, Inc., 804 F.3d 202, 223–25 (2d Cir. 2015), https://www.courtlistener.com/opinion/3137282/authors-guild-v-google-inc/.

[5] Thomson Reuters Enter. Ctr. GmbH v. Ross Intel. Inc., No. 1:20-cv-00613-SB, 2025 WL 458520 (D. Del. Feb. 11, 2025), https://www.bakerlaw.com/thomson-reuters-v-ross/.

[6] The New York Times Co. v. Microsoft Corp., No. 1:23-cv-11195 (S.D.N.Y. filed Dec. 27, 2023), https://www.courtlistener.com/docket/68117049/the-new-york-times-company-v-microsoft-corporation/.

[7] U.S. Copyright Office, Copyright and Artificial Intelligence, Part 3: Generative AI Training (Pre-Publication Version, May 2025), https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-3-Generative-AI-Training-Report-Pre-Publication-Version.pdf.

[8] Mark A. Lemley, How Generative AI Turns Copyright Upside Down, 25 Colum. Sci. & Tech. L. Rev. 190 (2024), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4517702.

[9] Courtney C. Radsch, The Case for Requiring Explicit Consent from Rights Holders for AI Training, Tech Policy Press (Jan. 17, 2025), https://www.techpolicy.press/the-case-for-requiring-explicit-consent-from-rights-holders-for-ai-training/.

[10] David W. Opderbeck, Copyright in AI Training Data: A Human-Centered Approach, 76 Okla. L. Rev. 951 (2024), https://digitalcommons.law.ou.edu/olr/vol76/iss4/3/.

[11] David Atkinson, Putting GenAI on Notice: GenAI Exceptionalism and Contract Law, 120 Nw. U. L. Rev. Online 27 (2025), https://northwesternlawreview.org/articles/putting-genai-on-notice-genai-exceptionalism-and-contract-law/.

[12] hiQ Labs, Inc. v. LinkedIn Corp., 31 F.4th 1180 (9th Cir. 2022), https://law.justia.com/cases/federal/appellate-courts/ca9/17-16783/17-16783-2022-04-18.html.

[13] Robert Kneschke v. LAION e.V., Case No. 310 O 227/23 (Hamburg Reg'l Ct. Sept. 27, 2024) (Ger.), https://www.wipo.int/wipolex/en/judgments/details/2381.

[14] GEMA v. OpenAI, Case No. 42 O 14139/24 (Munich I Reg'l Ct. Nov. 11, 2025) (Ger.), https://www.wipo.int/wipolex/en/judgments/details/3889.

[15] Taylor Wessing, AI and Copyright Litigation in the EU and the UK (2025), https://www.taylorwessing.com/en/interface/2025/online-and-ai-generated-content/ai-and-copyright-litigation-in-the-eu-and-the-uk.

[16] ANI Media Pvt. Ltd. v. Open AI OpCo LLC, CS(COMM) 1028/2024 (Delhi High Ct. July 24, 2026) (India), https://indiankanoon.org/doc/93327052/.

[17] Super Cassettes Indus. Ltd. v. Chintamani Rao, I.A. No. 13741/2006 in CS(OS) No. 2282/2006 (Delhi High Ct.) (India), https://indiankanoon.org/doc/576454/.

[18] ANI Media Pvt. Ltd. v. Open AI OpCo LLC, CS(COMM) 1028/2024 (Delhi High Ct. July 24, 2026) (India), https://indiankanoon.org/doc/93327052/.

[19] Id.

[20] Department for Promotion of Industry and Internal Trade, Ministry of Commerce & Industry, One Nation One License One Payment: Balancing AI Innovation and Copyright (2025), https://www.dpiit.gov.in/static/uploads/2025/12/ff266bbeed10c48e3479c941484f3525.pdf.

[21] U.S. Copyright Office, Copyright and Artificial Intelligence, Part 3: Generative AI Training (Pre-Publication Version, May 2025), https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-3-Generative-AI-Training-Report-Pre-Publication-Version.pdf.

[22] Bartz v. Anthropic PBC, No. 3:24-cv-05417 (N.D. Cal. 2026) (class action settlement), https://www.anthropiccopyrightsettlement.com/documents.

Suggested Citation:

Ashish Pratap Singh, Scraping the Internet for AI Training Data: Fair Dealing, Implied Consent, or Mass Copyright Violation?, NIDAN IPR Research Series 2026, NIDAN Centre for Legal and Policy Research (2026).