On September 25, 2026, early in the morning Japan time, OpenAI's crawlers changed how they read my sites. I saw it in the server logs. OAI-SearchBot (search) and GPTBot (training) stopped just fetching HTML and started running JavaScript and rendering pages.
The accepted wisdom has been that AI crawlers don't run JavaScript. The Vercel and MERJ study from December 2024 said none of OpenAI's three bots executed it. What I'm seeing is a change from that.
This post has the numbers from the logs first. After that comes a list of what I would do now if I wanted ChatGPT to mention a site. I couldn't find any announcement from OpenAI, so where I'm guessing, I say so.
The main site is @SOHO (atsoho.com), a Japanese freelance marketplace built on Next.js with the App Router. A few of my product sites run on the same server. I did not trust the user agent alone. Every request was checked against OpenAI's published IP ranges (gptbot.json, searchbot.json, chatgpt-user.json), and anything outside them was dropped as a fake.
| Date | OAI-SearchBot | GPTBot | ChatGPT-User | |---|---|---|---| | Sep 10 | 1,108 | 34 | 13,538 | | Sep 22 | 1,253 | 83 | 657 | | Sep 24 | 7,082 | 5 | 911 | | Sep 25 | 41,444 | 28,484 | 1,645 | | Sep 28 | 14,339 | 152,009 | 1,071 |
OAI-SearchBot jumped at 4 a.m. JST on September 25, and GPTBot took off at noon the same day. Until September 24 GPTBot came from one IP a day. From September 25 it came from 36 to 131.
When I looked inside, almost all of the extra requests were Next.js link prefetches, the ones with ?rsc= plus five random characters at the end. That string is not in the HTML. It only shows up when the page is rendered and its JavaScript runs.
・September 22: 80% of OAI-SearchBot requests were plain HTML fetches, 2% were prefetches ・September 28: 84% of OAI-SearchBot requests and 97% of GPTBot requests were prefetches
So 150,000 requests a day does not mean 150,000 pages read. Most of it is what a rendered page fires off on its own.
Each prefetch carries the page that triggered it in the Referer. Counting distinct Referers tells you how many pages the bot rendered.
| Date | Rendered by OAI-SearchBot | of which job pages | Rendered by GPTBot | |---|---|---|---| | Sep 18 | 56 | 2 | 17 | | Sep 25 | 861 | 108 | 1,994 | | Sep 28 | 539 | 36 | 4,395 |
Of the 4,309 pages GPTBot rendered on September 28, only 28% had been fetched from my server by anyone that day. My HTML is not cached at the CDN, so the CDN did not serve it either. My read is that OpenAI re-renders HTML it stored earlier in its own headless browser, and the only thing that reaches my server is the prefetches that fire during that render.
OpenAI's own docs say that if a site allows both bots, it may use the results from one crawl for both purposes to avoid duplicate crawling. GPTBot rendering what OAI-SearchBot fetched fits that.
On the same server, all four Next.js sites that have articles saw OAI-SearchBot go up 5 to 15x on September 24 and 25. They run different Next.js versions and deploy differently, and I did not change anything on my side that week. The change came from OpenAI.
But on Cloudflare Radar, OpenAI's share of all AI bot traffic stayed flat through September. It looks like rendering is only switched on for some sites.
It reaches static sites too. On one of my sites that doesn't use Next.js, OAI-SearchBot pulled images and CSS on September 25, which is what rendering looks like. A static page does not fire prefetches, though, so the request count barely moves. The spike on the Next.js sites is mostly prefetch amplification.
Around the same time, visits from ChatGPT to @SOHO's job pages went up, and daily signups rose about 1.7x. The number of people coming from ChatGPT did not change. Where they landed did. It used to be mostly blog posts, and from about September 25 it was mostly job pages. That lines up with job pages rendered by OAI-SearchBot going from 2 a day to 36 to 108. I cannot prove one caused the other.
These six come from the logs above and from OpenAI's crawler docs. Let OAI-SearchBot in