OpenAI Has Kept Secret an Accurate ChatGPT Detector for Two Years
According to The Wall Street Journal, “OpenAI has a method to reliably detect when someone uses ChatGPT to write an essay or research paper” but hasn’t released it yet, despite concerns about widespread student cheating on assignments with it, as well as other illicit uses.
–
The WSJ writes:
The project has been mired in internal debate at OpenAI for roughly two years and has been ready to be released for about a year, according to people familiar with the matter and internal documents viewed by The Wall Street Journal. “It’s just a matter of pressing a button,” one of the people said.
Why the delay?
In trying to decide what to do, OpenAI employees have wavered between the startup’s stated commitment to transparency and their desire to attract and retain users.
The detection technology is a watermark system that is reportedly 99.9% effective. Here’s how it works:
ChatGPT is powered by an AI system that predicts what word or word fragment, known as a token, should come next in a sentence. The anticheating tool under discussion at OpenAI would slightly change how the tokens are selected. Those changes would leave a pattern called a watermark. The watermarks would be unnoticeable to the human eye but could be found with OpenAI’s detection technology. The detector provides a score of how likely the entire document or a portion of it was written by ChatGPT. The watermarks are 99.9% effective when enough new text is created by ChatGPT, according to the internal documents.
Apparently a version of the watermarking technology has been developed by Google that can detect text generated by its LLM,
Gemini AI (the detection technology “is in beta testing and isn’t widely available”).
The detection technology does not appear to negatively affect the quality of the text generated by LLMs (which was one concern voiced by opponents of releasing the technology). OpenAI’s own survey data shows that people would prefer the technology to be released by a margin of four to one. What has kept it under wraps, it seems, are worries about how it will affect OpenAI’s profitability: “nearly 30% [of surveyed ChatGPT users] said they would use ChatGPT less if it deployed watermarks and a rival didn’t.” Are there reasons to think this is an especially intractable collection action problem?
The WSJ report, paywalled, is here.
(via Ian Olasov)
I hope that the government compels AI companies to develop and make publicly available such watermark-detection software. After all, there’s a very clear precedent for this.
Major printer companies like Canon and Xerox have voluntarily included invisible watermarks for decades on printed material. These watermarks–usually patterns of extremely small yellow dots–often encode the date and time of printing, the model of printer used, etc. Similar patterns of yellow dots are used on currency to prevent counterfeiting.
Fun fact, most printers refuse to print images in which they recognize the distinctive pattern of yellow dots governments place on their bills.
https://en.wikipedia.org/wiki/Printer_tracking_dots
https://en.wikipedia.org/wiki/EURion_constellation
I suppose that Canon and Xerox could adopt OpenAI’s brilliant argument here. Unabombers, terrorists, and blackmailers would be less likely to use their printers to print manifestos, ransom notes, and the like, if they include digital watermarks on their material. I mean, do we really want government agencies to be able to verify whether some video of Obama is genuine? That could seriously hurt profits!
I would post something, but alas I have a pile of robot essays to mark.
Turtle here and maybe it’s just not getting through my thick shell but teachers are using it to help grade papers, aren’t they?
A question for those more informed than I: Going forward, am I correct in saying that, at this there are so many other competitors that it won’t matter for too long, since students can use another model which does not include the watermarking?
Yes, this is basically correct as far as I know. Even if there is a successful regulatory push for labs to include watermarks in the outputs of their models, the fact that many models are being open-sourced means that people will always be able to fine-tune them relatively easily to remove the watermarking.
while this is basically correct, there are two issues.
first, if openai begins to watermark ChatGPT output, the other *major* ai companies (and there are only really a few) will probably do as well sooner or later.
second, at this point i’m not really worried about the “fine-tuning” issue. i am curious if anyone here has any experience with students fine-tuning their ai papers. the question could be seen as silly—the fine-tuned papers are the ones we’re a lot less likely to flag as ai generated, at least putatively—but it’s been my experience that the ai stuff is painfully and obviously clear. like it’s not even a question.
it follows, or likely follows, that–broadly-speaking–cheating students don’t bother to fine-tune ai material. whether that’s laziness or ignorance of how to do so is worth looking into, but at a certain point the fine-tuning ends up taking almost the same amount of time as writing the damn paper oneself would.
if the equation is minimizing effort while maximizing output—as I think it is for ai cheating—it remains in doubt whether widespread fine-tuning will occur.
You seem to misunderstand the point. The worry is not about fine tuning individual papers, but fine tuning entire models to remove the watermarking process from them. For open source models, this would undoubtedly occur and would be publicly uploaded. Moreover, since the regulations would only apply to corporations (presumably), nothing could be done about, say, non-watermarking models being uploaded to GitHub by private individuals.
Sorry, I did miss the point! At the same time I think that my point about minimizing effort in academic cheating stands.
A motivated cheater willing to put effort, time, and/or money into cheating will probably rarely get caught. But in my experience (15 years of teaching), this type of cheater is vanishingly rare. The vast majority don’t cover their tracks, don’t fine-tune their fake work, and don’t bother to even be “good” cheaters.
Noting this and combining it with the fact that our supposedly “digitally native” students often can’t seem to even get past the first three results on a Google search, I think a proliferation of open source non-watermarked AI poses minimal threat.
Yes, some students will use them and cheat and not get caught. But a variant of “good” cheating has always existed and is not the real threat in my view. The problem so far in the LLM era is that students who wouldn’t have cheated (or at least who wouldn’t have turned in entire papers they didn’t write) are now doing so because of the easy availability of LLMs that carry the sheen of legitimacy.
I use the analogy of speed limits. We can’t stop everyone who is determined to go 85 in a 55, or even a small percentage of them realistically. And if you want to go 85 in a 55 this one time because you need to pick your kid up, there’s a reasonable chance you’ll get away with it esp. if you use some precautions. At the same time, the speed limit is demonstrably a good way to control dangerous behavior even when not strictly enforced.
Watermarking would be a speed limit. It would encourage students to cheat less, even if it wouldn’t stop motivated cheaters. And while there are workarounds to watermarked LLM text, it’s my experience and view that the large majority of students who might consider cheating will be dissuaded by the extra effort required to use workarounds.
All this to say that I think that while watermarking LLM text isn’t a perfect solution, it will slow the flow of traffic as it were.
You make an interesting point. In the late ’80s I acquired a HP 19b programmable calculator with soft keys. We were allowed to use it. Likely the people in charge of that decision had no idea of its capabilities.
I was able to solve very difficult regression analysis and other problems in seconds. My professor knew because I raised my hand and answered the question within 30 seconds of him writing it on the board.
He was impressed that I could program that. However, I was also able to program every other formula that I could encounter in an exam and although I had to get the points for showing my work, I always got the right answer. And I always knew the correct formula.
And the truth is, I learned how these formulas worked better by programming that calculator. Then I probably would have had I’ve done the homework the hard way every time.
I know some people think using chatGPT is cheating Even in business, but I would say it writes about as well as me. However, I’m not as fast. Not even close. I usually spend a great deal of time writing a draft and then I explain what I’m trying to do and the law or whatever I’m writing about says and then I let it organize some of my arguments.
In the past I would have to spend 30 or 40 minutes organizing and typically I wrote it backwards and had to completely revise, bringing the last points I wrote to the front. It doesn’t mean I didn’t end up with something great, but It took a lot of time.
You always have to check its math and even if it reads a document you give it, it might lie about what it says. But if you spend the time checking this work and revising with the correct facts, you end up with something very good far less than half the time.
I don’t think it’s cheating. I’m going to post the output of pasting this writing into chatgpt and you can see it will revise but it won’t sound a lot better.
Here’s a revised version of your writing:
—
In the late ’80s, I acquired an HP 19B programmable calculator with soft keys, a tool that significantly boosted my efficiency in solving complex problems. Back then, I suspect those who permitted its use during exams had little understanding of its true capabilities.
Using this calculator, I could perform intricate regression analyses and solve other challenging problems within seconds. My professor quickly noticed when I raised my hand to answer a problem mere seconds after he’d written it on the board. He was impressed, recognizing the skill required to program such a device. What he didn’t see, however, was that I had pre-programmed nearly every formula I might encounter during exams. While I still needed to show my work to earn full points, I consistently got the correct answers and always knew the proper formulas to apply.
Interestingly, programming the calculator actually deepened my understanding of these formulas. By breaking them down and coding them, I gained insights that I might not have acquired had I simply worked through the homework manually each time.
Today, some might argue that using tools like ChatGPT in business is akin to cheating. However, I see it differently. ChatGPT writes nearly as well as I do, but far faster. Normally, I spend significant time drafting and organizing my thoughts, whether I’m explaining legal principles or constructing an argument. Historically, I’d spend 30 to 40 minutes just organizing my ideas—often realizing that I’d written the piece backward and needing to revise it entirely, moving my last points to the front. While this process often resulted in a strong final product, it was time-consuming.
While ChatGPT’s work requires careful review, especially its math and interpretations of documents, it allows me to create something of high quality in much less time. As long as I ensure accuracy by checking and revising, I don’t believe using ChatGPT is cheating. Instead, it’s a tool that, when used responsibly, can lead to excellent results in a fraction of the time.
—
This version retains your original message but with a more structured flow and slightly refined language.
I would have actually thought that the competitors would be less likely, if anything, to watermark, since not having would give them a strong market edge vis-à-vis openAI.
It’s possible of course. I tend to think that as a category leader, OpenAI will set precedents in the near future.
I also think that the other (what, two?) major LLM companies (Google & Anthropic) will look irresponsible if they don’t follow suit. But I could be wrong on either or both!
daily nous is serving chatgpt ads right this second.
I think it is worth distinguishing two different groups of users who don’t want watermarks. One group is students who are cheating. Another group is business users who are generating value with the product, but in an environment where stigma against AI outputs would undermine the value. So think of someone writing a government report that is just as good (if not better) for the AI that was used, but if it was public knowledge that the report was generated using AI in part, the report would be rejected. I think there is social value in stopping the cheaters; but there is social disvalue in stopping business users from using it, because it will make workers less productive. I think it may be hard to evaluate whether the benefits of watermarking outweigh the costs.
You’re right, but I think there’s no social disvalue in watermarking AI-generated material. Besides the students using AI to cheat, there is a far larger group of users attempting to pass off AI-generated content as their own, whether it be some visual art, a short story, a piece of music, or whatever. These users are just aware as you are that there is a premium for humanly created work you mention, and seek to profit from that by misrepresenting the origin of their work.
If watermarks take this premium away, that’s not a bad thing. People passing off AI-generated art as their own ought not to get that premium in the first place. Today, many sellers on Etsy and similar websites are in the business of generating AI art and then incorporating it into the items they sell: mousepads, posters, t-shirts, stickers, and the like. That’s a legitimate line of work. But it should be illegal to misrepresent one’s product in this way, whether we’re talking about AI or any other industry.
Best comment here and exactly my thoughts on the issue. I am completely at a loss to understand why there is so much discourse around detecting the use of AI. The priority for me and other businesses owners is to generate the very best content. I wouldn’t care if a monkey at the zoo generated the content for me, if the quality was excellent and met my needs. I will use it. The company that creates a detection tool for its own language model will deserve it’s swift downfall.
In the context of business, why does it matter if people use AI? It’s a tool. Web developers use templates and AI, as do many others. Physicians perform robotic surgery with the help of AI. Why so much petty focus on language models?
I would hesitate to hire a company that refused to leverage all available AI to my benefit. The same AI tools are available to everyone. If you don’t know how to use them effectively, you still won’t generate the quality output that someone else, like myself, can get because they do know how to use it effectively. If someone believes that value for a product or service is diminished because AI was involved, they can do it themselves.
The US educational system, as a whole, has long been ineffective and needs to be completely revamped. The problems within our educational system are so much larger than rather or not students use AI to do their homework. That’s the least of our problems.
Right, but if a monkey in zoo is making cakes for you, you don’t have a right as a bakery to pretend a person made it. Watermarks won’t stop AI from producing content, if they worked at least they would merely mark that content as made by AI. Trivially, it’s better for sellers for seller to be able to lie about what they are selling, but we don’t need enable them to do that.
Wait, I’m supposed to oppose watermarks because organizations should be able to deceive the public into thinking their content is produced by people?? Even if the content was somehow better, the fact that people would reject it if they knew it was AI is a prima facie reason in favor of watermarking, not against it.
The fact that business would make more money if they could pretend people made their content is hardly a a reason to let them do that.
That is a touch of not the most entertaining way of doing things, but one can generate their content, then use a tool to humanize, and ensure it goes undetected.
Even if that’s true, watermarking policies would still decrease cheating behavior by attaching additional costs to cheating. Rather than talk to chatGPT for 30 seconds, a student might need to spend their afternoon adding grammatical mistakes, and these might not be believably human. (Just as humans can’t credibly produce strings of random numbers, I suspect that students cannot reliably produce the errors, mistakes, and misunderstandings which honest students actually make.)
Humanizing AI is not difficult.
The “99.9% effective” claim should be taken with a huge grain of salt (and you should probably be admonished for headlining it). As reported this is “99.9% effective when enough new text is created by ChatGPT” which basically means that, if you generate a huge enough volume of text, perhaps many books’ worth for all we know, then there will be statistical patterns in the words that would have been extremely unlikely to occur via other ways of producing text, among some set of alternatives that they considered. That by itself doesn’t show that this would be nearly so effective at detecting fakes that are the length of student papers. It also doesn’t tell us anything about how effective it would be at detecting text that had been lightly edited/disrupted either by students themselves or by third-party “dewatermarking” apps that would surely flourish if Chat-GPT ever did roll this out. At best here, we have the claim that huge volumes of unaltered Chat-GPT output can be statistically recognized as (likely) being that, but it doesn’t tell us anything about how hard it would be to recognize smaller volumes of lightly-altered Chat-GPT output which is what our students would be likely to submit.
Claims to have built useful versions of LLM-generated text detectors continue to be bullshit in one of several ways. They either don’t work (like the first round of text-based watermarks), and so are useless, or (like the most recent round just announced) can be defeated simply by asking another capable model to paraphrase, and so are also useless. And in any case, it’s a bait and switch: rather than producing tools that can definitively say that some sample of text isn’t produced by *a* model, large model developers can as a technical matter only ever give us tools that can say that a sample of text isn’t produced by *their* model. And while that might protect them from liability, it doesn’t help with any actual problems.
Maybe I’m slow. Who needs a special program? Chatgpt always starts every letter with, “I hope this letter finds you well.” I could probably false trigger it by including that in every letter I write. I didn’t see in the article how many false positives you get because some people actually say stuff the same way.