There's an underappreciated comment in the other thread about SynthID and OpenAI [0] that captures what (IMO) the hacker ethos on this should be. We care about privacy, we should not accept tools that barcode our every digital move. (note that the counter of "well, they don't do that yet" is not particularly convincing)
Building a tool that tries (and probably fails) to remove the watermark (due to the arms race that large corporate machines will win) is tacitly accepting the barcode. The hacker ethos should be, first and foremost, to run open source models locally without relying on a corporation.
>due to the arms race that large corporate machines will win
Much like how the entirety of Hollywood, book publishers, academic publishers, and game developers have won against piracy despite being some of the largest corps on earth and dedicating untold billions to the issue over the past 30 years?
They didn’t win because of DRM. They won because of the regulations that grant a monopoly for a specific term in the form of a copyright. Society has recognized that incentivizing creative acts requires a temporary grant of monopoly to ensure the necessary scarcity to make money and recover the costs of creation. The real problem is Disney keeps expanding that time period so things never enter the public domain
This is again conflating at least two things and this is so prevalent in this context. Let us not conflate how annoying DRM:s are to us users that buy the things, with pirates thinking they somehow have a right to use any software without paying fairly for it. I would even go as far as to say that you pirates are the reason I have to have a DRM in the shit I bought and paid for.
> I would even go as far as to say that you pirates are the reason I have to have a DRM
I think this is largely an incorrect take. DRM is anti consumer, not anti piracy. In fact, it has done very little to deter actual piracy (and remember it only takes ONE person to break the DRM), while affecting some casual pirates and all legitimate users. In the process, they got rid of reselling stuff you own.
It's anticonsumer, not antipiracy, never forget that. It means something like this would have happened regardless of pirates.
They succesfully did away with 2nd hand markets and the concept of "owning" anything. So yes, I would imagine DRM would continue to exist without piracy.
I think so, because their main goal is to prevent unwanted use of the digital product -- to the detriment of end users -- in more ways than just piracy. In fact, they don't solve the piracy issue.
I am not sure how I am conflating two things, it would be helpful if you could expand or connect to my argument. Perhaps I am misunderstanding.
My argument is that the grant of monopoly is a regulatory decision and the real cause of "winning". No amount of DRM would confer the same benefit because the ability to bypass it through piracy would be totally legal with no economic or other consequences and so a robust cracking and distribution ecosystem would emerge. Thats a drastically different story than when napster gets shut down, and limewire gets shut down and pirate bay gets shut down every time it relaunches. Imagine a world where there is are 1000 pirate bays
Piracy is as easy now as it was pre-DRM. DRM is the digital equivalent to security screws on electronics, in that they’re a mechanism for lawyers to argue their client made an attempt despite being easily bypassed with a trivial amount of effort.
Exactly this. The real power is in the regulatory grant of a monopoly that comes with rights like the ability to sue for damages, issue take down notices etc. the DRM does allow restrictions on distribution because many people can’t be bothered to remove them, but more importantly the act of removing them is evidence of the intent to knowingly violate the copyright which might be harder to prove otherwise
And now that they're trying to push up the margins and the streaming ecosystem is fragmenting making everything into a series of bundles again, piracy is on the rise again.
They did win for a while because they stamped out 99% of piracy. In the early days of streaming it was legitimately difficult to argue for piracy. Streaming was just too convenient and too cheap.
But, they are greedy above all else. And so, we are once again seeing a resurgence of piracy. Large corporations seem to snatch defeat from the jaws of victory.
By the time you're building (or buying) the necessary highly esoteric and expensive ultracentrifuge setup I think you would be well outside the realm of "hobbyist" unless someone insists on the most unreasonably pedantic definition for the term.
Unless we're only considering final assembly. Just gotta get that weapons grade fissile material supplier lined up. That might or might not qualify as rich hobbyist territory depending on how high a price tag is permissible.
In theory, there's also direct laser-based isotope separation. It's a technology that is being actively suppressed, and that's one case where I very much in favor of that.
This subthread starts off with the argument that the big corps will never beat the little determined hackers, one of the founding myths of the early internet. And then every now and then a strong little branch of the argument runs up against an example and it becomes well sure, the little hobbyist hackers don't have anything there but that is because the big corps/gov/billionaires/whatever put so much into beating them.
I mean reading it all certainly sounds like the people on the little guy's side are overestimating the value of pluck, an observation Hollywood generally makes just before the heroes with pluck win for ever!
> And then every now and then a strong little branch of the argument runs up against an example and it becomes well sure, the little hobbyist hackers don't have anything there but that is because the big corps/gov/billionaires/whatever put so much into beating them.
It's almost never about the level of resources the organization puts in. The usual reason is that there isn't enough incentive to do it. What is a hobbyist going to do with a nuclear weapon? Why spend your time creating one if you, like the overwhelming majority of people, have no desire to blow up a city?
Preventing something that hardly anybody would be trying to do even if it wasn't being suppressed is a lot more practical than preventing something millions of people would do given the chance.
Yes. Winning against piracy doesn't mean you completely eliminate piracy. It means you scare enough people into not doing it and make it a bit harder to do for others.
Losing to piracy would see companies like Netflix and Spotify not thriving.
> It means you scare enough people into not doing it and make it a bit harder to do for others.
By which definition they utterly failed.
> Losing to piracy would see companies like Netflix and Spotify not thriving.
Not at all. Netflix and Spotify do well because they are a good value proposition for the average customer. Piracy is free at point of "purchase" but is (and always has been) expensive in terms of various sorts of overhead.
As long as enough people keep the pirate bays open, it will be there as an alternative when the services start their inevitable enshittification.
I for one do not enjoy the “Which service has the classic film I wanted to watch this week?” Nor having to switch services every time I want to see a new TV series.
We need (and have!) similar “free” alternatives to the watermarked generative services. Just like I hate the yellow dots on my printed images, I am not happy to have my creative assets (I do nothing nefarious) stained with SynthID.
> Winning against piracy doesn't mean you completely eliminate piracy.
But this is moving the goalposts. You can win against piracy either by making piracy less attractive or by making the paid offering more attractive. The first has utterly failed, piracy remains easy as a rule, and to the extent that they've succeeded it's not only disproportionately by doing the second thing, the DRM itself is a net negative because it has such a small effect on the ease of piracy while making the paid offering worse.
It doesn't make any sense at all. That's like saying browsing the internet with an ad blocker and other privacy tools is a tacit acceptance of tracking and ads, and that you should only visit websites that doesn't track or have ads.
It's already possible to lie with text. Pixels are pixels. If we can't blindly believe pixels to show the truth, we will be simply back to the pre-photography era which managed to have a concept of truth regardless.
For the umpteenth time, scale and ease of access and propagation matters.
A knife and a handgun aren’t comparable to a machine gun and a bomb. When you have equal access to all of those, the damage you can enact is exponentiated.
You could lie with text before, but it took effort and time and skill to do it convincingly. You could also lie with images but they took even more time and effort and skill, greatly limiting the pool of people who could do it and the possible damage.
When anyone anywhere can convincingly lie and have it do two laps around the world in a matter of minutes, the whole game changes.
It’s becoming very hard to believe that people making arguments like yours are doing so in good faith. Maybe you’re not even a person but a shill bot. That’s a very real and trivial possibility today, which is the whole point and illustrates the problem.
That’s a non-sequitur. This has nothing to do with “mainstream media”. Which, by the way, is not a singular entity you can blindly trust or distrust, don’t fall into that indoctrination trap. This is basic despotic citizen control: Get everyone to distrust those who hold them accountable; label them as irredeemable liars so you can do as you please.
Edit: Removed swipes while keeping the core argument.
You’re right. I should’ve done better and apologise. I have lightly edited the post above to keep the core point while removing the swipes.
I disagree with the irony point on the basis that none of us here (I assume) is a despot or in a position of power high enough to lie to the point of discrediting entire institutions to the masses. We can be victims of it, but not perpetrators. Also, nowhere have I accused anyone in particular of being a liar. Still, given that my ill chosen words will have no doubt clouded your understanding of the point—which is on me—I won’t hold that against you, I take responsibility.
I haven't heard anything about this from the mainstream media. What bothers me is the amount of misinformation being published on social media that people are taking as truth, but now with photorealistic photos and videos. It used to be you could reverse image search something and show that it came from a previous event, or find the original in the case of doctored images.
Now that anyone can manufacture convincing images out of whole cloth, it becomes impossible for the typical user to determine if what they're seeing is legitimate; this means not only that people can manufacture images to push a false narrative but also discredit legitimate images as falsified.
The 'mainstream media is bad' narrative is one manufactured almost entirely by right-wing influencers and politicians to create an environment where they can discredit any actual evidence that doesn't align with what they or their base want to hear. Once anyone with integrity is inherently suspicious, the narrative can be set by anyone to push their agenda, and because they've demonized actual news reporting they can point to any other outlet that counters their claims and say 'look, they're lying to your face, you can't trust them!'
> the main thing is tracking, ID verification, constant social credit-like ratings, etc. That's what is coming.
Do you mean things like masked armed federal agents demanding to see your proof of citizenship or they'll lock you up (or sometimes just locking you up anyway)? Is that what's coming? Is that what you're afraid of? Or is that fine if it only happens to people you don't agree with?
These talking points, as always, are fearmongering designed to trigger paranoia in people so that they'll go along with whatever the people in charge say will fix things. No one on the left has proposed any of these things, but people on the right are constantly saying that's whats coming because it's what helps them keep their base frightened and compliant, and you've fallen for it (or are part of it, I don't know).
> AI media generation is a red herring, the main thing is tracking, ID verification, constant social credit-like ratings, etc. That's what is coming.
It can be both things.
I worry about tracking and the post-truth world of everything fake and AI slop infiltrating everything. And like the person you're arguing with is saying, just because slop/fakery was possible before doesn't mean the widespread scale that AI and the internet have unlocked is of no concern.
It's the same walled garden principle. Benevolent Google will protect you from synthetic media by embedding invisible watermarks, from malicious apps by blocking sideloading, from porn by ID verification, etc etc. It's the same principle.
Are you opposed to people knowing that AI-generated content is AI-generated? Or are you just imagining that this is the first step towards 'someone' controlling your every move online somehow?
You know what would be a good counter to that? Don't use AI generation tools if they implement that. The argument that 'we should let AI generated media be indistinguishable from the truth because what if Google does something you don't like at some point' is pretty flimsy.
You and I did whenever we assumed the pixels were an upload of a digitized photo taken with an actual camera of, say, a historical event, or something in nature (e.g. some rare bird), a birthday party, etc.
The photo could be staged (e.g. Cottingley Fairies), it could be altered physically (like cutting or painting over, e.g. Stalin), it could be cropped intentionally to tell a different story (plenty of examples), and more recently it could be photoshopped, etc. All of these were possible, though harder than it is now, but let's not pretend we didn't "trust pixels". We all did, just as we trusted newspaper photographs. Now that era must come to a firm end, and I believe it's a tragedy.
Fair enough. While I would kind of wish AI could be reliably detected, deep down I know this is impossible and it would be pretty bad if we had, say, a prosecution that succeeded because "this 'provably-non-AI' photo places you at the scene of the crime" because only a few underground people know how to remove a watermark.
You raise an interesting point about the artificial generation of evidence used in court. In 1992, Michael Crichton wrote the book Rising Sun, which centers around the editing of security camera video footage to coverup a murder.
I also wonder if being able to prove that an image or video isn’t AI generated would lend credence to it, while in reality there are other methods to produce falsified video.
Well, you just have to convince the jury. The defense attorney will try to throw up all the reasons it could be falsified, but the prosecutor will say "All of that is unlikely - the defense attorney would like you to believe that farfetched story, but this is still a compelling piece of evidence." This is how it always is with any kind of testimony and evidence.
What stops someone from adding a watermark to an actually photographed (carefully framed?) picture to discredit it? There is no certainty either way, just suggestions from someone else about what the truth might be.
AI watermarks only give the illusion of maintaining the concept of truth. The government and corporations will still have access to un-watermarked models to destroy the truth with.
The concept of truth? A bit overblown don't you think? Because some guy can make a realistic looking fake videos that destroys the "concept" of truth? How?
Stalin had all the resources imaginables at his disposal.
Now Nancy, a tech-phobic waitress who has a grudge against her coworker can make up an entire scenario with one prompt and her colleagues might blindly believe her.
Let's not pretend they're the same thing.
Gen AI is inevitable. Watermarking is likely futile. But in my opinion it is still very important to discuss how, as a society, we're going to live in a post-truth world now that anybody can, IN SECONDS, not only fabricate a story but also spread it to thousands of people through their social media.
When that idea was originated, the advice was more like:
"Don't trust what you see on the Internet. Trust instead what you read in a reputable daily newspaper, or Peter Jennings or Tom Brokaw on the nightly news, or BBC World News."
Today, the Internet, especially the part which is not trustable, has nearly finished killing most of the "trustworthy" news sources, by outcompeting them for ad dollars - by being way better at targeting ads (e.g. Meta) and by scientifically perfecting addiction (e.g. TikTok). What remains is mostly controlled by governments and has far from a perfect record of being fact-based and impartial.[1] There are a ton of independent people out there in good faith posting facts on the Internet, but we just agreed that we shouldn't trust what we see on the Internet.
So doesn't this become "Don't trust anything"? And doesn't that, in practice, get implemented as "Don't trust anything that challenges what you believe to be true"? This feels like a really, really bad change to our society - and I'd argue it's already completely happened.
This isn't just ads, trust in the mainstream media, itself, is very low [1], deservedly so in my opinion. The continuous lies by omission, the outright incorrect headlines/articles that they edit after a day, the lock-step messaging, alignment, and avoidance of topics, pushed by their respective political parties/billionaire owners (6 companies own 90% of media [2]), made me switch to more independent journalists.
I have no objection to this -- I follow a few that I would say meet that definition well and which I trust. But boy do I worry that for 90% of the population, this translates to picking a bunch of enthusiastic propagandists whose bias is far worse than MSNBC, Fox News, or CNN ever were. I assume our craven and corrupt political parties will increasingly focus on propping up "independent journalists" who repeat their talking points for them.
Adequately implementing solves one problem (the making up a story because of a grudge), but creates a whole new set of likely much worse problems: how does one maintain a democracy / civil society? It's not just the trust of "social media" that you've eroded, you've almost certainly killed trust in traditional news reporting as well, especially considering just how much of traditional media is discovered via social media.
Effective democracy requires an informed voter base. Society requires its constituents to be invested in its continuity. Neither of those is achievable when we completely discard trust.
Yes, it's happened. Except a lot of people do have an exception - they'll trust the slop that reinforces their existing biases, or even if they know in their hearts it's not true, viewing their side's lies regularly still has an effect on the way they think.
Good point. Sometimes I wonder if social media, just almost every aspect of it, is the real cancer. Allowing just about anyone (globally) to anonymously deploy information warfare via the social media vector just seems bound to have horrible outcomes. It's just as bad with text as with images or video. Because of social media, we've trained at least 3 separate generations to self-sort into camps with customized ideological info sources that have incredibly-low standards for fact-checking and every incentive to tell their audience (1) exactly what they want and (2) whatever will enrage them most.
AI kind of makes this worse, but also only barely. Because most people really ought to know by now that almost any content could be AI, a video of, say, Trump kicking a baby or violating a goat wouldn't convince anyone that those acts happened (unless they already believed they happened).
Thing is, we're so flooded in biased BS, and no one has any incentive to produce non-sensational, factual news. I absolutely see 'post-truth' as the inevitability. You can't "weed a garden" when it is 100% weeds. The term "news" will cease to mean facts, and just become a branch of entertainment. Kind of the way "Reality TV" went from being supposedly a documentary (e.g. COPS) to just being a flavor of entertainment, where nothing needs to be real.
Stalin controlled the state. The state controls companies. Companies control watermarking.
This sort of solution to the fake image problem, makes it easier for stalin not harder. If everyone can make fake images that is one thing. If only the dictator can, well that is much worse.
Why are humans powerless to do anything about this? Aren't we making the technology? It's kind of a big problem for the future of the justice system and politics.
The watermarking should be on those things we want to verify as something that was not generated or manipulated. Something you'd add to, for instance, cameras. Putting them on the generated/manipulated is backwards as you can never get every model to watermark.
That model is equally bad though. Given that you're writing this in a discussion about gen AI watermarks, how in the world did you come up with the idea that Gen AI wouldn't be able to add a watermark?
Not that they "wouldn't be able".. that they wouldn't do it. For Gen AI watermarks to be useful all Gen AI systems need to add them and the incentives aren't there for that to happen. On the other hand the incentives are there for the non-generated sources to add it so they can differentiate themselves from the Gen AI media.
Perhaps not watermarks, but cameras could sign their pictures and put it in the extra data, that's not something that would be easy to add to fake pictures, or at least not the correct signature.
Maybe we do care about truth, freedom and privacy but the majority of rest of society will happily accept any T&Cs just to get access to whatever the next digital sliced pan is and as for truth and accountability, if they were two sides of the same coin on the ground people wouldn't bend down to pick it up as possesing it looks too much like responsibility and inconvenience.
I'm pretty sure watermarking is (or soon will be) a requirement for AI generated images in software used in the EU, as part of their regulations for AI transparency.
If i had a dollar for every time an American cried about literally any non-US jurisdiction having an iota of effect on them I could quit my job and leave this terrible website forever.
I disagree. Its mainly about having technical control and freedom. Reverse engineering how things work feels like peak hacker ethos. You don't have control of something if you can't remove it.
I think ethical considerations were always a bit secondary to technical power when it came to so called "hacker ethos".
After all, instructions on how to remove watermarks definitely feels like the sort of thing that would have been in phrack back in the day.
> Hackers believe that essential lessons can be learned about the systems—about the world—from taking things apart, seeing how they work, and using this knowledge to create new and more interesting things.
> Access to computers—and anything which might teach you something about the way the world works—should be unlimited and total.
> Mistrust authority—promote decentralization
> All information should be free
I phrased it a bit differently, and perhaps a little less sympathetically, but i think i was more or less saying the same thing.
In any case a tool like the article that strips watermarks seems exactly the sort of thing that would fit into what i quoted above. Its mistrusting authority - there is nothing more central authority then having a literal central authority adding hard to remove digital signatures to images. It promotes freedom of information - it supports explaining how watermarks work and what they are. Its fundamentally taking apart a system, which teaches us how the system works.
but I also live in a society that requires trust to function. making a tool the obliterates that trust(genAI imagery pipelines) then creating a tool that makes it trivial for normal people to remove any hint of controls over said trust eroding system is, toxic.
I get the argument about not putting in fingerprints that identify users, Good I agree. But this also removes the things that identify this as an AI image.
Now, what are the legitimate uses of that?
No really, why would I _need_ to remove a watermark for _legitimate_ purposes? Assuming that watermark is generic, rather than a fingerprint of a specific person
> No really, why would I _need_ to remove a watermark for _legitimate_ purposes?
When removing the watermark is easy, a very legitimate purpose of making the code to do it publicly available is to make a public demonstration that it's easy to do.
As for content use cases, suppose someone is using AI to modify their appearance because they're being unjustly targeted by an oppressive government. That government naturally bans doing that because they want to be able to identify and arrest their critics, so now if you make videos with your real face you get arrested but if you use a generated avatar then the watermark enables automated censorship because the government orders anything with the watermark to have its reach automatically restricted.
> suppose someone is using AI to modify their appearance because they're being unjustly targeted by an oppressive government
Then use a mask like everyone else. digital mask, one that obscures.
which is my main point, no, there isn't a legitimate need.
realtime avatars don't generally have invisible watermarks, also they are running from your machine, otherwise you've got a (normally credit card) trail to your front door. plus a video stream
also if you are generating stuff from a public provider, then tracing people isn't that hard to do.
As someone else pointed out: if watermarks are required, then everybody will assume an image without a watermark is the honest truth, which is obviously not true. Someone will end up in prison because of some image. This is bad.
but right now, we are eroding trust at an industrial scale.
There are no reliable tools for the end user, normal person, to work out if an image is AI or not. This erodes trust and lets bad actors get away with "oh thats AI generated" or use AI to defraud users.
But watermarks don't fix that because the bad actors could just use a model that doesn't include a watermark regardless of whether or not the ones that do can be removed. Foreign powers and monied interests were always going to have access to those and there are also already published local models that don't include them.
It's like making your image editing software watermark every image it edits in case someone photoshops a picture to show something that didn't happen. What's the point when anyone trying to fool people will always have access to ones that don't do that?
> Then use a mask like everyone else. digital mask, one that obscures.
It's not a matter of whether there are alternatives. If you can produce one without a watermark, and having the watermark allows the bad guys to cause trouble for you, then you have a legitimate reason to produce one without a watermark.
Wearing a mask also has different trade offs. A filter can change the shape of your face while continuing to allow you to show facial expressions.
> realtime avatars don't generally have invisible watermarks, also they are running from your machine, otherwise you've got a (normally credit card) trail to your front door.
"You don't have to remove the watermark because it didn't have one to begin with" isn't a way out. If anyone can create a video of themselves looking like $TARGET and saying despicable things without a watermark then you can't trust that a video without a watermark is real and the watermarks are pointless. Whereas if they're all supposed to have watermarks then you can't use the excuse that the video shouldn't have one to need removing.
> also if you are generating stuff from a public provider, then tracing people isn't that hard to do.
Only if the provider is subject to the jurisdiction/control of the user's oppressors.
The human ethos should be to never be misleading about the origin and truth of any content you create, forward, or pass on. If we care about honesty we should jail anyone who does so.
Even if you remove a watermark, the companies still have a record of which images they have generated and for whom. Even if you remove the obvious watermarks, all major image generators are using steganography to embed hidden information that you can't be sure were removed. This is a type of one-sided arms race where one player gets to be invisible if they want to.
Yes usually, since an important aspect of steganography is error correction. For example we know that SynthID is robust enough to survive resizing and small blurs.
Its what happens when people in power are paranoid dark-triad types and want to be able to catch anyone who threatens their power and stick it to them..
It already happened with Trump claiming any unfavorable content of his administration is "AI-generated" as a defense to dismiss real, unedited media. He literally said, “If something happens that’s really bad, maybe I’ll have to just blame AI.”
ie the video of garbage being thrown out the windows that his team already confirmed was real:
[0]: https://news.ycombinator.com/item?id=48200060