Why “Rotten Tomatoes” for News is a Terrible Idea

Mar 25, 2019

I just read an article about Credder, an organisation that wants to create an equivalent to “Rotten Tomatoes” for news. This a bad, bad idea. I can't stress this enough. Let me explain.


Flaw #1: Credibility (or lack thereof)


The first most glaring flaw that stands out to me is that Credder is a for-profit company that is not accountable to the public. If they manipulate ratings or have biases, there's almost no way for us to find out. And even if we do, what can we do? It's not like they're going to say in their TOS that their ratings are accurate. They'll have all sorts of legal notices and disclaimers protecting them. So what's to say that they won't mess with the system? Nothing. Sure, you could "trust" them, but look where trusting tech companies in the past has brought us. 

The fact that Credder is a for-profit which also aims to become a news platformi tself only further complicates the matter. Here's a quote from the aforementioned article:

...the startup wants to develop a sponsored-articles system in which an author writing on a specific field will be able to push a piece to a precise segment of readers. The platform will have to build an auction system that writers will use to reach the desired audience.

So... the ones with the most money get to reach a wide audience and spread their narrative? Does anyone else see the problem with that? This is definitely not a task for a for-profit company.

And here are a bunch of quotes from their TOS:



We grant you permission to use the Site subject to the restrictions in these Terms. Your use of the Site is at your own risk, including the risk that you might be exposed to Content that is offensive, indecent, inaccurate, objectionable, or otherwise inappropriate.

(emphasis mine on the second quote)

Now, this is, of course, expected. But if I company isn't willing to take the liability and say that all content on their platform is accurate, how can we trust them to rate the credibility of our news sources?

If you are taking on such a significant task and making a system which people from around the world are supposed to rely on, surely you're responsible for making sure that such ratings are accurate and actually reflect upon the publication/publisher which you're judging? How can you say "Here, this rating is to tell you whether or not a news publication is trustworthy." and at the same time say that... your rating itself may not be trustworthy? 


Flaw #2: There's not much that you can rate about news


News should ideally by as objective as possible. The only thing you can possibly do is fact-check it (and not even that in some cases - more on this later). But Credder doesn't say it's a fact checking service. It says it's a news "rating" service. But what is there to rate besides the accuracy of a news piece?

Here's a quote from their about page:

Credder doesn't tell you who to trust. In the same way, Yelp doesn't tell you where to eat and Rotten Tomatoes doesn't tell you what to watch. It is Credder's job to gather news consumers' feedback, hold the articles, authors, and outlets accountable with ratings, and try to be a delightful place to get your news.

In the aforementioned article it is even stated that the Credder team drew its inspiration from Rotten Tomatoes and its “Tomatometer”. But the comparison with Yelp or Rotten Tomatoes is... pointless. Whether or not food tastes good is highly subjective. Movies even more so. You can like or dislike a dish or a movie. But with news, it doesn't matter how you feel about it, all that matters is whether or not it is true.

Credder says it takes into account "news consumers' feedback" but what would they possibly say about a news piece? If they don't like it, if it's against the values they stand for, or if they feel threatened by it, they'll simple give it a poor "rating". If they like it or if it makes them happy, they'll give it a good one. That's the truth. They won't care whether or not the news is factual. But as I said previously, news cannot be "rated" or judged in accordance with how it makes you feel.

They say that they "hold the articles, authors, and outlets accountable with ratings", but these ratings will almost certainly always be subjective, and thus meaningless.


Flaw #3: Journalists aren't all that great either


From the article:

Like in RT, reviewers will be split between professionals and the general public.

But it's not unheard of for journalists to take sides, be biased, or be proponents and spreaders of false news. This is just one of many examples, but right-wing journalists in the US will always rate left-wing news agencies and their work poorly, and vice-versa. If you think otherwise, you're clearly not in touch with reality.

Only one of them spreads false information on a large scale, but the ratings of both will suffer. That's just how it is. Having journalists rate other journalists just as poor filter for quality as having the public rate them.

Now, you might (reasonably) ask why we can't just exclude right-wing news outlets from the rating process. Well, as the article correctly points out:

The diversity of the reviewers is also a big issue. Theoretically, it should represent a cross-section of the population.

You can't just exclude reviewers from one side of the political spectrum. That in itself would make the system partial and biased. And keep in mind, politics is only one of many issues where this applies.

I'd say that only a fraction of outlets and journalist have to be partisan for the system to fail. Credder claims that they evaluate the overall "rating" of a news piece after just taking 7 reviews. What if, instead of judging the news piece honestly, even two of those say that the news article is not trustworthy because it is not accordance with their personal views and doesn't validate these pre-existing notions? That comes out to trusted by around 71%. Would you want to read a news article that's apparently trusted by only 71% of journalists? Of course not. Journalism must be held to a higher standard.


Flaw #4: The system in itself is based on fallacious and erroneous assumptions


The same ones that I keep bringing up: that news is subjective, or that anyone will rate it properly. You can't trust humans to rate anything fairly, much less news.

Just take a look at their "notation system":

Do you see what the first thing they ask is what someone "thinks" of an article? How does it matter what I or anyone else thinks of it? ALL THAT MATTERS IS ITS ACCURACY.

Can you see anyone from one side of the political spectrum rate an article which says something bad about the parties they support fairly? Of course not. They'll say it's anything but trustworthy - illogical or biased or filled with mistakes or non-factual. They'll say it's sensationalist or satirical. They'll say that it's not credible, or that it lacks reliable sources. Now look at the screenshot again. Do you see the problem? This will happen with every topic out there that's even slightly controversial. It does not matter how many journalists or people are part of this system or how experienced they are. This problem won't go away.


Flaw #5: There's stuff that you can't even fact check, let alone rate


Here's some more food for thought. What if the news source is protecting a confidential source or insider? Of course they won't have reliable sources which they can disclose. Of course they can't be fact checked. Does that mean that they shouldn't publish such news?

What if it's an opinion piece? Opinion pieces aren't news, but it's widely accepted that news sources publish them. And for a reason. Opinion pieces can be important to inform public opinion. It can expose them to the thinking of experts and specialists in a field. But these are largely subjective. So you can't trust another human to honestly and truly "rate" these either.

When the New York Times first started publishing op-eds in 1970, this is what they said about them:

The objective is rather to afford greater opportunity for exploration of issues and presentation of new insights and new ideas by writers and thinkers who have no institutional connection with The Times and whose views will very frequently be completely divergent from our own.

Clearly, they are important, and can't just be done away with. But how do you rate an opinion piece?


Flaw #6: Humans are the weakest link


Here's another quote from the article:

The key question to how to prime the pump: growing an initial base of casual users, recruiting certified journalists and incentivizing them to review articles. Credder will rely on traffic sent by publishers page and platforms who should be happy to insert a badge in exchange on data relevant to their production. “Pros” reviewers will benefit from a boost of their social footprint for each contribution.

What's to say that this won't lead to an underground industry of paying off these "pro" reviewers to favorably publish and/or rate articles that portray a person or organisation in better light? What's to say that they can't be blackmailed if a news agency can go as far as to apparently blackmail the richest person in the world? There will undoubtedly be many determined to game the system, and you can't fight them all.

All in all, while what Credder is trying to do is admirable, in the end it's just a very unpractical and flawed system.

This is not to say, however, that something like AI would do a better job of "rating" news articles. It would probably do even worse. And it's not just that Credder's plan has major flaws - this is quite simply something that cannot be done. You just can't "rate" news organisations and pieces - at least, not in the manner in which Credder is approaching the problem.

You can't trust the public or other journalists to fairly and honestly judge a news source in an unbiased manner. However, I believe that a non-profit, audited, transparent and independent organisation (something along the lines of Media Bias/Fact Check, Politifact or Snopes) can fact-check news sources and measure their credibility in a manner which helps the general public.