Cover of Ernest HemingwayThe term crap detector was probably coined by Ernest Hemmingway, in an interview with The Atlantic magazine. The complete quote was:"To invent out of knowledge means to produce inventions that are true. Every man should have a built-in automatic crap detector operating inside him. It also should have a manual drill and a crank handle in case the machine breaks down."Howard Rheingold defined crap as "information tainted by ignorance, inept communication, or deliberate deception".
-- http://www.theatlantic.com/past/docs/issues/65aug/6508manning.htm
Pure spam content on the Internet have been pretty well filtered by Google and such. However, the spammers have gone to a different strategy: search engine optimization (i.e. how to spam the search results), and content farms, where low-quality articles are churned out by the dozens through automatic rewriters to get higher on search results.
It had gotten so bad that Google has essentially declared war on those content farms full of crap. However, that puts their objectivity at doubt. And they appear to be losing the battle, at least for now. Over 1/3 of Lifehacker readers voted that spammers have made significant foothold into Google's search results.
Howard RheingoldImage via WikipediaSo what can you do? If Google can't filter out crap for you, you must do it yourself. There are two ways... Through word of mouth, such as WOT (Web of Trust), SideWiki, and other comment systems, and through your own crap detector.
I've found multiple crap detectors online, including CRAP Test, Howard Rheingold's Crap Detection 101, Scott Berkun's Bull**** Test, and the FactChecked.ORG checklist. None of them seem to be very comprehensive, or specifically designed to check for crap online.
Here's my version specifically for Internet Research, inspired by those tests. I'll just call it "Kasey's Crap Test" (inspired by all the other Crap Tests above)
- Bias: Do you have a pre-existing bias regarding the hypothesis?
- Purpose: What is the purpose of the hypothesis? What is the ultimate goal?
- Evidence: What supports the hypothesis?
- Facts as evidence
- Fact or hypothesis?
- Are the facts current?
- Are the facts reliable?
- Testimony as evidence
- Is the testimony author qualified?
- Is the testimony author reliable?
- Is the testimony current?
- Logic as evidence
- Counter-Argument: What about the other side? Was it studied at all?
BIAS
Do you have a personal bias toward the topic / hypothesis? Do you know why?
Just by reading the information (or hypothesis), without reading any of the evidence, are you inclined to believe (or disbelieve) the hypothesis? If you do, this indicates you have some sort of a bias toward a particular side. Do you know why? Now that you are aware of the bias, can you keep an open mind?
Skepticism is not bias. Skepticism is "Really? Show me some proof either way." Bias is is tendency toward either Yes! or No!
PURPOSE
The author must have a purpose of writing an article or even a blog entry. Is the purpose simply to inform, to recruit you, to sell you something, or to sway your opinion? Is it the same as the hypothesis, or is there an ulterior motive?
Sales articles generally have separate hypothesis and purpose. The hypothesis is "Item X is good for you", and purpose is "Buy Item X".
Search the Beginning and the End
Sometimes, the answer is in the title, but sometimes, the purpose is hidden at the end. I would check the title, and the last paragraph, to see if the message is consistent, or is there an alternate purpose to the article. For example, some scam promoters write an article proclaiming their scam is NOT really a scam (i.e. "inform"), but the real purpose of the article is to actually recruit you into the scam.
Or even worse, the article's title can be "Truth about ______ Scam", but in the end is a recruiting link into that very scam.
Is there a way to give some feedback?
Authors who are interested in genuine exchange of information would welcome feedback. The fake ones and the closed-mind ones would have closed off comments on their blog and not even leave a "feedback" email address. Very few "sales articles" accept feedback at all.
Whether they accept feedback / comment gives you an idea whether they have looked for evidence and then reached an conclusion, or did they have a conclusion to start, then looked for evidence to support it, and is not interested in any feedback. The former is science. The latter is pseudo-science.
EVIDENCE
As Scott Berkun puts it very succintly, "How do you know what you know?" There are generally three types of supporting evidence: fact, testimony, or logic.
Facts as Evidence
Are the alleged facts actually facts or just more hypotheses? Do they actually support the hypothesis? Are the facts current? Are the facts reliable?
- Facts or Conclusions?
Are the "factual evidence" actually facts, or conclusions? What someone quotes as "fact" may be an a hypothesis or an observation, and it's worthless if you don't know the evidence that supports the hypothesis or lead to the conclusion. For example, if you see this headline
Then you see the one below.. Which is correct?
Actually, they both are correct. Both are referring to different information. The first headline was referring to ComScore stat that Android has surpassed Apple in US Smartphone market at the end of 2010. The second link was citing much older data, released by Nielsen in June 2010.
If you only look at the headlines you would never have known that. Both are observations, not facts. The actual data are the facts. The observation has a bias. The same facts may spawn multiple conclusions depending on the bias.
For example, both headlines appeared in January 2011, based on the same set of data: ComScore smartphone market share based on 3-month period ending November 2010. Both are observations, but one is very obviously biased.
Android Beats Apple in Market Share — LA Times
Blackberry still dominates Enterprise Market — BerryTimes
For example, both headlines appeared in January 2011, based on the same set of data: ComScore smartphone market share based on 3-month period ending November 2010. Both are observations, but one is very obviously biased.
Android Beats Apple in Market Share — LA Times
Blackberry still dominates Enterprise Market — BerryTimes
- Are the facts "current" or "timely"?
If it's research, is the research current? Did the research occurred many years ago, and new / better results became available? If there are differences between the results, have the differences been adequately explained? Have the research been repeated by a third-party?
A lot of "crap" on the Internet are undated stuff archived in "content farms" that were outdated almost as soon as they are published, but they still clog up the search results. You have to look for the date and other time indicators to determine the "freshness" of the data. The data may have changed significantly in the meanwhile.
In the two headlines above, as you can see, a lot of data changed in 6 months.
- Are the facts "reliable"?
Who did the research? Does the researcher(s) have anything to gain or lose from how the results turn out? Is the raw data available instead of the conclusions? (Remember, conclusion is an opinion/hypothesis, not fact)
Andrew Wakefield, the infamous "autism doctor" who suggested that MPR vaccine may have contributed to autism, was revealed to have received a lot of money from a lawyer group who wants to sue makers of MPR vaccines and was looking for "evidence".
It is best if the facts came from several different independently reliable sources. It is far less likely for data to be "faked" if they came from multiple reliable sources, or even multiple unknown but unrelated sources.
A newspaper is generally reliable, as they have a reputation to maintain. A world-renowned paper (New York Times, Wall Street Journal, etc.) even better.
Testimony as Evidence
Who is making the testimony? Does that person have the proper credentials to make such a conclusion? Do the testimony actually support the hypothesis? Has the person changed his mind since then? All testimony is tinged with bias.
- Is the Testimony Author Qualified?
If a lawyer claims that a certain medical supplement is 100% effective in helping weight loss, would you believe him? Probably not. He's not a medical professional and thus is not qualified in giving medical advice.
If a regular Joe off the street claims a certain "income opportunity" is 100% legal, and it had made him money so it cannot be a scam, would you believe him? Probably not. He's not a lawyer. What if it was your friend or relative telling you about this "income opportunity" instead?
Don't confuse this with 'ad hominem' attack. This only applies to TESTIMONIAL evidence, or in other words, an opinion. In other words, "(Based on my expertise) (I believe) this is good/bad." It sounds like a fact, but it is an opinion, based on his/her expertise / experience.
- Is the Testimonial Author Reliable?
Andrew Wakefield, the infamous "autism doc" had destroyed his credibility when it was revealed that a group of lawyers paid him a lot of money looking for evidence to sue makers of MPR vaccines.
Is his or her testimony corroborated by other unrelated people? Or contradicted? Why?
- Has the Testimonial author changed his mind since then?
If a person allegedly made a testimonial, check that person to make sure s/he had not changed his/her mind since.
Testimonials can be falsely attributed. Nobel Laureates have sued Mannatech for allowing associates to use their names to promote Mannatech products, implying some sort of endorsement.
Logic as Evidence
Logic can be used to support a hypothesis. Most people are familiar with logical deduction, or deductive reasoning. It can be written as follows:
For example:
However, what appears to be logic may in fact be logical fallacies. Here's one example:
Someone said "I took X, and I lost weight. Therefore X helps weight loss". Is that a logical statement? No, it is not. There is no proof that X is a significant factor in the weight loss unless there are some sort of scientific test(s) that ruled out all other factors. A single test can have flaws or bias. The tests should be repeated by disinterested third-parties that are not biased toward any one.
This is known as a "hasty generalization", trying to infer a generalization ("X helps weight loss") from a single case "I took X, and I lost weight" that may not be typical. http://www.logicalfallacies.info/presumption/hasty-generalisation/
There are three general types of logical fallacies:
Fallacy of Relevance -- trying to argue that something was related when it is not related
One type of "fallacy of relevance" is "bandwagon fallacy". An example would be:
Fallacy of Ambiguity -- trying to confuse the issue with word play or ambiguous interpretation
One type of Fallacy of Ambiguity is the "strawman fallacy", with the following example from Wikipedia
Fallacy of Presumption -- trying to presume something and use that as part of the argument, but that presumption itself is not proven
One type of fallacy of presumption is "Arguing from ignorance", also known as "appeal to ignorance" or "proving a negative". Here is an example:
For more about logical fallacies, see http://www.logicalfallacies.info/
COUNTER-ARGUMENT
Did the author study the opposite side of the hypothesis s/he is trying to prove or disprove? Did he acknowledge any objections, observations, facts, or such that support the other side? There is always more than one side of the story.
A sales article or one-sided heavily-biased article would only mention evidence that support one side of the conclusion. A more balanced article would mention some opposing evidence, but explain them away. How they are explained away is important, as it reveals how the author thinks, and whether he has blind spots or bias toward a particular conclusion.
Often a promoter of a scam will invent all sorts of explanations on why the scam is NOT a scam, usually by employing logical fallacies to dismiss legitimate concerns raised by critics.
CRAP DETECTION EXAMPLE ONE
Just for grins, I typed in "Idol White", and picked the first result to analyze it for "crap".
http://www.squidoo.com/idol-white-free-trial
Instead of going through line by line, I'll just list the factors I found.
Bias: if it's FREE and I want Whiter Teeth...
Purpose: Please try Idol White through my link! Definitely an ad.
Facts: None. Just personal testimonials, with not even a personal before/after pictures. The author appears to be biased.
Testimony: None. Author revealed NOTHING about him/herself except used Idol White. Not a dental hygienist or dentist or otherwise any sort of expert on teeth whitening or dental help. Reliability is zero. In fact, a study of the official Idol White website reveals that the links forwards to Idol White with an affiliate code.
Logic: None, but several logical fallacies, such as "great smile helps you attract opposite sex, so you should get Idol White".
Counter-argument: Nothing.
Thus, this article is pure "crap".
CONCLUSION
Crap Detection is not simple, but it is mainly tedious. Google and other search engines makes information access simple, but it is up to you to employ them to help your crap detector.
Testimonials can be falsely attributed. Nobel Laureates have sued Mannatech for allowing associates to use their names to promote Mannatech products, implying some sort of endorsement.
Logic as Evidence
Logic can be used to support a hypothesis. Most people are familiar with logical deduction, or deductive reasoning. It can be written as follows:
A) something; and
B) related something else
therefore
C) conclusion
For example:
A) All men are mortal. B) Socrates is a man. Therefore C) Socrates is mortal.Both A) and related fact B) must be true to make C) actually true.
However, what appears to be logic may in fact be logical fallacies. Here's one example:
A) Whenever it rains, you get wet. B) You are now wet.Therefore C) it is raining.Here, A) is not a properly written fact. While it is true, there are more than one reason for one to get wet instead of just rain. This is called "logical abduction". Basically, this is deductive reasoning applied BACKWARDS (and is therefore wrong.)
Someone said "I took X, and I lost weight. Therefore X helps weight loss". Is that a logical statement? No, it is not. There is no proof that X is a significant factor in the weight loss unless there are some sort of scientific test(s) that ruled out all other factors. A single test can have flaws or bias. The tests should be repeated by disinterested third-parties that are not biased toward any one.
This is known as a "hasty generalization", trying to infer a generalization ("X helps weight loss") from a single case "I took X, and I lost weight" that may not be typical. http://www.logicalfallacies.info/presumption/hasty-generalisation/
There are three general types of logical fallacies:
- fallacy of relevance
- fallacy of equivocation
- fallacy of presumption
Fallacy of Relevance -- trying to argue that something was related when it is not related
One type of "fallacy of relevance" is "bandwagon fallacy". An example would be:
"Ten million owners can't be wrong! Buy Product Y today!" (Implies Product Y is good)Whether Product Y is popular is not related to whether it is "good" or not.
Fallacy of Ambiguity -- trying to confuse the issue with word play or ambiguous interpretation
One type of Fallacy of Ambiguity is the "strawman fallacy", with the following example from Wikipedia
- Person A: Our society should spend more money helping the poor.
- Person B: Studies show that handouts don't work; they just create more poverty and humiliate the recipients. That money could be better spent.
Fallacy of Presumption -- trying to presume something and use that as part of the argument, but that presumption itself is not proven
One type of fallacy of presumption is "Arguing from ignorance", also known as "appeal to ignorance" or "proving a negative". Here is an example:
"If you can't prove that God does not exist, then God must exist."This is conclusion presumes that there are only two choice: God must either exist, or not exist, when in fact, God can be unknowable, or simply unknown. So there are at least two other choices. This tactic is often used by debaters to shift the burden of proof onto the opponent.
For more about logical fallacies, see http://www.logicalfallacies.info/
COUNTER-ARGUMENT
Did the author study the opposite side of the hypothesis s/he is trying to prove or disprove? Did he acknowledge any objections, observations, facts, or such that support the other side? There is always more than one side of the story.
A sales article or one-sided heavily-biased article would only mention evidence that support one side of the conclusion. A more balanced article would mention some opposing evidence, but explain them away. How they are explained away is important, as it reveals how the author thinks, and whether he has blind spots or bias toward a particular conclusion.
Often a promoter of a scam will invent all sorts of explanations on why the scam is NOT a scam, usually by employing logical fallacies to dismiss legitimate concerns raised by critics.
CRAP DETECTION EXAMPLE ONE
Just for grins, I typed in "Idol White", and picked the first result to analyze it for "crap".
http://www.squidoo.com/idol-white-free-trial
Instead of going through line by line, I'll just list the factors I found.
Bias: if it's FREE and I want Whiter Teeth...
Purpose: Please try Idol White through my link! Definitely an ad.
Facts: None. Just personal testimonials, with not even a personal before/after pictures. The author appears to be biased.
Testimony: None. Author revealed NOTHING about him/herself except used Idol White. Not a dental hygienist or dentist or otherwise any sort of expert on teeth whitening or dental help. Reliability is zero. In fact, a study of the official Idol White website reveals that the links forwards to Idol White with an affiliate code.
Logic: None, but several logical fallacies, such as "great smile helps you attract opposite sex, so you should get Idol White".
Counter-argument: Nothing.
Thus, this article is pure "crap".
CONCLUSION
Crap Detection is not simple, but it is mainly tedious. Google and other search engines makes information access simple, but it is up to you to employ them to help your crap detector.
No comments:
Post a Comment