Analytical Techniques for Coordinating Decentralized Journalism

I gained a lot of insight about the problems with American journalism from a lot of different expert perspectives throughout the production of The Echo Chamber documentary.

The media's objectivity standards limit the framing of issues to a very narrow spectrum of perspectives.

One way to overcome this limitation is for journalists to use much more sophisticated analytical techniques that have been developed by the Central Intelligence Agency. These techniques take a more scientific approach for weighing the available evidence against as set of possible explanations for the behavior and motivations of political leaders. These techniques provide very effective techniques for detecting political deception, and would be very beneficial investigative and analytical tool for journalism.

By combing this type of standardized analytical framework with the Internet, then it will be easier to harness the collective wisdom of a decentralized force of volunteer citizen journalists. By collaborating with these civically-engaged citizens, then the press should be able to challenge deceptive partisan framing and defy traditional media logic by investigating complex issues.

WHERE JOURNALISTIC OBJECTIVY BREAKS DOWN
RELAYING THEORIES NOT FACTS
CONTRADICTING PARTISAN FRAMING
ADDING THE SCIENTIFIC METHOD TO JOURNALISM
ORGANIZING DECENTRALIZED COLLABORATION
DISPROVING INSTEAD OF PROVING HYPOTHESES
ACH OVERCOMES LIMITATIONS OF WIKIPEDIA'S NPOV PHILOSOPHY
ADVANCED ANALYTICAL TECHNIQUES FOR DECEPTION DETECTION

WHERE JOURNALISTIC OBJECTIVY BREAKS DOWN #
The Gold Standard for American journalism is the "He Said / She Said" objectivity standard, which means that the bounds of debate is often determined by the Republican and Democratic Parties. The media frame stories by gathering conflicting quotes from sources in the Executive branch, Congressional leadership, a critical mass of Congressional members or partisan think tanks.

Todd Gitlin, professor of journalism and sociology at Columbia University, gives his perspective on this:

"The notion that one has discharged one’s journalistic duty by reporting what two people have seen -- or say they’ve seen -- or want you to know they’ve seen -- is ridiculous."

Politicians are mostly interested in preserving their own political power -- as well as their political party's. There are many issues where the Democrats and Republicans agree in order to minimize any potential negative electoral impact for taking a controversial stand on an issue. What happens to debate there is political consensus?

Gitlin says,

"When Democrats and Republicans agree, and all factions within these parties agree or are silenced, then the debate collapses. I mean, from the point of view of the "He Said / She Said" transcriptive theory of journalism, there is no debate -- there is nothing to talk about. And the circle of legitimacy has been compressed so that only those who fall within the consensus -- that’s marked out by all of these -- by this chorus of agreement -- get a hearing."

Bill Plante says that the American media is not usually "argumentative about the processes of government." This means that journalists will rarely challenge partisan framing of issues even when the major political institutions fail to provide credible theories to explain what's happening in the real world.

I asked if this an anomaly that needs to be corrected, and Plante responded,

"What needs to be corrected, in my view, is the lazy reliance on a stream of facts, which are presented to the public every day. There needs to be some interpretation of those facts. There needs to be some checking of those facts -- reality check if you will."

RELAYING THEORIES NOT FACTS #
Gitlin builds off of Plante's insight by saying,

"Journalists in general would bridle at the thought that what they’re doing is relaying theories. They would insist that all they’re doing is excavating facts. But of course, the question is: Which facts is one attentive to? How does one stress them -- play them up, play them down -- juxtapose them to other facts... include or exclude, frame, etc.?"

Gitlin makes an important distinction by emphasizing that the subjective juxtaposition of facts creates an overall "theory" that attempts to explain a set of events.

These theories are usually the officially sanctioned narratives provided to the media by the Republican and Democratic parties. The traditional inverted-pyramid style of reporting places the most official sources at the top of the story, which gives a disproportionate weight to any partisan deceptions or half-truths.

CONTRADICTING PARTISAN FRAMING #
It is rare to find journalists like Knight Ridder's Jonathan Landay and Warren Strobel who are willing to go out on their own and provide alternative theories that challenge the status quo. Together they won the Raymond Clapper Memorial Award for their coverage of "the Bush administration's use of faulty intelligence in planning the war in Iraq." We interviewed them both for The Echo Chamber documentary.

Landay says,

"In trying to get to the bottom of the use of the intelligence for the justification of the war, that requires a lot of investigative techniques. And that means -- in a lot of ways -- you have to sort of be an intelligence analyst. A lot of the most important stories I’ve done haven’t relied on secret sources or leaked documents. There’s a lot out there in the public domain that merely needs to be scrubbed and read over."

His comparison to being an intelligence analyst struck me as an important insight. Landay's dedicated scrutiny of the public record has enabled him to scoop the New York Times on a number of different intelligence stories.

Landay says,

"I’ve gone back and re-read virtually every speech that Vice-President Cheney’s made, that the President has made, that Colin Powell has made, news conferences, white papers -- There’s a lot out there that you can do without having to develop secret sources and getting people to leak documents to you."

Landay and Strobel are using facts from the public domain to piece together a narrative that diverts from the political framing provided by either the Republican or Democratic Party. They are implementing a basic journalistic philosophy of thoroughness and verification, and they have been scooping other news organizations who have a lot more resources.

Why don't more journalists do this? For one, Knight Ridder has a healthy distance from the senior officials -- they prefer the better-informed sources from the fringes of the intelligence bureaucracy. But there also needs to be a more sophisticated analytical technique that leaves an audit trail and allows journalists to make subjective judgments based upon a comprehensive analysis of the facts.

Journalists should have the freedom to do just more than repeat what He Said and what She Said -- They should be able to add, "We have determined what He Said is correct and what She Said is wrong."

Fears of political bias and other structural limitations have prevented journalists from exploring other reasonable theories that are coming from non-partisan experts or even from citizens on the blogosphere.

Now that the Internet has eliminated the space constraints, then the evidence used to make subjective decisions of "right" and "wrong" can be presented online in a comprehensive way.

Luckily, the Central Intelligence agency has been doing this for years, and journalists can learn a lot from their techniques.

ADDING THE SCIENTIFIC METHOD TO JOURNALISM #
During my research for this documentary, I came across Richards J Heuer's book "The Psychology of Intelligence Analysis" posted online by the CIA's Center for the Study of Intelligence.

I ended up reading the entire book and using a lot of the analytical and critical thinking techniques throughout the process of interviewing 45 different journalistic and political experts.

Heuer developed a useful analytical method called "Analysis of Competing Hypotheses" that seems to provide a much more comprehensive journalistic framework than the existing "He Said / She Said" objectivity standard.

Chapter 8 of Heuer's book begins with the following explanation:

Analysis of competing hypotheses, sometimes abbreviated ACH, is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.

ACH is an eight-step procedure grounded in basic insights from cognitive psychology, decision analysis, and the scientific method. It is a surprisingly effective, proven process that helps analysts avoid common analytic pitfalls. Because of its thoroughness, it is particularly appropriate for controversial issues when analysts want to leave an audit trail to show what they considered and how they arrived at their judgment.

This CIA analytical technique sounds like a perfect addition to the existing journalistic tool kit. This picture gives a nice conceptual overview of ACH, which essentially involves creating a matrix of hypotheses to explain motivations of political behavior versus all of the available evidence.

ORGANIZING DECENTRALIZED COLLABORATION #
Time constraints, space constraints, lack of adequate training and available manpower has inevitably prevented the field of journalism from adopting the degree of analytical sophistication required for Heuer's Analysis of Competing Hypotheses method.

But with the software improvements to the Internet that have been advancing the communications possibilities and connectivity, it is becoming increasingly possible to take a harness the collective wisdom of a decentralization labor force. ACH is a scientific approach that could provide a mechanism to harness the analytic power sitting beneath The Long Tail of the blogosphere.

Blogs have primarily been a subjective medium where individuals have been able to provide their own editorial analysis, theories that explain current events as well as their personal perspectives and emotions. While important insights can come from the blogosphere, the failure to comprehensively analyze all of the facts has resulted in an ocean of partisan opinion journalism.

The intelligent critical thinkers who make convincing arguments have been the ones who have risen to the top. But overall, the signal-to-noise ratio in the political blogosphere is pretty low.

ACH provides an analytical philosophy that uses the scientific method to give structure and order to this decentralized wisdom.

DISPROVING INSTEAD OF PROVING HYPOTHESES #
The paradigm shift of ACH for the political blogosphere is that it is a process of refuting hypotheses -- not a process of gathering evidence that confirm your beliefs. As Heuer explains:

"The principal concern is that if analysts focus mainly on trying to confirm one hypothesis they think is probably true, they can easily be led astray by the fact that there is so much evidence to support their point of view. They fail to recognize that most of this evidence is also consistent with other explanations or conclusions, and that these other alternatives have not been refuted."

It is human nature is to run away from cognitive dissonance instead of challenging your belief system. The ACH technique requires a loosening of dominant political paradigms just as a scientist has to loosen his / her expectations of whether or not a particular experiment will verify or disconfirm his / her hypothesis.

A scientist can't ignore experimental evidence that disconfirms their hypothesis because their credibility would be damaged within their academic community. But nothing prevents people from choosing to believe political deceptions -- in fact, this type of behavior is encouraged by the two-party monopoly.

ACH "requires an analyst to explicitly identify all the reasonable alternatives and have them compete against each other for the analyst's favor, rather than evaluating their plausibility one at a time."

ACH contradicts the traditional mindset of the political blogosphere, which is usually goes something like I have seen the truth. Here's a set of facts that should convince you that I'm correct.

Online political discourse is more about your team winning and the other team losing than it is about actually understanding the viewpoints of the political opposition or being open their alternative explanations.

ACH OVERCOMES LIMITATIONS OF WIKIPEDIA'S NPOV PHILOSOPHY #
Wikipedia has successfully created a way to collaboratively produce encylopedia articles that measure cultural beliefs. But their Neutral Point of View collaboration philosophy fails to adequately settle controversial disputes. NPOV gives equal weight to partisan subjective beliefs -- even when a comprehensive set of facts can clearly disprove one side or the other.

From a journalistic perspective, NPOV lacks an effective mechanism to reduce uncertainty and to make judgments about the most convincing theory.

Heuer's Analysis of Competing Hypotheses would be a much better collaboration philosophy -- it creates a comprehensive audit trail that can convincingly discredit politically-manufactured beliefs.

A moderated wiki could be a very effective method for tapping into the collective wisdom for collecting data for an ACH matrix. A journalist could ask citizens to brainstorm a set of possible hypotheses that explains political behavior and motivations. Then volunteers could work on filling in the ACH matrix by collecting confirming and disconfirming evidence for each hypothesis.

ADVANCED ANALYTICAL TECHNIQUES FOR DECEPTION DETECTION #
Heuer's Analysis of Competing Hypotheses technique is a good first step for coordinating a decentralized force of volunteer fact checkers and amateur intelligence analysts. There has to be a learning curve of basic critical thinking skills and a paradigm shift of disproving hypotheses instead of finding confirming evidence.

Google helped me discover that there is actually a "counter-deception" improvement to Heuer's technique that is a bit more advanced and complicated. Counter-deception means "detecting or recognizing a deception," and it was developed for a national security context for detecting enemy deception, but it could just as easily be applied in a journalistic context for detecting political deception.

The Mitre Corporation has developed this ACH-CD technique, and they have an unclassified white paper that:

"considers counter-deception [1] from a psychological, rather than cultural perspective. First, we summarize the cognitive aspects of counter-deception. Next, we describe a process developed in the intelligence community called the Analysis of Competing Hypotheses (ACH). We describe how we correct ACH to account for cognitive factors that make people poor at detecting deception. We call this modified process ACH-CD."

Mitre indicates that Heuer's ACH technique is very popular within the world of intelligence analysis.

"Increasingly the major U.S. intelligence agencies have advocated ACH to enhance analysis and estimation. For example, the U.S. Central Intelligence Agency offers its analysts workshops on "Alternative Analysis" methods [DI2002]. The republished Heuer's The Psychology of Intelligence Analysis in 1999 and posted Heuer's book on the CIA's Internet website. Heuer's book is widely used in analyst training at the various national intelligence agencies. Morgan Jones, who learned ACH as a CIA analyst, featured the technique prominently in his book, The Thinker's Toolkit: Fourteen powerful techniques for problem solving, which is also cited widely in U.S. Intelligence agency analyst training [Jones 1998]." (hypertext link added)

This gives me more confidence that this modified ACH-CD technique is worth exploring as a journalistic technique. With the right software, then it could be implemented to organize and coordinate a volunteer force of decentralized citizen journalists.

ACH-CD makes a number of modifications to a comprehensive deception detection technique developed by Paul E. Johnson, Stefano Grazioli, Karim Jamal and R. Glen Berryman. Their technique is explained in this Cognitive Science article called "Detecting deception: adversarial problem solving in a low base-rate world."

Johnson et al's deception detection technique provides a taxonomy for deception tactics and suggests heuristics to counter each of the tactics. The overall fraud detection model involves four steps that I have modified to apply to a journalistic context:

1.) Activation: Generate expectations for political statements and behavior based upon previous statements and behavior. Inconsistencies are determined by a large magnitude between the expected and the observed behavior.
2.) Hypothesis Generation: Proposes hypotheses to explain observed inconsistencies -- either intentional manipulation and deception, unintentional errors, or insufficient disclosure.
3.) Hypothesis Evaluation: Assesses the viability of every hypothesis based upon the available confirming or disconfirming evidence.
4.) Global Assessment: Combines all acceptable hypotheses and produces a final diagnostic outcome.

Mitre modifies the four steps of Johnson et al.'s counter-deception technique to apply to a national security context, but I would imagine that these following theoretical modifications would also be applicable to a domestic political context.

*Whaley & Busby’s "Congruity Theory & Ombudsman Method" [WB2002] explicitly address the problem of Poor anomaly detection by identifying data collection techniques likely to surface anomalies related to denial or deception tactics.
*R. V. Jones’s "Theory of Spoof Unmasking" [Jones1978, 1989, 1993] can be adapted to address the problem of Misattribution, i.e., to avoid attributing inconsistent or anomalous events to collection gaps or processing errors rather than to denial or deception tactics...
*We adapted Heuer’s "Analysis of Competing Hypotheses" [ACH, Heuer1999] technique for counter-deception analysis to address the Failure to link deception tactics to deception hypotheses...
* Finally, Johnson et al.’s "Cognitive Model of Fraud and Deception Detection" [Johnson2001] organizes these four parts into a counter-deception business process (Figure 1) and addresses the problem of Inadequate support for deception hypotheses.

Sounds simple enough doesn't it? (UPDATE: More details on this here.)Granted this may be a bit too complicated to modify and implement all at once. Applying a more scientific approach to journalism is certainly a daunting task. I have some idea for how it could all come together, but I need a lot more input as I try to implement some of these ideas to The Echo Chamber documentary.

These specific deception detection techniques may not provide the bestphilosophical framework for decentralized, collaborative journalism. But they do provide hopeful improvements to this evolving media ecosystem.

Combining the insights of as many different academic disciplines as possible seems to be good approach to figuring this puzzle out. I certainly can't integrate and implement all of these theories by myself. But by tapping into collective wisdom facilitated by the Internet, then there is nothing preventing me from recruiting a team of people to help make this happen.