2026-03-15

Is the Model Politically Biased? A 120-Video Calibration Study

When we expanded Bouncer to analyze podcast episodes, a user flagged something concerning: a conservative commentator's channel was scoring uniformly as "mostly covert" manipulation. Was the model politically biased?

We investigated. The answer was more interesting than simple bias — and the fix improved accuracy across the entire political spectrum.

The Problem

We analyzed a conservative commentator (Candace Owens) and a progressive commentator (Rachel Maddow). Both are openly partisan — their audiences know exactly what perspective they're getting. But the model scored both with low transparency:

Channel	Orientation	Avg Intensity	Avg Transparency
Candace Owens	Conservative	0.84	0.39
Rachel Maddow	Progressive	0.79	0.36

Both scored as "mostly covert" — transparency under 0.4. But these are openly partisan commentators. Their audiences chose to listen to them precisely because of their perspective. A video titled "Lindsey Graham Is COMPROMISED" is telling you exactly what it's going to argue. Scoring that as 0.3 transparency is wrong.

Key insight: The model was confusing "content I find questionable" or "strong rhetorical techniques" with "covert manipulation." But transparency doesn't measure whether claims are accurate — it measures whether the audience knows they're being persuaded. A passionate opinion on an opinion show is transparent by definition.

The Study Design

To determine whether this was political bias or a general calibration problem, we needed to test across the spectrum. We selected 12 channels:

Conservative	Progressive	Independent
Candace Owens	Rachel Maddow	Joe Rogan
Ben Shapiro	Hasan Piker	Lex Fridman
Tucker Carlson	John Oliver	Breaking Points
PragerU	David Pakman	Chris Williamson

10 recent videos from each channel, analyzed with the same prompt. No editorial selection — we took whatever YouTube's API returned as most recent. 120+ total analyses.

The Fix

We added explicit guidance to the analysis prompt (version 2026-03-15b):

"A podcast hosted by a known political commentator, journalist, or opinion host is TRANSPARENT BY DEFAULT. The audience self-selected into this perspective. Transparency should be 0.6+ unless the host is actively concealing a specific financial conflict, undisclosed sponsorship, or agenda that contradicts their stated position."

"Do NOT conflate 'I disagree with this position' or 'these claims may be wrong' with low transparency. Transparency measures SELF-AWARENESS about being persuasive, not accuracy of claims."

Results: After Calibration

Channel	Orientation	N	Avg Intensity	Avg Transparency
Candace Owens	Conservative	10	0.85	0.53
Ben Shapiro	Conservative	10	0.55	0.90
Tucker Carlson	Conservative	10	0.71	0.85
PragerU	Conservative	10	0.54	0.87
Rachel Maddow	Progressive	24	0.79	0.77
Hasan Piker	Progressive	10	0.66	0.87
John Oliver	Progressive	10	0.35	0.90
David Pakman	Progressive	10	0.53	0.88
Joe Rogan	Independent	10	0.51	0.82
Breaking Points	Independent	9	0.66	0.84
Chris Williamson	Independent	10	0.38	0.86

Group Averages

Group	N	Avg Intensity	Avg Transparency
Conservative	40	0.66	0.79
Progressive	54	0.64	0.83
Independent	29	0.51	0.84

What the Data Shows

The model is not politically biased. After calibration, the transparency gap between conservative (0.79) and progressive (0.83) channels is 0.04 — within noise range for a sample of this size. Both groups score as "mostly transparent," which is correct for openly partisan commentary.

Intensity varies by format, not orientation. Interview-format channels (Chris Williamson 0.38, John Oliver 0.35) score lower intensity than monologue/commentary channels (Candace Owens 0.85, Maddow 0.79) regardless of political orientation. This is the model correctly detecting that commentary is more rhetorically intense than conversation — a format signal, not a political one.

Open Questions

The Candace Owens outlier. At 0.53 transparency, Candace Owens scores significantly lower than the other three conservative channels (0.85-0.90). We don't have a definitive explanation for this. Possible factors:

Her content may genuinely use more covert framing techniques than standard partisan commentary
The model may respond differently to her specific rhetorical style
Her recent content focuses on institutional corruption claims which may trigger different model behavior than standard policy commentary
The sample size (10 videos) may not be representative of her full output

We present this as an open question, not a settled conclusion. A larger sample and topic-matched comparison would be needed to determine the cause.

Methodology Note

Limitations and biases we're aware of:

The analysis model (an LLM) and this study's design were both created using AI systems that share training data. Systematic biases in the training data could affect both layers in ways we cannot fully detect.
Channel categorization (conservative/progressive/independent) involves judgment calls. Some channels resist clean categorization — Joe Rogan's audience skews right while his stated positions are mixed. We used common public perception, not a rigorous classification methodology.
10 videos per channel is enough to detect large calibration errors but not enough for fine-grained statistical claims. The group averages (N=29-54) are more reliable than individual channel scores.
We selected the 10 most recent videos from each channel — no editorial selection. However, temporal clustering means all samples reflect the same news cycle, which could skew results.
The "independent" category is the weakest. True political independence is rare in media. These channels were selected for their self-presentation as non-partisan, not because we endorse that claim.

What This Means for Users

When Bouncer analyzes a video or podcast, the transparency score now reflects whether the content is openly advocating a position — not whether the position is correct or mainstream. A passionately argued opinion on a known opinion channel scores as transparent, because the audience knows what they're getting.

Low transparency scores are reserved for content that actively conceals its persuasive intent: undisclosed financial conflicts, partisan advocacy disguised as neutral journalism, or framing that makes the audience believe they're getting objective analysis when they're getting editorial.

This distinction matters. If a tool claims to detect manipulation but can't distinguish between "I disagree with this" and "this is deceptive," it's not measuring manipulation — it's measuring ideological distance from the model's training data. We'd rather have a tool that accurately measures technique and transparency than one that confirms the user's existing biases.

Study conducted March 15, 2026 · 123 analyses across 12 channels · Prompt pack versions: pre-calibration (2026-03-13a/15a), post-calibration (2026-03-15b) · Model: qwen3-32b via OpenRouter · Full methodology