Moral Disagreement and the Limits of AI Value Alignment: a dual challenge of epistemic justification and political legitimacy
Nick Schuster & Daniel Kilov (2025)
One-Sentence Thesis
The authors argue that current approaches to AI value alignment, including crowdsourcing, reinforcement learning from human feedback, and constitutional AI, fail to accommodate reasonable moral disagreement, posing a challenge for AI safety.
Argument Outline
- 1Introduction to AI value alignment and its importance for AI safety
- 2Discussion of the challenge of accommodating reasonable moral disagreement in AI decision-making
- 3Critique of current approaches to AI value alignment, including crowdsourcing, reinforcement learning from human feedback, and constitutional AI
- 4Analysis of the need for epistemic justification and political legitimacy in AI decision-making
Key Distinctions
Epistemic justification vs. political legitimacy in AI decision-making
Reasonable moral disagreement vs. unreasonable moral disagreement
Key Terms
Value alignment
Epistemic justification
Political legitimacy
Flashcards
18 cardsRelated Questions
In Nick Schuster & Daniel Kilov's "Moral Disagreement and the Limits of AI Value Alignment: a dual challenge of epistemic justification and political legitimacy", Anthropic develops which of the following?
Which of the following does Nick Schuster & Daniel Kilov contrasts with in "Moral Disagreement and the Limits of AI Value Alignment: a dual challenge of epistemic justification and political legitimacy"?
In Nick Schuster & Daniel Kilov's "Moral Disagreement and the Limits of AI Value Alignment: a dual challenge of epistemic justification and political legitimacy", Casper and Davies et al. identifies which of the following?
In Nick Schuster & Daniel Kilov's "Moral Disagreement and the Limits of AI Value Alignment: a dual challenge of epistemic justification and political legitimacy", Wellman cited which of the following?