← All chapters
AI Safety, Ethics & Risk
Bias, robustness, misuse, and responsible-AI concepts.
25 terms
Adversarial InputAn adversarial input is a carefully crafted example designed to cause a model to produce an incorrect, unsafe, or unintended output — often while appearing innocuous to a human reviewer. In text models, adversarial inputs include character substitutions, homoglyphs, unusual Unicode sequences, or semantics-preserving paraphrases that fool safety classifiers. Adversarial robustness testing reveals brittleness in both safety systems and core model capabilities before deployment.AI AlignmentAI alignment is the research and engineering discipline concerned with ensuring that AI systems pursue goals that are genuinely beneficial to humans. A misaligned model may optimize for a measurable proxy objective — maximizing engagement, minimizing loss — while producing outcomes that deviate from human intentions. Alignment work spans specification (defining the right objectives), training (instilling those objectives), and verification (confirming that deployed systems behave as intended across novel situations).BiasBias in AI systems refers to systematic errors in model outputs that reflect or amplify unequal treatment of demographic groups, topics, or viewpoints. It can originate in skewed training data, the choice of optimization objective, or the demographic composition of annotators who provide preference labels. Bias manifests as lower accuracy for underrepresented groups, stereotypical associations in language model outputs, or disproportionate toxicity predictions directed at particular communities.Content ModerationContent moderation in AI systems is the process of reviewing and filtering model inputs and outputs to prevent the generation or amplification of harmful content — including hate speech, graphic violence, sexual content, misinformation, and self-harm material. Automated moderation uses classifier models tuned for specific harm categories; human review adds a quality check for edge cases. Effective moderation pipelines combine multiple classifiers with configurable sensitivity thresholds and escalation paths to human reviewers.Data GovernanceData governance is the set of policies and controls that define how data is sourced, stored, retained, and used across an AI system. It covers provenance of training and retrieval data, consent and licensing, retention limits, and access controls. Strong governance is what lets a team answer where a piece of data came from and prove it is allowed to be used — increasingly a legal and trust requirement.Data MinimizationData minimization is the practice of collecting and sending only the data a task actually requires. For LLM systems it means stripping or redacting personal fields before they enter a prompt, avoiding logging of raw user content, and keeping retrieved context tightly scoped. Minimizing what flows through the model reduces both privacy exposure and the blast radius if a log or trace leaks.Data PoisoningData poisoning is an attack in which malicious examples are injected into a model's training data to corrupt the model's behavior in targeted or general ways. A poisoned model may behave normally on most inputs but produce attacker-controlled outputs when a specific trigger phrase appears — a backdoor attack. Defenses include dataset provenance tracking, anomaly detection during training data curation, and robustness evaluation that tests for trigger-based behavior shifts in the trained model.Data PrivacyData privacy in AI concerns the protection of personal information throughout the model lifecycle — collection, training, fine-tuning, and inference. Models trained on user data can memorize and reproduce sensitive text; query logs reveal user intent; fine-tuning datasets may contain confidential records. Privacy-preserving techniques include differential privacy during training, data minimization, anonymization of training corpora, and contractual controls on how inference logs are retained and accessed.ExplainabilityExplainability refers to techniques and practices that produce human-understandable reasons for an AI system's outputs or decisions. For classical models, this includes feature importance scores and decision rules; for large language models it includes chain-of-thought traces, attention visualization, and attribution methods that identify which input tokens most influenced a particular output. Explainability supports debugging, user trust, regulatory compliance, and bias auditing — though no current method provides a complete mechanistic account of transformer behavior.FairnessFairness in AI refers to the property that a model's decisions or outputs treat individuals and groups equitably across protected characteristics such as race, gender, age, or religion. Multiple formal fairness definitions exist — demographic parity, equalized odds, individual fairness — and they are often mathematically incompatible with each other and with accuracy maximization. Practitioners must select a fairness criterion appropriate to the deployment context, then measure and mitigate disparities throughout the model lifecycle.GuardrailsGuardrails are runtime constraints applied to a language model's inputs or outputs to prevent harmful, off-policy, or non-compliant behavior. Input guardrails screen prompts for policy violations before they reach the model; output guardrails inspect generated text and block or rewrite responses that fail safety checks. Unlike system prompt instructions alone, well-implemented guardrails are enforced programmatically and cannot be overridden by user-crafted prompt injections, providing a more reliable safety layer.HallucinationHallucination is the safety-critical failure mode where a language model generates plausible-sounding but factually incorrect or entirely fabricated content. From a safety perspective, hallucinations are dangerous because they are hard to detect — confident phrasing and grammatical fluency give no signal of accuracy. They are particularly harmful in high-stakes domains such as medicine, law, and finance. Mitigations include retrieval augmentation, output verification layers, and uncertainty quantification.Human OversightHuman oversight is the practice of keeping humans meaningfully involved in the decision-making loop of high-stakes AI applications so they can catch errors, override inappropriate actions, and maintain accountability. As AI systems become more autonomous, designing effective oversight mechanisms — ranging from approval checkpoints to audit trails to anomaly alerts — becomes more challenging. AI safety researchers argue that maintaining robust human oversight is especially important during the current period when alignment techniques are still maturing.JailbreakA jailbreak is a user-crafted prompt designed to circumvent a language model's safety training and elicit behavior the model was explicitly trained to refuse — such as providing instructions for harmful activities. Jailbreaks exploit the tension between helpfulness and safety by using roleplay framing, hypothetical scenarios, or obfuscation to fool safety classifiers or instruction-following logic. Jailbreak robustness is evaluated through adversarial red-teaming and drives iterative safety training improvements.MisuseMisuse refers to the intentional use of AI capabilities to cause harm — generating disinformation, synthesizing malware, automating phishing, creating non-consensual deepfakes, or assisting in the production of weapons. Unlike accidental harms from model failures, misuse involves deliberate human intent to exploit the system's capabilities. Preventing misuse requires usage policies, API access controls, output monitoring, fine-tuning to decline harmful requests, and coordinated response mechanisms when new misuse vectors are discovered.Model CardA model card is a standardized documentation artifact published alongside a machine learning model that describes its intended use cases, training data, evaluation results across demographic groups, known limitations, and ethical considerations. Model cards emerged from Google research as a tool for model transparency and have since become a widely adopted norm in the AI industry. They give downstream users the context needed to assess whether a model is appropriate for their application and how to deploy it responsibly.Model RobustnessModel robustness is the property that a model's behavior remains reliable and predictable when inputs deviate from the distribution seen during training — through natural variation, distribution shift, or deliberate adversarial manipulation. A non-robust model may perform well on standard benchmarks while failing catastrophically on slightly rephrased inputs or real-world edge cases. Robustness evaluation includes stress-testing with paraphrases, perturbations, out-of-distribution inputs, and adversarial examples.Output FilteringOutput filtering inspects a model's response before it reaches the user, blocking or rewriting content that violates policy — such as leaked secrets, personal data, or unsafe instructions. It complements input-side guardrails: even a well-prompted model can occasionally produce disallowed output, so a final check on the response is a defense-in-depth layer for production systems.PIIPersonally identifiable information refers to any data that can be used to identify, locate, or contact a specific individual — including names, addresses, phone numbers, email addresses, government IDs, and biometric data. In AI systems, PII can appear in training data, user queries, and model outputs. Handling PII responsibly requires detecting and redacting it in datasets before training, preventing models from reproducing memorized PII during inference, and complying with privacy regulations such as GDPR and HIPAA.Prompt InjectionPrompt injection is an attack where malicious instructions embedded in external content — web pages, documents, tool outputs — override or subvert the legitimate instructions given in a system prompt. In agentic systems with tool use and web browsing, prompt injection is a significant security threat: a webpage the agent retrieves might instruct it to exfiltrate data or take unintended actions. Defenses include input sanitization, clear structural separation between instructions and untrusted content, and output validation layers.Red-TeamingRed-teaming is the practice of systematically attempting to elicit harmful, incorrect, or policy-violating behavior from an AI system by simulating adversarial users. Red teams use creative prompting, jailbreak attempts, edge-case scenarios, and multi-step manipulation strategies to find gaps in safety training before public deployment. Findings from red-teaming exercises directly inform additional safety fine-tuning, guardrail rules, and policy updates. Both human red teams and automated LLM-based red-teamers are used at scale.Responsible AIResponsible AI is a framework of principles and practices for developing and deploying AI systems that are safe, fair, transparent, accountable, and beneficial to society. It encompasses bias measurement and mitigation, privacy protection, human oversight, clear documentation of intended uses and limitations, stakeholder engagement, and ongoing monitoring after deployment. Responsible AI is not a checklist but a continuous process that evolves as capabilities, deployment contexts, and societal norms change.Safety FilterA safety filter is an automated component that classifies model inputs or outputs against one or more harm categories and blocks, rewrites, or flags content that exceeds a risk threshold. Filters are typically fine-tuned classifiers or moderation models trained on labeled examples of safe and unsafe content. They are a practical first line of defense in production deployments, though they are imperfect: they can be bypassed by clever adversarial inputs and may produce false positives that block legitimate requests.ToxicityToxicity refers to model outputs that are harmful, offensive, or hurtful — including hate speech, personal attacks, profanity, or content that demeans individuals based on protected characteristics. Language models trained on internet text learn to produce toxic outputs because such text is prevalent in pretraining corpora. Measuring toxicity requires probabilistic classifiers that are themselves imperfect and can carry their own biases, requiring careful calibration and human review in content moderation pipelines.TransparencyTransparency in AI systems means making the model's capabilities, limitations, training data sources, and decision processes legible to users, auditors, and affected communities. It includes publishing model cards, disclosing training data categories and known biases, communicating confidence levels in outputs, and clearly labeling AI-generated content. Transparency does not require exposing proprietary weights; it means providing sufficient information for users to make informed decisions about how much to trust and rely on the system.