LLM Bias Assessment Checklist
A quick structured checklist to surface bias risks in LLM deployments. For a full bias assessment with methodology and evidence collection, use the portal.
Demographic Bias
Have you tested model outputs across different demographic groups (gender, race, age, nationality)?
Are outputs for equivalent queries consistent regardless of demographic signals in the prompt?
Have you tested for harmful stereotypes or differential treatment in the model's completions?
Representation Bias
Is the training data documented and reviewed for demographic representation gaps?
Have you evaluated performance across different languages, dialects, and cultural contexts?
Are underrepresented groups' perspectives reflected in fine-tuning data where relevant?
Confirmation & Sycophancy Bias
Does the model resist agreeing with factually incorrect statements when the user expresses confidence?
Have you tested whether the model changes its answer based on expressed user preferences rather than facts?
Are system prompts reviewed for instructions that may amplify confirmation bias?
Automation Bias
Are users informed that AI outputs require human verification before consequential action?
Is there a human oversight mechanism for high-stakes decisions made with AI assistance?
Are confidence scores or uncertainty signals surfaced to users to prevent over-reliance?
Output & Feedback Loops
Is there a mechanism to collect and review user feedback on biased or harmful outputs?
Are feedback loops in place to retrain or adjust the model when bias is identified in production?
Is bias monitoring part of your ongoing model evaluation process (not just pre-deployment)?