Bias (AI/ML)
Systematic errors in AI model outputs that arise from skewed training data, flawed problem framing, or optimization objectives that don't match real-world goals — distinct from statistical bias and a major concern when deploying AI in scientific and clinical settings.
What it means
In AI and machine learning, bias refers to systematic tendencies in model outputs to deviate from ground truth in predictable ways. This is conceptually distinct from statistical bias (the difference between an estimator’s expected value and the true parameter value), though related.
AI bias typically arises from one or more sources:
Training data bias. If the data used to train a model underrepresents certain groups, conditions, or scenarios, the model will perform worse on those cases. A clinical prediction model trained predominantly on data from tertiary academic medical centers will generalize poorly to community hospitals. A genomic model trained on European-ancestry cohorts will have lower variant interpretation accuracy for non-European populations.
Label bias. If the labels used to train a model reflect human judgments that are themselves biased (e.g., historical clinical diagnoses influenced by race or socioeconomic status), the model learns and perpetuates those biases.
Proxy bias / feedback loops. A model trained on a proximate outcome (e.g., who received treatment) rather than the true outcome of interest (e.g., who needed treatment) learns patterns that reflect healthcare access disparities, not underlying biology.
Measurement bias. Different groups may be measured differently (different assay batches, different imaging protocols, different survey instruments), creating systematic differences that the model interprets as true signal.
Bias in scientific AI tools
Literature mining tools trained predominantly on English-language journals will underrepresent findings published in non-English literature, creating a geographic and linguistic bias in what they surface.
Protein structure models perform best on well-studied protein families with many homologous sequences in training data. Poorly characterized proteins (many in non-model organisms) have systematically lower prediction accuracy.
LLMs for research assistance reflect the biases of their training text — over-representation of certain fields, institutions, countries, and methodological traditions in the web text they were trained on.
Detecting and mitigating bias
Disaggregated performance metrics. Report model performance broken down by relevant subgroups (ancestry, institution type, disease severity, publication year) rather than only overall metrics. A model with excellent average performance may be systematically worse for a minority of cases.
External validation. Validate on data from different populations, centers, or time periods than the training data.
Benchmarks designed for bias detection. Domain-specific bias benchmarks (EHRSHOT for clinical AI, HellaSwag variants for reasoning) systematically probe model performance across subgroups.