Build Fairness Library

Motivation: As models and datasets become increasingly large and complex, it is critical to evaluate the fairness of models according to multiple definitions of fairness and mitigate bias in learned representations. This library aims to make fairness metrics, fairness training tools, and bias mitigation algorithms extremely easy to use and accessible to researchers and practitioners of all levels.

Success Criteria:

• Create a fairness library, and apply it to the Textual Entailment model, publishing an analysis for where the present models fall short and where they should improve.
• Write a blog post and guide chapter and add a model and demo for the implementations of the fairness metrics and bias mitigation algorithms, and explain the broader impact.

Milestones

Implement the following:

Fairness Metrics

• Independence, Separation, Sufficiency
• Sparse Annotations for Ground-Truth
• Dataset Bias Amplification, Model Bias Amplification

Training-Time Fairness Algorithms (with and without Demographics):

• Through Adversarial Learning (with Demographics)
• Minimax (without Demographics)
• Repeated Loss Minimization (without Demographics)

Bias Mitigation Algorithms:

• Linear projection, Hard debiasing, OSCaR, Iterative Null Space Projection
• Bias direction methods: Classification Normal, Two Means, Paired PCA, PCA
• Contextualized word embeddings

Bias Metrics:

• WEAT, Embedding Coherence Test, NLI

Communication:

• blog post
• guide chapter
• demo
• contribute binary gender bias-mitigated model for SNLI to allennlp-models
• contribute binary gender bias-mitigated model for SNLI to demos