A checklist for developing fair models

With the preceding information, we can create a short checklist that can be used when creating fair models. Each issue comes with several sub-issues.

What is the goal of the model developers?

  • Is fairness an explicit goal?
  • Is the model evaluation metric chosen to reflect the fairness of the model?
  • How do model developers get promoted and rewarded?
  • How does the model influence business results?
  • Would the model discriminate against the developer's demographic?
  • How diverse is the development team?
  • Who is responsible when things go wrong?

Is the data biased?

  • How was the data collected?
  • Are there statistical misrepresentations in the sample?
  • Are sample sizes for minorities adequate?
  • Are sensitive variables included?
  • Can sensitive variables be inferred from the data?
  • Are there interactions between features that might only affect subgroups?

Are errors biased?

  • What are the error rates for different subgroups?
  • What is the error rate of a simple, rule-based alternative?
  • How do the errors in the model lead to different outcomes?

How is feedback incorporated?

  • Is there an appeals/reporting process?
  • Can mistakes be attributed back to a model?
  • Do model developers get insight into what happens with their model's predictions?
  • Can the model be audited?
  • Is the model open source?
  • Do people know which features are used to make predictions about them?

Can the model be interpreted?

  • Is a model interpretation, for example, individual results, in place?
  • Can the interpretation be understood by those it matters to?
  • Can findings from the interpretation lead to changes in the model?

What happens to models after deployment?

  • Is there a central repository to keep track of all the models deployed?
  • Are input assumptions checked continuously?
  • Are accuracy and fairness metrics monitored continuously?
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset