Model Cards for Model Reporting

Abstract

Trained machine learning models are increasingly used to perform high-impact\ntasks in areas such as law enforcement, medicine, education, and employment. In\norder to clarify the intended use cases of machine learning models and minimize\ntheir usage in contexts for which they are not well suited, we recommend that\nreleased models be accompanied by documentation detailing their performance\ncharacteristics. In this paper, we propose a framework that we call model\ncards, to encourage such transparent model reporting. Model cards are short\ndocuments accompanying trained machine learning models that provide benchmarked\nevaluation in a variety of conditions, such as across different cultural,\ndemographic, or phenotypic groups (e.g., race, geographic location, sex,\nFitzpatrick skin type) and intersectional groups (e.g., age and race, or sex\nand Fitzpatrick skin type) that are relevant to the intended application\ndomains. Model cards also disclose the context in which models are intended to\nbe used, details of the performance evaluation procedures, and other relevant\ninformation. While we focus primarily on human-centered machine learning models\nin the application fields of computer vision and natural language processing,\nthis framework can be used to document any trained machine learning model. To\nsolidify the concept, we provide cards for two supervised models: One trained\nto detect smiling faces in images, and one trained to detect toxic comments in\ntext. We propose model cards as a step towards the responsible democratization\nof machine learning and related AI technology, increasing transparency into how\nwell AI technology works. We hope this work encourages those releasing trained\nmachine learning models to accompany model releases with similar detailed\nevaluation numbers and other relevant documentation.\n

Related Publications