NIPS - Not Even Wrong? A Systematic Review of Empirically Complete\n Demonstrations of Algorithmic Effectiveness in the Machine Learning and\n Artificial Intelligence Literature

Abstract

Objective: To determine the completeness of argumentative steps necessary to\nconclude effectiveness of an algorithm in a sample of current ML/AI supervised\nlearning literature.\n Data Sources: Papers published in the Neural Information Processing Systems\n(NeurIPS, n\\'ee NIPS) journal where the official record showed a 2017 year of\npublication.\n Eligibility Criteria: Studies reporting a (semi-)supervised model, or\npre-processing fused with (semi-)supervised models for tabular data.\n Study Appraisal: Three reviewers applied the assessment criteria to determine\nargumentative completeness. The criteria were split into three groups,\nincluding: experiments (e.g real and/or synthetic data), baselines (e.g\nuninformed and/or state-of-art) and quantitative comparison (e.g. performance\nquantifiers with confidence intervals and formal comparison of the algorithm\nagainst baselines).\n Results: Of the 121 eligible manuscripts (from the sample of 679 abstracts),\n99\\% used real-world data and 29\\% used synthetic data. 91\\% of manuscripts did\nnot report an uninformed baseline and 55\\% reported a state-of-art baseline.\n32\\% reported confidence intervals for performance but none provided references\nor exposition for how these were calculated. 3\\% reported formal comparisons.\n Limitations: The use of one journal as the primary information source may not\nbe representative of all ML/AI literature. However, the NeurIPS conference is\nrecognised to be amongst the top tier concerning ML/AI studies, so it is\nreasonable to consider its corpus to be representative of high-quality\nresearch.\n Conclusion: Using the 2017 sample of the NeurIPS supervised learning corpus\nas an indicator for the quality and trustworthiness of current ML/AI research,\nit appears that complete argumentative chains in demonstrations of algorithmic\neffectiveness are rare.\n

Ähnliche Arbeiten