Joshand Hannah, a married couple in their 40’s, are applying for a business loan to help them realise their long-held dream of owning and operating their own fashion boutique. Hannahis a highly promising graduate of a prestigious fashionschool, and Joshis an accomplished accountant. They share astrong entrepreneurial desireto be ‘their own bosses’ and to bring something new and wonderful to their local fashionscene. The outside consultants have reviewed their business plan and assured them that they have a very promising and creative fashionconceptand the skills needed toimplement it successfully. The consultants tell them they should have no problemgetting a loan to get the business off the ground. For evaluating loan applications, Joshand Hannah’s local bank loan officer relies on an off-the-shelf software package that synthesises a wide range of data profiles purchased from hundreds of private data brokers. As a result, it has access to information about Joshand Hannah’s lives that goes well beyond what they were askedto disclose on their loan application. Some of this information is clearly relevant to the application, such as their on-time bill payment history. But a lot of the data usedby the system’s algorithms is of the kindthat no human loan officerswouldnormally think to lookat, or have access to—including inferences from their drugstore purchases about their likely medical histories, information from online genetic registries about health risk factors in their extended families, data about the books they read and the movies they watch, and inferences about their racial background. Much of the information is accurate, but some of it is not. A few days after they apply, Joshand Hannahget a call from the loan officer saying their loan was not approved. When they ask why, they are told simply that the loan system rated them as ‘moderate-to-high risk.’ When they ask for more information, the loan officer says he doesnot have any, and that the software company that built their loan system will not revealany specifics about the proprietary algorithm or the datasources it draws from, or whether that data was even validated. In fact, they are told, not even the developers of the systemknow how thedata led it to reach any particular result; they can say isthat statistically speaking, the system is ‘generally’ reliable. Joshand Hannahask if they can appeal the decision, but they are told that there is no means of appeal, since the system will simply process their application again using the same algorithmand data, and will reach the same result.
a) What sort of ethically significant benefits could come from banks using a big-datadriven system to evaluate loan applications?
b) What ethically significant harms might Joshand Hannahhave suffered as a result of their loan denial? Discuss at least threepossible ethically significant harms that you think are most important to their significant life interests.
