featured

When algorithms go rogue the havoc is all too human

Summer reading | 17 April 2019

Dubravka Cecez-Kecmanovic

Emerita Professor

Decisions are polluted as artificial systems reinforce programming biases

From recommending a movie on Netflix to processing a job application, algorithms are increasingly part of the decision-making processes of our everyday lives.

But alongside this has come a growing awareness of the negative effects of algorithmic decision-making – with individuals denied access to social security or health insurance, or even spending longer in jail, all without any human intervention.

Dubravka Cecez-Kecmanovic, a professor in the school of information systems and technology management at UNSW Business School, has coined the term algorithmic pollution to describe this phenomenon of unjustified, unfair, discriminatory or other harmful consequences of autonomous algorithmic decision-making.

Her co-authors are Richard Vidgen, also a professor at UNSW Business School, and Olivera Marjanovic, a professor at UTS.

“Managers and politicians typically get excited about new technologies, especially artificial intelligence (AI),” says Cecez-Kecmanovic.

“They believe that we’re developing powerful algorithms that are non-biased, efficient and better decision-makers than humans, which is a myth. Algorithms learn from human beings, and they learn their biases.”

'The new ideology is to replace human beings in the decision-making process. It's like the Wild West'
DUBRAVKA CECEZ-KECMANOVIC

Lack of understanding

Hiring algorithms, for example, draw on historical data to identify characteristics that predict job performance. As they learn from past decisions, algorithms reinforce historical biases, despite the best intentions of designers.

And algorithms often rely on unchecked and potentially inaccurate data sets, which increases the risk of wrong and unfair decisions.

Cecez-Kecmanovic cites the case of a highly trained and experienced job candidate who was not short-listed. An investigation found that the algorithm rejected the candidate based on data from pharmacies where the candidate was once prescribed anti-depressants.

Perhaps most bizarre of all is predictive policing. Almost 400,000 Chicago residents now have an official police ‘risk score’ calculated by an algorithm. While still secret and publicly unaccountable, these risk scores are used for decision-making and shape policing strategy.

“The new ideology is to replace human beings in the decision-making process,” says Cecez-Kecmanovic. “It’s like the Wild West.”

“There’s a worrying lack of understanding of the inherent limitations and dangers of algorithmic decision-making among politicians, managers and other professionals. While algorithms can be very useful in complex calculations and support decision-making, they cannot replace human judgment,” she says.

Unfair impacts

According to Naz Guler, a director at PwC who works in the area of transformation delivery: “Increasingly, both in business and in government, I’m seeing that decisions are being entrusted to technology, and there’s a growing belief that ‘tech is always right’.

“I see increasing investment into AI systems and a growing reliance on AI models, without having a clear understanding of their capabilities, knowledge or training processes,” Guler says.

“We need to slow down and think clearly about what we’re doing, and the potential unfair impacts we will have. I would like to see businesses doing more to establish the foundations of trust in their algorithms and models.”

Guler sees the drive towards algorithmic solutions as the result of the convergence of huge increases in technology (such as data storage and computer power) plus the exponential growth of the volume of available data.

“There’s also been a realisation that data itself is really valuable,” she says.

AI has enormous potential to improve public policy and services, as Guler believes data analytics can enable governments to develop policies to create a more equitable society, with better personalisation and customisation of services.

“When it’s augmenting human efficiencies, it contributes in a positive way,” she says. “The danger comes when we rely on data-driven decisions with no human intervention or consideration.

“In my view, if there’s a bias within the data, you will get inadvertent decisions. A good example here is facial recognition systems, where there are a lot of examples of racial bias.

“There’s a lack of transparency about what data we’re using and how that data is relevant in any given decision,” Guler says.

'There’s a lack of transparency about what data we’re using and how that data is relevant in any given decision'
NAZ GULER

Perpetuating prejudice

Perhaps the most unsettling area where algorithms are increasingly taking over from human decision-making is in law enforcement. Predictive policing systems such as Predpol, which use past data on crimes in order to focus policing resources into certain geographical areas, are used widely in the US.

“In a resource-limited world, the idea of reducing crime by spending less money is very attractive,” says Lyria Bennett Moses, a professor and director of the Allens Hub for Technology, Law and Innovation at UNSW Law.

“And the words that get used to describe predictive policing – objective, scientific, data driven – all have a positive spin.”

But while Bennett Moses admits that these systems can be good at predicting location-based crime such as burglary, a predictive policing system inevitably focuses on where crime is reported, rather than where it happens.

These tools are less useful at predicting crimes with little location-based correlation, such as domestic violence (which are generally under-reported) or crimes policed in a racially based way, such as those connected with the use of offensive language.

“Offensive language occurs a lot at, say, sporting events or in pubs,” she says. “But where it actually gets reported is in areas like Sydney’s Redfern, with its large Indigenous population.

“If you police a community a lot, then you notice more of the crime that happens there. And that’s what goes into the database.”

Other algorithmic systems, such as COMPAS, are now creating risk assessment scores that are used in decisions on criminal sentencing and parole. Pro Publica has found that this tool has a higher false positive rate (falsely flagging danger) for African Americans, but it goes beyond racial bias.

A typical question that a system will ask is, are your parents still married, and if not, how old were you when they divorced? Presently, if an offender provides the ‘wrong’ answer to this, they could get refused parole, because they fall into a group that’s more likely to reoffend if released early.

“The basic idea here is wrong,” says Bennett Moses. “There are some kinds of decisions where these factors should be ignored – even if they are statistically relevant.

“We’re relying on systems that affect people’s lives, so we need to take a step back. It’s not just the machine – racial skewing is not the machine. If the past data suggests that black people commit crimes in certain places (and disproportionately so due to targeted policing) then that’s where the machine will look.

“It becomes a perpetuator of prejudice.”

Human responsibility

So, if the problem is clear, what then of the solution? Can businesses and governments police this themselves?

“We certainly need more regulation, and it needs to be well-evidenced and based on research,” says Cecez-Kecmanovic.

“The new General Data Protection Regulation (GDPR) in the EU is a good model to follow. Google and Facebook are already worried about these laws, so I think that the EU is on the right track.”

“’Responsible AI’ is a bit of a buzzword at the moment, AI that’s designed to draw in human values,” says Guler. “But the problem with this is – which values? Current business leaders in this sector are all North American, and for the most part, white men.”

Guler sees a good parallel here with recent discussions over bio-ethics.

“There’s an idea that we should be lining up AI in terms of human rights. We need to think about fairness, transparency, and integrity in decision making,” she says.

“For any decisions made by algorithms, there has to be a human responsibility identified,” says Cecez-Kecmanovic.

“If algorithms are our future, then understanding, fighting against and preventing algorithmic pollution may save our collective dignity and humanity.”

Social Impact Digital Data

Republish this article

Republish

You are free to republish this article both online and in print. We ask that you follow some simple guidelines.

Please do not edit the piece, ensure that you attribute the author, their institute, and mention that the article was originally published on Business Think.

By copying the HTML below, you will be adhering to all our guidelines.

Press Ctrl-C to copy

<h1>When algorithms go rogue the havoc is all too human</h1>

<figure><img src="https://assets-us-01.kc-usercontent.com:443/4df0558d-5779-0012-00f2-b3fa06d6c950/3949e4de-e03d-4897-bc12-cfb8ff61ad43/algorithms_hero.jpg?w=1320" alt="Algorithms rogue" /><figcaption>Algorithms rogue</figcaption></figure>
 From recommending a movie on Netflix to processing a job application, algorithms are increasingly part of the decision-making processes of our everyday lives.
But alongside this has come a growing awareness of the negative effects of algorithmic decision-making – with individuals denied access to social security or health insurance, or even spending longer in jail, all without any human intervention.
Dubravka Cecez-Kecmanovic, a professor in the school of information systems and technology management at UNSW Business School, has coined the term algorithmic pollution to describe this phenomenon of unjustified, unfair, discriminatory or other harmful consequences of autonomous algorithmic decision-making.
Her co-authors are Richard Vidgen, also a professor at UNSW Business School, and Olivera Marjanovic, a professor at UTS. 
“Managers and politicians typically get excited about new technologies, especially artificial intelligence (AI),” says Cecez-Kecmanovic. 
“They believe that we’re developing powerful algorithms that are non-biased, efficient and better decision-makers than humans, which is a myth. Algorithms learn from human beings, and they learn their biases.” 
<blockquote>'The new ideology is to replace human beings in the decision-making process. It's like the Wild West'DUBRAVKA CECEZ-KECMANOVIC</blockquote>
<h4>Lack of understanding</h4>
Hiring algorithms, for example, draw on historical data to identify characteristics that predict job performance. As they learn from past decisions, algorithms reinforce historical biases, despite the best intentions of designers. 
And algorithms often rely on unchecked and potentially inaccurate data sets, which increases the risk of wrong and unfair decisions. 
Cecez-Kecmanovic cites the case of a highly trained and experienced job candidate who was not short-listed. An investigation found that the algorithm rejected the candidate based on data from pharmacies where the candidate was once prescribed anti-depressants. 
Perhaps most bizarre of all is predictive policing. Almost 400,000 Chicago residents now have an official police ‘risk score’ calculated by an algorithm. While still secret and publicly unaccountable, these risk scores are used for decision-making and shape policing strategy.
“The new ideology is to replace human beings in the decision-making process,” says Cecez-Kecmanovic. “It’s like the Wild West.”
“There’s a worrying lack of understanding of the inherent limitations and dangers of algorithmic decision-making among politicians, managers and other professionals. While algorithms can be very useful in complex calculations and support decision-making, they cannot replace human judgment,” she says.
<h4>Unfair impacts</h4>
According to Naz Guler, a director at PwC who works in the area of transformation delivery: “Increasingly, both in business and in government, I’m seeing that decisions are being entrusted to technology, and there’s a growing belief that ‘tech is always right’.
“I see increasing investment into AI systems and a growing reliance on AI models, without having a clear understanding of their capabilities, knowledge or training processes,” Guler says. 
“We need to slow down and think clearly about what we’re doing, and the potential unfair impacts we will have. I would like to see businesses doing more to establish the foundations of trust in their algorithms and models.”
Guler sees the drive towards algorithmic solutions as the result of the convergence of huge increases in technology (such as data storage and computer power) plus the exponential growth of the volume of available data.
“There’s also been a realisation that data itself is really valuable,” she says.
AI has enormous potential to improve public policy and services, as Guler believes data analytics can enable governments to develop policies to create a more equitable society, with better personalisation and customisation of services.
“When it’s augmenting human efficiencies, it contributes in a positive way,” she says. “The danger comes when we rely on data-driven decisions with no human intervention or consideration.
“In my view, if there’s a bias within the data, you will get inadvertent decisions. A good example here is facial recognition systems, where there are a lot of examples of racial bias.
“There’s a lack of transparency about what data we’re using and how that data is relevant in any given decision,” Guler says.
<blockquote>'There’s a lack of transparency about what data we’re using and how that data is relevant in any given decision'NAZ GULER</blockquote>
<h4>Perpetuating prejudice</h4>
Perhaps the most unsettling area where algorithms are increasingly taking over from human decision-making is in law enforcement. Predictive policing systems such as Predpol, which use past data on crimes in order to focus policing resources into certain geographical areas, are used widely in the US.
“In a resource-limited world, the idea of reducing crime by spending less money is very attractive,” says Lyria Bennett Moses, a professor and director of the Allens Hub for Technology, Law and Innovation at UNSW Law. 
“And the words that get used to describe predictive policing – objective, scientific, data driven – all have a positive spin.”
But while Bennett Moses admits that these systems can be good at predicting location-based crime such as burglary, a predictive policing system inevitably focuses on where crime is reported, rather than where it happens. 
These tools are less useful at predicting crimes with little location-based correlation, such as domestic violence (which are generally under-reported) or crimes policed in a racially based way, such as those connected with the use of offensive language.
“Offensive language occurs a lot at, say, sporting events or in pubs,” she says. “But where it actually gets reported is in areas like Sydney’s Redfern, with its large Indigenous population.
“If you police a community a lot, then you notice more of the crime that happens there. And that’s what goes into the database.”
Other algorithmic systems, such as COMPAS, are now creating risk assessment scores that are used in decisions on criminal sentencing and parole. Pro Publica has found that this tool has a higher false positive rate (falsely flagging danger) for African Americans, but it goes beyond racial bias. 
A typical question that a system will ask is, are your parents still married, and if not, how old were you when they divorced? Presently, if an offender provides the ‘wrong’ answer to this, they could get refused parole, because they fall into a group that’s more likely to reoffend if released early.
“The basic idea here is wrong,” says Bennett Moses. “There are some kinds of decisions where these factors should be ignored – even if they are statistically relevant.
“We’re relying on systems that affect people’s lives, so we need to take a step back. It’s not just the machine – racial skewing is not the machine. If the past data suggests that black people commit crimes in certain places (and disproportionately so due to targeted policing) then that’s where the machine will look.
“It becomes a perpetuator of prejudice.”
<h4>Human responsibility</h4>
So, if the problem is clear, what then of the solution? Can businesses and governments police this themselves?
“We certainly need more regulation, and it needs to be well-evidenced and based on research,” says Cecez-Kecmanovic. 
“The new General Data Protection Regulation (GDPR) in the EU is a good model to follow. Google and Facebook are already worried about these laws, so I think that the EU is on the right track.”
“’Responsible AI’ is a bit of a buzzword at the moment, AI that’s designed to draw in human values,” says Guler. “But the problem with this is – which values? Current business leaders in this sector are all North American, and for the most part, white men.”
Guler sees a good parallel here with recent discussions over bio-ethics. 
“There’s an idea that we should be lining up AI in terms of human rights. We need to think about fairness, transparency, and integrity in decision making,” she says.
“For any decisions made by algorithms, there has to be a human responsibility identified,” says Cecez-Kecmanovic.
“If algorithms are our future, then understanding, fighting against and preventing algorithmic pollution may save our collective dignity and humanity.”

Comments