After months of struggle, the association Squaring the Net managed to obtain the source code of the algorithm used by the CAF to combat benefit fraud. It reveals “predictive surveillance with dystopian overtones”, which “deliberately targets the most precarious people”.
To fight against fraud, the CAF (Family Allowance Fund) uses an algorithm. Its objective is to estimate the risks of fraud in order to enable its investigators to correct them. A practice that has been known for a long time, but which CAF had never wished to detail. We now know why.
In an investigation published on November 27, Quadrature du Net reveals several lessons on its study of the CAF algorithm, which it was able to obtain after months of waiting. The French association for the defense of rights and freedoms on the Internet was not entitled to the source code of the algorithm currently used, but to those used between 2010 and 2014 and between 2014 and 2018. The CAF assumes that it does not want publish all the variables it uses, a behavior judged “nauseating” by Quadrature. The association does not mince its words in its article where it denounces a “predictive surveillance with dystopian overtones” and even questions the relevance of all this struggle, more ” policy ” And ” policewoman “ than social.
A “suspicion score” attributed to 32 million people
What Quadrature du Net reveals is the existence of what the association calls a “suspicion score”, between 0 and 1 and automatically assigned, by a machine, to each beneficiary.
Based on several social parameters, such as family situation, professional situation, financial situation, possible disability and place of residence, but also information collected by its agents (type and amounts of services received, frequency of connections to web space, time since the last trip to reception, number of emails exchanged, time since the last check, number and types of declarations, etc.), the CAF assigns a risk score to each of its registrants. In total, 32 million people, including 13 million children, are noted.
Unsurprisingly, the way the algorithm works naturally discriminates against the most precarious people. To prove it, Quadrature du Net simulated five suspicion scoreswith five very different profiles:
- A “wealthy” family with high income.
- A “modest” family where both parents earn the minimum wage.
- A single parent who earns minimum wage.
- A family in which both parents are beneficiaries of minimum social benefits,
- A family where one parent is a worker with a disability.
The results, presented in the graph above, show to what extent the CAF algorithm does not take into account any human specificity. It’s all just a line of code, which makes him wary of people who earn less.
The fact that a disabled allowance is also considered as a risk multiplier also poses numerous ethical problems, in the same way as the situation of single mothers, which the CAF is investigating further. The CAF algorithm assumes that the most precarious people are those who would tamper with their income the most — which de facto spares those with the greatest stability. This raises even more questions about the variables hidden by the CAF. What did she want to hide?
“The algorithm deliberately targets the most precarious”, affirms Quadrature du Net in its investigation. The association has also put the CAF source code online, so that the curious can take advantage of it.
According to Quadrature du Net, the suspicion score increases:
- if you have low income,
- if you are unemployed,
- if you benefit from RSA,
- if you live in a disadvantaged neighborhood,
- if his rent is too expensive
- if his income is not stable.
“ What our graph shows is precisely that socio-economic variables have a preponderant weight in the calculation of the score », she explains.
Is there a fairer system?
Can CAF do better? That’s the whole debate. If effectively tracking down social fraud is a legitimate objective for a public body, Quadrature du Net underlines that “There cannot be an algorithm model that does not target the most disadvantaged, and more broadly those who deviate from the standard defined by its designers. »
In its investigation, Quadrature du Net questions the relevance of this entire fight which, according to it, does not detect enough over-perception to justify such a great restriction on personal data. La Quadrature du Net also highlights “a fight against welfare” who forgets that many people do not receive their aid and that anti-fraud systems have always led to an increase in declaration errors. In short, the CAF algorithm is debated.
If you liked this article, you will like the following: don’t miss them by subscribing to Numerama on Google News.