Gender, algorithmic bias and responsibility in AI

An account of a training on women's leadership and of the internal workshop it opened, on gender, power and decision-making in the algorithmic era.

In this article, I tell the story of an initiative by the Impostors' Club, the affinity group for exchange and support among Taqtile's women, that began as a training only for them and ended in a provocation for the whole company.

The training was Women in Charge: what nobody told you about reaching the top, with Jéssica Paraguassu. Out of it came a workshop open to the whole team, on the theme Who programs the future? Gender, power and decision-making in the algorithmic era.

The importance of naming the discomfort

As Naomi Sato wrote in "What kind of discomfort do you want to sustain?", discomfort in the corporate environment has many faces, but for women it is far more relentless. This is not news, but her account shows it up close.

Everyone has felt a certain discomfort, a difficulty in expressing themselves and feeling heard or recognized in corporate environments. The thing is that, for women, this is a reality that starts much earlier, going back to childhood and upbringing.

Present, or even intensified in corporate environments, these discomforts add up and lead to self-silencing. Part of it fatigue, part of it disbelief that they will be heard and recognized.

The thing is that in this process not only women are being silenced, but a whole perspective, repertoire and experience of the world that would contribute to activities, including business ones. Jéssica not only made this clear in her workshop, but showed that keeping this discussion away from the business is a mistake. In the video we published on LinkedIn, she gives a good example.

That is why it is important to create moments in which these discomforts can be named and debated. The Impostors' Club stands for exactly that: organizing spaces only of women and for women may seem a small step, but it has real value in expanding women's representation in leadership positions.

Women's participation in the world of technology

As Jéssica brought up in her workshop, in the last 20 years there has been a drop in the share of women graduating in Computer Science in the United States, from 37% to 19% (NCES). Not only that: studies indicate that only 11% of leadership positions in AI are held by women, and that, globally, 22% of AI professionals are women (Global Gender Gap Report, World Economic Forum, 2023).

In Brazil, although women's participation in the technology sector grew 60% between 2015 and 2022, 83.3% of the market is still made up of men (CNN, CAGED). The figure becomes even more significant when we find out that 63% of the people working on microwork platforms in Brazil are women.

That is because it is through microwork, not only in Brazil, that language models are trained. And, unlike more formal and well-known positions, pay and working conditions are far worse (see the BBC report).

Indeed, as the film Hidden Figures (2016) portrays, the idea that women are not interested in computing and technology is quite inaccurate. They were not only part of the founding of the field but continue to make an impact to this day.

What does it have to do with us?

For anyone who wants to create relevant products, solve business problems and innovate, or simply be a more engaged colleague, this subject matters too.

You have probably heard some of these figures:

• 97% of AI-generated "CEOs" are white men, while a "social worker" is almost always Black (Bloomberg).
• Facial recognition error reaches 34.7% for Black women, versus 0.3% for white men (Gender Shades).
• Language models are 4 times more likely to associate women with "home" than men (Unesco).

However, this subject does not end with the biases of Artificial Intelligence. As Jéssica put it while presenting this whole scenario to us, it is reflected in our work too. Because this bias not only makes invisible the participation and contribution of a significant part of the population, including in technology development, but shows that women's perspective is being disregarded.

Keeping this scenario in the large-scale adoption of AI amplifies the risks and problems, including making this part of the population even more invisible. Not by chance, as early as 2021 a recommendation on the Ethics of Artificial Intelligence was formally approved at Unesco by its 193 member states. It was the first global normative framework on the subject.

However, beyond becoming familiar with guidelines and frameworks (as we explore below), it is important to recognize in ourselves, especially in men, the reality of this "silent silencing". Because these perspectives, lost through the constant discomfort and silencing imposed on women, hinder innovation processes or any business problem solving. As Jéssica said, often:

"We think we are making rational decisions, from the most rational perspective possible, but could it be that the questions we are asking, the research we are running, the views we are taking are leading us to look at the world from a perspective that is predominantly male?"

What can we do differently?

That was one of the central proposals of the workshop. Working in detail on the notion that AI is not neutral, and that technical decisions distribute risks and benefits to specific social groups, the training brought a series of provocations.

We cannot cover here all the recommendations, nor the depth with which she treated the subject (look up her training); we have organized part of what was presented. Among the references was the work of Catherine D'Ignazio and Lauren F. Klein (MIT, 2020) on Data Feminism as a working method.

Framework: the 7 fundamental principles

Catherine D'Ignazio and Lauren F. Klein, MIT Press (2020)

1. Examine power. Map who benefits and who is harmed by the data
2. Challenge power. Build data that challenges, rather than legitimizes, inequalities
3. Elevate emotion & embodiment. Recognize bodies and feelings as legitimate dimensions of knowledge
4. Rethink binaries & hierarchies. Abandon categories that naturalize oppression (binary gender, race, etc.)
5. Embrace pluralism. Multiple perspectives produce more complete knowledge
6. Consider context. Data does not speak for itself; context makes the data
7. Make labor visible. Make visible who collects, annotates and cleans

Another model presented was the Algorithmic Impact Assessment (AIA), a standard created by the Canadian government that sets strict rules for the use of Artificial Intelligence and algorithms in public services. The AIA is a tool for assessing impact with a gender, race and vulnerability lens before products are deployed, revisited continuously.

1. Map affected stakeholders. Who is affected by the system's decision? Who becomes invisible?
2. Classify risk by category. High, medium or low, with a gender, race and class lens
3. Audit training and output data. Equity metrics, under-representation, differential error
4. Define mitigations and oversight. Human-in-the-loop for sensitive decisions; a channel to contest
5. Continuous review. The system changes, the world changes; the assessment needs to be alive

For those who want to go deeper, Jéssica put together a bibliography:

• BENJAMIN, Ruha. Race After Technology. Polity, 2019.
• BUOLAMWINI, J.; GEBRU, T. Gender Shades. MIT Media Lab, 2018.
• CASILLI, Antonio. Esperando os robôs. Ed. Unicamp, 2022.
• CRAWFORD, Kate. Atlas of AI. Yale Univ. Press, 2021.
• CRIADO PEREZ, C. Invisible Women. Vintage, 2019.
• D'IGNAZIO, C.; KLEIN, L. Data Feminism. MIT Press, 2020.
• EUBANKS, Virginia. Automating Inequality. St. Martin's, 2018.
• FEDERICI, Silvia. O Ponto Zero da Revolução. Elefante, 2019.
• GEBRU, T. et al. "Datasheets for Datasets". CACM, 2021.
• GEBRU, T. et al. "Stochastic Parrots". FAccT, 2021.
• GONZALEZ, Lélia. Por um feminismo afro-latino-americano. 1988.
• GRAY, M.; SURI, S. Ghost Work. HMH, 2019.
• HARAWAY, Donna. Manifesto Ciborgue. 1985 (Brasil: Hedra, 2009).
• HARDING, Sandra. The Science Question in Feminism. 1986.
• HICKS, Mar. Programmed Inequality. MIT Press, 2017.
• MITCHELL, M. et al. "Model Cards". FAT*, 2019.
• NOBLE, Safiya U. Algorithms of Oppression. NYU Press, 2018.
• WAJCMAN, Judy. TechnoFeminism. Polity, 2004.
• UNESCO. Challenging systematic prejudices. 2024.
• PretaLab + Olabi. Quem Coda BR. Pesquisas 2020-2023.

How we are internalizing the provocations at Taqtile

From the training on, some questions started circulating internally. Reinforced by the Impostors' Club at the opening of the workshop, more than counting how many women there are or are joining the company, we need to look at positions of leadership and influence. How and with whom we are making decisions, which points of view are being considered and how this affects the diversity of the team and of the people who use our products are some of the questions all of us can ask.

We also discussed how the subject ties into Taqtile's main objective for 2026: working with what is emerging in technology, exchanging what we learn and developing ourselves and the businesses we collaborate with. Understanding that every technical decision we make in our day to day leaves Taqtile at scale, leading this transition requires understanding the biases of what we are accelerating. That is the understanding we took from the meeting into our projects.

Our thanks to Jéssica, to the Impostors' Club and to Tuanny Martins and Naomi Sato, who organized the meeting and the training with the help of Bianca Lima, Ysabella Andrade and Erica Urushibata.

If you want to follow what we publish on responsible AI, gender and technology applied to business, subscribe to the Taqtile newsletter. It is where we share our latest experiments, cases and projects.

Jéssica Paraguassu is a master's student in World Political Economy at UFABC, where she coordinates the Gender and Technology Studies Group, with research focused on the impacts of artificial intelligence on women's work and lives. With more than 15 years of corporate experience and creator of the Women in Charge methodology, which has reached more than 2 million women, she works at the intersection of academic production and practical transformation, dedicated to building a more inclusive future in the face of digital transformations.