Their method of identifying genders is to use a data source of Github user emails and cross reference those with a Google+ account. Then they scrape the Google+ account and attempt and try to automatically determine the gender. Using this method, they are able to identify only 35% of the users involved in these PRs.
I find this suspect, because, anecdotally, I can look almost any author of a PR and determine their gender to a high degree of certainty. Probably well over 90% of the time, just from their name + profile picture + handle. Try it for yourself..look at the latest commits on a random project and see how obvious the genders are most of the time.
So the claim that "when a woman is identifiable, PRs are merged less" is totally suspect, because they themselves can only identify the genders of a small percentage relative to what a normal human can identify. If people can identify the genders way more often and accurately, then the claim being made is bogus. Perhaps there is a correlation of strongly signalling your gender (to the point where an inaccurate method of gender-identification has no problem) to being a below average developer.
If you make to the end of the article, they do account for exactly this issue by comparing cases where they could find an out-of-band gender indication (google+) but where the name was not identifiable via github name/profile.
From the article:
> For gender-neutral profiles, we included GitHub users that used an identicon, that Michael’s tool could not infer a gender for, and that a mixed-culture panel of judges could not guess the gender for.
Only 35% percent of the accounts have their gender listed in a linked Google+ account. Checking someone's social media profile is a relatively sure way of automatically determining the gender of a lot of people. The authors did use another automated tool to see if they could figure out the gender of users from their Github profile as well, which is something they needed for the second part of their analysis. They don't specify how accurate that procedure was, so it's possible that they are more accurate than you think.
Of course, there is still the issue that they have effectively limited their sample to people with Google+ accounts which may affect the results of the study. Given that men's acceptance rate also dropped when their gender was identifiable (but not by as much) gives credence to the idea that there might be a flaw in their Github profile analyzer.
Well, they used two steps. First they identified a sample set of contributors that have self identified, thus validated to some extent, their gender.
Further down they then distinguish between contributors where the gender can be inferred from looking at their name & profile picture. Splitting the group of those 35% which were identified via Google+ into two separate groups - identifiable vs. non-identifiable.
Open Closed Merged Merge Rate 95% Confidence Interval
Women 8,216 21,890 111,011 78.6% [78.45%, 78.87%]
Men 150,248 591,785 2,181,517 74.6% [74.56%, 74.67%]
> The hypothesis is not only false, but it is in the opposite direction than expected; women tend to have their pull requests accepted at a higher rate than men! This difference is statistically significant (p < .001).
Many hypothesis are discussed in an attempt to explain this and the paper is well worth the read. Thanks to a documented GitHub API this stuff gets quite scientifically correct - everything is backed up with real-world data.
More quotes:
> To summarize this paper’s observations:
> 1. Women are more likely to have pull requests accepted than men.
> 2. Women continue to have high acceptance rates as they gain experience.
> 3. Women’s pull requests are less likely to serve an immediate project need.
> 4. Women’s changes are larger.
> 5. Women’s acceptance rates are higher across programming languages.
> 6. Women have lower acceptance rates as outsiders when they are identifiable as women.
"For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable."
And then you discover in Questions Section;
"Our analysis (not in this paper -- we've cut a lot out to keep it crisp) shows that women are harder on other women than they are on men. Men are harder on other men than they are on women."
https://peerj.com/questions/2002-do-you-have-data-on-the-gen...
Good grief. You'd think, considering the near constant criticism of men in technology for gender bias, that fact might have warranted a mention as an aside at least.
One bias I don't see often mentioned in research papers is for researchers (or those who present the work of researchers) to draw light to findings supporting certain narratives while leaving findings supporting another narrative only available to the very small percentage of people who go and read the full report.
Sampling bias, perhaps? Or how that thing is called? If there's, say, 20% of women in a field and 80% men, and you get treated badly twice as likely by an average woman than by an average man, you'd still observe (in absolute terms) more instances of bad treatment from your male peers. Surely this could distort one's perception.
Most of the even-half-informed criticism I see is about systems for gender bias, and specifically acknowledges that people regardless of gender who have been brought up in a world with biases have internalized those biases.
As a man in technology, I can't say I've ever felt personally attacked for gender bias.
I guess it depends on how you look at it. When I see phrases like "boy's club" (not in this paper), it implies to me that there's a group of men who work (consciously or unconsciously) to keep women out. I'm aware that it's not at all an uncommon finding that women are pretty hard on other women, yet I think that receives very little play in the popular media.
Perhaps I do take it too personally - I do my very best to be fair and reasonable with all my fellows, whatever their various deviations (or not) from the industry norm might be. It's upsetting to me to be tarred with this guilt-by-association. I find it particularly disheartening in the venue of OSS: I've always viewed it as a truly fantastic collective charitable effort, yet lately it seems to be getting depicted more and more as some kind of refuge for white men to exclude everyone else.
I feel the same guilt, I'm a student - I'm excluded from scholarships, meetups and career events because of my gender, that plus the regular criticism aimed at the industry - I feel ashamed every time I'm asked to fill out an equality form. I don't know what to do about it.
Don't let it effect you? Or, "I look to a day when people will not be judged by the color of their skin, but by the content of their character" can be applied to sex and any other identifiable trait as well.
I think we have a winner. Most likely difference - corporate vs non-corporate contributors.
The paper's conclusion that "women are more competent overall" doesn't seem very plausible to me. If there was really a significant, consistent boost simply from being female then we'd see software companies that consist mostly of women or which are run by women outcompete opposite companies at least some of the time, and it'd be an effect that'd have been detected before. I'm struggling to think of any examples of this.
Based on my own experience of github and open source, however, the only times I have ever seen women contribute to open source, they were always being paid to do so. I cannot recall a single time that I came across a pull request or even just a regular patch from a hobbyist woman. Not sure why that is, but it has been a constant throughout my career - men do hobby programming and women don't.
Given this difference I'd expect contributions from women to be larger overall, tackling grunge work that volunteers wouldn't want to do, solves some real business problem, and more frequently be code that was tested for a while internally first so is less likely to have quality problems.
The fact that men are also less likely to have pull reqs accepted when their gender is known also seems likely to be a correlation with submitting from personal gmail accounts vs corporate accounts.
> Based on my own experience of github and open source, however, the only times I have ever seen women contribute to open source, they were always being paid to do so. I cannot recall a single time that I came across a pull request or even just a regular patch from a hobbyist woman. Not sure why that is, but it has been a constant throughout my career - men do hobby programming and women don't.
Possibly related anecdata: from my not insignificant, but still far from comprehensive experience nearly all the women who are developers in my area (not SF—I know nothing about that market aside from what I read) are with bigger companies (not necessarily software companies), and in fact there are quite a few women working as developers with those companies.
OTOH, startups and small software businesses around here do hire lots of women—but mostly as graphic designers, or in sales/marketing, or as project managers. Very, very few women even interview for startup software development jobs here (for whatever reason—I don't claim to have an explanation).
Presumably something risk related? Men take more risks, it's documented, testosterone related. Working for a startup is risky.
At Google there was (when I worked there) perhaps about 20-30% female engineering staff. Or maybe it was a bit lower, I don't recall, but it was pretty high. Yet as far as I recall, the new projects were typically started by men, I mean the bottoms-up ones. I never noticed it at the time, but thinking back, I'm struggling to think of cases where a woman started a new project with her 20% time (or even took it, actually). That might be just lack of data points of course. Projects created by employees weren't at all rare there, but it still wasn't something that happened every day.
Many women in software, by my experience, are from slightly poorer parts of the world: eastern Europe, Russia, China, India. I suspect to them programming is just a job; a way to earn good money. But it's not something they're especially passionate about in the single minded obsessively focussed way that men can be (and I'm guilty of that).
Clearly larger companies are likely to attract more risk averse candidates but they also typically offer better health benefits, better work/life balance, and are more likely to have an hr department that can maybe mitigate some of the worst sexism that crops up in our industry.
There's also probably a bit of a gravity affect happening. My entire career I've never been on a team with even a single other woman and it would certainly be a nice thing to go somewhere where that wasn't the case, so maybe it's just easier for companies that employee women to hire more women.
This is also something I have observed in my job. I am in a typical enterprise software shop and I feel like women are more likely to see programming as "just a job", while men are more likely to put more passion into it. But most people in general see it as "just a job", so maybe hobbyist female programmers are just rare because female programmers in general are rare.
I'm a woman, I program professionally and as a hobby, I have friends who do too. I think maybe the biggest problem is visibility. I work on things and have a lot of code on github but I don't really get involved with open source projects in my free time.
For me it's a combination of not wanting to deal with the very abrasive personalities idolized in some open source communities and just preferring to spend my free time hacking on throw away code to experiment with new ideas. I write clean readable tested production code at work, I want my hobby programming time to focus on impractical educational exploratory programming.
It also just feels like I have a lot less time to write code than my coworkers, even though I'm the only person on my team without kids. Traditional gender roles are evolving but it still feels like I'm left with more life stuff to deal with.
It's worth noting that the number of pull requests by women is far, far smaller than that by men. So either there are far less women active on github, or women are more careful when submitting pull requests.
The reasons for being more careful could be all over the place, ranging from actually being better programmers to being more likely to receive criticism for a bad pull request.
In an industry that's somewhat hostile to women, it wouldn't be surprising that only the most competent women stick around, whereas less competent men stick around more than less competent women, dragging average competency of men in the industry down, even if the total number of competent men might be higher than the total number of competent women.
I'm saying this in a comment that explains this point. The world is bigger than this one study.
I'm explaining how hostility to women can result in a higher acceptance rate of their pull requests. As you could have seen from the numbers in this study, women do far, far fewer pull requests than men. Why? Perhaps, because only the best women remain, and they only submit their best pull requests. That drives the average for women up, while driving the total number down.
And I don't think the acceptance rate of pull requests is a likely vector of discrimination anyway; they're fairly anonymous and focus on the code. But the hostility absolutely does exist in other areas: small talk, jokes, hiring practices, sexual harassment, etc.
I don't think the drop is the important bit - men's AR also drops when gender identifiable.
In fact, they both drop when identified to a point which (as far as I can tell from the graph; figures aren't given) they are within each other's confidence interval.
So actually, we're more critical of everyone when their gender is disclosed; equally so.
Which I find surprising. I'd have thought if anything, we'd be more trusting and supportive of the person disclosing a picture, name, etc. (the criteria used for gender identification).
I'd have thought if anything, we'd be more trusting and supportive of the person disclosing a picture, name, etc. (the criteria used for gender identification).
Why though? My experience is that aside from everyone's monkeysphere, people generally tend to hate other people.
Social media where you interact with a higher than Dunbar number of people (ie Not Facebook) is often just a nasty status game.
Humanizing a pull request can bring all sorts of baggage from that into the evaluation of a contained block of ideas.
I'm personally somewhat biased (probably unfairly) against people that use a full name and photo rather than a handle, if I don't know them. I just have this gut feeling that their pull request is probably "I've rewritten this cli tool in node.js!" or something. It's a purely tribal thing, but that may explain it to some extent (then again, I feel like github is the home of people like that, so it should work as an in-group marker? Dunno)
The same conclusion for men, using the same data set (figure 5) is this: Men's acceptance rates are 69%, but drop to 64% when the gender is identifiable.
So to summarize the two: The overall acceptance rate drops from ~70% to ~63.5% when the gender is identifyable. Very interesting.
Yeah, sorry. I was editing that comment as I initially skimmed it. I added in their final summaries and they do include that fact which you mentioned.
I'm not proud of that statistic. I used to be firmly on the fence regarding the subject of women being discriminated against in tech. With facts like that, there's no denying it whatsoever. We have a big problem.
Ignoring the fact that this difference could easily be explained away by some other factor when it is considered. It may have nothing to do with it. Causal relationships are so tricky when dealing with statistics.
PR closes can be for a variety of factors besides programmer competence or gender bias. For example, point #4 says that "women's changes are larger", whereas many large projects are particularly wary of large PRs with multiple changes. That's not to say that the large PR would break something, but many times I've seen a PR closed with a request to break it down into multiple smaller PRs that are more easily reviewed and integrated.
> 6. Women have lower acceptance rates as outsiders when they are identifiable as women.
They should have added this:
7. Men have lower acceptance rates as outsiders when they are identifiable as men.
Seeing those two together - equally justified by the text - suggests it might just be a small bias (~10%) against "outsiders" who use their real name in favor of "outsiders" who use hacker aliases...regardless of gender.
Be careful implying causation here. We would want to investigate who identifies themselves as any sex vs who remains unidentified (or less obviously identified) and the skill levels represented in each group. Since GitHub does not request your gender for your profile, they used Google+ profiles, which I think would significantly slew the results; they did not sample from all pull requests, but from those that were linked to a G+ account AND whose owners decided to post their gender.
> Specifically, we extract users’ email addresses from GHTorrent, look up that email address on the Google+ social network, then, if that user has a profile, extract gender information from these users’ profiles. Out of 4,037,953 GitHub user profiles with email addresses, we were able to identify 1,426,121 (35.3%) of them as men or women through their public Google+ profiles.
> This is awful - I think it would be good to have some discussion around what we can do as a community to improve this?
I don't agree that the result is as strong as "awful" indicate because the paper concludes that
> Women had higher acceptance of pull request than men
> Identifiable women had higher acceptance of pull request than men
> Insider identifiable women had higher acceptance of pull request than men
> Both outsider identifiable women and outsider identifiable men had lower acceptance rates than their insider counterparts
The study proposes the existence of gender bias because, in this last point, the drop for women was higher than the drop for men, regardless of all the other instances where the opposite was true.
I don't believe the study is representative for the whole industry at all (GitHub being mainly remote, voluntary and due to the FLOSS philosophy mostly liberal) but this seems to be one to be celebrated because of the positive conclusion it showed for women.
Given that from this article's stats it even admits there are 15 times more male self-identified programmers then women on github, a 4% difference in pull requests accepted is tiny.
The data is just convenient for anyone with a political agenda to make big claims. If the paper wanted to be genuine they could have gone further - what was rejected? Was there a valid reason? etc
Based on the findings about men, it seems that both men and women experience a drop when they identify gender. Women experience a greater drop than men, but it is worth considering this trend is not found in the treatment of any one gender.
So the question remains, why do both genders experience a drop. Once we answer that, we can look into why women experience a worse drop.
The difference between men and women when their gender is identifiable is 0.5% (63.5 vs 64.0). Lets first have some good discussion on how significant that difference is.
[Edit]
Retracting my comment below, after the parent comment was edited to provide more context.
---
This quote doesn't seem representative of the whole story that is easily discoverable in the abstract.
> Surprisingly, our results show that women's contributions tend to be accepted more often than men's. However, when a woman's gender is identifiable, they are rejected more often. Our results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless.
This is indeed in their abstract, but it's misleading. Take a look at page 15 of the preprint. Their data shows that gendered profiles have lower acceptance rates for both genders; that is, pull requests from a gender-neutral profile are more likely to be accepted regardless of the gender of the person behind it.
This paper gives the impression that the authors started out hoping to find evidence of discrimination in the form of a lower pull acceptance rate for women than men, they found the opposite, and then they data mined until they could find something they could use to argue there's an anti-woman bias.
The summary seems to be all positive regarding women's contributions except for the last one:
"Women have lower acceptance rates as outsiders when they are identifiable as women.".
Looking at the data, they say that the acceptance rate for female outsiders is 62.5%, but they don't say what it is for men. Looking at the chart it looks like ~63.5%. And acceptance rates for both genders drop by >=10% going from non-gendered to gendered. Can we infer then that the bias is somewhat across the board for pull requests from easily gendered users?
it looks like the criteria for determining gender is to pull it from a google+ profile that shares an email address with the github account, and is not using any information from the github profile.
It seems likely to me that the bias here is gmail users versus non-gmail (corporate domain?) users.
That's incorrect. The study identified the gender of the Github users twice. They got their 'canonical' gender for users using your method (and removed from their sample users who could not be identified with this method,) but later to figure out whether a user's gender was identifiable, they used the Github user's own profile.
Non-Google+ users were not analyzed at all. That could possibly still influence the results, but not the way you are suggesting.
Hmm..., this could very well modify the results. Do they present the numbers when they limit it to only people with google+ accounts or to only people using @gmail as their email address?
It does seem like this would be an even more important conclusion. Whether you're a man or a women, your PR will much more likely be accepted of your profile doesn't reveal your gender (at least not in the sense that creators of this study can identify it).
Given that it seems like they analysed real-world data rather than producing synthetic experiments, I wouldn't put too much stock in the findings. For instance, in a hypothetical world where all businesses contributing to Github projects are perfectly sexist, only ever assign male coders to core engineering and only ever assign female coders to UX design, UX design is intrinsically more controversial and Github maintainerers accept patches purely on the degree of public support for their contents, one would expect a far lower degree of acceptance for patches from identifiable (as businesses presumably make their workers operate under real-name accounts) female accounts without this containing any signal about sexism on the Github side.
A better methodology for a study like this would be to identify a number of people planning to make sporadic contributions to projects they have no connection to and making them submit the pull requests under fresh fictional profiles with randomised gender. This way, any irrelevant correlations between the kind of pull request being made and the requesting party being identifiable as female would be eliminated.
> Experiments and retrospective field studies each have advantages. The advantage of experiments is that they can more confidently infer cause and effect by isolating gender as the predictor variable. The advantage of retrospective field studies is that they tend to have higher ecological validity because they are conducted in real-world situations. In this paper, we use a retrospective field study as a first step to quantify the effect of gender bias in open source.
The research is presented in context, an important and valid context, too. At the end, they discuss potential explanations and never go so far as to say "And all of this is clearly because ______".
A controlled study would be a great followup, but I'd sure put stock in these compelling and well documented findings.
I would tend to agree with you. When you use public data you get all the implied imperfections of public data. You can make some vague inferences but since the controls aren't in laboratory conditions the potential for biased results is too high.
If they showed individuals with already high acceptance rates received lower acceptance rates on the same code when submitted from a gender identifiable account then I'd find it to be more compelling.
> Women’s acceptance rates are 71.8% when
they use gender neutral profiles, but drop to 62.5% when their gender is identifiable
What is also true from the graphs, but the author does not emphasize is this: Men's acceptance rates are 69%, but drop to 64% when the gender is identifiable.
This is taken from figure 5. So in general, it seems the acceptance rate drops in both cases, as soon as the gender is identifyable. I do wonder why this is? The two just don't add up. It seems that the type of PRs that the authors have maanged to have the gender identified, simply have much lower acceptance rates.
It could be something like hobbyist vs. professional programming. I would bet that pull requests generated as part of someone's job are typically higher quality: they're more thoroughly used/tested before submission, the authors can spend more time on them, etc. These are more likely to be submitted from "work accounts", which can be deliberately vague (e.g., if shared) or just not fleshed out very well.
It might be possible to flesh this out by looking at the domain names.
Maybe, when you reveal things about yourself, people are more likely to find things they dislike about you. Having a recognizable gender on github would often include things like: a picture (your face is ugly!), your name (your ethnicity sucks!), your country (your country sucks!), your place of work (your company sucks!) and maybe your side project blog (your side projects are stupid!).
Or maybe more experienced/open-source involved programmers have a stronger proclivity for choosing abstract user handles? There's no end to this guessing game, all they revealed was some correlation (and not a very strong one at that), and this by itself means next to nothing.
That is a possible explanation. Naively I would have thought the opposite - the more you know about someone, the more likely you accept their PR. But this is a statistically wide enough gap for both cases, and also a statistically more significant drop in acceptance of women's PRs.
I would say "the more you know about someone like you"...
To me that would correlate with other studies based on first impressions, hiring, etc., where we sometimes subconsciously seek out people like us. Similar to another comment, giving any information about you allows me to put you in a box I have created, whether positive or negative.
There's a large drop for both men and women between project outsiders who do not identify a gender on GitHub and those that do. So that's one very clear effect. If you want to have your contributions accepted on GitHub, male or female, and you are an outsider, do not identify yourself.
However, it was only possible to identify the gender of about a third of people overall from publicly available information.
So for the conclusions of the study to be valid, one must eliminate the possibility that there is a gender imbalance in those whose gender is able to be identified by publicly available information (which would not represent discrimination, since that information is self-reported). Until one can eliminate this, one may not be measuring discrimination, but effectively just a correlation between availability of public information on programmers and their skill level.
Moreover, it looks like the gender information was entirely constructed from Google+. Yet there is no information on what percentage of users' of each gender actually had Google+ pages. This is extremely relevant to the conclusions as there could also be correlations with the usage of Google+ by programmers of a certain skill and gender.
> Out of 4,037,953 GitHub user profiles with email addresses, we were able to identify 1,426,121 (35.3%) of them as men or women through their public Google+ profiles.
I wonder if there's a gender bias with regard to making your Google+ profile public (or having one in the first place).
I also wonder how they actually verified the public G+ profiles. I know that a significant number of women in my circle of acquaintances deliberately flag themselves as male.
I randomly lie about my gender, age, and anything that could be used to identify me. I learned this from others, and have also encourage my spouse/children/etc. to do the same (which they seem to do).
There is no compelling reason for me to provide any real information about myself online.
>Our analysis (not in this paper -- we've cut a lot out to keep it crisp) shows that women are harder on other women than they are on men. Men are harder on other men than they are on women.
I think we should be very careful how we choose to interpret the results here. There are a lot of unknowns and variables presented in the paper (how they're determining gender, for example.)
I'm not sure there's enough data here to create an understanding about the relationship between merge rate and gender. Various external issues could be the cause of the difference (I don't believe either gender is more capable) - for example (not sure if this is true, but it portrays how these stats could be skewed): In India, there's a higher incentive to be recognized in the community because competition is stiffer, so more people submit PRs. In this case, people who generally wouldn't submit a PR because of their lack of expertise are now submitting PRs on the chance that it may get accepted. The result is a greater number of PRs with lower quality.
These are the effects of having a global community. There are so many external factors I don't really think we can determine much of anything from these results. It's an interesting analysis regardless though.
> Specifically, we extract users’ email addresses from GHTorrent, look up that email address on the Google+ social network, then, if that user has a profile, extract gender information from these users’ profiles.
I was curious so I looked up the Google+ API (https://developers.google.com/+/web/api/rest/) and I do not see an "email lookup" endpoint. You can only search for generic information in public profiles. Searching for my own email, which is public in Google+, using that particular endpoint yielded no results.
I call shennanigans that 1/3 of the accounts in the dataset had discoverable profiles this way, and they were retrieved in a manner that is acceptable by Google.
To me, the real horror show is the 1:18 ratio between contributors who publicly identify themselves as a Woman and contributors who publicly identify themselves as a Man[1].
There's so many hypotheses one could make about that (ratios among software engineers in general, differing interest in programming as a hobby beyond work, the confrontational culture among projects, hiding ones identity to avoid creeps, etc). It's a good start and we could really use further research.
[1] Sorry that's worded so awkwardly, but it's hard to phrase that without implying something unsupported by this particular data.
> To me, the real horror show is the 1:18 ratio between contributors who publicly identify themselves as a Woman and contributors who publicly identify themselves as a Man[1].
To me, comments like yours highlight the toxic nature of the software industry.
It is an expectation in software dev that we do most of our learning and development on our own time and part time work is generally frowned upon. GitHub accounts and side projects are becoming a larger part of interview processes.
How do you think this looks to prospective female employees? Basically it gives the impression that they have to be prepared to work 45-50 hours a week in order to be competitive.
This isn't compatible with many women's focus on flexible work for when they have children.
Notice that none of this is inherent to software development. Companies could provide proper on the job learning and development. Companies could be more open to part time work. Companies could use better proxies for developer skill than personal contributions to GitHub.
I think you are reading stuff into my comment that I simply did not say or imply; worse, the things you think I said you describe as "toxic"!
I don't think side projects are necessary to further your career, and it's been 20 years since I worked on one myself.
You do make good points that an individual may not have the opportunity to engage in a side project for reasons not related to competence. Your hypothesis that the hardship faced by many women once they have children may contribute to the statistic is well-worth further investigation -- like I said, there's many possible hypotheses to investigate, and more than one factor may contribute to the overall outcome. As an interviewer, I've only once seen an outside project on a resume and I considered it a plus; that may have been a mistake on my part.
If you really want to encourage women then help improve the industry by pushing for 20% time, pushing for flexible work practices, pushing for internal promotions, pushing for salaries to be adjusted to better reflect market rates when they get out of alignment, etc.
Remember this is not peered reviewed. Others haven't looked at it to decide if it had proper controls, good enough samples, or even valid data. So take this with a grain of salt until it's properly peered reviewed.
I almost wonder if it's worse to publish non-peered-reviewed research or not. I mean it could be great if it's solid research but if there are flaws (even minor) it could undermine other studies, change people's minds, etc.
We considered a GitHub profile as gender neutral if all of the following conditions were met:
- an identicon (rather than a profile image) was used,
- the gender inference tool output a ’unknown’ for the user’s login name and display name, and
- none of the panelists indicated that they could identify the user’s gender.
Across both panels, panelists inspected 3000 profiles of roughly equal numbers of women and men.
Given that acceptance rates dropped for both men and women when this method determined that a user was not gender-neutral, I wonder if use of a profile picture or real name is the real killer here. Perhaps the kind of person to include identifying information in their Github profile is less likely to have good pull requests? More likely, perhaps having any identifying information at all gives people an excuse to reject your pull requests? That would explain why there was a drop for both sexes but a slightly stronger one for women. If you are identifiable from your profile image or name, people are more likely to reject your PR on the bases of your ethnicity, appearance, religion, or gender, with gender being only a part of the effect.
"For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable. There is a similar drop for men, but the effect is not as strong."
Why leave out an exact number for men whose gender is identifiable?
For regular contributors, women appear* to accept a man's work (slightly) more than they will a woman's, and men appear to accept a woman's work more than they will a man's. Since there are more men than women, this means that on the whole women are privileged.
* appear: maybe men or women have a different internal bar for how polished they'll make a pull request (how afraid they are of rejection, etc). the study looks at profiles that are closeted vs out as a gender. If you reveal gender on purpose, this tells us something about you, presumably, but for convenience the difference between closeted and out is taken to signify "bias against [out] men/women". Much of this is not statistically significant, probably (study doesn't give enough info, and suspiciously did an "insider" vs "outsider" analysis rather than a pooled analysis, suggesting they didn't like what they found until they split into two pools).
For unsolicited outside contributions, closeted men seem to get rejected more than closeted women (men's bar is lower?). Out men get rejected more than closeted men or women. Out women get rejected more than closeted men or women. The key is: for this category, people appear to be less biased against out men than out women (but somehow people prefer contributions from hidden-gender folks?).
Anyway, this is interesting stuff but I'm not sure what to take from it. I do expect more low-quality outside-team submissions from men than women, and I might judge them unfairly if they were out men, but this is just a random hunch and I doubt it would affect me much (probably I wouldn't notice).
Remember when people on the internet werent pushed towards using real names? Gender bias didn't exist because only people you told knew your gender. Such gay times they were!
If that was how it was for you, you were hanging out in a pretty crummy part of the internet, which is really sad considering how easy it is to find a different part. You missed out on some quality times.
That is a completely different argument to the one you made originally - "some parts of the internet weren't sexist" vs "people weren't sexist because they couldn't figure out your gender" (which is the one I disagreed with")
Yes, I do remember that (having been using the internet several years before there was any such thing as a web browser), and I dispute your claim, albeit on the purely anecdotal basis of spending years on Usenet.
When you're contributing to a project, performance is only one important thing. It also matters whether you are furthering the goals of the project, communicating well with the other members, and even if you have the social skills to lead a project.
that counts to performing good in a job, too.
It's just I hate people who have biases against, anything.
What I really wanted to say is that I don't get these articles, people should never evaluate people against their outside. no matter if an IT job or anything. the outside appearance has nothing todo with how you really are.
and i dislike it so much. I mean yes I know that some groups of people can't go well with others, but thats just something in our minds, because of our past.
Because it's a well-known factor for discrimination, to the point of limiting people's opportunities to perform in the first place. Until there's rough gender parity that resembles the distribution of gender in the general population fairly closely, you're likely to see significant structural inequalities that hinder open participation in the first place.
Myself, I view this as a big problem and think it needs to b researched and actively mitigated. I'm a man, but I score much more like a woman on transgender-related psychological scales despite not taking hormones or engaging in SRS. When I play online videogames I sometimes identify as female (based largely on whim) and I can't help noticing that my female avatars have to put up with an exponentially larger amount of random bullshit from other players. I'm not cool with this, but neither do I want to live in a world where people feel the need to obscure or neutralize their gender, since that's a fairly fundamental part of most people's identity.
Here is one (IMO) much more complete abstract (not to call it tl:dr) that covers the main hypothesis, and all the variables studied
Our main research question was: to what extent does gender bias exist among people who judge GitHub pull requests?
To answer this question, we approached the problem by examining whether men and women are equally likely to have their pull requests accepted on GitHub, then investigated why differences might exist.
We extract users' email addresses from GHTorrent, look up that email address on the Google+ social network, then, if that user has a profile, extract gender information from these users' profiles. Out of 4,037,953 GitHub user profiles with email addresses, we were able to identify 1,426,121 (35.3%) of them as men or women through their public Google+ profiles.
== Results ==
Are women's pull requests less likely to be accepted?
The hypothesis is not only false, but it is in the opposite direction than expected; women tend to have their pull requests accepted at a higher rate than men!
== Do women's pull request acceptance rates start low and increase over time? ==
[B]etween 1 and 64 pull requests, women's higher acceptance rate remains. Thus, the evidence casts doubt on our hypothesis.
== Are women making pull requests that are more needed? ==
[T]he result suggests that women's increased success rate is not explained by making more immediately needed pull requests.
== Are women making smaller changes? ==
(...) Women make pull requests that add and remove more lines of code, modify more files, and contain more commits.
== Are women's pull requests more successful when contributing code? ==
(For instance, changes to HTML could be more likely to be accepted than changes to C code, and if women are more likely to change HTML)
[W]omen's acceptance rates dominate over men's for every programming language in the top ten, to various degrees
== Is a woman’s pull request accepted more often because she appears to be a woman? ==
For insiders, we observe little evidence of bias when we compare women with gender neutral profiles and women with gendered profiles, since both have about equivalent acceptance rates.
For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable. There is a similar drop for men, but the effect is not as strong.
Women have a higher acceptance rate of pull requests overall (as we reported earlier), but when they’re outsiders and their gender is identifiable, they have a lower acceptance rate than men.
I hope that helps those who won't read the whole paper but feel the need to comment on it.
My opinion on the paper is that the abstract doesn't reflect adequately the content of the paper.
The last assertion, that the "results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless" seems to indicate that the paper results didn't confirmed the desired hypothesis but that there was need to highlight any incidence of gender bias anyway.
The conclusion that there is gender bias on the acceptance of pull requests by women while outsiders (as opposed on women as insiders or men in general) seems to be excessive sub-categorization in order to find a supporting result as the hypothesis doesn't hold for women as whole nor for the subcategory of identifiable women.
It is a very good result, demonstrating that, at least for something so unrelated to the average workplace as Github repositories can be (being in general remote, voluntary and more likely than not to liberal in tendencies due to the FLOSS philosophy), that an useful line of code is an useful line of code, regardless of who wrote it.
Sigh. Authors are unaware of Simpson's paradox. Such analyses are meaningless if you don't control for such obvious factors as the distributions of projects people of different genders choose to contribute to. See the classic college admissions bias study to understand why: https://en.m.wikipedia.org/wiki/Simpson%27s_paradox
You have a good point, but you've used the format of a middlebrow dismissal to express it—i.e. acting like others don't know something well-known, and snarking about it. That makes it a bad HN comment even if you're right. A better HN comment would bring up the issue neutrally and then either examine the OP to see how it deals with it, or ask a question about how it does.
Grandparent, you should have said something along the lines of "authors didn't give an indication of knowing the Simpson's paradox even though their paper was a great opportunity to do so". Or hear from forum lawyers :)
Can you give an example of a type of project which would theoetically have different gender distributions of contributors than other projects? That doesn't seem obvious to me.
Have no idea. But also have an inkling that distributions would differ (and not necessarily the way you'd expect) in projects that require a specialized skill set, such as e.g. top notch design skills or knowledge of DB internals and such.
The point being, as stated the whole thing very much resembles the Berkeley gender discrimination study, so proposed improvements are quite obvious: break things down further by project, see if you can glean something from per-project (or at least per-domain) distribution. There might be nothing there, mind you, and the stated conclusions might still hold, but without controlling for the confounding factor there's no way to tell, and the conclusion of this study is therefore not above suspicion.
A good start is to call out misogyny when you see it, online or anywhere else.
This is disappointingly rare, in my experience.
Edit: what a shotgun blast of downvotes and straw men. People are lining up to say all sorts of utter bullshit. For example:
* "Where does this indicate misogyny?"
* "The paper is about GitHub, which has a reputation for the opposite."
* "A good start is to stop calling things misogyny just because you're bad at reading statistics."
* "Of course it's important to recognize when it's actual misogyny and not just disagreement with the code."
The paper's findings are clear as a fucking bell: women have equal acceptance of pull requests, except when the pull request obviously comes from a woman. Fucking TEXTBOOK misogyny, and you're an idiotic trog if you don't think so. (Looking RIGHT AT YOU here, pluma.)
You can flag them now. The karma threshold is 30 and you're well over that.
To flag a comment, you need to click on its timestamp to go to its page, then click the 'flag' link at the top. A couple of restrictions: (1) you can't flag direct replies, and (2) flag links disappear after 2 weeks, which is also when voting expires on a post.
>Our analysis (not in this paper -- we've cut a lot out to keep it crisp) shows that women are harder on other women than they are on men. Men are harder on other men than they are on women.
Did you read the paper? Where does this indicate misogyny?
Alternative hypothesis: Men are harder on other men because they can be more honest, with less fear that any criticism will be misconstrued into something that ends with an -ism.
You should avoid sarcasm on HN because it doesn't lead to useful or interesting discussion, and because it's indistibguishable from the viewpoints some people actually hold.
Note that the first point is still true if you include the /s tag.
Conflating gender bias and misogyny is wrong. Gender bias is wrong, but people could have unconscious gender bias, which they need to work on and correct.
However, to call them misogynists is completely wrong and unhelpful and will only cause them to dismiss anything further. Misogyny is when men truly think women are beneath them, like that men's group that recently wanted to legalize rape.
For example, recently someone came into a chat room and announced "Hello gents". I immediately corrected him with "You mean "ladies and gents"" and he apologized profusely. This is unconscious gender bias, it's not misogyny.
Call gender bias what it is, gender bias, but don't overreact and call it misogyny when it's not. There are true misogynists out there, and to put those two people in the same camp is wrong and unhelpful and will only drive people away.
That's a fair distinction, but I feel the core of my point remains - to say that bias doesn't exist or distract from that fact with other arguments is utter hogwash.
The amalgamation of gender bias against women is misogyny. People like to wax poetic about the obviously misogynistic boogeymen like roosh or redpillers, but ignoring the reality of the situation -- that misogyny us upheld unwittingly by people that most would consider 'normal' -- doesn't really help anyone. It's like saying that racism is what the Klan does, not what normal people do!
It is not misogyny, and if you keep conflating them, then you will lose the support of most people with common sense.
Yes, there is gender bias, and yes we need to fight it vigorously. I do not want my daughter living in a world where she is considered an afterthought, or even a single atom inferior to a male. I want her growing up knowing that she is a full equal to any male on every level. And it's an uphill battle because of gender bias.
Not all misogyny is gender bias, but I think there is a surjection. It's not an uphill battle because of gender bias -- it wouldn't exist at all without it. Socialization is the primary difference between the sexes, and that is how inequality perpetuates itself.
Hm, is your hypothesis that code from women who are identifiable as women is more often disagreeable than code from women who are not identifiable as women?
Consider the method they used to identify gender - they based it on the email being linked to a Google+ account. This rules out most submissions made as part of corporate work.
So if corporate users submit better code than hobbyists then you would expect a drop. And you do see a significant drop across the board for gender-identifiable vs gender-indeterminate contributions.
However rare it might be, the devastating affects of false positives make this a terrible idea. Just ask Tim Hunt, Matt Taylor, or the men targeted by "donglegate".
Anti-sexist reactions are very rare compared to instances of sexist behavior in my experience. If anti-sexist behavior was as rampant and casual as sexist behavior, those kind of blowups wouldn't happen.
There are only two appropriate ways to handle it: if the offense is not egregious, address the incident with the offender in private. If the offense is egregious, or if you do not feel comfortable enough to discuss it with the offender, then you should report it to whatever authority figures are responsible for the context of the incident.
"Calling out" the behavior consistently creates vigilantism, and puts an unfair burden on the accused. In instances of sexism I have never, ever seen the accused treated as "innocent until proven guilty" in the eyes of the online hate mobs.
Tim Hunt's experience was bad enough that he admitted to thoughts of suicide. This is a man fully respected in his field and has a knighthood, but still had difficulty facing the onslaught of hatred.
The paper is about GitHub, which has a reputation for the opposite. Can you point out an instance where you think misogyny wasn't called out on GitHub?
This isn't about GitHub the company, but rather is about GitHub users. I'm not sure if you may be alluding to some of the allegations of misconduct against GitHub employees, but this piece isn't about GitHub employees.
I think the proper idiom is "pouring gasoline on the fire". Especially considering the authors say they intentionally cut the analysis "to keep it crisp" when the analysis actually disagreed with the spirit of the abstract.
>Surprisingly, our results show that women's contributions tend to be accepted more often than men's. However, when a woman's gender is identifiable, they are rejected more often. Our results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless.
If gender isn't identifiable, why is it surprising at all that women have a higher acceptance rate? This is really annoying me, anyone else notice this?
> Our results suggest that although women on GitHub may be more competent overall
and stopped. No. Just, no. I'm inclined to disbelieve any study that claims one gender is better at programming than other, and that claim makes me automatically dismiss the rest of its conclusions.
See, I'm inclined to believe it. If for no other reason than it just seems intuitive. Women who go into technology know it's an uphill battle. They work that much harder and honestly I'd expect their work to be better, on average, particularly for open source projects because they are genuinely interested in improving the code. They always have to contend with the thought, "Will people reject this because I'm a woman? Better make sure it's as good as it possibly can be so they can't reject it based on that."
It's an interesting hypothesis, but I don't think I agree. The selection pool for women is smaller in STEM, therefore less women means less exceptional women.
> particularly for open source projects because they are genuinely interested in improving the code
I reject this on the grounds that there's no reason why women would be more interested in improving the code than men.
> "Will people reject this because I'm a woman? Better make sure it's as good as it possibly can be so they can't reject it based on that."
I'd be surprised to learn that this was the thought process of the majority of women, let alone most of them.
> selection pool for women is smaller in STEM, therefore less women means less exceptional women
This is only true if you assume women in STEM are a random sampling. If you consider survivorship bias, it's quite possible that the cohort of women who persist are better, despite being fewer in number.
The same is also true for men, unless you start with the belief that the misogyny is a baseline factor that women have to deal with. But that's what this analysis is trying to figure out. Chicken and egg maybe?
The argument seems to be mathematical, not categorical, this is what I think is being claimed -- "there are many more men on github than women, thus there is stronger selection pressures on the women who are on github. Whether men are more or less competent than women IN GENERAL is a separate question from which is, on average the more competent sub-population on github."
Of course they are. But if there is significant discrimination against women, maybe women either get really good or leave OSS/programming altogether. Leading to the remaining women (the sample) being "more competent".
Actually, if you look at their numbers (or the summary numbers provided by the top-voted comment), the statement "women on github are more competent than men" seems to be a result of selection bias rather than "one sex is better than the other".
It appears there are more than 150k men on github but less than 10k women. But if you look at real life, men don't out-number women 15-to-1, which seems to suggest programming on github draws in more men than women. I suggest that, whereas in addition to professional men, tons of boys will make a hobbyist account on github to leave inane comments on open-source projects, there are very few hobbyist girls that fall into this category; that is, a woman only makes an account on github if she is a professional in the industry.
Naturally, if you compare a group of professionals against a group of professionals + college-rockstar-ninja-guru-hackathon-"yeah, but is it optimized?" amateurs, the group of professionals without the baggage would be "better".
Anyway, that's my take on what the numbers say; maybe I'm wrong, in which case, I for one welcome our new female overlords; if they must put us all to death, may they do so with snu-snu
"Women on GitHub" != "women". Nobody is suggesting women are better at programming. The results are showing that women who submit GitHub PRs are more competent. There could be any number of reasons for that:
- women are more timid, so they don't even try submitting a PR until they're sure their work is good
- less determined women (who would perhaps also be less competent) are turned off by the culture of the industry
- women in the industry hide behind gender-neutral profiles until they have more experience
It doesn't claim that. You're conflating "women on GitHub vs men on github" with "a random sampling of men and women".
Given that in the above study there seems to be dramatically fewer women than men contributing to GitHub, take the assumption that in the general population men and women have similar distributions of programming competence, and it seems women are more strongly selected against to contribute on GitHub than men. It seems reasonable that the women who push would have traits associated with greater competence. Thus men and women have equal ability, but women on GitHub can on average be more competent than men on GitHub.
You're being downvoted, but what if the result was "Our results suggest that although men on GitHub may be more competent overall"?
Then you'd probably be upvoted.
Also, I refuse to believe that genders are obviously different in various physiological and mental traits (e.g. height, spatial awareness, ability to host a baby) except the ones that are socially uncomfortable to us.
I suspect you're right about the first part. That makes me sad, because I'd hope we'd be more even-keeled as a community.
I haven't seen any evidence that the second is true, though. I've not heard of a plausible mechanism by which - all else equal - men would be better at abstract reasoning than women or vice versa.
I haven't seen any evidence that it's false, though, either, and there are obviously physical and mental differences between sexes/races/whatever, so it seems more likely to me that this would follow the rule, rather than be the exception.
> I'm inclined to disbelieve any study that claims one gender is better at programming than other
The paper makes no such claim. It hypothesizes that women on GitHub may be more competent than men on GitHub.
There is a ton of selection pressures involved that determine which set of men and women on Earth end up in the set of people that contribute to open source code on GitHub.
So, even if it turns out that one gender is wildly more competent on GitHub than the other, that may or may not say anything about biology or overall gender. Instead, it may simply point to some gender-asymmetry of those selection pressures.
Your summary of the claim is not the same as the claim. "Women on github" are not "women in general."
There is a well established idea that glass ceiling-type barriers require higher competence from women to push through the barriers holding them back than men face.
Though I suspect you'll find the last sentence of this quote off-putting, it's often summarized as the following:
> “Whatever women do they must do twice as well as men to be thought half as good. Luckily, this is not difficult.” - Charlotte Whitton.
IMO that's a knee-jerk reaction to which you shouldn't succumb. The study isn't saying that "one gender is better at programming"; it's saying there's weak evidence that GitHub users of one gender might be slightly more competent.
There are many reasons that might be true without any gender being better programmers in general.
I love the way that you decided to dismiss the claim on the basis of your personal prior beliefs without bothering to assess the quality of the evidence proffered in support of it. This is not to say the claim is necessarily correct (I see some people critiquing bits of the methodology and of course extraordinary claims require extraordinary evidence), but it's unusual to see such a clear statement of cognitive bias as to cause someone to admit up front that they literally chose to ignore the evidence because they were so repelled by the claim of what it demonstrated.
My contentious "personal prior beliefs" are that men and women are equally good at science. My cognitive bias is that I am highly suspect of evidence that purports to show that one is better than the other at thoughtwork. I find that repellant and against everything I've been raised to believe true. So yes, you've got me on this one.
Well I have the same priors but I'm willing to at least consider the possibility that I'm wrong about this. Admittedly I'm married to a lady with EE degree so this wouldn't be the first time I've had to consider that. Inb4 n=1 etc. :-)
Yeah, my mom was an EE-in-practice and my wife's a surgeon. I'm totally open to being proven wrong. :-) I just don't think this is the study that's going to do it.
I am wondering if the data had been the reverse would they have ever made that claim, or would the difference had been explained as the result of some other process?
I guess competent is meant to be shorthand for "likely to get a Pull Request merged" - I agree that competent is a overextension of that, and very subjective. The actual study is more thoughtful than the abstract might suggest.
Why? Brain structure is different there might be other real differences. However, the data doesn't show that at all. It merely shows acceptance is higher. I am therefore left wondering are these non-controversial changes? For example, how many of these are documentation changes?
I don't know if that's true, but I haven't heard a viable explanation about why the opposite would be plausible, either. I'm not a neuroscientist, but I don't know of any reasons why one gender would be better at mental tasks than another.
Look at nature. Sometimes the male is dominant, sometimes the female is dominant (or whatever feature you want to focus on). We are different. There is nothing wrong with that.
I find this suspect, because, anecdotally, I can look almost any author of a PR and determine their gender to a high degree of certainty. Probably well over 90% of the time, just from their name + profile picture + handle. Try it for yourself..look at the latest commits on a random project and see how obvious the genders are most of the time.
So the claim that "when a woman is identifiable, PRs are merged less" is totally suspect, because they themselves can only identify the genders of a small percentage relative to what a normal human can identify. If people can identify the genders way more often and accurately, then the claim being made is bogus. Perhaps there is a correlation of strongly signalling your gender (to the point where an inaccurate method of gender-identification has no problem) to being a below average developer.