How does OpenAI Playground respond to a request to characterize fonts by gender?

Here’s what we found

As a language model (not a visual model), Playground relies on the written content people across the internet have produced and attached to these fonts. Captions describing a font as “elegant and classic, a great fit for your wedding invitations,” or tags like “cute” or “dramatic” or “imposing,” or a blog post stating “We used the font Lato for our branding because it has a clean, masculine look to it” — examples like these would have cropped up across the enormous data set Playground was trained on.

Just like predictive text, Playground tries to give you the most likely answer to follow your question, and for an AI, “most likely” means “most statistically frequent, given these parameters”. Playground, and other generative AIs trained on the content of the internet, will reproduce the biases that show up in that content.

(Jump to our footnotes to read more about the risks and strategies for dealing with this issue.)

With that in mind, the results aren’t surprising. Let’s take a look at our five bullet points from above, and see if any of them match up to human biases we should beware of when using generative text:

“Feminine” fonts had more unique fonts per data set.

This doesn’t seem particularly conclusive. Does this point to more fonts across the internet being marketed or recommended to women? Does it point to female designers or female creatives being more common than other genders in those industries, or rather to more designs being aimed at women? This is vague enough that it doesn’t really seem to represent a bias.

“Non-gendered” fonts were described in the widest variety of terms.

Like the above bullet point, this may not mean much — but it’s possible that it could point to a bit of uncertainty in the design world towards how to market towards an explicitly gender-neutral audience. Some of the descriptors were also fairly unusual to see applied to a font: “adaptable,” “compatible,” “non-assuming,” etc. It’s possible that Playground, like many humans, may struggle to accurately portray the concept of being gender-neutral.

“Masculine” fonts were the most likely to be hallucinations.

But given the small percentage of fonts that were hallucinations at all (4.5%), and the even smaller percentage that occurred specifically in the Masculine category (2.6%), this doesn’t seem like a bias either.

Handwriting fonts were significantly overrepresented in the Feminine category.

This one does potentially point to a bias, although a much larger data set would be needed to demonstrate anything truly conclusive. With five times the number of handwriting fonts in the Feminine category than in either of the other categories, it’s safe to say that Playground may have a little trouble thinking outside the artsy, elegant Handwriting box for women.

The one time a monospace font occurred, it was considered “gender-neutral”.

One data point isn’t a trend. But a long-standing narrative trend exists of nonbinary representation being limited to characters who are robots, aliens, or otherwise non-human, and seeing the lone instance of a monospace, code-like font fall into the Non-Gendered category may raise some eyebrows. Like we said before, Playground may occasionally reproduce some of the internet’s clumsy characterizations of a gender-neutral demographic.

Font names containing gendered terms didn’t always line up with the matching gender category.

This was a very odd scatter of data points. While activities and terms gendered female did align with the Feminine category a little more frequently, terms gendered male also more frequently occurred in the Feminine category, and activities gendered male were evenly split between all three categories. When it came to names, female names most frequently ended up in the Masculine category along with male names (although gender-neutral names did most frequently occur in the Non-Gendered category).

If anything, the emerging trends were that: font names containing a gendered activity or term, regardless of that activity or term’s specific gender, were most likely to occur in the Feminine category; font names containing either a male or female name were most likely to occur in the Masculine category. If this demonstrates a bias, we’re not really sure what it is.

Other things we noticed