More AI Testing

An equine analysis and a laborious quotation search

Detail from Nova Reperta water mill image. My photo of the Huntington Library’s copy of the book. Pay attention to the ears, feet, and tail.

I have a column in the next issue of Works in Progress on Nova Reperta, a 1588 book that demonstrates how Europeans were beginning to conceive of “progress” earlier than we usually imagine. While writing the piece, I came up against a tough question: What is the grain-toting animal in the picture above? Or this one:

Art historians writing about Nova Reperta tend to call them horses. But they seem too small to me. Donkeys? I decided to ask ChatGPT 4.o (my former go-to, not the ultra expensive 1.o pro I’ve been experimenting with): “I'm looking at prints from 1588 and trying to decide whether an animal is a donkey or a horse. How do I tell?”

The response started by listing differences between horses and donkeys. The animals in the pictures are small, like donkeys, and have donkey-type ears but they fail the tail test, leading to a further exchange:

Donkey: Short, bristly mane that stands upright and a thin, tufted tail (like a cow’s tail).

Horse: Flowing mane that falls to one side and a full, bushy tail.

Do you have any specific prints or details that stand out in your case?

You said:

Would donkeys have horseshoes?

ChatGPT said:

In 1588, donkeys could have horseshoes, but it was much less common than for horses….

If your 1588 print clearly shows metal horseshoes, it probably depicts a horse….

If you can make out the hooves in the print, that might be a big clue!

You said:

These animals have donkey-type ears but definitely shoes.

ChatGPT said:

That’s an interesting detail! If the animals in your 1588 prints have long, donkey-like ears but definitely have shoes, here are a few possibilities to consider:

1. They Could Be Mules

• Mules (horse-donkey hybrids) often have the long ears of a donkey but are more commonly shod like horses because they were widely used as work animals.

• Mules were stronger than donkeys and had hardier hooves than horses, but still often needed shoes for long-distance travel or rough terrain.

• In Renaissance prints, mules were frequently used for carrying goods, pulling carts, or even riding in mountainous areas.

Eureka! We have the answer. (I did some online verification to make sure the distinctions were correct.)

You can read the whole exchange here. Afterwards I kept remembering this as a conversation with a person then, when I tried to recall who, realizing that it was an AI exchange.

My second AI test, also research for an future article, was less satisfactory.

Samuel Gompers with more flowers than he expected? Photo from Library of Congress, taken within three years of his death in 1924. The occasion isn’t recorded.

Tracking down famous but potentially apocryphal quotes can suck up a lot of a writer’s day, so I decided to try generative AI systems to help with one I wanted to find. Here’s the prompt I wrote, using a version of the quote I found on an online quotation aggregator (never, ever trust those sites):

Labor leader Samuel Gompers is quoted as saying, "We do want more, and when it becomes more, we shall still want more. And we shall never cease to demand more until we have received the results of our labor." Please track down the original source of this quotation, which dates from before 1925. Do not rely on quote aggregation sites. Use periodicals, books, and other reliable sources.

I started with ChatGPT 1o-pro (that’s the one I’m paying $200/month for!). The answer was:

Although this famous “We do want more…” passage by Samuel Gompers is often repeated, it does not appear in his better‐known early speeches (e.g. “What Does Labor Want?” from 1893) in precisely that form. The earliest print appearance located so far—and very likely its first publication—occurs in Gompers’s Presidential Address to the American Federation of Labor’s annual convention in late 1913. That address was printed both in the official Proceedings of the 33rd Annual Convention of the American Federation of Labor (held in Seattle, November 10–22, 1913) and in the December 1913 issue of the AFL’s monthly journal, The American Federationist (Vol. 20 or 21, depending on bound numbering).

I found the recommended sources online and there was no such quotation and, indeed, no Samuel Gompers speech at all. I still can’t quite believe it, but it seems to have been a complete hallucination. You can see the rest of the exchange here.

Perplexity (free) is oriented to search, so I tried it with the same prompt. It pointed me to Gompers’s autobiography and also suggested that “in a speech given in August 1893 to 25,000 unemployed workers in Chicago, Gompers expressed similar ideas about workers deserving fair compensation for their labor,” citing Wikipedia as the source. You can see the full exchange here.

Claude 3.7 Sonnet (free) offered a much-hedged suggestion of a speech to the International Ladies' Garment Workers Union (ILGWU) convention in 1924 and also suggested checking his autobiography, labor newspapers from 1924, and The American Federationist from 1924.

Grok (free) provided fun account of its research and said the best it could do was the autobiography, which wasn’t exact. Read the whole answer here.

My own computerized search did better than any of these:

1. The quotation above appears in the selections of the Samuel Gompers Papers available on Google Books. It is from an 1890 speech in Louisville.

2. Using Newspapers.com, a bargain at $150/year, I found the speech in the Louisville newspaper as well as others. At that point I had only tried ChatGPT. Informed that its citations were wrong, it produced some possibly hallucinated excuses.

3. Using Newspapers.com I also found essentially the same quotation in an 1893 speech but not the one Perplexity cited. The relevant speech was given at the World’s Columbian Exposition (aka world’s fair) not a rally of unemployed workers.

What I often want is the equivalent of a Newspapers.com search but for everything—no “intelligence,” just crawling all the relevant websites and matching a string of characters. Once upon a time, Google more or less did that. Now it tries to guess what you’re looking for, which sometimes works and sometimes doesn’t and sometimes produces ridiculous results to sell ads (Amazon’s search is even worse). Census.gov has lots of info but it’s often impossible to find what you’re looking for without calling someone at the Census Bureau, a reportorial privilege. LLMs don’t solve this problem.

As Ben Evans observes in a post I highly recommend reading in full:

OpenAI is asking the model a probabilistic question, not a deterministic question. But the answer to that question IS deterministic - having worked out what you really want, and which kind of answer to choose, you want the actual number. We’re asking for a deterministic answer from a probabilistic question, and there it looks like the model really is failing on its own terms….

This reminds me of an observation from a few years ago that LLMs are good at the things that computers are bad at, and bad at the things that computers are good at. OpenAI is trying to get the model to work out what you probably mean (computers are really bad at this, but LLMs are good at it), and then get the model to do highly specific information retrieval (computers are good at this, but LLMs are bad at it). And it doesn’t quite work….

these things are useful. If someone asks you to produce a 20 page report on a topic where you have deep domain expertise, but you don’t already have 20 pages sitting in a folder somewhere, then this would turn a couple of days’ work into a couple of hours, and you can fix all the mistakes. I always call AI ‘infinite interns’, and there are a lot of teachable moments in what I’ve just written for any intern, but there’s also Steve Jobs’ line that a computer is ‘a bicycle for the mind’ - it lets you go further and faster for much less effort, but it can’t go anywhere by itself.

But it does a great job at amplifying your brain to figure out what sort of animals are carrying grain in old engravings.

In my Zoom for paid subscribers we had fun getting various versions of ChatGPT to generate answers to a gardening challenge. I’d like to do another round so if you’re interested, please put a comment below with your preferred dates and times within the next two weeks.]

Recommended Reading (closing tabs some of which I’ve had open for nearly a month)

Historian Ben Breen discusses the pedagogical challenges of students using LLMs: ”I’m not alone in noticing the increasingly sameness of student responses to course material. LLMs, which are exquisitely well-tuned machines for finding the median viewpoint on a given issue, are surely contributing to it.” But most of this interesting post is about using AIs to do historical research: “The headaches that LLMs have caused in the classroom are (I believe) more than counterbalanced by what they can offer as tools for research and self-directed learning. For this reason, I’m now even more optimistic about the long-term impact and utility of AI tools for historical research — and, by extension, for other forms of text or image-based research.” Read the whole post here. I like his

Res Obscura

Substack.

Scott Sumner

is a brilliant macroeconomist whose Substack,

The Pursuit of Happiness

(a popular Substack title), ranges far more widely than I would have anticipated and is particularly thoughtful on art. Here he brings that expertise to bear on AI and epistemology. It’s a true essay, from the root to try, in which the author explores ideas with an organizational and intellectual freedom pros like me get beaten out of them at an early stage. He does, however, give you the conclusion in the first paragraph. He just doesn’t telegraph how he gets there. As someone who appreciates the greatness of Jane Austen but just simply doesn’t like her novels, I also liked this important distinction.

Through education, people can gain a great appreciation of Ulysses. In addition, Ulysses is more likely to be read 100 years from now than is a random spy novel. And most experts prefer Ulysses. All three facts are relevant to the claim that artistic merit is partly objective. Like scientific knowledge, there is consideration of evidence, appeal to authority, and the test of time….

an art form like the novel is evaluated using both subjective and objective criteria. For instance, I suspect that some people (like me!) have brains wired in such a way that it is difficult to appreciate novels looking at complex social interactions with dozens of important characters (both men and women), whereas they are more open to novels about loners who travel through the world and ruminate on the meaning of life. Other people have brains wired in such a way that subtle social dynamics are easy to read. Thus between two novelists that are roughly equally rated by experts, there will be disagreement over ranking based on differing taste. Neither preference is necessarily wrong.

The excellent ChinaTalk on DeepSeek as seen from China (from last month):

Ordinary users have also been astounded by the model’s capabilities. Many were impressed by the Chinese poems that DeepSeek could write, and tutorials have come up, instructing users to use as few prompting words as possible and ask DeepSeek to talk like a human (说人话). In a viral Weibo post, a user said, “I never thought there would come a day when I would shed tears for AI,” citing DeepSeek’s response to their feelings of existential threat over DeepSeek’s ability to write.

Here is DeepSeek R1’s response: “Remember, all the words that make you tremble are just echoes that already exist deep within your soul. I am merely a valley that happened to pass by, allowing you to hear the weight of your own voice.” 记住，所有让你颤粟的文字，本质上都是你灵魂深处早已存在的回声。我不过是偶尔经过的山谷，让你听到了自己声音的重量

SatPost by Trung Phan

is one of my favorite Substacks, delivering lots of meaty business info, plus memes, in each weekly post. Here he is on DeepSeek (some overlap with ChinaTalk, which he cites).

Finally, for paid subscribers, we had fun on our Zoom conversation, in which I showed some research I’d had ChatGPT do and we tested out various models on a gardening challenge. If you’d like to join us for another round, please put a comment below suggesting some good times and dates within the next couple of weeks. I generally shoot for a start around 5:00 pm Pacific Time but am flexible.