AI as critic

I listened recently to a radio program in which a collection of prominent entrepreneurs discussed some of the key examples this year of overhyped marketing. As an aside, one of the participants mentioned the disturbing error rate in ChatGPT’s responses to questions of fact, e.g. name ten distinguished alumni of the Fitzwilliam College, Cambridge? It’s interesting how some commentators judge claims to intelligence on the basis of accuracy with facts.

I now resist the temptation to ask questions of the LLM. I get more out of it if I tell it things and see what response I get. That’s treating the AI as a critic rather than a flawed information source.

This post is short as I’m working on a book on AI in the urban context. One chapter seeks to summarise the main features of LLMs and relate these to aspects of the city.

I called on ChatGPT to anticipate objections to the content and delivery of the chapter. It highlighted the limitations of my summary and advocated for more detail. Its hard to unfold a narrative where you expect to outline the detail later on in the work.

The main criticism focussed on my comparison between human language learning and LLM training, which it suggested could be construed as oversimplified. It suggested that I remind the reader that there are social, cultural, and cognitive factors that are not directly comparable to the statistical learning processes in LLMs.

My response was to note that the complexity of human language acquisition and use is not in dispute. Rather, it is significant that the functions I have outlined seem to be sufficient for AI language models to deliver performance that resembles and exceeds certain aspects of intelligent human conversation.

That high level of LLM performance indicates the extent to which the corpus, the source text and the language it deploys, embodies the social, cultural, and cognitive complexities of human language development and of human discourse.

Though the developers of LLM methods have created algorithms that effectively operationalise very sophisticated pattern detection and processing, the prime cognitive claim of these platforms resides in the authorship of the source texts on which the models are trained rather than the algorithms. That observation is a further endorsement, if we need it, of the power of human culture and of the instrument of human language.

This to me is a reasonable use of conversational AI in academic writing. It can offer review and critique quicker and more comprehensively than one might expect from a colleague or human reviewer. It is then up to the writer to amend or otherwise respond to the critique in their own words.

Note

• WordPress’ AI also offered a critique of this post: “The content provided offers a valuable reflection on the use of conversational AI in academic writing. The comparison between human language learning and LLM training could benefit from acknowledging the distinct social, cultural, and cognitive factors at play. Additionally, delving further into the limitations of the AI’s summary and incorporating more detail may enhance the chapter. Consider addressing the critique while emphasizing the unique capabilities of AI as a reviewer, ensuring a thoughtful response in the writer’s words.”
• Featured image is by ChatGPT: “Here is the graphic banner you requested, depicting a database of authoritative texts with a grunge, post-apocalyptic aesthetic. If you need any adjustments, please let me know.”