5.5.1 A Pragmatic View of AI Chatbots: Part 1
What are Chatbots?
Chatbots are automated computer applications that can have text conversations with humans. These systems are based on machine learning frameworks, otherwise known as natural language processing that is a form of artificial intelligence. Chatbots are conventionally activated by a person entering a text prompt into an input window on a webpage. The text input triggers an enormous number of probabilistic calculations and returns a text output as a response. Continuous sequences of inputs and outputs within a single session are classified as ‘chats’. The system can often use the entire content of the ongoing chat as a ‘context window’ to condition the responses of further enquiries within the same session. The term bot is an abbreviation of ‘robot’ and in this context refers to some kind of automated program operating over a network like the internet. These automated bot systems execute operations at speeds vastly exceeding human cognitive processing capabilities.
General purpose commercially available chatbots such as Google Gemini, Claude, and ChatGPT, are now used by hundreds of millions of people, and so have rapidly acquired great cultural significance. They are probably the most common type of machine learning application (or form of ‘artificial intelligence’) that most people will encounter, when searching the internet, pursuing an educational goal, using a machine translation, using speech to text generation, interacting with a company online, or using text in the course of their work. These applications can create flowing, grammatical and apparently coherent and persuasive natural language text output of all kinds. The outputs include very detailed reports on almost any conceivable subject for which publicly accessible text is available. AI chatbots can also generate computer code used in programming and undertake creative writing of jokes, stories, poems, and song lyrics. The reason these applications are able to generate text is that they have been ‘trained’ on vast repositories of information available via the internet. Training allows the chatbots to predict which sequences of words produce a suitable enough output for a human to read in response to his or her text input. In a psychological sense there is no ‘intelligence’ involved since these systems achieve their impressive outputs by probabilistic calculations.
Despite limitations chatbots make excellent proof-readers and will correct spelling, grammar and style, word inversions and duplications, editorial errors generated by copying and pasting and at times offer what might be described as helpful insights on a piece you might have written. If you are a borderline dyslexic like me then they are a very significant editing tool.
Stating the Obvious
In the summer of 2025 one prominent philosopher of science in the UK said to me that he was a philosophical ” reliabilist”. He also made it clear that he did not “trust” chatbots based on large language models because their output could not “be relied upon”. For me the whole idea of ‘trust’ in artificial intelligence (AI) seems inappropriate. I would no more ‘trust’ AI output than I would an inanimate gouge when I am turning at the wood lathe. In both cases I am using an inanimate tool and so need to know how I can use them appropriately, efficiently and safely.
Although the outputs of chatbots are derived from vast amounts of human language use they are not direct stores of books, papers, magazine or webpages, PDF or word processor files or any other kind of text, surprising though that may seem! In other words, there is nothing in the underlying trained generative AI models which power chatbot applications, that can be read by humans since they do not even store machine-readable sequences of words or symbols let alone anything that would make sense to a human. They do not even operate with long words without breaking them into bits! Surprisingly, the information chatbots contain is merely related to the associations of symbols and words as they are used in natural languages, such as English, and in formal logic expressions, or mathematics and in statements expressed in programming languages. Even though chatbots have significant shortcomings, their creation is an incredible feat of those brilliant people involved in the long drive to invent machine learning systems, which work as well as they do today.
Although current chatbots can produce large amount of fluent prose they do not encode direct representations of the physical world. The output of chatbots merely mimics human language use. They are very sophisticated text producing machines with no sensations or human-like awareness of the environment. It is only the human reader with a sensory connection to the world that can claim to ascribe sentient meaning to the text output. However creating a meaningful text stream in response to human generated text is an ability we now share with text generating machines.
One crucial difference between the human and the bot is that we can be self-motivated to initiate language use. At a more functional and pragmatic level we can view the chatbot as a symbol handling machine driven by an enormous information store. The fact that the information is encoded in a way which is alien to human intuitions about learning and memory and reasoning makes the operation of these systems seem all the more remarkable.
Chatbots do not have human-like episodic or procedural memories. They also lack feelings, emotions, motivations, intentions, forward planning, biological drives or survival instincts. Commentators say that they use “confident language” even when making obvious errors. This is unacceptable anthropomorphism since the bots have no sentience or consciousness.
It is a very important feature of chatbots that they generate errors. We can accept this view unequivocally as the companies that produce chatbots append an error warning to all outputs of chatbots, such as the message used by Google ” AI and can make mistakes”. These errors are often inappropriately referred to as ‘hallucinations’. Only biological systems with senses can, by definition, produce hallucinations. Hicks and colleagues point out the chatbots cannot “see” and they cannot “misperceive” because they have no perception. The Danish medical researchers Østergaard and Nielbo point out that the term ‘hallucination’ used in this context “is a highly stigmatizing metaphor” and so should be avoided. The term confabulation is also used to chatbot describe errors. However this term also has overtones of human central nervous system dysfunction resulting from brain injury or degenerative disease which are wholly inappropriate when describing functioning computer systems. Clearly it is better to refer so called ‘hallucinations’ as ‘machine generated errors’. In view of the errors that are produced it would be extremely naive and philosophically imprudent to expect infallibility or even human-like understanding. Nor should we treat current chatbot systems as if they have agency since output is triggered by input from the human agent that has the biological drive to act. Our actions in the world are a prerequisite for survival and reproduction and are just one of a myriad of characteristics that help to distinguish us from text generating machines.
Hype
Every time the CEO of a certain chatbot company makes fabulous claims or issues dire warnings about his company’s applications ask yourself what is the financial motivations in making such statements. When you read tech news articles about the latest changes in AI systems after some minor upgrade, consider the long view. Think of previous unreliable features and ask yourself has the technology fundamentally changed? Are the performance metrics still less than 100%? What does less than 100% imply for your personal use? Do those metrics have any relevance to your personal prompts? If so, what subtle and hard to spot errors are they still likely to generate? In addition, what biases and expressions of ignorance will chatbots reproduce from human sources?
When well meaning people warn that chatbots and other AI systems will be socially disruptive take them very seriously! When others warn that the machines will “want” to “take over”, laugh!, because such teleological claims should be the subject of extreme skepticism. As one member of an AI company said to me, “Where is human agency in that scenario? Just switch them off.”
Text Output of Highly Variable Quality But Not Bullshit
At a philosophy of science conference, I asked the very personable Michael Hicks whether or not he regretted the title of his paper ‘CHATGPT Is Bullshit‘. Unsurprisingly the title has attracted much attention and more than 1 million accesses (!!), which Mike and his colleagues were pleased to receive, so he had no regrets in that regard. The major claim in the paper can be easily expressed; transformer based large language models are not truth-apt. They are instead merely designed to produce fluent text output that makes grammatical sense to readers.
I do not like the anthropomorphic connotation that the indeterminate output of a text generating machine has the same undesirable characteristics as some notorious humans. Humans have a perspective on the physical world and have volition to form language strings that they consider to be honest or deceptive. They can also choose to exercise diligence or indifference in their pronouncements. In other words humans make epistemic choices that are not open to text generating machines. The term BS is therefore best thought of as human-generated, just like “hallucinations”.
I think it is possible Hicks and colleagues have swallowed a myth put about by the AI companies concerning our direct interaction with the ‘AI models’. Chatbots are by definition computer applications in which, I have been told, the companies have “tremendously valuable intellectual property” that extends beyond the weights of a very expensively generated model. For example part of the effort in ‘training’ machine learning models involves so called ‘reinforcement learning from human feedback‘ in which the output of the models is ‘aligned’ to human expectations and preferences. In other words the AI researchers are aiming to generate useful output not BS. It is a matter of empirical assessment as to what extent they have achieved their epistemic goals not a matter of philosophical definition.