1976
Joseph Weizenbaum, Computer Power and Human Reason, W. H. Freeman Co., San Francisco 1975
The idea of a stored program computer leads immediately to studying mental processes as abstract computer programs. Artificial intelligence treats problem solving mechanisms non-biologically, and modern cognitive psychology makes information-processing models of the human mind. Alan Turing and John von Neumann thought of this even before the first computers were working. Both studies proved fruitful though difficult and have been pursued with ever increasing vigor.
Progress in either study, like Darwinism and like most progress in medicine and biology, moves the scientific picture of man's nature directly away from the subjectivity preferred by modern literary culture. Full success, like successful genetic engineering, will present individuals and society with a bewildering collection of options. Weizenbaum fears the options he can imagine and the rationalist world-view that computer-modeling reinforces.
He criticizes all present work in artificial intelligence, information-processing-based psychology and computer linguistics as mere technique. In particular he regards the computer linguists as hackers whose work there is no point in studying, but he explicitly puts no limit on the potential problem-solving capability of computers except when understanding humans is required. His point is moral, and his arguments use the 1960s technology of moralistic invective.
He finds it immoral for a scientist to adopt certain hypotheses even tentatively, to perform certain experiments or propose certain applications--not because they are dangerous or won't work, but because they are ``obscene''. He distinguishes between not closing one's mind to a hypothesis (OK) and tentatively adopting it (possibly immoral). Also information processing models of man are OK in principle provided one recognizes that they can't model any ``authentically human concern'', but no work meeting his criteria is mentioned.
The objectionable hypotheses, experiments and applications include the theory that man is a simple organism in a complex environment, the idea that all reality can be formalized, the idea that what a judge knows can be told to a computer, some experiments with recombinant DNA, connecting animal brains to computers, psychological testing, and using a computer program for psychiatry. Here are some of the arguments:
On psychiatry-- ``What can the psychiatrist's image of his patient be when he sees himself, as a therapist, not as an engaged human being acting as a healer, but as an information processor following rules, etc.?''
On connecting computers to animal brains--``The first kind'' [of application] ``I would call simply obscene. These are ones whose very contemplation ought to give rise to feelings of disgust in every civilized person.''
On a proposed moratorium on some DNA experiments--``why do they feel they have to give a reason for what they recommend at all? Is not the overriding obligation on men, including men of science, to exempt life itself from the madness of treating everything as an object, a sufficient reason, and one that does not even have to be spoken?''
On science in general and pure science in particular--``Not only has our unbounded feeding on science caused us to become dependent on it, but, as happens with many other drugs taken in increasing doses, science has been gradually converted into a slow acting poison.'' and, ``Scientists who continue to prattle on about 'knowledge for its own sake' in order to exploit that slogan for their self-serving ends have detached science and knowledge from any contact with the real world''.
A moral principle--``Those who know who and what they are do not need to ask what they should do.''
Success in modeling the mind will raise policy issues with both moral and factual aspects. However, the public entitled to decide them has more immediate concerns; imagine asking the 1976 presidential candidates to debate whether computer programs should do psychiatry while there are none that can. When they become concrete, they must be discussed in terms of costs and benefits and not in terms of ``obscenity''.
As in Darwin's time, science--especially genetics, psychology, sociology and (now) computer science--is being morally pressed to fit its theories to ``religion''. Many have given in; few will speak out for studying the genetics of human behavior, computer scientists in unrelated fields claimed to have proved that the ABM couldn't work, and physicists claim to show that nuclear explosions can have no peaceful use. When scientists forget their duty to pursue truth wherever the search leads, when they start selecting facts to support comforting world-views or the policies of the good guys, they lose much of their value to society.
John McCarthy Computer Science Department Stanford, California 94305