Friedberg (1958 and 1959) discussed a completely general way of representing behavior and provided a way of learning to improve it. Namely, the behavior is represented by a computer program and learning is accomplished by making random modifications to the program and testing the modified program. The Friedberg approach was successful in learning only how to move a single bit from one memory cell to another, and its scheme of rewarding instructions involved in successful runs by reducing their probability of modification was shown by Herbert Simon (a now substantiated rumor froma 1987 personal communication) to be inferior to testing each program thoroughly and completely scrapping any program that wasn't perfect. No-one seems to have attempted to follow up the idea of learning by modifying whole programs.
The defect of the Friedberg approach is that while representing behaviors by programs is entirely general, modifying behaviors by small modifications to the programs is very special. A small conceptual modification to a behavior is usually not represented by a small modification to the program, especially if machine language programs are used and any one small modification to the text of a program is considered as likely as any other.
It might be worth trying something more analogous to genetic evolution; duplicates of subroutines would be made, some copies would be modified and others left unchanged. The learning system would then experiment whether it was advantageous to change certain calls of the original subroutine to calls of the modified subroutine. Most likely even this wouldn't work unless the relevant small modifications of behavior were obtainable by calls to slightly modified subroutines. It would probably be necessary to provide for modifications to the number of arguments of subroutines.
While Friedberg's problem was learning from experience, all schemes for representing knowledge by program suffer from similar difficulties when the object is to combine disparate knowledge or to make programs that modify knowledge.