GARVEY@SRI-AI.ARPA (Tom Garvey) (06/21/86)
I think the notion of V&V for expert systems highlights a number of points about the field. First, in the words of David Mizell (formerly of ONR), "AI is being overbought." People that should know better are taking an attitude that there are sufficient useful AI systems out there that we should be concerned with formal notions of their capabilities. In point of fact, AI is very much a research topic (I almost said science), and for most problems we are struggling to find any solution at all, much less one that will be operationally useful and verifiable. The traditional rationale for attempting an "AI" solution to a problem is that we don't know how to solve the problem directly (if we did, why screw around), or that our problems come from a large class of ill-specified problems where flexibility in the problem-solving approach is of paramount importance (otherwise, ...). AI approaches typically involve non-deterministic processes such as context-sensitive search (frequently in large, ill-structured knowledge-bases), and their performance is therefore extremely difficult to describe much less quantify. (We don't do a very good job of V&V on deterministic systems yet.) Even statistical validation (i.e., try a million random test cases and measure resulting performance) will be questionable, as characterizing an appropriate set of test cases spanning the range of possible or likely inputs will be extremely difficult. At this point, I view most expert system development as not much more than programming in a new language. The language offers ease of specification and representation of certain types of information (oops, knowledge), but does not lend itself well to either V&V, maintenance, or robust operation. To the extent that we use expert system developments to help understand and structure problems, these shortcomings are not too significant; to the extent that we view the systems as the solutions themselves, the shortcomings are significant. All this doesn't help your quest much, but perhaps it will help lower your expectations. Cheers, Tom