doug@eris (Doug Merritt) (04/26/88)
Crossposted from sci.lang; edit Newsgroups if appropriate! In article <201@heurikon.UUCP> lampman@heurikon.UUCP (Ray Lampman) writes: >In article <9141@agate.BERKELEY.EDU> doug@eris.UUCP (Doug Merritt) writes: >> I've written a program that categorizes files by the apparent language >> they're written in. > [...] be possible to identify just about any language. Will your program >recognize computer as well as human languages? Yes, it does. It is extremely thorough, compared with the Unix "file" program. But it is biased towards things I can test. Lisp, for instance, is recognized by a simple heuristic to avoid the problem of tangling with the issue of zillions of dialects. It currently recognizes perhaps a dozen common programming languages, a half dozen human languages, transportable file formats (e.g. GIF, arc, sit, pit, zoo, etc), executables (Unix, Amiga, Atari ST, MacIntosh), about thirty (maybe sixty, it's late, my mind is fading) files that are purely native to the Amiga (devices, sound, graphics, animation, fonts, etc), since that's the target machine, and a couple dozen formats from other machines that seem likely to pop up. I've got the initial documentation finished, but I'm too tired to read it to refresh my memory. So I keep trying to stretch its capabilities. I'm currently investigating some information I found about ASCII transliterations of Hebrew, more because I've got the info than because I think I'll run into any Hebrew. If I can dig up cluster frequencies or something, I'd like to recognize ASCII translits of common Romance languages, Japanese and Russian. And maybe others. Why not? Sorry, Pig Latin and Jive show up as English. :-) But it knows the difference between Usenet news files and Unix mail, for instance. Handy. >How about providing a way of `teaching' your program about languages it >does not yet recognize? I'm working on it... >If I can provide a sample text of my favorite >language `X', can your program assimilate the sample and recognize other >samples of the same language? What should the program do if there is no >statistical difference between a language it already `knows' and a new >one you are trying to teach it? Hope some of this is useful, Well, it's impossible to do a perfect job, so inevitably I accept that it'll be wrong sometimes. Finnegan's Wake would give it a heart attack. :-) Also, it would be *very* slow (cpu intensive) to do even the best job possible. So I pick a reasonable tradeoff. As for using it on arbitrary new languages via example, that sounds like a tough objective, but I'll think it over. Porting it to a non-Amiga machine would be mostly easy, just chop off the parts that are system specific or you don't care about (e.g. analysing Amiga device names) and replace a few centralized i/o routines. It's not *quite* ready for full release (still hoping to find more statistics), but I could give out pieces of source code to aid in particular specialized purposes if anyone is dying for it. -Doug- Doug Merritt doug@mica.berkeley.edu (ucbvax!mica!doug) or ucbvax!unisoft!certes!doug or sun.com!cup.portal.com!doug-merritt