htdig is a native (C), lucene is written in Java. I understand Lucene is actually the 5th generation of a project that began many years ago by an expert in NLS technology. I have a strong suspicion that Lucene is very sophisticated, perhaps a little more than htdig, and I also have some doubts as to how sophisticated the ordering of searches are in htdig as compared to lucene, considering lucene is probably friendly to i18n standards of ordering and the best htdig will be able to do is a lexigraphic ordering. You won't notice the difference in English, but there will be a big one in Chinese.

I have used lucene on my website. The API was so easy I was up and running in about 3 hours, and I coded it all in JSP (lazy, but works).