Has anyone developed a JNI class to create custom borders (masked edges) on a java.awt.Window (or better yet: javax.swing.JWindow)?

Not exactly what you asked for, but there is an artile in javaworld that describes how to do custom windows, eg. round windows;

http://javaworld.com/javaworld/javatips/jw-javatip86_p.html

You can probably use a similar technique.