Can I use a in-house CA to sign a JCE provider implementation?

You can, but if I'm not mistaken that provider will be rejected by Sun's JCE implementation. Because they require trusted certificate signed by root sun's certificate. That's the idea why providers should be signed.

Selfsigned certificate might be created by standard keytool from the Java SDK.