There is a file named 'netbeans.conf' which can be found in <NetBeans_home>\etc\ which you can edit to increase the amount of memory that NetBeans runs with. Edit the property named 'netbeans_default_options'. Increasing the -Xms and -Xmx options will increase the amount of RAM avialable to the JVM that NetBeans will be running inside of.
Example:
netbeans_default_options="-J-Xms384m -J-Xmx512m -J-XX:PermSize=32m -J-XX:MaxPermSize=96m -J-Xverify:none"

This example will allow NetBeans to run with a minimum of 384 megabytes of RAM and a maximum of 512 megabytes of RAM.