There are different types of OutOfMemory errors:
java.lang.OutOfMemoryError: Java heap space
Increase the -Xms and -Xmx. I'd make sure they are set at least 256m and generally it's a good idea to set them to the same value.
java.lang.OutOfMemoryError: PermGen space
Add either -XX:+CMSPermGenSweepingEnabled or increase the PermGen size: -XX:PermSize=256m
java.lang.OutOfMemoryError: GC overhead limit exceeded
Add more heap, the garbage collector can't free enough memory with each cycle. Also try turning on GC logging.
java.lang.OutOfMemoryError: unable to create new native thread
Decrease your heap :) This means that you have too much memory allocated to the heap that the OS doesn't have enough memory to create threads..
Two last things, the above can be configured in jboss/bin/run.conf.
Also when starting JBoss see what -X parameters are being passed to the JVM, it prints this information by default, verify that it's what you expect it to be.