oracle.apps.ad.util.TimedProcess$TimedProcessProcess Not enough space

When I ran ./adstpall.sh I was getting the below errors.

and when checked the log, I see the below strange messages.



Executing service control script:
/product/app/ICMQC/inst/apps/ICMQC1_icmqapp02cdp/admin/scripts/adopmnctl.sh start
Timeout specified in context file: 100 second(s)

RC-50208: Exception in method TimedProcessProcess.run
Raised by oracle.apps.ad.util.TimedProcess$TimedProcessProcess: java.io.IOException: Cannot run program "/bin/env": error=12, Not enough space


RC-50208: Exception in method TimedProcessProcess.run
Raised by oracle.apps.ad.util.TimedProcess$TimedProcessProcess: java.io.IOException: Cannot run program "/product/app/ICMQC/inst/apps/ICMQ
C1_icmqapp02cdp/admin/scripts/adopmnctl.sh": error=12, Not enough space
script returned:



### hmm strange I have enough space of disk then why "Not enough space" errors.

checked few other things like DB,Listener,space all is clean.

Now when I tried to start the individual process some came up and some errored out saying they are already runing even though they were not running.



/product/app/ICMQC/inst/apps/ICMQC1_icmqapp02cdp/admin/scripts/apmnctl.sh start

06/03/11-13:14:40 :: adapcctl.sh: starting OPMN if it is not running

opmnctl: opmn is already running.


Now I tried to stop using adstpall.sh


icmqapp02cdp /product/app/ICMQC/inst/apps/ICMQC1_icmqapp02cdp/admin/scripts >./adstpall.sh

You are running adstpall.sh version 120.10.12010000.4


Enter the APPS username: APPS

Enter the APPS password:
Error occurred during initialization of VM
Could not reserve enough space for object heap

adstpall.sh:Exiting with status 1


ahaaaaaaaa, here we go now I got the real error so" we dont have memory for heap"


Since we have three applications running on the same VM, decided to cut down the heap footprint for one of the application:

In the context_file on the app nodes:
changed from:

-server -verbose:gc -Xmx2048M -Xms512M

to

-server -verbose:gc -Xmx256M -Xms64M


for all oacore,jvm,opmn and so on

ran the autoconfig.sh to make sure these takes effect.



icmqapp02cdp /product/app/ICMQC/inst/apps/ICMQC1_icmqapp02cdp/admin/scripts >./adautocfg.sh
Enter the APPS user password:
-----

AutoConfig completed successfully.




Now tried the adstpall.sh

icmqapp02cdp /product/app/ICMQC/inst/apps/ICMQC1_icmqapp02cdp/admin/scripts >./adstpall.sh

You are running adstpall.sh version 120.10.12010000.4


Enter the APPS username: APPS

all the processes were brought down cleanly.......



adstpall.sh:Exiting with status 0



Now tried to start the services:

All enabled services for this node are started.

adstrtal.sh: Exiting with status 0


all came up clean..I am a happy camper.

lesson learnt, we have more applications running on the same box, try to do the math and allocate the sizes accordingly.