srvctl tracing
srvctl fails on one node but works on another
./srvctl
./srvctl: line 179: $JRE $JRE_OPTIONS -classpath $CLASSPATH $TRACE oracle.ops.opsctl.OPSCTLDriver “$@”
the obvious checks were already done :
check timestamp of srvctl and its size against another node
diff srvctl against another server
check version of java & path to version of java
environment variables match
o/s strace of srvctl did not give any clues to the problem.
other crs commands worked e.g. crsctl
no failures in /var/log/messages
To enable oracle tracing of srvctl:
oracle +ASM1 > export SRVM_TRACE=true (MOS 178683.1)
oracle +ASM1 > srvctl
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
oracle +ASM1 > free -m
ps -ef | grep java
(pid 17377 is process for the oc4j CRS process)
oracle  17377    1 0 09:22 ?       00:00:30 /oracle/GRID/11203/jdk/jre//bin/java -server -Xcheck:jni -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Dstdstream.filesize=100 -Dstdstream.filenumber=10 -DTRACING.ENABLED=false -Doracle.wlm.dbwlmlogger.logging.level=INFO -Dport.rmi=23792 -jar /oracle/GRID/11203/oc4j/j2ee/home/oc4j.jar -config /oracle/GRID/11203/oc4j/j2ee/home/OC4J_DBWLM_config/server.xml -out /oracle/GRID/11203/oc4j/j2ee/home/log/oc4j.out -err /oracle/GRID/11203/oc4j/j2ee/home/log/oc4j.err
JRE_OPTIONS is a variable in the srvctl script where the default is JRE_OPTIONS=””
New Java process when running srvctl from another window with JRE_OPTIONS=”” and SRVM_TRACE=true
(the DTRACING is due to SRVM_TRACE set to TRUE)
oracle  15025 15019 0 13:26 pts/0   00:00:00 /oracle/GRID/11203/jdk/jre//bin/java -DORACLE_HOME=/oracle/GRID/11203 -classpath /oracle/GRID/11203/jlib/netcfg.jar:/oracle/GRID/11203/jdk/jre//jre/lib/rt.jar:/oracle/GRID/11203/jdk/jre//jre/lib/i18n.jar:/oracle/GRID/11203/jlib/srvm.jar:/oracle/GRID/11203/jlib/srvmhas.jar:/oracle/GRID/11203/jlib/srvmasm.jar:/oracle/GRID/11203/jlib/supercluster.jar:/oracle/GRID/11203/jlib/supercluster-common.jar:/oracle/GRID/11203/ons/lib/ons.jar:/oracle/GRID/11203/srvm/jlib/srvctl.jar:/oracle/GRID/11203/jlib/gns.jar -DTRACING.ENABLED=true -DTRACING.LEVEL=2 -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/oracle/GRID/11203/srvm/admin/logging.properties oracle.ops.opsctl.OPSCTLDriver
oracle  15942 15905 0 13:26 pts/0   00:00:00 /oracle/GRID/11203/jdk/jre//bin/java -DORACLE_HOME=/oracle/GRID/11203 -classpath /oracle/GRID/11203/jlib/netcfg.jar:/oracle/GRID/11203/jdk/jre//jre/lib/rt.jar:/oracle/GRID/11203/jdk/jre//jre/lib/i18n.jar:/oracle/GRID/11203/jlib/srvm.jar:/oracle/GRID/11203/jlib/srvmhas.jar:/oracle/GRID/11203/jlib/srvmasm.jar:/oracle/GRID/11203/jlib/supercluster.jar:/oracle/GRID/11203/jlib/supercluster-common.jar:/oracle/GRID/11203/ons/lib/ons.jar:/oracle/GRID/11203/srvm/jlib/srvctl.jar:/oracle/GRID/11203/jlib/gns.jar –DTRACING.ENABLED=true -DTRACING.LEVEL=2 -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/oracle/GRID/11203/srvm/admin/logging.properties oracle.ops.opsctl.OPSCTLDriver
New Java process when running srvctl from another window with JRE_OPTIONS=”” and SRVM_TRACE unset
oracle  26374 26366 0 13:28 pts/0   00:00:00 /oracle/GRID/11203/jdk/jre//bin/java -DORACLE_HOME=/oracle/GRID/11203 -classpath /oracle/GRID/11203/jlib/netcfg.jar:/oracle/GRID/11203/jdk/jre//jre/lib/rt.jar:/oracle/GRID/11203/jdk/jre//jre/l                                    ib/i18n.jar:/oracle/GRID/11203/jlib/srvm.jar:/oracle/GRID/11203/jlib/srvmhas.jar:/oracle/GRID/11203/jlib/srvmasm.jar:/or                                    acle/GRID/11203/jlib/supercluster.jar:/oracle/GRID/11203/jlib/supercluster-common.jar:/oracle/GRID/11203/ons/lib/ons.jar                                    :/oracle/GRID/11203/srvm/jlib/srvctl.jar:/oracle/GRID/11203/jlib/gns.jar -Djava.net.preferIPv4Stack=true -Djava.util.log                                    ging.config.file=/oracle/GRID/11203/srvm/admin/logging.properties oracle.ops.opsctl.OPSCTLDriver
with JRE_OPTIONS=”-Xms256M -Xmx1024M” and SRVM_TRACE unset set
oracle    695  685 0 13:35 pts/0   00:00:00 /oracle/GRID/11203/jdk/jre//bin/java -Xms256M -Xmx1024M -DORACLE_HOME=/oracle/GRID/11203 -classpath /oracle/GRID/11203/jlib/netcfg.jar:/oracle/GRID/11203/jdk/jre//jre/lib/rt.jar:/oracle/GRID/11203/jdk/jre//jre/lib/i18n.jar:/oracle/GRID/11203/jlib/srvm.jar:/oracle/GRID/11203/jlib/srvmhas.jar:/oracle/GRID/11203/jlib/srvmasm.jar:/oracle/GRID/11203/jlib/supercluster.jar:/oracle/GRID/11203/jlib/supercluster-common.jar:/oracle/GRID/11203/ons/lib/ons.jar:/oracle/GRID/11203/srvm/jlib/srvctl.jar:/oracle/GRID/11203/jlib/gns.jar -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/oracle/GRID/11203/srvm/admin/logging.properties oracle.ops.opsctl.OPSCTLDriver
Conclusion: The o/s was unable to satisfy memory requirement for srvctl intermitently. JRE_OPTIONS was set with a small and large memory value which worked but this was just a co-incidence. Even after JRE_OPTIONS was set back to null , srvctl carried on working.
Â
Discussion ¬