If Application Server Hang on AIX platform.
To troubleshoot the hang, please collect the following doc during the time that
the hang is occurring. login as "root",
1. netstat -an > netstat.txt (if webserver is remote, gather netstat output on
Appserver machine AND webserver machine)
2. ps -ef > ps.txt
3. df -k > df.txt
4. lsof -p [PID of hung appserver] > lsof.txt (You need to install "lsof" utility)
5. Can you serve a static document from the webserver?
6. tprof -k -s -e -x sleep 60
7. ./dbxtrace_aix.sh -a [PID of hung appserver] > dbxtrace.txt (wait until finished)
8. WAIT TWO MINUTES
9. kill -3 [PID of hung appserver]
10. WAIT TWO MINUTES
11. kill -3 [PID of hung appserver]
12. WAIT TWO MINUTES
13. kill -3 [PID of hung appserver]
The kill -3's should create javacore.txt files in the working directory of WAS,
normally Websphere\Appserver\bin directory.
Please send in the following files:
1. Everything in <was root>\logs directory (from the WebSphere machine)
2. If the webserver is remote, send the trace.log.<webserver name>.<date> (3.5.X),
native.log (4.0.X), or http_plugin.log (5.0.X) file from the webserver machine.
3. dbxtrace.txt
4. All javacore?????.txt files created
5. netstat.txt, ps.txt, df.txt
6. Send in all files generated by tprof (especially _prof.all). These files will
be in the directory from which the tprof command was run and will begin with "_".
7. Additionally, for WebSphere,
(1) uname -a > uname.txt
(2) lslpp -i |grep AIX_ML | sort > aix_sw.txt
(3) errpt -a > errpt.txt
(4) httpd.conf of HTTP Server
(5) WAS_HOME/bin/admin.config
(6) WAS_HOME/bin# ./XMLConfig.sh -adminNodeName <server> -export xmlconfig.xml
(send "xmlconfig.xml")
(7) WAS_HOME/properties/com/ibm/websphere/product.xml
8. If you think the problem caused by JDBC Connection Pool, send me all Java source
programs you have.
Appendix (사전 install)
1. lsof - LiSt Open Files
http://www.javaservice.net/~java/bbs/read.cgi?m=unix&b=unix&c=r_p&n=959319048
2. dbxtrace.sh
a. "dbx" 라는 프로그램이 설치되어 있는지 확인해 보십시요. 만약 설치되어 있지
않다면, 설치를 하도록 하십시요. 대부분 이미 설치되어 있을 겁니다.
b. 아래에 첨부한 dbxtrace.sh 라는 shell script 를 다운받아 적당한 디렉토리에
넣어두고, 실행권한을 부여하십시요.
# cp /tmp/dbxtrace.sh /usr/local/bin/
# chmod 755 /usr/local/bin/dbxtrace.sh
dbxtrace.sh script download
http://w3.hursley.ibm.com/java/communication/aix/service/AIX/Tools.html
http://w3.hursley.ibm.com/java/communication/aix/service/AIX/dbxtrace
c. root 유저에서 다음의 명령을 통해 system fullcore 옵션을 true 로 변경하시요.
옵션확인방법 : lsattr -El sys0 | grep fullcore
옵션변경방법 : chdev -a fullcore=true -l sys0
d. /usr/WebSphere/AppServer/bin/ 디렉토리에 있는 javacore* 및 core 파일은
(현재 개발중이시니) 더이상 필요없으며, 가급적 삭제하시고, 각종 로그들도
삭제하여 /usr 디스크의 free space 를 가급적 500MB 이상을 확보하십시요.
 dbxtrace.sh (5456 Bytes)
 lsof_aix_binary.zip (232832 Bytes)
출처 : 자바서비스넷 이원영(javaservice@hanmail.net) |