HP-UX上一次高CPU占用问题的分析

一、背景：

      本周，另一项目组的一个应用系统更换服务器，由原来的HP-UX11.11迁移到HP-UX11.23上，考虑到是操作系统都是HP-UX，所以直接从源服务器把整个weblogic安装目录（包括域目录）直接复制到目标服务器上，能正常启动和运行。但是发现在执行某些操作的时候，CPU占用很高（weblogic所在的java进程会100%占用4~5颗CPU），应用系统的表现就是反应很慢。

       想登录Admin Console看看线程情况也登录不进去。

 

 

二、分析过程：

1、根据CPU占用极高的情况来判断，可能是线程很忙，而无法登录Admin Console可能是因为Admin在与出问题的Managed Server进行通讯时发生阻塞所致；如果应用系统中出现死循环，可能会导致这种现象出现，但迁移前后应用系统没有改动，所以死循环这个原因可能性不大；

 

2、为了查看线程使用情况，用kill -3 <pid>作ThreadDump，先后作了7次，然后对线程情况进行分析，发现有很多线程在读写数据库：

 

"ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006d5f0 nid=21 lwp_id=428827 runnable [0x29f41000..0x29f40530]
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at oracle.net.ns.Packet.receive(Unknown Source)
	at oracle.net.ns.DataPacket.receive(Unknown Source)
	at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:930)
	at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:894)
	at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:383)
	at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1973)
	at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteDescribe(TTC7Protocol.java:850)
	- locked <439f1b58> (a oracle.jdbc.ttc7.TTC7Protocol)
	at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2599)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2963)
	at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:961)
	- locked <a91e7bb8> (a oracle.jdbc.driver.OracleStatement)
	- locked <439f1bc0> (a oracle.jdbc.driver.OracleConnection)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:757)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:392)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1509)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1402)
	at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:295)
	at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:266)
	at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:449)
	at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:359)
	at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:80)
	at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:87)
	at weblogic.jdbc.pool.Driver.connect(Driver.java:148)
	at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:507)
	at weblogic.jdbc.jts.Driver.connect(Driver.java:139)
	at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:305)
	at com.ursa.j2ee.J2eeDataSource.getConnection(J2eeDataSource.java:121)
	at com.ursa.acf.component.datasource.impl.StrutsDSManager.getConnection(StrutsDSManager.java:107)
	at com.ursa.acf.component.datasource.DatabaseHelper.getConnection(DatabaseHelper.java:40)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization.b(UrsaOrganization.java:217)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization. (UrsaOrganization.java:122)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganizationManager.getOrganization(UrsaOrganizationManager.java:64)
	at com.ursa.acf.plugins.organization.OrganizationHelper.getOrganization(OrganizationHelper.java:233)
	at com.ursa.acf.plugins.post.ursaimpl.AbstractPost.getOrg(AbstractPost.java:482)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.a(RoleMember_SelfOu_WithO_EXPOU_PT.java:75)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.getMembers(RoleMember_SelfOu_WithO_EXPOU_PT.java:109)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.checkUser(RoleMember_SelfOu_WithO_EXPOU_PT.java:50)
	at com.ursa.acf.plugins.role.ursaimpl.UrsaRole.checkUser(UrsaRole.java:1202)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:594)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:407)
	at com.ursa.acf.plugins.workflow.utils.UserHelper.getAdminShenFen(UserHelper.java:240)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.doQuery(SWListQueryJJG.java:251)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.acfExecute(SWListQueryJJG.java:67)
	at com.ursa.acf.core.servlet.AcfCoreAction.execute(AcfCoreAction.java:55)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:996)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cn.com.pansky.portal.sso2.client.SsoFilter.doFilter(SsoFilter.java:92)
	at com.itss.xz.plugin.portal2.sso.OaSsoFilter.doFilter(OaSsoFilter.java:33)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.ursa.acf.core.EncodingFilter.doFilter(EncodingFilter.java:46)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6458)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3661)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2630)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

 

"ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006d1b8 nid=18 lwp_id=428824 runnable [0x2a181000..0x2a180530]
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
	at oracle.net.ns.DataPacket.send(Unknown Source)
	- locked <53bfdb28> (a java.net.SocketOutputStream)
	at oracle.net.ns.NetOutputStream.flush(Unknown Source)
	at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.net.ns.NetInputStream.read(Unknown Source)
	at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:930)
	at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:894)
	at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:383)
	at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1973)
	at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:1231)
	- locked <528ca5a8> (a oracle.jdbc.ttc7.TTC7Protocol)
	at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2616)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2963)
	at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:961)
	- locked <a9200028> (a oracle.jdbc.driver.OracleStatement)
	- locked <52850b28> (a oracle.jdbc.driver.OracleConnection)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:757)
	at weblogic.jdbc.common.internal.ConnectionEnv.test(ConnectionEnv.java:392)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1509)
	at weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1402)
	at weblogic.common.resourcepool.ResourcePoolImpl.releaseResource(ResourcePoolImpl.java:517)
	at weblogic.jdbc.common.internal.ConnectionPool.release(ConnectionPool.java:604)
	at weblogic.jdbc.common.internal.ConnectionPoolManager.release(ConnectionPoolManager.java:105)
	at weblogic.jdbc.wrapper.PoolConnection.doClose(PoolConnection.java:164)
	at weblogic.jdbc.wrapper.PoolConnection.close(PoolConnection.java:124)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization.b(UrsaOrganization.java:229)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganization. (UrsaOrganization.java:122)
	at com.ursa.acf.plugins.organization.ursaimpl.UrsaOrganizationManager.getOrganization(UrsaOrganizationManager.java:64)
	at com.ursa.acf.plugins.organization.OrganizationHelper.getOrganization(OrganizationHelper.java:233)
	at com.ursa.acf.plugins.post.ursaimpl.AbstractPost.getOrg(AbstractPost.java:482)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.a(RoleMember_SelfOu_WithO_EXPOU_PT.java:75)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.getMembers(RoleMember_SelfOu_WithO_EXPOU_PT.java:109)
	at com.ursa.acf.plugins.role.ursaimpl.RoleMember_SelfOu_WithO_EXPOU_PT.checkUser(RoleMember_SelfOu_WithO_EXPOU_PT.java:50)
	at com.ursa.acf.plugins.role.ursaimpl.UrsaRole.checkUser(UrsaRole.java:1202)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:594)
	at com.ursa.acf.plugins.purview.ursaimpl.AbstractPurview.checkUser(AbstractPurview.java:407)
	at com.ursa.acf.plugins.workflow.utils.UserHelper.getAdminShenFen(UserHelper.java:240)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.doQuery(SWListQueryJJG.java:251)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.acfExecute(SWListQueryJJG.java:67)
	at com.ursa.acf.core.servlet.AcfCoreAction.execute(AcfCoreAction.java:55)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:996)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cn.com.pansky.portal.sso2.client.SsoFilter.doFilter(SsoFilter.java:92)
	at com.itss.xz.plugin.portal2.sso.OaSsoFilter.doFilter(OaSsoFilter.java:33)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.ursa.acf.core.EncodingFilter.doFilter(EncodingFilter.java:46)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6458)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3661)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2630)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

 

还有一些是：

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=10 tid=00074508 nid=100 lwp_id=428908 runnable [0x26400000..0x26400530]
	at weblogic.socket.PosixSocketMuxer.poll(Native Method)
	at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
	- locked <3ba3f728> (a java.lang.String)
	at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

 

也有一些在输出日志：

"ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006e568 nid=32 lwp_id=428840 runnable [0x29701000..0x29700530]
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:66)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:110)
	- locked <3b80ca68> (a java.io.BufferedOutputStream)
	at java.io.PrintStream.write(PrintStream.java:258)
	- locked <3b80ca50> (a java.io.PrintStream)
	at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
	at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
	at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:408)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
	- locked <3c12d840> (a java.io.OutputStreamWriter)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
	at java.util.logging.StreamHandler.flush(StreamHandler.java:224)
	- locked <3c12d700> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
	at java.util.logging.Logger.log(Logger.java:424)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:659)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:141)
	at com.ursa.acf.util.SQLUtils.getContents(SQLUtils.java:398)
	at com.ursa.acf.plugins.workflow.def.dbimpl.DbProcessAdmin.getAttrs(DbProcessAdmin.java:40)
	at com.ursa.acf.plugins.workflow.def.dbimpl.DbProcessAdmin.setID(DbProcessAdmin.java:56)
	at com.ursa.acf.plugins.workflow.WorkflowHelper.getAllProcessAdmin(WorkflowHelper.java:148)
	at com.ursa.acf.plugins.workflow.utils.UserHelper.getAdminShenFen(UserHelper.java:220)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.doQuery(SWListQueryJJG.java:251)
	at com.itss.xz.bg.swgl.action.SWListQueryJJG.acfExecute(SWListQueryJJG.java:67)
	at com.ursa.acf.core.servlet.AcfCoreAction.execute(AcfCoreAction.java:55)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
	at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:996)
	at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:419)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at cn.com.pansky.portal.sso2.client.SsoFilter.doFilter(SsoFilter.java:92)
	at com.itss.xz.plugin.portal2.sso.OaSsoFilter.doFilter(OaSsoFilter.java:33)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at com.ursa.acf.core.EncodingFilter.doFilter(EncodingFilter.java:46)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6458)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
	at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
	at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3661)
	at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2630)
	at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

 

剩下的多数就是在waitForRequest：

"ExecuteThread: '61' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=00072078 nid=74 lwp_id=428882 in Object.wait() [0x27780000..0x27780530]
	at java.lang.Object.wait(Native Method)
	- waiting on <3ba1f7e8> (a weblogic.kernel.ExecuteThread)
	at java.lang.Object.wait(Object.java:429)
	at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
	- locked <3ba1f7e8> (a weblogic.kernel.ExecuteThread)
	at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)

 

      从以上情况，首先看到有比较多的数据库操作，考虑会不会是数据库出现问题或应用程序中数据库操作部分效率不高，但是本次迁移没有动数据库，应用程序也没改动，所以这种可能性也不是很大；

      再综合这些线程来看，这些线程都在处理I/O相关的事情（除去那些“waitForRequest”的线程），会不会是I/O效率低导致的问题呢？我以前曾经遇到过因为weblogic上的“Native I/O”选择没有打开而导致性能低下的案例，这次会不会也是这个问题呢？登录到Admin Console（服务器已经重启过了），看到出问题服务器的“Native I/O”选择是选上的，看来也不是“Native I/O”的问题了。

      在ThreadDump日志里，最严重的一个地方是一个线程在写日志：

"ExecuteThread: '39' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=00070188 nid=52 lwp_id=428860 runnable [0x28802000..0x28800530]
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:66)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)
	- locked <3b80ca68> (a java.io.BufferedOutputStream)
	at java.io.PrintStream.flush(PrintStream.java:136)
	- locked <3b80ca50> (a java.io.PrintStream)
	at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:410)
	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
	- locked <3c13bb50> (a java.io.OutputStreamWriter)
	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
	at java.util.logging.StreamHandler.flush(StreamHandler.java:224)
	- locked <3c13ba10> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
	at java.util.logging.Logger.log(Logger.java:424)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:659)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java:141)

            ……

 ，另外有55个线程在等待：

"ExecuteThread: '22' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0006e9a0 nid=35 lwp_id=428843 waiting for monitor entry [0x294c2000..0x294c0530]
	at java.util.logging.StreamHandler.publish(StreamHandler.java:173)
	- waiting to lock <3c13ba10> (a java.util.logging.ConsoleHandler)
	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:87)
	at java.util.logging.Logger.log(Logger.java:424)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:587)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:659)
	at org.apache.commons.logging.impl.SLF4JLocationAwareLog.info(SLF4JLocationAwareLog.java
                ……

 从这些现象综合来看，应该还是I/O效率的问题。

       我再次到网上搜索，找到Weblogic的老家关于I/O调优的部分：

http://download.oracle.com/docs/cd/E13222_01/wls/docs100/perform/WLSTuning.html#wp1152246

 

里面讲到了“Performance Packs”的问题，根据该文指示的方向：

http://download.oracle.com/docs/cd/E13196_01/platform/suppconfigs/configs81/81_over/overview.html

，我核对了一下操作系统与weblogic版本的对应的Performance Packs，发现在“HP-UX 11i v2 on PA-RISC”上，并不支持“weblogic SP3”的“Performance Packs”（http://download.oracle.com/docs/cd/E13196_01/platform/suppconfigs/configs81/81_over/overview.html）。

 

最后，我分析的结论是：

在迁移时weblogic是直接Copy到目标机上的，复制过去的Performance Packs(lib/hpux11/libmuxer.sl)不适合操作系统的版本“HP-UX 11i v2 on PA-RISC”，导致I/O效率低下，进而导致线程池繁忙，占用CPU高。

解决办法是：重新安装weblogic8.1 SP4以上版本，启用“Native I/O”。

 

=============================

另外，有同事发现weblogic运行在Development Mode，这也可能是影响性能的因素之一，已改成Produectio Mode，正在观察效果。