14 Troubleshooting JVM Diagnostics

JAM Agent: loadNative Exception loading
[/tmp/libjamagent_nz.so.1] /tmp/libjamagent
_nz.so.1: libnnz11.so: cannot open shared object
file: No such file or directory. JAM Agent:
Please make sure that libnnz11 and libclntsh is
in the LD_LIBRARY_PATH

This error occurs if the NZ libraries are not in LD_LIBRARY_PATH.

To resolve this issue, add the libnz11.so and libclntsh.so.11.1 files to the LD_LIBRARY_PATH.

Jam Communication : Open wallet error : 28759
Jam Communication : setup_auth failed 28759

This error occurs if an unrecognized wallet resource locator has been used to open the wallet.

To resolve this error, ensure that the wallet location is preceded by file.

Jam Communication : Open wallet error : 28759 
Jam Communication : setup_auth failed 28759

This error occurs when the wallet file cannot be opened by the JVM Diagnostics Agent.

Ensure that the file path to the wallet is correct and the correct permissions have been provided. While specifying the wallet location in the web.xml file, do not specify the absolute path to the wallet.

Jam Communication : nz initialize SSL handshake
error 28864 Jam Communication : nzos_Write error
- 29031 AM Console: Invalid connection from
 10.228.234.246.55859

These errors occur if the JVM Diagnostics Manager has been launched in non-secure mode.

javax.crypto.IllegalBlockSizeException: Input
length must be multiple of 8 when decrypting
with padded cipher
at com.sun.crypto.provider.SunJCE_f.b(DashoA13?
*..)
at
com.sun.crypto.provider.DESedeCipher.engineDoFin
al(DashoA13? *..) 
at javax.crypto.Cipher.doFinal(DashoA13? *..) 
at jamagent.jamagent.decrypt(jamagent.java:725)
at jamagent.jamagent.decrypt(jamagent.java:741)
at jamagent.jamagent.<init>(jamagent.java:799)
at jamagent.jaminit.init(jaminit.java:50)

This error occurs if the encrypted password specified for the JVM Diagnostics Agent is incorrect. Enter the correct password in web.xml file.

Jam Communication : Setting up context
Jam Communication : nz initialize SSL handshake
error 29024 
Jam Communication : nzos_Write error :- 29031

This error occurs if the wallet password is incorrect or the certificate for the JVM Diagnostics Agent has not been certified by the same CA as the JVM Diagnostics Manager.

Jam Communication : nz initialize SSL handshake
error 28860 
Jam Communication : nzos_Read error : 28865
Console Err code: -1716291680 Receiving data on
port 3600 from 127.0.0.1:57187, FD 450: Broken
pipe

JAM Agent: Conn reset by console
JVM Diagnostics Manager Error:
JAM Communication: nz initialize SSL handshake 
error 29049
JAM Communication: nzos_Read error: 28865
Console Err code: -1716291680 Receiving data on
port 3600 from 127.0.0.1:58221, FD 450:
Transport endpoint is not connected

This error occurs if the JVM Diagnostics Agent is running in non-secure mode and is trying to connect to a secure JVM Diagnostics Manager

DBWait link not displayed on JVM Threads Real
Time Analysis page.

No data displayed in Top DBStates / SQLs tables.

Cross tier functionality errors may occur due the following:

• Incorrect database credentials

• Database Agent errors

If the database credentials are incorrect:

• Navigate to Middleware > JVM Diagnostics > Setup > Databases page and verify the credentials for the registered database to be monitored.

• Enter the id of user who has installed the database in the OS User field.

• Specify the database application user credentials in the DB User field.

• Specify the database system user credentials in the DB User (Explain Plan) field.

If database agent errors occur, ensure that the database agent is running on a machine on which the database is installed with the correct IP address and port number.

• Navigate to the Threads > Real Time Analysis page for the JVM. If you see a thread that is in the DB Wait state, it should be a hyperlink. Click on the hyperlink to drill down to the database details and view the database session information including the SQL statements executed.

• If the DB Wait state is not a hyperlink, navigate to the Middleware > JVM Diagnostics > Setup page and set the Cross Tier Log Level to 6 and send AD4J Manager logs report issue.

weblogic.transaction.internal.TimedOutException:
Transaction timed out after 30 seconds

This error occurs if the Poll Duration has a large value and results in a timeout.

This error does not affect the Trace functionality and can be ignored.

ScriptException: Error occured while performing
deploy: The action you performed timed out after
600,000 milliseconds

This error occurs when you are deploying the JVM Diagnostics Agent. To resolve this issue check if the lock for the WebLogic Administration Console on which the JVM Diagnostics Manager has already been acquired. If it has been acquired, release it and run the script again.

• Login to the WebLogic Administration Console: http://<machine address>:<webogic port>/console.

• If you see the Activate Changes and Undo All Changes buttons in the left pane, click on these buttons to clear them. If the buttons are not cleared, click Undo All Changes and run the script again.

glibc detected * free(): invalid next size
(fast): 0x0965d090" ./loadheap.sh: line 237:
32357 Aborted ./bin/${bindir}/processlog
in=$infile hdr=${sumdata} obj=${objdata}
rel=${reldata} root=${rootdata}
osum=${objsumdata} rrel=${rootrel} heap=${heap
_id} skip=$skipgarbage db=$dbtype $* 
Error processing file /tmp/heapdump6.txt 

Check if the heapdump operation has been successfully completed. Open the heapdump6.txt file and check if there is a heapdump finished string at the end of the file. If you see this string, load the finished dump file.

Heapdump already in progress, cannot take another
heapdump

Check if the heapdump operation has been successfully completed. Open the heapdump6.txt file and check if there is a heapdump finished string at the end of the file. If you see this string, restart the jamagent and run the heapdump operation again.

loadheap.sh created unusable unique indexes.

Run the loadheap/sql/cleanup.sql shipped with loadheap.zip to fix the unique indexes.

JAM Console: Socket timed out after recv --
client adc2100083.us.oracle.com:7001 is not
Active [0] secs JAM Console jamlooptimeout =
[3]JAM CONSOLE: JVM 1 is not active JAM Cons Err
Processing Request: 128 JVM 1 is not active
jamDAL: jamreq returned 128 return status < 0
from jamDalInst.processRequest 

To resolve this error, increase the Agent Request Timeout (secs) and Agent Loop Request Timeout (secs)

This page cannot be displayed at the current JVM
optimization level.

When the optimization level is set to 0, the Show Heap (Memory) operation is very time consuming and is not supported.

Method locals on Real Time Thread page not
displayed.

Method locals is not supported if optimization level is set to 0.

Agent is up and running but is not displayed in
the real time pages.

If the log file shows JAMMANAGER: OLD AGENT or NULL POOl or wrong jamoptimization level, this indicates that the old jamagent/Dbagent is being used.

To resolve this issue, download the latest jamagent from Setup > Download page.

You do not have the necessary privileges to view
this page

Ensure that you have the required JVM Diagnostics Administrator or User privileges to view the JVM Diagnostics data.