8 Troubleshooting

This chapter discusses diagnostic techniques and aids for determining and resolving problems with data conversion, truncation, and conversation startup. It also describes how to collect the data when the debugging (trace) option is on.

You will want to trace the PL/SQL stored procedures only when you suspect problems. Do not run with tracing enabled during normal operations, because it will affect performance.

This chapter contains the following sections:

TIP Definition Errors

TIP definition errors occur when a TRANSACTION, CALL, or DATA entry in the PG DD is not properly defined.

Use the REPORT with DEBUG statement to list the PG DD contents and GENERATE DIAGNOSE(PKGEX(DR)) option to include corresponding ID numbers in the TIP.

Table 8-1 shows the mnemonic used to represent ID numbers and their correspondence with the following:

  • PGAU REPORT with debug listings, GENERATE traces and TIPs

  • PG DD tables and columns from which ID numbers are selected

  • Oracle sequence objects from which ID numbers originate

Table 8-1 PG DD ID Numbers in Correspondence

PGAU REPORT/TIP PDGG table(col) Sequence Object

v# transaction version

pga_trans(version)

pga.transvers

v# call version

pga_call(version)

pga.callvers

v# data version

pga_data(version)

pga.datavers

t# transaction id#

pga_trans(trans#)

pga.transeq

c# call id#

pga_call(call#)

pga_call_parm(call#)

pga.callseq

d# data id#

pga_call_parm(data#)

pga_data(data#)

pga_fields(data#)

pga.dataseq

f# field id#

pga_fields(fld#)

pga.fieldseq

q# qualifier id#

pga_data_values(qual#)

pga.fieldseq

a# trans attribute id#

pga_trans_values(attr#)

pga_trans_attr(attr#)

pga.tattrseq

a# field attribute id#

pga_data_values(attr#)

pga_data_attr(attr#)

pga.dtattseq

e# environment

pga_environments(env#)

pga.envrseq

l# compiler/language

pga_compilers(comp#)

pga.compseq

These ID numbers can be used to associate the conversions performed in the TIP with the definitions stored in the PG DD.

The PG DD diagnostic references appear in TIPs generated with the PKGEX(DR) option as single line Comments:

-- PG DD type idno=nnn ...

The PG DD diagnostic references appear in REPORT with DEBUG listings before or to the right of their related definition entry as end-delimited Comments:

/* idno=nnn */

Refer to Appendix A, "Database Gateway for APPC Data Dictionary" for more information about PG DD, including a complete list of dictionary tables.

Problem Analysis with PG DD Diagnostic References

TIPs should be generated by the PGAU GENERATE command with the PKGEX(DR) diagnostic option, to include PG DD reference Comments in the TIP. These diagnostic references are Comments only and do not affect the runtime overhead of the TIP. Refer to GENERATE in Chapter 2, "Procedural Gateway Administration Utility" for a description of the PKGEX (DR) parameter.

  1. Before defining the PL/SQL package, identify the transaction name, ID number (t#), and version (v#) from the TIP specification within the TIP.

  2. Invoke PGAU REPORT WITH DEBUG specifying the same transaction name and version.

    REPORT selects definitions from the PG DD and produces a listing showing the DATA, CALL, and TRANSACTION definitions and the ID number of each user-supplied definition.

  3. Compare the reported definitions with those used in the remote transaction program and identify all corresponding exchanges and the data formats transmitted.

  4. Look for and investigate any mismatches, such as:

    • different numbers of send/receive calls

    • different sequence of send/receive calls

    • different parameter lists on send/receive calls

    • different data fields within each exchanged parameter

    • different lengths for each exchanged parameter

    • unsupported datatypes for each exchanged parameter

    • improperly initialized control fields for:

      • repeating group counts

        IBMVSCOBOLII affected clauses include

        OCCURS n TIMES DEPENDING ON field

      • remapped group criteria

        IBMVSCOBOLII affected clauses include

        REDEFINES field1 WHEN field2 = criteria

Problem Analysis with PG DD Select Scripts

PGAU GENERATE error messages and TRACE(OC) entries reference SQL SELECT statements. Refer to Table 8-2 for the meaning of the name designations for each entry.

Table 8-2 Meaning of TRACE(OC) Output

Name Entry

SED

Select Environment Data

STL

Select Transaction (latest version)

STV

Select Transaction (specific version)

STC

Select Transaction Calls

SPD

Select Parameter Data

SF

Select Fields

SFA

Select Field Attributes

SXF

Select conversion Formats

SXA

Select Attribute conversions

The SQL*Plus test scripts in Table 8-3 are provided to perform the identical SELECTS as GENERATE performs to determine which PG DD rows are being used when the TIP is generated. These files are loaded into the %ORACLE_HOME%\\dg4appc\\admin directory on Microsoft Windows or into the $ORACLE_HOME/dg4appc/admin directory on UNIX based systems, during installation.

Table 8-3 SQL*Plus Test Scripts and Their Corresponding Entries

Script Entry

pgddsed.sql

Select Environment Data

pgddstl.sql

Select Transaction (latest version)

pgddstv.sql

Select Transaction (specific version)

pgddstc.sql

Select Transaction Calls

pgddspd.sql

Select Parameter Data

pgddsf.sql

Select Fields

pgddsfa.sql

Select Field Attributes

pgddsxf.sql

Select Conversion Formats

pgddsxa.sql

Select Attribute conversions

The scripts are shown in the same order used by GENERATE and each script prompts the SQL*Plus user for the required input. The information retrieved from a previous select is often used as input to a subsequent select. If a you suspect that a PG DD field entry has produced inaccurate data, browse the .sql files listed above to determine the source of the problem. These files are loaded into the %ORACLE_HOME%\\dg4appc\\admin directory on Microsoft Windows or $ORACLE_HOME/dg4appc/admin directory on UNIX based systems, during installation.

Data Conversion Errors

Data conversion errors are usually the result of:

  • incorrect determination of datatype

    or

  • incorrect specification of data position.

PGAU determination of the datatype is based on the values found in the PG DD, pga_fields(mask), and pga_fields(maskopts) columns. PGAU generates PL/SQL code to perform conversions based on the mask value:

  • PIC X converted to CHAR with the same character length

  • PIC G converted to CHAR with the same character length

  • PIC 9 converted to NUMBER

Character datatype is presumed for all PIC X and PIC G mask values and conversion errors are more likely the result of position, length, and justification errors.

Determination of numeric datatype depends on several factors, including the combination of mask and maskopts values and how they apply to the actual remote host data in its internal format. Values for mask, maskopts, and data might conflict in unexpected ways. For example, an option such as USAGE IS COMP might be overridden if the data is in display format. While compilers occasionally perform such overrides correctly, they can cause unexpected results when exchanging data with systems coded in other languages.

To notify the user of such overrides, a warning function has been included in the following UTL_PG functions:

  • MAKE _NUMBER_TO_RAW_FORMAT

  • MAKE_RAW_TO_NUMBER_FORMAT

  • NUMBER_TO_RAW

  • RAW_TO_NUMBER

Problem Analysis with TIP Runtime Traces

TIPs should be generated by the PGAU GENERATE command with the PKGEX(DC) diagnostic option to include TIP data conversion trace logic in the TIP. TIP function call trace logic is always included in every TIP. This is runtime trace instrumentation and has some overhead when tracing is enabled, but negligible overhead when tracing is disabled. Refer to GENERATE in Chapter 2, "Procedural Gateway Administration Utility" for more information.

  1. Regenerate TIPs with the PKGEX(DC, DR) options and recompile the TIP body file, tipname.pkb. Avoid recompiling the TIP specification.

  2. Revise the application that calls the TIP initialization function (tipname_init) to pass the trace flags parameter with data conversion and function call tracing enabled. Refer to "Controlling TIP Runtime Data Conversion Tracing".

    If the problem causes an exception to be raised in the TIP and the application contains an exception handler, the application exception handler should be Commented out to prevent it from handling the exception and preventing the exception point of origin from being reported. When the TIP exception is next raised, its source line number in the TIP is reported. Record this information.

  3. Execute the application with diagnostic TIP initialization.

    If the TIP trace pipe inlet overflows due to the application calls causing the TIP to write trace messages in the TIP trace pipe inlet, you have one minute from the start of the overflow condition to begin Step 4 and empty the TIP trace pipe.

    Otherwise, exception "ORA-20703 PGA-TIP: pipe send error" is issued, ending the diagnostic session, possibly before any relevant trace information is generated.

  4. Retrieve and record the TIP trace message stream.

    Use SQL*Plus to connect to the same Oracle user ID executing the application or the user ID under which the TIP is executed. This establishes a second session from which the trace pipe outlet can be read, preventing the TIP trace pipe from overflowing at the TIP trace pipe inlet.

    1. Issue the command:

      set serveroutput on size nnnnn

    2. Issue the command to record the trace output:

      spool tipname.trc

    3. Issue the command to retrieve the trace stream:

      exec rtrace('tipname');

      If the application is long-running, repeat this command as often as needed until all trace messages have been retrieved.

  5. If any exceptions are raised, note their prefix, number, and full message text.

  6. Analyze the TIP trace message stream. A normal trace is shown for the pgadb2i TIP in Appendix F, "Administration Utility Samples".

TIP Runtime Trace Controls

Runtime trace control is the second parameter specified on a TIP initialization call. It is a CHAR(8) datatype of the following form:

rc := yourtip_init(trannum,'wxyz0000');

Table 8-4 describes the value of positions one to four:

Table 8-4 Values of Positions 1 through 4 on Second Parameter of TIP Call

Item Description

position 1 (w)

controls UTL_RAW warning. A value of 0 suppresses warnings; a value of 1 issues warnings.

position 2 (x)

controls the function entry/exit tracing. A value of 0 suppresses the function entry/exit tracing; a value of 1 enables the function entry/exit tracing.

position 3 (y)

controls data conversion tracing. A value of 0 suppresses data conversion tracing; a value of 1 enables data conversion tracing.

position 4 (z)

controls gateway exchange tracing. A value of 0 suppresses gateway exchange tracing; a value of 1 enables gateway exchange tracing.

Positions 5 through 8 are reserved and ignored.

Generating Runtime Data Conversion Trace and Warning Support

Use PGAU to regenerate the TIP and specify the GENERATE parameter DIAGNOSE(PKGEX(DC)). This includes runtime PL/SQL code in the TIP which tests for and displays warnings of correct, but possibly unexpected NUMBER_TO_RAW and RAW_TO_NUMBER conversions.

Refer to GENERATE in Chapter 2, "Procedural Gateway Administration Utility" for more information about this parameter.

Recompile the TIP body under SQL*Plus. Avoid recompiling the TIP specification.

Controlling TIP Runtime Conversion Warnings

After the TIP has been regenerated, the issuance of runtime warnings is under control of the application. By default, warnings are suppressed and are only issued when they are enabled.

Errors and exceptions are always issued if they occur.

To enable the issuance of warnings, an additional parameter must be supplied when calling the TIP initialization function. This parameter is a CHAR(8) datatype and each character position controls a particular TIP runtime diagnostic function.

To enable warnings in yourtip, the client application should call the TIP initialization function with the statement:

rc := yourtip_init(trannum,'10000000');

The following is input to the TIP trace pipe inlet at initialization time:

"UTL_PG warnings enabled"

Controlling TIP Runtime Function Entry/Exit Tracing

To enable function entry/exit tracing in yourtip, the client application should call the TIP initialization function with the statement:

rc := yourtip_init(trannum,'01000000');

The following is input to the TIP trace pipe inlet at initialization time:

'function entry/exit trace enabled'
'tipname_init entered'
'time date/time stamp'

Controlling TIP Runtime Data Conversion Tracing

To enable data conversion tracing in yourtip, the client application should call the TIP initialization function with the following statement:

rc := yourtip_init(trannum,'00100000');

The following is input to the TIP trace pipe inlet at initialization time:

'data conversion trace enabled'

Controlling TIP Runtime Gateway Exchange Tracing

To enable runtime gateway exchange tracing in yourtip, the client application should call the TIP initialization function with the following statement:

rc := yourtip_init(trannum,'00010000');

The following is input to the TIP trace pipe inlet at initialization time:

'gateway exchange trace enabled'

Suppressing TIP Warnings and Tracing

After debugging is finished, there are two ways to suppress the following:

  • data conversion tracing

  • conversion warnings

  • function entry/exit tracing

  • gateway exchange tracing

You can:

  1. Call the TIP initialization function without passing any diagnostic control parameters:

    rc := yourtip_init(trannum);

  2. Call the TIP initialization function passing a revised diagnostic control parameter which disables all tracing and warnings:

    rc := yourtip_init(trannum,'00000000');

    A third method, described in Method C, removes the logic for:

    • data conversion tracing

    • conversion warnings

  3. Generate the TIP again without:

    PKGEX(DC)

Or you can recompile the previous version of the TIP body if it was saved.

Methods A and B allow you to use the same TIP without alteration, but without tracing or warnings. These methods are reversible without alteration or replacement of the TIP. Tracing and warnings can be redisplayed should a problem recur.

Method C also suppresses data conversion tracing and warnings and incurs reduced overhead by avoiding tests, but is not reversible without regenerating the TIP or recompiling an alternate version with data conversion tracing and warning diagnostics imbedded.

The logic for function entry/exit and gateway exchange tracing is included in every TIP and cannot be removed. It can be disabled by method A or B.

Problem Analysis of Data Conversion and Truncation Errors

Oracle Database Gateway for APPC data lengths are limited by PL/SQL to 32,763 bytes per APPC exchange and PL/SQL variable.

The following steps can be used to diagnose data conversion or truncation errors.

Refer to Chapter 3, "Creating a TIP" to review the proper values and definitions referenced in items 1 through 4 below:

  1. Ensure that the COBOL definitions used in the RHT match the input to PGAU;

  2. Ensure the RHT transmission buffers are of sufficient length;

  3. If your gateway uses SNA: Ensure the RHT APPC call addresses the correct transmission buffer and uses the correct data length;

    If your gateway uses TCP/IP: Ensure the RHT I/O PCB call addresses the correct transmission buffer and uses the correct data length

  4. Ensure the client application has declared the correct TIP datatypes used as arguments in the TIP calls.

  5. Ensure that the client application is calling the TIP functions in the proper sequence (init, user-defined..., term), and that any input data to the RHT is correct. Also ensure that if multiple user-defined functions exist, they are being called in the proper sequence and passed the correct input values, if any.

    DBMS_OUTPUT calls can be inserted in the client application to trace its behavior.

    For more information about calling TIP functions in proper sequence, refer to the chapter on configuring the Oracle database for first time installations, in the Oracle Database Gateway for APPC Installation and Configuration Guide for IBM AIX on POWER Systems (64-Bit), Linux x86-64, Oracle Solaris on SPARC (64-Bit), and HP-UX Itanium or Oracle Database Gateway for APPC Installation and Configuration Guide for Microsoft Windows.

  6. Optionally, regenerate the TIP with diagnostic traces included and enable them. The following traces are particularly useful:

    • data conversion trace

    • function entry/exit trace

    • gateway exchange trace

    Refer to "Problem Analysis with TIP Runtime Traces" for more information about traces; refer also to GENERATE in Chapter 2, "Procedural Gateway Administration Utility".

    Note that the output of the trace is different for a gateway using SNA than for a gateway using TCP/IP. However, the method of invoking the trace is the same regardless of which communication protocol you are using.

    On Microsoft Windows, the gateway server tracing must also be enabled in %ORACLE_HOME%\\dg4appc\\admin\initsid.ora. Set the parameters SET TRACE_LEVEL=255 and SET LOG_DESTINATION=C:\oracle\pga\12.1\dg4appc\log

    On UNIX based systems, the gateway server tracing must also be enabled in $ORACLE_HOME/dg4appc/admin/initsid.ora. Set the parameters SET TRACE_LEVEL=255 and SET LOG_DESTINATION=/oracle/pga/12.1/dg4appc/log

    Refer to "Gateway Server Tracing" in this guide for more information about tracing.

    Rerun the client application and examine the trace (see the next step for details).

    To disable the trace, reset

    SET TRACE_LEVEL=0

  7. Examine the trace output.

    The TIP trace output can be saved in a spool file, such as:

    spool tipname.trc

    TIP trace output is written to a named DBMS_PIPE and can be retrieved under SQL*Plus by issuing the following command:

    exec rtrace('tipname');

    or it can be purged by issuing the following command:

    exec ptrace('tipname');

    Note:

    tipname is case-sensitive and must be specified exactly as it is in the TIP.

    Gateway server trace output is written to a log file in a default directory path specified by the SET LOG_DESTINATION gateway parameter in %ORACLE_HOME%\\dg4appc\\admin\initsid.ora for Microsoft Windows and in $ORACLE_HOME/dg4appc/admin/initsid.ora for UNIX based systems. For example, on Microsoft Windows:

    SET LOG_DESTINATION=C:\oracle\pga\12.1\dg4appc\log

    On UNIX based systems:

    SET LOG_DESTINATION=$ORACLE_HOME/dg4appc/log/

    Refer to "Gateway Server Tracing" for more information.

    The gateway server log file can be viewed be editing the file or by issuing other system commands that display file contents. The log file can also be copied and saved to document problem symptoms.

Gateway Server Tracing

The gateway contains extensive tracing logic in the gateway remote procedural calls (RPCs), and the APPC-specific code. Tracing is enabled through gateway initialization parameters or dynamic RPC calls to the gateway. The trace provides information about the execution of the gateway RPC functions and about the execution of the APPC interface. The trace file contains a text stream written in chronological sequence of events. The trace is designed to assist application programmers with the debugging of their OLTP transaction programs and Oracle applications that communicate with those transaction programs through the gateway.

A single trace file is created for an entire gateway session from the time the database link is opened until it is closed. The trace can be directed to a specific path/filename or to a path (directory) only. In the first case, the file is overwritten each time a new session begins for the gateway being traced. When the trace target is a directory, a separate file with a generated name (containing the operating system process ID) is written for each gateway session. The latter approach must be used whenever the gateway to be traced might be the target of new sessions after the desired trace is written but before it can be copied and saved. Conversely, in some situations you might choose to create a distinct gateway system identifier used solely for tracing, and direct its trace to a single specific filename. This avoids the problem of an ever-increasing set of trace files when, for example, repeated attempts are necessary to reproduce or debug a problem. A fixed filename should never be used if there is any chance that an unexpected gateway session could overlay a useful trace.

Defining the Gateway Trace Destination

This section describes how to define the destination of trace files to the gateway, and how to cause the gateway to create the trace files during initialization. Note that this does not enable any gateway tracing, it merely defines the destination of any trace output produced when the gateway tracing is enabled.

  1. Choose a gateway system identifier to trace. Decide whether you will be tracing an existing gateway system identifier or a new one created specifically for tracing. If a new system identifier will be used, configure the new system identifier exactly the same as the old one by creating a new initsid.ora (a copy of the old), entries in listener.ora as necessary, and a new Oracle database link.

    Test the new system identifier to ensure it works before proceeding.

  2. For Microsoft Windows, in %ORACLE_HOME%\\dg4appc\\admin, edit the initsid.ora file so it contains the following:

    SET TRACE_LEVEL=255
SET LOG_DESTINATION=logdest

    For UNIX based systems, in $ORACLE_HOME/dg4appc/admin, edit the initsid.ora file so it contains the following:

    SET TRACE_LEVEL=255
SET LOG_DESTINATION=logdest

    where logdest is the directory path for the trace output. The logfile is usually in %ORACLE_HOME%\dg4appc\log for Microsoft Windows and $ORACLE_HOME/dg4appc/log for UNIX based systems. Refer to the earlier discussion about "Problem Analysis of Data Conversion and Truncation Errors" for more information.

    Note:

    Misspelled parameter names in initsid.ora are not detected. The parameter is ignored.

Once these two steps are completed, the gateway opens the specified trace file during initialization. Each session on this system identifier writes a trace file as specified by the SET LOG_DESTINATION parameter described in Step 2 above.

If a directory path was specified, each trace file has a name of the form:

sid_pid.log

where sid is the gateway sid and pid is the operating system process ID of the gateway server expressed in decimal.

Enabling the Gateway Trace

There are two ways to enable the gateway server tracing. The first is to set the tracing options in the gateway initialization file, initsid.ora. The second is to use the additional PGA remote procedural call (RPC) function, PGATCTL, to dynamically control the tracing from within the Oracle application. The first method causes tracing to be performed for all users of the gateway system identifier and is recommended only when the use of the gateway system identifier can be limited to users actually needing the trace. The second method is more flexible and allows the application programmer to selectively trace events on a single gateway session without affecting the operation of other users' gateway sessions.

Before the gateway server trace is enabled, perform the tasks listed in "Defining the Gateway Trace Destination".

Enabling the Gateway Trace Using Initialization Parameters

Edit the initsid.ora file, and add the following line at the end of the file (or, if a SET TRACE_LEVEL parameter is already specified, modify it):

SET TRACE_LEVEL=trace

where trace is a numeric value from 1 to 255 indicating which traces are to be enabled. For further information on the use of this parameter, refer to "PGA Parameters" in Appendix A, "Gateway Initialization Parameters for SNA Protocol" of the Oracle Database Gateway for APPC Installation and Configuration Guide for IBM AIX on POWER Systems (64-Bit), Linux x86-64, Oracle Solaris on SPARC (64-Bit), and HP-UX Itanium or Oracle Database Gateway for APPC Installation and Configuration Guide for Microsoft Windows

Once this step is completed, tracing is enabled for the desired gateway system identifier.

Enabling the Gateway Trace Dynamically from PL/SQL

The following is only needed for user-written TIPs. PGAU-generated TIPs automatically include the following facilities. Refer to "Controlling TIP Runtime Gateway Exchange Tracing" for more information.

Make the following changes to the PL/SQL application that calls the Transaction Interface Package(s) to execute remote transaction(s).

  1. Add a call to PGATCTL before any calls to TIP initialization functions are made:

    PGATCTL@dblink(convid,
               traceF,
               traceS);

    Where Table 8-5 describes the parameters in PGATCTL:

    Table 8-5 PGATCTL Parameters

    Parameter Description

    dblink

    is the name of the database link to the gateway

    convid

    For a gateway using SNA: Conversation identifier returned by the PGAINIT function to be used to identify the conversation.

    For a gateway using TCP/IP: Socket file descriptor returned by the PGAINIT function to be used to identify the conversation

    traceF

    is the trace control function to be performed.

    traceS

    specifies which traces are to be enabled, as described previously in the discussion of the SET TRACE_LEVEL initialization parameter.

    This call sets the trace flags for all new conversations started after the call to the value specified by traceS.

  2. Recompile the PL/SQL application to pick up the new trace call.