Getting Started

The material in this chapter supplements information in the Databridge Client Administrator's Guide.

As stated previously, the Databridge Client for Kafka uses client control tables in the same way as relational database clients. These control tables are common to all clients and have the same layout. See Databridge Client Control Tables in the Databridge Client Administrator's Guide for a complete description of these tables.

All Databridge Client commands must be run with the current directory set to the working directory associated with the data source, as these commands rely on the files in the config sub-directory and write the output to the logs folder. To customize how DMSII data sets get mapped to their corresponding tables, you must write user scripts in SQL. For instructions, see Customizing with User Scripts in the Databridge Client Administrator's Guide.

Creating Client Control Tables

Before you clone a DMSII data source, you must create the Client control tables using the Client configure command.

Create the global working directory.
Create the data source working directory.
Set the current directory to be the data source working directory.
Create a config sub-directory.
Run an export command to create a text configuration file in the config folder.
Edit the text configuration file to setup the Oracle connection parameters as well as the Kafka specific parameters. Refer to the Getting Started section of the Databridge Client Administrator's Guide.
Run an import command to create a binary configuration file in the config folder which the Client will use by default.
Run a configure command to create the Client control tables.

The material in the Creating Client Control Tables section of the Databridge Client Administrator's Guide applies to the Kafka Client.

Defining a Data Source

Refer to Defining a Data Source section of the Databridge Client Administrator's Guide. All information in this chapter applies to the Kafka Client except for the data table creation and index creation user scripts as these do not apply to the Kafka Client.

The Decoding DMSII Dates, Times, and Date/Times section of the Databridge Client Administrator's Guide applies to the Kafka Client with the exception that the dates are written out using the format specified by the span_date_format parameter in the [Kafka] section of the client configuration file.

The Creating Indexes for Tables section of the Databridge Client Administrator's Guide does not apply to the Kafka Client.

The Databridge Client Administrator's Guide section on Adding a Non DMSII Column applies to the Kafka Client; in fact it supports more types of external columns than the relational database clients do. The table below shows the values for all the external column types supported by the Kafka Client.

Bit	Value	User Column Name	Description
1	1	update_type	Database update type, as follows: 0 for extract 1 for create 2 for delete (bit 10 must also be enabled) 3 for modify Note: You cannot use this value at the same time as bit 11.
2	2	update_time	Time the update was written to the flat file.
3	4	update_ts	N/A
4	8	audit_ts	DMSII audit file timestamp. This column is set to NULL during the initial clone. Note: You cannot use this bit at the same time as bit 13.
5	16	audit_filenum	Audit file number
6	32	audit_block	Audit block serial number (ABSN)
7	64	source_name	Data source name
8	128	source_id	Data source identifier as defined in the DATASOURCES Client control table
9	256	my_id	N/A
10	512	deleted_record	N/A
11	1024	update_type	Expanded database update type as follows: 0 for extract 1 for create 2 for delete 3 for modify NOTE: If you reuse the key for this record, inserting the new duplicate record removes the key. You cannot use this value at the same time as bit 1 or bit 10. Bits 10 and 11 are compared in Preserving Deleted Records. This bit and bit 1 work in the same way, except that this bit preserves the deleted image.
12	2048	source_id	Data source identifier as defined in the DATASOURCES Client control table (key item)
13	4096	audit_ts	Expanded audit file time. This column contains the DMSII audit file timestamp during updates and the starting time of the data extraction during extraction. NOTE: You cannot use this bit at the same time as bit 4.
14	8192	user_column1	Generic user column whose entry is left as NULL.
15	16384	sequence_no	N/A
16	32768	delete_seqno	N/A
17	65536	create_time	Time when the record was created in the flat file (PC time).
18	131072	user_column2	N/A
19	262144	user_column3	N/A
20	524288	user_column4	N/A
21	1048576	change_code	The one letter change code for an update A=Add, D=Delete, M=Modify
22	2097152	strnum	The structure number for the data set.
23	4194304	strname	The data set name.
24	8388608	rectype	The record type for variable format data sets.
25	16777216	modflag	A flag that indicates if an update is part of a MODIFY BI/AI pair. A value of 1 for an ADD or a DELETE, indicates that the ADD was originally the AI or DELETE was originally the BI of a MODIFY BI/AI pair.
26	33554432	mode	The mode of the data set as stored in the ds_mode column of the DATASETS Client control table. 0=CLONE, 1=FIXUP, 2=UPDATE.
27	67108864	fmt_level	The format level for the data set.
28	134217728	audit_seg	The segment number of the audit location within the audit file (audit_seg column in DATASETS Client control table)
29	268435456	audit_inx	The index of the audit location within the segment of the audit file (audit_inx column in DATASETS Client control table)
30	536870912	stacknbr	The stack number of the corresponding task on the MCP.

Note

Values for All the Externals Column Types Supported by the Kafka Client

The value for the Bit column in this table is equal to the value in the dms_subtype column of the DATAITEMS Client control table. The exception are bits 14, 18, 19, and 20 which result in a dms_subtype of 0. Bits are numbered from right to left; the right-most bit is 1.

The Generate Command

The generate command creates a schema file for every table associated with the data sets being replicated. The files, which are named tablename_schema.json, reside in the "dbscripts" directory. This file describes the layout of the corresponding table in JSON format. An example of such a file for the table customer is below:

{
    "namespace": "TESTDB",
    "name": "customer",
    "pk_name": "cust_rsn_set",
    "pk_members": "(thersn)",
    "fields": {
        "update_type": "int",
        "thersn", "string",
        "acct_number", "int",
        "address", "string",
        "phone_number", "string"
     }
}

In addition to creating the schema files, the generate command also creates the topic_config.ini file in the config folder that allows you to associate a topic name with one or more tables. By default the client uses the topic name of source_tablename for the various tables. This can be changed by editing the file and setting the topic name to use the desired names. For an example of such a file for the data source TESTDB refer to example 1-1 in chapter 1.

Under normal circumstance the above mentioned files are only updated when the parent data set has bit 2 of the status_bit column set. Adding the -u option to the command causes all the schema files and the topic_config.ini file to be updated. The client always reads the topic_config.ini file before saving the topic names. If the file does not exist it gets created with the default topic names.

The Process Command

The process command is the main command of the Databridge Kafka Client. This command creates output streams for all data set tables where the active column is 1 in the corresponding entries of the DATASETS client control table. Since the define command initializes the ds_mode column to 0 (indicating that the data set needs to be cloned), all the selected data sets are cloned the first time you run a process command.

Note

If you do not select specific data sets in the data set global mapping script, the Databridge Client automatically clones all data sets except for remaps, the restart data set, and the global data set.

Once the data sets are cloned, subsequent runs of the process command will store updates in the Kafka repository. For insert operations all columns will be present, for delete operations, only columns that are keys will be present in the update record. In the case of updates, the keys and changed columns will be present in the record. All records contain the update_type field, which is 0 for extract records, 1 for insert operations after the data extraction completes, 2 for delete operations, and 3 for update operations.

The process command to update the data in the Kafka repository can be scheduled. The schedule becomes effective after the process command is run for the first time. For more information, see the Scheduling Updates section in the Databridge Client Administrator's Guide.

Before running the process command, ensure that the current directory is set to the working directory that was created for this data source.

For more information on the process command refer to the Databridge Client Administrator's Guide.

The Clone Command

Refer to The Clone Command section of the Databridge Client Administrator's Guide for documentation on this command.