Altered Data Sets
In This Chapter
This chapter gives you programming tips and examples for altering data sets.
Overview
An altered data set is a DMSII data set to which you apply a data item conversion routine (custom reformatting routine) to reformat data items in a data set to different layouts.
Note
The Databridge API is not involved in altering data sets; however, ALGOL programming is required.
You can accomplish any of the following tasks by altering a data set:
-
Flatten OCCURS clauses—Involves changing an occurring item to a series of individual items.
However, if you plan to clone the data set with the Databridge Clients, you may want to use the Databridge Clients to flatten OCCURS clauses. Refer to the Databridge Client Administrator's Guide for more information.
-
Subdivide compound items—Allows the secondary database to search and retrieve data for the individual items rather than the compound item as a whole.
-
Convert or format dates—Involves changing the date from one format to another.
The Databridge Clients, however, provide date formats that are often less expensive to implement. Refer to the Databridge Client Administrator's Guide for more information on how the Databridge Clients format and convert dates.
-
Expand compressed data—Allows you to expand data that has been stored in a compression format (such as a digital picture) or a delimited format.
-
Convert data in a proprietary format to a well-known format—Involves changing the data from one kind of format to another.
-
Merge a list data items in to a single data item
To alter a data set, you must modify one of the provided sample reformatting procedures or write your own reformatting procedure and declare it to be internal or external to your tailored support library.
In addition, you must list (declare) the data items you want to alter in the ALTER section of the DBGenFormat parameter file. By making the ALTER declaration, you indicate which reformatting routines you want DBGenFormat to apply to the data items you named. Then, whenever a DBGenFormat-generated formatting routine (such as COMMAFORMAT or BINARYFORMAT) encounters data items from the ALTER declaration data set statement list, that DBGenFormat formatting routine calls the particular reformatting routine indicated in the ALTER declaration. See
ALTER Declaration Syntax for more information about how to determine what kind of a reformatting routine you want to write. For example, if you are reformatting similar items, such as timestamps, you would typically use the same reformatting routine to reformat all of them.
Finally, you must compile a tailored support library and enter the name of the tailored support library in the appropriate Accessory parameter file. When this process is completed, the Accessory can use your altered data set.
The provided sample reformatting procedures are listed as follows:
- PATCH/DATABRIDGE/SAMPLE/SUPPORT/REFORMAT for internal reformats—Sample External Reformatting Procedure shows how to use an internal reformatting procedure.
- SYMBOL/DATABRIDGE/SAMPLE/REFORMAT for external reformats—Sample External Reformatting Procedure shows how to use an external reformatting procedure.
The following sections include altered data set examples:
-
ALTER Declaration Syntax contains an example ALTER declaration.
-
Example Altered Data Set for Flattening OCCURS shows how to use altered data sets to flatten OCCURS.
Altering a Data Set
To alter data sets, complete the following steps:
-
Read this entire chapter so that you get an understanding of how the code you write for your altered data sets relates to the actual ALTER declarations you make in the DBGenFormat file.
See ALTER Restrictions for more information about making ALTER declarations.
-
Look at the data items you want to convert so you can get an idea of how many reformatting routines you need to code. Keep in mind that several ALTER declarations can call the same reformatting routine.
-
Familiarize yourself with the following samples, which illustrate several ways to apply reformatting procedures:
- Sample Internal Reformatting Procedure
- Sample External Reformatting Procedure
- Example Altered Data Set for Flattening OCCURS
-
Use CANDE or another editor to retrieve the DBGenFormat parameter file DATA/GENFORMAT/SAMPLE/CONTROL.
For a general description of the DBGenFormat parameter file, refer to the Databridge Host Administrator's Guide.
-
Rename the file, as follows:
DATA/GENFORMAT/databasename/CONTROLwhere databasename is the name of the database that contains the items you are altering and for which you are creating the tailored support library.
-
Make the following declarations in the DBGenFormat parameter file (DATA/GENFORMAT/databasename/CONTROL):
-
Declare the reformatting procedure.
See Declaring Internal and External Reformatting Procedures for instructions.
-
Declare all of the altered data sets.
See ALTER Declaration Syntax for instructions.
-
-
Save DATA/GENFORMAT/ databasename /CONTROL.
-
Write the reformatting procedure as follows:
Note
The reformatting routines must be in ALGOL; however, you can write routines that invoke a COBOL library.
If you Do this Declared an internal reformatting procedure Write the reformatting procedure patch file, and give it the name you specified in the DBGenFormat parameter file.
NOTE: It is recommended that your reformatting procedure use a case statement to identify the individual reformatting routines.Declared an external reformatting procedure Write the reformatting procedure library source file.
Compile the library containing the external reformatting procedure as the name you specified in the DBGenFormat parameter file.
NOTE: It is recommended that your reformatting procedure use a case statement to identify the individual reformatting routines.
You do not have to complete the external library file before going on to the next step. You must, however, finish writing and compiling it before you run an Accessory that uses the external reformat. -
Compile the tailored support library as follows:
START WFL/DATABRIDGE/COMP ("SUPPORT", "databasename" ["logicaldatabasename"])Where Is "SUPPORT"The literal that represents the DBSupport program
The quotation marks are required."databasename" The name of the database for which you are creating the tailored support library
The database name can include a usercode and pack, which are used to locate the database DESCRIPTION file, as follows:
"(usercode)databasename ON packname"
The quotation marks are required.This WFL compiles layout tables for each data set in the database designated by databasename or logicaldatabasename. This results in the new tailored support library titled as follows: OBJECT/DATABRIDGE/SUPPORT/databasename
— or —
OBJECT/DATABRIDGE/SUPPORT/databasename/logicaldatabasenameThese data set-specific layout tables contain the offsets and sizes of individual data items, including those in the ALTER data set declaration.
Caution
If you have two databases with the same name under different usercodes, and you are running Databridge from a third usercode, be careful when you create a tailored support library. In this case, the second library you compile overwrites the first, because Databridge strips the usercode and pack name from the database name to create the tailored support library title.
-
Enter the name of the tailored support library in the appropriate Accessory parameter file, as follows:
For Do this Databridge Clients In the DBServer parameter file, enter the tailored support library name for the SUPPORT option.
For more information, refer to the Databridge Host Administrator's Guide.Other Accessories In the Accessory parameter file, enter the tailored support library name for the SUPPORT option.
For more information, refer to the Databridge Host Administrator's Guide.
What to Do Next
Repeat these steps for each data set you want to alter.
If you encounter problems when creating or compiling your altered data set, see the chapter on Troubleshooting. This chapter provides specific troubleshooting tips for writing reformatting procedures and working with altered data sets.
ALTER Restrictions
You must be aware of the following restrictions when you use the ALTER declaration:
- If you alter a GROUP item, the reformatting routine must format the entire group.
- You cannot alter an item subordinate to a GROUP OCCURS. In this case you must alter the entire GROUP.
- If you alter an item with an OCCURS clause, the reformatting procedure must reformat all occurrences at once. (The source length is the total size for all occurrences.)
- If you alter an item in a data set that has more than one data item with the same name, only the last item is altered. (This can happen if a data item is found in more than one of the variable-format parts of a variable format data set.)
-
If the reformatted item is to be signed, you must have at least one space between the "S" and the declared size, as in the following example:
ACCT-BALANCE NUMBER (S 11, 2); -
If the reformatted item is an occurring GROUP (or FIELD), then the OCCURS clause must immediately follow the word GROUP (or FIELD), as in the following example:
MONTHLY-SUMMARY GROUP OCCURS 12 (SALES NUMBER (8); ... ); -
Items altered to type IMAGE are treated as ALPHA items by the Databridge Clients, except that the Databridge Clients do not translate or interpret IMAGE items.
- To merge data items, the data items must be adjacent and must exist in the same parent group.
ALTER Declaration Syntax
You must make one ALTER declaration for each data set that contains data items you want to alter. The following is the ALTER declaration for the DBGenFormat parameter file (DATA/GENFORMAT/databasename/CONTROL):
ALTER datasetname
(
[uservalue] originaldataitemname newitemtype(n)
[uservalue] originaldataitemname newitemtype(n)
[uservalue] dataitemname1, dataitemname2, ... AS newdataitemname
newitemtype(n) <–– This syntax specifies data items to be merged into one.
...
);
| Where | Is |
|---|---|
| datasetname | The name of the data set from which you want to convert data items The data set name must match the data set name in the DASDL. |
[uservalue] or [DEFINEuservalue] or [REDEFINE] |
A value that indicates the type of alteration made by the reformatting procedure You must include the brackets. The user value corresponds to a reformatting routine and can be any integer greater than or equal to 0. It is usually less than 1024 so that the reformatting procedure can use it as a case value (as in the example reformatting procedures). You can use the same integer (and therefore call the same formatting routine) for more than one data item. If you are reformatting similar items (for example, timestamps) you would typically assign the same user value to each one so that the reformatting procedure uses the same code (reformatting routine) to reformat all of them. Note for merging data items: If [uservalue] is an integer, the Reformat routine is called with the source offset of the first data item and the total length of all of the items. A value of DEFINE introduces a virtual data item into the altered data set. DEFINE does not apply to merging data items. A value of REDEFINE redefines the data in place rather than using the reformatting procedure. Use the REDEFINE command to subdivide elementary items and flatten OCCURS (see Example Altered Data Set for Flattening OCCURS). Two qualifications exist for using REDEFINE, as follows:
|
| originaldataitemname | The name of the data item as it appears in the DASDL |
| newitemtype(n) | The new data type for the changed data item or merged data items where newitemtype is the new type, such as ALPHA, IMAGE, or NUMBER, and n is the size of the field (NUMBER items can have a scaling factor and sign, as in NUMBER (S, 6,2). See Virtual and Alter Data Item Types for Databridge specific data item types you can specify. |
| dataitemname1, dataitemname2, dataitemnamex | A comma-separated list of adjacent data items in the same parent group that will be merged into a new data item |
| newdataitemname | The name of the merged data item. This is the name that Accessories will see in place of the listed data items. |
REDEFINE Errors
If the REDEFINE size differs from the original size, DBGenFormat displays the following error:
dataitem original size: origsize but REDEFINE size: newsize
If the original data item was not on a byte boundary but the REDEFINE data type requires it to start on a byte boundary (as for GROUP items), DBGenFormat displays the following error:
REDEFINE of dataitem requires byte-alignment
Example 1
This example demonstrates how to merge data items using the REDEFINE command and a reformatting routine coded in the ALGOL procedure.
Assume that the original DASDL for the ORDERINFO data set contains the following:
ORDERINFO DATASET
(
ORD-YY NUMBER (4);
ORD-MM NUMBER (2);
ORD-DD NUMBER (2);
...
ORD-CITY ALPHA (16);
ORD-STATE ALPHA (2);
ORD-ZIP NUMBER (4);
...
);
However, you want to merge the year, month, and day data items into one date item, and you want to merge the city, state and zipcode into one alphanumeric item.
In the DBGenFormat parameter file, you could write the ALTER declaration for the ORDERINFO data set as follows:
ALTER ORDERINFO
(
[REDEFINE] ORD-YY,ORD-MM, ORD-DD AS
ORD-YYMMDD NUMBER (YYMMDD);
[2] ORD-CITY, ORD-STATE, ORD-ZIP AS
ORD-ADDR-CSZ ALPHA (45);
);
Example 2
This example demonstrates how to alter a data set using the DEFINE command, the REDEFINE command, and reformatting routines coded in the ALGOL procedure.
Assume that the original DASDL for the BANK data set contains the following:
BANK DATASET
(
BANK-ID NUMBER (4);
BANK-NAM ALPHA (11) INITIALVALUE "BRANCH NAME";
BANK-ADDR ALPHA (30);
BANK-TS REAL; % timestamp
BANK-ROUTE NUMBER (9);
);
However, you want to use the ALTER declaration to change the BANK data items as follows:
- Change BANK-ID from NUMBER (4) to NUMBER (6)
- Change BANK-ADDR from ALPHA (30) to a group containing three elementary data items
- Change BANK-TS (timestamp) from REAL to ALPHA (30), which contains a readable date and time
- Change BANK-ROUTE from NUMBER (9), to a group containing three data items
- Add a BANK-PRES virtual data item of ALPHA (40)
In the DBGenFormat parameter file, you would write the ALTER declaration for the BANK data set as follows:
ALTER BANK
(
[1] BANK-TS ALPHA (30); % was REAL
[2] BANK-ID NUMBER (6); % was NUMBER (4)
[DEFINE 4] BANK-PRES ALPHA (40); % virtual
[REDEFINE] BANK-ADDR GROUP % was ALPHA (30)
(
BR-CITY ALPHA (18);
BR-STATE ALPHA (2);
BR-ZIP ALPHA (10);
);
[3] BANK-ROUTE GROUP
(
BR-1 NUMBER (2);
BR-2 NUMBER (3);
BR-3 NUMBER (4);
);
);
In this example, the DBGenFormat formatting routines call the reformatting procedure to reformat TS, BANK-ID, and BANK-ROUTE.
Each [uservalue] in the ALTER declaration corresponds to a specific reformatting routine (that you have coded) in the reformat procedure. When the DBGenFormat formatting routines receive a BANK record, they call the reformatting procedure (once for each data item) with the following information:
- The value that corresponds to the specific reformatting routine in the reformat procedure (which is 1, 2, or 3 in this example)
- The original location and size of BANK-ID, BANK-ROUTE, and BANK-TS
- The location and size of where the reformatting procedure should place BANK-ID, BANK-ROUTE, and TS
When the formatting routines call the reformatting procedure for the virtual item BANK-PRES, they supply the 4 as the [uservalue], but the source offset and size is 0 because there is no source item. The reformatting routine must retrieve the data from some external source (such as another database, file, and so on) and copy it into the destination array.
The DBGenFormat formatting routines do not call any reformatting routines for BANK-ADDR because a REDEFINE command redefines the data in place.
Note
BANK-ROUTE cannot be a REDEFINE because GROUP items are multiples of whole bytes, and, therefore, are required to be byte-aligned.
Declaring Internal and External Reformatting Procedures
Reformatting procedures for altered data sets must be declared as internal and external reformats in the DBGenFormat parameter file. Declare the reformatting procedures in the DBGenFormat parameter file using the syntax below in this section, as well as the Declaring External Reformats section.
Consider the following information before you choose whether to declare an internal or external reformat:
| Internal Reformat Description | External Format Description |
|---|---|
| Internal reformats are compiled as patches to your tailored support library. This requires that you recompile DBSupport via WFL/ DATABRIDGE/COMP each time you update the internal reformatting procedure. | External reformats are linked at run-time to a user-written format library, so they can be recompiled any time without having to recompile DBSupport. |
| Internal reformats do not have to specify how to link to DBEngine or DBSupport. | External reformats must link to DBSupport and DBEngine at the proper time. |
| Internal reformats do not have to verify that their interface version matches DBEngine. | External reformat libraries must ensure that their interface version matches DBEngine. |
Declaring Internal Reformats
To declare the altered data set reformatting procedure as an internal reformat, use the following syntax in the DBGenFormat parameter file (the comments in the file indicate where this declaration should go):
INTERNAL REFORMAT reformattingprocedure IN "patchfiletitle"
where reformattingprocedure is the name of a reformatting procedure, and patchfiletitle is the title of the ALGOL file that you created as a patch for DBSupport.
Declaring External Reformats
If you want to write your own ALGOL library for a reformat, you can reference it in the tailored support library by using the following syntax in the DBGenFormat parameter file (the comments in the file indicate where this declaration should go):
EXTERNAL REFORMAT reformattingprocedure IN "objectfilename"
where reformattingprocedure is the name you have given to the reformatting procedure and objectfilename is the file title of your compiled ALGOL library code.
Writing an Internal Reformatting Procedure
If you declared an internal reformat in DBGenFormat, you must write an ALGOL patch file containing the reformatting procedure that converts altered data items. The patch file may include global declarations in addition to the reformatting procedure itself. The patch file should not include the EXPORT declaration for the reformatting procedure. DBGenFormat automatically generates the appropriate EXPORT declaration.
See PATCH/DATABRIDGE/SAMPLE/SUPPORT/REFORMAT in Sample Internal Reformatting Procedure as an example of an internal reformatting procedure.
Writing an External Reformatting Procedure
If you declared an external reformat in DBGenFormat, you must write your own library program that contains the reformatting procedure and does the following:
-
Sets the $ INCLUDE_ENGINE option (and the $INCLUDE_SUPPORT option if you call any DBSupport entry points) and includes SYMBOL/DATABRIDGE/INTERFACE using the following ALGOL $INCLUDE statements:
$SET INCLUDE_ENGINE
$INCLUDE "SYMBOL/DATABRIDGE/INTERFACE"or
$SET INCLUDE_ENGINE INCLUDE_SUPPORT
$INCLUDE "SYMBOL/DATABRIDGE/INTERFACE"Note
Do not invoke the DBLINKENGINE define in your library. DBSupport automatically links your library to the correct instance of DBEngine. Do not attempt to call any DBEngine entry points before the library freezes. Otherwise, your library will link to a different instance of DBEngine than the Accessory, and it might return incorrect information and errors.
-
Calls DBINTERFACEVERSION to verify that your program was compiled against the same API file (SYMBOL/DATABRIDGE/INTERFACE) as DBEngine.
Note
Do not call DBINITIALIZE. DBINITIALIZE will undo the initialization that the Accessory has already completed.
See OBJECT/DATABRIDGE/REFORMAT in Sample External Reformatting Procedure as an example of an external reformatting procedure.
Example: Internal Reformatting Procedure
The ALGOL source code for this example is as follows:
$ SET OMIT 09900000
------------------------------------------------------------------------
09900100
09900130
Module: PATCH/DATABRIDGE/SAMPLE/SUPPORT/REFORMAT 09900140
09900150
Project: Databridge 09900160
09900170
Description: Databridge Sample Reformatting Patch 09900180
09900190
(c) Copyright 2019 Micro Focus or one of its affiliates. 09900290
09900430
------------------------------------------------------------------------
09900440
09902000
Modification history 09902100
-------------------- 09902200
09902300
Version 30.001 09902400
Initial release. 09902500
09902600
This is a sample patch to DBSupport for reformatting 09902700
data items in conjunction with the GenFormat ALTER construct. 09902800
To include this patch in DBSupport put the following
declaration 09902900
in the GenFormat parameter file: 09903000
09903100
INTERNAL REFORMAT IN "PATCH/DATABRIDGE/SAMPLE/SUPPORT/REFORMAT"
09903200
09903300
Version 41.471 09903400
1 Added cases 6 and 7 to illustrate handling virtual data items 09903500
declared with the [DEFINE n] syntax in GenFormat. 09903600
Case 6 also illustrates the necessary code to handle formatting
09903700
a null record when the reformatting routine normally stores a 09903800
constant value. 09903900
09904000
Version 41.474 09904100
2 Added defines for 8-bit offsets and 8-bit sizes and changed 09904200
examples accordingly. 09904250
09904260
Version 61.001 09904300
1 The Reformat routine now returns false (indicating failure) if
a09905000
fault is detected. 09906000
End History 09999990
$ POP OMIT 09999999
50000000
50002000
50005000
string TSMsg; % timestamp message 50006000
50007000
real FaultNbr; 50007100
ebcdic array 50007200
FaultHistory [0:79]; 50007300
50009000
boolean procedure Reformat (UserValue, UpdateInfo, 50010000
% -------- 50011000
SourceRec, SourceOfs, SourceSz, 50012000
DestRec, DestOfs, DestSz); 50013000
50014000
% Custom reformatting of a data item. This user-written 50015000
% procedure converts a data item used in a non-standard way into50016000
% a "standard" data item defined in the GenFormat parameter
file50017000
% using the ALTER declaration. 50018000
50019000
% For example, a "days-since" data item might be converted to 50020000
% a YYYYMMDD date. 50021000
50022000
% Returns true if item successfully reformatted. 50023000
50024000
value UserValue, SourceOfs, SourceSz, DestOfs, DestSz; 50025000
50026000
integer UserValue; 50027000
% Input: user-specified value associated with the data 50028000
% item (from GenFormat) 50029000
50030000
array UpdateInfo [0]; 50031000
% Input: information describing the update 50032000
50033000
array SourceRec [0]; 50034000
% Input: dataset record containing source item 50035000
50036000
integer SourceOfs; 50037000
% Input: offset of the source item in SourceRec 50038000
% (4-bit digits) 50039000
50040000
integer SourceSz; 50041000
% Input: size of the source item in SourceRec 50042000
% (4-bit digits) 50043000
50044000
array DestRec [0]; 50045000
% Output: reformatted dataset record 50046000
50047000
integer DestOfs; 50048000
% Input: offset of the destination item in DestRec 50049000
% (4-bit digits). 50050000
50051000
integer DestSz; 50052000
% Input: size of the destination item in DestRec 50053000
% (4-bit digits) 50054000
50055000
begin 50056000
hex array 50057000
Source4 [0] = SourceRec, 50058000
Dest4 [0] = DestRec; 50059000
50060000
ebcdic array 50061000
Source8 [0] = SourceRec, 50062000
Dest8 [0] = DestRec; 50063000
50063100
define SourceSz4 = SourceSz #; 50063200
define SourceSz8 = (SourceSz / 2) #; 50063300
50063400
define DestSz4 = DestSz #; 50063500
define DestSz8 = (DestSz / 2) #; 50063600
50064000
define SourceOfs4 = SourceOfs #; 50064100
define SourceOfs8 = (SourceOfs / 2) #; 50064200
50064300
define DestOfs4 = DestOfs #; 50064400
define DestOfs8 = (DestOfs / 2) #; 50064500
50066000
own integer 50066200
BankIDOfs; 50066250
own integer 50066300
BankIDSz; 50066350
EBCDIC value array 50066400
BankName (80"BANK"); 50066600
50066800
own boolean 50067000
Initialized; 50067050
50067100
procedure Initialize; 50067200
% ---------- 50067400
begin 50067600
array ITEM_INFO [0 : II_ENTRY_SIZE - 1]; 50067800
integer BankStrNum; 50068000
50068200
% Get the offset and size of BANK-ID. 50068400
50068600
DBStrNum (BankName, BankStrNum); 50068800
DBItemInfo (BankStrNum, 0, "BANK-ID", ITEM_INFO); 50069000
BankIDOfs := ITEM_INFO [II_OFFSET]; 50069200
BankIDSz := ITEM_INFO [II_SIZE]; 50069400
50069500
Initialized := true; 50069600
end Initialize; 50069800
50070000
50070200
if ^ Initialized then 50070400
begin 50070600
Initialize; 50070800
end; 50071000
50071200
on anyfault [FaultHistory: FaultNbr], 50072100
begin 50072200
DBDisplayFault ("Reformat: ", FaultNbr, FaultHistory); 50072300
Reformat := false; 50072350
end; 50072400
50072500
Reformat := true; 50073000
50074000
case UserValue of 50075000
begin 50076000
1: % timestamp 50077000
50078000
% call Engine to convert 50079000
DBTIMESTAMPMSG 50080000
(real (Source4 [SourceOfs4], SourceSz4), 50081000
TSMsg); 50082000
50083000
% see if it will fit 50086000
if length (TSMsg) > DestSz8 then 50087000
begin 50088000
TSMsg := take (TSMsg, DestSz8); 50089000
end; 50090000
50091000
% copy the timestamp message into dest 50092000
replace Dest8 [DestOfs8] by 50093000
TSMsg, 50094000
" " for DestSz8 - length (TSMsg); 50095000
50096000
2: % bigger branch id 50097000
50098000
replace Dest4 [DestOfs4] by 50099000
4"00", 50100000
Source4 [SourceOfs4] for 4; 50101000
50102000
3: % split out branch address 50103000
50104000
replace Dest8 [DestOfs8] by 50105000
Source8 [SourceOfs8] for 30, 50106000
"-" for 30, 50107000
"your town here ", 50108000
"your region "; 50109000
50110000
6: % Bank president name virtual data item. 50110020
% If this is a null record then we want this to 50110040
% be null also so that the client isn't 50110060
% confused about what is the null value. 50110080
50110100
if NullRecord (UpdateInfo) then % null record 50110120
begin 50110140
replace Dest4 [DestOfs4] by 4"F" 50110160
for DestSz4; 50110180
end 50110200
else 50110220
begin 50110240
replace Dest8 [DestOfs8] by 50110260
" " for DestSz8; 50110280
replace Dest8 [DestOfs8] by 50110300
"Pres. Greenspan"; 50110320
end; 50110340
50110360
7: % Bank phone number virtual data item 50110380
% pieced together from a constant and the 50110400
% BANK-ID. 50110420
50110440
replace Dest8 [DestOfs8] by 50110460
"202-555-", 50110480
Source4 [BankIDOfs] for 4 50110500
with HEXTOEBCDIC; 50110520
50110540
else: % unrecognized UserValue 50111000
% copy data as-is 50111100
50111200
replace Dest4 [DestOfs4] by 50111300
Source4 [SourceOfs4] 50111400
for min (SourceSz4, DestSz4); 50111500
50111600
Reformat := false; 50112000
end; 50113000
50114000
end Reformat; 50115000
50116000
Example: External Reformatting Procedure
The ALGOL source code for this example is as follows:
$ SET OMIT 09000000
------------------------------------------------------------------------
09000100
09000110
(c) Copyright 2019 Micro Focus or one of its affiliates. 09000120
09000130
Module: SYMBOL/DATABRIDGE/SAMPLE/REFORMAT 09000140
09000150
Project: Databridge 09000160
09000170
Description: Databridge Sample Reformatting Library 09000180
09000190
09000430
------------------------------------------------------------------------
09000440
09002000
This is a sample reformatting library to illustrate how to reformat
09002100
data items in conjunction with the GenFormat ALTER construct. 09002200
09002300
Modification history 09002400
-------------------- 09002500
09002600
Version 25.001 09002700
1 Initial release. 09002800
09002900
Version 30.001 09003000
1 Added fault-trapping code to handle SEG ARRAY ERR, INVALID 09003100
INDEX, etc. faults caused by this library. 09003200
09003300
2 If this routine receives an unrecognized UserValue it will 09003400
now copy the source data to the destination without 09003500
modification. Previously it did nothing in this situation. 09003600
09003700
Version 41.484 09003800
1 Added cases 6 and 7 to illustrate handling virtual data items 09003900
declared with the [DEFINE n] syntax in GenFormat. 09004000
Case 6 also illustrates the necessary code to handle formatting
09004100
a null record when the reformatting routine normally stores a 09004200
constant value. 09004300
09004400
2 Added defines for 8-bit offsets and 8-bit sizes and changed 09004500
examples accordingly. 09004600
09004700
Version 41.485 09004800
3 Added initialization code to check DBInterface version. 09004900
09005000
Version 51.501 09005100
End History 09005200
$ POP OMIT 09005300
09005400
09005600
$ VERSION 06.003.0000
09999900Version
46000000
begin 46000100
46000200
$ SET INCLUDE_ENGINE 46000300
$ INCLUDE "SYMBOL/DATABRIDGE/INTERFACE" 46000400
46000500
string TSMsg; % timestamp message 46000600
46000700
real FaultNbr; 46000800
ebcdic array 46000900
FaultHistory [0:79]; 46001000
46001100
boolean procedure Reformat (UserValue, UpdateInfo, 46001200
% -------- 46001300
SourceRec, SourceOfs, SourceSz, 46001400
DestRec, DestOfs, DestSz); 46001500
46001600
% Custom reformatting of a data item. This user-written 46001700
% procedure converts a data item used in a non-standard way into46001800
% a "standard" data item defined in the GenFormat parameter
file46001900
% using the ALTER declaration. 46002000
46002100
% For example, a "days-since" data item might be converted to 46002200
% a YYYYMMDD date. 46002300
46002400
% Returns true if item successfully reformatted. 46002500
46002600
value UserValue, SourceOfs, SourceSz, DestOfs, DestSz; 46002700
46002800
integer UserValue; 46002900
% Input: user-specified value associated with the data 46003000
% item (from GenFormat) 46003100
46003200
array UpdateInfo [0]; 46003300
% Input: information describing the update 46003400
46003500
array SourceRec [0]; 46003600
% Input: dataset record containing source item 46003700
46003800
integer SourceOfs; 46003900
% Input: offset of the source item in SourceRec 46004000
% (4-bit digits) 46004100
46004200
integer SourceSz; 46004300
% Input: size of the source item in SourceRec 46004400
% (4-bit digits) 46004500
46004600
array DestRec [0]; 46004700
% Output: reformatted dataset record 46004800
46004900
integer DestOfs; 46005000
% Input: offset of the destination item in DestRec 46005100
% (4-bit digits). 46005200
46005300
integer DestSz; 46005400
% Input: size of the destination item in DestRec 46005500
% (4-bit digits) 46005600
46005700
begin 46005800
hex array 46005900
Source4 [0] = SourceRec, 46006000
Dest4 [0] = DestRec; 46006100
46006200
ebcdic array 46006300
Source8 [0] = SourceRec, 46006400
Dest8 [0] = DestRec; 46006500
46006600
define SourceSz4 = SourceSz #; 46006700
define SourceSz8 = (SourceSz / 2) #; 46006800
46006900
define DestSz4 = DestSz #; 46007000
define DestSz8 = (DestSz / 2) #; 46007100
46007200
define SourceOfs4 = SourceOfs #; 46007300
define SourceOfs8 = (SourceOfs / 2) #; 46007400
46007500
define DestOfs4 = DestOfs #; 46007600
define DestOfs8 = (DestOfs / 2) #; 46007700
46007800
own integer 46007900
BankIDOfs; 46008000
own integer 46008100
BankIDSz; 46008200
EBCDIC value array 46008300
BankName (80"BANK"); 46008400
46008500
define NullRecord (UI) = (UI [UI_STRIDX] = 0) #; 46008600
% true if null record 46008700
46008800
own boolean 46008900
Initialized; 46009000
46009100
procedure Initialize; 46009200
% ---------- 46009300
begin 46009400
array ITEM_INFO [0 : II_ENTRY_SIZE - 1]; 46009500
integer BankStrNum; 46009600
46009700
DBMTYPE DBRslt; 46009800
46009900
DBRslt := DBInterfaceVersion (DBV_VERSION, "Reformat:"); 46010000
if DBRslt NEQ DBM_OK then 46010100
begin 46010200
DBDisplayMsg (DBRslt); 46010300
MYSELF.STATUS := value (TERMINATED); 46010400
end; 46010500
46010600
% Get the offset and size of BANK-ID. 46010700
46010800
DBStrNum (BankName, BankStrNum); 46010900
DBItemInfo (BankStrNum, 0, "BANK-ID", ITEM_INFO); 46011000
BankIDOfs := ITEM_INFO [II_OFFSET]; 46011100
BankIDSz := ITEM_INFO [II_SIZE]; 46011200
46011300
Initialized := true; 46011400
end Initialize; 46011500
46011600
46011700
if ^ Initialized then 46011800
begin 46011900
Initialize; 46012000
end; 46012100
46012200
on anyfault [FaultHistory: FaultNbr], 46012300
begin 46012400
DBDisplayFault ("xReformat: ", FaultNbr, FaultHistory); 46012500
Reformat := false; 46012550
end; 46012600
46012700
Reformat := true; 46012800
46012900
case UserValue of 46013000
begin 46013100
1: % timestamp 46013200
46013300
% call Engine to convert 46013400
DBTIMESTAMPMSG 46013500
(real (Source4 [SourceOfs4], SourceSz4), 46013600
TSMsg); 46013700
46013800
% see if it will fit 46013900
if length (TSMsg) > DestSz8 then 46014000
begin 46014100
TSMsg := take (TSMsg, DestSz8); 46014200
end; 46014300
46014400
% copy the timestamp message into dest 46014500
replace Dest8 [DestOfs8] by 46014600
TSMsg, 46014700
" " for DestSz8 - length (TSMsg); 46014800
46014900
2: % bigger branch id 46015000
46015100
replace Dest4 [DestOfs4] by 46015200
4"00", 46015300
Source4 [SourceOfs4] for 4; 46015400
46015500
3: % split out branch address 46015600
46015700
replace Dest8 [DestOfs8] by 46015800
Source8 [SourceOfs8] for 30, 46015900
"-" for 30, 46016000
"your town here ", 46016100
"your region "; 46016200
46016300
6: % Bank president name virtual data item. 46016400
% If this is a null record then we want this to 46016500
% be null also so that the client isn't 46016600
% confused about what is the null value. 46016700
46016800
if NullRecord (UpdateInfo) then % null record 46016900
begin 46017000
replace Dest4 [DestOfs4] by 4"F" 46017100
for DestSz4; 46017200
end 46017300
else 46017400
begin 46017500
replace Dest8 [DestOfs8] by 46017600
" " for DestSz8; 46017700
replace Dest8 [DestOfs8] by 46017800
"Pres. Greenspan"; 46017900
end; 46018000
46018100
7: % Bank phone number virtual data item 46018200
% pieced together from a constant and the 46018300
% BANK-ID. 46018400
46018500
replace Dest8 [DestOfs8] by 46018600
"202-555-", 46018700
Source4 [BankIDOfs] for 4 46018800
with HEXTOEBCDIC; 46018900
46019000
else: % unrecognized UserValue 46019100
% copy data as-is 46019200
46019300
replace Dest4 [DestOfs4] by 46019400
Source4 [SourceOfs4] 46019500
for min (SourceSz4, DestSz4); 46019600
46019700
Reformat := false; 46019800
end; 46019900
46020000
46020100
end Reformat; 46020200
46020300
export Reformat; 46020400
46020500
freeze (temporary); 46020600
end. 46020700
Example: Altered Data Set for Flattening OCCURS
This section shows you the declarations you must make in order to flatten OCCURS using a REDEFINE command. Notice that no reformatting routines are used.
DASDL Declaration
This sample is the original DASDL declaration.
G DATA SET
(G-1 GROUP OCCURS 2
(G-ALPHA ALPHA (10);
G-NUM NUMBER (5);
);
G-KEY ALPHA (10);
);
ALTER Declaration in DBGenFormat
Make the following declaration in DATA/GENFORMAT/ databasename /CONTROL:
ALTER G
(
[REDEFINE] G-1 GROUP
(G-ALPHA-1 ALPHA (10);
G-NUM-1 NUMBER (5);
G-ALPHA-2 ALPHA (10);
G-NUM-2 NUMBER (5);
);
);
Example: Databridge NewId
The Databridge NewId (new identifier) feature is an implementation of pseudonymization. The goal is to substitute meaningful names and numbers for any personally identifiable data in a database while retaining the relationships that exist between records either in the same data set or different data sets.
For example, if an identifier links a customer to accounts associated with that customer, the substituted identifier will still maintain that linkage. This condition is called tokenization. Each unique identifier is mapped to a single token value. For instance, all instances of "Bill" might be mapped to “Chyna.”
The resulting database should not have any data that can be tied to a real person, as required by GDPR and other auditing requirements for test and development.
This feature is implemented in Databridge using extensions to the GenFormat ALTER declaration and a proprietary REFORMAT routine that maps the original data values to sensible but fictitious values, such that the same original value for a data item is always mapped to the same fictitious value.
To illustrate, here is the DASDL declaration for a CUSTOMER data set.
We create three customer records. If we clone the customer data set normally, without any tokenizing, it would look like this in the client database.
An ALTER of the CUSTOMER data set in the GenFormat file can specify that we want certain data items to be tokenized. In the screen shot above we want to tokenize all of the data items except the ones highlighted in yellow. The ones we do want tokenized are put in the ALTER declaration with one of the new X_... keywords, which include:
X_F_NAME
X_L_NAME
X_M_NAME
X_FL_NAME
X_FML_NAME
X_LFM_NAME
X_STREET
X_ADDRESS
X_CITY
X_STATE
X_EMAIL
X_PHONE
X_COUNTY
X_COUNTRY
X_POSTCODE
X_ZIPCODE
X_SSN
X_NUMBER
X_DATE
X_NULL
X_ZERO
X_SPACES
X_STAR
The ALTER might look like this.
The special REFORMAT function is provided in a separately licensed library (called OBJECT /DATABRIDGE/NEWID in the GenFormat excerpt below). The REFORMAT function contains pools of different kinds of names drawn from public lists. The list of first names comes from the Census Bureau’s list of most common first names. The street names are from San Francisco. Cities are from Georgia, and so on. Phone numbers and SSNs are just pseudo random numbers.
After compiling DBSupport with these changes and making sure the SOURCE declaration in the DBServer parameter file specifies the right DBSupport, we can do another clone and the results would look like this.
For comparison, here is the original data.
Notice that cust_id, cust_acct_id, and cust_type have their original values but the other fields have been changed to fictitious, but readable, values. Also note that the first and third records originally had cust_state =”NY” and both were converted to “NV” (for Nevada).
If a caret character, ^, follows the X_... keyword, the data will be uppercase.
Let’s change the ALTER so that the name is in “last, first middle” format and make the street address and city uppercase.
After recompiling DBSupport we do another clone. This time it looks like this:
Now the first and last names are in a different order (along with a middle name) and the address and city are capitalized. All of the other items stayed the same.
GenFormat and the REFORMAT routine make intelligent guesses regarding the format of numeric items such as SSN, ZIP code, and phone based on the length of the data item; they decide whether or not to insert delimiters and what kind of delimiters. For example, an 11-character ALPHA for an SSN will have two hyphens inserted into the 9-digit value.
We rely on the customer to create the ALTER declarations that specify which data items to tokenize and what type of data they contain. Generally speaking, the percentage of the database that contains personally identifying data is very small. The data regarding quantities, part numbers, dollar amounts, etc., typically do not need to be tokenized (although this implementation allows for it using the X_NUMBER token type).
The customer could completely wipe out the contents of a data item by using the X_NULL, X_ZERO, X_SPACES, and X_STAR token types.
DBEnterprise also supports all of this functionality using a special DLL. The customer makes the same changes to the GenFormat file as outlined above. There are no other changes other than installing the DLL in the Config directory. The DLL produces exactly the same mappings from the original data to the tokenized values so that the client can switch back and forth between DBEnterprise and DBServer.