Extracting source and data from IBM i

Posted on by

There are many different ways of getting source code and data out of the IBM i.

The method described in this article does it automatically for you.

The utility consist of a Windows Command (CMD) file and a VBScript. They perform all the tasks needed by having the Windows console FTP client doing the hard work. Before using the utility, please read how to use and configure it.

Full source and download are at the buttom.

How does it work

The utility will create a folder having the name of the library on your PC. The utility will then transfer source files and physical data files. For the source files, the utility will create a folder, named as the source file, on your pc and tranfer all source members as text files into that folder. Each data file is first converted to a CSV file on the IBM i and then transferred to the folder named as the library.

The file/folder structure of a library transferred with this will utility will look like this:

For the library ‘JWTOOLS’ the structure is:

Folder in which the ‘BackUpSrcDta’ utility is installed. Note the ‘JWTOOLS’ folder and the ‘JWTOOLS.DspFD’ file.
Content of the folder ‘JWTOOLS’.
Content of the ‘QRPGLESRC’ folder.

The utility makes use of the Windows FTP console client to do the hard work. The utility creates script files that the FTP client executes. A number of log files is generated during the run of the utility. You will have to check these when the utility is finished.

The files ‘*.DspFD’ and ‘*.MBRLIST’ contains the output from the IBM i command ‘DSPFD’. To get information about the text associated with a file or a member, you must look into these files.

Configuring the tool

To extract one or more libraries you must configure the utility. First install the two files ‘BackUpSrcDta.CMD’ and ‘BackUpSrcDta.vbs’ into a folder on a drive with sufficent space to hold the data and source files. Then open the file ‘BackUpSrcDta.CMD’ in a text editor. You can see the section where you must add the libraries that you want to extract from the IBM i:

Section of ‘BackUpSrcDta.CMD’ where you configure which libaries to extract.

In the picture above, you can see that I have put in QGPL and MYLIB to be extracted. Change this to the libraries that you need. Save the file and then launch the CMD file from a Command prompt.

Things to be aware of using this utility

This utility was written to extract source code and data from the IBM i. There is no counterpart to upload data and/or source code.

Before you start, please consider the following:

  • Does source contain binary data?
    A number of tools exist that can insert binary data into the source code, typically to display comments in colours. The data that is insert, is binary and will not translate as a colour tag then the source is transferred to the PC.
  • Does physical data files contain binary data?
    The utility relies on the command CPYTOIMPF to do the translation from the native format to CSV files. This means that packed, zoned, Date, Time etc. fields are translated correctly. However, if you have put binary data into char fileds, these binary data will not be translated correctly.
  • Are there any physical files that contains program described data?
    If there are, then you must consider to extract the data and transfer it using another method.

Source for utility

Download zip with source from here: BackUpSrcDta.zip

Source list 1: BackUpSrcDta.CMD:

@Echo Off
 REM
 REM BackUpSource
 REM ------------
 REM Jesper Wachs, version 1, January 2020.
 REM
 REM Utility to extract source and data files from IBM i.
 REM
 REM Leave :SetupConfig here.
 Call :SetupConfig
 REM
 REM Here you must put all the libraries that you want to extract.
 REM
 Call :GetSourceAndData "QGPL"
 Call :GetSourceAndData "MYLIB"
 REM
 REM Perform cleanup and exit.
 REM
 Goto :Exit
 REM
 REM SetupConfig
 REM -----------
 REM Setup configuration values.
 REM
 :SetupConfig
 set scriptfile="%~dp0%~n0.script"
 set logfile="%~dp0%~n0-1.log"
 set logfile2="%~dp0%~n0-2.log"
 Echo.
 set /p ibminame= Enter name or IP of IBM i:
 Echo.
 set /p user= Enter id for iSeries userprofile:
 Echo.
 set /p pwd= Enter password for iSeries userprofile:
 Echo.
 if exist %scriptfile%      del %scriptfile%
 if exist %logfile%         del %logfile%
 if exist %logfile2%        del %logfile2%
 if exist "%~dp0%~n0-3.log" del "%~dp0%~n0-3.log"
 if exist "%~dp0%~n0-4.log" del "%~dp0%~n0-4.log"
 if exist "%~dp0%~n0-5.log" del "%~dp0%~n0-5.log"
 Exit /b
 REM
 REM GetSourceAndData
 REM ----------------
 REM Subroutine that will initiate the retrieval of Source code and Data files.
 REM
 :GetSourceAndData Library
 REM
 REM Check that the library exists.
 REM
 REM
 REM Connect and test if target is an IBM i.
 REM
 Echo.
 Echo Verifying system, user id and password.
 Echo.
 Echo Please wait….
 Echo.
 Echo open %ibminame% > %scriptfile%
 Echo %user% >> %scriptfile%
 Echo %pwd% >> %scriptfile%
 Echo quote system >> %scriptfile%
 Echo Quit >> %scriptfile%
 REM
 REM Execute FTP
 REM
 FTP -s:%scriptfile% > %logfile% 2>&1
 REM
 REM Check that we are connected to an IBMi.
 REM
 FIND /V "530 " < %logfile% > %logfile2%
 If errorlevel == 1 (
 Echo.
 Echo ====================================================================
 Echo Error occured!
 Echo --------------
 Echo Either the user is/password is not correct or the system is not an
 Echo IBM i.
 Echo ====================================================================
 Echo.
 Echo Check the log below:
 Echo ------------------------[ Start of log ]----------------------------
 Type %logfile%
 Echo -------------------------[ End of log ]-----------------------------
 Pause
 Goto Exit
 )
 if not exist "%~1" md "%~1"
 set lib=%~1
 Echo open %ibminame% > %scriptfile%
 Echo %user% >> %scriptfile%
 Echo %pwd% >> %scriptfile%
 Echo quote rcmd chkobj %lib% *lib >> %scriptfile%
 Echo Quit >> %scriptfile%
 REM
 REM Execute FTP
 REM
 FTP -s:%scriptfile% > %logfile% 2>&1
 REM
 REM Check that we are connected to an IBM i.
 REM
 FIND "550-" < %logfile% > %logfile2%
 If errorlevel == 1 Goto C250
 Goto Cont1
 :C250
 REM
 REM In this case, the system must have returned 250 which means ok.
 FIND "250" < %logfile% > %logfile2%
 if errorlevel == 1 (
 Echo.
 Echo ====================================================================
 Echo Error occured!
 Echo --------------
 Echo Test for lib %lib% did not execute correctly.
 Echo ====================================================================
 Echo.
 Echo Check the log below:
 Echo ------------------------[ Start of log ]----------------------------
 Type %logfile%
 Echo -------------------------[ End of log ]-----------------------------
 Pause
 Goto Exit
 )
 :Cont1
 Echo open %ibminame% > %scriptfile%
 Echo %user% >> %scriptfile%
 Echo %pwd% >> %scriptfile%
 Echo quote rcmd dspfd %lib%/all type(basatr) output(*outfile) outfile(qtemp/dspfd) >> %scriptfile%
 Echo get /QSYS.LIB/QTEMP.LIB/DSPFD.FILE .\%lib%.DspFD >> %scriptfile%
 Echo quote rcmd dltf qtemp/dspfd >> %scriptfile%
 Echo Quit >> %scriptfile%
 REM
 REM Execute FTP
 REM
 FTP -s:%scriptfile% > %logfile% 2>&1
 REM
 REM Check that we are connected to an iSeries.
 REM
 FIND "550-" < %logfile% > %logfile2%
 If errorlevel == 1 Goto C2501
 Goto Cont2
 :C2501
 REM
 REM In this case, the system must have returned 250 which means ok.
 FIND "250" < %logfile% > %logfile2%
 if errorlevel == 1 (
 Echo.
 Echo ====================================================================
 Echo Error occured!
 Echo --------------
 Echo Performing the command DSPFD on the library %lib% did not execute
 Echo correctly.
 Echo ====================================================================
 Echo.
 Echo Check the log below:
 Echo ------------------------[ Start of log ]----------------------------
 Type %logfile%
 Echo -------------------------[ End of log ]-----------------------------
 Pause
 Goto Exit
 )
 :Cont2
 REM
 REM Get and filter the list of files.
 REM
 CScript.exe "%~dp0BackUpSrcDta.vbs" "%user%" "%pwd%" "%ibminame%" "%lib%" "%~dp0%lib%.DspFD"
 Exit /b
 REM
 REM End of job
 REM ----------
 REM
 :Exit
 if exist %scriptfile%  del %scriptfile%
 set scriptfile=
 set logfile=
 set logfile2=
 set user=
 set pwd=
 set ibminame=

Listing 2: BackUpSrcDta.vbs:

Rem
 Rem BackUpSource
 Rem ------------
 Rem Jesper Wachs, version 1, January 2020.
 Rem
 Rem Global variables.
 Option Explicit
 Dim f, fso, arg, library, user, pwd, ibmi, fileName, fileType, dataType, oneLine
 Rem
 Rem GetDataFile
 Rem -----------
 Rem Gets a data file and store it as a CSV file.
 Rem
 Rem Be aware that if your datafiles have more than one member, this method
 Rem must be changed to get all the members. This code below will only read
 Rem the first member.
 Rem
 Rem Also, check files transferred to the PC. If Alphanumeric fields contains
 Rem binary data, you will need another method to retrieve the data, as this
 Rem method will translate data to ASCII and bonary data will get corrupted.
 Rem
 Sub GetDataFile(user, pwd, host, lib, file)
    Dim outFile, objFile, objShell
    outFile = ".\" + lib + "\" + file + ".script"
    WScript.Echo "Getting data file ..: " + lib + "/" + file
    Set objFile = fso.CreateTextFile(outFile, True)
    objFile.WriteLine "open " + host
    objFile.WriteLine user
    objFile.WriteLine pwd
    objFile.WriteLine "quote rcmd cpytoimpf " + lib + "/" + file + " TOSTMF('/home/tempfile.pf') rcddlm(*CRLF)"
    objFile.WriteLine "ascii"
    objFile.WriteLine "get /home/tempfile.pf .\" + lib + "\" + file + ".PF"
    objFile.WriteLine "del /home/tempfile.pf"
    objFile.WriteLine "quit"
    objFile.Close
    Set objShell = WScript.CreateObject ("WScript.shell")
    objShell.run "cmd /c ftp -s:" + outFile + " >> BackUpSrcDta-3.log 2>&1", , True
    objShell.run "cmd /c del " + outFile, , True
    Set objShell = Nothing
 End Sub
 Rem
 Rem GetSourceFile
 Rem -------------
 Rem Gets a source file.
 Rem
 Sub GetSourceFile(user, pwd, host, lib, file)
    Dim outFile, objFile, objShell
    outFile = ".\" + lib + "\" + file + ".script"
    WScript.Echo "Getting source file : " + lib + "/" + file
    Set objFile = fso.CreateTextFile(outFile, True)
    objFile.WriteLine "open " + host
    objFile.WriteLine user
    objFile.WriteLine pwd
    objFile.WriteLine "quote rcmd dspfd " + lib + "/" + file + " TYPE(MBRLIST) OUTPUT(OUTFILE) OUTFILE(QTEMP/DSPFDMBR)"
    objFile.WriteLine "ascii"
    objFile.WriteLine "get /qsys.lib/qtemp.lib/dspfdmbr.file .\" + lib + "\" + file + ".MBRLIST"
    objFile.WriteLine "quote rcmd dltf qtemp/dspfdmbr"
    objFile.WriteLine "quit"
    objFile.Close
    Set objShell = WScript.CreateObject ("WScript.shell")
    objShell.run "cmd /c ftp -s:" + outFile + " >> BackUpSrcDta-4.log 2>&1", , True
    objShell.run "cmd /c del " + outFile, , True
    Set objShell = Nothing
    GetSourceMembers user, pwd, host, lib, file, ".\" + lib + "\" + file + ".MBRLIST"
 End Sub
 Rem
 Rem GetSourceMembers
 Rem ----------------
 Rem Gets source members and store them as a folder with text files, one of each member
 Rem
 Sub GetSourceMembers(user, pwd, host, lib, srcfile, file)
    Dim f, oneLine, filename, library, member, mbrtype, objFile, outFile, objShell
 Set objShell = WScript.CreateObject ("WScript.shell")
 objShell.run "cmd /c md .\" + lib + "\" + srcfile, , True
 outFile = ".\" + lib + "\" + srcfile + ".script"   
    Set objFile = fso.CreateTextFile(outFile, True)
    objFile.WriteLine "open " + host
    objFile.WriteLine user
    objFile.WriteLine pwd
    objFile.WriteLine "ascii"
 Set f = fso.OpenTextFile(file)
 Do Until f.AtEndOfStream
       oneLine = f.ReadLine
   filename = trim(Mid(oneLine, 14, 10))
   library = trim(Mid(oneLine, 24, 10))
   member = trim(Mid(oneLine, 65, 10))
   mbrtype = trim(Mid(oneLine, 167, 10))
   If member > "" Then
      objFile.WriteLine "get /qsys.lib/" + library + ".lib/" + filename + ".file/" + member + ".mbr .\" + lib + "\" + srcfile + "\" + member + "." + mbrtype
   End If
 Loop
    objFile.WriteLine "quit"
    objFile.Close
 objShell.run "cmd /c ftp -s:" + outFile + " >> BackUpSrcDta-5.log 2>&1", , True
    objShell.run "cmd /c del " + outFile, , True
    Set objShell = Nothing
 f.Close
 End Sub
 Rem
 Rem Main line
 Rem ---------
 Rem
 Set arg = WScript.Arguments
 user = arg(0)
 pwd = arg(1)
 ibmi = arg(2)
 library = arg(3)
 filename = arg(4)
 Set fso = CreateObject("Scripting.FileSystemObject")
 Set f = fso.OpenTextFile(filename)
 Do Until f.AtEndOfStream
    oneLine = f.ReadLine
 fileName = Trim(Mid(oneLine, 14, 10))
    fileType = Mid(oneLine, 34, 1)
    dataType = Mid(oneLine, 62, 1)
    Rem
    Rem We want to read file that are Physical Data and Source files
    Rem
    If fileType = "P" Then
       If dataType = "D" Then
          GetDataFile user, pwd, ibmi, library, fileName
       ElseIf dataType = "S" Then
          GetSourceFile user, pwd, ibmi, library, fileName
       End If
    End If   
 Loop
 f.Close