https://www.novaworkssoftware.com/samples/RetrieveLibraryExample.php

How this handler page functions is a bit beyond the scope of this project, but I set it up so if you click on the above link you can see the PHP source code of the file. It’s very basic, it simply gets the $_POST array variable (which is where PHP stores post parameters) and if the ‘type’ value in that array is ‘CIK’, it prints out a CIK library, if it is ‘OWNERS’, it prints out a reporting owners library. This is only a basic example, but ideally you would have some sort of authentication so a user would have to have a password posted, and you would have some way of building the library files out of a database. That is more into the realm of web development than scripting though, so this basic example just illustrates what is possible.

Our script for this week is:

/*
 * Download CIK and Reporting Owner libraries
 */

#define LIBRARY_LOCATION "https:\\\\www.novaworkssoftware.com\\samples\\RetrieveLibraryExample.php"

    int                 download_cik(int f_id, string mode);
    int                 download_rol(int f_id, string mode);
    int                 download_library(int f_id, string mode, string type);
                                                                        
int setup() {                                                           
                                                                        
    string              fnScript;                                       
    string              item[10];                                       
    int                 rc;           
                                                                        
    // build item for download ciks hook
    item["Code"] = "EXTENSION_DOWNLOAD_CIKLIB";                         
    item["MenuText"] = "&Download a CIK Library";                               
    item["Description"] = "<B>Download a CIK Library</B>";      
                                                                        
    // make sure it's not on menu already
    rc = MenuFindFunctionID(item["Code"]);                              
    if (IsNotError(rc)) {                                               
      return ERROR_NONE;                                                
      }                                                                 
                                                                        
    // add it to the menu
    rc = MenuAddFunction(item);                                         
    if (IsError(rc)) {                                                  
      return ERROR_NONE;                                                
      }                                                                 
    // hook it
    fnScript = GetScriptFilename();                                     
    MenuSetHook(item["Code"], fnScript, "download_cik");                                        
    
    // build item for download owners hook
    item["Code"] = "EXTENSION_DOWNLOAD_ROL";                            
    item["MenuText"] = "&Download a Reporting Owners Library";                          
    item["Description"] = "<B>Download a Reporting Owners Library</B>"; 
                                                                        
    // make sure it's not on the menu already
    rc = MenuFindFunctionID(item["Code"]);                              
    if (IsNotError(rc)) {                                               
      return ERROR_NONE;                                                
      }                                                                 
                                                                        
    // add it to the menu
    rc = MenuAddFunction(item);                                         
    if (IsError(rc)) {                                                  
      return ERROR_NONE;                                                
      }                                                                 
    // hook it
    fnScript = GetScriptFilename();                                     
    MenuSetHook(item["Code"], fnScript, "download_rol");            
    return ERROR_NONE;                                                  
    }
    
int download_cik(int f_id, string mode){
    // download library with correct parameter
    download_library(f_id,mode,"CIK");
    return ERROR_NONE;
    }
    
int download_rol(int f_id, string mode){
    // download library with correct parameter
    download_library(f_id,mode,"OWNERS");
    return ERROR_NONE;
    }
    
int download_library(int f_id, string mode, string type){
    
    string              res;
    string              params[];
    string              settings_loc;
    string              lib_folder;
    int                 rc;
    
    if (mode!="preprocess"){
      return ERROR_NONE;
      }
    
    // set the type of library to download
    params["type"] = type;
    
    // request library from server
    res = HTTPPost(LIBRARY_LOCATION, params);
    
    // get library folder location from settings file
    settings_loc = GetApplicationDataLocalFolder();
    settings_loc+=GetApplicationName()+" Settings.ini";
    lib_folder = GetSetting(settings_loc,"EDS","Library");
   
    // check if we're using a test folder, if not make sure we are
    if(IsRegexMatch(lib_folder,".+(_downloaded)")==false){
      lib_folder = lib_folder+"_downloaded";
      }
    
    // ensure folder exists
    if (IsFolder(lib_folder)==false){
      rc = CreateFolder(lib_folder);
      if (IsError(rc)){
        MessageBox('x',"Cannot create test folder");
      return ERROR_EXIT;
      }
    
      // store folder location, and load force preferences to load
      PutSetting(settings_loc,"EDS","Library",lib_folder);
      LoadPreferences(settings_loc);
      }
    
    // clear and reset the appropriate library
    if (type=="CIK"){
      EDGARClearLibrary(EAL_CIK_LIBRARY);
      rc = EDGARImportLibrary(EAL_CIK_LIBRARY,res);
      }
    if (type=="OWNERS"){
      EDGARClearLibrary(EAL_RO_LIBRARY);
      rc = EDGARImportLibrary(EAL_RO_LIBRARY,res);
      }
    
    // display a success / fail message based on the library import
    if (IsError(rc)){
      MessageBox('x',"Failed to import library, code %0x",rc);
      return ERROR_EXIT;
      }
    else{
      MessageBox('i',"Successfully imported library.");
      return ERROR_NONE;
      }
    }

int main(){
    setup();
    return ERROR_NONE;
    }                                                   

In the declarations area at the top of the script, I define a constant variable, LIBRARY_LOCATION. This is just the web address of the sample PHP page I put up that the script is going to talk to. If you want to point it to a different page, you would change this value. This script uses a slight variation on the format of several we’ve written before. Many of them have the standard setup, run, and main functions. Well, this one has two hooks, and we want them to both do something slightly different, so instead of a single function for run, I used two functions called download_cik and download_rol in place of run. They are both called by the hook processor, and then they control what gets passed to the download_library function, which is the one that does all the actual work..

#define LIBRARY_LOCATION "https:\\\\www.novaworkssoftware.com\\samples\\RetrieveLibraryExample.php"

    int                 download_cik(int f_id, string mode);
    int                 download_rol(int f_id, string mode);
    int                 download_library(int f_id, string mode, string type);

The setup function is identical to other setup functions we’ve discussed in the past (except for there being two hooks in it instead of just one, but that doesn’t change how it works at all) so I won’t go into depth covering how those work. First, lets take a look at our actual hooked functions, download_cik and download_rol. They are both very simple. All they do is call download_library, which does all the actual work. This function takes the additional string parameter ‘type’, which lets it know which menu function was pressed.

int download_cik(int f_id, string mode){
    // download library with correct parameter
    download_library(f_id,mode,"CIK");
    return ERROR_NONE;
    }
    
int download_rol(int f_id, string mode){
    // download library with correct parameter
    download_library(f_id,mode,"OWNERS");
    return ERROR_NONE;
    }

Inside the download_library function, after the variable declarations and the if statement to quit if not preprocess, we get into the actual web request. First, we need to build an array of parameters to send to our web page. The string array params works for this. The key value of ‘type’ in the params array is set to be the type of library we are trying to download. Once the parameter array is set up, we can actually use our HTTPPost function to send the request to our web page. The web server receives our POST request, along with the parameters we sent, and then sends back a string variable, which is just the result of the post. Discussing exactly how that PHP script works is beyond the scope of this post, but it’s a very simple one in this case, it’s giving us back a library sample file in CSV format. This is stored in the res variable.

    // set the type of library to download
    params["type"] = type;
    
    // request library from server
    res = HTTPPost(LIBRARY_LOCATION, params);

Now that we have the CSV of a library file, we actually have to do something with it in Legato for it to be of any use. First, we need to get the location of our current library directory, and ensure that we append an “_downloaded” to it. Otherwise, setting our downloaded sample library would delete our current library, and that would probably be a bad thing. So we get the settings file location by building the path to it with the GetApplicationDataLocalFolder and GetApplicationName functions, then use the GetSetting function to pull the library path from the application settings. Using the function IsRegexMatch, we can then check to make sure it ends in “_downloaded”.

    // get library folder location from settings file
    settings_loc = GetApplicationDataLocalFolder();
    settings_loc+=GetApplicationName()+" Settings.ini";
    lib_folder = GetSetting(settings_loc,"EDS","Library");
   
    // check if we're using a test folder, if not make sure we are
    if(IsRegexMatch(lib_folder,".+(_downloaded)")==false){
      lib_folder = lib_folder+"_downloaded";
      }

After we’ve ensured our path ends in “_downloaded”, we need to make sure that folder actually exists. If it doesn’t exist, we need to create it. Whenever creating a folder, be sure to check the result code as well, because if the folder wasn’t created as expected, we want the script to throw an error message, and then exit. Otherwise, it would continue on, and that could lead to undefined behavior. After the folder is created (if it didn’t already exist), we need to store the new setting in the settings file using the PutSetting function, and then force a reload on the preferences so GoFiler knows which library folder to use when reading and writing libraries.

    // ensure folder exists
    if (IsFolder(lib_folder)==false){
      rc = CreateFolder(lib_folder);
      if (IsError(rc)){
        MessageBox('x',"Cannot create test folder");
      return ERROR_EXIT;
      }
    
      // store folder location, and load force preferences to load
      PutSetting(settings_loc,"EDS","Library",lib_folder);
      LoadPreferences(settings_loc);
      }

For the last step, we need to check which type of library we are importing, and then for either one, clear the library with the EDGARClearLibrary function and use the EDGARImportLibrary function to import it into GoFiler. Then, we can check the error code, and display either a success message or an error code, to let the user know the operation was completed.

    // clear and reset the appropriate library
    if (type=="CIK"){
      EDGARClearLibrary(EAL_CIK_LIBRARY);
      rc = EDGARImportLibrary(EAL_CIK_LIBRARY,res);
      }
    if (type=="OWNERS"){
      EDGARClearLibrary(EAL_RO_LIBRARY);
      rc = EDGARImportLibrary(EAL_RO_LIBRARY,res);
      }
    
    // display a success / fail message based on the library import
    if (IsError(rc)){
      MessageBox('x',"Failed to import library, code %0x",rc);
      return ERROR_EXIT;
      }
    else{
      MessageBox('i',"Successfully imported library.");
      return ERROR_NONE;
      }

This is a pretty basic example, but it illustrates that you can pass information to a web page with a parameter array, and then use dynamically generated information from that page in GoFiler. As an example application of a script like this, I have written a set of script functions that can log into our portal system, GoFiler.Online, ask for a library for the user who has logged on, and then the library file is built on the fly for the individual user. This allows us to have a centralized repository of CIK entities that isn’t stored on a local network, but is accessible from any computer with an internet connection. It also allows users to make edits to the library through the portal system itself, instead of having to control the library through GoFiler Complete.

Other information can be pulled from web pages though, the library is just an example. Preferences could be retrieved, templates, SEC filings, pretty much anything could be retrieved by using the HTTPPost or HTTPGet functions, and then used by Legato to help automate parts of your process.

Steven Horowitz has been working for Novaworks for over five years as a technical expert with a focus on EDGAR HTML and XBRL. Since the creation of the Legato language in 2015, Steven has been developing scripts to improve the GoFiler user experience. He is currently working toward a Bachelor of Sciences in Software Engineering at RIT and MCC.

Additional Resources

Novaworks’ Legato Resources

Legato Script Developers LinkedIn Group

Primer: An Introduction to Legato