Building on what we did last week, this week we’ll add more information to our EDGAR Filing log. Last week, we examined a script that automatically ran after a user in GoFiler uses the File Live, File Test or File Test As Agent menu function. This function created a filing log by adding a new row into a simple CSV file to record information. Now we’ll add more information to the filing log from two sources: an SDK function to access data after GoFiler files the document and the Accession Notice file in the Mailbox directory.
Friday, October 07. 2016
Legato Developers Corner #4: Recording Filing Changes
SDK functions can access information about the filing once the document is filled, such as its accession number. The Accession Notice file holds data about previous filings, including the filing status and the number of documents in the filing. Before proceeding, I suggest you delete your old mailbox log file that was created by the previous version of our script created.
Again, this script is a .ms file so it runs at startup. It has the same three user functions as well: setup, main, and run. Two new functions are added to the script. In addition, we will modify the run function to add more field headings to our CSV table as well as give it the capacity to check previous filings for missing information. The following outlines what we will explore this week:
- Parsing a string response
- Checking for accession notice files
- Using a for loop to go over all the entries in the log and check their status
Our Sample Script
// GoFiler Legato Script - Record Filings
// ------------------------------------------
//
// Rev 10/07/2016
//
// (c) 2016 Novaworks, LLC -- All rights reserved.
//
//
// Notes:
// - None.
int setup ();
int run (int f_id, string mode);
string parse_response (string response);
string check_status (string accession);
// set up the script and add it to the menu bar. runs on startup automatically.
int setup() {
string fnScript;
int rc;
// get the name of this script file
fnScript = GetScriptFilename();
// hook the "run" function of this script to execute when the new menu
// option is activated.
MenuSetHook("EDGAR_SUBMIT_LIVE", fnScript, "run");
MenuSetHook("EDGAR_SUBMIT_TEST", fnScript, "run");
MenuSetHook("EDGAR_SUBMIT_TEST_AGENT", fnScript, "run");
return ERROR_NONE;
}
// runs every time the menu button is clicked on
int run(int f_id, string mode){
// test/live code.
string f_code;
// the actual data file.
string data_file;
// where the data file is stored on my PC.
string data_file_path;
// contents of a data file;
string data_file_contents[][];
// index to start writing data;
int index;
// array of response information from filing
string response_array[];
// string response from the filing
string response;
// counter variable
int ix;
// an accession number of a filing to check status of
string accession;
// the response from the check status function
string check_status_response;
// the response from the check status function as an array
string check_status_response_array[];
// only execute on post-process of function.
if(mode!="postprocess"){
return ERROR_NONE;
}
// get the response code from the File function ran, and convert it to our CSV format
response = parse_response(GetMenuFunctionResponse());
// convert the parsed response string to an array so we can set it into our table
response_array = ExplodeString(response,",");
switch(f_id){
case MenuFindFunctionID("EDGAR_SUBMIT_LIVE"):
f_code = "LIVE";
break;
case MenuFindFunctionID("EDGAR_SUBMIT_TEST"):
f_code = "TEST";
break;
case MenuFindFunctionID("EDGAR_SUBMIT_TEST_AGENT"):
f_code = "TEST";
break;
}
// get the location of the data file
data_file_path = GetSetting("settings","filing_history");
// if we don't have a data file, ask where we should store it.
if(IsFile(data_file_path) == false){
data_file_path = BrowseSaveFile("Select Save File","CSV Files|*.csv","",0,"CSV Files|*.csv");
// if the user cancelled, stop here.
if (GetLastError()==ERROR_CANCEL){
return ERROR_CANCEL;
}
// save name of storage file
PutSetting("settings","filing_history",data_file_path);
// set headings for CSV file
data_file_contents[0][0]="Name";
data_file_contents[0][1]="Live/Test";
data_file_contents[0][2]="Date";
// new fields filled in by response code
data_file_contents[0][3]="Form Type";
data_file_contents[0][4]="Primary CIK";
data_file_contents[0][5]="Agent CIK";
data_file_contents[0][6]="File Name";
data_file_contents[0][7]="Accession Number";
// fields filled in by check filing status function
data_file_contents[0][8]="Status";
data_file_contents[0][9]="Documents";
}
else{
// get data from existing file
data_file_contents = CSVReadTable(data_file_path);
}
// get index of where to put our information.
index = ArrayGetAxisDepth(data_file_contents);
// set the new data into the index
data_file_contents[index][0] = GetUserName();
data_file_contents[index][1] = f_code;
data_file_contents[index][2] = GetLocalTime(DS_DATE_AT_TIME);
// new fields filled in by response code
data_file_contents[index][3] = response_array[0];
data_file_contents[index][4] = response_array[1];
data_file_contents[index][5] = response_array[2];
data_file_contents[index][6] = response_array[3];
data_file_contents[index][7] = response_array[4];
// check all previous filings to add in missing status fields
for(ix = 1; ix<index; ix++){
// if the filing doesn't have a status
if (data_file_contents[ix][8]==""){
// get the accession number of the file to look up
accession = data_file_contents[ix][7];
// look up the status of this filing, and put it into our table
check_status_response = check_status(accession);
// make sure our status check returned information before trying to use it
if (check_status_response!=""){
check_status_response_array = ExplodeString(check_status_response,",");
data_file_contents[ix][8]=check_status_response_array[0];
data_file_contents[ix][9]=check_status_response_array[1];
}
}
}
// write output and quit
CSVWriteTable(data_file_contents,data_file_path);
return ERROR_NONE;
}
// take the response from file, and convert it into a CSV format
string parse_response(string response){
// the response we will return
string parsed_response;
parsed_response+= GetParameter(response,"FormType");
parsed_response+=",";
parsed_response+= GetParameter(response,"PrimaryCIK");
parsed_response+=",";
parsed_response+= GetParameter(response,"AgentCIK");
parsed_response+=",";
parsed_response+= GetParameter(response,"File");
parsed_response+=",";
parsed_response+= GetParameter(response,"AccessionNumber");
return parsed_response;
}
// checks the status of all previous filings to see if they can be updated.
string check_status(string accession){
// the mailbox folder
string mailbox_folder;
// GoFiler's settings file
string settings_file;
// the name of the accession notice file to check
string accession_notice;
// the contents of the accession notice as a string
string accession_file_text;
// the information from the accession notice
string accession_info[];
// the acceptance code
string acceptance_code;
// /the acceptance code as a text string
string acceptance_string;
// the string we're actually returning
string status_string;
status_string = "";
settings_file = GetApplicationDataLocalFolder()+GetApplicationName()+ "Settings.ini";
mailbox_folder = GetSetting(settings_file,"EDS","Mail");
accession_notice = AddPaths(mailbox_folder,accession+".txt");
// check to make sure this accession notice actually exists in our mailbox
if (IsFile(accession_notice)){
// read the file to a string
accession_file_text = FileToString(accession_notice);
accession_info = EDGARResponseProps(accession_file_text);
status_string ="";
// convert the string from a response code to an actual text string for storage
acceptance_code = accession_info["ResultCode"];
switch(TextToInteger(acceptance_code)){
case (EM_RESULT_TEST_FAIL):
acceptance_string = "Test Filing Failed";
break;
case (EM_RESULT_TEST_PASS):
acceptance_string = "Test Filing Passed";
break;
case (EM_RESULT_TEST_PASS_XBRL_FAIL):
acceptance_string = "Test Filing Passed/XBRL Fail";
break;
case (EM_RESULT_LIVE_FAIL):
acceptance_string = "Live Filing Failed";
break;
case (EM_RESULT_LIVE_PASS):
acceptance_string = "Live Filing Passed";
break;
case (EM_RESULT_LIVE_PASS_XBRL_FAIL):
acceptance_string = "Live Filing Passed/XBRL Fail";
break;
}
status_string+= acceptance_string;
status_string+= ",";
status_string+= accession_info["Documents"];
return status_string;
}
// if the accession doesn't exist, return an empty string; there's no info to check
else{
return status_string;
}
}
// every script requires a main function when it's run from the IDE
int main() {
// make sure the function is set up.
setup();
return ERROR_NONE;
}
Script Walkthrough
We begin with our setup function, which hooks our function to the listed menu functions. As we did last week, we will hook to the EDGAR_SUBMIT_LIVE, EDGAR_SUBMIT_TEST, and EDGAR_SUBMIT_TEST_AGENT menu function codes. Again, every menu function in GoFiler has a unique code, and you can download an Excel file (.xls) that contains a list of every GoFiler menu function and its code by clicking here or you can contact us at legato@novaworkssoftware.com if you have questions.
Now in the run user function we can specify what we want to happen in this script. Again, we want to use the mode parameter to limited our script to executing on the “postprocess” stage. At this point, we process the response from the filing that triggered the script. This is a combination of two functions: the parse_response user function and the SDK function GetMenuFunctionResponse.
The parse_response function takes a string as its parameter, and this string must be formatted as returned by the GetMenuFunctionResponse SDK function. Using this format, the GetParameter SDK function can return specific information that we request from the string. The parse_response function then rebuilds this data into a comma-delimited list format and returns that string to the calling run function. This prepares our filing information for the CSV table. Note the use of the += operator to concatenate the string together piece by piece.
// take the response from file, and convert it into a CSV format
string parse_response(string response){
// the response we will return
string parsed_response;
parsed_response+= GetParameter(response,"FormType");
parsed_response+=",";
parsed_response+= GetParameter(response,"PrimaryCIK");
parsed_response+=",";
parsed_response+= GetParameter(response,"AgentCIK");
parsed_response+=",";
parsed_response+= GetParameter(response,"File");
parsed_response+=",";
parsed_response+= GetParameter(response,"AccessionNumber");
return parsed_response;
}
Now let’s look onward in the run function. The data returned by the parse_response function is exploded by its delimiter (the comma). At this point we use a switch statement on the f_id variable again to determine which menu action resulted in the script executing so we can mark this as a test or live filing.
As we did last week, we prepare our data file. Once the data file is ready, the script reads the contents of the file using the CSVReadTable SDK function to find the appropriate place to add our new information. Again, these functions and their uses were covered in last week’s blog. Now that we have the position within the file via the ArrayGetAxisDepth SDK function (which should be at the end, given this is the most recent filing), we can add our new information to the file. We do this by adding the new parsed response data after the user name, f_code, and time indices that we explored in the previous blog.
// get index of where to put our information.
index = ArrayGetAxisDepth(data_file_contents);
// set the new data into the index
data_file_contents[index][0] = GetUserName();
data_file_contents[index][1] = f_code;
data_file_contents[index][2] = GetLocalTime(DS_DATE_AT_TIME);
// new fields filled in by response code
data_file_contents[index][3] = response_array[0];
data_file_contents[index][4] = response_array[1];
data_file_contents[index][5] = response_array[2];
data_file_contents[index][6] = response_array[3];
data_file_contents[index][7] = response_array[4];
At this point we have our recent filing’s data in the CSV array. What we’d like to do as well is fill out the information that’s missing in other filings. This is where the for loop comes into play. We will also make use of an additional user function: the check_status function.
// check all previous filings to add in missing status fields
for(ix = 1; ix<index; ix++){
// if the filing doesn't have a status
if (data_file_contents[ix][8]==""){
// get the accession number of the file to look up
accession = data_file_contents[ix][7];
// look up the status of this filing, and put it into our table
check_status_response = check_status(accession);
// make sure our status check returned information before trying to use it
if (check_status_response!=""){
check_status_response_array = ExplodeString(check_status_response,",");
data_file_contents[ix][8]=check_status_response_array[0];
data_file_contents[ix][9]=check_status_response_array[1];
}
}
}
Our for loop iterates through the array of records. If that record does not have a status, we store the accession number and pass it to our check_status function.
// checks the status of all previous filings to see if they can be updated.
string check_status(string accession){
// the mailbox folder
string mailbox_folder;
// GoFiler's settings file
string settings_file;
// the name of the accession notice file to check
string accession_notice;
// the contents of the accession notice as a string
string accession_file_text;
// the information from the accession notice
string accession_info[];
// the acceptance code
string acceptance_code;
// /the acceptance code as a text string
string acceptance_string;
// the string we're actually returning
string status_string;
status_string = "";
settings_file = GetApplicationDataLocalFolder()+GetApplicationName()+" Settings.ini";
mailbox_folder = GetSetting(settings_file,"EDS","Mail");
accession_notice = AddPaths(mailbox_folder,accession+".txt");
// check to make sure this accession notice actually exists in our mailbox
if (IsFile(accession_notice)){
// read the file to a string
accession_file_text = FileToString(accession_notice);
accession_info = EDGARResponseProps(accession_file_text);
status_string ="";
// convert the string from a response code to an actual text string for storage
acceptance_code = accession_info["ResultCode"];
switch(TextToInteger(acceptance_code)){
case (EM_RESULT_TEST_FAIL):
acceptance_string = "Test Filing Failed";
break;
case (EM_RESULT_TEST_PASS):
acceptance_string = "Test Filing Passed";
break;
case (EM_RESULT_TEST_PASS_XBRL_FAIL):
acceptance_string = "Test Filing Passed/XBRL Fail";
break;
case (EM_RESULT_LIVE_FAIL):
acceptance_string = "Live Filing Failed";
break;
case (EM_RESULT_LIVE_PASS):
acceptance_string = "Live Filing Passed";
break;
case (EM_RESULT_LIVE_PASS_XBRL_FAIL):
acceptance_string = "Live Filing Passed/XBRL Fail";
break;
}
status_string+= acceptance_string;
status_string+= ",";
status_string+= accession_info["Documents"];
return status_string;
}
// if the accession doesn't exist, return an empty string; there's no info to check
else{
return status_string;
}
}
This user function makes use of numerous SDK functions to access the application mailbox for a particular filing notice as denoted by the parameter accession. Once the file is located, the function then accesses it;s data via the SDK function EDGARResponseProps, which returns an array of response properties. Switching through the “ResultCode” property (which must first be converted to integer), we can set a proper string describing the result of the filing. After that, we add the number of documents in the filing by concatenating the data stored at the “Documents” key in the array of response information. The check_status function then returns the resultant string for the filing linked with that accession number.
You should note that the check_status function only checks filings that were done before the one that triggered our script. It doesn’t actually get the status of the filing you just did because there would have to be a delay between when the filing was done and when the mailbox actually gets the accession notice. A possible modification for later would be to spawn a background thread and run the check_status function a minute or two after the filing is done since the Mailbox had been updated. Alternatively, you could add a post process function when the mailbox receives mail to update the status of our log file.
The loop inside the run function continues like this, filling in status information as necessary for all the files up to the most recent filing (denoted by the index variable).
After that, we can use the CSVWriteTable SDK function to write our modified table out to the log file and complete our processing. This script demonstrates how you can not only store information in a CSV log for your most recent filing, but fill in the gaps in previous filings using a few SDK functions and a loop.
 Steven Horowitz has been working for Novaworks for over five years as a technical expert with a focus on EDGAR HTML and XBRL. Since the creation of the Legato language in 2015, Steven has been developing scripts to improve the GoFiler user experience. He is currently working toward a Bachelor of Sciences in Software Engineering at RIT and MCC. |
Additional Resources
Quicksearch
Categories
Calendar
Feeds
