String Testing and Translation

This week we will be covering scanning, testing, and some functions for manipulating textual string data.

Introduction

In certain cases, it is necessary to manually scan and test string content. One could manually step through a string character by character and perform specific tests. This would be slow and cumbersome. Fortunately, the Legato SDK contains hundreds of functions to process and manipulate strings. In this article, we will be discussing parsing, basic boolean test functions, word analysis functions, and some conversion functions.

Boolean and Bitwise Results

Functions that test or get data frequent either return a boolean value or a bitwise result. While they are both fundamentally numbers, the boolean data type is used to indicate that if the value is 0, it is FALSE. A non-zero (usually 1) is TRUE. Bitwise return values are typically defined as a dword, a 32-bit unsigned integer. Conventionally the top bit (0x80000000) is not used as part of the result since it normally indicates the value is a formatted error code.

With boolean return values, it is easy to perform operations such as:

if (HasText(mystring)) { ... }

Bitwise return values can contain a lot of information and are normally tested with bit and ordinal masks. This will be covered in detail later. Since working with values like 0x0001000 can be clumsy, the SDK contains definitions for all common bitwise return functions.

A sample use of bitwise return values might look like this:

result = AnalyzeText(mystring);

if ((result & TEXT_TYPE_MASK) == TEXT_TYPE_HEADING) { ... }

A Short Review of the Word Parse Object

Generally speaking, we have to have data to test. The easiest way to look through a string of text is by using the Word Parse Object. The object supports a series of functions that are specifically tailored to “parse” or process textual data. The general-purpose parse runs in three modes: text, tags, and program. The text mode provides word parsing for reading general text. The tags mode is tailored to work on XML, HTML, or SGML tags and character entities. Finally, program mode is made to parse typical program or script text.

We will be focusing on the default parsing mode or text mode (WP_GENERAL), which stops on word spaces, returns (line endings), and punctuation within the textual information.

Basic Operation

The general steps are as follows: create (get handle), load/set data, and iterate until the data has been exhausted. New data can be repeatedly loaded to the same object to process multiple buffers or lines. After completion, the Word Parse Object handle should be closed.

As each item is parsed, the leading spaces and statistics are stored. For example, the caller can check to see if there are leading spaces and even get the white space character as a string. 

Once the source data is set, the source variable can be changed or released. The Word Parse Object makes an internal copy of the data.

Setting Up a Parse Operation

The first action is to create a Word Parse Object and retrieve a handle. That handle is then used in subsequent operations to move through the text and examine each parsed item. For example:

        handle          hWP;
        string          s1, s2;
        int             spaces, count, pos;
        
        s1 = "My favorite pastime is  waiting for my browser to load a page.\rEnd.";

        hWP = WordParseCreate();
        if (hWP == NULL_HANDLE) {
          MessageBox('x', "Error on handle");
          exit;
          }
        WordParseSetData(hWP, s1);
        s2 = WordParseGetWord(hWP);
        while (s2 != "") {
          count++;
          pos = WordParseGetPosition(hWP);
          spaces = WordParseGetSpaceSize(hWP);
          AddMessage("   %3d %3d %3d :%s:", count, pos, spaces, s2);
          s2 = WordParseGetWord(hWP);
          }

        CloseHandle(hWP);

The result in the log:

     1   2   0 :My:
     2  11   1 :favorite:
     3  19   1 :pastime:
     4  22   1 :is:
     5  31   2 :waiting:
     6  35   1 :for:
     7  38   1 :my:
     8  46   1 :browser:
     9  49   1 :to:
    10  54   1 :load:
    11  56   1 :a:
    12  62   1 :page.:
    13  67   1 :End.:

In this case, the parse object is created with the default mode (text). A string is added to the parse object and then each successive word is retrieved along with certain attributes. When added to the log, we surround the returned string value with “::” to illustrate that the string does not contain white space. Note as shown in the log, the first entry has no leading spaces. There is an additional space before “waiting” and a return before “End”.

Functions are provided to retrieve and change the parsing position. In addition, a parse object can be used repeatedly, provided the parse mode remains the same.

Skipping Through a String

Another option while parsing is to skip through a string looking for text or spaces. A series of functions are provided that take a string and a zero-based index as a parameter and then return a new index position. 

Once positions have been established, they can be used with the word parser or functions such as the GetStringSegment function. 

Testing a Word

To avoid having to test each character in a word to determine its type, a large set of SDK functions are provided to test a string, or in some cases a character, as meeting a particular criterion. The following is a partial list of the ‘Is’ or ‘Has’ functions that return TRUE (1) or FALSE (0) depending on the result.

Some of the above functions will also accept and test an individual character. Functions that test only characters are also available:

Another powerful tool is the GetWordType function. The GetWordType function analyzes the content of a provided word and returns the type and attributes. The prototype:

dword = GetWordType ( string data );

The data parameter is a string containing a word without leading or trailing spaces. The returned value is a 32-bit dword (or int, unsigned) containing bitwise information. The results can be any of the following:

Depending on your programming background, bitwise operation may be a bit foreign. They are widely used under the hood in many environments and can be very efficient at conveying a lot of information in a small form factor. Generally, the binary information is segmented into flags and ordinals. Flags are simple. If a bit is set, then the condition is true. Ordinals, on the other hand, require a mask to filter the group associated bits. Those bits in turn represent one of a set of conditions. For example, the resulting dword can be filtered by ‘ANDing’ the result with the WT_TYPE_ITEM_MASK value:

        code = GetWordType(word);
        switch (code & WT_TYPE_ITEM_MASK) {
          case WT_TYPE_UNKNOWN:
            break;
          case WT_TYPE_WORD:
            break;
          case WT_TYPE_NUMBER:
            break;
          case WT_TYPE_NUMBER_SERIAL:
            break;
          case WT_TYPE_LEADER:
            break;
          case WT_TYPE_RULER:
            break;
          case WT_TYPE_CURRENCY_LEADER:
            break;
          case WT_TYPE_NIL:
            break;
          case WT_TYPE_DATE:
            break;
          }

Each case section can then count or act upon the details of the item. For example, if the type is date, then the WT_DATE_ items can be tested to narrow the type of date.

The GetWordType function is useful for aggregating information from a text stream to perform high level analysis. For example, a line of text can be parsed, information accumulated, and the first and last word data examined to determine the probability of line being a heading, part of a paragraph, or perhaps a row of a table.

Analysis is performed on a gross level basis. That is, types of characters are counted and then run through logic to perform a basic analysis. For example, if one or two dashes are present without text, the content will be considered a “nil” value as would be seen in a table. On other hand, three dashes would be considered as a possible rule or visual aid.

In addition, there are the related functions GetListType and GetNumericType, which are similar in operation to GetWordType but return data specific to values as list and numbers, respectively.

The words to test should be passed to the function without spaces. If the Word Parse Object is employed with WP_GENERAL mode, the data returned is compatible with analysis.

Converting Common String Forms

The Legato SDK also contains a number of functions for performing common string conversions and operations:

Changing case is a common operation, which can be performed using the MakeLowerCase and MakeUpperCase functions. The ChangeCase function is substantially more sophisticated allows for the processing of sentences of data in a number of modes, such as title capitalization.

Expanding Our Example

Let us add a few things to the above example:

        handle          hWP;
        string          s1, s2, s3;
        dword           type;
        int             spaces, count, pos;
        
        s1  = "On July 27, 2016, the Company: (i) purchased a dog; (ii) found a vet; ";
        s1 += "(iii) purchased a dog bed; and, (iv) spent $110 on a doggy ID chip. ";

        hWP = WordParseCreate();
        if (hWP == NULL_HANDLE) {
          MessageBox('x', "Error on handle");
          exit;
          }
        WordParseSetData(hWP, s1);
        s2 = WordParseGetWord(hWP);
        while (s2 != "") {
          count++;
          pos = WordParseGetPosition(hWP);
          spaces = WordParseGetSpaceSize(hWP);
          type = GetWordType(s2);
          s2 = ":" + s2 + ":";
          s2 = PadString(s2, 12);
          switch (type & WT_TYPE_ITEM_MASK) { 
            case WT_TYPE_UNKNOWN: s3 = "Unknown"; break; 
            case WT_TYPE_WORD: s3 = "Word"; break; 
            case WT_TYPE_NUMBER: s3 = "Number"; break; 
            case WT_TYPE_NUMBER_SERIAL: s3 = "Number (Serial)"; break; 
            case WT_TYPE_LEADER: s3 = "Leader"; break; 
            case WT_TYPE_RULER: s3 = "Ruler"; break; 
            case WT_TYPE_CURRENCY_LEADER: s3 = "Currency"; break; 
            case WT_TYPE_NIL: s3 = "Nil"; break; 
            case WT_TYPE_DATE: s3 = "Date"; break; 
            default: s3 = "";
            }
          AddMessage("   %3d %3d %3d 0x%08X %s %s", count, pos, spaces, type, s2, s3);
          s2 = WordParseGetWord(hWP);
          }

        CloseHandle(hWP);

The result in the log would appear as:

     1   2   0 0x00310000 :On:         Word
     2   7   1 0x10310000 :July:       Word
     3  11   1 0x00230002 :27,:        Number (serial)
     4  17   1 0x00030005 :2016,:      Number (serial)
     5  21   1 0x00110000 :the:        Word
     6  30   1 0x00B1003A :Company::   Word
     7  34   1 0x02110000 :(i):        Word
     8  44   1 0x00110000 :purchased:  Word
     9  46   1 0x00110000 :a:          Word
    10  51   1 0x0011003B :dog;:       Word
    11  56   1 0x02110000 :(ii):       Word
    12  62   1 0x00110000 :found:      Word
    13  64   1 0x00110000 :a:          Word
    14  69   1 0x0011003B :vet;:       Word
    15  75   1 0x02110000 :(iii):      Word
    16  85   1 0x00110000 :purchased:  Word
    17  87   1 0x00110000 :a:          Word
    18  91   1 0x00110000 :dog:        Word
    19  96   1 0x0011003B :bed;:       Word
    20 101   1 0x0011002C :and,:       Word
    21 106   1 0x02110000 :(iv):       Word
    22 112   1 0x00110000 :spent:      Word
    23 117   1 0x08020004 :$110:       Number
    24 120   1 0x00110000 :on:         Word
    25 122   1 0x00110000 :a:          Word
    26 128   1 0x00110000 :doggy:      Word
    27 131   1 0x00210000 :ID:         Word
    28 137   1 0x0091002E :chip.:      Word

We are using the PadString function to make a fixed size field in the log for the word, and we are still maintaining the ‘::’ convention that contains the word. The return value from the GetWordType function is both translated to a friendly string and printed in hexadecimal form in the log. Note that the words “27,” and “2016,” are considered serial numbers, as in a list that could appear within narrative as opposed to a table cell.

Conclusion

Since Legato is a part of GoFiler and GoFiler specializes in converting and editing text, many of the foundational string functions are exposed as script functions. If you cannot find a function to match your particular operation, contact technical support as it may already exist.

Scott Theis is the President of Novaworks and has been involved in the EDGAR industry for over thirty years. He has worked with the EDGAR system at multiple levels: as a financial printer, a member of the EDGAR design team, and as a software developer. He has extensive expertise with EDGAR, HTML, XBRL, and other programming languages.

Additional Resources

EDGAR 2016 Filing Peak Schedule (www.sec.gov)