Appendix - Additional Information

Document Text Extraction

When processing a submitted document Classification Server will first attempt to extract text then use this extracted text in the classification process. Without any meaningful textual content the classification process is irrelevant, for example, there is no way to classify an image file because the only text the file generally contains is meta information encoded within the file structure (Classification Server does not do OCR). With files like this it is perfectly valid to either see an error generated by the Classification Server request or no classifications returned from the process at all. Indeed, it is quite common in a correctly configured production installation to regularly see errors generated for certain documents.

If Classification Server is unable to extract text this could be due to any of the following situations:

1. Classification Server does not handle the specific file format/version that is being used - Semaphore uses a third party library for content extraction that is very good but there are some formats that are simply not handled. The general list of file formats which this third party handles can be found below but note that support for things like specific versions of the file format (that are generated by different versions of applications) may differ. This results in errors such as “Unspecified error reading”, and “Type identified correctly but no available filter to handle content”.
2. Classification Server cannot extract text from secured documents - Often for security and technical reasons Semaphore cannot extract text from documents that have been secured with a password or encrypted in some way. This results in errors such as “Password Protected or encrypted file FILENAME” and “Could not open the FILENAME” file FILENAME reason: This document requires authentication“. - Classification Server cannot parse the file provided - Even if the file can be opened in the application that generated it the third party libraries may not be able to support the specific information included in it. Often this can be because the third party does not handle all variations of the format that are available or that the published specification used does not include it (that is, the application is generating information in a format the third party does not know about). This results in errors such as ”Unknown exception caught!“, ”Can not open the FILETYPE file FILENAME : The file is damaged and could not be repaired“, ”Unspecified error reading“, ”Failed to obtain content handle for FILENAME“, ”Adobe PDF parse Failed on page X“, ”File is invalid xml“, ”XML error” and “Unspecified error reading embedded object”.

Specific File Format Content Extraction Notes

The following are some comments regarding what content is extracted from specific file types:

1. “SWF” is generally a Adobe Flash file which Semaphore can only extract a very small amount of textual information, if any at all. There is generally no textual content that can meaningfully be used for classification.
2. “MP3” is generally an audio format that Semaphore can only extract a small amount of textual information, if any at all, mostly based on the ID3 tags. There is generally no textual content that can meaningfully be used for classification.
3. “MPG” is generally a video format that Semaphore can only extract a small amount of textual information, if any at all, mostly based on file id only. There is generally no textual content that can meaningfully be used for classification.
4. “OVG” is generally an “image overlay” file format that Semaphore does not support. With such files, even if Semaphore did support them, there is generally no textual content it can meaningfully use for classification.
5. “MP4” is generally an audio or video format that Semaphore can only extract a small amount of textual information from, if any at all, mostly based on the metadata. There is generally no textual content that can meaningfully be used for classification.
6. “TIF” is generally an image format that Semaphore can only extract a small amount of textual information, if any at all, mostly based on the metadata. There is generally no textual content that can meaningfully be used for classification.
7. “MSG” is generally an email format that may contain elements Semaphore cannot process such as (some) embedded objects.
8. “CHM” is generally aa “Microsoft Compiled Help” file which is a archive-based format. In archive formats, such as ZIP, Semaphore can only process file names within the file and not the files contained in the archive themselves so classification of such content is generally not advisable as the results are not useful (even if they are generated).
9. “DGN” is a CAD file format which Semaphore does not support. From such formats (much like any image file) there is generally no textual content that can meaningfully be used for classification..
10. “RAR” and “ZIP” is generally an archive-based format. In archive formats Semaphore can only process file names within the file and not the files contained in the archive themselves so classification of such content is generally not advisable as the results are not useful (even if they are generated).
11. “PUB” is generally a Microsoft Publisher file that Semaphore can only extract a small amount of textual information, if any at all, mostly based on file id only. There is generally no textual content that can meaningfully be used for classification.

For those file types Semaphore cannot handle or from which little meaningful data can be extracted it is recommended that you exclude them from classification.

Supported document formats

The following is a sample of the many file formats supported by Classification Server. Contact Progress if a format you require is not listed (as it very well may be supported).

“classify.py/classify.exe” full command reference

The “options” are as follows:

Classification Server request options:

XML File Submission

Rules written for XML files submitted for classification (directly in rules or via the rulebase templates) can specifically target XML elements present in the file (this is not to be confused with a XML request submitted to Classification Server - this is where the request references, or includes, a separate XML file for classification). For example if the following XML file is referenced in a classification request:

<?xml version="1.0" encoding="UTF-8"?>
<book>
  <author>A B Smith</author>
  <bio_info url="http://www.example.com/absmith">
    <born>1/1/1970</born>
    <facebook>absmith</facebook>
  </bio_info>
  <text>Some text here</text>
  <more_text>Some title text</more_text>
</book> 

Classification Server supports the limited use of “XPath”-style references to the fields (see field for further details) present in the XML document so, in this case, we could reference “book/text” in the “field” element of the rulebase to target the “Some text here” text. For example:

<phrase field="book/text" data="Some text here" />

To allow normal rulebases, which almost always use “body” restricted rules, work without modifications the actual XML document elements are placed within the “body” section of the request. So, in our example we could access the same “Some text here” information using “body/book/text”.

For convenient usage of any style of XML document the attribute values are also addressable using the XPath convention of prefixing “@” to refer to an attribute rather than an element name. From the above example:

<text field="bio_info/@url" data="http://www.example.com/absmith" />

For the scoping of the search, attribute values are treated exactly as if they were child elements and so the values of attributes are selected for the scope of a search as well as any child elements. For example:

<phrase field="person" data="Fred Bloggs" />

will find a match in

<person name="Fred Bloggs" /> 

or

<person>
  <name>Fred Bloggs</name>
</person>

Though in the first case the field the name is found in is actually “@name” rather than “name” which allows you to distinguish between the two cases if required.