This example shows various ways to handle invalid encodings in text files.
This sample script copies its input file to the output file while substituting the bytes that are invalid in the given encoding with the replacement character (U+FFFD). Additionally, records containing invalid bytes will be written to an error file, optionally prepended with their offsets in the original file.
The following script parameters can be used to modify the script's behavior:
-
@inputFile- File to read and validate. If the input file name isEncodingErrorSample.txt:- The output file will be
EncodingErrorSample.filtered.txt. - The errors file will be
EncodingErrorSample.errors.txt.
- The output file will be
-
@codePage- Code page to validate the file with. The following code pages are supported:Constant Name CP Number @codePageUtf865001 @codePageUtf161200 @codePageUtf16be1201 @codePageUtf3212000 @codePageUtf32be12001 -
@rowDelimiter- This string is used to break the input file into records. Row delimiters must occur in the input file at least once every 4MB. The output file will have row delimiter after the last record even if the input file did not have it. -
@removeInvalidInputRecordsFromOutputfile- Iftrue, the records containing invalid bytes will not be copied to the output file. -
@replaceInvalidBytesInErrorsFile- Iftrue, invalid bytes in the records written to the error file will be substituted with the replacement charcter (U+FFFD). This is useful if you want to analyze the errors with a tool (e.g., editor) that complains about invalid encoding. -
@offsetFormatInErrorsFile- This specifies a C# composite format string that is used to output offsets of invalid records to the error file. The string should contain a format item with index0, e.g."{0}:\t". Ifnull, the offsets will not be written.