Friday, July 9, 2021

HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: MalformedInputException: Input length = 1

Working on Polybase implementation where data stored as parquet file has to be imported in Synapse (SQL DWH). 

So based on number of objects, created respective polybase(s). After executing few of them successfully, I got below error for one of them


HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: MalformedInputException: Input length = 1


To troubleshoot  checked the error statement ---

HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: MalformedInputException: Input length = 2 


It appeared something in Input has length = 2, (size mismatch) but upon checking file found no such fields.

Why it happened


I have created the external table with varchar as the datatype for all the columns so that all datatypes can be handled.

             CREATE EXTERNAL TABLE SampleTbl(
ID VARCHAR(200),
                        NOTE VARCHAR(1000),
ROWGUID VARCHAR(200),
CreateDate VARCHAR(200)
             )


But there's an exception, columns having datetime values and that's the reason for the error.  

Although external table will be created, the error will come while inserting in destination table, where expected is datetime value but trying to be inserted is varchar.

The error message doesn't say so 😉


What to do


So while creating external table,  check for columns having datetime datatype. If there is then specify it .

            CREATE EXTERNAL TABLE SampleTbl(
ID VARCHAR(200),
                        NOTE VARCHAR(1000),
ROWGUID VARCHAR(200),
CreateDate DATETIME
            )









No comments:

Post a Comment

If you have any suggestions or questions or want to share something then please drop a comment