Binary file with Endianness is expressed as big-endian and field PIC.

[sfuenteRoot]

we are migrating a process developed in dataStage to scala, using bookshelves. The Copyboock is defined and we manage to read the file in DF with select. But the fields that in the original process we read as binary (Raw) we fail to interpret correctly.

in the DS process, is define like

this is loaded into a database, converting it to hexadecimal, and we can see it, for example so:

but in DF, we do not know how to treat it and we see it like this:

   .withColumn("CMAVCRPE_P55_TEXT_ori", (col("CMAVCRPE_P55_TEXT")))
   .withColumn("CMAVCRPE_P55_TEXT_hex", hex(col("CMAVCRPE_P55_TEXT")))

Then, we think that we are not performing the reading correctly and that bits are lost.

Thank you

[sfuenteRoot]

The schema print:

| | | |-- CMAVCRPE_P55: struct (nullable = true)
| | | | |-- CMAVCRPE_P55_LEN: integer (nullable = true)
| | | | |-- CMAVCRPE_P55_TEXT: string (nullable = true)

But i want to change to binary, like:

[yruslan]

It seems that the new feature that is not released yet could help here.
#624

You can try it out by building the master branch, or by waiting a few days when the release is planned.

The fields that you want to be 'raw' the pic should look like this:

   09.  ZMAVCRPE_P55_TEXT  PIC X(63) COMP.

(note 'COMP')

[sfuenteRoot]

Thank you, for your prompt reply. We look forward to the deployment of the next version.

[yruslan]

2.6.8 is released. You can define binary fields as above (PIC X(63) COMP.)