Weird invisible character in file

Hardware and technical stuff
1

so i used the head command to try and read the first part of a file my company has to process at a unix prompt. it returned something like an upside down question mark and 400 and the prompt was on the same line, so unix had a problem reading it. opening the file up in windows showed nothing unusual.

doing a head filename > test.txt and then opening in windows showed the file’s first ten lines but in between each character was a box character. not only that, but it i deleted the box and tried to do an undelete using my text editor’s undo, it wouldn’t do it, like it didn’t know what the character was.

dos2ux did nothing to fix it.

what’s this weird invisible character and how do i get rid of it?

2

Use a hex editor, preferably one with a search and replace function.
Although it does sound liek the text file is in unicode.

3

there are limits to how much i am allowed to alter data from a client. if i did have a hex editor, what would i be doing?

4

‘od -x filename’ will give you a hex dump of the file. Though it does sound like it might just be a Unicode byte order marker at the start of the file.

5

Watch out for “low values” if this file was generated from a data file on a mainframe somewhere. I still don’t understand those damn things and they always wreck data files for everyone else.