Diet sabotage, editors, and crappy write(fd, &mystruct, sizeof(mystruct))-style log formats

[talarubi]

So DietPower, the food logger I use, decided as long as I'm sitting here, I could wait two minutes for the food log to open. It was already slow, but now it's firmly in the realm of code that doesn't scale.

C'mon, it's clearly a C job. No excuse. I know people who write faster apps in Perl!

Another irritating thing is it's missing an export option, so the only way to get the logs out involves manual inspection, and then writing the code. The latter's not so hard, but I always used to cringe at binary files. Maybe the braver people would drag out a hex editor.

Well, the first thing I reach for is EDIT.COM. That used to be because it's all I had that reliably saved binaries... too bad it doesn't do hex, right? But after awhile I noticed that an ascii view is really useful, quite possibly moreso.

For one, you get the best use of screen space, which reveals a lot of patterns. Intel code is very dense, with the occasional word-size data thrown in. C structs with small numbers (e.g. most of them) leave a lot of zero padding. Pointers and offsets can be seen growing throughout the file. Text, of course, is immediately readable.

Repeated data shows up especially well, since nearly every byte has a unique glyph. The exceptions, 0x00/0x20/0xff, are blank. Happily, this includes various representations of nil: space, the integers 0 and -1, floating-point zero--and unused numeric precision.

Other values show up too, for example values centered around 0x3f (?, @, A) which are sign/exponent bits for small positive floats. "  Ç┐" happens to be 0xbf800000, or -1.0. At left, it marks unspecified nutritional values. (Sloppy coding has introduced other negative values as well!)

If the file is just a series of structs, it shows up as angular striations, betraying the length and number of the records. Usually somewhere nearby is the count. If you know your hex and ascii (essentially base-16/256), you'll quickly find it, and now you can write a skeletal reader.

And then you can go try all the program's special cases, and make note of what it spits out. This is how I wrote my OpenCanvas reader. Maybe I can get DietPower's logs into a spreadsheet or something...

It does give me thoughts, though. Using EDIT works great, but it's still a pain to do the math. I wonder if there's a better editor out there. I think I should be able to select regions, say "this looks like a record", and have the thing pattern-matched down the whole file. Then I could say "these are float fields", and it would update everywhere.

I think I would have circular tooltips, where you hover over some data and it shows you all the different values it could be in each representation it knows, and then the most likely one. (0's or 1's on the left = small int, high bits near 0xbf, 0x3f = small float, compare to values in other instances of the record...) Then you'd right-drag through a pie menu to choose one of them.

There could even be a hex view, either through a lens tool, or using a color mapping on top of the numbers, to show patterns. Or you could space things out and display assembler inline with all the other data, using heuristics (statistics, file headers) or manual selection to identify the code.

Ideally it would come with different plugins for the known formats, and people could write their own. You could use it as a teaching tool for Operating systems 101, or to help debug things, or repair corrupt files.

Ohwell, I'm probably dreaming. But it would be helpful right now.