7-bit sysex scrambling from mios

MIDIbox Tools & MIOS Studio
1

So I’m experimenting with developing a PC editor for my pedal board project. At the moment i’m just reading a syx file downloaded from a bankstick

I’m hoping someone can enlighten me as to the 7-bit scrambling.

I have assumed it’s the first byte of 8 that contains the first bit of the following 7.

0ABCDEFG, 0AAAAAAA, 0BBBBBBB, 0CCCCCCC, 0DDDDDDDD, 0EEEEEEE, 0FFFFFFF, 0GGGGGGG.

So to get the value of Bankstick address 0x0000

Given that the header is 11 bytes long, it should be:

value = ((12thByte & b01000000) << 1) | 13thByte

Can anyon help me out?

Cheers!

2

No, this kind of scrambling isn’t used by MIOS, instead the bits are packed without de-positioning.

You will find two useful perl scripts, which include the algorithms, in the MIOS source code package under tools/hex2syx.pl and tools/syx2asm.pl

Best Regards, Thorsten.

3

whoa…  my head hurts…

they really are scrambled

Are there some notes that explain the concepts of the algorithms?

I’m having some trouble following the math.

Cheers!

4

Alright, I’ve got the basic idea sorted. It’s actually fairly straight forward.

Nothing like some sleep to clear your head.

Now where’s my copy of Java for Dummies ;D

5

There is a Java based scrambling routine in the MIOS Studio package - but it doesn’t descramble.

Unfortunately Java is not like e, where you can easily define a bitvector with unlimited length, pack the 8bit value array into this vector, and unpack a 7bit value array from the vector. Thats basically the approach. In “e” a single line of code, in Java/C… hm… perhaps some more.

Best Regards, Thorsten.

6

haha, yeah I discovered that the hard way.   :frowning:

my second attempt is to grab 1 bit at a time and place it directly in the correct byte, it’s pretty ugly but works so far.

7

With Java (where memory consumption doesn’t matter), you could maybe also do it the following way: convert the decimal value of each 8bit byte into binary strings, and concatenate them one after another to a long, new string. Thereafter split the string into chunks of 7 characters, and convert back binary->decimal to get the 7bit value.

Pseudo code:

  string := “”

  foreach byte

     string := string + decimal_to_binary(byte)

  while lenth(string) && (length(string) >= 7) {

    SendMIDI(binary_to_decimal(string[0:6]))

    string := string[7:end]

  }

(this is no official programming language, I only want to demonstrate the algorithm…)

Best Regards, Thorsten.

8

Hello number-crunching friends!

I’m also trying to figure that out for a patch editor written in C;

however, I am not sure about the length of the returned sysEx header and would appreciate some clarification.

I get these (hex-) values when I request the first bankstick’s content:

[tt]f0 00 00 7e 40 00 02 40 00 01 00 02 12 6a 32 69[/tt]

from the syx2asm.pl I see the header is:

[tt]f0 00 00 7e 40[/tt]

from the doku I guess the next numbers are:

[tt]00  [/tt] the device ID

[tt]02  [/tt] “WRITE_CMD”

[tt]40  [/tt] means it’s a BS memory, incrementing later on (0x41, 0x42…)

[tt]00 01[/tt] address high and low byte

[tt]00 02[/tt] counter high and low byte?

Then my data would start with [tt]12[/tt], is this right?

I’m a bit confused about the counter low byte which is [tt]7f[/tt] later on…

I hope this question is not too silly, but my eyes start to flicker of the endless rows of numbers I looked at this evening

Best,

Michael

9

The term “counter” could be missleading, it actually means: SysEx block size / 8

It’s normaly the same for all blocks. 256 byte blocks are usually prefered, this results into a “counter value” of 0x100 >> 3 = 0x20

Best Regards, Thorsten.

10

thanks for your answer, Thorsten.

Unfortunately I’m still struggling with the byte-decoding.

The syx2asm.pl script produces the correct output: the first byte is 0x04, which is fine, because it’s the version compatibility info for v0.4 (see attached code snippets and ACSensorizer Patch Description)

  • however, if I use the suggested dec2bin/bin2dec method, I get a wrong result:

convert the decimal value of each 8bit byte into binary strings, and concatenate them one after another to a long, new string. Thereafter split the string into chunks of 7 characters, and convert back binary->decimal to get the 7bit value.

So in my case:[tt]

0x12 = ‘0001 0010’

      ’ 000 1001’ = 0x09[/tt]

but 0x09 is not the 0x04 I was hoping to for  :cry:

I get the feeling, I’m missing something pretty obvious…

Best Regards,

Michael

0: f0 00 00 7e 40 00 02 40 00 01 00 02 12 6a 32 69  ...~@..@.....j2i
16: 54 5e 30 1c 4c 06 00 00 00 00 00 00 00 0c 40 00  T^0.L.........@.
32: 00 00 00 00 00 60 18 0c 06 03 01 00 00 00 00 00  .....`..........
48: 00 02 02 01 41 00 50 30 1c 00 00 00 00 02 41 20  ....A.P0......A
64: 50 28 07 03 01 60 40 20 10 08 04 00 00 00 00 00  P(...`@ ........
80: 00 00 00 00 00 00 00 00 00 10 08 04 02 00 20 10  .............. .
96: 08 04 01 00 40 20 40 00 00 00 00 00 00 00 00 04  ....@ @.........
(...)
[/code]

[code=syx2asm.pl asm-output ----- this is correct]org 0x400000
db 0x04, 0x4b, 0x53, 0x2d, 0x35, 0x2f, 0x30, 0x39   ;; (0x400000)
db 0x30, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00   ;; (0x400008)
db 0x64, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x01   ;; (0x400010)
db 0x81, 0x81, 0x81, 0x81, 0x00, 0x00, 0x00, 0x00   ;; (0x400018)
db 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07   ;; (0x400020)
db 0x00, 0x00, 0x00, 0x00, 0x14, 0x14, 0x14, 0x14   ;; (0x400028)
db 0x07, 0x06, 0x07, 0x04, 0x04, 0x04, 0x04, 0x04   ;; (0x400030)

and here the output from my calculations:

    0: 12 6a 32 69 54 5e 30 1c 4c 06 00 00 00 00 00 00  .j2iT^0.L.......
   16: 00 0c 40 00 00 00 00 00 00 60 18 0c 06 03 01 00  ..@......`......
   32: 00 00 00 00 00 02 02 01 41 00 50 30 1c 00 00 00  ........A.P0....
   48: 00 02 41 20 50 28 07 03 01 60 40 20 10 08 04 00  ..A P(...`@ ....
   64: 00 00 00 00 00 00 00 00 00 00 00 00 00 10 08 04  ................
   80: 02 00 20 10 08 04 01 00 40 20 40 00 00 00 00 00  .. .....@ @.....
   96: 00 00 00 04 66 1f 10 46 40 00 00 02 48 04 02 01  ....f..F@...H...
[/code]

[code=Binary representation]
    0: 30 30 30 31 30 30 31 30 30 31 31 30 31 30 31 30  0001001001101010
   16: 30 30 31 31 30 30 31 30 30 31 31 30 31 30 30 31  0011001001101001
   32: 30 31 30 31 30 31 30 30 30 31 30 31 31 31 31 30  0101010001011110
   48: 30 30 31 31 30 30 30 30 30 30 30 31 31 31 30 30  0011000000011100
   64: 30 31 30 30 31 31 30 30 30 30 30 30 30 31 31 30  0100110000000110
   80: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
   96: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000

    0: 09 12 24 49 13 26 4d 1a 35 6a 54 28 51 23 46 0c  ..$I.&M.5jT(Q#F.
   16: 19 32 64 49 13 26 4d 1a 34 69 52 25 4a 15 2a 55  .2dI.&M.4iR%J.*U
   32: 2a 54 28 51 22 45 0b 17 2f 5e 3c 78 71 63 46 0c  *T(Q"E../^<xqcF.
   48: 18 30 60 40 00 01 03 07 0e 1c 38 71 62 44 09 13  .0`@......8qbD..
   64: 26 4c 18 30 60 40 00 01 03 06 0c 18 30 60 40 00  &L.0`@......0`@.
   80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
   96: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[/code]
11

When you are starting at the 12th, and not at the 13th byte, it makes sense:

First 8 bytes of BankStick content:

0x04, 0x4b, 0x53, 0x2d, 0x35, 0x2f, 0x30, 0x39

Binary representation:

00000100 01001011 01010011 00101101 00110101 00101111 00110000 00111001

Scrambled SysEx Stream:

0x02 0x12 0x6a 0x32 0x69 0x54 0x5e 0x30 0x1c 0x4c 0x06

Binary representation:

0000010 0010010 1101010 0110010 1101001 1010100 1011110 0110000 0011100 1001100 0000110

Now we concatenate the binary values to a single string (remove spaces)

00000100010010110101001100101101001101010010111100110000001110010011000000110

And split it into 8bit chunks:

00000100 01001011 01010011 00101101 00110101 00101111 00110000 00111001 …

Convert it back into hex (or decimal) values:

0x04 0x4b 0x53 0x2d 0x35 0x2f 0x30 0x39

Surprise! :wink:

Best Regards, Thorsten.

12

When I read your initial question, I guess, that the 0x40 was confusing you.

The 8th and 9th byte are the address, 0x4000 means: BankStick, offset 0

Best Regards, Thorsten.

13

I’ve had some success, but i’ve hit a brick wall

I’m dealing with uploading/dowloading a whole 32k bankstick

I’ve successfuly got the sysex message translated correctly to bankstick format.

The initial method was skip the first 12 bytes, run the rest of the file through the algorithim. Then every 0x400 bytes I skip 12 bytes (i think it’s was 12 anyway, might have been 8 or 10 :-[ my code is another computer).

This worked great 100% accurate using a sysex file made with hex2syx.pl

Getting it back to a sysex message has proved much more difficult.

So to convert to sysex. it’s 11 bytes of header and info.

followed by 0x400 bytes of bankstick data (in bankstick format). Which should be 0x492 bytes of sysex data.

followed by the checksum and 0xF7

Total 0x4A0 bytes.

loop that 32 times

but for some reason I fall 3 bytes short. Adding 3 blank bytes  before the checksum almost worked. But my bankstick data ends up being offset +2 every 0x400 bytes. (using my sysex to bankstick code)

So I go back to my sysex to bankstick conversion. To do it properly (hoping to find an answer).

I take the header and info (11 bytes worth), convert the next 0x492 bytes (7-bit to 8-bit), get the checksum (to eventully compare it) and F7.

Then loop the sequence 32 times to get the full bankstick.

Taking the same sysex file I used earlier, The first loop works great, my first 0x400 bankstick bytes are correct, but as of the second loop the data is incorrect.

I can relate to AC here… there has to be something obvious I’m missing.

edit: added a bit more clarity

14

Why do I get the feeling, that your function is not adding the one bit, to convert 7b to 8b - either on the first or the last byte… Just a gut feeling…

15

yay, it works!!

thanks so much to all of you.

I was making two errors: First, I have indeed been confused by the 0x40 (Bankstick address in the header), forgetting the low byte and therefore assumed to start with the 12th byte, but it’s of course the 11th byte to begin with;

secondly I misread the 0x20 (lenght of sysEx) for a 0x02 (12th byte)… typical kind of inattention-error, that’s why I always failed in math at school ::slight_smile:

I’m going to make some more tests and then I will post some lines of C-code that may be useful for others (and probably for a Java conversion as well).

Best,

Michael

…see the names in the third descrambled output? “KS-5/09” and “KS-5/SP” :smiley:

SysEx data output:
    0: 02 12 6a 32 69 54 5e 30 1c 4c 06 00 00 00 00 00  ..j2iT^0.L......
   16: 00 00 0c 40 00 00 00 00 00 00 60 18 0c 06 03 01  ...@......`.....
   32: 00 00 00 00 00 00 02 02 01 41 00 50 30 1c 00 00  .........A.P0...
   48: 00 00 02 41 20 50 28 07 03 01 60 40 20 10 08 04  ...A P(...`@ ...
   64: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 08  ................
   80: 04 02 00 20 10 08 04 01 00 40 20 40 00 00 00 00  ... .....@ @....
   96: 00 00 00 00 04 66 1f 10 46 40 00 00 02 48 04 02  .....f..F@...H..
  112: 01 00 60 30 18 0c 05 0c 40 66 63 4f 7f 7f 7f 44  ..`0....@fcO...D
  128: 16 4e 67 43 10 00 00 00 00 18 6d 66 23 31 7e 7f  .NgC......mf#1~.
  144: 3f 5f 60 44 5a 4c 5a 35 17 54 6a 03 01 40 04 02  ?_`DZLZ5.Tj..@..
  160: 00 40 20 10 23 10 00 00 00 00 00 00 58 50 02 01  .@ .#.......XP..
  176: 00 40 20 00 00 02 00 00 00 40 40 30 20 14 0c 00  .@ ......@@0 ...
  192: 00 00 00 00 00 50 28 14 0a 01 00 20 28 0c 00 00  .....P(.... (...
  208: 00 00 0d 16 23 6d 7e 7f 3f 5f 6f 70 00 08 00 00  ....#m~.?_op....
  224: 04 02 01 00 40 08 04 02 01 00 20 10 08 10 00 00  ....@..... .....
  240: 00 00 00 00 00 00 00 57 30 15 0a 20 00 00 00 52  .......W0.. ...R


Binary 7-bit output:
    0: 30 30 30 30 30 31 30 30 30 31 30 30 31 30 31 31  0000010001001011
   16: 30 31 30 31 30 30 31 31 30 30 31 30 31 31 30 31  0101001100101101
   32: 30 30 31 31 30 31 30 31 30 30 31 30 31 31 31 31  0011010100101111
   48: 30 30 31 31 30 30 30 30 30 30 31 31 31 30 30 31  0011000000111001
   64: 30 30 31 31 30 30 30 30 30 30 31 31 30 30 30 30  0011000000110000
   80: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
   96: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  112: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  128: 30 31 31 30 30 31 30 30 30 30 30 30 30 30 30 30  0110010000000000
  144: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  160: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  176: 30 30 30 30 30 30 31 31 30 30 30 30 30 30 30 31  0000001100000001
  192: 31 30 30 30 30 30 30 31 31 30 30 30 30 30 30 31  1000000110000001
  208: 31 30 30 30 30 30 30 31 31 30 30 30 30 30 30 31  1000000110000001
  224: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  240: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000


Converted 8-bit output:
    0: 04 4b 53 2d 35 2f 30 39 30 30 00 00 00 00 00 00  .KS-5/0900......
   16: 64 00 00 00 00 00 03 01 81 81 81 81 00 00 00 00  d...............
   32: 00 01 02 03 04 05 06 07 00 00 00 00 14 14 14 14  ................
   48: 07 06 07 04 04 04 04 04 00 00 00 00 00 00 00 00  ................
   64: 00 00 00 00 08 08 08 08 02 02 02 02 01 01 01 04  ................
   80: 00 00 00 00 00 00 00 00 99 8f 90 8d 00 00 00 52  ...............R
   96: 02 02 02 03 03 03 03 02 8c 81 9b 1c ff ff ff c4  ................
  112: 2d 3b 3c 32 00 00 00 00 63 6e 64 6c 7f 7f 7f 7f  -;<2....cndl....
  128: 04 4b 53 2d 35 2f 53 50 30 30 02 02 01 01 01 04  .KS-5/SP00......
  144: 64 00 00 00 00 00 0b 14 01 01 01 01 00 00 00 80  d...............
  160: 00 01 02 03 04 05 06 00 00 00 00 00 14 14 14 14  ................
  176: 04 02 05 03 00 00 00 00 69 64 7b 7f 7f 7f 7f 7f  ........id{.....
  192: 00 02 00 00 08 08 08 08 02 02 02 02 01 01 01 04  ................
  208: 00 00 00 00 00 00 00 00 57 60 54 52 00 00 00 52  ........W`TR...R
  224: 02 02 02 02 03 03 03 02 8c 56 9b 53 ff ff ff c4  .........V.S....
  240: 23 27 14 13 00 00 00 00 69 64 7b 7f 7f 7f 7f 7f  #'......id{.....
16

Nice work AC!!

Have you had any success going back to sysex?

I’m flat out at work for next 2 weeks. So I won’t have time to work on it for a while. :frowning:

17

nah, I’m not that far  :slight_smile:

next step is to split up the numbers and feed them in a table.

But it’s the same here: I have an exam end of the week, so I can resume working on it next week soonest (and I better won’t dare to overwrite my BS with cheesy data until I’m sure it works ;D )

Best,

Michael

18

hmmm, I had brain wave last night… Can someone help me out with this…

when converting 7bit to 8bit, it should be 1024 hex bytes to a section, times 32 to get a full bankstick.

From reading a sysex file in a hex viewer, there is 1182 bytes to a section.

So minus the header and info (11 bytes), minus footer (2 bytes).

Leaves 1170 bytes that make up the bankstick (in 7bit).

Given that 8 bytes of syx are converted to 7 bytes hex.

1170 divided by 8 = 146.25

146.25 * 7 = 1023.75

what am I missing?   ???

Why do I get the feeling’ date=’ that your function is not adding the one bit, to convert 7b to 8b - either on the first or the last byte… Just a gut feeling…

[/quote’]

2 bits maybe?

19

Not sure about your algorithm, but 1182 - 11 - 2 = 1169  :wink:

Uhm… those fractions don’t always work out, so make sure you round up to the nearest whole bit. No quantum PICs just yet :frowning:

20

Hi Durisian,

as I said, I’m not finished yet, but by using the approach Thorsten suggested, it seems to work so far.

But remember: haven’t tested enough and the checksum is totally ignored, so there is a 1-byte shift in the output; I only used it to read the BS contents – never in the other direction, to write back the edited data… normally I wouldn’t post it like this :wink:

It’s in C, the second snippet using some additional (easy to remove) Objective-C;

the only C-lib-function used (IIRC) is strcat(), which is simply combining two strings to one (but note the end-byte [tt]\0[/tt])

void dec2bin(int x, unsigned char *result) {
int i;
result[0] = 0;
for (i = 7; i >= 0; i--) {
strcat(result, x & (1 << i) ? "1": "0");
}
}

long bin2dec(unsigned char *s) {
long r=0;
for (; *s; r = (r<<1) | (*s++ - '0'));
return r;
}
[/code]

and this is the current decoding snippet; [tt]sxd[/tt] is an NSData Object (Objective-C), that is interchangeable with a binary SysEx-File. But before working on the bytes, I put them in a C-buffer, so conversion should be relatively easy:

[code=SysEx decoder - early beta, use at your own risk]
-(void)printAcd {
// prints sensorizer patch data (sxd with stripped sysEx headers)

if(sxd == nil) {
// try to write data to file (for debug)
// [sxd writeToFile:@“~/ACSensorizer_PatchManager/BS_dir/ACS_BS_NSData.acd” atomically:YES];
// try to read data from file (for debug)
[self setSxd:[NSData dataWithContentsOfFile:@“~/ACSensorizer_PatchManager/BS_dir/ACS_BS_NSData.acd”]];
}

unsigned long l = [sxd length];
unsigned long i,k;
unsigned char buf[l];
[sxd getBytes:buf length:l];
for(i=0;i<l;i++) {
switch(buf[i]) {
case 0xf0: // skip header
i += 11;
break;
case 0xf7: // skip EOX
i++;
break;
// IMPORTANT: CHECKSUM NOT ATTENDED; THIS HAS TO BE DONE TO MAKE IT REALLY WORK
}
[acd appendBytes:&buf[i] length:sizeof(buf[i])];
}
// debug out
l = [acd length];
[acd getBytes:buf length:l];
// printf(“\n\nSysEx data output:\n”);
// hexview(buf, 256);

// now we want to convert the bits and bytes from sysEx 7-bit back to the original 8-bit format
l = [acd length];
char nbuf[l];
[acd getBytes:nbuf length:l];
unsigned char b[l*8]; // binaries string
unsigned char a[l]; // 8-bit numbers array
unsigned char s[8]; // 7-bit binary
// build one long string of binary coded decimals
for(i=0;i<l;i++) {
dec2bin(nbuf[i],s);
for(k=0;k<7;k++) {
s[k] = s[k+1];
}
s[7] = ‘\0’;
strcat(b, s);
}
// debug out
// printf(“\n\nBinary 7-bit output:\n”);
// hexview(b, 256);

// split b to 8-bit chunks
unsigned char bchunk[9];
for(i=0;i<l;i++) {
for(k=0;k<8;k++) {
bchunk[k] = b[(i*8)+k];
}
bchunk[8] = ‘\0’;
a[i] = (unsigned char) (bin2dec(bchunk));
}
// print hexview of b
// printf(“\n\nConverted 8-bit output:\n”);
printf(“\n8-bit ACS data, BS#%i\n”, bankstick);
hexview(a, l);

}

The conversion needs some time, but compared to the time the SysEx dump needs to arrive via MIDI, it’s more than okay :slight_smile:

Hope this helps,

best

Michael