Programming Bankstick Files

Although this query relates to my organ project, this is likely to be of general interest.

QUERY

I believe that I can write sysex files to a Bankstick using the MIOS Sysex implemetation commands, using command 03 and then Command 02.

To test that the files are stored I would then, using Commands 03 and 01, read from the Bankstick.

I would use MIDIOX to send and receive the data files.

Is this procedure correct?

Do I need to prepare the Core module other than giving it an ID and programming the bootstraps, MIOS and Midio128?

I do realise that ultimately the send routines to transmit the sysex messages to my sound module will be controlled by a button handler.

Regards Robin Fawell

Yes, in principle thats correct.

There is a tutorial application in the MIOS download section which explains BankStick transfers step-by-step, search for “bs_xfer_v1_4.zip”

Best Regards, Thorsten.

Dear Thorsten

Thank you for the last reply.

You referred me to the BS Transfer Method.  In Step 1 you state that the file Main.asm includes P=EEPROM16.  It contains “LIST      P=PIC18F452, R=DEC”.  There is no reference to EEPROM16.

I am not ready yet to modify the main.asm file and it seems easier (for me) to use the MIOS Sysex implementation method.

It might be better if I explained in detail what I am trying to do.

I have recorded using Midiox several sysex messages.  These are “setup” instructions for the Roland SC8850 Sound Module.  I want to store these in my Bankstick module (0-7). This is built but not tested.

Checksum

In the sysex implementation text page 3, it refers to "checksum: inverted  checksum from ah to the last byte of <dump>.

In hex to Sysex converter you refer to "-(sum of bytes) & )x7f.  

Are we counting 8 bit or 16 bit bytes?

If we count the bytes fom ah to the last dump byte in binary invert the result ie 0101 becomes 1010 and then “AND” the result with 7f (0111 1111) would this be correct?

Do we always have to use perl to convert to syx from hex to send midi messages via Midi in?

Thanks Robin

I left this out of my preceding message.

In checking through the Sysex Implementation text under the command 01 examples I think there may be an error in the sysex message for the Internal Flash..  I believe that the ch parameter should be 0x20.

If Counter = 0x08000 When this is shifted 10  and "AND"ed with 7f the result is 0x20  (binary 0010 0000) not 0x10.  Of course I may be wrong and misunderstood the basis of the calculation.

Regards Robin

Hi Robin,

you are right, I forgot to update the documentation about the EEPROM16 type, it must be PIC18F452 - EEPROM16 doesn’t work due to an imperfection in the MPASM assembler

you are also right with the documentation error concerning the internal flash readout counter - ch must be 0x20

But regarding the SysEx upload: it isn’t possible to write any plain MIDI data (like SysEx strings) into the BankStick, they have to be encoded by the hex2syx script to another format. It’s too complicated to convert an existing data dump into the MIOS upload format without this script.

So, what you have to do: you have to write a script which converts your binary files to assembler text, so that the assembler can generate a .hex file, which can be thereafter converted to a .syx file by hex2syx.pl

The assembler format can be found in bankstick_content.inc

Best Regards, Thorsten.

Hi again,

I’ve updated the bs_xfer package (MIOS Download -> bs_xfer_v1_6.zip)

It now also contains a script which allows you to convert a plain binary file to assembler text, it can be found in the tools directory (tools\bin2asm.pl)

So, here the flow to bring plain binaries into the BankStick:

  • use bin2asm.pl to convert your binary file to assembler
  • you can copy&paste several assembler files to a single assembler file. Only requirement: the overall number of bytes should not exceed the 32768 boundary (0x407fff) if you are using a 32k BankStick, or 65536 (0x40ffff) if you are using a 64k BankStick
  • use MPASM to get a .hex file
  • use hex2syx.pl to get a MIOS upload file

You will notice that this conversion process simplifies the creation of the BankStick content a lot, since you are able to move and rearrange the converted binary files with a text editor to free BankStick addresses

Best Regards, Thorsten.

Thanks Thorsten for such a fast response.

May I ask you to clarify the checksum queries.

Regards Robin

Hi Robin,

I thought that the checksum question became obsolete since you will use the hex2syx.pl script which calculates the checksum automatically.

Maybe you’ve misinterpreted the term “sum of bytes”. It should mean: the checksum is the addition result of all bytes within the address/counter field and all bytes of the dump field. Than it will be negated, so that the checksum within the SysEx file + the calculated checksum (which will be calculated by MIOS during the upload) results to zero

The checksum is a 7-bit value, therefore only the first 7 bits are used (thats the “& 0x7f”)

Best Regards, Thorsten.

Dear Thorsten

Thanks for straightening me out.

I was in the middle of assembling a binary file complete with 01, 03 commands.

I am going to practice with my largest file, “reset” which will go to BS 00.

I will generate a reset.inc file using bin2asm.pl.

I will use MPASM to generate a .hex file.

I will convert this to a syx file using hex2syx.pl and then upload this using Midiox.

MPASM

Do I use the default settings?

I assume the source file is Reset.inc

Processor EEPROM or 18F452?

Which generated file?

Tab size 8 ?

Thanks again Robin

Hi Robin,

I would suggest to create a copy of the main.asm which which comes with the bs_xfer package, because it already contains all required settings (and therefore documentates the requirements and answers your questions)

From main.asm you can include one or more .inc files which contain the converted binaries.

Hint: after the assembler has been passed, you will find the address offsets to the converted binaries in main.lst - they are starting at 0x400000, the last 4 digits (16 bits) are equal to the BankStick addresses.

Best Regards, Thorsten.

Dear Thorsten

I have received the following response from the core module via Midi out after sending a test  .syx file.

F0 00 00 7E 40 00 0F 07 F7

F0 00 00 7E 40 00 0F 7D F7

F0 00 00 7E 40 00 0F 3A F7

F0 00 00 7E 40 00 0F 6E F7

F0 00 00 7E 40 00 01 F7

The file was sent to BS 7.  I have worked out that the messages are to do with WREG.

The last is a request for an upload, I think.  

Does this message seem OK?

Regards Robin

PS device ID is 0x00

Hi Robin,

yes, the whole upload procedure was successfull:

>  F0 00 00 7E 40 00 0F 07 F7  

is an acknowledge message (0F) which reports that the received checksum matched with the calculated checksum, that the data has been copied into the BankStick and that it has been verified (read-back and compare)

The value behind 0F (here: 07) is the same checksum like reported by the hex2syx.pl script.

You will receive an error message (0E) if any step fails (e.g. BankStick not connected, checksum error, verify error, etc…)

>  F0 00 00 7E 40 00 01 F7

is the upload request of the first level bootstrap loader.

A reset instruction will be issued after up- or downloads via MIOS, therefore this message appears like on a common startup

So - now you are sure that the binary has been copied into the BankStick.

Best Regards, Thorsten.

Dear Thorsten

I am progressing slowly but have reached another difficulty.

So far I have uploaded a large and medium converted binary file to BS 0 and 7.

The large file “Reset”  bin file was recorded as 16470 bytes by Midiox.  After conversion to Reset.syx the file size measured by Midiox was 59025 bytes.

During the upload Midiox received 51 Sysex acknowledgements. .

For the medium file the respective bytes were 960 and 3560 with 5 ack.

As you stated in your last message it seems as if data is being stored.

I have tried many different ways to read the stored data using the sysex read procedures with Midiox with no success.  Is it possible to use the sysex Read 01 Command for this task?

I am sorry to trouble you again so soon.

Regards Robin

Hi Robin,

it’s normal that the resulting file is larger than the original binary after the hex2syx conversion – this is due to the required conversion algorithm which converts 8-bit values to 7-bit values. MIOS converts them back to the 8-bit values during the upload.

So, normaly you should get about 1184 bytes per 1k block (original size: 1024 bytes)

But it’s very strange that the converted .syx files are much larger.

How did you convert the binary exactly?

What is the content of the resulting assembler file? You should see the address offsets within the files (they are inserted as comments at the end of each line), what is the last offset value?

Best Regards, Thorsten.

Thanks Thorsten

I recorded the sysex files using MIDIOX.

I saved the data from MIDIOX as a .txt file.

I created an .asm file from the .txt file using bin2asm.

Using MPLAB I made an .hex file using the .asm file as source file. Note this is the only file I used.

I converted this file to a .syx file using hex2syx.

I used MIDIOX to send the data to the Core via Midi in.

The last offset on the “Reset” fle is 0x40C0A0.  There is a change in the format up to 0x4080B0.  The data up to that point was db followed by eight double bytes ;; and then the address eg 40000 (1st add).

After this there are numerous db data sets per line.

As if the line return had been lost.

Regards Robin

Hi Robin,

I saved the data from MIDIOX as a .txt file.

This could be the reason why the converted .syx file is more than 3 times larger than the original file.

You’ve to save the SysEx data in binary format - just like a common .syx file. Thereafter use bin2asm.pl to convert this binary to a assembler file.

Check the content of the generated assembler code, the data behind the db statements must match with the data displayed in the MIDI-Ox window.

Best Regards, Thorsten.

Dear Thorsten

Following your guidance I have produced a new reset.syx file.  Originally as recorded with 16470 bytes after conversion to a syx file now 20137. (both measured by MIDIOX.

By the way with MPLAB I did not use main.asm with a Reset.inc file.  I made a reset.asm and used it (only) as a source file.  Was this OK?

Can I use Command 01 to read the BS?  As you probably realise I keep thinking that the MIOS system will respond to Midi sysex messages on Midi In.  Plese let me know how to proceed.

Thanks Robin

Hi Robin,

alright - the size of 20137 sounds correct, these are 17 converted 1k blocks, and the original file was a little bit larger than 16k - perfect!

Yes, now you could read out the BankStick again and convert the resulting .syx to a .asm file like described in the bs_xfer package. However, it’s more like a toy, but it helps you to 100% ensure that the file has been uploaded.

For a first test it’s ok when you assemble the reset.asm file directly, but later you propably want to have multiple files in a single BankStick, and for this it would be easier to include all data files into a single assembler file, so that the assembler can take care about the data locations.

Ok, now you should proceed with the function which sends the SysEx

string, see:

http://www.midibox.org/cgi-bin/yabb/YaBB.cgi?board=mios;action=display;num=1078174868

Later you should think about how to organize the data. I would suggest

to create a table which contains:

  • the bankstick number
  • the offset address
  • the size of the SysEx string

But I fear that I don’t have enough time to explain more details before my holidays… :-/

Best Regards, Thorsten.

Dear Thorsten

I can see the point of having multiple sysex strings held in each bankstick and being uploaded in one operation.

I have tried many times using MPlab to include the .inc files into the.asm file without any result.  The build is signalled as successful but I finish up with the original .asm file.

I note that each file (.asm or .inc ) starts with Address 400000.   Should not the .inc files  start with  different address(es)?

There are also comments in each .inc file re deleting the line where there is a LIST instruction.  Should these be followed?

Please confirm that I should be using MPLAB to make the composite .asm file and advise  any other instruction needed.

Regards Robin

Since sending the above message I wondered whether I should be composing the composite “write to bankstick” by cutting and pasting within MIDIOX.

Hi Robin,

did you solve the problem in the meantime?

Root cause: overlapping data areas.

You’ve to remove the “org” statements and to add a single one (which points to 0x400000) to the main.asm, or you have to relocate the data blocks by hand to free addresses.

Best Regards, Thorsten.