As the Transterpreter on the Arduino is getting more capable we are starting to write more and more occam programs. As these occam programs do not change the runtime itself, our current strategy of compiling the program bytecode into the runtime is becoming boring. Uploading the 16 KB interpreter together with a few hundred bytes of bytecode wastes quite a bit of time per upload. Thus, our next step is to separate the bytecode from the interpreter so that either can be uploaded independently of the other.
Currently we are linking the Transterpreter bytecode together with the
interpreter without any of the metadata normally contained in a
This metadata consists of information such as memory requirements for the
bytecode, debugging, symbol information, and other things. We really don't want
to be uploading much more than the bytecode to the Arduino though, as it has a
limited amount of flash. If we are interested in using the debugging
information, we are probably better off sending the relevant state back from the
Arduino and making the host computer perform a mapping from the current program
state to useful debugging information, using the debugging symbols found in the
Fortunately Carl thought of this while we were sleeping and introduced various changes (changeset: 5880, 5881, 5882, and 5883) to the plinker in order to be able to exclude certain sections of the bytecode file.
The Surveyor port of the Transterpreter uses a virtual machine with two contexts. One context executes the firmware code (occam code statically compiled into the Transterpreter) and the other context executes user code, which is only uploaded at runtime. In order to upload user code, the firmware is able to read bytecode sent over the network, store this into RAM, and later execute it.
On the AVR chips we don't have enough RAM available to store bytecode in RAM (we only have a few kilobytes). Instead it must be stored in flash (we generally have tens of kilobytes of flash). Currently we are storing bytecode in flash, but only as a side effect of compiling the bytecode directly into the interpreter. Instead we want to be able to upload either the interpreter or the bytecode into flash memory, without disturbing the other.
Avrdude is the tool which we use to upload the firmware. It reads a number of
ihex (amongst others). We are using
as this is what the Arduino makefiles generate by default and it seems like a
Before going further we decided to start by establishing that we can in fact
upload bytecode to flash at a location after the Transterpreter VM, ie without
any undesirable side effects such as the entire flash being erased. After
satisfying that we are indeed able upload 'stuff', without erasing the VM, given
the right flags and the appropriate addresses2 written in the
ihex file, we
proceed to look for an easy way to convert a bytecode file to
Our first idea was to use
objcopy in order to generate an
ihex file. This
almost works, except that when passing a start address (the location where we
want our bytecode written in flash) to
objcopy it is not used in the address
portion of the
ihex data record, but instead added on as a special command at
the end of the
avrdude unfortunately seems to ignore the start
address command, and only obey the addresses that are part of each data record.
As the data records written by
objcopy always start at address zero, this
unfortunately overwrites our virtual machine. After some poking around we
decided that was no other way forward than to write a small tool that can
ihex file given any data and a start address:
We now have the ability to upload any data we like to the Arduino, which
enables us to upload the
.tbc (Transputer bytecode) files generated by our
tools. We are careful (I think) to strip information out of the file that we
don't need, such as debugging information and the symbol table. Other
information is necessary for the correct execution of our programs though, such
as memory requirements. In order to obtain this information and to correctly
locate the bytecode within the uploaded data, we need to be able to correctly
parse TEncode, the format of the
.tbc files, based on the IFF format3.
libtvm source code already contains a decoder for TEncode, as the posix
interface to the Transterpreter already loads
.tbc bytecode files.
Unfortunately we cannot use this decoder due to the 'Harvard Machine' "problem".
Since we are storing the bytecode in flash on the AVR we need to use special
instructions to access the bytecode. The TEncode decoding routines in
are not aware of the program vs. data memory distinction and only currently work
with data memory. Instead of trying to generalise these routines, which are able
to decode more of the TEncode format than we currently care about, we instead
wrote a few AVR specific decoding functions which work with the AVRs flash
As of changeset 5884 we can upload occam programs (as TEncode data) to flash independently of the interpreter, which, upon reset, will load the uploaded code out of flash and execute it.
At this point Matt decided that we should eyeball how fast the interpreter is on the Arduino. Given our current setup of 12 LEDs in a circle, turning these on and off as fast as possible would identify if we are running very slowly (ie we see the lights blinking).
After we had identified that there was no visible blinking going on with the LEDs, we decided to try to see if we could guage the blink speed with our Saleae USB logic analyser. Unfortunately we were unable to test the blinkenlights speed using the logic analyser. The software is currently Windows only and Linux and OS X drivers and software have been promised, but are not currently available. Our only option currently is therefore to run the device in a virtual machine. While we have been told by the manufacturer that this should not be a problem, we have been unable to sample at data rates above one megahertz, presumably due to the virtualised USB interface. The software also crashes a lot, but again, this might be due the fact that the analyser is being run through a virtual machine. We have tried both Virtual Box 3.0 and VMWare Fusion 2.0.X.
So we were not able to identify exactly how quickly the LEDs were blinking, but our visual inspection assured us that it is plenty fast! Since the Arduino is running at 16MHz and has a rating of 1 MIPS per 1MHz our virtual machine would have to be pretty inefficient in order to blink the LEDs slow enough that the delay would be visible. We decided however that we had more important things than to sit and count instructions or go over to the physics department to borrow an oscilloscope.
Adam also fixed the compilation of the
forall library (occam's standard
library) on 16 bit machines in changeset 5885. We will need use
most interesting programs so this fix had to happen sooner or later.
Additionally occbuild is now more aware of what architecture it is compiling for
in changeset 5886, which ensures that we build 16 bit bytecode for 16 bit
targets. Various less interesting changes happened in 5888, 5889,
5890 fixed a problem where we were not clearing memory too late, which meant that some state which had already been set up in the memory allocated for the soon to be run occam program would be overwritten. This only manifested itself if a program actually ran to completion and quit, which is not actually something that occam programs tend to do a lot4.
In addition to our Arduino boards we also have a Sanguino. This is a board much like the Arduino, but made specifically for use in the RepRap project. Since we have it, and they are so similar, we thought we should demonstrate the Transterpreter running on it as well. Changeset 5887 commits some (for now, commented out) makefile magic to do this.
objcopy to generate some testing files. We were not able
objcopy to output addresses in the
ihex file in the way we
wanted which initially made testing that we could write to arbitrary
addresses in flash difficult. We decided manually edit the addresses in
the file and see if
avrdude would ignore the checksum (which include the
address of course).
avrdude does not ignore the checksum, but it
helpfully calculates and displays the correct checksum for the line,
making it easy to further patch the
ihex file until all the checksums
are correct. ↩
occam programs are often made up of parallel processes. These parallel
processes are most often written in a manner such that they will never
exit, ie using a
WHILE TRUE loop in their body. ↩