Interesting serial port driver hacks

So ... say you've got a Mac. And you've upgraded to El Capitan.

Now your Keyspan serial adapter doesn't work (I have a USA-19HS and older USA-28x) - this used to be a built-in driver, but I guess Apple dropped it. This makes me sad, because Keyspan adapters generally were excellent and didn't cause kernel panics. One obvious option is to install the driver. http://www.tripplite.com/support/USA19HS
 

Say you also have a Prolific-based adapter - not surprising - these are everywhere for a good price. But the last time I installed the drivers I had kernel panics - of course it might be unrelated. You might have to install the native drivers if you want to use a Mac application with it, or see the options below...

Generally, apart from VirtualBox, I never want to install non-Apple drivers/Kext ever, really. (I've considered uninstalling VirtualBox in the past - but currently my machine seems stable even running VirtualBox ... thanks Apple ... famous last words, I know).

But installing drivers is the standard option, I'll admit.

You have the following other choices, as far as I can tell:

1. Use a computer with Yosemite or before. If that sounds like a pain, that's because it is. Especially as your whole family appears to be upgrading.
2. Get an FTDI-based serial adapter off Amazon. Hope it's not a dodgy FTDI clone. This is the route I'd go if it wasn't present season. I *suspect* that the Mac has an FTDI driver built in because it works with things like the Forth board on my the Display Board (which has an FTDI to serial adapter) without installing any drivers...
3. Downgrade to Yosemite.
4. Use a non-Mac ... (NOTE: My Windows PC machine can't sleep without a blue-screen half the time because of serial adapter drivers - none of the ones mentioned here, I should say. Yes I have evidence (a) it started crashing right after installing and (b) it also mentions the specific two drivers in the blue screen. But my work requires them, so hey...).
5. Use a Mac ... but use Ubuntu running in VirtualBox! Ubuntu has Prolific drivers built-in.

VirtualBox has an interesting ability to pass USB traffic to the guest OS and use guest-native drivers, even if the host doesn't have a driver!! So, basically you never need to install a Mac driver (if you can use a Linux app rather than a Mac app).

This link might be helpful for VirtualBox or just normal Linux people ... http://ram.kossboss.com/prolific-rs232/

P.S. To use a Keyspan driver under Linux, you'd need to install it yourself. Debian didn't like the licensing terms, apparently.

Automatic testing and Regression tests

For a couple of minutes a day, I've been creating regression tests for one of my background projects - LuaZ80. I started this to identify which instructions were implemented, and which were working properly.

https://github.com/robzed/LuaZ80

Before that I was using a test file to single step the instructions.

Why I didn't make regression tests before because as well as finding which instructions are implemented (at all or properly) they've found a load of edge cases I'd have never found by manual testing.

Well, one reason is the massive effort required to do so ... I guess several times the amount of time to write the original compiling interpreter!

However, without them I have a project which doesn't work properly - based on the number of mistakes and minor errors I've found. And what use is that?

Additionally I have something to check the operation in the future when changes are made - the true meaning of regression tests.

Read out loud blog posts

Not sure why I didn't think of this before, but of course modern computers have text-to-speech and can read text aloud.

So I got it to read my post from yesterday - instantly found two errors (which then I had to correct) and also several poorly written sentences (which I left because life is short and it's probably better to make new blog posts better).

What made me realise this was browsing Mariner Software's writing software and finding this http://marinersoftware.com/products/narrator/ which allows setting different voices in different pieces of text. But for simple use cases the built-in stuff is enough.

Other notes and links:

Firefox has Text-to-speech add-ons.

Linux can use Festival http://www.xenocafe.com/tutorials/php/festival_text_to_speech/index.php but there are others as well.

Raspberry Pi Text to speech ... http://elinux.org/RPi_Text_to_Speech_%28Speech_Synthesis%29 ... could be interesting for Embedded speech!

Windows Text to Speech is also available: http://answers.microsoft.com/en-us/windows/forum/windows_7-pictures/how-to-configure-text-to-speech-in-windows-7/83f67763-9d2b-416b-ad03-0e40ea97585b?auth=1

For Mac users, Safari and Text Edit read things aloud. Go to 'System Preferences' ->  'Dictation and Speech' to configure the voices.

New test version of Forlorn Fox

So, apparently it's been 3 months since the last public test version. There have been lots of internal changes and only a few visible external changes.

More annoying, immediately after packaging it, we find a simple but yucky bug - the glyph editor doesn't work at all. And the fix is trivial - literally one require line.

We decided to release with the fault anyway - because 3 months is way too long between test releases. And we can always release another version next week - the whole process of release for all four platforms is probably under two hours end-to-end. So I have to wonder why we waited so long - I guess it's the feeling of one more thing.

We spend time fixing the build-and-package earlier this year in fixing this after the 0.03 release took ages to put together.

Currently the upload to web site is manual - but it's relatively trivial to do. But we've of course talked about automating that as well.

Next up are various extra game features and maybe another mini-game, although we've been talking about fixing one specific 'structural' problem: the single player and server player (for network) share a lot of game logic code, but the network client player game is different in a lot of ways. This causes us problems every time we add a game feature - we effectively have to add it twice and debug it twice. One alternative is to run all games as a type of client. But that's quite a big change - but one which might be necessary.

Software and the Future of humanity

My day job is proprietary and closed-source software, and is a lot of other peoples I know. It seems to work well, most of the time.

But unlike most other technologies, the workings are almost totally hidden without massive effort with binary software.

The future of humanity is at a turning point regarding knowing what software actually does inside - and this disadvantage doesn't just affect individuals but governments and corporations as well.

At the moment 'we' are trying to trying to tie things down with operating system permissions and network settings; I think this effort will fail - even though it's only just started. Already simple Android applications ask for nearly all permissions and most programs require a set of network permissions to make them work properly. Apple and others can revoke iOS authors licenses killing all copies of that software. But again - the software has to be very bad for that for happen and won't happen in most cases. So how can we tie them down after the fact? I don't think we can.

There are options other than free software and open source. Partially open solutions, for instance, where you can see the code (and maybe even change it), but not redistribute it might be the right license for some software. Wasn't Minecraft like this? There are others - not all were as successful as Open Source but some were.

I also think that people making money of the same, basically unchanged, software for 30+ years seems silly - so free/open source solutions (e.g. LibreOffice and GNU/Linux) will perhaps provide competition and drive all software forward.

To clarify this last point - unlike artistic endeavours where there is something intrinsically special, most software is functional. You can argue that Minecraft is special and straight copies are morally wrong - but we wouldn't want to say no one could produce a computer/video game ever again - a game based on blocks, or with crafting, etc.. That would be silly. The same applies for Novels - Harry Potter is not the first or last wizard/witch at school book, and we wouldn't want that. But for functional software the gap is even narrower between the original and some other version - and this is how it should be.

I also occasionally release open-source/free software, but I know lots of people won't be able to get paid that way - and software takes a long time to write, document, finish and release.  Therefore I don't think free and open-source software can fill all gaps for everyone and every-company all of the time. But there needs to be some way of getting the middle ground accepted. Microsoft tried to do this - but people don't trust them and the software they tried it on could have easily been Open Sourced properly, in my opinion - or at least had lass onerous restrictions.

I'm not sure I know what the solution is. But I can see the problem.

Forlorn Fox 0.03 Windows and Linux

I'd like to say thanks to Tony for sorting out the Linux and Windows builds of Forlorn Fox. It's taken a lot longer than expected... we sort of thought it would be easy ...

Oh well... It can only get easier ... right?

http://redgameworks.com

Forlorn Fox - 0.03 dev test version release

So, myself Tony and Kenny have been working on a multiplayer, plan view, simple graphics game on-and-off for most of 2014. We are all very part time, but it has a few interesting features.

We finally got our act together and are releasing a very early test version.

There is not much there, but there is:

• Networking multi-player, with a LAN-based game finder
• A graphics editor
• Some rooms
• Selectable player characters
• A simple coin-rush sub-game
• A marginally scary 'eye-beast'
• All game code in Lua
• C++ engine
• Linux/Mac/Windows (at the moment)

Currently it will be hosted on my personal web pages at robprobin.com, but you can go immediately there via redgameworks.com. The files should be up today or tomorrow.
 

Halloween 2014

So, I went to switch my LED Display Light box from the Christmas display to the Halloween display. A simple job you might think. Of course, I ended up with a bunch of problems. For some reason switching the display always causes problems - mostly because I'm always doing it last minute. I wrote myself detailed instructions last year to help speed the process, but that didn't help.

A simplified list of what happened:

• To reprogram the LED Display light box I need a USB cable ... of course I can't find a USB A to USB B full size cable (my normal one used for this is cool - it lights up orange! Something like this). So into the loft/attic to find one in the large stash of random (mostly useless) cables. 
• For some reason CoolTerm wouldn't allow me to change RTS?!?! What?!?! On the embedded board (which uses an NXP LPC processor) DTR is used to drive the reset pin and the RTS is used to select programming mode or normal run mode). 
• Try downloading the new version of CoolTerm (1.4.4 rather than 1.3.0). Doesn't help.
• Try two other machines (with 10.8 and 10.9 on them) - they don't recognise the device at all.
• Start to suspect driver on back Yosemite ... I'm not sure whether it's a built in driver or the FTDI driver (the board has a USB-Serial chip from FTDI on it). Downoad the FTDI VCP driver. Doesn't help.
• Ok, so back onto the my wife's machine. Install the FTDI driver. Install CoolTerm. It recognises it the LED Display and RTS works! (So it sounds like it might be a problem with RTS and Yosemite (Mac OS X 10.10), assuming the FTDI driver installed ok).
• Haven't got the Forth code on this machine. Off to my public GitHub area to download it.
• Start to follow the instructions. Realise the command to erase isn't listed in my instructions. DOH! Go to the BV511 v1 Forth board instructions. So the command is 'new'.
• After a couple of aborted copy/pastes (including convincing my machine that '.fth' files really are text.) I finally get the two big files onto the board. Does it work? Of course not.
• I have a vague recollection of some problem to do with the port initialisation code. Of course, I don't remember exactly why or what, but the instruction hint at loading the second file in two halves, saving and testing after the first half. It might be that the Halloween code wasn't as mature as the Christmas code (something I never got around to fixing).
• I try this ...and initially the code turns on / off all LEDs as expected. But not after a reboot. I try loading the rest of the code, and it doesn't work. I think maybe setting the top level command as the word to run on boot and rebooting will help (desperation? maybe). To do this you use 'save word-name', all the saves write from on-chip RAM to on-chip Flash. Something goes wrong during the save and it locks. Arrggg... Reboot ... Nope, the Forth engine doesn't come up with a command line.
• Ok, back to my machine to reflash the chip. Of course, my lpc21isp programmer doesn't work because it was an ancient built. Get the latest code off SourceForge and compile it (by just typing make).
• Try running the programmer with the basic BV511 hex file ... but it doesn't program - probably because the RTS control isn't working. 

At this point, it's time for an early tea before we take the kids out in costume.

There are three options going forward.

• Reflash the board with the Forth firmware and just load the Forth code on.
• Do the above but merge the code so we can switch between Halloween and Christmas with a single high-level file.
• Switch the control board (the Forth-based BV511) for a Linux board - like a Raspberry Pi. There might be interesting I/O questions, and it isn't as quick a fix. However, it's likely to be much easier to update - and we can do it wirelessly via SSH.

Sounds like there might be more of this story in the future.

Part 2 of How to disassemble Sinclair ZX Spectrum Games

In part 1 of this series we covered the different data types and a summary of some legal implications.

In this part we cover creating a disassembly of the Z80 machine code. We will use a disassembler to create the assembler mnemonics - the human readable code version of the number that the Z80 processor interprets. We are making an assumption that most of the game is in Z80 machine code, rather than in BASIC or some other high level language. This is mostly true for commercial games but not all. We’ll look at the BASIC loader later.

Games not written Z80 machine code or Sinclair BASIC would need an interpreter written in Z80 machine code anyway, so this step is not lost.

If you want a game to disassemble (and haven’t chosen already) then Gulpman might be a reasonable choice because I’ve already done a full disassembly over here, which you can compare your progress with. You can get Gulpman from World of Spectrum but you’ll need to use an emulator to convert this info an SNA file to use with the tools in this article.

File Formats

Tape formats (tax and tap) are really no good because they don’t contain the memory contents. You could convert them to an memory image (or ’snap shot file) using a ZX Spectrum emulator.

Two example ‘snap shot’ formats are .sna http://www.worldofspectrum.org/faq/reference/formats.htm#File and .z80 http://www.worldofspectrum.org/faq/reference/z80format.htm. This are documented on the great world of Spectrum site. For a 48K Spectrum, both have a header followed by data. The only ‘problem’ with the .z80 format is that the data is compressed. We would need to ‘uncompress’ the data before we use it. If you want to do this, the format is well documented and you could write a simple program to uncompress the data.

A raw memory dump of the RAM area (from 0x4000 to 0xFFFF) would also be acceptable. (You could also do a raw memory dump of the ROM could be used from 0x0000 to 0x4000 - however, there are plenty of annotated disassemblies of the standard ZX Spectrum ROM on the Internet, including a copy of the book that was published in the 80’s).

Therefore I’ll assume you are using .sna format below. The utilities I’ll provide with this series of articles assume that format at the moment.

A Disassembler

You will need a disassembler to convert the Z80 machine code into assembler mnemonics - you can find several on the net.

The one I used is here http://robprobin.com/pmwiki.php?n=Main.ZXSpectrumTools - and is modified by me. (Please respect the wishes of the original author as per the top of the file).

This disassembler ASSUMES .sna format (it’s hard coded) when calculating the address of the instructions (and making the labels for the disassembly), however, it would be easy to change this assumption.

You will need to compile that with a C++ compiler. For example, at the terminal prompt / command line type (assuming you have GCC installed) you could tun the following commands:

For Windows:

    g++ Z80DIS_rob.cpp -o Z80DIS_rob.exe

For Linux or Mac:

    g++ Z80DIS_rob.cpp -o Z80DIS_rob


Once compiled, you can use it against the snapshot you are trying to disassemble:

On Windows:
    Z80DIS_rob game.sna > game_disassembly.txt

On Linux or Mac:
    ./Z80DIS_rob game.sna > game_disassembly.txt

Output

The file created (game_disassembly.txt in the above case), has both data and code disassembled into instructions. This has several disadvantages:

    1.    The header at the start will be disassembled. Since this is data not code, the instructions here will be not be valid.
    2.    Any data in the game will be disassembled. Again - instructions here will be rubbish.
    3.    Sometimes the instructions generated will ‘consume’ the first bytes of valid code.

Especially in the last case, some remedial work is required, manually editing the instructions that this point.

For example:

For a hex dump:

3E 3E 00

Would be disassembled as:

LD A, #3EH
NOP

But if we know the first byte is never executed and is in fact data, then the disassembly looks like:;

DB 3EH
LD A, #00H

Unfortunately, often, You only find this out as you start to understand how the code works. Have your Z80 instruction reference handy! (see last section for web references)


In the next parts we will look at an example disassembly in more detail and also at extracting text strings, hex for data and graphics.

I hate random crashing, especially Seg Faults :-(

I hate seg faults. Luckily this one was easy enough to track down.

Basically we had migrated a bunch of debug calls to a unified interface (so that things like Android targets could use specific calls rather than the generic mixture of std::cout and printf which go nowhere(?)). Printf-style formatting was useful, which lead us to using vsprintf.

Only problem: buffer was a fixed 256 bytes long. A really dislike C-style code for bugs like this. (Our project is meant to be 'pure C++').

When I introduced Lua stack traces, guess what get blown through?

So, initially, I allocated the buffer to be 20000 on the heap (rather than the stack). This, of course, is a hack that might come back to bite me.

So I looked at vsnprintf ... which is supported on a bunch of targets - Microsoft and Apple mention it. So does the Linux documentation. Also C++11 and C99. Don't quote me on this however. Also - do all targets implementations zero terminate corrrectly if they run out of buffer space?

Really, what I want is a self-allocating version. Allocating a huge number - even with a limited function is sort of rubbish. vasprintf - which is supplied by glibc/Linux/Unix/MacOSX looks like the way to go. Problem is which platforms is this supported on? And does this mean we will end up wasting time by implementing our own. There are a couple of forums where people have given ideas on this - but still, it's a couple of hours finishing and debugging that is nothing to do with actually writing your application.

I feel like I've spent enough time on this already - time where I could have been working on the actual game. All to avoid a stupid inconvient seg fault - now or in the future. Which is basically due to the naive C string container.