zxsp 0.8.pre27: Multiface ONE, 128 and +3

Hello,

Three new members in the family of supported extensions: The Multiface ONE, 128 and +3 from Romantic Robots.

This time i scrutinized the circuits into depth and therefore the emulation should be near perfect. :-) There were different roms used in the MF1, evtl. also in the MF128, i don't know – i have included the most common.

The Multiface ONE had a Kempston-style joystick port which could be disabled by cutting a wire, in case another device also used this port. You can disable this port in zxsp too. If you attached the Kempston interface (e.g. you have enabled the auto-attach option in the preferences) and then at some time attach the Multiface ONE, then disabling this port is not strictly required, because an already attached USB joystick will happen to be magically plugged in both interfaces, and then both devices will react the same when a program reads a byte from the Kempston port. But disabling this port in the MF1 is probably more intuitive and leads to less unexpected misbehavior.

While looking at the circuits i noticed some interesting points:

The Multifaces do only react to an NMI generated by the own red button, not to any NMI.

The presence of the Multiface ONE could be detected by the running program if it paged in the MF1 rom, and probably at some point games started to test for it and complained if they detected this device.

The presence of the Multiface 128 could no longer be detected by the running software; Romantic Robots had added a flag which stored the visibility state of the interface, i called it the camouflage flag. Same time they wrote "PIRACY IS ILLEGAL! MULTIFACE 128 is NOT designed to encourage piracy!" into their manual – Words and actions are not always the same.

The Multiface +3 was unusable if a program mapped ram to address $0000. Romantic Robots silently did not mention this in their manual. But the circuit of the MF+3 is very clear: It has a flag which monitors the "special mode" bit written to the MMU of the Spectrum +3 and, if set, ignores any press of the button. The reason is quite clear: Amstrad had reworked the ZX Spectrum bus connector of the +3 and added a second line to disable the internal rom at address $0000 with highest comfort, but no possibility to disable ram at this address. Yes, the Spectrum was not their most beloved child…

zxsp displays the state of the most interesting flags in the Multiface, which is, for the MF1, whether an NMI is pending (and the NMI routine very likely already has been executed and any more pressing this button has no effect), for the MF128 whether it is visible and for the +3 whether the ZX Spectrum MMU was set to "special mode" which means ram at $0000 and a disabled Multiface.

The red button of all Multifaces is gated by the output of the NMI pending flip flop which is set by this button: If the button set the FF, then the FF's output immediately disabled the button. To my best understanding this is unnecessary and it does not work:

• I believe it was meant to immediately stop the signal from the button, so that recursive NMI processing in the CPU could not happen. But the NMI is edge triggered: once active, it can be held active for years without triggering another NMI.
• Once the FF was reset (by software) this disable signal disappeared and the FF was immediately set again, if the button was still pressed.

The last point might be tested with the real device if you press the red button and keep it pressed while you type 'r' on the keyboard to exit the Multiface menu. Then the Multiface should be immediately entered again. But it's not sure whether it will, because for this the NMI must go away for a long enough time so another edge can be detected by the Z80 CPU, but it only goes away for a tiny moment, just the round-trip time through one gate and the flip flop. Note: this is not reproduced in zxsp.

So, to all of you who owned a Multiface: Test your skills and revive memories. ;-).

     ... Kio !

zxsp 0.8.pre26: hE(ll)(oo) k(ee)(OO)

Got it?

"Hello Kio !"

Yeah, my Spectrum speaks.

Inspector for the Currah µSpeech digital speech unit

Thanks to the work of Joe Zbiciak and Frank Palazzolo, who had reverse engineered the SP0256 speech processor, i was able to implement an emulation of this chip and then of the Currah µSpeech interface in zxsp.
The SP0256-AL2 contains a cascade of six 2-pole filters, a rom with filter coefficients for British/American language and a micro sequencer, which feeds these values into the filter coefficients. All digital. The output is an 8 bit PWM.

Despite of the very comprehensive informations about the SP0256 chip on their website there were still some bits not clear to me. I filled in the gaps by own tests and looking at what other people did. (not too many of them…) And i emulated the double RC filter after the PWM output. Yeah!

I added an option to switch between the original 8 bit sound and the unrestricted approx. 14 bit output, but i can't hear any difference. :-)

At the bottom of the inspector widget i added a scroller which displays the spoken allophones.

Though most allophones are spoken as expected, i believe some are emulated wrong, e.g. "1" and "Y" sound a little bit like a "bass bump" and "?", spoken as "kw(ee)r(ee)", sounds more like "kneenee". But there is little i can do about this.

I found that there are three pairs of allophone tokens in the µSpeech unit which refer to the same allophone: "(ck)" and "(gg)" which refer to KK2, "(oo)" and "(eau)" for OW and "(aa)" and "(ay)" for allophone EY. On the other side, tokens for two allophones are missing: DH2 and GG2. Given the fact, that "CLEAR" and "CLS" are both spoken as "CLEAR", the µSpeech seemed to be released a little bit fast these days.

Technically the Currah µSpeech isn't a Z80 peripheral. It's more like a 6502 add-on. It uses memory-mapped i/o! I had to extend my Z80 emulation to handle that sort of things…

Final problem was the rom switching for this little beast: It toggle's between it's 2k rom and the ZX Spectrum's internal rom when a RST7, that is, when the regular timer interrupt is executed. Actually, the way i handled rom switching up to now could not handle this: It detected an opcode fetch at address 0x0038 in one rom, switched the rom and restarted the CPU at this address. Then it detected an opcode fetch at address 0x0038 in the other rom, switched the rom and restarted the CPU at this address. And so forth.


A final word about the Blog Editor: It's getting worse every day i write my next post. It's really so bad now, that i write my blog post in a text editor on my Mac and copy&paste it into this editor.

zxsp 0.8.0pre25: 4.75 cm/s

Hello,

long time not seen…

This release brings bug fixes, improved tape handling and the ability to load C sources.

Fixed bugs

(some are only intermediate bugs, not actually present in pre24)

• Kempston mouse interface: Mouse grabbing & tracking did no longer work in Qt 5.x
• Investigated and fixed audio interrupt drop-outs (thanks, Apple!)
• Investigated and fixed paintEvent() overpainting child widgets in Qt 5.4 (thanks, Qt!)
• Investigated regular unusually long interrupt runtime: WLAN-internet bridge! (thanks, Apple!)
• Instand loading crashed at end of tape if tape was "running"
• Inves could not load from tape recorder
• Crash when accessing beyond allocated tracks of floppy, e.g. on empty side
• Save tape utterly broken, crashed in CSW buffer decoding

Tape recorder

Load and save tape is now fully supported in all models.
Spectrum-type machines can load and save .tap, and .tzx.
ZX80 machines can load and save .o, .80 and .tzx.
ZX81 machines can load and save .p, .81, .p81 and .tzx.
The Jupiter Ace can load and save .tap and .tzx.

Additionally all models can load from real audio files like .wav and .aiff, but saving to audio file is not yet supported. E.g. you can insert a .wav file in the tape recorder and choose "save as" from the context menu and save it to a .tzx file.

All models can instant-load and instant-save tapes, if the rom routines are used. Loading tape files like a snapshot, e.g. by double clicking or by reload recent file, will set the tape recorder to instant-load, so, unless custom loading routines are used, loading a tape file is instantaneous.

While playing tapes now the typical sound is audible, which was missing for a while.

Record and play back from your Mac's audio-in and audio-out device is not yet fully supported. For recording i recommend to record the audio file with Audacity, but i have not yet tested whether it suffers from dropped audio interrupts, as zxsp did and, as far i can tell, even iTunes does (for playing). For playing the tape recorder of any machine can be used, though the audio level is currently a little bit low, because i didn't want to annoy you too much every time you load a tape. :-) Of course for playing real audio files any audio program can be used as well. ;-)

Recording your old tapes into .tap or .tzx files is now very easy: Just insert the audio file into the tape recorder and choose "save as" from the context menu to save it in another format.

Load assembler and C source files

Assembler source files are assembled using a built-in version of zasm, and the result, whatever it is, is loaded. zasm supports .sna, .z80, .o, .p and .tap files.

zxsp comes with a built-in executable of sdcc, a multi-target C compiler. Take a look at the zasm manual to see how to include C source files.

zxsp 0.8.0pre24: OMG!

Hello,

this is a bugfix release. A bugfix release was planned anyway, but suddenly it became very urgent, for two reasons:

1. dyndns.com will stop their free DDNS service.
2. a severe bug in zxsp made it crash when a usb joystick was attached.

New Domain

Please bookmark this new domain for all zxsp related web resources:

• k1.spdns.de

The links in the zxsp.zip are also updated, yo you may use them.
The URL of the blog does not change! :-)

Bug Fixes

• Scanning for USB joysticks crashed, eventually right from the start
• Reloading a HD image in a machine which can't attach the DivIDE interface crashed
• Loading two or more files simultaneously crashed if they a loaded different models

• Visualisation of compound keys on ZX128 and later did no longer animate the compound keys
• Joysticks in TS2068 were swapped, fire button bit was wrong (bugger, WoS!)
• Opening joystick inspector could reassign USB joysticks

Misc.

I have slightly modified the key mapping: The ESC key now generates the typical BREAK key(s) of the current machine: mostly SPACE + CAPS SHIFT on ZX Spectrum models, simply SPACE on ZX81 models. Activating extended mode now requires SYMBOL SHIFT + CAPS SHIFT to be pressed (as on real model) where i suggest to use CTRL + ALT on the real keyboard because the are adjacent.

final note:

OMG!

Why does nobody report when zxsp crashes? Bugs which are reported are fixed much faster, really!

    ... Kio !

P.S.:

Oh, i forgot the image: Right now i'm listening to:

You Are Adult.tap

zxsp 0.8.0pre23: DivIDE Hard Disc Interface

Hello everbody,

good news: this is no April's fool. zxsp 0.8.0pre23 has just been released.

New in this version: 

Emulation of the DivIDE hard disc interface.
Some bugs eliminated.
Some small improvements.
Lots of new bugs. :-/

Due to sucking Qt this version of zxsp is still linked against Qt 5.11; not 5.2, not 5.2.1 and not 5.3beta. Please visit QTBUG-37706 and vote it up.

And if you are on the Qt bug tracker site, vote this up as well: QTBUG-37918. And this one: QTBUG-33530. And this one: QTBUG-34160. And this one: QTCREATORBUG-10276. Just to make a starting point...

Open File Requester

Qt fails to open the file requester box in the requested directory for quite a few versions now. I have finally added a work-around.

Qt under certain conditions returns a directory though only files were requested. I have added a catch.

Debugger Disassembler View

Fixed a problem where the disassembler was not informed about memory configuration changes and would crash when reading from old memory.

Fixed colouring of assembler instruction at the CPU's programme counter.

Fixed that setting r/w/x breakpoints in the disassembler view corrupted the memory byte.

Added requested feature: Display in disassembler view now follows the PC while debugging and slow motion of the CPU (≤1000Hz).

Release Key "A" Error

When key "A" and a modifier key is down, if you release the modifier also released key "A" though it was still down. This is fixed in zxsp linked against Qt 5.3, but in this version, which is linked against Qt 5.11, it is replaced by some other wrong behaviour... Luckily enough this is pretty rare and only affects key "A" because the key code of this key is 0x00.

DivIDE Compact Flash and IDE Hard Disc Interface

This is the great new addition: support of the DivIDE interface. The interface can be added to most machines with a ZXSP system bus. zxsp remembers the recently used rom and hard disc file. Initially it defaults to esxDOS 0.8.5. From the context menu of the DivIDE inspector you can create new hard disc files which are preset with SYS/ and BIN/ folders for esxDOS 0.8.5. From my web site you can download hard disc image files for some different dos/versions with a bunch of demos and some games. Don't expect them all to work, at least not on every model. B-)

The yellow jumper: If you own a DivIDE interface you'll know it: it enables the rom (actually the automatic rom paging on certain hook addresses) and disables writing to the rom. If it is removed you can load a .tap file (or similar) and write it to the rom. I suggest to save the rom thereafter or next time you'll have to load it from .tap again.

The push button: This is the NMI button. if you click on it, it should start the waiting dos on your DivIDE interface which will show the hard disc's contents and provide some other functions as well. You can load quite a lot of different types of files in esxDOS: .tap, .scr, .sna, .z80, .trd and maybe others. But many files don't work / on every model. They don't work on the real thing as well (mostly, probably).

The hard disc image files: zxsp knows .img, .dmg, .iso and .hdf files. It does not support all variants of the .img and .dmg files, actually only the very basic uncompressed formats. You can mount them in OSX as well and fill in files. Then unmount them and insert them into the DivIDE interface. Note that the DivIDE interface can write to the disc image and eventually corrupt it, though this should not happen. From the DivIDE inspector context menu you can write protect the disc or you can write protect the image file itself in OSX. Creating a disc file which works with fatware is very tricky in OSX. Actually i don't know how to do it. But phoenix^ra supplied me with a .hdf image which works with fatware and i have made a .img file from it. You can download it from my web site. See link above.

Known and unknown Bugs

The current release of zxsp has some known bugs at the moment of release (as opposed to bugs known after releasing ;-) )

TS2068 emulation has some problems: it frequently crashes in Basic, e.g. if you try to save or load a file. Timex Command Cartridges (dock roms) seemingly work fine. If someone has an idea why Basic crashes every now and then i'll be thankful to know. I'm stuck.

Writing to tape probably only works if you enable instand load/save and write using the standard rom routines only.

There's still a lot on the TODO list.

    ... Kio !

zxsp 0.8.0pre22: Timex Sinclair 2068 family; Updated

Hello everybody,

and on it goes: 3 new models added! Ok, basically it's only one model which comes in 3 flavours:

The Timex Sinclair 2068 Family

• The Timex Sinclair 2068 (USA)
• The Timex Computer 2068 (Portugal)
• The unipolbrit 2086 (Poland)

Technically they are very similar, not to say identical, but there are minor differences:
The US version runs with 60 Hz, the European variants with 50 Hz frame rate.
The Polish version had only one joystick port, the right port was replaced by a parallel "Centromix" port. But, *wonder*wonder*, *my* U2086 had 2 joystick ports! So currently the U2086 is also emulated with 2 joystick ports and no parallel printer port. I currently don't know of any software using the printer port anyway. Even the precise usage of some control pins are still discussed.

TCC dock inspector

These computers had a Rom module port, the so called Timex Command Cartridge dock. These are stored on .dck files nowadays and i have included some samples in this distribution. Only a very limited number of titles were ever produced by Timex – Androids, Budgeter, Casino1, Crazy Bugs, Flight Simulator, Pinball, States And Capitals, and Penetrator – but there are also some Sinclair Interface II adaptations and some emulators for various variants of the ZX Spectrum, especially the ZX Spectrum emulator itself. This was even bundled with every U2086 computer AFAIK.

Note: if someone has games or demos which use the AY sound chip and joysticks of these machines could he please be so kind and send me a copy? Thanks in advance!

Some people say that the "86" in unipolbrit 2086 is not a spelling error but a pun on 8086 or an indication of the year of release – but then "Centromix" is probably no spelling error either... B-)

Update: Instant load tape files is currently broken on these machines. You must disable instant tape loading and load them in realtime. This will be fixed in the next release.

Other Changes

I have added recent-files menus to the SPECTRA and Sinclair Interface II inspector as well to the Dock inspector. Inserting a rom now also adds it to the global "recent files" list as well so that CMD-R reloads it.
The TK95 got a rear-side joystick connector image and a real photo of the keyboard – not sure whether this is an improvement thou.

zxsp 0.8.0pre21: misc. old stuff

Hi Folks,

bad news: i can't make up my mind for a picture. A blog post without a photo? – wait – there will be one, at the end of the post, maybe. :-)

New in this version:

misc. old stuff

"misc." consists of the following:

Video Beam Indicator

Yeah! Finally, the video beam indicator is back again. Right now only for the color machines, but the b&w machines know exactly where the beam currently is anyway. ;-)

The video beam indicator is a red&yellow blinking cursor at the current position of the video beam. It is activated when the cpu clock is throttled down to 100 kHz or less or when you single step in the debugger. This is an aid for anybody who wants to write a demo with some critical timing. For those who only want to play games it's a "nice to have" for curiosity. B-)

Load assembler source

Now it's possible (again) to load snapshots and tapes from assembler source. The source will be compiled by the built-in zasm assembler and the resulting snapshot will be loaded as if it was initially selected to load. The source file is stored in the "recent files" list so that reloading it is easy.

It's not trivial to get the headers of .sna or .z80 files right, so i have included template files in this distribution.

Supported target file types for assembler sources are .sna, .z80, .ace, .rom and .tap.

Load .rom and .scr files

Also, it is now possible again to load .rom and .scr files. 

Load .rom files: Based on the rom size, the currently running machine and the default machine in the preferences, the application choses the best fitting machine and uses this to load the rom file. If am Interface 2 or a SPECTRA interface is attached, then 16K roms will load as a rom cartridge. Else they will replace the built-in rom.

Load scr files: If the current machine is a color machine, then the .scr screenshot is loaded into the currently displayed video ram of that machine. Else, based on the default machine in the preferences, a color machine is chosen and started, but probably the screenshot will not be displayed very long in this case. 

Load .tap and .dsk files

Tapes and disks can now be loaded like a snapshot. zxsp choses the best fitting machine based on the file type, the currently running machine and the default model in the preferences. Note: a tape or disk normally does not contain any easily retrievable information about the required machine, so you should select the desired machine before loading a tape or disk.

Load .dsk files: That's fairly easy, as these disk files can only be loaded into a ZX Spectrum +3. zxsp just tries to select the British or Spanish version based on the currently running machine and the default model, if the currently running machine is not a +3.

Load .tap files: These tape files can only be loaded into a color model or into the Jupiter Ace. zxsp tries to detect whether this is a Jupiter Ace tape and loads it into a Jupiter Ace, else it uses the currently running machine, if it is a color model, else it tries to determine the best model based on the default model, starts it and loads the .tap file. Most .tap files can be instant-loaded entirely, giving the impression of loading a snapshot. If the program uses it's own tape loader code, instant loading does not work and you have to wait until the tape is loaded. You may open the tape recorder window (if it isn't opened by default) and wind back and fore if required.

Load .tzx files: These tapes may be loaded into every model. zxsp can't tell beforehand which might be the best fitting one. So it loads a .tzx file always into the current machine. You should always start the desired machine before loading a .tzx file. Frequently .tzx files contain programs which use custom loaders. Then instant loading will not work and you have to wait for the game loading as normal.

Loading a tape like a snapshot vs. inserting a tape into the tape recorder
Tapes and disks are no snapshots. They contain the data of their respective represented media. When you double-click or open a tape or disk file with "open file", then zxsp secretly loads a snapshot which was saved at a point where it started to load a program from tape or disk and inserts the requested tape or disk into the respective drive. Additionally, it sets the tape recorder to instant-load. The result is, that it looks like zxsp is loading the tape or disk like a snapshot, though you now know it isn't.

This is a convenience feature, though. Normally you should launch the desired model, insert the tape into the tape recorder (or the disk into the disk drive) and type LOAD "" or similar. Then you get back more of the original feeling. You can select instant-load and auto-start-stop the tape here as well.

Load .tap files into the Jupiter Ace
You may not know it, but the Jupiter Ace had no load tape command with wildcard and it had no auto-start feature. So zxsp takes some effort to supply the tape loader snapshot with the required name. But it does not know which command is required to start the game or to load subsequent data blocks. Once the first dictionary is loaded you are alone. You may open the tape recorder and wind the tape fore and back to see the names of following blocks, but you'll have to pick the info on how to load them and how to start the game from it's documentation.

Display parameters for the b&w Ulas

I reworked the display of video parameters in the Ula inspector for all models, but especially the b&w models, which up to now just showed some default values collected from various sources in the net. They now show the actually used display parameters, though some, most notably the cpu cycles per scan line, still just show the nominal values.

Select 50 Hz or 60 Hz model for the TK clones

I have added a 50 Hz / 60 Hz switch to the Brazilian TK models. I'm not sure about the cpu clock for the color 50 Hz models. If someone owns a TK85, TK90X or TK95 could he please determine the model (50 vs 60) and tell me the quartz frequencies? 

Fixes:

Fixed a problem with the .ace files saved by zxsp. 

The two final bytes were missing and made them not load. :-|

Fixed spurious crash when switching between models.

This was annoying me for a long time now and i really thought it was Qt to blame. But it was my fault and the error only popped up at an unexpected place, misleading me for quite a while. Under certain race conditions a mutex was released twice and then another thread waiting for this thread to terminate crashed in it's wait position, somewhere deep in the library function. :-/ 

Actually this fix made me issue version pre21 today.

Image

An image, an image ...

.ace sample program loaded from assembler source

animated gif file made with zxsp

zxsp 0.8.0pre19: New stuff for Jupiter Ace, Sound Fx and TS2020 tape recorder

Moin,

Timex Sinclair TS2020

a new version of zxsp is out now. There is no 'big' thing as in the last release, but several minor improvements and some 'less important' additions.

Jupiter Ace

Some errors in handling .tap files for Jupiter Ace have been eliminated. .tap files can be 'instant loaded' now. .ace snapshots can be loaded and saved. I have included 'WORMS' from the Jupiter Cantab demonstration tape in this release. A tiny game, just to show it works. To load it, type load worms get graphics game. There is no wildcard load and no auto-start on the Jupiter Ace!

Timex Sinclair TS2020

The b&w machines now use a TS2020 for their tape recorder. Not sure whether i should also use it for all other machines as well, except for those with built-in tape recorder. What's your opinion?
As with the +2 / +2A recorder, the pause button is used in conjunction with wind and rewind to auto-stop at block boundaries.

All tape recorders

I have added sound effects to the tape recorders. I hope you like it. :-)
I have also added new menu entries to the context menu, including a 'recent tapes...' menu.

New Qt version

This is the first release which uses Qt 5.x. This enables me to send bug reports for the current version. B-) The only visible difference (to my knowing) is the superfluous animation when starting/leaving full screen mode.

Have fun!

zxsp 0.8.0pre18: +3 Floppy Disc Drive support (updated)

zxsp 0.8.0pre18 is out now!

New supported machines in zxsp:

ZX Spectrum +3 English and Spanish version.

ZX Spectrum +3 disc drive

The NEC µPD765 emulation seems to work good, but sometimes i think zxsp reacts differently than the real machine. There is still plenty left for testing. The FDC765 is not very well documented. There are plenty of data sheets for the various clones* around, but somehow they have all just copied NECs data sheet, more or less.

*) The FDC in my +3 is a Zilog Z0765A.

Supported file format is currently .dsk only, others will follow in the next release.

The disc drive is the first item with sound effects! :-) I had a hard time making recordings of the various sounds. I have at least 10 microphones at home (telephone, cellphone, iPad, iMac, even ZX Spectrum recording hardware) but nothing really good or in working order. I finally resorted to the recordings i made with my iPad except for the stepping noise, which is 'hand crafted'.

In the floppy disc controller window you can insert and eject the disc by clicking on the slot or the eject button. The context menu provides options for creation of new discs and access to recently used discs.

Discs can be flipped side B up by clicking on the "B" label. You probably have to format most discs on this side before you can use it. This is done in Basic with:

format "A:"

where "A:" is the drive, not the side. ;-)

Discs can and probably should mostly be write protected. This can be checked and toggled in the context menu, when a disc is "inserted" or "ejected". (The "ejected" state is as shown in the above image.) The write protection state is simply the write protection state of the disc file.

If you double-click a .dsk file it will mostly open in a ZX Spectrum +3, except if your default machine or the currently open machine is a Spanish +3.

Please test the +3 disc emulation and report any problems. (or whether it works for you.)

    ... Kio !

backup daemon for OSX (and maybe other Unixes) – updated

Hi2all,

project of this month was a backup daemon. It's just a console application, no GUI, and meant to run whenever my Mac is on. And it is always on. ;-)

Whenever it comes to backups i have a bad feeling: what if i lose everything stored at home? Maybe fire, water is a little bit impossible here, or burglary? A thief won't stop at my Mac but also take the external hard drive and probably all memory sticks as well. And if i hide them very well then it's likely that accessing them is so awkward, that i only update them every now and then, and in case i need it it's probably several months old.

In the domestic most valued computer magazine c't was an article about how to solve this with BoxCrypter and BitTorrent Sync. BoxCrypter to encrypt you files and BT Sync to upload it to a friend, who does the same vice versa.

I tried it with a friend and found this:

If you want to encrypt more than one directory (with it's sub directories) or if you want to encrypt file names as well, you must buy BoxCrypter. Then it provides you with one or more virtual drives which you can use like normal drives, and stores the encrypted data in a directory bundle, which you can backup with BT Sync. But these virtual drives are no full replacements for standard drives, OSX complains here or there about not supported features. And i don't know what happens if you put your user directory into a BoxCrypter volume...

BitTorrent Sync seems to work reliably but it is slow. For unknown reasons it only transmits in bursts and thus uses only a fraction of the already not-so-high upload bandwidth.

After looking at this for two weeks i decided to actually write my own backup daemon.

Kio's backup daemon

The program can be found here:  kio's backup daemon.

A sample config file can be found at  ~/.backup_daemon/config.txt.

If you give it a try please report your experience.

It is now feature-complete and most bugs are eliminated so i think i can offer it for testing to others. If you feel awkward when you think of your backup and the 'worst case', then it may well be worth a try.

The current features are:

• console application, no GUI. You can make it auto-start whenever you login and hide it in the dock.
• Backup: synchronize a remote directory with your local directory in regular intervals.
• File transfer: maintain 'push' and 'pull' folders for automatic transfer of files between you to your friend(s).
• Encrypted connection, backups are stored encrypted, file and directory names are encrypted as well.
• Upload speed limiter.
• Daily snapshots of backups, except if hard links are not supported. (mostly NAS)
• Include and exclude files or subdirectories from uploading.
• Upload to a peer over the internet
• Synchronize a backup folder locally, e.g. a DropBox folder, NAS or external drive.

You need (except for pure local usage):

• a domain name. Use any dynamic name service you like.
• an open port, forwarded by your router, if any.
• a friend

Does not work with:

• WebDAV shares.

There will be sporadic updates in the next months.

Behavior of sync type "backup"

In this mode the daemon tries to synchronize a local directory with a remote directory. Files which vanished remotely are deleted locally too, modified files or new files are downloaded. Once per day a snapshot is created which contains hard-linked files to other snapshots and the current backup, therefore using only very little additional space on the disk. Files which vanished are moved to the last snapshot if no previous copy exists.

This works well with local disks, but not so good with Network Attached Storage 'NAS', because within the last ten years Apple did not find the time to add a 'hard link' command to the Apple File Protocol 'AFP'. So basically, this does not work with NAS. In this case the backup daemon still creates snapshot directories, but they will only contain the deleted files. Just in case that you need to recreate them.

You can circumvent this problem in the same way as Time Machine does: With Apple's DiskUtility create a SparseImage on the NAS and mount this image locally. Determine the mount point and use this path for your backup destination.

Currently 'backup' folders are polled every 12 hours.

Behavior of sync type "push" / "pull"

In this mode the daemon downloads all new files i finds in the remote folder. Once download is complete, the file is added to a done list which is stored in "~/.backup_daemon/NAME.done".
So you can remove the file from the receiving folder without triggering a second download of this file.
Files which vanish remotely are not deleted locally, so the sender can remove his files at some point as well.

Currently 'pull' folders are polled ever 30 minutes.

Command line arguments

While the daemon can be started just 'as is', it may be started with one of two command arguments as well:

's' (single character 's'): Only the server is started.
'c' (single character 'c'): Only the clients are started.

This can be used to start 2 instances which can be debugged or stopped independently.

Settings in ~/.backup_daemon/config.txt

The backup daemon needs a config file, which must be stored at '~/.backup_daemon/config.txt'. (note: the location has changed.) The tilde '~' indicates your home directory.

All changes to the config file only take effect when the backup daemon starts. So if you update something here you must stop and restart the daemon. For an example also see ~/.backup_daemon/config.txt.

verbose : NUMBER
Defines how much log output will be produced.
Possible values are: 0 (nearly none) to 4 (each file transferred produces a log line)

upload_speed : NUMBER
Defines a speed limit for sent data. Setting this to 0 disables the limit.
The speed is set in bits per second and may be followed by a unit 'k' or 'M'.
For a typical ADSL connection with 1500 kbit/sec upload speed this might be set to 1000k.

num_clients : NUMBER
Defines how many client workers are created. 'clients' handle the receive side of a connection. 4 clients are recommended.

num_servers : NUMBER
Define how many servers are started at most. A server handles the send side of a connection. They are created whenever a client connect to this daemon. If more than N clients try to connect at the same time, only N connections are granted and the others are rejected and will retry later. There should be at least 10 servers allowed.

self : n=MY_NAME  : h=MY_ADDRESS  : p=MY_SERVER_PORT : secret="MY_LOGIN_SECRET"
Defines some settings concerning your own computer.
MY_NAME is your nickname. Any client connecting to you will check this name.
MY_ADDRESS is your static IP address or your static or dynamic server name. Remote clients must know this address to connect to your server.

MY_SERVER_PORT is the socket port which your server uses for incoming connections. Remote clients must know this port address and it must be forwarded in your router, if there is one between your computer and the internet line.

MY_LOGIN_SECRET this is the password which is exchanged and tested with a challenge - response test. Obviously any client must know this too.

The 'self' settings are used to create a 'peer' entry as well, so you can connect connect to yourself to make backups to a local disk.

peer : n=MY_NAME  : h=MY_ADDRESS  : p=MY_SERVER_PORT  : secret="MY_LOGIN_SECRET"

Define the same settings for any peer which is allowed to connect to your server. 

peer : *

Defines that any peer is allowed to connect to your server. But they will only have access to folders which are marked with 'p=*'.

push: n=MY_BACKUP : t=BACKUP : p=HIS_NAME : d="/MY/ROOT/DIR" : s="MY_FOLDER_PASSWORD" : x="EXCLUDED" : i="INCLUDED"

Define a folder, which is exported by your server.

MY_BACKUP is the nickname for this export. This must match a corresponding 'pull' entry in the remote clients' settings.

BACKUP is 'backup' for backup-style exports (directories which you want to backup to your friend's HD) or 'push' for a push folder which transmits anything put in here to your friend(s).

HIS_NAME nickname of your friend. This entry can appear multiple times: once for each friend which shall have access to this folder. If you add a p-setting with value '*' (one character) then this folder is exported to anybody who can connect to your server. For a distribution folder which shall be exported to anybody who knows about your server you must add a peer '*', see above.

MY/ROOT/DIR is the base path to your exported directory.

MY_FOLDER_PASSWORD is a password which is used for encryption. all files and file names are encrypted with this password. If you use this in a 'push' folder then all friends should use this password for decryption in their 'pull' jobs, or they'll only get the encrypted files which is probably useless. If you use this for a backup folder, then don't tell your friend your password. Then your files will be stored encrypted on your friend's HD. 

EXCLUDED contains a partial path of files which shall be excluded from the exported directory listing. Initially all files are included. Any file whose path starts with this string is excluded. Except if the string starts with '*' then this indicates a file type which will be excluded. Any amount of excluded file paths may be defined for an exported folder.

e.g.:

• x="" excludes all files
• x="." excludes all hidden files in the root level of this push folder.
• x="aa" excludes all files starting with "aa"
• x="a/bb" excludes all files starting with "bb" in folder "a"
• x="*.txt" excludes all files in all folders which end on ".txt"

INCLUDED contains a partial path of files which shall be included in the exported directory listing. Initially all files are included. Any file whose path starts with this string is excluded. Any amount of included file paths may be defined for an exported folder.

If entries for both included and excluded files match then the longer and more specific entry wins.

e.g.

• x="" i="Documents" excludes everything except (probably) folder "Documents"
• i="photos" x="photos/private" exports all 'photos' except (probably) folder 'photos/private'

pull: n=MY_BACKUP : t=BACKUP : p=HIS_NAME : d="/MY/ROOT/DIR" : s="MY_FOLDER_PASSWORD" : x="EXCLUDED" : i="INCLUDED"

This is essentially the same as for 'push' just for the client. If the server exports an encrypted directory and the client has no encryption password set on the pull job, then the directory tree is stored encrypted on his disk. The client may also filter what to download with 'included' and 'excluded' partial paths though normally this won't be used. It is only useful if you want to retrieve only a few files from your backup.

Restore from a backup

What to do when the worst case happens and you need your data back from your friend?

1. Stop your backup daemon and ask your friend to stop his backup daemon.
2. Ask your friend to replace the backup pull job by a push job with identical settings, except the path must be extended with "/current_backup".
3. Replace your backup push job by a pull job with identical settings, except you might want to change the destination path.
4. Start your backup daemon and ask your friend to do the same.

Modified scenario: You want to restore from a local disk, e.g. a NAS. Then "you" and "your friend" are the same person. ;-)

Modified scenario: You want to restore only certain files. Then add an appropriate 'exclude' and 'include' entry to the pull job.

Modified scenario: Your friend brings your backup on an external drive: Add a push job for the external drive to your config.txt as well and remove the line for the upload limit.

Hints

• Add the backup daemon as a start item to your login profile.
• You can test that restoring your backup will work if your friend also adds a push job for the backup folder and you a pull job to a temp directory, eventually with an include/exclude filter.

My Annual Rant II

Hi,

sometimes things don't work, it's not your fault and you cannot fix it. And sometimes this comes in close succession. So here is my 2nd annual rant for this year:

Chapter I

I'm working on a backup daemon, because i'd like to have a copy of important data outside of my home, just in case. As usual this takes some days until it works, i make local tests and fix bugs and try to move the software to a good state. Then tests with a connection over the internet; and finally this works too.

After that i thought of reasonable extensions to the daemon. 5 GB of webspace come to mind. They are mounted via WebDAV and can be accesses as any local file system. Nearly.

First i had to adjust the character set for encrypted file names until this seemed to be no longer the cause of transmission problems.

Then: symbolic links are not supported. Ok, it's mostly for files only anyway, so i accepted this.

Next is: i cannot set the modification date on files. I use this as a flag whether the file is fully synced. If mdate and file length match, then the file is not further scrutinized but deemed to be up to date. I modified the backup daemon to accept any mdate which is newer than the original file's mdate.

Next is: frequent errors, irregular with no pattern. Except: 9 of 10 errors are timeout errors. I work around to catch and retry in case of timeout and this seems to work. Running the daemon for some time updated more and more files to there final size. But some files failed over and over again.

Common pattern: long files. Approx. 15 files and they all were in range 10 to 50 MB. I examined the problem further: It seems that data written to a WebDAV file is not uploaded immediately, but only buffered internally. When the file is closed all data is uploaded. And, unbelievable, the close() system call has a timeout. It's generous: 1 minute. And then it fails with timeout error... and nothing is saved! So there's no real chance to upload long files this way.

I tried to enforce upload in chunks by calling fsync() after each chunk of data. What the WebDAV driver does here is: it uploads all the current file, not only the modified data, and now, after a long while, while it uploads the growing file again and again, it fails with timeout error when it reaches the critical file size. :-/

Here i gave up. WebDAV is not supported.

Chapter II

That was last week. This week i developed the idea, that i could make daily snapshots of the current backup and hard link the files to save space. After all, 'Time Machine' does it as well. And we are Unix, aren't we? After two days it worked, tested locally. What i got back from my friend was:

Client1: caught error: create hardlink "/Volumes/K/1/2013-07-15/,NSMHyr<1 {i>+_AbQ5Tc<89I3t0>Er''dGEb(xZNjJ{!#yiO4rmSE8_ILhE": Operation not supported (45)

(funny filenames, eh?)

Yes: create hard link not supported. He has a Mac like me and stores my backup on his NAS. The NAS is a Linux box and surely supports hard links. So it's the file transfer protocol: AFP. After 12 or 13 years of being unixoid Apple did not manage to implement hard links into their Apple File Protocol. This sucks! But how does 'Time Machine' do this? After all, they not only create hard linked files but also directories (according to internet sources) which is generally deprecated!

'Time Machine' creates a SparseImage filesystem file on the NAS and mounts this locally. Wow! That's what i call a workaround. So my friend can just do the same and never forget to mount this volume, or i discard the daily snapshot feature. But i don't want to drop it, so i rewrote the daemon to ignore the failure of creating a hard-linked snapshot of the current backup and just resume work. The risk is, that in case i need the backup it may just have been crippled to the state on my Mac before i can restore it... :-/

Chapter III

In my work i write apps for Android. Today i wrote a screen to input a bunch of dates and times. I didn't want the DatePicker and the TimePicker to popup over and over again, so the user should be able to enter date and time in text edit fields. There are specializations for these and similar cases, and i flagged the text fields as 'date' and 'time'. 'date' works. 'time' does not allow me to enter a colon. Actually, not any character besides '0' to '9'. Has Google ever tested this? There is no workaround or solution for this problem, so i used a nasty workaround and require the user to enter a period instead of a colon... :-/

And here i felt it's time for my second annual rant this year.

    ... Kio !

now i feel better. ;-)

zxsp 0.8.0pre16: SPECTRA Interface

Hello,

the next version of zxsp is out right now. The major new thing is support for Paul Farrow's SPECTRA video interface. But let's start with the small modifications.

Rob Probin has pointed me to an improvement for the undocumented bits in the F register in the Z80 emulation which i now implemented. Hopefully programmes crash more accurate now. B-)

I replaced the default external AY extension, which was an AY-Magic, with the Didaktik Melodik, because i didn't get answer to my request for admittance to use the AY-Magic image.

Now to the big add-on:

Paul Farrow's SPECTRA video interface

Paul Farrow's SPECTRA video interface

This is a RGB-Scart interface to connect your ZX Spectrum to a TV set with much better image quality than with the standard HF lead.

It also incorporates new colour modes, a rom slot with Interface1-compatible paging, a Kempston joystick interface and an Interface1-compatible RS232 port.

Of course the "better image" is not emulated in zxsp, because you already get the best quality here :-) but colour modes, rom and joystick interface are emulated. The RS232 port is postponed for a while.

The new colour modes

Screenshot made with zxsp

There are 32 combinations for new modes combined from

• extra colours: provides 64 colours
• half cell mode: set colours for left and right 4 pixels in character cell separately
• 2-byte attributes: doubles the available information which can be stored for attributes
• 8, 4, 2 and 1-line mode: increase vertical colour resolution

Some combinations are a little bit pointless, but most provide certain trade-offs between memory consumption and added colour capabilities.

In addition the interface contains 2 x 16 kByte of ram which allows double buffering of video drawing.

All colour modes are supported by zxsp, you can also make GIF images and movies in these modes.

ROM support

The interface has a Sinclair Interface 2-like rom slot or, optionally, a DIL socket, but zxsp basically emulates the IF2 slot. There is the option to enable rom switching on the 3 positions where the Sinclair Interface 1 pages in and out. So you can "insert" an Interface 1 rom or a rom which is somehow based on this rom. You can test that it works if you enter the Basic command

    OPEN #4;"m";1;"foo"

which doesn't pass the parser of the plain Basic rom but works when an Interface 1 rom is inserted and the rom hooks are enabled, though it will only tell you that the microdrive 1 is not present.

Joystick support

The interface contains a Kempston-style joystick port and is emulated by zxsp.

RS232

The interface contains a RS232 port which works identical to the Interface 1 port, except that it only supports the RS232 mode of this port. This is not yet emulated in zxsp.

.Z80 file support

I have extended the .Z80 file standard to contain the SPECTRA interface. I just dump the information here, so that everybody who is interested can add support to his own emulator:

I have used 1 formerly unused bit to indicate presence of the SPECTRA interface:

• Bit 3 in Byte "rldiremu'".

If this bit is set, then two bytes are added to the up-to-now defined header after byte "port_1ffd" introduced by xzx/warajewo:

• Byte "spectra_bits" and
• Byte "spectra_port_7fdf".

The first one contains some state information while the latter one the state of the port 0x7FDF colour mode register. 

The bits are:

• Bit 0: new colour modes enabled
• Bit 1: RS232 enabled            
• Bit 2: Joystick enabled  
• Bit 3: IF1 rom hooks enabled
• Bit 4: rom paged in      
• Bit 5: port 239: Comms out bit
• Bit 6: port 239: CTS out bit 
• Bit 7: port 247: Data out bit

After the header, pages with ID 12, 13 and 14 may be present.

• Page 12: rom data of inserted rom
• Page 13: ram data of ram bank 0
• Page 14: ram data of ram bank 1

If page 12 is not present, then no rom is inserted.

If page 13 is not present, then it shall be copied from the contended ram page.

Same for page 14.

The zip archive of this release comes with some examples and links to my and Paul's website where you can get some more. Let's see how many people find it fancy to write a game or demo which uses this interface. You now have an emulator for convenient development. :-)

    ... Kio !

p.s.: Someone promised a download link here, so here's a link to the official download page: http://k1.spdns.de/Develop/projects/zxsp-osx/distributions/. The downloads are at the end of the page.

zxsp 0.8.0pre15: Screenshots for TC2048 and b&w machines

Hi,

i was busy working on GIF file export in zxsp, which i had temporarily disabled in 0.8.0pre14. GIF file export is now working for all models, that is, in addition to the ZX Spectrum compatible models you now can make screen shots from the TC2048 and ZX80, ZX81 and Jupiter ACE screens.

For the TC2048 i decided on a compromise, after i ran into a widely unsupported feature of gif files, the pixel aspect ratio. Screenshots of the TC2048 are made in the same 1x size as those of a ZX Spectrum. In most cases you will probably use the standard 32 column mode anyway and then this is what you want. If you make a screenshot when (parts of) the display is in 64 column mode, the image will be smoothly scaled down to 256 pixels width.

Weird Science Overscan Demo

Also new is that you can include full border animation in the gif file movies. There is an option right under the "Record gif movie" item to include or exclude border animation. In most cases this makes no difference, but when you make a movie from the loading process of a programme, then it's surely preferable not to include the border animation, because this will explode file size. On the other hand you may want to make a movie from a demo which performs some tricks with the border; then you want to record border changes.

While i was over the b&w gif movie support i also fixed some errors in the display of the ZX80 and ZX81 compatible machines, especially in 60 Hz mode.

TC2048 64 Column Mode

ZX81 Silly Basic Demo

    ... Kio !

zxsp 0.8.0 pre 14: TC2048 Screen Modes

Hello everybody,

zxsp 0.8.0 pre14 is out now.

With this version zxsp gets (hopefully) full support for the Portuguese Timex TC2048 computer. Feel free to test it and compare it to a real machine and report any bugs you find. Otherwise they won't be fixed. :-)

As a temporary drawback creation of GIF images and movies is not supported in this version.

Screenshot from the TC2048Hi-ResColourDemo

In the course of TC2048 screen mode implementation i wanted to redesign some aspects of the screen output, basically make it more easy for me. For that i wanted to replace the native OpenGL calls with a convenience layer of Qt classes: QPainter and QImage. But to make the story short: i ran into another OpenGL-related bug, which will probably fixed in Qt 5.1. And for some other unclear problems i'm still stuck in Qt 4.8.4. What i have done is: i have split rendering of frames from the the actual display update, so that i don't have to deal with OpenGL stuff when modifying to the screen rendering routines. This works pretty well now. I have removed lots of optimizations, because it seems that OpenGL overhead is dominant compared to rendering of large arrays of pixels. So i render lots of pixels. Always. And then copy exactly one large image of the screen to the graphics card. 50 times a second. Demos which tended to not get 100% screen hits or stuttered a little, now run 100% smooth. At least on my iMac.

Here are the first screen shots:

Program used for this demo

The online editor of Blogger is such a crap thing... sorry...

Screen with 1:1 zoom

I'll have a look whether this can be smoothed instead of omitting every other pixel.

Screen with 2:1 zoom

zxsp 0.8.0 pre13: Tape Recorder Animation

Hello everybody,

after working for some time on my hardware projects, i'm back to zxsp for the next weeks or months.
I have released a new version yesterday.

What's new?

I have added animation to the tape recorder windows. Ok, version 0.7.2 had an animated tape recorder too, but for 0.8.0 this is new. :-)
There is the well-known green Walkman-style tape recorder, which is currently used for all models without built-in tape recorder, and internal tape drives for the plus2 and plus2A.

Working with the tape recorder

The buttons work mostly as on the original machine. Use EJECT to eject or insert a tape. The PAUSE button has an additional function: If down while winding back or forth the tape will stop at the start or end of the current data block. The green Walkman-style recorder has separate buttons for this.

Right-click in the tape recorder window to bring up an options menu. Here you can create and insert a new empty tape or temporarily switch instant-load or auto-start-stop on and of. The latter two options are set globally for new machines in the Preferences.

What's next?

If nothing breaks i'll now work on the additional screen modes for the Timex machines.

Greetings,

    ... Kio !

Erlangen Computer Disaster 2013

Hi,

after i had to setup my Mac from backup, there are some bits left worth to note:

Restoring from Time Machine Backup was very reliefing and worked with almost no hassle; ignoring the fact, that for some reason restoring the last snapshot did not work, but the 1 hr older was not really worse. Worse was, that the backup was > 10 days old, because it's done on an external drive which i don't want to power up ever 15 minutes, which it does if left mounted. So it's mostly unmounted.

curious: iMail marked ~ 2000 mails as unread. I had to "read" them again.

silly: Restoring my LAN setting did not work. I had to correct it manually when i discovered that my Mac k1.spdns.de/ was not reachable.

bugger: 2 days later i discovered megabytes of web server error messages, because my script couldn't store preview thumbs in the preview folder. I had to correct permissions. I think this did no harm except that preview images were created anew each time a page was accessed.

that really sucks: Today i noticed that the current download version of zxsp was not in the download folder. :-/ I had to create it anew. Now it's slightly larger again, this really depends on what i have compiled in when i compiled Qt. This is a little bit silly; do i actually need a Qt installation for each project, so that the libs included in the app only contain the binaries the app actually uses? And then, of course, i made an error in the deployment process and built the pre11 app with an internal version of pre12. And some folks have already downloaded it before i noticed this mistake. For that reason there won't be a version pre12, the next will be pre13 for the sake of the "check update" function.

    ... Kio !

p.s.: http://www.youtube.com/watch?v=KCRqAzCevsY

p.p.s.: And there's a new background for the blog. :-)

Good News & Bad News

Hi,

short story:

1. My Mac is online again. 
2. There won't be a PowerPC version of zxsp unless something very unlikely happens.

long story:

I tried to create a PowerPC version of zxsp for all those who have a nicely working Mac and don't want to buy a new one just for Intel's sake.

I need to compile Qt, because the precompiled version is Intel-only. done.

I need to figure out which options to use for ./configure. done.

I tried to compile. failed.

I'm lacking a PowerPC compiler.

I'm researching in the net. done.

Checked whether there is something i can install with MacPorts. riente.

I need to install XCode 3.2.6 to get a powerpc gcc and thereafter XCode 4.0 to fix what's messed up.

I found XCode 3.2 on my Snow Leopard install disk and installed it. Asked me to quit iTunes. iTunes wasn't running. Couldn't be quit. Couldn't be force quit. Mac couldn't be shut down. 

Power cycled my Mac. done.

Installed XCode 3.2 without having started iTunes before so it couldn't leave some trails in the system. worked.

XCode 3.2 provided an older version of the powerpc gcc. grrr...

I tried to compile zxsp. failed.

I searched, found and downloaded XCode 3.2.6, whoopy 4.1GB. done.

I installed XCode 3.2.6, or, erm, at least, what it liked to install. Basically everything except the development tools itself. grrr...

I checked whether i still could compile zxsp. failed.

I installed XCode 4.0.5 from the AppStore. done.

I checked whether i still could compile zxsp. failed.

I checked whether i could use XCode. Basically yes.

I tried to compile a project. Initially failed, but after updating some settings worked. done.

I checked whether i still could compile zxsp. failed.

I tried to compile Qt 4.8.3 from the saved sources. failed.

Meanwhile i had disposed the XCode 3.2 installation. But i thought, i could have saved the slightly older powerpc gcc files. I looked into the bundle of the XCode installer on the Snow Leopard disk and installed only those packages which seemed to be appropriate. If they were installed, i didn't find them.
I installed XCode 3.2 aga...
Kernel Panic.
Mac rebooted.
Kernel Panic.
Mac rebooted.
Kernel Panic.
Mac rebooted.
Kernel Panic.
I booted from the Snow Leopard installation disk.  ok.
Checked file system. ok.
Restored from Time Machine backup. 1h later: Failed.
Restored from older Time Machine Backup. 3h later: worked.
Booted. ok.
Checked online state. k1.dyndns.org not responding.
Checked local web server: running & serving.
Checked DynDns IP assignment. ok.
Fixed LAN settings. k1.dyndns.org online again.
Downloaded Qt source. (Qt stuff wasn't in the backup) done.
Compiled Qt. worked.
Compiled zxsp. worked.
Will i give it another try? Not on this Mac.

I'll accept a donated Mac Mini or similar...

(That's the very unlikely thing to happen...)

    ... Kio !