Startup process

The Atari Lynx has a hardcoded bootup procedure involving initialization of the hardware and decryption followed by de-obfuscation of the header of the inserted cartridge. The boot sequence is important as it cannot be altered until the execution and control is rendered to the boot loader. Also, the code only partially initializes the Mikey and Suzy chips, so a loader should perform any additional (re)initialization. In this chapter the boot sequence is explained in detail. The encryption process and headers are described in other chapters.

Mikey boot process

Like any 6502, 65SC02 or 65C02 processor the Mikey chip has 3 vectors at the very end of the 64KB memory range. The startup vectors for the Lynx are hardcoded at $FFFA to $FFFF inside the boot ROM code.

The boot vector determines the behavior at startup as its value determines the initial value of the program counter PC . Essentially it holds the start address of the code to run at boot time. The value is $FF80 pointing to an address located inside of Mikey’s ROM area containing the startup code for the Atari Lynx from $FF00 to $FFF8. The execution of the boot sequence starts there.

1. Initialization of essential hardware registers and erasing memory:
   Set Mikey registers and erases RAM memory to zeros.
2. Perform decryption process of cartridge header frame:
   The process for decryption is a reverse of encryption. More information on encryption can be found in the corresponding chapter
3. Execute decrypted loader located at $0200

If during the boot process there turns out to be something wrong a routine with the bootloader or cartridge contents, the Lynx will not start up correctly. It will either freeze or display an “INSERT GAME” message on screen.

Common cases of error situations are:

• Cartridge not inserted: Static INSERT GAME displayed
• Decryption failure: fast flashing INSERT GAME
• Hash value mismatch of cartridge contents: Lynx freezes

The last case is only happening in the loader used in Atari published games, as it checks almost all content of the cartridge and will loop forever if any changes were detected. A longer explanation is available in the chapter on boot loaders.

Initialization of Mikey

The ROM will perform these steps to start and initialize the Lynx, load data from the cartridge and execute the program or game that is on it. The first action is a check for real hardware, by inspecting the hardware version of the Atari Lynx, which is 1 for all revisions across Lynx 1 and 2.

; Start of boot sequence at $FF80
   LDA SUZYHREV   ; Load hardware version (always 1.0 for hardware)
   BEQ $0000

Whenever the expected value of 1 (or any non-zero value for matter) is not returned, the branch is taken away and the code goes to test code at $0000. The test code probably was available during hardware creation and testing in the early stages of the Lynx lifecycle.

   PLA
   PLA
   PLA
   PLA
   LDY #CART_POWER_OFF  ; Corresponds to #2
   STY IODAT            ; Set cartridge power off
   INY
   STY IODIR            ; External power and cart address to output
   STY MAPCTL           ; Mikey and Suzy addresses are RAM
   STZ $00              ; Set $0000 to 0
   JMP ClearMemory

The four calls to PLA cause required RAS signals for the RAM memory.

The hardware registers of IODAT and IODIR are explained in more details in the chapter on cartridges. The boot sequence sets the pins for external power and cart address to output. While this is correct for the cart address, the detection of external power is an input bit, so must be set to accordingly before IODAT expects to read the correct value for an external power source from bit 0 EXTERNAL_POWER.

The memory map control register MAPCTL is set to #$03 which evaluates to MIKEY_SPACE and SUZY_SPACE being set. The effect is that RAM is mapped to the corresponding address spaces of Suzy ($FC00-$FCFF) and Mikey ($FD00-$FDFF) being mapped to RAM memory. It also means that the ROM space ($FE00-$FFF8) and vector space ($FFF9-$FFFF) are still referring to the ROM and vectors instead of RAM. This makes absolute sense, as the ROM area is currently being read from for the code of the boot sequence.

The next step is a jump to a routine near the beginning of the ROM space to clear the RAM memory for almost the entire range from $0000 to $FFFF.

   ; Clear all memory from $0003 to $FFFF to 0x00
ClearMemory
   STZ dest+1
   LDA #$00
.3 STA (dest),y
   INY
   BNE .3
   INC dest+1
   BNE .3

With the memory at $0000 already zeroed at the first part, and dest+1 being $0001 the first two bytes of RAM memory are set to zero. However, as Y is equal to #$03 when entering this routine, $0002 is not set to zero apparently. The rest of the memory range $0003 to $FFFF is being written to, with most having the effect of setting the RAM memory to #$00. The address range for ROM and the vectors are not mapped to RAM and are read-only because of it, so writing zero to it doesn’t have any effect. However, checking for a condition of dest smaller than $FE00 would be slower during booting and would cost additional bytes of the limited ROM space.

The boot code will continue with the initialization of Mikey. It reads pairs of values for both the offset from the start of Mikey address space $FD00 and the corresponding values.

InitMikey
   LDX #MIKEYVALS_COUNT
.4 LDA MikeyValues-1,x
   LDY MikeyOffsets-1,x
   STA $FD00,y
   DEX
   BNE .4

The effect of the initialization is that video is enabled at 60 Hz with the timers set correctly, and the color palette populated with black and yellow colors for pen 0 and F.

The offsets and values are used in reversed order, so the final action is to acknowledge Suzy, allowing Mikey to be put to sleep when Suzy is busy. This is recommended to avoid Mikey not going to sleep during Suzy activity. Suzy will not be used during booting in a happy flow scenario, so it is purely for error situations, as will become evident later. Also, a comment in the Atari boot loader eludes to not setting the SDONEACK.

* Note:  I'm not clearing Suzy's Done-Acknowledge flag.  This presumes that
* all programs will clear it before starting, which all good programs do
;???  STZ SDONEACK

Decryption of header

The next part of the boot sequence will prepare for and then execute the decryption of the first frame of the encrypted header. Typically, the boot loader consists of two frames, where the boot ROM only decrypts the first frame and the loader uses the same code in Mikey’s ROM to decrypt the second. Note that the ROM does not require or depend on the existence of two frames. Some homebrew games have a microloader in a 1-block single frame header.

The decryption algorithm uses Montgomery multiplication to perform the calculation of the exponent of the encrypted data block of 51 bytes, followed by a modulo division by the modulus of the key. The exponent in the public key is the prime number 3, so the operation to perform is a power of 3. The modulus itself is embedded in the ROM at location $FF9A.

The Montgomery multiplication helps perform these two parts of the calculation without having to do a complex division. The implementation uses self-modifying code, so it has to be located in RAM address space. Therefore, the decryption process starts by copying 256 bytes containing the decyption code from $FEC1 to $5000.

      WORK_ORG .EQU $5000

; Copy 256 bytes from $FEC1 to $5000
copy  LDA Decrypt,x     ; Decrypt is function at `$FEC1`
      STA WORK_ORG,x
      INX
      BNE copy

The implementation of the decryption process is less than the 256 bytes being copied. The last code and data is located at $FF7F, so 190 bytes would have sufficed. As before, it simply is easier to copy more to RAM and save bytes in the code. The RAM area of $5000 to $50FF can be overwritten by the program later anyway.

Before calling the decryption routine, the ptr variable at location $06 and $07 in zero-page should contain the destination address of the decrypted data.

; Start decryption process for first frame
   STZ ptr        ; Destination address of decrypted data (low)
   LDA #2
   STA ptr+1      ; Destination address of decrypted data (high)
   STZ checksum   ; Initialize transition byte

As can be seen in the code fragment above, the ptr implicit address is set to $0200, where the decrypted frame will be stored. This is also the hardcoded jump address where boot loaders always start execution after decryption has completed.

The data to be decrypted is loaded from the start of the first (zero-th) block on the cartridge.

   LDA #0         ; Prepare for first block
   JSR SetCartBlock

   LDA RCART_0    ; Read cart (using strobe CART0)
   CMP #$FB       ; First byte has two's complement of number of blocks in first frame 
   BCC retry      ; If value is less than #$FB it is not a correct header
   STA num_blocks ; Save block count
   STZ GREENF     ; Color F to black
   STZ BLUEREDF   ; 

A check is made to make sure that the block count is not larger than 5, by evaluating whether the two’s complement of the count is not smaller than #$FB. This would indicate an error in the data, as the frames cannot be larger than 256 bytes and consequently cannot have more than 5 blocks.

Whenever an error situation is detected a retry routine is called. The purpose of this code is to start Suzy initialization and keep counting the number of times in a powerdowncount double byte counter. When the counter reaches 65536 the boot sequence will loop forever, halting further execution and shutting down the Lynx console.

retry
   INC powerdowncount
   BNE InitSuzy
   INC powerdowncount+1
   BNE InitSuzy
   STZ SYSCTL1    ; Power down console
.6 BRA .6         ; Give up and loop forever 

The following step of the process copies a block from the frame in reverse order and starts the actual decryption using montgomery multiplication. It will repeat this process for the required number of blocks as indicated in the num_blocks:

   STA checksum   ; Remember last byte
   INC num_blocks ; Next block
   BNE ReadBlock  ; Continue reading and decrypting next block of data from cart

During the algorithm for decryption several validations are made to check for error conditions, such as the existence of non-zero values in the first bytes of each block, as well as the block not being larger than the modulus. After decryption of a block the start of each block is compared with #$15 which is the hard-coded marker byte that should be present.

Errors during startup

In case of any errors a retry is attempted. Suzy will be initialized, with the sole purpose to display the characteristic INSERT GAME sprite.

The initialization of Suzy reads values and offsets into Suzy space at $FC00, similar to the initialization of Mikey. The values are read in reverse order.

InitSuzy
   LDX #SUZYVALS_COUNT-1
.7 LDA SuzyValues,x
   LDY SuzyOffsets,x
   STA $FC00,y
   DEX
   BPL .7
   STZ CPUSLEEP   ; Reset CPU bus request flip flop (draw INSERT GAME sprite)
   STZ SDONEACK   ; Clear SDONEACK

The values and offsets are as follows:

The values show how the video base address is set to $2000, well above the destination of the first frame at $0200 and before the copied decryption code at $5000. In addition, the horizontal and vertical offsets are 0. Most importantly, the sprite data structure is pointing to $5082 inside the copied range. In that location the sprite’s control block (SCB) data is stored to define the sprite to display.

   .BYTE $05      ; SPRCTLO (1 bit/pixel, no flipping, non-collideable sprite)
   .BYTE $93      ; SPRCTL1 (Totally literal, HSIZE and VSIZE specified, drawing starts at upper left quadrant)
   .BYTE $00      ; SPRCOLL
   .WORD $0000    ; Address of next SCB
   .WORD $5092    ; Address of SPRDLINE (sprite data)
   .WORD $0080    ; HPOSSTRT 128
   .WORD $0048    ; VPOSSTRT 72
   .WORD $0400    ; HSIZE (magnify by 4 horizontally)
   .WORD $0400    ; VSIZE (magnify by 4 vertically)
   .BYTE $F0      ; Palette colors

The SCB defines a sprite with 4x4 magnification of a literal two color (1-bit) color sprite. The two colors in the palette are F and 0 for yellow and black. Its bottom right corner will be displayed at (128,72), because the drawing starts at the upper left quadrant. This implies that the sprite will be rendered upside down and mirrored left to right. The pixel data of the sprite is:

.OR $FF53

.BYTE 04 E2 EA 87    ; ...000.0...0.0.0.0000...
.BYTE 04 FA AA B7    ; .....0.0.0.0.0.0.0..0...
.BYTE 04 F2 08 97    ; ....00.00000.000.00.0...
.BYTE 04 FA 4A F7    ; .....0.00.00.0.0....0... 
.BYTE 04 E2 E8 87    ; ...000.0...0.000.0000...
.BYTE 02 FF          ; ........................
.BYTE 05 B5 11 68 FF ; .0..0.0.000.000.0..0.000
.BYTE 04 B5 D7 2D    ; .0..0.0...0.0...00.0..0.
.BYTE 04 B9 91 0D    ; .0...00..00.000.0000..0.
.BYTE 04 B5 DD 4D    ; .0..0.0...0...0.0.00..0.
.BYTE 05 19 11 68 FF ; 000..00.000.000.0..0.000
.BYTE 00             ; End of quadrant

The initialization ends with setting SPRGO to #$01 which effectively starting drawing the sprite in yellow pixels on a black background, followed by putting the CPU to sleep using CPUSLEEP and acknowledging the wakeup with SDONEACK.

The retry routine and in effect the Suzy initialization is called only when either the block count is not correct or there are errors detected in the decryption process. This explains the two cases of the INSERT GAME being displayed:

1. Typically the block count is not valid when the cartridge is not (correctly) inserted and it could not be read. It will retry many times and keep displaying the static sprite, never getting past the point of the validity check.
2. Right after the check of the block count the F color is changed from yellow to black. It causes the sprite to no longer be visible during the decryption process. In case of an error, the use of the retry routine now causes re-initialization of Suzy which displays the sprite again in yellow. This is the reason for the flashing sprite, as it alternates between being visible and invisible at a very fast rate.

Boot loaders

If all goes as expected during the boot process and all validations are asserted, the execution will continue at $0200 after decryption has stored the first frame there. At this point, only Mikey is initialized.

The last action of the boot ROM is to render control to the code in the decrypted frame by making a jump to the hard coded load address. Each game cartridge must have a first header that is decryptable by the algorithm executed by Mikey. This made it hard to create cartridges of your own, before the encryption process and its keys were discovered and reverse-engineered.

Hint

The Atari loaders use the decryption routine RSA_LOAD located in Mikey ROM at $FE4A to decode the second frame.

The RSA_LOAD routine can be used in similar fashion by your code to decrypt specific files or areas. It does require setting ptr to the destination address, such as $0300, as well as positioning the cartridge read location to the place to read. Of course, the copied ROM range at $5000 should still be in memory.

As the jump after decoding is hardcoded to be $0200, your program should modify the instructions there to return execution to the desired point in your own code.

Reaching this point means that no errors occurred. It also implies that Suzy has not been initialized yet. It is up to the boot loader to perform this task if it uses video, for example to display a title screen.