Bsrgvty

The BRK instruction on the MOS 6502 seems to be one of the more
ill-documented features of the processor. The 1976 preliminary data
sheet from MOS indicates that it's a 1-byte instruction using
the "implied" addressing mode (i.e., it has no operands), as does the
the 1980 CSG data sheet.¹ (Edit: the MCS6500 Microcomputer
Family Programminng Manual does describe the details and
intention of the BRK instruction in §9.11 pp. 144-146.)

However, unlike an IRQ interrupting any other 1-byte instruction, the
program counter pushed on to the stack when a BRK is executed does
not point to the byte immediately after the opcode but instead to the
second byte after the opcode. (This is mentioned nowhere in those
two data sheets.²)

Given the lack of documentation about this, I'm guessing that this was
unintentional³ (though evidence otherwise is welcome!), because it
wastes the byte after the BRK if the interrupt handler does an RTI
to the address pushed by the BRK after handling it. (If there was no
RTI, the byte need not be wasted: control would never flow past the
BRK and so another routine could have an entry point immediately
after it. This was the case on, e.g., the Apple II, where a BRK would
stop whatever program was executing and enter the monitor.⁴)

I'm looking for examples in "real-world" code (as opposed to
demonstration or example code) of how this "spare byte" was used in
systems with IRQ routines that did an RTI to return to the address
pushed by the BRK instruction.⁵ This could be use by the IRQ routine
itself (as a parameter, or similar, e.g. as Woz sugested in his
November 1977 BYTE article on Sweet 16) or something else.

Ideally the examples would be from widely distributed software for
6502-based general purpose computers. Again, I'm not too interested in
demo code that shows how it could be used in a theoretical
real-world program. But if you feel you have an example that has
something useful to offer beyond the limited technical details of
how to implement this, feel free post an answer about it.

¹_{Both have a note "See Fig. 1" beside the instruction in the
instruction set/opcde table, but none of the figures in either
document are numbered, and none seem to be relevant. I welcome
clarification of this.}

²<sub>The ignored byte after the BRK is kind of implied by the cycle
breakdown of the BRK instruction on page A-11 of the 1976 MCS6500
Microcomputer Family Hardware Manual, where after the T0
cycle reads the opcode the subsequent T1 cycle puts on the address bus
"PC + 1 (PC on hardware interrupt)" before pushing the PC on to the
stack in T2 and T3. But documenting it as a one-byte instruction or
opcode with no operands, despite the byte after it being skipped,
persists even today in the 2018 WDC W65C02S data sheet.


     This post on forum.6502.org does say, "WDC
describes BRK as '...a 2 byte instruction. The NMOS and CMOS devices
simply skips [sic] the second byte (i.e. doesn’t care about the second
byte) by incrementing the program counter twice.'" However, it doesn't
give a source for that quote.</sub>

³_{By "unintentional" I mean that having that unused byte after the
BRK instruction was not a design goal. In this context, "we've got
an unused byte there but it isn't a problem and that lets us save
transistors (or whatever) elsewhere" does not count as
"intentional."}

⁴_{See the IRQ routine at $FA86 in the BIOS listing on page 81
(PDF page 84) of the Apple II Reference Manual.}

⁵_{An IRQ routine could use bit 4 of the program status register
value pushed on to the stack to determine it had been invoked by a
BRK and then use the PC pushed on to the stack to find the spare
byte. Wilson Mines gives details on how to do this.}

On the BBC Micro, the byte after the BRK instruction held the error number, followed by the error message string terminated with 0x0D. CALLing the address of the BRK instruction would cause an error to be raised.

Update with example, as a runnable BBC BASIC program:

10 DIM b% 32
20 ?b%=0: REM BRK
30 ?(b%+1)=42: REM Error number 42
40 $(b%+2)="StackExchange"
50 CALL b%
>RUN

StackExchange at line 50
>PRINT ERR
 42
>_

The BRK instruction on the MOS 6502 seems to be one of the more ill-documented features of the processor. [...]
Given the lack of documentation about this

It is documented quite well and in depth in the corresponding MCS 6500 Microcomputer Family Programming Manual of January 1976 (and all follow ups). Check page 144 and after for description, reasoning and examples.

(This is mentioned nowhere in those two data sheets.)

Data sheets aren't meant as full documentation, but rather giving an overview.

However, unlike an IRQ interrupting any other 1-byte instruction,

This might be a first misunderstanding. BRK isn't a 1-byte instruction interrupted by IRQ, but rather an instruction working like IRQ. The difference is in cause and effect - here the way the instruction works. With an external interrupt it gets handled before the PC is advanced on the next instruction (technically by inserting a BRK). With a BRK the instruction gets fetched and processed (over two cycles) before the PC is pushed.

Still this doesn't make it a two byte instruction, it stays a one byte instruction with the workings of advancing PC by two before pushing.

I'm guessing that this was unintentional (though evidence otherwise is welcome!),

I'd say it was quite intentional. With the 6500 being designed as an extremely low cost CPU, it was what could be achieved with a bare minimum in additional circuitry. So instead of setting up a separate vector the IRQ vector was used - in fact, BRK processing is even used to make IRQ/NMI/RES happen.

The increment by two is part of this too, as that's the default behaviour of 6502 instructions, as all take two cycles at least. Single byte instructions need to explicitly disable the increment during the second cycle. Since BRK isn't a normal instruction, intended to be executed in a regular sequence and situation, adding that correction wouldn't make any difference.

BRK was never intended to be more than a debugging aid. A use that never comes up in regular use, thus its rather cumbersome detection and the need to readjust PC as well did not do much harm. In fact, being a debugging tool, readjusting the PC to the original instruction address and replacing the instruction overwritten (*1) had to be done anyway.

It's all about offering an in-place debugging-aid with the least effort possible.

Similar the use of x'00' as opcode as any (back then) (E)PROM can always be patched to contain all zero in a location, enabling debugging of PROM code. Similar is forcing the data bus to zero, to detect a special case during problem analysis, an easy task, not requiring a lot of hardware.

All of this has to be seen interlinked.

but it does allow you to do something interesting, which is to use the byte after the BRK as a parameter to the IRQ routine.

Yes, that has been done many times. After all, and at first sight, it seems like a really nifty SuperVisor Call type of instruction. Looking closely it does involve quite some address juggling. So much so that more performance-aware developers usually preferred to use JSR calls instead, as their return address can be used directly with less stack mangling as well (*2).

I'm not terribly interested in examples from embedded systems

Sure, valid point - still, to understand why things have been made the way they are, it's always a good idea to keep in mind that the 6502 wasn't developed with a general purpose computer in mind, but embedded use. The whole point was to make a very cheap CPU and support it with (for that time) quite complex and versatile I/O companion chips integrating several previous separate components into one (e.g. 653x type).

*1 - Or emulating it and advancing the PC to after the whole instruction

*2 - See the ProDOS MLI as main example, but IIRC Woz used it already in 1977 to call Sweet-16 sequences.

The SOS operating system written for the Apple /// used BRK for operating system calls. The byte following the BRK holds the OS function number.

Of course, the two bytes after that are also used as inline data, so you can argue that it's not a true example of BRK with a signature byte. Still, it's an example of "real world" code using inline data after a BRK.

Regarding the RTI: looking at this disassembly, page 112 in the PDF, IRQ.RCVR appears to be the BRK entry point. It eventually jumps to DISPATCH on page 118. At the bottom of page 119 it invokes an RTI instruction.