Bsrgvty

Preface: This question does in part intersect with Use of undocumented opcodes, but targets especially the 8086 instruction handling.

I was reading Tanenbaum's "Operating Systems, Design and Implementation" 3rd edition (The MINIX Book), where I encountered the following quote that surprised me:

For instance, the 8086 and 8088 processors do not support detection of illegal instruction operation codes, but this capability is available on the 286 and above, which trap on an attempt to execute an illegal opcode.

What did the 8086 and 8088 do when it an encountered an illegal instruction?

Illegal opcodes were just instructions that hadn't been fully defined by the chip designers – a little like Undefined Behaviour in C, but much more predictable. Many people called these "undocumented instructions", because they functioned just like ordinary instructions, on the particular versions of the particular chips on which they are found. There was no special handling to prevent these instructions from executing.

Most of them were just NOPs (either because they weren't wired up to anything or because they did stuff like writing a register to itself) or duplicates of other instructions (because the instruction decoder ignored some bits when it didn't need to pay attention to them), but some of them were more interesting.

For instance, SALC, which does:

if (carry flag set)
 AL = 0xFF;
else
 AL = 0x00;

This can be used as a translation layer between C code and certain assembly returning conventions. It is equivalent to SBB AL, AL, except that it does not clobber flags.

For more information, see:

• 
  Stephen Kitt's answer to Use of undocumented opcodes.


• 
  supercat's answer to Use of undocumented opcodes, which has a good explanation of where they come from.


• A related question on Stack Overflow

It might be better to think about it this way:

On the 80186 and above, a new thing was defined called an "illegal instruction", and this new thing came with a new behavior -- a #UD exception that was generated when one was encountered. Before the 80186/80286, there was no such thing as an illegal instruction, just undocumented ones.

Your question then becomes, "what did the 8086 and 8088 do when encountering undocumented instructions" and the answer is simply that the behavior was undocumented. Some of them appeared to do nothing (i.e. they produced the same apparent result as a NOP instruction), while others did odd things, or simply the same thing as another opcode.

Here's a WayBack link that may shed more light on the situation:
http://web.archive.org/web/20190321200321/http://www.os2museum.com/wp/undocumented-8086-opcodes-part-i/

To understand this, you need a general idea of how processors work at the raw hardware level (or at least how they worked before the concept of microcode was developed).

Basically, as the processor would read the bytes of a single assembler language instruction, those bytes would be fed as an input to a network of logic gates called the instruction decoder; this network was designed to produce as output the signals to enable or disable various internal components of the processor according to what the instruction was supposed to do, so that the processor ends up doing the required operation.

As a result, if you viewed the instructions of older, simpler processors as bit patterns, you might notice certain groupings; for example, if the top three bits of the instructions would be set so, you might know that it must be a jump instruction of some type; other patterns might identify other general types of instructions.

The design team for a new processor architecture would develop a reasonable set of instructions, the various networks of logic gates to implement them, and the instruction decoder to manage them all. Back when the 8086 and 8088 processors were designed, it was not yet economically viable to add extra logic to the instruction decoder to exclude any "undocumented instructions", so any bit pattern fed into the instruction decoder would do something, but only the documented instructions would be guaranteed to exist and work the same also in the future members of the same processor family.

Sometimes the undocumented instructions actually ended up doing something useful, but relying on them was risky, as it might mean any program using a particular undocumented instructions might only run successfully that particular processor model. On a newer model, the previously undocumented instructions might actually been used for some new functionality, or they might have become completely useless as a result of a redesign of the instruction decoder.

With 80186 or above, using a number of logic gates to trap the undocumented instructions was economically feasible, and it became a compatibility feature: a program might prefer to use a new instruction available in newer processors only, but could provide a routine to emulate that instruction in software within an "illegal instruction" trap handler.

Based on the other answers, the proper answer is: they do whatever the instructions bits tell them too.
Since apparently they aren't validated, they always do something, even if that something is nothing.

Same as trying to turn on a lamp on an illuminated room- you won't notice it is there and it is on.

EDIT: I had a question in the answer which I removed, which was "Another question you may ask is: if an instruction does not exist, how does the processor knows it length?", which was replied in the comments below.