Background

A Windows 10 host is running VMWare Workstation Pro 15.1.0. The guest OS is running X11. The pointer hardware is a Logitech Marble Mouse having configuration information on the Ubuntu community site. On Windows, scrolling by holding the small right button works with TrackballScroll. The following image shows how xorg references Logitech's buttons:

The VMWare .vmx file has the following configuration settings, which allows more than 3 different buttons to trigger mouse events in the guest OS:

usb.generic.allowHID = "TRUE"
mouse.vusb.enable = "TRUE"
mouse.vusb.useBasicMouse = "FALSE"

With these lines all mouse button events are sent to the guest OS.

The output from xinput list in the guest OS is:

xinput list 
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ VMware VMware Virtual USB Mouse id=7 [slave pointer (2)]
⎜ ↳ VMware VMware Virtual USB Mouse id=8 [slave pointer (2)]
⎜ ↳ VirtualPS/2 VMware VMMouse id=10 [slave pointer (2)]
⎜ ↳ VirtualPS/2 VMware VMMouse id=11 [slave pointer (2)]

Running xinput test-xi2 --root details events for all four buttons 1, 8, 9, and 3 (from the above picture):

EVENT type 5 (ButtonRelease)
 device: 8 (8)
 detail: 1
EVENT type 5 (ButtonRelease)
 device: 8 (8)
 detail: 8
EVENT type 5 (ButtonRelease)
 device: 8 (8)
 detail: 9
EVENT type 5 (ButtonRelease)
 device: 8 (8)
 detail: 3

This shows that the button click events are being passed from the host to the guest successfully using device 8. Of slight concern is that there are two mice assigned the same name, but having different ids:

⎜ ↳ VMware VMware Virtual USB Mouse id=7 [slave pointer (2)]
⎜ ↳ VMware VMware Virtual USB Mouse id=8 [slave pointer (2)]

Problem

Scrolling by holding down the small right button (9) while moving the trackball fails. Running xinput test-xi2 --root shows the failure:

EVENT type 4 (ButtonPress)
 device: 8 (8)
 detail: 9
EVENT type 1 (DeviceChanged)
 device: 2 (7)
 reason: SlaveSwitch
 Reporting 7 classes:
 Class originated from: 7. Type: XIButtonClass
 Buttons supported: 11
X Error of failed request: BadAtom (invalid Atom parameter)
 Major opcode of failed request: 17 (X_GetAtomName)
 Atom id in failed request: 0x282
 Serial number of failed request: 34
 Current serial number in output stream: 34
 Button labels: "Button Left" "Button Middle" "Button Right"

The file .../xorg.conf.d/50-trackball.conf configures X11 as follows:

Section "InputClass"
 Identifier "Logitech Marble Mouse"
 MatchIsPointer "true"
 #MatchProduct "Mouse"
 #MatchVendor "VMWare"
 Driver "libinput"
 Option "ScrollMethod" "button"
 Option "ScrollButton" "9"
 Option "MiddleEmulation" "true"
 Option "HorizontalScrolling" "false"
 Option "ButtonMapping" "1 2 3 4 5 6 7 9 8"
EndSection

The output from xinput list-props 7 shows:

Device 'VMware VMware Virtual USB Mouse':
 Device Enabled (124): 1
 Coordinate Transformation Matrix (126): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
 libinput Calibration Matrix (261): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
 libinput Calibration Matrix Default (262): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
 libinput Natural Scrolling Enabled (263): 0
 libinput Natural Scrolling Enabled Default (264): 0
 libinput Send Events Modes Available (244): 1, 0
 libinput Send Events Mode Enabled (245): 0, 0
 libinput Send Events Mode Enabled Default (246): 0, 0
 libinput Left Handed Enabled (265): 0
 libinput Left Handed Enabled Default (266): 0
 libinput Scroll Methods Available (267): 0, 0, 1
 libinput Scroll Method Enabled (268): 0, 0, 1
 libinput Scroll Method Enabled Default (269): 0, 0, 1
 libinput Button Scrolling Button (270): 9
 libinput Button Scrolling Button Default (271): 2
 libinput Middle Emulation Enabled (272): 0
 libinput Middle Emulation Enabled Default (273): 0
 Device Node (247): "/dev/input/event5"
 Device Product ID (248): 3599, 3
 libinput Drag Lock Buttons (274): <no items>
 libinput Horizontal Scroll Enabled (275): 1

Question

What needs to change in the X Configuration File so that the small right button when held allows vertical scrolling using the trackball in an X11 session that's running inside VMWare?

Related

Related links include:

• https://wiki.archlinux.org/index.php/Logitech_Marble_Mouse


• https://help.ubuntu.com/community/Logitech_Marblemouse_USB


• https://unix.stackexchange.com/questions/58117/determine-xinput-device-manufacturer-and-model


• https://www.x.org/releases/current/doc/man/man5/xorg.conf.5.xhtml


• https://www.mankier.com/4/libinput


• http://www.russet.org.uk/blog/3156

Additional Details

Some extra information about attempts to resolve the problem and ways that work but are less than ideal.

Disconnect from Host

Note that it is possible to disconnect the mouse from the host OS by using the Connect option, which then permits scrolling as desired. The downside is that this makes switching between the guest and host systems a bit awkward in that it takes several steps to switch between systems. Ideally, the guest OS would use a single mouse click to give it focus, scrolling would "just work", then Ctrl+Alt to release control back to the host OS.

Changing Driver

Changing the driver from libinput to evdev does not work, such as:

 Driver "evdev"

xinput

Even though VMWare lists two entries for the USB Mouse, id=8 does not appear to have any effect; whereas, id=7 can be disabled using:

xinput set-prop 7 124 0

Where 7 is the mouse ID, 124 is the Device Enabled setting, and 0 indicates false. This disables the pointer altogether; changing 0 to 1 re-enables the pointer.

Alternative Solutions

Configuring the trackball from the command-line is an acceptable solution, though changing 50-trackball.conf is preferable.

Firstly, when entries are duplicated in xinput users appear uninterested in removing duplicates. They seem more interested in applying configuration changes to both first and duplicate entries in xinput with a script like this. Although you are only changing device ID 7 it could be beneficial to make the same changes to device ID 8.

The bad atom error message seems to be related to SSH:

• 
  Why do remote programs crash with an X Error of failed request: BadAtom?. VMware does have a configuration option for X11Forwarding to fix this too: Solved: No able to perform vmrun via ssh
  

I found the following VMWare links to research. None provided a definitive answer but the questions and answers might provide clues:

• 
  Workstation 14 - no Logitech mouse wheel in guest - Solution was to disable Logitech smooth scrolling


• 
  Cannot scroll in VMs with trackpad - Not solved but help article to read was recommended
  


• 
  Horizontal scrolling with a trackball mouse? - Not answered


• 
  Mouse click partially works!!! - Not answered


• 
  Mouse Click Release - One user found removing Kensington driver and using generic driver solved issue.