Can mouse / cursor position input be mirrored to the target computer through the Tinypilot?
Also can simultaneous key combos e.g. shift + something be sent easily?
Also wondering what the device & drivers are displayed as on the target computer.
I have a device called USBtoUSB made by Hagstrom electronics, which I was hoping would allow me to do something just like what the Tinypilot does - it takes Serial COM messages from a Sender computer and converts them to USB keyboard data (it just shows as USB keyboard on the target comp) for the target computer. It is a physical device connected physically between the two computers, one of them I can remote into and the other is the non-networked, headless computer etc.
However this device would have required me to write the code in C# or Vb /.net and didn't come with anything other than very very limited example .exes that only allow you to send one string of ASCII characters at a time by hitting Enter, and did not seem to send special keys like Alt/Ctrl etc at all.
As for the mouse position part of it, the programs provided with it involved clicking on different buttons in a GUI window to make the mouse jump to a numerical offset position on the target computer as well and it was nigh on unusable for anything that required time efficiency or requiring moving mouse to various different not-predetermind positions.
I wanted to try to see if it was possible to write in C# or Vb a program that would do a more realtime mirroring of my sender computer keystrokes (all types of keys) and mouse movements but it seems very difficult.
It looks like the Tinypilot would work as a solution for me instead. I actually already have my HDMI capture part sorted via a capture box into the networked/sender computer, which I also remote into and view the HDMI signal from target computer via OBS.
The keystrokes and maybe mouse data is the only thing left to try to get working, via the tinypilot hopefully.
Michael Lynch @michael2021-03-08 22:42:58.481ZCan mouse / cursor position input be mirrored to the target computer through the Tinypilot?
Yep, TinyPilot supports mouse forwarding.
Also can simultaneous key combos e.g. shift + something be sent easily?
Yes, TinyPilot supports sending keys with modifiers like shift, alt, etc.
I wanted to try to see if it was possible to write in C# or Vb a program that would do a more realtime mirroring of my sender computer keystrokes (all types of keys) and mouse movements but it seems very difficult.
It sounds like TinyPilot should do what you need already. If you do want to script it, TinyPilot doesn't expose an API, but it's open source, so you can peek in the code to see how the mouse and keyboard interaction works. Most of the logic is in the hid directory.
For example, here's a simple script I wrote to jiggle the mouse cursor every 3 seconds.
- BIn reply torcdc⬆:Gerald Parker @BrothaNature
I ordered the USBtoUSB device as well. You are correct, they provide very limited examples however, you do NOT have to use C#. I was able to communicate with the device using Python. The device itself is language agnostic and only cares about the bytes it receives from the computer. The language you use is just the instructions that tell the computer what data to send. This can be done with any language that has a serial port library (Python, Golang, Node.js etc..) Follow the steps below for a python example:
-
You need to make sure the USBtoUSB device driver is installed on your computer. You can install from here: https://ftdichip.com/drivers/vcp-drivers/
NOTE: I used the "setup executable" installer found under the comments column of the windows 10 installer -
Open windows Device Manager and look under Ports, when you plug the COM PORT side of the device into the computer, it should appear here under Ports. Mine was COM3.
NOTE: If you have not installed the driver it will not show up. -
Install Python on your comptuer if it is not already installed. I have version 3.6.8
-
Run pip install pyserial to install Pyserial
-
Open a text editor to create a python script and enter the following, but replace the port value with your COM port:
`
import serialserialPort = serial.Serial(port = "COM3", baudrate=19200,
bytesize=8, timeout=2, stopbits=serial.STOPBITS_ONE)serialPort.write(b"Sending a message")
`6 Plug the KBD/Mouse side of the device into the computer you want to control
7 Run the python script
Hopefully this helps!
- Ddragon788 @dragon788
Kind of crazy they charge $90 for what is basically a Rubber Ducky USB connected via serial, and is also something you could easily do with the Raspberry Pi Zero. Hook a serial UART adapter into the GPIO pins and configure the Pi Zero in gadget mode (what TinyPilot uses too), and you can write a little bridge that you can send keystrokes from the serial connection to the HID if you want to rely on a wired no-network connection adapter. The beauty of the Pi solution (and what TinyPilot offers) is being able to actually see the HDMI output of the target device, which can also add additional features that get kind of awesome/scary, like being able to OCR the screen, or use pyautogui with screen captures (or better yet OpenOCR) to "read" dialogs and click on buttons automatically.
Also scary is how easy it was to find this wireless version of the Rubber Ducky that can be interacted with via serial over the wireless for programmatic sending of keystrokes.....
https://www.amazon.com/Zopsc-Injector-Plastic-Material-Keystrokes/dp/B07TN6M8V6
-