How To Send Inputs to Multiple Windows / Minimized Windows with Python. Or Die Trying

Let's explore using SendMessage to send automated inputs to multiple windows at once, or to windows that are minimized or in the background. I'll share my progress and discuss the suitability of using this method for botting.

View this code on GitHub: https://github.com/learncodebygaming/...
Join us on Discord:   / discord  

Today I want to answer the most asked question on my channel:
How do I send mouse and keyboard inputs to multiple windows at once
How do I send inputs to a minimized window or background window
How can I keep using my mouse and keyboard while also sending automated inputs?

I'm guessing most of you want to know how to do this because you've written a bot, and you want to run multiple bots at once, or you want your bot to run in the background while you continue using your computer.

And my initial reaction when I started getting this question was: just use virtual machines. You could have multiple VM's running in the background and those would be isolated from whatever you're doing on your main desktop. I've confirmed that this does work, but the problem is the performance of virtual machines is typically not very good. At least with Hyper-V, even basic games show considerable lag. You might be able to find better performance by setting up hardware graphics acceleration for your VM, but that's not the route I decided to explore today.

So what other options do we have?

About a year ago now, as I was working on my OpenCV series, I got a tip from a viewer telling me about this SendMessage() function he was using to send inputs to a window in the background. It's a Windows API function he was calling via pywin32, and it allows you to send messages, including mouse and keyboard inputs, to a specific window. And there's a couple StackOverflow discussions about SendMessage(), but other than that, there really isn't a whole lot I could find about working with this. So let's try it out and see what we can do with it.

I first tested a simple example with Notepad, and this gave me more problems than I was expecting.

The Notepad window has an inner window called 'Edit' that you need to target if you want to send inputs to the main text area. Even after I solved that, I wasn't able to get any results using WM_KEYDOWN and WM_KEYUP. Only by using WM_CHAR with ord() was I able to get text to appear in Notepad. Both SendMessage() and PostMessage() work this way.

Next I decided to take this to a simple browser game. I found that WM_KEYDOWN and WM_KEYUP did work in Chrome, and there weren't any inner windows to worry about either.

The first challenge I faced here was how to press down multiple keys at the same time. If you sleep() between WM_KEYDOWN and WM_KEYUP that's blocking, so other code can't execute at the same time. You could solve this by using threading, but this time I decided to use sched to schedule each command in queue, and then run that queue once it's been built. This worked really well for one game, so now I wanted to scale it up to play games in multiple windows.

Unfortunately in Chrome, I couldn't get SendMessage to work when the window wasn't focused. So I ended up using SetForegroundWindow() to quickly bring focus to the window that I'm sending the next command to. This does work, but because setting the foreground window takes a little bit of time you have to be careful to not have two commands run at the same time. I wasn't very careful about how I avoided those collisions here, so that's something you'll want to develop if you plan on using this method of quickly swapping between windows. The more copies of the game you're running, the more likely you are to run into these conflicts.

As a side note, if you're going to be swapping window focus anyway, you might find better performance using SendInput(), or even just PyAutoGUI or PyDirectInput as you normally would. The only benefit that SendMessage() or PostMessage() give us is that we can target a specific window handle with them.

So now I kinda have a solution for botting in multiple windows at once, but we still have the issue of wanting to send inputs to minimized or unfocused windows so that we can run our bots in the background.

I found that how SendMessage and PostMessage behave are very dependent on what program you're trying to automate. For example with Firefox, I was able to send inputs to Mario even when the browser was minimized or in the background. But this wasn't quite a perfect solution because Firefox was laggy compared to Chrome, and I couldn't get SendMessage to work with multiple tabs or multiple windows in Firefox unless I resorted back to our SetForegroundWindow method.

Continue reading here: https://learncodebygaming.com/blog/ho...

#programming #python