[comp.windows.x] How to simulate mouse events from a data tablet

davis@3d.enet.dec.com (Peter Davis) (05/29/91)

In article <64370@bbn.BBN.COM>, schroder@bbn.com (Ken Schroder) writes...
>I'm trying to integrate a data tablet with the mouse using X11R4.
>I'd like the cursor to move whenever I move either the mouse of the
>cursor/pen on the tablet.  I'd like X clients to receive a button
>event whenever a button on either the mouse or tablet is pressed
>or released.  Has anyone already done this?
> 
>I've found that using XWarpPointer works fine to drive the pointer
>from the tablet.  The problem is getting a suitable value for the
>window pointer passed in the event structure to XSendEvent.  Does
>anyone know the correct value so the server will map screen (x,y)
>to window and window relative (x,y)?
>
Gee, I'm trying to do almost this exact thing.  I'm looking at a whole
bunch of input devices, and I want to be able to simulate mouse movements
by sending events, and then send any additional data that the devices can
input via the ClientMessage mechanism.

To do motion, I found I had to:

   1	XQueryPointer to get the current position,
   2    find the lowest offspring of the pointer window which still contains
	the pointer position,
   3	XTranslateCoordinates to get the coordinates relative to that
	offspring,
   4	XSendEvent to send a motion event to that window,
   5	XGrabPointer
   6	XWarpPointer
   7	XUngrabPointer
   8	XQueryPointer to see if the pointer window has changed.  If it has,
	repeat steps 2, 3, and 4 for the new window.

To do button transitions, I had to:

   1	XQueryPointer to get the current position, window, etc.
   2	find the lowest offspring of the pointer window which still contains
	the pointer position,
   3	XTranslateCoordinates to get the x, y in the coordinates of that
	offspring window,
   4	Figure out what the button/key mask should be, and
   5	XSendEvent to send the button event to that window.

I still don't have this completely working, so I won't vouch for the
completeness of the above procedures.  If anyone has any improvements or
suggestions, I'd be glad to hear 'em.

Thanks.
-pd

marbru@auto-trol.com (Martin Brunecky) (05/30/91)

In article <1991May29.150652.20665@engage.pko.dec.com> davis@3d.enet.dec.com (Peter Davis) writes:
>
>Gee, I'm trying to do almost this exact thing.  I'm looking at a whole
>bunch of input devices, and I want to be able to simulate mouse movements
>by sending events, and then send any additional data that the devices can
>input via the ClientMessage mechanism.
>
>To do motion, I found I had to:
>
>   1	XQueryPointer to get the current position,
>   2    find the lowest offspring of the pointer window which still contains
>	the pointer position,
>   3	XTranslateCoordinates to get the coordinates relative to that
>	offspring,
>   4	XSendEvent to send a motion event to that window,
>   5	XGrabPointer
>   6	XWarpPointer
>   7	XUngrabPointer
  .....
    Can you elaborate on why you have to XGrabPointer ? (I don't, and have
    no problems I can see - so far).
    Plus, I also found I have to look at what events those windows are
    interested in. The "propagate" mechanism did not work for me, so I am
    looking for the "topmost" window containing the pointer AND interested
    in a particular event.

>
>To do button transitions, I had to:
>
    ....(more of the same as above, without grabs - deleted).....

    I found I can easily send a button event. The problems arise when the
    client receiving this button event either:
    - assumes a passive grab
    - grabs the button explicitly.

    For example, trying to resize window from a "simulated" mouse under
    most widnow managers won't work - those guys grab the "real" mouse
    button - which is not depressed.
    Grabbing the pointer for motions might help, but I am worried it will
    confuse other things ....


-- 
=*= Opinions presented here are solely of my own and not those of Auto-trol =*=
Martin Brunecky                      marbru%auto-trol@sunpeaks.central.sun.com
(303) 252-2499                       (better avoid:  marbru@auto-trol.COM )
Auto-trol Technology Corp. 12500 North Washington St., Denver, CO 80241-2404 

davis@gauss.enet.dec.com (Peter Davis) (05/30/91)

In article <1991May30.022513.15790@auto-trol.com>, marbru@auto-trol.com (Martin Brunecky) writes...
>
>    Can you elaborate on why you have to XGrabPointer ? (I don't, and have
>    no problems I can see - so far).
>
	I thought I had to grab the pointer in order to be able to move
	it around in windows outside my control.  I don't remember testing
	this, however, so perhaps it's not necessary.
>
>    Plus, I also found I have to look at what events those windows are
>    interested in. The "propagate" mechanism did not work for me, so I am
>    looking for the "topmost" window containing the pointer AND interested
>    in a particular event.
>
	As far as I know, there's no way, or at least, no easy way to find
	out which window to send the events to.  Using the lowest window
	containing the pointer seems to work most of the time, but not
	always.  As I mentioned. I still don't have my code completely
	working yet, so I don't know all the ramifications.  Maybe I should
	try your approach.
> 
>    I found I can easily send a button event. The problems arise when the
>    client receiving this button event either:
>    - assumes a passive grab
>    - grabs the button explicitly.
> 
>    For example, trying to resize window from a "simulated" mouse under
>    most widnow managers won't work - those guys grab the "real" mouse
>    button - which is not depressed.
>    Grabbing the pointer for motions might help, but I am worried it will
>    confuse other things ....
>
	It may be that there's no way to really do this other than through
	some X extension, such as the input extension. This would require
	rebuilding the X server and possibly the operating system kernel
	as well, so it's not well suited for quick testing and prototyping
	of different input devices.

	I'd be interested in seeing what you're doing.  If you want to
	exchange code, we can do it here or by mail.

	Thanks.
	-pd

michael@xzaphod.uucp (Michael R. Miller) (05/31/91)

In article <1991May29.150652.20665@engage.pko.dec.com> davis@3d.enet.dec.com (Peter Davis) writes:
>
>In article <64370@bbn.BBN.COM>, schroder@bbn.com (Ken Schroder) writes...
>>I'm trying to integrate a data tablet with the mouse using X11R4.
>>I'd like the cursor to move whenever I move either the mouse of the
>>cursor/pen on the tablet.  I'd like X clients to receive a button
>>event whenever a button on either the mouse or tablet is pressed
>>or released.  Has anyone already done this?
>>	...
>
>Gee, I'm trying to do almost this exact thing.  I'm looking at a whole
>bunch of input devices, and I want to be able to simulate mouse movements
>by sending events, and then send any additional data that the devices can
>input via the ClientMessage mechanism.
>	...
>To do motion, I found I had to:
>	...
>To do button transitions, I had to:
>	...
>I still don't have this completely working, so I won't vouch for the
>completeness of the above procedures.  If anyone has any improvements or
>suggestions, I'd be glad to hear 'em.
>
>Thanks.
>-pd

Suggestion:  Modify the input devices management code in the server to
look for additional input devices.  This is what I did to solve the
problem.

Presumption (perhaps a big one):  You have (or can obtain) the source
for your server.

Michael R. Miller
uunet!xzaphod!michael

davis@3d.enet.dec.com (Peter Davis) (05/31/91)

In article <1991May30.022513.15790@auto-trol.com>, marbru@auto-trol.com (Martin Brunecky) writes...
>    Can you elaborate on why you have to XGrabPointer ? (I don't, and have
>    no problems I can see - so far).

I had thought that I needed to have the pointer grabbed in order to be able
to move it anywhere on the screen.  I've since gone back and tested and found
this is not the case, so you don't really need to grab the pointer, except...

XWarpPointer generates it's own motion events.  However, the key/button mask
in those events is that of the "real" pointing device, rather than the
"fake" device you're simulating.  So, to get the events to have the correct
key/button mask, I think you have to grab the pointer, move it, and then
explicitly send events to the affected window(s).

>    Plus, I also found I have to look at what events those windows are
>    interested in. The "propagate" mechanism did not work for me, so I am
>    looking for the "topmost" window containing the pointer AND interested
>    in a particular event.

Yeah, I probably should do that too.  I don't think there's any trivial way
to determine which window should get the events.  Nothing I've tried seems to
work for window manager decoration.

>    I found I can easily send a button event. The problems arise when the
>    client receiving this button event either:
>    - assumes a passive grab
>    - grabs the button explicitly.
> 
>    For example, trying to resize window from a "simulated" mouse under
>    most widnow managers won't work - those guys grab the "real" mouse
>    button - which is not depressed.
>    Grabbing the pointer for motions might help, but I am worried it will
>    confuse other things ....

If the window manager grabs the "real" mouse, that shouldn't precluded its
getting synthetic events from your input device.  The problem may be related
to the button state mask, as I mentioned earlier.  It may also be that the
window manager is explicitly ignoring events which have the send_event flag set
to true, but that would be really nasty.

So far, I think it's mainly a problem of figuring out which window to send
to.  Any suggestions, comments, etc. are welcome.

If you're interested in exchanging code to have a look at what each other
is doing, it might save us both some time.  Send me some mail if interested.

Thanks.
-pd

marbru@auto-trol.com (Martin Brunecky) (05/31/91)

In article <1991May30.191703.11099@engage.pko.dec.com> davis@3d.enet.dec.com (Peter Davis) writes:
>
>>    Can you elaborate on why you have to XGrabPointer ? (I don't, and have
>>    no problems I can see - so far).
>
>XWarpPointer generates it's own motion events.  However, the key/button mask
>in those events is that of the "real" pointing device, rather than the
>"fake" device you're simulating.  So, to get the events to have the correct
>key/button mask, I think you have to grab the pointer, move it, and then
>explicitly send events to the affected window(s).

    Which means you have to properly simulate all enter/leave events. No
    thanx, I am giving up on that one -).
   
>
>Yeah, I probably should do that too.  I don't think there's any trivial way
>to determine which window should get the events.  Nothing I've tried seems to
>work for window manager decoration.
>
        ....Window wins[...] - stack of windows containing the pointer
        /* find the window which wants our event, give up on don't propagate */
        for ( --iw; iw >= 0; --iw)
        {           
           XWindowAttributes attr;
           if ( XGetWindowAttributes ( dpy, wins[iw], &attr ))
           {
                if (attr.all_event_masks & mask       ) break;
                if (attr.map_state != IsViewable      ) continue;
                if (attr.do_not_propagate_mask & mask ) return(FALSE);
           }
        }



-- 
=*= Opinions presented here are solely of my own and not those of Auto-trol =*=
Martin Brunecky                      marbru%auto-trol@sunpeaks.central.sun.com
(303) 252-2499                       (better avoid:  marbru@auto-trol.COM )
Auto-trol Technology Corp. 12500 North Washington St., Denver, CO 80241-2404 

davis@3d.enet.dec.com (Peter Davis) (06/03/91)

In article <1991May31.163832.22933@auto-trol.com>, marbru@auto-trol.com (Martin Brunecky) writes...
>In article <1991May30.191703.11099@engage.pko.dec.com> davis@3d.enet.dec.com (Peter Davis) writes:
>>
>>>    Can you elaborate on why you have to XGrabPointer ? (I don't, and have
>>>    no problems I can see - so far).
>>
>>XWarpPointer generates it's own motion events.  However, the key/button mask
>>in those events is that of the "real" pointing device, rather than the
>>"fake" device you're simulating.  So, to get the events to have the correct
>>key/button mask, I think you have to grab the pointer, move it, and then
>>explicitly send events to the affected window(s).
> 
>    Which means you have to properly simulate all enter/leave events. No
>    thanx, I am giving up on that one -).
>

It's worse than that.  It seems that grabbing the pointer causes it's own
events, at least according to xev.  When I try it, I get all kinds of
EnterNotify, LeaveNotify, FocusIn, FocusOut, and KeymapNotify events that seem
to be all tied to grabbing the pointer.

It's beginning to look like you have to modify the server, or use an input
extension to accomplish this correctly.  It's too bad.  You can almost do it
from the application level, which sure would be nice.

-pd

davis@3d.enet.dec.com (Peter Davis) (06/03/91)

In article <1991May31.163832.22933@auto-trol.com>, marbru@auto-trol.com (Martin Brunecky) writes...
>In article <1991May30.191703.11099@engage.pko.dec.com> davis@3d.enet.dec.com (Peter Davis) writes:
	.
	.
	.
> 
>    Which means you have to properly simulate all enter/leave events. No
>    thanx, I am giving up on that one -).
>

I think what's really needed here are XWarpButton and XWarpKey functions.
XWarpPointer only does half the job.  There's really no reason why you should be
able to warp the pointer position, but not be able to control the other input
device values as well.

michael@xzaphod.uucp (Michael R. Miller) (06/05/91)

In article <1991Jun3.142241.14494@engage.pko.dec.com> davis@3d.enet.dec.com (Peter Davis) writes:
>
>In article <1991May31.163832.22933@auto-trol.com>, marbru@auto-trol.com (Martin Brunecky) writes...
>>In article <1991May30.191703.11099@engage.pko.dec.com> davis@3d.enet.dec.com (Peter Davis) writes:
>	.
>	.
>	.
>> 
>>    Which means you have to properly simulate all enter/leave events. No
>>    thanx, I am giving up on that one -).
>>
>
>I think what's really needed here are XWarpButton and XWarpKey functions.
>XWarpPointer only does half the job.  There's really no reason why you should be
>able to warp the pointer position, but not be able to control the other input
>device values as well.

What is needed is to modify the server.  The server was designed with a decent
extension mechanism.  Extensions are treated as (almost) first class citizens.
Why not use the tool specifically designed to accomodate this situation?

The problem with trying to do OUTSIDE the server what should be done INSIDE
is the duplication of all that very good code to handle generation and
delivery of events.  Also you are missing that wealth of information buried
inside the server.

The XInput extensions were created to address this problem.  While not a
complete solution, it'll probably fulfill most needs.  One area I found as
"missing" was concurrency of use.  I had a need to be able to accept input
from a digitizing tablet, a touchscreen and a mouse as well as the keyboard
without any application changing the active pointing device.  This would
allow the user to simultaneously use the touchscreen and the mouse (Etch-a-
sketch time again) without any application intervening.  It took a little
effort to do this but the solution of modifying the server is the only
practical solution to your problem.

Someone will say: "But we don't have the source for the X server."  Please
recognize that MIT expected this to be source code available to the users.
The fact that you obtained the compiled version from some vendor of X that
doesn't offer the source code makes your job, perhaps, impossible to solve.
Any other solution will be inadequate for anything but a tightly controlled
X application environment.

Fortunately the X source tree ports to many environments fairly easily
these days.  Writing a DDX, especially for a device that is well designed
for X, is not terribly difficult (yeah, it takes some effort but so do
most things).  A competent C programmer should be able to do a DDX for
a board in a reasonable amount of time.  It'd take a much less than a
year to get the X11R4 server itself into robust shape for one good
programmer -- including your extensions (speaking from personal experience).
Fortunately the task is easily split into pieces so a team can do it.

Yes this is more expensive than buying a onesy or twosy from your
friendly OS/X11 vendor.  But if you've got 100+ machines to outfit, the
price of all this software work suddenly looks much better.

Michael R. Miller
uunet!xzaphod!michael

davis@3d.enet.dec.com (Peter Davis) (06/05/91)

In article <1991Jun04.235308.21266@xzaphod.uucp>, michael@xzaphod.uucp (Michael R. Miller) writes...
>In article <1991Jun3.142241.14494@engage.pko.dec.com> davis@3d.enet.dec.com (Peter Davis) writes:
>>
>>In article <1991May31.163832.22933@auto-trol.com>, marbru@auto-trol.com (Martin Brunecky) writes...
>>>In article <1991May30.191703.11099@engage.pko.dec.com> davis@3d.enet.dec.com (Peter Davis) writes:
>>	.
>>	.
>>	.
>>> 
>>>    Which means you have to properly simulate all enter/leave events. No
>>>    thanx, I am giving up on that one -).
>>>
>>
>>I think what's really needed here are XWarpButton and XWarpKey functions.
>>XWarpPointer only does half the job.  There's really no reason why you should be
>>able to warp the pointer position, but not be able to control the other input
>>device values as well.
> 
>What is needed is to modify the server.
> [elaboration deleted]

Yes, modifying the server (and the O/S kernel/device drivers) is one approach.
However, it has several drawbacks:

   o	It requires server level expertise to develop and maintain.

   o	It must be constantly re-implemented and re-tested against O/S
	and server upgrades.

   o	It's not very attractive to an unsophisticated customer.  ("Here,
	just save your old sources, load this tape, then re-build your O/S
	kernel, and then re-build your X server and you're all set.")

   o	It makes it difficult to quickly test out new input devices.

Perhaps the "correct" solution would be to have X provide a facility for
down-loading server extensions.  This would solve some problems today, such
as how to applications which depend on certain extensions run on servers
which don't have them.  It would also make it easy to implement an extension
once, and have it be insulated from future O/S or server changes.

eli@ima.isc.com (Elias Israel) (06/05/91)

In article <1991Jun5.140807.13727@engage.pko.dec.com>,
davis@3d.enet.dec.com (Peter Davis) writes:
|> Yes, modifying the server (and the O/S kernel/device drivers) is one
|> approach.
|> However, it has several drawbacks:

Hacking the server is really the only approach that's going to do the
right thing with all of the events, even if you were of a mind to try
and duplicate event propagation code in a client (yuk). The only way
that you're really going to get the right behaviour out of the system is
to add your tablet to the server, using the input extension already in
the MIT code.

This approach isn't without it's problems. Getting extension events to
Xt-based clients is -- err --  tricky, as others have already noted.

|>    o	It requires server level expertise to develop and maintain.

True, but as I said, it's the only way to get the job done, really.

|>    o	It must be constantly re-implemented and re-tested against O/S
|> 	and server upgrades.

My guess would be that the input extension will not change very much in
the future. I doubt that it would take a whole engineer's time to keep
up with that extension, or with changes to the extension interface in
the server. (Mind you, this is a *guess*. I don't claim to know what Bob
S. and Keith P. are thinking)

As for changes to the O/S, you'd have to manage those even if you didn't
make a server change. Either way, when your O/S changes, you'll have to
hack drivers.

|>    o	It's not very attractive to an unsophisticated customer.  ("Here,
|> 	just save your old sources, load this tape, then re-build your O/S
|> 	kernel, and then re-build your X server and you're all set.")

Installing a new driver in the kernel is something that our end-users do
all the time. There's no reason why they can't handle it if they have
the right tools. (And the tools I'm talking about aren't even
particularly user-friendly). Having end-users hack the server is a
problem, I'll admit. But if you can send them a fresh binary instead of
telling them to re-build their server, you might be able to get away
with it. Depends on your business model, I guess.

|>    o	It makes it difficult to quickly test out new input devices.

I'm not sure that it would be significantly easier the other way.
Testing devices is difficult. It always has been, whether you're talking
about kernel drivers or X input.

|> Perhaps the "correct" solution would be to have X provide a facility for
|> down-loading server extensions.  This would solve some problems today, such
|> as how to applications which depend on certain extensions run on servers
|> which don't have them.  It would also make it easy to implement an extension
|> once, and have it be insulated from future O/S or server changes.

This isn't possible, as far as I can see. Every time the staff at MIT
has written a new extension, they've found a new area of the server code
that needs to be expanded or modularized, or sometimes just changed to
give extension developers a new hook. It's hard to imagine how you can
completely insulate extensions from changes in the server, or how you
could download an extension, except in the trivial sense of dynamically
loading an object file into a running server. (And even that requires
some trickery.) Extensions are rather intimately tied up in the rest of
the server code; much more so that it might seem at first.

On the other hand, I don't think that the burden of keeping up with the
server changes is likely to be very large in the future. (Putting aside
for a moment the multi-threaded server experiment, the fate of which is
still undecided as far as I know.)

Ordinarily, all of the signs point away from using extensions when
you're trying to solve an X problem. In this case, however, extensions
are the only game in town.

Elias Israel		   | "Justice, n. A commodity which in more or
Interactive Systems Corp.  | less adulterated condition the State sells
Boston, MA		   | to the citizen as a reward for his allegiance,
eli@ima.isc.com  	   | taxes, and personal service."
eli@village.boston.ma.us   |     -- Ambrose Bierce, _The Devil's Dictionary_