Discussion:
[fullscreen] Problems with mouse-edge scrolling and games
(too old to reply)
Ian Langworth
2014-02-22 01:00:02 UTC
Permalink
Hi everyone,

We're building a console- and native-quality game in the browser using
JavaScript and WebGL. You can see a very early version of the game in this
video:
. It's a realtime strategy (RTS) game,
like StarCraft[1] or Command & Conquer[2], and moving the cursor to the
edge of the screen is the primary way that users move around the map.

We have a single requirement: Moving the cursor to, and possibly past, the
edge of the window or screen should pan the camera in that direction.

In windowed mode there are a couple of problems: Players might click
outside the window accidentally, pixels outside the window are wasted
screen real estate, mousemove events aren't fired when the cursor is
outside the window, and sometimes the cursor moves so quickly that it
misses the area of the window where we do edge detection.

The better option is to go fullscreen with the Fullscreen API, but this has
problems as well. There are certain behaviors that occur when moving the
cursor to the top and bottom of the screen, which I've illustrated in the
following videos using a quick edge-scrolling demo[3]. (Please ignore the
problems with cursor detection in the Windows videos. It only occurred
while the screen recording software was active.):

Chrome, Windows 7
https://www.dropbox.com/s/t9kyo8s5am76ezg/chrome%20win7%20edges.avi
The fullscreen notice appears every time the cursor is moved to the top.

Chrome, Mac OS X
https://www.dropbox.com/s/ke1mr5te2dwgvor/chrome%20osx%20edges.mov
A poor experience with the menu and address bars appearing at the top
and Dock appearing at the bottom.

Firefox, Windows 7
https://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges.avi
No problems with the fullscreen experience!

Firefox, Mac OS X
https://www.dropbox.com/s/0bkdx71ir0yhw88/firefox%20osx%20edges.mov
Menu bar appears at top, along with some mysterious window chrome, and
the Dock appears at the bottom as well.

These behaviors are to remind the user about fullscreen and lessen the
chance of phishing. But, as you might imagine, they are troublesome when
you're in the heat of battle and are trying to crush your opponent. I would
like to solicit the list for opinions on how we can improve the fullscreen
UI for games and give players the best experience possible. Some options
that come to mind:

(A) Provide an alternate fullscreen API or option which provides a more
complete experience, but with more dire warnings to the user. For example,
pass in a flag to requestFullscreen() which gives the page full control
over the screen (no fullscreen reminder notifications, menu bars, or Docks)
and even the keyboard (so games can use the Escape key), but entering this
mode presents a much more intense warning to the user like the invalid
certificate warnings.

(B) Maintain the current API, but put an option in the preferences or
flags which makes the browser's fullscreen mode more complete and without
interruption. Hardcore gamers will likely accept this extra step if it
means getting an optimal experience.

(C) Propose a new Mouse-Edge Detection API, which might solve these
problems and provide games with better cursor detection overall.

I appreciate any and all feedback.

[1]


[2]


[3] http://jsfiddle.net/statico/F8sjw/show/
Florian Bösch
2014-02-22 22:22:39 UTC
Permalink
Pointerlock should solve these problems in the following fashion:

- When the user clicks into the app, request pointerlock
- Use it to give him a cursor drawn by you
- That way you can keep the interaction inside your game and accurately
detect borders etc.

I run http://webglstats.com/ and It also records the availability of
pointerlock. This is relative to everybody who has webgl and it currently
stands at 94.1%.

Caveat, I think Firefoxes implementaiton of the Pointerlock API might not
follow the specification yet to make it possible to have pointerlock
without fullscreen. I know that it's possible to get pointerlock in google
chrome without fullscreen.

Links:

- http://www.html5rocks.com/en/tutorials/pointerlock/intro/
- http://www.w3.org/TR/pointerlock/
Post by Ian Langworth
Hi everyone,
We're building a console- and native-quality game in the browser using
JavaScript and WebGL. You can see a very early version of the game in this
video: http://youtu.be/NiCy5igO9-I . It's a realtime strategy (RTS)
game, like StarCraft[1] or Command & Conquer[2], and moving the cursor to
the edge of the screen is the primary way that users move around the map.
We have a single requirement: Moving the cursor to, and possibly past, the
edge of the window or screen should pan the camera in that direction.
In windowed mode there are a couple of problems: Players might click
outside the window accidentally, pixels outside the window are wasted
screen real estate, mousemove events aren't fired when the cursor is
outside the window, and sometimes the cursor moves so quickly that it
misses the area of the window where we do edge detection.
The better option is to go fullscreen with the Fullscreen API, but this
has problems as well. There are certain behaviors that occur when moving
the cursor to the top and bottom of the screen, which I've illustrated in
the following videos using a quick edge-scrolling demo[3].. (Please ignore
the problems with cursor detection in the Windows videos. It only occurred
Chrome, Windows 7
https://www.dropbox.com/s/t9kyo8s5am76ezg/chrome%20win7%20edges.avi
The fullscreen notice appears every time the cursor is moved to the top.
Chrome, Mac OS X
https://www.dropbox.com/s/ke1mr5te2dwgvor/chrome%20osx%20edges.mov
A poor experience with the menu and address bars appearing at the top
and Dock appearing at the bottom.
Firefox, Windows 7
https://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges.avi<https://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges..avi>
No problems with the fullscreen experience!
Firefox, Mac OS X
https://www.dropbox.com/s/0bkdx71ir0yhw88/firefox%20osx%20edges.mov
Menu bar appears at top, along with some mysterious window chrome, and
the Dock appears at the bottom as well.
These behaviors are to remind the user about fullscreen and lessen the
chance of phishing. But, as you might imagine, they are troublesome when
you're in the heat of battle and are trying to crush your opponent. I would
like to solicit the list for opinions on how we can improve the fullscreen
UI for games and give players the best experience possible. Some options
(A) Provide an alternate fullscreen API or option which provides a more
complete experience, but with more dire warnings to the user. For example,
pass in a flag to requestFullscreen() which gives the page full control
over the screen (no fullscreen reminder notifications, menu bars, or Docks)
and even the keyboard (so games can use the Escape key), but entering this
mode presents a much more intense warning to the user like the invalid
certificate warnings.
(B) Maintain the current API, but put an option in the preferences or
flags which makes the browser's fullscreen mode more complete and without
interruption. Hardcore gamers will likely accept this extra step if it
means getting an optimal experience.
(C) Propose a new Mouse-Edge Detection API, which might solve these
problems and provide games with better cursor detection overall.
I appreciate any and all feedback.
[1] http://youtu.be/Qb0VzbxdP4U
[2] http://youtu.be/l41hG-fVDN4
[3] http://jsfiddle.net/statico/F8sjw/show/
Florian Bösch
2014-02-22 22:39:19 UTC
Permalink
Post by Florian Bösch
Caveat, I think Firefoxes implementaiton of the Pointerlock API might not
follow the specification yet to make it possible to have pointerlock
without fullscreen.
Just checked this against a test I wrote a while ago
http://codeflow.org/issues/pointerlock-test.html and consulted the
pointerlock stats for IE. You can get pointerlock in both firefox and
chrome without fullscreen (you can also get pointerlock and fullscreen
simultaneously). IE seems to support pointerlock in versions that have
webgl, but they seem to have added it only recently so support is still low
(32%).
Thibaut Despoulain
2014-02-23 00:57:35 UTC
Permalink
The issue with pointerlock is that it requires the app to draw its own
cursor instead of the OS cursor, which has at least two significant
drawbacks:

- Significant pointer lag between movement input and actual movement on
screen compared to an OS-drawn cursor (yes, even at 60fps),
- Inability to use DOM elements with mouse events for a game overlay/HUD..

We could work around #2, but #1 is a no-go for a twitch game such as a RTS
where mouse accuracy and responsiveness is so critical.

Something like a different kind of pointer lock that uses the OS cursor and
only prevent the cursor from exiting the window canvas would be optimal.
Post by Florian Bösch
Post by Florian Bösch
Caveat, I think Firefoxes implementaiton of the Pointerlock API might not
follow the specification yet to make it possible to have pointerlock
without fullscreen.
Just checked this against a test I wrote a while ago
http://codeflow.org/issues/pointerlock-test.html and consulted the
pointerlock stats for IE. You can get pointerlock in both firefox and
chrome without fullscreen (you can also get pointerlock and fullscreen
simultaneously). IE seems to support pointerlock in versions that have
webgl, but they seem to have added it only recently so support is still low
(32%).
Brandon Jones
2014-02-23 01:15:28 UTC
Permalink
Pointer lock does indeed provide a workaround for this, but it's worth
noting that with the current architecture of Chrome a software cursor will
experience visible lag, which these types of games are highly sensitive to.
I'm not sure about the latency of Firefox or others, but in general a
hardware cursor is preferable any time you have a visible cursor. As a
result this is still a use case worth considering.

--Brandon
Post by Florian Bösch
- When the user clicks into the app, request pointerlock
- Use it to give him a cursor drawn by you
- That way you can keep the interaction inside your game and
accurately detect borders etc.
I run http://webglstats.com/ and It also records the availability of
pointerlock. This is relative to everybody who has webgl and it currently
stands at 94.1%.
Caveat, I think Firefoxes implementaiton of the Pointerlock API might not
follow the specification yet to make it possible to have pointerlock
without fullscreen. I know that it's possible to get pointerlock in google
chrome without fullscreen.
- http://www.html5rocks.com/en/tutorials/pointerlock/intro/
- http://www.w3.org/TR/pointerlock/
Post by Ian Langworth
Hi everyone,
We're building a console- and native-quality game in the browser using
JavaScript and WebGL. You can see a very early version of the game in this
video: http://youtu.be/NiCy5igO9-I . It's a realtime strategy (RTS)
game, like StarCraft[1] or Command & Conquer[2], and moving the cursor to
the edge of the screen is the primary way that users move around the map.
We have a single requirement: Moving the cursor to, and possibly past,
the edge of the window or screen should pan the camera in that direction.
In windowed mode there are a couple of problems: Players might click
outside the window accidentally, pixels outside the window are wasted
screen real estate, mousemove events aren't fired when the cursor is
outside the window, and sometimes the cursor moves so quickly that it
misses the area of the window where we do edge detection.
The better option is to go fullscreen with the Fullscreen API, but this
has problems as well. There are certain behaviors that occur when moving
the cursor to the top and bottom of the screen, which I've illustrated in
the following videos using a quick edge-scrolling demo[3].. (Please ignore
the problems with cursor detection in the Windows videos. It only occurred
Chrome, Windows 7
https://www.dropbox.com/s/t9kyo8s5am76ezg/chrome%20win7%20edges.avi
The fullscreen notice appears every time the cursor is moved to the top.
Chrome, Mac OS X
https://www.dropbox.com/s/ke1mr5te2dwgvor/chrome%20osx%20edges.mov
A poor experience with the menu and address bars appearing at the top
and Dock appearing at the bottom.
Firefox, Windows 7
https://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges.avi<https://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges..avi>
No problems with the fullscreen experience!
Firefox, Mac OS X
https://www.dropbox.com/s/0bkdx71ir0yhw88/firefox%20osx%20edges.mov
Menu bar appears at top, along with some mysterious window chrome, and
the Dock appears at the bottom as well.
These behaviors are to remind the user about fullscreen and lessen the
chance of phishing. But, as you might imagine, they are troublesome when
you're in the heat of battle and are trying to crush your opponent. I would
like to solicit the list for opinions on how we can improve the fullscreen
UI for games and give players the best experience possible. Some options
(A) Provide an alternate fullscreen API or option which provides a more
complete experience, but with more dire warnings to the user. For example,
pass in a flag to requestFullscreen() which gives the page full control
over the screen (no fullscreen reminder notifications, menu bars, or Docks)
and even the keyboard (so games can use the Escape key), but entering this
mode presents a much more intense warning to the user like the invalid
certificate warnings.
(B) Maintain the current API, but put an option in the preferences or
flags which makes the browser's fullscreen mode more complete and without
interruption. Hardcore gamers will likely accept this extra step if it
means getting an optimal experience.
(C) Propose a new Mouse-Edge Detection API, which might solve these
problems and provide games with better cursor detection overall.
I appreciate any and all feedback.
[1] http://youtu.be/Qb0VzbxdP4U
[2] http://youtu.be/l41hG-fVDN4
[3] http://jsfiddle.net/statico/F8sjw/show/
Florian Bösch
2014-02-23 08:55:12 UTC
Permalink
The issue with pointerlock is that it requires the app to draw its own
cursor instead of the OS cursor
I fully agree with motivation, it is usually preferrable to give the user
an OS-themed cursor (not always, but often).
- Significant pointer lag between movement input and actual movement
on screen compared to an OS-drawn cursor (yes, even at 60fps),
http://codeflow.org/issues/software-cursor.html

My observation from testing on linux is that I can't distinguish latency
for the software cursor from the OS cursor (or not by much anyway) in
google chrome. In firefox however there's noticable lag. Mileage may vary
for other platforms.
- Inability to use DOM elements with mouse events for a game
overlay/HUD.
The test I've written here http://codeflow.org/issues/software-cursor.htmlalso tests mouse event synthesis (as hinted at by an example in the
pointerlock spec) and it works satisfactory in chrome and firefox on linux.
To get all subtleties right one would also have to do most other mouse
events I guess (mouseover, mouseout, mousemove etc.).
Chrome a software cursor will experience visible lag, which these types of
games are highly sensitive to.
Not as much lag as I expected, see this test:
http://codeflow.org/issues/software-cursor.html
I'm not sure about the latency of Firefox or others
Pretty bad, see this test: http://codeflow.org/issues/software-cursor.html
, but in general a hardware cursor is preferable any time you have a
visible cursor. As a result this is still a use case worth considering.
It's not always preferrable (sometimes people want custom cursors). But in
general it is preferrable.


When providing the OS cursor there's several issues to be addressed:

- If you are in fullscreen, capture the cursor to the viewport but show
the OS cursor, that does *NOT* mean you'll want the annoying slide-down
that some browsers provide if you hit the top edge. If you wanted that,
you'd never have restricted the cursor in the first place. I would suggest
as a resolution that if somebody requests cursor restriction, and gets it,
that the slide down is never shown regardless if the OS cursor is shown or
not.
- It should be possible to show and hide the OS cursor separately, but
retain pointerlock. The reason for that is that you might want to avoid
synthesizing your own cursor, but you may also want to keep pointerlock
because your game switched to some UI interaction for instance from an FPS
interaction mode. I would suggest this could be resolved with an additional
API call of the form: document.hideCursor() and document.showCursor() or
something similar. If the sequence is myContainer.requestPointerLock();
document.showCursor(); the cursor is shown, but confined to the container.
Outside of being in pointerlock, these calls would have no effect.
Florian Bösch
2014-02-23 13:58:47 UTC
Permalink
- Inability to use DOM elements with mouse events for a game
overlay/HUD.
The test I've written here
http://codeflow.org/issues/software-cursor.html also tests mouse event
synthesis (as hinted at by an example in the pointerlock spec) and it works
satisfactory in chrome and firefox on linux. To get all subtleties right
one would also have to do most other mouse events I guess (mouseover,
mouseout, mousemove etc.).
I did some more testing on this. Synthetic mouse events are limited in that
they cannot trigger some state like hover styles. Other things like
focus/blur require special handling. It's also left to the synthetic event
creator to synthesize compound events like over/out/enter/leave and their
semantic as supplying the primitive mouse events (click, mousemove) doesn't
automatically synthesize the others.
Brandon Jones
2014-02-23 15:18:24 UTC
Permalink
Several points to add:

- it's possible to theme the OS cursor using custom images with CSS.
https://developer.mozilla.org/en-US/docs/Web/CSS/cursor/url
- The reason the cursor is hidden when the pointer is locked is that some
OSes don't have the ability to report relative mouse movement correctly at
screen edges. This requires the cursors to constantly be reset to the
center of the screen, which obviously would look strange if the cursor was
visible.
- You already mentioned some issues with synthetic mouse events, but
unfortunately it's worse than you suspect. For example: sending a synthetic
click event to a checkbox doesn't actually change it's checked state. (Not
the last time I tried anyway) select controls also have a hard time with
synthetic events, and there's a whole host of other sub rely broken things.
:(

Thanks for the tests!
Post by Florian Bösch
- Inability to use DOM elements with mouse events for a game
overlay/HUD.
The test I've written here
http://codeflow.org/issues/software-cursor.html also tests mouse event
synthesis (as hinted at by an example in the pointerlock spec) and it works
satisfactory in chrome and firefox on linux. To get all subtleties right
one would also have to do most other mouse events I guess (mouseover,
mouseout, mousemove etc.).
I did some more testing on this. Synthetic mouse events are limited in
that they cannot trigger some state like hover styles. Other things like
focus/blur require special handling. It's also left to the synthetic event
creator to synthesize compound events like over/out/enter/leave and their
semantic as supplying the primitive mouse events (click, mousemove) doesn't
automatically synthesize the others.
Florian Bösch
2014-02-23 15:30:42 UTC
Permalink
Post by Brandon Jones
- it's possible to theme the OS cursor using custom images with CSS.
https://developer.mozilla.org/en-US/docs/Web/CSS/cursor/url
Although that doesn't absolve vendors from fixing the latency issue even if
native pointers where to be made available during pointerlock. The reason
is that cursors come in more flavors than an image. For example they could
come in some variety of 3D rendered representations useful for the game in
question.
Post by Brandon Jones
- The reason the cursor is hidden when the pointer is locked is that some
OSes don't have the ability to report relative mouse movement correctly at
screen edges. This requires the cursors to constantly be reset to the
center of the screen, which obviously would look strange if the cursor was
visible.
Isn't that only a concern if you want to capture the cursor, not when you
display the OS cursor?
Post by Brandon Jones
- You already mentioned some issues with synthetic mouse events, but
unfortunately it's worse than you suspect. For example: sending a synthetic
click event to a checkbox doesn't actually change it's checked state. (Not
the last time I tried anyway) select controls also have a hard time with
synthetic events, and there's a whole host of other sub rely broken things.
:(
Is there a motivation not to make it work? Clickjacking?
Florian Bösch
2014-02-24 10:17:21 UTC
Permalink
On Mon, Feb 24, 2014 at 1:16 AM, Thibaut Despoulain
Post by Florian Bösch
http://codeflow.org/issues/software-cursor.html
My observation from testing on linux is that I can't distinguish latency
for the software cursor from the OS cursor (or not by much anyway) in
google chrome. In firefox however there's noticable lag. Mileage may vary
for other platforms.
This is true, but sadly your test runs on an empty animation frame. If
your main thread is doing a lot of work already (barely hitting the 60fps
mark, as it is the case for demanding games), the lag will be much more
perceptible as you will most likely drop a frame every now and then.
I regard dropping below 60fps as an application defect for a realtime
interactive application where the users input is time critical.

*Reasons things drop below 60fps*

- You trigger GC-ing
- Your JS main thread takes up more than 16ms
- Your GPU processing time takes up more than 16ms

*Detecting the problem*

- You'd detect GC-ing by measuring the rate at which
requestAnimationFrame is fired. Gaps that appear periodically are typically
GC related.
- JS main thread time can be accurately measure with performance.now()
- GPU time will be measurable with EXT_disjoint_timer_query
http://www.khronos.org/registry/webgl/extensions/EXT_disjoint_timer_query/

*Solving the problem*

- Incremental GCs for JS will help. But they're not here yet, so one way
you can do something right now is to be very careful about the use of [],
{}, function(){}, new, DOM, createElement, innerHTML etc. It is possible to
eliminate GC caused frame drop almost entirely that way.
- Set yourself a time budget for the JS-main thread (that's well below
16ms) that you'll want to reach even on the lower-end of the expected
hardware. First try to keep everything below that budget in tests. If
something simply cannot be done in that time, split up into multi-frame
processing or shuffle it out to a webworker. Finally, if you still run into
the problem, keep constant tap on the JS time per frame, and when you run
over it reduce some processing (as in LOD, quality degradation etc.)
- EXT_disjoint_timer_query is not yet available, but once it does become
available you can use it to perform accurate testing on the lower end of
the hardware spectrum to identify rendering issues. And you can also use it
to dynamically react to pending performance issues by dialing down the LOD.

*Critical features for this use-case vendors have to offer, and soon*

- Showing the OS cursor, but containing it into a bounding region as
determined by a dom element (such as body, other elements)
- Implementing incremental GCs
- Implementing EXT_disjoint_timer_query
- Reducing input -> output lag

I will point out that I have consistently called for fixes in these areas
during the last 2-3 years on the various discussions/specs/tickets on these
topics. So it's not surprising to me that these are issues now somebody
actually tries to make a real world, real time application in a browser.
I'm very glad that Artillery is doing the fine work of pushing this
boundary. I'd also like to ask the vendors to go the last mile as quickly
as feasible. Solving these issues is in the end, not just crucial for
realtime applications. Web applications get a lot of criticism for not
being able to compete in quality with native applications. This stems in
large part, from the fact that it is hard to make web applications as
responsive and snappy as native applications.
Glenn Maynard
2014-02-24 16:18:20 UTC
Permalink
Post by Florian Bösch
Post by Florian Bösch
http://codeflow.org/issues/software-cursor.html
My observation from testing on linux is that I can't distinguish latency
for the software cursor from the OS cursor (or not by much anyway) in
google chrome. In firefox however there's noticable lag. Mileage may vary
for other platforms.
This is true, but sadly your test runs on an empty animation frame. If
your main thread is doing a lot of work already (barely hitting the 60fps
mark, as it is the case for demanding games), the lag will be much more
perceptible as you will most likely drop a frame every now and then.
I regard dropping below 60fps as an application defect for a realtime
interactive application where the users input is time critical.
First, mouse input is always time-critical, even if the application itself
is not. Even for a spreadsheet, the mouse must still move responsively,
since pointing devices are difficult to use with any delay. Some games
only run graphics at 30 FPS (like it or not), but the pointer if any must
still update at 60 FPS.

Second, system-driven pointers allow better responsiveness than is possible
at the application level. They can process mouse input and update the
mouse cursor right when the backbuffer is flipped, giving the lowest
possible latency. Applications (especially web apps) simply can't do that.

Going forward this could be even more severe. For example, AirPlay video
streaming has significant video delay, but they could send mouse movement
over a sideband and draw it on the rendering device, allowing responsive
mouse movement where it would otherwise be impossible.

It's not the application's job to keep the mouse cursor responsive, it's
the system's. Hiding the system mouse cursor and drawing one manually is
always a bad idea.
--
Glenn Maynard
Florian Bösch
2014-02-24 16:30:59 UTC
Permalink
Post by Glenn Maynard
It's not the application's job to keep the mouse cursor responsive, it's
the system's. Hiding the system mouse cursor and drawing one manually is
always a bad idea.
That's a wonderful argument. And now we look at an FPS game, or an Oculus
Rift title, or something that requires something else than a picture cursor
like say, an on terrain selection, a bounding box selection, a 3D ray
picking selection/cursor, or anything like that.

Always a bad idea, sure. How about you think about that again hm?
Glenn Maynard
2014-02-24 16:40:44 UTC
Permalink
(More reasons: it's very likely that you'll end up implementing a cursor
with different motion and acceleration, a different "feel", than the real
mouse cursor. It also breaks accessibility features, like mouse trails.)
Post by Florian Bösch
Post by Glenn Maynard
It's not the application's job to keep the mouse cursor responsive, it's
the system's. Hiding the system mouse cursor and drawing one manually is
always a bad idea.
That's a wonderful argument. And now we look at an FPS game, or an Oculus
Rift title, or something that requires something else than a picture cursor
like say, an on terrain selection, a bounding box selection, a 3D ray
picking selection/cursor, or anything like that.
Always a bad idea, sure. How about you think about that again hm?
This doesn't seem to relate to the discussion, which is about mouse
pointers.
--
Glenn Maynard
Florian Bösch
2014-02-24 16:50:46 UTC
Permalink
Post by Glenn Maynard
(More reasons: it's very likely that you'll end up implementing a cursor
with different motion and acceleration, a different "feel", than the real
mouse cursor. It also breaks accessibility features, like mouse trails.)
Oh I agree, if your usecase fits a mouse cursor of the style that the OS
offers, it's definitely preferable to have the OS mouse cursor. And I
distinctly remember arguing prior to the pointerlock specification that it
would be immensely useful to have the ability to show the cursor, but
capture the pointer inside an area. And I was pointed, at the time, to the
solution to draw a software mouse cursor... So now what we're having this
discussion, I find appropriate to have pointed out as a possibility (to
draw the software cursor), which met some, let's call them "difficulties".
Now, I also just made a nice list which mentions 4 things to do, among
them, add this ability. I consider this now settled, because I think we all
agree on this. We might just need to agree on the limitations as they
partain to relative mouse movement reporting when you're showing the OS
cursor. And I think that's relatively easy to agree on, you can't rely on
relative motion outside of the constrained area if you show the OS cursor.
Post by Glenn Maynard
This doesn't seem to relate to the discussion, which is about mouse
pointers.
It is about input -> output, essentially. Intput as it comes from your
pointing device, and output, as it reflects in your application. The OS
mouse cursor is but one example of such reflection. There are many more
usecases that would like to benefit from a low input -> output latency, one
example of which is the variety of "software cursors" which may take the
shape of a picture, or something else completely. But moreover there are
usecases such as viewing controls that have the exact same need, notably
FPS shooters, for instance, or Oculus rift titles and so forth. So, it
matters a great deal to the, because you can alleviate some of the issues
while simultaneously solving a whole host of other issues.
Vincent Scheib
2014-02-24 18:07:43 UTC
Permalink
Thank you Ian for the clear use case write up (and to Artillery for
progress on a great product), Florian Brandon and Glenn for discussion &
contributions.

As Florian mentioned, we did discuss this use case in 2011 and at that time
I attempted to summarize the situation in the pointer lock spec FAQ [1].
Please do read that discussion of the related 3 topics of clipping, motion
while clipped, and raw data.

One way to move forward on those topics is to prototype functional
solutions on Windows, Mac, and Linux and illuminate a practical solution
that browsers may adopt. I did an investigation in 2011 and did not find a
solution. For clipping only I believe Mac was the only platform I didn't
find a solution for. Windows has ClipCursor() and Linux has XGrabPointer().
Once we know we can implement the functionality, we can discuss how to
express this in an API.

[1]
https://dvcs.w3.org/hg/pointerlock/raw-file/default/index.html#why-bundle-all-functionality-hiding-cursor-providing-mouse-deltas-instead-of-using-css-to-hide-the-cursor-always-providing-delta-values-and-offering-an-api-to-restrict-the-cursor-movement-to-a-portion-of-the-web-page
Florian Bösch
2014-02-24 18:37:05 UTC
Permalink
Windows has ClipCursor() and Linux has XGrabPointer(). Once we know we can
implement the functionality, we can discuss how to express this in an API.
Would using Quarz CGWarpMouseCursorPosition work where you'd clamp the
passed position into the desired rectangle?

https://developer.apple.com/library/mac/documentation/GraphicsImaging/Reference/Quartz_Services_Ref/Reference/reference.html#//apple_ref/c/func/CGWarpMouseCursorPosition
Vincent Scheib
2014-02-24 18:41:15 UTC
Permalink
Post by Florian Bösch
Windows has ClipCursor() and Linux has XGrabPointer(). Once we know we
can implement the functionality, we can discuss how to express this in an
API.
Would using Quarz CGWarpMouseCursorPosition work where you'd clamp the
passed position into the desired rectangle?
https://developer.apple.com/library/mac/documentation/GraphicsImaging/Reference/Quartz_Services_Ref/Reference/reference.html#//apple_ref/c/func/CGWarpMouseCursorPosition
I believe no, because it would allow the pointer to escape the region
before being warped back, permitting escape if clicked at that time as well.
Florian Bösch
2014-02-24 18:47:00 UTC
Permalink
Right, so you'd CGAssociateMouseAndCursorPosition(false) and then use
CGWarpMouseCursorPosition or CGDisplayMouseCursorToPoint to move it where
you want it to be, but clamped inside the rect. As long as you keep pumping
the event loop separately that does this (as fast as possible) it shouldn't
be perceptively different from an OS cursor.
Post by Vincent Scheib
Post by Florian Bösch
Windows has ClipCursor() and Linux has XGrabPointer(). Once we know we
can implement the functionality, we can discuss how to express this in an
API.
Would using Quarz CGWarpMouseCursorPosition work where you'd clamp the
passed position into the desired rectangle?
https://developer.apple.com/library/mac/documentation/GraphicsImaging/Reference/Quartz_Services_Ref/Reference/reference.html#//apple_ref/c/func/CGWarpMouseCursorPosition
I believe no, because it would allow the pointer to escape the region
before being warped back, permitting escape if clicked at that time as well.
Brendan Eich
2014-02-24 17:47:18 UTC
Permalink
Post by Glenn Maynard
It's not the application's job to keep the mouse cursor responsive,
it's the system's. Hiding the system mouse cursor and drawing one
manually is always a bad idea.
Agreed!

In the same vein, programmers cannot avoid GC pauses without relying on
pause-free or at least incremental GC (which BTW some browsers' JS
engines have already, e.g., SpiderMonkey in Firefox), or as a real
alternative, cross-compiling C or C++ for example via Emscripten, to
allocate heap memory from a typed array.

Florian, your goals are good, but the means to those ends must involve
better runtimes or compilers -- not on JS programmers working harder to
avoid GC while still somehow allocating objects frequently and even
implicitly.

/be
Thibaut Despoulain
2014-02-24 18:11:43 UTC
Permalink
@Florian:
That is very well put. 

Trust me, we are well aware of time budgets and GC pauses. The thing is that even when planning frame budgets and object allocations ahead, you still have external parameters you cannot really control (ie. Another tab running flash or some other intensive task, another software running in the background, etc.). This is why an OS-drawn cursor is so critical (in our use case that is, since we rely on a simple 2d cursor styled with CSS and not a 3d one)

Furthermore, when targeting low-end hardware, a 30fps render loop is enough for a decent graphical experience, while being completely mediocre and unplayable on the input side.

As for your last point, "Critical features for this use-case vendors have to offer, and soon”, we completely agree and would love to see those features implemented sooner than later.

-- 
WebGL guru @Artillery
Mail: ***@artillery.com
Tweet: @BKcore
From: Brendan Eich Brendan Eich
Reply: Brendan Eich ***@secure.meer.net
Date: February 24, 2014 at 9:47:24 AM
To: Glenn Maynard ***@zewt.org
Subject:  Re: [fullscreen] Problems with mouse-edge scrolling and games
Post by Glenn Maynard
It's not the application's job to keep the mouse cursor responsive,
it's the system's. Hiding the system mouse cursor and drawing one
manually is always a bad idea.
Agreed!

In the same vein, programmers cannot avoid GC pauses without relying on
pause-free or at least incremental GC (which BTW some browsers' JS
engines have already, e.g., SpiderMonkey in Firefox), or as a real
alternative, cross-compiling C or C++ for example via Emscripten, to
allocate heap memory from a typed array.

Florian, your goals are good, but the means to those ends must involve
better runtimes or compilers -- not on JS programmers working harder to
avoid GC while still somehow allocating objects frequently and even
implicitly.

/be
Florian Bösch
2014-02-24 18:19:15 UTC
Permalink
Post by Brendan Eich
Post by Glenn Maynard
It's not the application's job to keep the mouse cursor responsive, it's
the system's. Hiding the system mouse cursor and drawing one manually is
always a bad idea.
Agreed!
Like I say, some usecases are fine with OS cursors. But that doesn't mean
that somehow, vendors are absolved from improving input -> output latency
issues even if pointerlock is updated to allow OS cursor showing, which I'm
all for. There are a lot of usecases that involve pointing devices, and
pointing metaphors, or view controls, virtual helmets, and so forth, that
cannot properly function with a high input -> output latency. For this
reason it's imperative not only to address the ability to make the OS
cursor visible, but also to continue working on low latency input -> output.
Post by Brendan Eich
In the same vein, programmers cannot avoid GC pauses without relying on
pause-free or at least incremental GC (which BTW some browsers' JS engines
have already, e.g., SpiderMonkey in Firefox), or as a real alternative,
cross-compiling C or C++ for example via Emscripten, to allocate heap
memory from a typed array.
Florian, your goals are good, but the means to those ends must involve
Post by Brendan Eich
better runtimes or compilers -- not on JS programmers working harder to
avoid GC while still somehow allocating objects frequently and even
implicitly.
I agree that things aren't today how they should be for realtime
applications with GCs. And it's true that GCs are getting better. But, it
is the status of today, that a JS programmer has to work harder to make a
glitch/stutter/jerk free realtime applications. A better GC can improve
this situation. However, that doesn't mean that you can forget about GCing
and frame budgets. A realtime programmer will always have to be conscious
not to overload the GC even if it's incremental. Because when the
incremental GC would not manage to get rid of the garbage faster than it is
produced, it would have to resort to more drastic pauses to rectify that
situation. Fortunately the act of being GC conscious for a non incremental
GC, and for an incremental one, is very similar. You try to avoid
triggering it. So in that, you can start that work today, it will not be in
vain once GCs get better in some far flung future.
Glenn Maynard
2014-02-24 19:21:56 UTC
Permalink
Post by Florian Bösch
Like I say, some usecases are fine with OS cursors. But that doesn't mean
that somehow, vendors are absolved from improving input -> output latency
issues even if pointerlock is updated to allow OS cursor showing, which I'm
all for.
Only if low latency is important to the application. It's very important
for some games (rhythm games), moderately important for some (first-person
games controlled with a mouse, which in turn is more sensitive than with a
gamepad), and not very important at all for others (turn-based strategy).
How much time to spend on this is a business decision, since this can be
very time-consuming and can require visual tradeoffs.

There are a lot of usecases that involve pointing devices, and pointing
Post by Florian Bösch
metaphors, or view controls, virtual helmets, and so forth, that cannot
properly function with a high input -> output latency.
For this reason it's imperative not only to address the ability to make the
Post by Florian Bösch
OS cursor visible, but also to continue working on low latency input ->
output.
True, but tangental. :) (My last job was making music rhythm games, which
are more sensitive to consistent framerates than any other game genre I
know of--one dropped frame can wreck someone's game--so I'm sympathetic to
the cause.)
Post by Florian Bösch
The better option is to go fullscreen with the Fullscreen API, but this
has problems as well. There are certain behaviors that occur when moving
the cursor to the top and bottom of the screen,
I think that going fullscreen is the right approach, since locking the
mouse into the window while not fullscreen is really weird and rare, at
least in Windows. By going fullscreen, this hooks into the same UI design
to allow the user to "escape". Even if this was supported in a window,
there'd still have to be some UI to tell the user how to exit, which could
end up having the same problem.

I've been annoyed by the edge-of-screen browser behavior too. It's a part
of the screen where you might want to put something, like navigation UI. I
haven't come up with a better solution, though. I don't think having a
"stronger" fullscreen mode that asks the user for more permission will fly.
Browsers try very hard to avoid asking for special permissions--people
will just agree without reading it, then won't know how to escape from the
app.

I think that for your use case of edge scrolling, having a fullscreen
notice appear at the top is OK (if a little ugly), as long as it's
transparent to mouse events so you can still see the mouse moving around
(or else you might see the mouse move to 20 pixels from the top, then never
see it actually reach the top, so you'd never start scrolling). Menus and
address bars appearing seems like a bug. That makes sense for the
fullscreen you get by hitting F11 in Windows or Command-Shift-F in OSX, but
application fullscreen should just act like a game, and keep as much as
possible out of the way.
--
Glenn Maynard
Florian Bösch
2014-02-24 19:32:13 UTC
Permalink
Post by Glenn Maynard
I think that going fullscreen is the right approach, since locking the
mouse into the window while not fullscreen is really weird and rare, at
least in Windows.
It's quite common for games to have a cursor, grab the pointer and not be
fullscreen. Of course most games that allow for this, use software cursors,
and are apparently not having much problems with it.
Post by Glenn Maynard
By going fullscreen, this hooks into the same UI design to allow the user
to "escape". Even if this was supported in a window, there'd still have to
be some UI to tell the user how to exit, which could end up having the same
problem.
A sidenote, if you have more than one monitor, going fullscreen will not
lock the pointer on screen, obviously. It's not terribly common perhaps to
have more than one monitor, but it's also not that rare.


I've been annoyed by the edge-of-screen browser behavior too. It's a part
Post by Glenn Maynard
of the screen where you might want to put something, like navigation UI. I
haven't come up with a better solution, though. I don't think having a
"stronger" fullscreen mode that asks the user for more permission will fly.
Browsers try very hard to avoid asking for special permissions--people
will just agree without reading it, then won't know how to escape from the
app.
I think that for your use case of edge scrolling, having a fullscreen
notice appear at the top is OK (if a little ugly), as long as it's
transparent to mouse events so you can still see the mouse moving around
(or else you might see the mouse move to 20 pixels from the top, then never
see it actually reach the top, so you'd never start scrolling). Menus and
address bars appearing seems like a bug. That makes sense for the
fullscreen you get by hitting F11 in Windows or Command-Shift-F in OSX, but
application fullscreen should just act like a game, and keep as much as
possible out of the way.
For a fast paced game where you might click and select and do whatnot,
having a slidedown from the top of the window when you hit the border is
not acceptable. People will click it accidentally a lot for instance when
doing a selection. Not being able to offer a fullscreen button in the game
is also bad UX. You end up with explanations for the user to "Please press
F11 to get into fullscreen". You should never have to explain to a user
what ritual he has to perform, if instead you can trigger that action
without the ritual.
Thibaut Despoulain
2014-02-24 20:43:44 UTC
Permalink
Exactly. Having an “exit from fullscreen” bar or some OS auto-hide elements like docks and task bars popping up  at the edges when you play an intense twitch game is simply horrendous. We have several playtests feedbacks arguing in that direction.

Not to mention that a lot of our alpha players have multi-monitors configurations where the mouse is “escaping” on the secondary monitor, even while full screened, which is a terrible experience.

-- 
WebGL guru @Artillery
Mail: ***@artillery.com
Tweet: @BKcore
From: Florian Bösch Florian Bösch
Reply: Florian Bösch ***@gmail.com
Date: February 24, 2014 at 11:32:13 AM
To: Glenn Maynard ***@zewt.org
Subject:  Re: [fullscreen] Problems with mouse-edge scrolling and games
On Mon, Feb 24, 2014 at 8:21 PM, Glenn Maynard <***@zewt.org> wrote:
I think that going fullscreen is the right approach, since locking the mouse into the window while not fullscreen is really weird and rare, at least in Windows.
It's quite common for games to have a cursor, grab the pointer and not be fullscreen. Of course most games that allow for this, use software cursors, and are apparently not having much problems with it.
 
 By going fullscreen, this hooks into the same UI design to allow the user to "escape".  Even if this was supported in a window, there'd still have to be some UI to tell the user how to exit, which could end up having the same problem.
A sidenote, if you have more than one monitor, going fullscreen will not lock the pointer on screen, obviously. It's not terribly common perhaps to have more than one monitor, but it's also not that rare.
 

I've been annoyed by the edge-of-screen browser behavior too.  It's a part of the screen where you might want to put something, like navigation UI.  I haven't come up with a better solution, though.  I don't think having a "stronger" fullscreen mode that asks the user for more permission will fly.  Browsers try very hard to avoid asking for special permissions--people will just agree without reading it, then won't know how to escape from the app.

I think that for your use case of edge scrolling, having a fullscreen notice appear at the top is OK (if a little ugly), as long as it's transparent to mouse events so you can still see the mouse moving around (or else you might see the mouse move to 20 pixels from the top, then never see it actually reach the top, so you'd never start scrolling).  Menus and address bars appearing seems like a bug.  That makes sense for the fullscreen you get by hitting F11 in Windows or Command-Shift-F in OSX, but application fullscreen should just act like a game, and keep as much as possible out of the way.

For a fast paced game where you might click and select and do whatnot, having a slidedown from the top of the window when you hit the border is not acceptable. People will click it accidentally a lot for instance when doing a selection. Not being able to offer a fullscreen button in the game is also bad UX. You end up with explanations for the user to "Please press F11 to get into fullscreen". You should never have to explain to a user what ritual he has to perform, if instead you can trigger that action without the ritual.
Ian Langworth
2014-02-26 01:11:24 UTC
Permalink
Thank you all. This is wonderfully helpful.
Post by Vincent Scheib
One way to move forward on those topics is to prototype functional
solutions on Windows, Mac, and Linux and illuminate a practical solution
that browsers may adopt. I did an investigation in 2011 and did not find a
solution. For clipping only I believe Mac was the only platform I didn't
find a solution for. Windows has ClipCursor() and Linux has XGrabPointer().
Once we know we can implement the functionality, we can discuss how to
express this in an API.
- The reason the cursor is hidden when the pointer is locked is that some
OSes don't have the ability to report relative mouse movement correctly at
screen edges. This requires the cursors to constantly be reset to the
center of the screen, which obviously would look strange if the cursor was
visible.
An OS-level visible cursor with pointer lock would be the holy grail. Maybe
we can spend some time revisiting the implementations of these to prove out
the functionality and push the spec forward a little.

Thibaut Despoulain
2014-02-24 00:16:47 UTC
Permalink
Post by Florian Bösch
http://codeflow.org/issues/software-cursor.html
My observation from testing on linux is that I can't distinguish latency
for the software cursor from the OS cursor (or not by much anyway) in
google chrome. In firefox however there's noticable lag. Mileage may vary
for other platforms.
This is true, but sadly your test runs on an empty animation frame. If your
main thread is doing a lot of work already (barely hitting the 60fps mark,
as it is the case for demanding games), the lag will be much more
perceptible as you will most likely drop a frame every now and then.

The reason the cursor is hidden when the pointer is locked is that some
Post by Florian Bösch
OSes don't have the ability to report relative mouse movement correctly at
screen edges. This requires the cursors to constantly be reset to the
center of the screen, which obviously would look strange if the cursor was
visible.
Would that prevent vendors from implementing a pointer "edge grabbing"?
(preventing the cursor to exit the window but keeping the OS cursor as-is
while inside of it)
Post by Florian Bösch
Post by Brandon Jones
- it's possible to theme the OS cursor using custom images with CSS.
https://developer.mozilla.org/en-US/docs/Web/CSS/cursor/url
Although that doesn't absolve vendors from fixing the latency issue even
if native pointers where to be made available during pointerlock. The
reason is that cursors come in more flavors than an image. For example they
could come in some variety of 3D rendered representations useful for the
game in question.
Post by Brandon Jones
- The reason the cursor is hidden when the pointer is locked is that some
OSes don't have the ability to report relative mouse movement correctly at
screen edges. This requires the cursors to constantly be reset to the
center of the screen, which obviously would look strange if the cursor was
visible.
Isn't that only a concern if you want to capture the cursor, not when you
display the OS cursor?
Post by Brandon Jones
- You already mentioned some issues with synthetic mouse events, but
unfortunately it's worse than you suspect. For example: sending a synthetic
click event to a checkbox doesn't actually change it's checked state. (Not
the last time I tried anyway) select controls also have a hard time with
synthetic events, and there's a whole host of other sub rely broken things.
:(
Is there a motivation not to make it work? Clickjacking?
Thibaut Despoulain
2014-02-24 00:19:34 UTC
Permalink
Also: At the moment we're using CSS cursors to give visual feedback on
in-game hover/action states which works pretty well and is a breeze to
implement.


On Sun, Feb 23, 2014 at 4:16 PM, Thibaut Despoulain
Post by Florian Bösch
http://codeflow.org/issues/software-cursor.html
My observation from testing on linux is that I can't distinguish latency
for the software cursor from the OS cursor (or not by much anyway) in
google chrome. In firefox however there's noticable lag. Mileage may vary
for other platforms.
This is true, but sadly your test runs on an empty animation frame. If
your main thread is doing a lot of work already (barely hitting the 60fps
mark, as it is the case for demanding games), the lag will be much more
perceptible as you will most likely drop a frame every now and then.
The reason the cursor is hidden when the pointer is locked is that some
Post by Florian Bösch
OSes don't have the ability to report relative mouse movement correctly at
screen edges. This requires the cursors to constantly be reset to the
center of the screen, which obviously would look strange if the cursor was
visible.
Would that prevent vendors from implementing a pointer "edge grabbing"?
(preventing the cursor to exit the window but keeping the OS cursor as-is
while inside of it)
Post by Florian Bösch
Post by Brandon Jones
- it's possible to theme the OS cursor using custom images with CSS.
https://developer.mozilla.org/en-US/docs/Web/CSS/cursor/url
Although that doesn't absolve vendors from fixing the latency issue even
if native pointers where to be made available during pointerlock. The
reason is that cursors come in more flavors than an image. For example they
could come in some variety of 3D rendered representations useful for the
game in question.
Post by Brandon Jones
- The reason the cursor is hidden when the pointer is locked is that
some OSes don't have the ability to report relative mouse movement
correctly at screen edges. This requires the cursors to constantly be reset
to the center of the screen, which obviously would look strange if the
cursor was visible.
Isn't that only a concern if you want to capture the cursor, not when you
display the OS cursor?
Post by Brandon Jones
- You already mentioned some issues with synthetic mouse events, but
unfortunately it's worse than you suspect. For example: sending a synthetic
click event to a checkbox doesn't actually change it's checked state. (Not
the last time I tried anyway) select controls also have a hard time with
synthetic events, and there's a whole host of other sub rely broken things.
:(
Is there a motivation not to make it work? Clickjacking?
Loading...