Multitouch control surface

[DSDlover]

As a long time engineer/producer, I love the convenience of working in the box and mixing in a hybrid environment. Aside from sound quality arguments which I would prefer not to open this forum up to, it seems to me the main challenge remains the user interface. Great advancements have been made in control surface technology in the form of the Digidesign ICON format and the Euphonix MC Mix series. But new technologies in multitouch are becoming more and more readily available at a very inexpensive price.

for those who are interested:

Jeff Han demos his breakthrough touchscreen | Video on TED.com

YouTube - Emulator for Traktor Pro running on big multitouch screen.

FAMILIES - Overlay | DISPLAX - Interactive Systems

Some of you may have seen the new DP ipad control surface.

I am interested in a redesign of the software products that many of us use and love for true interactivity in a multitouch environment that gets away from the keyboard and mouse approach and brings the software to a console like experience where the eyes and the hands are in the same place(phew wow, that's one serious run on sentence!). Theoretically, this should allow the brain to spend more energy around the auditory experience as opposed to trying to learn tactically where plugins are mapped to physical controls.

I would like to start a conversation about what users might find to be useful or challenging, including haptic issues and solutions.

Please don't pollute this forum with reasons why it can't be done or conspiracy theories about why such and such company will never allow it. But constructive criticisms about such technology and its usability are encouraged.


Thank you!

[TRA]

Originally Posted by DSDlover

As a long time engineer/producer, I love the convenience of working in the box and mixing in a hybrid environment. Aside from sound quality arguments which I would prefer not to open this forum up to, it seems to me the main challenge remains the user interface. Great advancements have been made in control surface technology in the form of the Digidesign ICON format and the Euphonix MC Mix series. But new technologies in multitouch are becoming more and more readily available at a very inexpensive price.

for those who are interested:

Jeff Han demos his breakthrough touchscreen | Video on TED.com

YouTube - Emulator for Traktor Pro running on big multitouch screen.

FAMILIES - Overlay | DISPLAX - Interactive Systems

Some of you may have seen the new DP ipad control surface.

I am interested in a redesign of the software products that many of us use and love for true interactivity in a multitouch environment that gets away from the keyboard and mouse approach and brings the software to a console like experience where the eyes and the hands are in the same place(phew wow, that's one serious run on sentence!). Theoretically, this should allow the brain to spend more energy around the auditory experience as opposed to trying to learn tactically where plugins are mapped to physical controls.

I would like to start a conversation about what users might find to be useful or challenging, including haptic issues and solutions.

Please don't pollute this forum with reasons why it can't be done or conspiracy theories about why such and such company will never allow it. But constructive criticisms about such technology and its usability are encouraged.


Thank you!

This is awesome stuff. Multitouch is the future of computing. How many years until keyboards are no longer the norm? If Steve Jobs gets his way it'll be sooner than later.

[sparqee]

What I want is a touch screen that has a kind of dynamic braille. Imagine a screen where every pixel can raise or lower itself by half a millimeter. Where a virtual button is depicted on the screen those pixels raise up to give the fingers tactile feedback.

I've played with music interfaces (virtual mixers, synth programming screens, etc.) on touchscreens and the lack of tactile feedback is a show stopper for me. In other words: touchscreens fail to stimulate my sense of touch. Smooth unyielding glass forces me to stare at the screen more intensly than I do with a regular LCD screen while using a mouse. Since I need to judge my movement exclusively with visual feedback I need to see the control but my finger is in the way of the control I'm trying to look at. Catch 22. Perhaps it might help if touching a point on the screen zooms in that section thereby making the control more easily adjustable (like the iPhone does when you touch a spot in a text edit screen).

I really am looking forward to the next generation of touchscreen technology! I just want it to be more touchable!

edit: Hmm... this looks interesting: http://www.ubergizmo.com/15/archives...n_devices.html

[DSDlover]

Sparquee - you make an excellent observation about the tactile feel. I definitely think that "zoom to function" will be a necessary part of any GUI integrated with multitouch. As well a "pickup where left off" functionality so that parameters don't jump to where ever the user happens to have their hands.

Thanks for the braille link! That looks like it could really have promise if they can effectively get it to work. Brilliant!

Keep the thoughts and links coming....

[nucelar]

WTF?!!?!
Is there some cosmic thought-crosstalk going on?? I have been thinking about a hybrid between braille display and touchscreen just today, I googled braille devices, made some concept doodlings then I open GS and THERE'S A THREAD ABOUT IT! Amazing...
In fact I've been thinking about how only the faders could be both tactile and haptic, and how it could be simplified..
But everything seems to be incredibly expensive.
An 80-character braille display (480 dots) costs almost 8k. These things use tiny electomechanical solenoids which are individually addressable.
But the info in this this link is very promising:
New Technology Brings Blind Computing into 21st Century | Tech News Daily
IMO we are still some years away from multitouch screen AND haptic.

OTOH what I'd really like to see is multitouch editing of audio regions, like the editing tools in protools, but with natural multitouch gestures.

[sparqee]

Development of new tech is always expensive. Early prototypes are typically cost prohibitive and often done for the sake of public proof of concept. An important roll we might play is in conceiving of mass market opportunities of a technology that will suit our needs. We want a tactile multi-touch screen for music production. Not a big global market there. The new tech has to compete with both hardware (i.e. analog mixers and HW DAW control surfaces) as well as the now standard ITB mouse & LCD screen environment. However, this is a computer interface technology we're talking about so there will be many much broader applications if it can be made. The real money will be made in these other applications (i.e. mass market consumer electronics) so we need to find something in development and help promote it for our use as well as those mass market uses. That will bring in investment capitol and get the technology moving from the lab into people's hands.

Responsive, functional touch screens have been a dream of mine since the 90's. It wasn't until the past few years that I came to realize that tactile feedback is going to be a necessity for my user satisfaction and I think once such a thing exists most people are going to demand it as well. People don't miss it now because they don't know any better but once the virtual buttons on their iPhone can be felt with the finger tip... there will be no going back.

As a side note; many people in the music industry aren't aware that Ray Kurzweil invented one of the first text to braille scanning devices. Because of low demand they were pretty much hand made and thusly quite expensive. The target audience was small but the reality of blind people being able to read any book in the library was incredibly exciting. Maybe we can get his genius mind interested in our dream.

[gsilbers]

damn. isnt there the cheaper version of this tech?

ipad with "air display" seems cool but there is lag and faders are small. (small screen)


there is also this
KEYTEC is a leading touch screen manufacturer since 1987.* We produce award winning add-on touch screen, integrated touchscreen LCD/CRT monitors, built-in touch screen kit for system integrators, large-size IR touchscreen, laser touch interactive dev

we have a mac touch screen here at the studio but its so unresponsive and slow that we rather use the mouse.

[DSDlover]

Absolutely Sparquee. The haptic issue is one that is a common concern and perhaps luxury considering the challenges/convenience. One of my concerns is how do you treat the surface so the natural grease on your finger tips do not cause resistance and a stuttered response to input.

would it be reasonable to accept a touch screen sub 1k price range and be able to add on, say an MC Mix for another 1k for real faders, but as a scalable solution? (hiding the faders for instance from the screen if one had real faders to interact with, or just the bank that one is working with)

do we think there should be a plug and play standard that all software protocols adhere to (zooming, pinching, etc) or should manufacturers of the software be responsible for innovating their own solutions at the peril of interoperability issues (think key commands in Logic vs Protools vs Nuendo)?

Are there any special issues regarding soft synths? I know that one of the problems I see in Logic for instance is the graphic interface and parameter sets are often incredibly complex, think ES2 or Structure.

I love this braille concept, but in the absence of this technology being affordable to a broad based musical demographic, does anyone feel like there are definite issues that could be tailored to working around this problem?



Gsilbers - do you know what type of touch screen is on the mac you have? Latency is obviously a huge problem if it exists in any form. I remember when AT&T developed the first digital mixing desk controlled from an analogue SSL via the recall system. Because the system relied on the recall system to constantly scan, updates took seconds (like 5-10) to hear the results and it was kind of a joke. Obviously we are not talking about the same problem here, but I'd be curious to have a benchmark of unacceptable latency where it's just easier to use the keyboard and mouse.

[DSDlover]

I don't know why that emoticon is in my message - but point being I'm concerned about fingers jumping across the screen because of oils on the skin - maybe this is or is not a problem

[sparqee]

I currently use an AlphTrack for its motorized fader, a Griffin PowerMate knob to zoom vertically and horizontally, a custom made midi controller with 16 digital encoder knobs and 64 buttons to do track selection and control the UAD 4K channel strip plugin (my go to plugin typically inserted on most tracks). I also use extensive shortcuts on my Qwerty keyboard and I'm thinking about setting up Cubase command mapping for my 88 note Yamaha keyboard (e.g. hitting C3 on my Yamaha might mute insert 3 on the selected track, etc.)

As you can see, I like physical controllers but I also value space efficiency. My studio is small and I prefer to have just one set of physical controlls that affect the selected track rather than having many redundent channels like a HW mixer (I really just don't have the space for that). If I were going to keep my current setup and make just one major change it might be to have one large touchscreen LCD monitor set at a 45 degree angle (like my old drafting desk). It would have to have lightning fast response to touch and would have a good auto zoom. I would still probably use my AlphTrack fader as I just can't imagine a touch screen fader that would feel & work as well. I would still use my custom 4K plugin controller but would use the touch screen to edit other plugins and do basic DAW manipulation.

on a side note; I was just playing with the touch pad on a new Sony Vaio and it has a somewhat bumpy surface that I really like. Much more satisfying to use than the smooth surface of my MacBook Pro touch pad.

[Geert van den Berg]

Originally Posted by DSDlover

I am interested in a redesign of the software products that many of us use and love for true interactivity in a multitouch environment that gets away from the keyboard and mouse approach and brings the software to a console like experience where the eyes and the hands are in the same place

For a console like experience you don't need multitouch but a good board with faders IMHO. And such controllers already exist. It does not feel natural trying to grab a virtual fader with no feedback, with a real fader you can feel its position without looking at it.

[champ]

Originally Posted by Geert van den Berg

For a console like experience you don't need multitouch but a good board with faders IMHO. And such controllers already exist. It does not feel natural trying to grab a virtual fader with no feedback, with a real fader you can feel its position without looking at it.

Touchscreen can't replace faders but using touch screens for drawing automation, EQ setting, fast routing etc. can be pretty useful.

[slaves666]

to bad this never worked out...

[Rockshamrover]

I don't really want touch screen as it just aint natural. It would end up as a big game of twister

I'd like a big mouse mat thing that lets me control by touch everything I see on the screen, without having to take my eyes off the screen.

It would be operated using both hands for maximum er integration with all the parameters on screen and would be really cheap

[matt thomas]

Originally Posted by slaves666

to bad this never worked out...

I had one of those for a while, it was great, when it worked, which unfortunately wasn't very often..

matt

[yeloocproducer]

I read on one of the mac secret type sites awhile ago that Apple has a patent out on a touchscreen that raises up with textures, like you guys are describing.

This was at least a year or two ago, but I don't have a link. They're on it.

[clonewar]

The touchscreen monitor technology is there.. the biggest holdup to this is on the DAW side. DAW's are coded to respond to mouse input and can only handle one mouse click event at a time, they need to be updated to accept multiple input events simultainiously to work like we want them to with touch screens.

There's one piece of audio software that I know of that supports this now.. SAC (Software Audio Console). It's a virtual FOH mixer from the guy that makes SAW Studio, and he's said that the multi touch technology is coming to SS.

[sparqee]

Originally Posted by clonewar

The touchscreen monitor technology is there.. the biggest holdup to this is on the DAW side.

The responsiveness of touch screens I've used are not up to my expectations. There is too much latency for the most part and making tightly controlled small adjustments is difficult. Mixing on a touch screen is like trying to mix on a console while wearing gloves.

I really consider touch screens to still be in their infancy (at least from a professional, accuracy critical point of view) and I think it's well worth everyone's time to keep pressing for better touch screen technology. If we allow the consumer electronics level of "acceptability" to drive development then we're going to be stuck with using iPhones to control our DAW's.

For me multi-touch isn't even an issue until someone can come up with a way of making usably responsive virtual faders and even then, my style of writing automation doesn't require 8 fingers on 8 sliders. I'm happy to adjust one fader at a time.

p.s. that video above is a perfect example of the latency I'm talking about. If you watch his fingers and the sliders you can see that the sliders lag behind noticeably. You'll also notice how carefully he has to tap the smaller buttons. This kind of careful manipulation leads to strain. It's far more tiring to use a touch screen than real hardware. With a real console you're constantly resting your hands and fingers (however so lightly) on the surfaces of the desk, faders, buttons, etc. This in my opinion is the greatest hurdle to overcome. Your work environment needs to be comfortable and ergonomic and I don't find current touch screens to be either.

[DSDlover]

Thanks for the continued input. For those who are not interested in multitouch or think it can't be done, this is not the forum for you.

I grew up on SSL, Neve VRs, Neve 8068, and array of other popular analog consoles, so I am well aware of the benefits of the real thing. I still use these consoles very often, but assigning budgets for even major label projects (which is all I do) to only work this way is increasingly becoming impossible due to the current music business climate. In fact, if it were my way, we'd all still be using tape and consoles and forcing musicians to actually perform instead of asking the engineer to make them sound like they can play - BUT, I digress. Reality is a different story. The challenge of replicating that experience at a lower cost is becoming necessary for engineers and beneficial to musicians who lack funding and have great ideas.

I currently own a Digidesign/AVID dCommand and in the past have owned the Logic/Mackie control and Steinburg Houston. And I have used the Euphonix MC Mix quite a bit. Before any of these "solutions" presented themselves, I tried Wacom tablets. None of these solutions have proved to be satisfactory.

As a Grammy winning engineer with the ear of a number of manufacturers, I am trying to collate a list of concerns to provide guidance for a product that would solve the problems presented by the new In The Box environment we now find ourselves in, and which I am sure will continue to proliferate.

The obvious reason for multitouch is bringing the eyes and hands together for a more console like experience. The second obvious reason is that no hardware controller can be versatile enough to satisfy all the parameters of every plugin. If you've ever used Logic with a MC Mix, you know what I mean! Try mapping out Structure to knobs and faders! Likewise, with the dCommand and dControl, non standardized plugins generally are mapped very poorly. Even for standardized plugs, the direction and placement of the parameters are not consistent. Also frustrating that standard plugins may have important parameters not originally conceived of by the hardware manufacturer that are not available on the control surface. If you have used Waves SSL, and API compressor and EQs with ICON series controllers you know about the consistency issues. You can assign specific parameters to faders on the ICON series which is fantastic if you need to automate a parameter, but still requires a setup step which interrupts the creative process.

I understand that faders as touch screen may be unsatisfactory for some and for those who require that, plenty of solutions exist. Obviously resolution on screen at the moment is probably insufficient and would require a "zoom to function" to work well in a touchscreen environment.

What excites me about mulitouch is malleability, low cost of manufacture (ie cheap for you and me), and low cost of maintenance.

Keep the comments coming.

Anyone have any thoughts on pop up displays, screen sets, standardizing physical movements to commands, etc?

[DSDlover]

Great video Clonewar! Thanks for that.

Quote from Sparqee:
"p.s. that video above is a perfect example of the latency I'm talking about. If you watch his fingers and the sliders you can see that the sliders lag behind noticeably. You'll also notice how carefully he has to tap the smaller buttons. This kind of careful manipulation leads to strain. It's far more tiring to use a touch screen than real hardware. "

Sparqee, do you envision a kind of solution that might help with these concerns? I understand the latency issue and agree completely that it is unacceptable. I wonder if the latency you see on the screen is just feedback or actual recognition.

I think your criticisms are 100% on. can you envision software features that would overcome these hurdles?

[elambo]

Originally Posted by clonewar

DAW's are coded to respond to mouse input and can only handle one mouse click event at a time, they need to be updated to accept multiple input events simultainiously to work like we want them to with touch screens.

That would be a helluva update for many DAWs.

[sparqee]

Originally Posted by DSDlover

Great video Clonewar! Thanks for that.

Sparqee, do you envision a kind of solution that might help with these concerns? I understand the latency issue and agree completely that it is unacceptable. I wonder if the latency you see on the screen is just feedback or actual recognition.

I think your criticisms are 100% on. can you envision software features that would overcome these hurdles?

I wish that I did have some detailed programming suggestions on how to improve touch screen hardware and the associated drivers but my experience in software development has nothing to do with hardware. I post my comment simply as input to those that have the skills to make my world a little bit happier and I appreciate all of their hard work. thumbsup

My vision for TS (touch screen) DAW UI goes something like this: I touch a button to open up a plugin and that plugin opens. Even if I leave my finger on the screen no zoom should appear. When I touch a knob on the plugin the plugin zooms up with my finger centered on the value pointer of the knob. The knob is 2" across and I can easily move my finger in a circle to rotate the knob and set the value with precision. If I need more precision I can place two finger tips on the knob and it goes into a "fine control" mode (similar to holding the shift key while setting a parameter with a mouse). I can then move my finger to another plugin knob to adjust that value or I can tap the screen with two fingers to close the zoom or with three fingers to close the plugin.

I'm already a big fan of using multiple monitors and I can imagine that I would have my TS angled in front of me for easy access and another monitor above and behind (in a more traditional angle) for viewing the entire project window. One possible scenario might be that the TS in non-zoom mode actually mirrored the main screen so that I can reach out and touch anything but once the TS goes into zoom mode I can still look up at the main project screen and see everything. I don't want to have to be zooming in and zooming out all the time just to look at my project window. I suppose if the TS was big enough (perhaps 2'x3') then maybe the zoom mode wouldn't block too much of the screen. <shrug>

I like to rest my fingertips on my control surface but I could probably get used to curling my fingers enough that my nails would rest on the TS and not trigger touch response. I might need to wear something over my thumb though. Wow! Here's a crazy idea; wouldn't it be cool if the TS could recognize a finger tip from a finger nail and you could use your nail for fine adjustments? Yeah, I'm starting to get excited about the possibilities of multi-hand, multi-finger gestures. Not necessarily for simultaneous adjusting of multiple parameters but more as shortcut methods of accomplishing common tasks. Those of us that play piano & guitar are used to moving fingers independently to produce chords and multi-touch gestures are similar in concept really.

[DSDlover]

Thanks Sparqee.

This is the kind of response I'm looking for, if others would like to contribute with ideas like this. I don't expect anyone to post programming code or even have coding skills. These ideas should just be what would make a multitouch experience enjoyable enough to sway anyone into saying, "you know, I didn't think I would ever want to work this way, and now I can't think of working any other way".

Great stuff, keep the ideas coming

Sparqee, I have had very similar thoughts about "zoom to function" where a small parameter zooms in for precise control and then can be put away for macro functions/view. of course there would need to be some kind of gesture method of shift/option/command swiping tracks for selection, copy, paste, etc

as far as where one might rest their hands, perhaps a bumper rest like most consoles have would be sufficient to rest hands and/or arms - clearly some elbow rest would be nice. some kind of screen lock function would be good as well.

I mean there are various options in terms of scalability that make this idea really cool. So if one is on a tight budget, they can buy a 23" screen, but maybe this can be setup in a modular fashion as well. For those who want mulitple screens so if feels like a large format console, expansion could be the solution. One could buy more 23" screens or just a much larger piece of glass. The manufacturer I posted in my initial thread, Displax, features a "Skin" product which is available from 30 to 116" (almost 10' - that's definitely large format console size). I can see sizes even being set up for film dubbing rooms. For those who need real faders, a real fader pack (or mulitple), like MC mix could be set up in front of the screen and a screen set could be setup to hide faders from the screen that would normally have the active bank displayed. A company, such as Argosy, I'm sure would be more than happy to provide a solution for those who want mutliple screens, fader packs with elbow bumpers in place.

[DSDlover]

Originally Posted by clonewar

The touchscreen monitor technology is there.. the biggest holdup to this is on the DAW side. DAW's are coded to respond to mouse input and can only handle one mouse click event at a time, they need to be updated to accept multiple input events simultainiously to work like we want them to with touch screens.

There's one piece of audio software that I know of that supports this now.. SAC (Software Audio Console). It's a virtual FOH mixer from the guy that makes SAW Studio, and he's said that the multi touch technology is coming to SS.

yeah I'm not at all familiar with this product or know anyone who's ever used it, but then I'm a studio guy, so...

having said that, it appears that this touch screen is $500. I mean, 500 bucks?! that's the price of an iPad without the bells and whistles...

[emkay]

Originally Posted by nucelar

WTF?!!?!
Is there some cosmic thought-crosstalk going on?? I have been thinking about a hybrid between braille display and touchscreen just today, I googled braille devices, made some concept doodlings then I open GS and THERE'S A THREAD ABOUT IT! Amazing...
In fact I've been thinking about how only the faders could be both tactile and haptic, and how it could be simplified..
But everything seems to be incredibly expensive.
An 80-character braille display (480 dots) costs almost 8k. These things use tiny electomechanical solenoids which are individually addressable.
But the info in this this link is very promising:
New Technology Brings Blind Computing into 21st Century | Tech News Daily
IMO we are still some years away from multitouch screen AND haptic.

OTOH what I'd really like to see is multitouch editing of audio regions, like the editing tools in protools, but with natural multitouch gestures.

First time I've read the word "haptic". Looked it up. Amazing technology. Thanks for introducing this to me....

[emkay]

Originally Posted by emkay

First time I've read the word "haptic". Looked it up. Amazing technology. Thanks for introducing this to me....

A nice compromise with both tactile and touch technologies. But at a premium at approx $8,000 US!


SmartAV » Tango

[emkay]

Originally Posted by DSDlover

yeah I'm not at all familiar with this product or know anyone who's ever used it, but then I'm a studio guy, so...

having said that, it appears that this touch screen is $500. I mean, 500 bucks?! that's the price of an iPad without the bells and whistles...

Unfortunately looks like it only supports the Windows platform.
Just checked out the site. Sent a message asking if it's compatible with Logic? Then noticed the system requirements....oh well

[DSDlover]

Originally Posted by emkay

Unfortunately looks like it only supports the Windows platform.
Just checked out the site. Sent a message asking if it's compatible with Logic? Then noticed the system requirements....oh well

emkay, no such platform currently exists for mac or pc that supports any common audio programs. MOTU has an iPad app that controls Digital Performer, but it appears to be pretty basic.

It may be possible for a third party to create a controller that is Eucon based or for the manufacturers themselves to recode their interfaces to be multitouch ready.

The purpose of this forum is to vet ideas on how to make a platform that users would be excited to use

[DSDlover]

Originally Posted by sparqee

on a side note; I was just playing with the touch pad on a new Sony Vaio and it has a somewhat bumpy surface that I really like. Much more satisfying to use than the smooth surface of my MacBook Pro touch pad.

I agree that some sort of film over glass, such as a "bumpy" textured plastic surface could be useful to minimize resistance of hands against glass. the surface next to the physical faders on a dCommand is one such idea.

interestingly as it relates to latency response, the 3M product has a latency of 15 ms and the Displax product has a latency of 17ms.

interestingly "visual signals take about 50 milliseconds to travel from the eye to the brain" as quoted from:

Brain, not eyes, composes colors

in fact this is rather fast as normal within tolerance latencies for pattern shift visual evoked responses during EEG brain tests are between 90ms-140ms

Auditory response on the other hand is as fast as 3ms from ear drum contact to brain stem reception as measured by EEG testing. Interesting eh?!

(if people want references, I'll have to find the medical papers, but trust me, these measurements are accurate)

Granted when you move a physical fader, you are shaving off 15-17ms (assuming that say an SSL G series computer or Flying Faders running on Windows 3.0 on a 486 computer-hahahaha, aka gerbils running inside, actually is recording and playing back the Flying Faders on servo motors with fishing tackle/VCA data instantaneously - a rather bold assumption!!!). But perhaps multitouch automation systems could take into account, this 15-17ms latency, post write automation pass and shift automation accordingly.

[sparqee]

I was reading this wikipedia entry: Touchscreen - Wikipedia, the free encyclopedia and found it interesting that the University of Glasgow found in studies that haptic feedback increased speed of touchscreen data input by 20%, reduced errors by 20% and lowered the user's cognitive load by 40%. I assume that "lowered the user's cognitive load by 40%" means that users had to pay less strict attention to the process of using the screen. For example I can type without looking at my logitech keyboard but there's no way I can do that with a TC, for that matter I can even type on a virtual keyboard faster with a mouse than with a TC.

I remember the membrane switches on the DX7 and how people complained about the lack of satisfying "click" when you pressed one but those bottoms gave more feedback than a touch screen. I always thought that the real problem with those buttons is that you couldn't feel the edges so you had to spend too much time *looking* at them to make sure your finger was in the right place.... but I digress.