CommonSense controller for digital piano project?

[deskman]

DMA - awesome project - kept me up until 4am last night hahaha.

A friend is converting an old piano keyboard to a MIDI controller with optical sensors. This will allow silent play with an awesome wooden grand piano keyboard via a computer Virtual Instrument (VI) program like PianoTeq.

Do you think the CommonSense controller could be easily adapted for that use?

A few parameters:

1- There are 88 keys and 3 pedals on the piano. That's a lot of inputs.

2- 128 MIDI velocities per key (that is the speed a key moves and is used by the VI to calculate volume).

3- Need ultra-precise timing from keyboard to computer (very low latency & very low jitter)

4- Digital piano makers like Yamaha can get latency to about 5 ms with dedicated internal systems (e.g. time from finger's first contact to key until sound emerges from headphone out).

5- Using the Yamaha as just a keyboard outputting to MIDI either with:
---- internal USB interface OR
----via MIDI Din cables then user USB interface

Then running a piano VI program on a decent computer, latency is about the same. This requires ASIO drivers, some tweaking and some luck

6- I suppose optical sensors could be run in analogue or digital scheme. Most digital pianos use 2 or 3 cheap rubber buttons per key as detailed here http://www.kawaivpc.com/en/features/

DIY guy is a Bulgarian programmer who was a beta tester of a cool Itailian VI called PianoTeq. He purchased an old wooden grand piano keyboard (only), some Vishay CNY70 sensors, teensy board, and posted a few concept videos. The thread is here

http://forum.pianoworld.com/ubbthreads. ... 921/1.html

https://youtu.be/ikgZ0o7evnU

https://youtu.be/3b0y96ll1xA

In the second video, you see a big fuzzy white hammer which normally would hit the strings and stops the key movement. You also see an aluminum bar with black rubber which was added by CyberGene to stop the keys in place of the strings.

EDIT- Photo attached with one optical sensor on the right side of the picture key hanging by 4 yellow wires...

If this should be in a different thread or section please feel free to move.

[DMA]

CommonSense will be hard to adapt - there's no notion of speed
But I need to know more about schematics of this thing.

PSoC 5 LP has USB MIDI component (but I see this problem is solved by teensy already) and a good 1Msps ADC. 91 input is not much of a problem - make 3-5 sensor subassemblies, connect by I2C or SPI.

[deskman]

Thanks DMA - appreciate your response.

CommonSense controller could be a great solution but as you note getting the velocity information would be an issue.

Fully bypassing the very slow MIDI transmission speeds is a tough challenge. . .

[DMA]

Can I look at schematics of this all? sensor looks complicated

[DiodeHead]

o man the internals on this pianos are awesome, I would love to see the key mechanism made too, in a way so you can regulate the tension on the keys and have custom tension depending on your hand strength.

that optical sensor looks interesting, and for the midi handling I don't think you would have any problems, I started some while ago a harmonic table (like the axis 64), I thought that was a very interesting thing to have to study music theory.

but as always that is in the todo list, but I remember doing the DIN connector midi implementation and that was easy since you only had to use the Arduino serial port since if I remember correctly, midi is just a serial protocol.

Code: Select all

void notaOn(byte canal, byte nota, byte vol)
{
  Serial.print(144 + canal - 1, BYTE); // byte de estado 144 + canal - 1
  Serial.print(nota, BYTE); // Altura de la nota
  Serial.print(vol, BYTE); // Velocidad o volumen
}

and this is all you needed to send a note. Over USB I bet is more convoluted to use.

regards

[deskman]

I will ping CyberGene at the PianoWorld. He is in Bulgaria so takes a few days to respond due to time differences!

I think he is still noodling on how to best implement this.

____
Most commercial digital pianos use 3 cheap rubber button sensors per key to measure each key. Factory mounting of the sensors is rather precise and I imagine the makers run a quick calibration routine after assembly.

Optical sensors are used by a few of the really expensive commercial digital pianos (think $10k+). To simplify, CyberGene could run one sensor per key analogue. Or run three sensors digital.

In the photo above, the optical sensor is dangling from the yellow wires (it is cut off but if you click the photo you can see it). There should be say a good white surface on the key on the final version to optimise performance of the sensor.

In the photo, you also see three pots on the breadboard; those are for CyberGene's manual calibration. For 88 keys that gets expensive and unwieldy to build. Also difficult to calibrate and recalibrate hundreds of pots manually. So a software solution would be much easier.

____
After keystroke, data needs to arrive at the computer in ~2ms in my view. That would give the computer and virtual instrument software and DAC say ~4ms for processing and output. Some musicians will notice "round trip" latency (as I defined above) in the 6ms range.

I think speed may still be an issue as good players are running well over 10 signals per second. And large chords should be reproduced in parallel. Not sure how much CyberGene will be able to resolve here. He could send data uber quickly to laptop then remap to MIDI but that would take some effort to get fast, low jitter, glitch free.

____
Funny the techno musicians that connected synths to the Atari systems of the mid 80s claimed uber low latency and jitter. Just a rock solid connection among the player and computer that we have a tough time replicating today. Same observation I see here with the typing keyboards. . .

[deskman]

o man the internals on this pianos are awesome

The piano internals are awesome indeed! CyberGene purchased a very old internals from a dead grand piano.

I would love to see the key mechanism made too, in a way so you can regulate the tension on the keys and have custom tension depending on your hand strength.

Interesting point. The idea was to get a professional grade wooden grand piano "action" rather than a replica. Then his professional piano tuner will refurbish the "action" to play like new. The tuner has some ability to regulate this to CyberGene's wishes.

Big surprise was that the ancient wooden action was only 85 keys. But his tuner will just make 3 new ones hahaha.

but I remember doing the DIN connector midi implementation and that was easy since you only had to use the Arduino serial port since if I remember correctly, midi is just a serial protocol...Over USB I bet is more convoluted to use.

There is some manipulation and conversion to get the data into the laptop. Getting it with uber-low latency is the tough part in my view.

[CyberGene]

Hi guys, I've seen you're interested in my project I'll be very short since I am kind of busy these days...

One of the most important parts of my project is that I don't rely on ADC because it is slow, even though the ADC on the Teensy board is pretty fast. I mean it is fast by itself but I need to be able to measure 88 keys and, as you will see further, I also need to measure three separate distance points per key and all that makes it unacceptable for real time piano playing. The sensors I use are CNY70 which is basically a LED and phototransistor. The latter changes its collector-emitter current based on proximity of objects. Basically it is an analog output sensors. I need to have three separate distance points measured by the sensors: strike point (I don't have strings, just a rail which hammers hist) which is the closest distance to the sensor. Another one which is close to the strike point (it's a point after escapement, which in piano mechanics means the point where the hammer detaches from the key mechanism and moves by the momentum). I need to measure the time hammer travels between these two points to calculate its velocity. The third distance is used to determine whether the key is still being held (hammer is raised which on the piano keeps the string vibrating freely; when released the dampers will stop it). So, since I can't used ADC, I need to somehow translate the analog output of the sensor into three binary signals corresponding to each distance. They are "high" at rest and become "low" when the hammer is at a closer distance that the corresponding distance-point (e.g. when the hammer is at the strike point all three binary outputs will be "low").

So, how to do that? Here comes the great suggestion by the other PW member who recommended using a LM339N comparator. Each comparator has 4 pairs of voltage inputs and 4 binary outputs. On each pair you provide a reference voltage and the voltage from the CNY70. If the voltage becomes higher than the reference, then the output changes. And so the output of each CNY 70 is fed to three comparator inputs, there are also three trimpots providing three reference voltages. The big advantage is the comparator is extremely fast! The disadvantage is I need many comparators and so many trimpots... But at lease I solve the problem of speed.

Now, in order to read all the keys, I've grouped the keys into 18 groups of 5 keys each (the last group is of 3 keys, not 5). Each group produces 5 * 3 = 15 binary values that I feed into Teensy digital inputs. In order to select groups, I use octal-bus transceivers SN74LVC245AN (those are chips with 8 inputs, 8 outputs, a direction pin and an on/off pin). Basically a transceiver will repeat the inputs to the outputs in the selected direction (input to the output in my case) and what's important here is that by using the on/off pin I can select which transceivers are active at that moment. Furthermore, the switching is also extremely fast! So, for each group of 5 keys (15 binary values) I use two transceivers (16 inputs, 1 unused). In each scanning pass I have only two transceivers active.

So, I use 18 digital outputs from Teensy to select a pair of transceivers at a time, and I read 15 inputs from the selected pair. That's 33 digital pins of Teensy used. It has 52. I will use some of them to read the triple pedal in a similar fashion.

Hope that helps.

Cheers!

[DiodeHead]

In the photo, you also see three pots on the breadboard; those are for CyberGene's manual calibration. For 88 keys that gets expensive and unwieldy to build. Also difficult to calibrate and recalibrate hundreds of pots manually. So a software solution would be much easier.

what is he calibrating with the pots?? that would be helpful to know to help him find a software solution.

And large chords should be reproduced in parallel. Not sure how much CyberGene will be able to resolve here.

https://www.midi.org/specifications-old ... di-message

since it's a serial protocol to send a chord, no matter how long it is, you just send one note after another, I think some synth implement parallel chords with SYSEX, but they are SYStemEXclusive so what it works for Yamaha could not work for Roland and so on to give an example.

what I find more problematic is making the sustain pedal realistic. without it, the only thing you have to do is send notes with the velocity being how hard your hited it and then send a noteOFF, since in a grand piano even if you leave the key pressed the note won't sustain unless you have the pedal press, isn't it?. I think the sustain pedal is not as easy as hooking it up to an interrupt line and sending a noteOFF when you depress the pedal since even with the pedal they won't sustain forever, right?

[DiodeHead]

haha I'm too slow, some of my questions have been answered while I wrote the post

[deskman]

Hi guys

Thanks for dropping by CyberGene and updating details on your mechanical wood piano project!

I see some people are "turbo charging" sampling rates on ADC on the Teensy to over 300ksps with some easy tweaking outside library. From a practical perspective you will be lower but still get pretty fast! Some samples:

https://forum.pjrc.com/archive/index.ph ... 2-p-2.html

You can do similar ADC tweaking with Arduino to reach 350ksps

https://www.youtube.com/watch?v=glulIeL2lxA&t=0s

https://www.midi.org/specifications-old ... di-message

since it's a serial protocol to send a chord, no matter how long it is, you just send one note after another, I think some synth implement parallel chords with SYSEX, but they are SYStemEXclusive so what it works for Yamaha could not work for Roland and so on to give an example.

We could bypass some MIDI 1980s protocol and go fast or go serial.

what I find more problematic is making the sustain pedal realistic. without it, the only thing you have to do is send notes with the velocity being how hard your hited it and then send a noteOFF, since in a grand piano even if you leave the key pressed the note won't sustain unless you have the pedal press, isn't it?. I think the sustain pedal is not as easy as hooking it up to an interrupt line and sending a noteOFF when you depress the pedal since even with the pedal they won't sustain forever, right?

The good news is that the Virtual Instrument software has good pedaling algorythms. Most commercial digital pianos use some cheap carbon pots for encoding; some of these fail very quickly due to physical contact issues. So a simplified DIY optical sensor scheme also works here for the three pedals; will require some extra calibration at the start.

[DMA]

I don't rely on ADC because it is slow, even though the ADC on the Teensy board is pretty fast. I mean it is fast by itself but I need to be able to measure 88 keys and, as you will see further, I also need to measure three separate distance points per key and all that makes it unacceptable for real time piano playing.

What prevents you from using several MCUs? Teensy is a bad choice - ADC on teensy 2.0 was _incredibly_ sucky. You couldn't get anywhere close to advertised sampling rate.
PSoC can actually scan at advertised 1Msps, because you can drive it from the programmable hardware part. Still, there's more keys than pins so you'll need several MCUs anyway. Not 18 modules though. And at 3 modules in parallel you'll have a decent resolution - 31kHz. If you go to 8bit resolution - closer to 50 even. That's not LM339A's 1.3us, but may be you don't need such precision. You can also measure speed directly - by measuring the distance hammer travels after escapement point is reached. You'll just need 1 scan. I don't know how long the hammer flies though.

Now, in order to read all the keys, I've grouped the keys into 18 groups of 5 keys each

What is your scanning frequency? How often you get 2 events at once - i.e. miss an event? Like "the key was released, now it already striken the string"?

[deskman]

How often you get 2 events at once - i.e. miss an event? Like "the key was released, now it already striken the string"?

Could you kindly clarify this question? I am assuming that there will be just one optical sensor per (note) wooden keystick providing "velocity" data every scan.

[DMA]

How often you get 2 events at once - i.e. miss an event? Like "the key was released, now it already striken the string"?

Could you kindly clarify this question? I am assuming that there will be just one optical sensor per (note) wooden keystick providing "velocity" data every scan.

There's no velocity data. Velocity is calculated.
There is poor man's 2-bit ADC made from 3 comparators on every key. No velocity, only key positions. Positions are "key in flight", "key hit the string" and "key is resting on the sustain bar". Lack of signals means "key is released".
Then there is an 18-input MUX which reads outputs of those ADCs, 5 at a time.
What is the clock frequency of the MUX? How often "key in flight" and "key hit the string" events are detected in a single measuring pass?

Simpler question is "how fast is the fastest keystroke?"
What is the shortest time between "key left escapement" and "key hit the string"?

[deskman]

What is the clock frequency of the MUX? How often "key in flight" and "key hit the string" events are detected in a single measuring pass?

Simpler question is "how fast is the fastest keystroke?"
What is the shortest time between "key left escapement" and "key hit the string"?

Thanks DMA - incredibly helpful inquiry!

I will leave response to CyberGene as he has actual measurements from his wooden keyboard and a better view on his system goals.

[CyberGene]

I see a lot of questions...

Let me first start by saying that I am first of all a high-level programmer by profession, I mostly write in Java which isn't helping me a lot besides the basic coding skills I am entirely new to controllers, low-level programming etc. I am also an advanced amateur at the piano which is helping me in answering questions related to the required dynamics, playing technique, how the piano action works in very precise details and how/when/where to measure velocity, how to regulate the action (a term that's used to denote the intricate balance between all the mechanical parts and settings that ultimately determine how responsive the action is). I am a total noob in terms of electronics and just to have an idea how bad I am, I was recently wondering if I should put the LED resistors at the LED anode or cathode Or whether I can use only a single resistor for all LED-s (those of the CNY70). I didn't know what a comparator is or transceiver... I asked on various forums like this one and I guess it would have been great if I started on this one.

That all is the reason why I am still deciding how to finish the project because I am not very confident at that stuff.

Anyway, here's some answers. The hammer velocity should be measured at string strike point, i.e. a terminal velocity. Since I have decided on optical sensors and not pressure/strain sensors, I have to measure time between two points instead. Due to piano specifics, the only reliable velocity should be measured after escapement point. The hammer usually has about 1mm travel after that point until it hits the string. In my measurements that equates to a duration close to 1ms, maybe a bit less, maybe more... But approximately a velocity of 1m/s. I've read somewhere that velocities up to 2-3m/s can be reached by a piano hammer, depends on how heavy someone bangs, I didn't thrash my action a lot. Furthermore, the 1mm is just a reference point but I haven't regulated the action yet, so it's only visually, might be 0.5mm, might be 1.5mm.

Anyway, the three points I measure are:
1. Key is being depressed at least to some degree (I don't use that for any calculation, only to know when to send key off. On a real piano there are string dampers which are lifted when the key is being pressed halfway, allowing the string to vibrate freely. If you keep the key pressed beyond that point, the string will vibrate until released. Once released I will send key-off). Which is why I need that point. The proper way is to measure the key. I am actually measuring that point at the hammer instead but it doesn't matter since the hammer angle will follow the key press depth in exact fashion for at least half of the way, i.e. until that damper release point. Just for information, after the hammer rebounds from the string, it will hit a spring loaded escapement lever, so the hammer will drop and thus won't follow the key angle but it's still enough and above the damper release point I am interested in, so I am OK. Kawai also do that. Yamaha measure keys separately to know key on/off but I think that's rather overkill.
2. Hammer reached escapement point. I start measuring at that point.
3. Hammer hit the string (in my case the stop rail)

As I said, velocity needs to be measured between 2 and 3. I can't afford measuring below escapement because just for example one can press the key very slowly until it just reaches the escapement point and then very forcefully press the key and be able to throw the hammer. If I started measuring before that, that would have screwed my measurements since a lot of time would have elapsed while the hammer was just being held jut before escapement.

Now, you offer various other solutions and I am totally open to any of them. As I said, I am a total newbie in electronics and controllers, etc. so any help would be appreciated

Why I need the trimpots. Because I need to set those three points. Since those are optical sensors that just hanging above a wooden surface of the hammer, this is very unpredictable in terms of precision. Also, the piano regulation although a fine work isn't an exact science so hammer blow/strike distances can vary. I will use software calibration, but I also need the trimpots to calibrate my poor-man ADC-s. If I use fast ADC instead, then I can rely only on software calibration.

I haven't even ordered the transceivers nor measured any sequential reading to be able to know how fast I can scan. I've only tested POC of a single key I was going to order some transceivers and test a few groups of keys and then would know for sure.

BTW, I have a 10-month daughter so the spare time I have for the project is almost close to zero. But I am in no hurry either.

So any help is appreciated, I'll try to answer when/if I can

Cheers!

(Sorry for my English which isn't probably very good)

P.S. Edited my wording a little

[CyberGene]

So, since I don't see any further replies. Let's assume the highest possible terminal velocity is 2m/s. Escapement to string distance is 1mm. That equates to shortest duration of 0.5ms. In order to detect something like that I should scan at least twice as fast, meaning 4000 times a second. Multiply that by 88 keys so it's 352ksps. Meaning even a poor Arduino can do the task. Am I right? If so, no need for my complicated schematic with so many trimpots, comparators and transceivers.

If all that's true, then the problem I need to solve is how I can feed 88 analog inputs into a limited number of analog pins on my Teensy or Arduino? Maybe I can use that MUX thing (I need to read more about that) but isn't that supposed to work only for digital signal and not analog ones?

[DMA]

Don't think arduino is up to the task - kinetis ADC is shitty. Also you're applying Kotelnikov's theorem wrong - you'll be able to detect max. speed, but you won't be able to discern it from lower speeds. So you'll need oversampling.

Problem of feeding 91 inputs is not actually a problem - just have 3 PSoCs. Forget about arduino and teensy - their ADCs are shitty, there's too little analog inputs. If you're sticking to arduino - your solution is probably optimal. A bit expensive - but for a single device cost doesn't matter much.

[CyberGene]

Thanks, I was thinking about that "double" the sampling rate intuitively but thinking more about that you're right I'll need much more sample points than that to get a good precision.

Very initially, while still not knowing anything about Teensy and Arduino, I was thinking that the solution would probably be solved by some custom PIC controllers - cheap and small, using a few of them, probably linked to a bus... But as I said my knowledge on that is so little and noob I never elaborated on that since people recommended Teensy. I didn't know what PSoC is before you mentioned it. Seems like a nice solution indeed, the only problem is I am not sure if I will manage to do the things, it will require much lower level programming that Arduino/Teensy. I don't know... Do you think it's manageable?

[DMA]

Everything is in C, and all hardware functions are _functions_, not "set this register to that value". See, for example, https://github.com/dmaone/CommonSense/b ... ore/core.c
And programmable hardware is basically legos. The only downside is that development environment is only available for windows.

[HuBandiT]

just a few brief ideas:

- you could go acoustic and put a tiny microphone capsule to be hit by the hammer; potentially behind some individual damping or blocking to prevent capsule damage; and measure hammer speed indirectly from the loudness of the sound/vibration it generates - just like it happens in the real piano
- serial midi (5-pin DIN connector) has a baud rate of 31250, uses 8 bit bytes, I guess 1 start bit and 1 stop bit, so max byte rate is 3125 Hz, one note down message requires 3 bytes (release 2 bytes - average 5 bytes), so that gives you a 625 Hz aggregate note rate. USB 2.0 MIDI will have an 1 kHz frame rate (or theoretically 8 kHz if using USB 2.0 Hi-Speed), however look at the USB MIDI specification to see whether they allow more than one byte per USB frame per MIDI channel - if they do, then you can win some latency here. if not, you can still go multi-MIDI-channel to send notes in parallel. PITA in MIDI organ keyboards where - due to organs being often played in multiple layered voices (registers) - they would have to send several notes per keypress. for even lower latency you could do an Ethernet-based MIDI implementation
- 180 BPM music (not usually played on a piano as it is very fast) is 3 Hz when playing quarter notes, sixteenth (1/16) notes are 12 Hz, (1/32) notes are 24 Hz - but you probably cannot hit keys that fast; the Guiness world record in 2012 was 765 notes per minute which is 12,75 Hz. assuming you have 10 fingers, and hit a separate key with each of them, and repeat that with 12,75 Hz that is a note rate of 127,5 Hz. (it is left as an exercise to the interested reader to ponder what happens when one hits multiple keys with each finger - for a realistic estimate do consider hand size/reach though) finally, you should also consider the case of 4 handed play.
- more on the topic from the very informative Colin Pykett
- if you consider aftertouch - which you should since you have a felt pad under the keys allowing the keys to move slightly in response to varying pressure, and a separate sensor for the position of each key - then that quickly explodes the number of potential MIDI messages sent per second, so you probably will need Ethernet MIDI in that case.
- since you have a position sensor on each key, you should also do key position detection and reporting, for those synthesizers which do take into account the position of the key/dampener; it also enables lovely proportional action organ playingfor upcoming organ synthesizers (most modern - wired/electric action - pipe organs cannot do this and lack this subtlety in their play; only a few can); apologies, I do not know the MIDI details for this - maybe there are none yet, in which case feel free to invent your own, working together with the synthesizer developers
- some midrange microcontrollers offer 10 million 12 bit samples per second, with 50 input channels; for a full 88-key keyboard you could reach about 113 kHz sampling rate _per key_, if you have three sensors per key, you could have a 37 kHz sampling rate _per sensor_ (and maybe you only need two since now you have quite precise position sensing); very conservatively let's halve that (to allow timing for multiplexing, etc.) to reach 18 kHz sampling rate _per sensor_. with your half millisecond hammer travel time that is a minimum of 9-10 samples per hammer travel, and each one of those samples is a 12 bit position value, to which you could fit a linear regression to get a quite precise velocity value (way better than the 7 bit/128 velocity values traditional MIDI allows - luckily recently there is a MIDI extension to 14 bits for velocity values so you won't lose date); and that is only for the fastest/loudest notes - softer notes will have many more samples, offering even more precision.
- do consider that your hammers, after (and probably even before) hitting the target, will do non-trivial mechanical things (compress, bounce back, oscillate sideways, etc.) so prepare for a thorough analysis and modelling of hammer-sensor system behaviour (and multiple redesigns of your sensor arrangement); I recommed max speed sampling of single hammers during development; remember, you have 10 million samples per second; that will allow you to learn a very great deal about the dynamic behaviour of your system, and to develop a pretty accurate model. since your mechanical system is not rigid, and will very slightly change after playing every note, you most certainly will want to prepare for various auto-calibration scenarios.
- you will also want to allow for the player (or the tuner) to calibrate the keys, e.g. in terms of evening out the force/velocity response across the keyboard, etc. interactively (that is, by hitting keys)
- adding configuration possibilities for velocity curves, aftertouch behaviour, etc. are left as an exercise for the software developer
- do consider adding proportional sensing for all three pedals as well (if your mechanism includes pedals)

[HuBandiT]

I just saw there are videos. My further recommendations:

- measure velocity at the original position: the hammer
- ditch the rail (unless it is part of the original mechanism)
- mount a strip of (IR) transparent plastic or glass above the hammers where the strings would be hit; thick enough to get its lower surface (on which the hammers will hit) to your sensor optimal (= highest signal) distance (which is often not zero distance); rigid enough that it does not get bent by the hammers hitting on it from below
- mount the optical sensors on top of that strip, looking down onto the hammer tops
- the sensors will see the approaching hammer tops and continuously measure the distance as they approach and bounce back
- you only need one or at most two sensors per note (resulting higher sampling rate from my example above): one for the hammer, another for the key position; alternatively, depending on the hammer travel and your sensor range, you could be fine with just one sensor, if it sees "deep enough" to the bottom position of the hammer; you might consider a sensor that "sees farther" than your current 5 mm one; measure, calculate, think, ask

[Arakula]

Thanks DMA - appreciate your response.

CommonSense controller could be a great solution but as you note getting the velocity information would be an issue.

Fully bypassing the very slow MIDI transmission speeds is a tough challenge. . .

Not really. MIDI uses 31250 bps with 1 start/1 stop/8 data bits, i.e., roughly 3 bytes per millisecond. Since this is a piano with not exactly a lot of additional controllers, most of the time Running Status can be employed, keeping the time to transmit a single Note On message well below 1ms. Even if you press 30 keys down at once with a brick, transmission of the flurry of events would take less than 20ms. Ironically, if you send single note events over a class-compliant USB MIDI interface, transfer can be slower than the original MIDI from 1982. Complete buffers are a different thing, of course.

[deskman]

Not really. MIDI uses 31250 bps with 1 start/1 stop/8 data bits, i.e., roughly 3 bytes per millisecond. Since this is a piano with not exactly a lot of additional controllers, most of the time Running Status can be employed, keeping the time to transmit a single Note On message well below 1ms. Even if you press 30 keys down at once with a brick, transmission of the flurry of events would take less than 20ms. Ironically, if you send single note events over a class-compliant USB MIDI interface, transfer can be slower than the original MIDI from 1982. Complete buffers are a different thing, of course.

Thanks for coming by Arakula!

Hopefully nobody will be using bricks except in rare fits of frustration hahaha. But we probably have to plan for 10 fingers and 2 feet simulatenously (probably more). And as HuBandiT noted, sometimes two people play together (e.g. teacher and student). So NKRO gets to well over 12, I suppose.

Also, it would be great to get all info to the computer in less than 2ms based on my very rough measurements. There is no good science on latency percpetion; it certainly varies by person and probably improves by training. Regardless, getting "round trip" latency low enough for decent musicians is a challenge for home-gamers.

Not sure how much system needs to be tied to the old midi protocol. And it might be just easier to send data to the computer rapidly then dress it up in software.

[deskman]

- more on the topic from the very informative Colin Pykett

Thanks for your thoughts here HuBandiT. A few quick thoughts.

The mic sensor is an interesting approach but I might have some problems with noise as I use loudspeakers during the day and headphones at night.

For "simplicity", it might make sense to avoid measuring the hammer entirely. That is the approach taken by big players like Yamaha and Kawai. Measuring the key also allows system to be used on regular digital keyboards (which don't usually have hammers).

The calibration and configuration really needs to be done in software. Analogue domain gets very overwhelming.

Thanks for the very helpful links to Colin Pykett. His site is a gold mine. Some superb articles and a great index with more articles:

http://www.pykett.org.uk/picscannermidi.htm
http://www.pykett.org.uk/a_midi_pedalboard_encoder.htm
http://www.pykett.org.uk/completed_work.htm#SubjAction

[DiodeHead]

The mic sensor is an interesting approach but I might have some problems with noise as I use loudspeakers during the day and headphones at night.

then piezos would be a better option, that is what is used in electronic drums. But you would encounter a lot of problems with that too, I used to have a TD-3 and I had to be careful paying because of the crosstalk if you played too energetically.

[Arakula]

Hopefully nobody will be using bricks except in rare fits of frustration hahaha. But we probably have to plan for 10 fingers and 2 feet simulatenously (probably more). And as HuBandiT noted, sometimes two people play together (e.g. teacher and student). So NKRO gets to well over 12, I suppose.

Any normal master keyboard transfers states for all keys. There's no "key rollover". There may be voice stealing going on in the sound generation part, but definitely not in the control part.

Also, it would be great to get all info to the computer in less than 2ms based on my very rough measurements. There is no good science on latency percpetion; it certainly varies by person and probably improves by training. Regardless, getting "round trip" latency low enough for decent musicians is a challenge for home-gamers.

Sure, the faster the signal is in the computer, the better. But the biggest latency adder in this case is still the sound generation part, not the note transfer. This also adds some milliseconds, and getting the complete chain down to an absolutely imperceptible range (for normal human beings; there will always be exceptions) below 3-5ms is ... let's say, "financially challenging".

Not sure how much system needs to be tied to the old midi protocol. And it might be just easier to send data to the computer rapidly then dress it up in software.

If you're planning to use off-the-shelf-software, you'll need to convert to a protocol spoken by the sound generating software at some point; but that doesn't have to be directly in the controller. If you do it in the computer, you can easily send hundreds of simultaneous MIDI messages. A 3GHz multicore machine doesn't even notice the difference.

BTW, it takes ~10ms for the nerve signal from your fingertip that "the key has bottomed out so a sound should start" to reach your brain. Experience just allows you to anticipate that. And it takes ~3ms for a signal from a speaker, or an excited string, at a distance of 1m to reach your ears (well, that is faster with headphones, of course). I wouldn't get overly anxious about miniature timing issues in the single millisecond range.

[deskman]

The mic sensor is an interesting approach but I might have some problems with noise as I use loudspeakers during the day and headphones at night.

then piezos would be a better option, that is what is used in electronic drums. But you would encounter a lot of problems with that too, I used to have a TD-3 and I had to be careful paying because of the crosstalk if you played too energetically.

Interesting that you mention electronic drums. Roland sells some pro electronic drums (including the big round drum pads and a custom computer with LCD screen) that some people are claiming have below 2ms round trip latency (that is time from stick hit to sound output from headphone)! After reading that, I dropped by my local music store and for sure the tound trip latency felt well below the ~5ms I measured on my digital piano.

I felt a great connection with the insturment and it felt 100% like real drums. Sound quality was mediocre so I would guess the connection was uber low-latency and/or low jitter.

That said, I don't know if dropping total round trip latency to 2ms on a digital piano (with VI runnning on a laptop) would be possible or matter as it is such a different instrument from electronic drums. . .

Also, it would be great to get all info to the computer in less than 2ms based on my very rough measurements. There is no good science on latency percpetion; it certainly varies by person and probably improves by training. Regardless, getting "round trip" latency low enough for decent musicians is a challenge for home-gamers.

Sure, the faster the signal is in the computer, the better. But the biggest latency adder in this case is still the sound generation part, not the note transfer. This also adds some milliseconds, and getting the complete chain down to an absolutely imperceptible range (for normal human beings; there will always be exceptions) below 3-5ms is ... let's say, "financially challenging".

Right. Let's assume we will use fast, optimised home computers. Beyond that, we can not change the sound generation speed. I suppose we still have two main areas we can "easily" accelerate:

1. Headphones to reduce reproduction latency of speakers (>3ms depending on distance between speakers and player)
2. Improve transfer speed of key data to computer (particularly for larger chords, which start to face longer delays)

If you're planning to use off-the-shelf-software, you'll need to convert to a protocol spoken by the sound generating software at some point; but that doesn't have to be directly in the controller. If you do it in the computer, you can easily send hundreds of simultaneous MIDI messages. A 3GHz multicore machine doesn't even notice the difference.

CyberGene was a beta tester for PianoTeq and has contact with a developer of Garritan CFX so those guys might be able to provide some optimised tips for their respective software packages.

BTW, it takes ~10ms for the nerve signal from your fingertip that "the key has bottomed out so a sound should start" to reach your brain. Experience just allows you to anticipate that... I wouldn't get overly anxious about miniature timing issues in the single millisecond range.

This is the most interesting part for me. I think timing and perception varies a lot by person and may be improved by training (think F1 drivers, video gamers, musicians). Published "research" is in very early stages so not very useful.

For musicians using a "virtual keyboard", some suspect that Jitter is much more problematic than latency. The importance of jitter is very difficult to prove.

For sure, we recognize the rare musician with a rock-solid internal clock. Maybe we can observe this with say jazz or funk masters playing "in the groove." And electronic pop musicians still muse about the "tight timing" from the 80s MIDI gear; and the old 80s Atari computers.

- At the extreme, huge jitter among notes would make it difficult to play music as the sound "feedback" of every note would be haphazard and confusing.

- Rock musicians can adjust for some latency in large concert areans; but long latency in a virtual instrument at home is irritating and unrealistic.

A close friend ran an interface development team at a major game maker. HIs work focused on minimizing latency and minimizing jitter at system level, let's say among input controllers and output to screens/speakers. I will try to get him to drop by.

[deskman]

So still researching this. Two interesting products noted below:
_____

There was a Texas-based open-sourced MIDI keyboard that was released around 2011 and then a much cheaper version was released around 2017. The 2011 board was foldable for easy transport and well reviewed but super expensive .

https://www.gearslutz.com/board/electro ... ponce.html

2017 version used the PIC24FJ128GC010 and the nifty Vishay optical sensors TCPT1350X01.

Basic concept is on the kickstarter page. Below are some quick schematics and the assembly guide has some great photos of the insides. Really a nifty project.

https://www.kickstarter.com/projects/12 ... controller
https://web.archive.org/web/20170113065 ... sembly.pdf
https://web.archive.org/web/20170111184 ... -guide.pdf
https://web.archive.org/web/20180223074 ... ematic.pdf
https://web.archive.org/web/20170302104 ... matic2.pdf
https://web.archive.org/web/20170302160 ... ematic.pdf
https://web.archive.org/web/20170111232 ... rmware.pdf

The 2017 version had some issues and was terminated after initial distribution

https://www.gearslutz.com/board/electro ... ponce.html
_____

Wooting finished the Kickstarter for their second gaming keyboard. Expected release Winter 2018, still working on tooling now.

Wooting summarise their architecture below (keys are multiplexed into 6 linear groups, microcontroller scans board at 1kHz). More interestingly they got all keys to be analogue at around about 150$.

http://blog.wooting.nl/what-influences-keyboard-speed/

A post-doc researcher in Human-Computer interaction (interesting job!) measured performance improvements with beta firmware:

https://twitter.com/sunjun_kim/status/9 ... 7173860352

[DMA]

wooting texts are a bit bulshitty. like "we use analog input so we can't have matrix". Also they don't need debouncing because IR switches don't bounce. What they need is EMI rejection. Which they can do with jacking up scanning speed to, say, 5kHz and have sub-1ms latencies (which are not that useful because USB limits latency to 1ms, but still).
4ms is kinda long. I was able to make 2ms latency with 4-deep debouncing buffer.

I'm surprised those wooting guys are not on DT. With ideas from CS they could make their keyboard much cheaper and better.