Requesting help: New Model F w/ dead keys

[1coin3lives]

I have a problem with a New Model F and I’m hoping the community can help.

Since the keyboard arrived, it has had an issue wherein the top row of keys (1-9, etc.) is dead. I’ve been back and forth via email with Joe attempting to troubleshoot it, but so far to no avail.

I’ve disassembled it, reflowed solder to every weak spot around the connection between the PCB and keyboard (it had a number of poor solder joints out of the box), but that’s made no difference.

Now I’ve taken it apart down to the PCB and have tested it using Pandrew and a single flipper.

Here’s what I get (screenshots attached)

- With no flipper applied, as soon as the Signal Level Monitor starts, there are numbers for all keys except the top row and one key in the second row.
- With the flipper applied to a key, the signal numbers change across many keys on the board, but the top row and one key in the second row - those remain at 0.
- After removing the flipper, the signal numbers continue to change, with the same “0” response from that top row and one key in the second row

Unfortunately, I don’t know what any of that means, and it sounds like Joe doesn't either, since he sent me here to ask.

So I’m throwing myself on the mercy of those more knowledgable.

Can anyone provide suggestions or interpret the Pandrew results?

[Muirium]

Anyone got the matrix diagram for these keyboards? It looks like a whole broken row to me. But the underlying logic may not match the physical layout.

Broken on the PCB or on controller or somewhere in between? That’s what you must work out with a multimeter.

Shame these boards ship out with such collossal faults as this. And I don’t mean rhetorically!

[BuGless]

Do any EE's have recommendations ...

In order to be able to help out solving these kinds of problems, we'd need some more info from Ellipse; what would help are:
- A schematic which includes every labeled keypad (capacitive sense part).
- A PDF version of the PCB traces with different colours per layer, all superimposed.

[Muirium]

Are Ellipse’s PCBs open source?

Also, a question for OP: do you have a key installed on that lonely red position on the second row? I’m guessing it’s the Q key so you should! But there are many pads on Model Fs for optional keys, especially around Shift and Return. When stabilisers are there instead, the pads correctly show no signal. Like this:

This seems unlikely for your Q key though!

[1coin3lives]

Also, a question for OP: do you have a key installed on that lonely red position on the second row? I’m guessing it’s the Q key so you should!

If I understood the question, yes there's a 'Q' key on the physical keyboard but when I took this Pandrew snapshot I had it stripped down to the controller and PCB, using a single flipper to actuate keys. So no, I never placed the flipper on the 'Q' key position. And like the numeric row, that position (not sure if it's Q or not but that makes sense) always reads 0.

[Muirium]

Ah, you tested every sense pad directly? Very thorough.

If you still have the PCB out, I’d suggest trying to follow the matrix visually to see if all of those dead keys have a single point of failure. Do they all share the same electrical row?

[1coin3lives]

The first row (with the majority of dead keys) appears to be a separate connection from the second row "Q" that also shows up as dead. Picture attached to illustrate. I already reflowed solder on both of those connections, on both sides of the PCB (you can see my small sharpie dot marks by the ribbon cable).

[Muirium]

Hmm. Maybe, that’s the *column* side of the PCB. The other set of pads will be hooked up with a perpendicular set of connections to complete the matrix.

Note: matrices are often strange compared to the basic physical layout.

[BuGless]

Judging from the PCB layout, it seems that all keys that register zero all the time are connected to either ribbon connector pin 9 or 10 at the top (the two marked with the black marker, counting from the left).

So most likely, the signal on those two pins is not reaching the processor; or the two pins are being shorted (with each other).
The latter being the most likely explanation, actually. Check the resistance between the two with connector and processor board connected.

[pandrew]

By far the most likely explanation is that both rows are shorted to ground. The short could be happening for various reasons. It could be as simple as bad soldering somewhere, but also, check if you still get those zeros even when testing with the keyboard out of the case, there's a known common fault whereby sometimes some solder-joints pierce through electrical/kapton tape, and end up shorting out against a case. The common remedy is to just add a couple more layers of tape.

There can also be other things causing this, in order of decreasing likelyhood:

• As mentioned above, rows could be shorted to ground.
• The positive input of two of the comparators could be disconnected. Or the biasing resistor pack pins could be disconnected. if the floating pin potential remains below the minimum value the DAC can output, then it will always show zero. What happens if you temporarily with a screwdriver short a good row to a bad row while watching signal level monitor? Does the good row go to zero, or the bad row go high, or perhaps neither? If the good row goes to zero, then you probably have a short to ground. If the bad row goes high, then the biasing resistor pack is not making contact with that row. If nothing changes, then the problem is somewhere beyond the comparator, or perhaps the comparator input could be disconnected.
  Try shorting here a good row to a bad row:

  

  try_shorting_here_as_an_experiment.png (14.5 KiB) Viewed 8477 times

• (this next one only if you're using firmware which was built before December) It's possible two of the comparator outputs are disconnected somewhere close to the microcontroller, or perhaps that the pull-ups on the comparator output are disconnected. Can you confirm you're using freshly compiled firmware?
• two negative input(s) on the comparator could be disconnected. Normally with something like that I'd expect randomness, perhaps all 1023, not zeros, but maybe there's some weak pull-up in the comparators that pulls them high, resulting in a 0 output, so I guess it's possible. Check these solder joints visually:

  

  check_these.png (17.22 KiB) Viewed 8477 times

• Two outputs of the comparators could be shorted to ground. What happens if you temporarily short a good comparator output to a bad comparator output with a wire? Do the zeros turn into non-zeros? or some other non-zeros turn into zeros? Note; it's okay to short these outputs for experimentation because the comparator is open-collector:

  

  what_happens_if_you_short_these_opamp_outputs.png (16.08 KiB) Viewed 8477 times

• If you've soldered with the keyboard plugged in, then depending on your soldering equipment, a ground loop could potentially kill an comparator input. Never solder with plugged in electronics. This is just a general warning. I am assuming you had this exact problem before you reflowed anything, in which case you can't blame soldering.
• static electricity could have fried part of an comparator chip or an input of an atmega. Extremely unlikely, very far fetched, but I guess it's technically possible.

For reference the wcass-drawn controller board schematics mostly match original xwhatsits: https://static.wongcornall.com/ibm-caps ... _schem.png
Only differences are those related to usb-c resistors and such, and the ordering of the rows is shuffled.

Do you have a multimeter to check things with?

[Ellipse]

Thanks pandrew, very helpful. While the factory and I both 100% test every key before shipping, these keyboards do get bounced around in shipping unfortunately. It does look like there's a short to ground if the numbers display 0, as I learned from pandrew. Usually when I see all the numbers around 110 or something and they don't change when any key is pressed, it could indicate another problem. Maybe some of the solder got bridged between two connections, either on one or the two sides of the controller or one of the two sides of the large capacitive PCB. I have done this before over the past couple years while doing QC, and the pandrew utility is great for helping catch these issues.

[1coin3lives]

Thanks Pandrew. Some follow ups:

1. Shorting the two example pins seems to bring other connections to 0. See attached pic which was taken while the two pins were shorted.

shorting U4 pins.jpg (164.76 KiB) Viewed 8274 times

2. I've been testing out of the case, including the utility screen shots I posted before and the ones I'm posting now.
3. I definitely didn't solder with the keyboard plugged in.
4. I don't know how to verify or replace the firmware but I'll explore that.
5. The solder joints on U4 look fine to me. At the very least they're consistent with all the other solder joints.
6. Shorting the top two U4 pins creates additional "0" values. See attached screen grab.

Shorting RP4 pins.jpg (164.5 KiB) Viewed 8274 times

[1coin3lives]

Also, resistance between the two pins I have marked on the PCB reads 198k with the keyboard connected.

[pandrew]

Shorting the two example pins seems to bring other connections to 0

Can you confirm you shorted these two pins together to get the above result?
download/file.php?id=74716

Cause your resulting screenshot doesn't look like what I would expect the additional zeros are not related to any of the two pins I said short together, in fact the additional zeros are coming from one more pin to the left. Could it be that the additional zeros happened not due to electrical shorting, but due to physically moving something? Ar the middle numbers (current signal value) sticky after releasing the short? (note: the middle number is current value, the top number is the maximum value since signal level monitor had been open, and the bottom value is minimum since it's been open.)

Also could you put your multimeter into DC voltage measurement mode, put the keyboard into row/dac tester or column tester mode (do it in order for the columns to become quiescent, and you can get a better measurement), and then measure the voltage between ground and the 8 row pins?
The expected value is defined by the voltage divider that's connected to the resistor networks, so should be 5V * 1K / (1K + 4.7K) = 0.877V. Do you see any difference between the good and bad rows in this regard?

[1coin3lives]

Shorting the two example pins seems to bring other connections to 0

Can you confirm you shorted these two pins together to get the above result?
download/file.php?id=74716

Yes, those are the pins I shorted. I don't think anything physically moved; the pcb was pretty steady when I did that.

Regarding that second part, I don't know where I'd be connecting to ground to take the measurement?

[pandrew]

So if anything I would have expected Tab and W to go to zero when shorting those pins, or nothing additionally going to zero. The E that you can see gone to zero has nothing to do with those. So it must have gone to zero for a different reason. Maybe everything is just failing intermittently? I don 't know, can you repeat the experiment?

Regarding ground:
So the order of pins on the ribbon cable connection from left to right is:

• 2 ground pins
• 8 row pins
• 2 ground pins
• 16 column pins
• 2 ground pins

You can connect the black wire to any ground pin, and measure each of the rows by touching the red wire to it.

[1coin3lives]

The two row pins I have marked with sharpie dots measure .815, and the other row pins measure .817 or .818

Repeating the shorting experiment, here's the result:

shorting v2.jpg (161.19 KiB) Viewed 8182 times

[1coin3lives]

BTW the positive values were changing all over the place when I took that screen shot - they weren't static. Only the 0 values were static. I don't know if that's expected or not.

[Muirium]

A bit of ripple is standard behaviour on my vintage IBMs.

[pandrew]

The two row pins I have marked with sharpie dots measure .815, and the other row pins measure .817 or .818

That seems to be okay, doesn't give me any concern. The differences from the ideal value of .877 can be attributed to voltage drop on USB +5V and/or imperfect precision of the resistor divider circuitry. In fact it seems like the analog row itself is not shorted to ground, and also the biasing resistor pack is not disconnected from the row.

Thiis result makes me expect that shorting these two pins (marked with red):

2pins.png (249.87 KiB) Viewed 8121 times

(which are the same as I drew up before on the controller side)
Would have no effect.

However it bothers me that the last screenshot you posted has a bunch of zeros and is in general extremely noisy, I would have expected it to look exactly like the first screenshot, so we need to be able to figure out why that happens. It is either another symptom, or maybe you were shorting the wrong thing, or maybe you were accidentally shorting something by the controller being in contact with some conductive surface. In any case something caused those zeros to appear, and we need to figure out what caused it to happen, before we can proceed probing deeper in the controller.

BTW the positive values were changing all over the place when I took that screen shot - they weren't static. Only the 0 values were static. I don't know if that's expected or not.

It is absolutely expected for the middle numbers to be changing +- a couple digits all the time, that's just noise. Just to clarify, inside each box there are 3 numbers. The top number if the maximum number seen since the signal level monitor has been opened. the middle number is the current value, and the bottom number is the minimum value since the window has been opened.
If the bottom number of a square reaches 0, it cannot change ever again, since there can't be a value lower than 0. (Same happens if the top value reaches 1023 on original-xwhatsit/wcass, or 4095 on the newer th xwhatsit and SMD derivatives)

Can you repeat the shorting of those 2 pins experiment again, and make sure that:
1) if you have done any other touching, or accidental shorting of any other pins, then you closed and re-opened the signal level monitor, to avoid dirty-ing up the collected min/max data
2) make sure that you wait at least two full refreshes of all keys before taking a screenshot (you will notice that the refresh rate is slow)
3) if you do see more zeros pop-up, other then the keys that were already zero in your very first screenshot from this thread, then can you figure out what actions caused those zeros to appear? I can't think of a reason why shorting the two pins I mentioned would affect the bottom row. And even more so, given the 0.8XX volts measurement results, I expected no additional keys going to zero from the shorting of those 2 pins.

[pandrew]

In the interest of going faster, I will assume that the conclusion will be that the zeros from the last screenshot was a result of accidental touching, or shorting the wrong pins. If that is the case, please perform the following experiments:

1) Measure voltage between these pins, while "Row/Dac tester" is open:
Expected result: 0.8XX volts
(note: red = red wire, blue = black wire of multimeter)

exp1.png (14.41 KiB) Viewed 8110 times

2) Measure voltage between these pins, while "Row/Dac tester" is open
Expected result: 0.8XX volts

exp2.png (13.53 KiB) Viewed 8110 times

3) In the "ROW/DAC tester" slide the threshold value to about 75%. (approx. 1023 * 75/100 = 768 raw threshold value)
Please tell me what the "Threshold voltage:" box says.
Then measure the voltage between the following two pins:
The expected value is that given by the "threshold voltage" box.

exp3.png (13.58 KiB) Viewed 8110 times

4) In the "ROW/DAC tester" slide the threshold value to about 75%. (approx. 1023 * 75/100 = 768 raw threshold value)
Please tell me what the "Threshold voltage:" box says.
Then measure the voltage between the following two pins:
The expected value is that given by the "threshold voltage" box.

exp4.png (13.64 KiB) Viewed 8110 times

5) In the "ROW/DAC tester" slide the threshold value to about 75%. (approx. 1023 * 75/100 = 768 raw threshold value)
Please tell me what the "Threshold voltage:" box says.
Then measure the voltage between the following two pins:
The expected value is that given by the "threshold voltage" box.

exp5.png (13.7 KiB) Viewed 8110 times

6) Can you take a picture of the controller, where the components are placed, to see which exact variant you have?
7fixed) Do you see any indication visually of something shorting these 3 pins? Perhaps a solder bridge? Look very closely with a lot of light and maybe a magnifying class, might not be obvious:
EDIT: whoops, initially I had the wrong pins circled here, I edited the post and now the image is fixed: Also please confirm your results are based on this fixed image.

check_visually.png (13.82 KiB) Viewed 8104 times

Do you have a hot air station to remove SMD components if the need will arise?

[1coin3lives]

I'll take some measurements later, but in the meantime:

- Under significant magnification, I see no evidence of a solder bridge or any other defect in the area pointed out from the fixed image.
- I don't have a hot air station, no. I'm just a keyboard enthusiast with very modest electronics knowledge/skills.

More info later... and thanks again for your help.

[pandrew]

Can I add another measurement:
8)
Disconnect the controller from the computer.
Put the multimeter into Diode measurement mode. (in this mode it shows voltage drop across a diode when tested with a given current)
Red wire to red, Black write to blue.
And let me know what the measurement is.

diode_mode.png (12.46 KiB) Viewed 8023 times

9) Same as 8) in diode mode, with the controller disconnected, but do it on the other chip:

diode_mode2.png (12.19 KiB) Viewed 8023 times

Also can you tell me what model multimeter you have?

[1coin3lives]

1) and 2) - I get .813v
3) Threshold voltage reads 1.40 when it's set to 768. Voltage measured by the first image is 1.415, which would seem to be correct.
4) Voltage measured by the second image is also 1.415
5) Voltage measured by the third image is .813, which doesn't match the 1.4v threshold reading
6) Picture is attached
8) and 9) for the diode test I get a value of 700 for the first test, and 698 for the second.

The multitester is a Cen-Tech 98025 from Harbor Freight; further evidence of my electronics novice status.

unnamed-3.jpg (783.12 KiB) Viewed 7964 times

[1coin3lives]

Bump to request additional feedback from anyone who might be able to provide it. Thanks in advance!

[pandrew]

Hey, sorry, I missed your messages somehow, (and honestly not even sure how this thread popped up again in front of me today)
If you're still here, then:
Please repeat measurement 5 paying attention to be 100% sure you're measuring the correct pins.
If it still reads 0.8V, (with the ROW/DAC tester open set to 75%), then it means that that pin at the "red" pad is not correctly soldered to the pad underneath it, or perhaps there could be a break in the trace that connects to that pad under the chip.
If you have a soldering iron, try to re-solder that pin. If it doesn't help, then try running a wire from the red pad of measurement 5, to the red pad of measurement 4.