IBM 5251 terminal USB conversion (but not what you think)

[Pingmeide]

This morning I read the documentation again and I stumbled accross the "DP8344B Biphase Communications Processor"-documentation, which I downloaded.

Regardless of the current state of this project (which I fully support), wouldn't it be easier to use this "device", as it (as far as I understand) provides all the information, timing and interfacing to the Twinax-Connection we need ? What I miss (or don't understand, to be honest) is how this chip could be interfaced to the USB port. Eventually this may be the biggest problem here.... ?

I found supposedly new DP8344B's on ebay at "https://www.ebay.de/itm/DP8344BV-PLCC84 ... SwlptZ73sz" for EUR 11,25 each.

Just to hear your comments - can't we use this chip plus suppporting hardware, as it provides all signals we need on the twinax-connection ? There is some assembler-code in the DP's documentation but I have no clue what it is good for....

A short comment would be very much appreciated, what the (dis-)advantages would be. (Beside the availability of the chip.)

Many thanks !

[inmbolmie]

A short comment would be very much appreciated, what the (dis-)advantages would be. (Beside the availability of the chip.)

The availability of the DP8344 is not an issue, but there are three main reasons for me not to use it for this project:

• One, as you said, is that the DP8344 is a pre-USB era design, so you will have to throw more components just for getting a USB interface
• The second is that to program for that device I would need to figure out how to get a development board, code toolchain and learn its assembly language, and for that I would have to spend a ton of time and effort. With the Teensy it's just plugging it, installing the software and in ten minutes you are already running code.
• The third is that even with the two previous points solved, the obtained knowledge will be wasted as it is so specific for that device that it won't be applicable to any other project.

As an alternative I only seriously considered using a FPGA, as that could theoretically reduce the required parts count and be more flexible and suitable for a more compact and mass-production friendly design. But I finally considered to be easier and more productive going with a micro-controller board.

[Pingmeide]

Hi, thanks for the detailed clarification ! Yes, these are major stumbling-blocks for not using this chip, I agree with you !
So simply continue with the setup you have developed and already invested time and money.

In some older post you mentioned that there is some python (?) code, that is required to run on the (Linux)-PC to send / receive the data from / to the USB-Port. Have you made it available in the meantime ? (At least I can't find a link).

This would enable me to analyse the code and understand it. (You mentioned some mapping between the keys of the Terminal and the PC is required).
I am planning to attach an IBM 3477 Terminal to the Interface - so I could make some necessary preprarations well before receiving the interface-hardware. (For some time in the future when you declare the project to be "finished".)

Many thanks and
Have a nice Sunday !

[inmbolmie]

It’s time for the application layer (layer 3) stuff, the Python script for protocol conversion and session management.

I have uploaded the code (5250_terminal.py) as well as some basic instructions to run and configure it to the Github repository:

https://github.com/inmbolmie/5250_usb_converter

The first two layers give us the ability to send and receive frames to and from the terminals, but the 5250 terminals and the Linux compatible terminals, specifically DEC VT type terminals, are two completely different worlds:

• The 5250 terminals are block-oriented terminals. They provide a persistent buffer where we can write and read text characters to display on the screen. We have to interact with the buffer using commands that manage the displayed text itself as well as several internal registers to change text styles, control the cursor, copy data across the buffer, etc.
• The DEC VT terminals are character-oriented terminals. We interact with this type of terminals directly sending and receiving a serialized character stream, like in the good old times of mechanical teletypes. What these terminals receive they directly print it on the screen, and what you type in its keyboard is also directly sent to the computer as another character stream.

As we can only send plain text to a character based terminal, to have a greater control over what is displayed on the screen we need to include in the character stream what we call “control characters”. The control characters are analogous to the commands in the block oriented terminals, and using the we can make things like clearing the screen, positioning the text cursor, changing text style and many other things.

So with the block-oriented terminals we have a “buffer” and “commands” to manage that buffer, and with the character-oriented terminals we have a character stream where we mix standard displayable characters with non-displayable control characters to manage the display.

There are two main kinds of control characters in a DEC type terminal:

• ASCII control characters, that are simply the characters from the ascii table below the “space” character (character 32 or 0x20 in hexadecimal). Those are characters like line feed, carry return, tabulators, etc. They are also typically used as keyboard-generated control characters, usually pressing in the keyboard the Control key followed by another key (as an example, Control + c generates the character ETX or End of Text).
• Escape characters, that are extended control sequences that begin with the “ESC” character code (0x1B) followed by one or more characters as command modifiers. The meaning of those escape sequences are terminal-specific so we need to target an adequate terminal for the conversion.

ASCII control chars


For this project we are currently implementing only the Escape sequences for the VT52 terminal, as it it simple enough for a quick implementation, and somehow matches the capabilities of the IBM 5251 terminal. In the future it would be nice to have a full VT100 or ANSI X3.64 support for this converter, as this will allow to have a nearly total compatibility with most modern text-based software. Maybe this is a little overkill for a 5251 terminal but could be adequate for more modern terminals like the IBM 3196.


VT52 escape sequences


The structure of the Python program (5250_terminal.py) is as follows:

• The program starts a thread to manage the serial port communication, to and from the Teensy. That resource is shared between all the managed terminals.
• One thread is started for each managed terminal that starts a shell process and captures its standard input and output for that terminal.
• The output generated by the shell process (character stream) is captured and converted to a 5250 command sequence. For that, the program makes an ASCII to EBCDIC to character conversion for standard text characters, and for the control characters (ASCII and Escaped) it converts them to 5250 commands that emulate the control character actions in the 5250 terminal. So a character stream is converted to a command sequence for the 5250
• The input received from the 5250 keyboard in the form of keyboard scancodes is converted to ASCII character data, that can be regular text characters (uppercase, lowercase or "alt" depending on the status of “shift”, “alt” and “caps lock” keys) or escape sequences if we press the configured “control” key plus another key.
• The main thread ends up displaying a command prompt where the user can execute some administrative and debugging commands, like restarting a terminal, sending commands to it or sending text input to the shell (the same as typing it in the terminal)

Conversion program command prompt

cmdline.png (46.33 KiB) Viewed 19602 times

I won’t give here a full description of the VT52 escape codes, but as an example this is how the ESC_E escape code is converted to 5250 commands (ESC_E is clear screen, that we will receive from the shell as the two-character sequence 0x1B 0x45 in Hex):

• Move address counter to upper left corner with LOAD_ADDRESS_COUNTER command
• Move reference counter to lower right corner with LOAD_REFERENCE_COUNTER command
• Send CLEAR command to erase screen contents between the address counter and the reference counter
• Position the cursor and address counter to the upper left corner with the LOAD_CURSOR_REGISTER and LOAD_ADDRESS_COUNTER commands
• Send a END_OF_QUEUE command to indicate the end of command sequence

Every escape code has to be translated to a similar command sequence to run it in the 5250 terminal.

You also have a brief description of the available 5250 commands in the “5251 Maintenance Information Manual”, previously linked in this thread.

One drawback of this adaptation is that not every ASCII character has a correspondence in EBCDIC, and even due to differences in the localized character set of some terminals there are some weird character mappings that could negatively affect the user experience. For that reason the Python script supports user defined character mappings that you can configure to better suit your needs. For example in my case for the spanish 5251 character set, I have redefined these character mappings that correspond to ASCII character that are absent from EBCDIC code page 037 or get weird mappings from the default conversion:

Code: Select all

'['->(0x4A)
']'->(0x5A)
'^'->(0x95)
'#'->(0xBC)

Those mappings make sense in my case, but for other systems, different or no mappings at all should be configured by the user. I selected those mappings only because the mapped characters “look better” or “look more similar” to the original ASCII characters.

To have a feeling about the capabilities of the VT52 terminal and check if you could be really interested in running it over a 5250 you can use in Linux this command line to emulate it in a xterm window:

Code: Select all

xterm -ti vt52 -tn vt52 &

That's more or less what you will find in the real converted 5251 terminal.

My emulation is really slightly better than that, because I’ve added some very basic capabilities to detect VT100 Escape sequences, but that should give you a broad idea of its general compatibility.

So with the current state of the project you should consider those three main limitations of the conversion:

• Only V52 support, no VT100 or ANSI at the moment. This can change in the future.
• You will need to configure your keyboard mappings, maybe in the future this would be improved with the inclusion of more mappings, some kind of autodiscovery or a better interface.
• You will get some weird character conversions, others will be missing and you probably will have to configure some manual character mappings. This cannot be improved as it is a limitation of the terminal itself.

I still have to upload the PCB files, I want to make some final adjustments to them.

[Pingmeide]

Hi,
what an effort - I studied all that you wrote and I was astonished how long the "short piece of Python"-software is.

At the moment I have printed out the code and I am analyzing what it does, so I can eventually provide you with clues etc.

Will get back to you after the weekend.

BTW - I found some other persons owning a Twinax-terminal and looking for a connection to the ubuntu box, so I sent them the link to this page - let's see what will happen !

Cheers !

[Pingmeide]

Hi, here's a quick update - your comment on an "auto-configuration-feature" made me searching the net ...

On this link :
https://www.netphantom.com/APIDocumenta ... m5250.html

I found some information that you can query the terminal about its settings etc - as far as I understand.

Here's a snippet from the reply you can query the terminal for :

29 Device Type
X'01' - 5250 Display or 5250 Emulation
30-33 C'cccc' Device Type (e.g. 3180 for 3180 Mod 2)
34-36 C'ccc' Device Model (e.g. 002 for 3180 Mod 2)
37 Keyboard ID
X'02' - Standard Keyboard
X'82' - G Keyboard
38 X'00' Extended Keyboard ID
39 X'00' Reserved
40-43 X'xxxxxxxx' Display Serial Number
44-45 Maximum number of input fields
X'0100' - Typically = 256 input fields
46-48 X'00' Reserved (set to zero)
49-53 Controller/Display Capability


Moreover there is a reference to this document : IBM 5250 Information Display System Functions Reference Manual
Link :
http://bitsavers.informatik.uni-stuttga ... _May80.pdf

Details on the 5250 datastream are here:
http://bitsavers.informatik.uni-stuttga ... _Dec88.pdf

Perhaps this is a quick information for further ideas... I will check after the weekend and will post here.

[inmbolmie]

Hi, here's a quick update - your comment on an "auto-configuration-feature" made me searching the net ...

Unfortunately all that documentation is about the SNA/SLDC protocol, that is a network protocol related but totally different from the Twinax protocol. As eeguru said earlier in this thread, there could be some mapping equivalences between the two protocols, but it is difficult to determine and can't be used in this project at this stage.

Details on the 5250 datastream are here:
http://bitsavers.informatik.uni-stuttga ... _Dec88.pdf

Also that document describes the 3270 protocol, that is similar but different to the 5250 protocol. IBM had two different terminal series at that time, 3270 for the mainframes and 5250 for the minicomputers, using the same transmission media but different protocols.

[inmbolmie]

I've uploaded the PCB files to the repository. So now, "version 1" of this project is ready and anyone with a desire to do this by itself has all the necessary information published.

The final result looks like this:

PCB.png (197.36 KiB) Viewed 19494 times

I've also been finally able to test the converter with real Twinax cables, and it works fine, at least with short cables.

I think I'll make a small order of PCB's to some professional manufacturing service. For making like 10 pieces there are very cheap providers nowadays. Although the "self made" PCB works fine, soldering the components on a two layer self made PCB is a torture and the result not very aesthetic.

[inmbolmie]

I expect to have the first batch of PCB's in my hands next week, they are already made and in transit from the fabricator, so if anybody is interested (out of who already have contacted me) please send me a PM. I will make only a handful of devices and only by demand, as the components aren't specially cheap.

[Pingmeide]

Relating to eegru's request for "5250 Information Display to System/36 and System/38 System Units Product Attachment Information". Since there was no answer from IBM yet I have posted a request for that document to a german IBM i / AS/400 user-group. Eventually this will help since I already got some questions what I am looking for in detail. As soon as there is an outcome I will report here.
Cheers

[inmbolmie]

I have received the final PCB boards, much more pleasant to look at that the DIY thing.

board_final.png (1.47 MiB) Viewed 19322 times

Assembled board

Spoiler:

board_assembled.png (1.44 MiB) Viewed 19322 times

Everything works fine with them, there are no layout errors in the final design so I feel that for a first version, this project is finished and is sufficiently documented. I will close now the production of a small batch of converters with the people that have showed interest.

After that, if anyone is interested in a spare PCB for making its own converter send me a PM.

BTW I’ve decided to put a USB-B external connector in the box. The twinax connector and its attached cable are so sturdy that I feel that the micro-usb connector is a joke in comparison, mechanically speaking. I fear that in the long term the micro-usb will be ripped up by the tensions generated by the twinax cable, so a stronger connector is needed.

I’ve also added some vent holes as the DC converter becomes quite hot inside the sealed box. I’ve tested it for 12 hours of continuous operation without a glitch.

Spoiler:

box.png (877.13 KiB) Viewed 19322 times

Later I will post a video showing the current performance of the 5251 terminal with the converter.

[AJM]

That looks very professional. And I totally agree to use a proper connector like USB-B instead of such flimsy micro, mini or C connectors.

[Pingmeide]

Yes, this really looks very professional ! Congratulations ! Well done !

Regarding my older posts - there was either no reply (IBM) or I was provided with information we already have access to. So no new information on this, sorry. But I will continue to search for the document eegru is looking for . (5250 Information Display to System/36 and System/38 System Units Product Attachment Information).

[SneakyRobb]

This project is redic and I love it. Excellent work

[Weezer]

Agreed with the above posts. This project is awesome. I'm glad someone finally was able to revive these (in my opinion) beautifully styled beam spring terminals.

[zzxx53]

Interesting project!
Apparently someone made a similar converter for 3270 around the same time as OP.
https://ajk.me/building-an-ibm-3270-terminal-controller

[inmbolmie]

Interesting project!
Apparently someone made a similar converter for 3270 around the same time as OP.
https://ajk.me/building-an-ibm-3270-terminal-controller

Wow, thanks for pointing it out!

It was a big coincidence, both in timing and in concept. In January-February I searched throughfully online for projects like this and I was unable to find anything similar. It is great to have also available an option for 3270 conversion, that means that most of the IBM Beamspring era terminals, either 3270 or 5250 will now be usable with modern systems. Maybe that will contribute for some of those terminals not to end being trashed.

Like with eeguru project, it seems that this kind of conversion were in the minds of many people just waiting to become a reality.

The global concept is the same than mine, a microcontroller for USB conversion, some additional hardware for the physical interface and a Python script server-side for terminal conversion.

There are some differences in the approach, in the 3270 converter the author opted for using vintage chips, but it seems that that particular chips are not programmable, or at least more application specific as there is one chip for reception and one for transmission. So in his converter I suppose that the data link layer stuff is entirely managed by the vintage chips, while in my 5250 converter is managed in software by the Teensy. So it is a choice between more powerful micro against more functionality in the adapter chips.

Also for terminal conversion he uses a specific library for in-memory terminal emulation (pyte), while I make a direct stream conversion implementing the VT52 control character set, keeping very little state information in the server side.. I'm not sure about the performance of that method, if he has to refresh the entire screen for every change, or maybe he has some optimizations in place. But I will review that in order to get a method for getting also vt100 emulation in my converter, if it is possible to get it more easily than implementing the whole VT100 control characters in the way I did with VT52.

[lucar]

You made an awsome work!!

[inmbolmie]

For those brave users who will receive the first batch of converters, I have added to the Python script the following key mappings:

• 5251_US
• 5251_DE
• 3196_DE

As I have no reliable way of testing them they may need some adjustments. In the README are some instructions to modify those configurations and, if you need it, to watch the scancodes generated by your terminals.

To run the script as is for one of those mappings, simply put the terminal address plus ":" plus the key mapping name, like:

Code: Select all

python3  5250_terminal.py  0:5251_DE

..for a 5251 with german keymap at terminal address 0.

You could also edit your copy of the script to modify the default values for keyboard mapping and terminal address:

Code: Select all

#Configure the defaulf dictionary to use if nothing is specified in the command line, from those defined in scancodeDictionaries
DEFAULT_SCANCODE_DICTIONARY='5251_ES'

#Configure the defaulf station address if nothing is specified in the command line
DEFAULT_STATION_ADDRESS=0

[inmbolmie]

Also I want to comment that the first (and probably last) batch of converters is now closed, but I have spare PCB's if in the future anyone is interested. But it is so cheap to order that online nowadays that going directly that route is also a good option.

Archivo_001(10).png (1.13 MiB) Viewed 18871 times

[Weezer]

Also I want to comment that the first (and probably last) batch of converters is now closed, but I have spare PCB's if in the future anyone is interested. But it is so cheap to order that online nowadays that going directly that route is also a good option.


Archivo_001(10).png

Dang! I really wanted one of these.

[ramnes]

[AJM]

In case there will by another Deskthority-Award in the future, my vote will definitely go to inmbolmie!
Not only did he develop and build a great piece of kit, he is also a very generous and helpful person.
Besides building and offering his converters for the costs of the material, he even sent me one of his IBM twinax cables, after I had trouble sourcing one myself (for the price of shipping!). At the same time he tested and modified his script, so it would work with the subsystem for Linux of Windows and I wouldn't have to setup a whole separate Linux system. And then - after I finally started testing the device, he was again very helpful with any question I had and often answered within an hour.
What an awesome bloke!

[inmbolmie]

Well, you are welcome AJM, thanks also to all the other "early adopters" as you really risked your money for something that might or might not work with your terminals.

[Redmaus]

Would this be possible with a 5291 terminal? It uses twinax.

[inmbolmie]

Would this be possible with a 5291 terminal? It uses twinax.

At the moment it has been successfully tested with some 5251's, a couple of 3196's and one 3477. The 5291 if I'm not wrong is the intermediate model between 5251 and 3196, so if it's Twinax and the terminal is operative, "it should work".

[listofoptions]

likely some more interesting information about the 5250 terminals will be found here

http://bitsavers.org/pdf/ibm/system34/f ... chment.pdf

http://bitsavers.org/pdf/ibm/system34/f ... _Feb80.pdf

[inmbolmie]

likely some more interesting information about the 5250 terminals will be found here

http://bitsavers.org/pdf/ibm/system34/f ... chment.pdf

http://bitsavers.org/pdf/ibm/system34/f ... _Feb80.pdf

Thanks for the pointers, the first document is very useful, I already had it printed, section 11-11 contains the best available description of the command opcodes encoding. The second is new to me and contains the schematics for the System34 Twinax driver I was looking for, seem to be pretty cryptic but I will try to decipher them to see if anything useful can be extracted.

[listofoptions]

I noticed you have either a panel mount or a pcb mount twinax connector on your boards there, got any part numbers?

[inmbolmie]

I noticed you have either a panel mount or a pcb mount twinax connector on your boards there, got any part numbers?

Sorry, no part numbers nor any markings whatsoever, just generic parts picked randomly off Ebay.