SAML, RHEL7 and you

Updating to a new IDP with Django 2.2 & Djangosaml2 on Python 3.6

The Bechtel Center @ Purdue uses a lightweight Project Management System that I created using Django 2.2, the Djangosaml2 project built on PySAML2, and Python 3.6. Now in 2023 both the Python interpreter version and the Django version are very much End of Life.

Purdue has been updating its SAML provider as a member of Internet2 / InCommon and switching my development environment SP to the new credentials failed to log in with the following error:

ERROR ::2023-11-10 16:47:47,443::sigver sigver.py:850 ::returncode=1
error=func=xmlSecTransformNodeRead:file=transforms.c:line=1549:obj=unknown:subj=xmlSecTransformIdListFindByHref:error=1:xmlsec library function failed:href=http://www.w3.org/2009/xmlenc11#aes128-gcm
func=xmlSecTransformCtxNodeRead:file=transforms.c:line=694:obj=unknown:subj=xmlSecTransformNodeRead:error=1:xmlsec library function failed:name=EncryptionMethod
func=xmlSecEncCtxEncDataNodeRead:file=xmlenc.c:line=905:obj=unknown:subj=xmlSecTransformCtxNodeRead:error=1:xmlsec library function failed:node=EncryptionMethod
func=xmlSecEncCtxDecryptToBuffer:file=xmlenc.c:line=715:obj=unknown:subj=xmlSecEncCtxEncDataNodeRead:error=1:xmlsec library function failed:
func=xmlSecEncCtxDecrypt:file=xmlenc.c:line=623:obj=unknown:subj=xmlSecEncCtxDecryptToBuffer:error=1:xmlsec library function failed:
Error: failed to decrypt file
Error: failed to decrypt file "/tmp/tmptkxst803"

output=
ERROR ::2023-11-10 16:47:47,458::sigver sigver.py:850 ::returncode=1
error=func=xmlSecTransformNodeRead:file=transforms.c:line=1549:obj=unknown:subj=xmlSecTransformIdListFindByHref:error=1:xmlsec library function failed:href=http://www.w3.org/2009/xmlenc11#aes128-gcm
func=xmlSecTransformCtxNodeRead:file=transforms.c:line=694:obj=unknown:subj=xmlSecTransformNodeRead:error=1:xmlsec library function failed:name=EncryptionMethod
func=xmlSecEncCtxEncDataNodeRead:file=xmlenc.c:line=905:obj=unknown:subj=xmlSecTransformCtxNodeRead:error=1:xmlsec library function failed:node=EncryptionMethod
func=xmlSecEncCtxDecryptToBuffer:file=xmlenc.c:line=715:obj=unknown:subj=xmlSecEncCtxEncDataNodeRead:error=1:xmlsec library function failed:
func=xmlSecEncCtxDecrypt:file=xmlenc.c:line=623:obj=unknown:subj=xmlSecEncCtxDecryptToBuffer:error=1:xmlsec library function failed:
Error: failed to decrypt file
Error: failed to decrypt file "/tmp/tmpmpjdtmyj"

After going down several rabbit-holes, testing different signing algorithms, I finally read the error message carefully.

The error comes from the xmlsec1 binary which is used to perform the actual cryptographic functions needed by PySAML2, namely signing, verification and encrypt/decryption, and actually clearly states the issue in the first line:

error=func=xmlSecTransformNodeRead:file=transforms.c:line=1549:obj=unknown:subj=xmlSecTransformIdListFindByHref:error=1:xmlsec library function failed:href=http://www.w3.org/2009/xmlenc11#aes128-gcm

 Breaking it down into pieces and translating:

error=func=xmlSecTransformNodeRead:file=transforms.c:line=1549:obj=unknown:

 - Error reading a node in the target XML file due to an unknown object (executable code)

file=transforms.c:line=1549:obj=unknown:

 - Error reading a node in the XML file

subj=xmlSecTransformIdListFindByHref:error=1:xmlsec library function failed:href=http://www.w3.org/2009/xmlenc11#aes128-gcm:

 - The missing executable code referred to in the target XML file is identified by the URL 'http://www.w3.org/2009/xmlenc11#aes128-gcm'

Specifically the last part "aes128-gcm" refers to an encryption/decryption algorithm based on the AES algorithm. Now encryption is a very complex and difficult subject, as security is almost impossible to generalize, and even minor changes in how it applies can completely change its effectiveness. However it seems that the specific implementations in the 2001 W3C XML Encryption Standard require AES CBC and were completely vulnerable to attack, detailed by Matthew Green here. 

However, if we look at the xmlsec1 changelog here https://www.aleksey.com/xmlsec/news.html, AES CBC support was added to the xmlsec1 (openssl) edition in version 1.2.27 on  October 23 2018, and the version shipped with RHEL 7 is xmlsec1 (openssl) 1.2.20. Creating a local copy is not easy as xmlsec1 relies on libXML, OpenSSL, and other system packages, and compiling/linking against secure versions of them is not a trivial exercise.

Luckily they were able to downgrade the requirements back to AES128-CBC for my specific SP and mitigate the risks as all IDP/SP request/responses are only available over SSL, however time to finish up the OS2 replacement built on RHEL 8, Python 3.11 with a more modern xmlsec1!

Getting undercurl working in Neovim over SSH with ncurses 5

 ...or enabling Neovim undercurl on Red Hat Enterprise Linux 7

Firstly what is and why would I want it? I believe undercurl is a powerful option for your editor to give you feedback on what you have been writing, and Mozilla, Microsoft, Google, etc. have all implemented it many products.

As I switch away from Visual Studio Code to Neovim, basing my config on the excellent video from ThePrimeagen, I have faced several issues getting a modern copy of Neovim (>0.9) running. The source of the issues is both the age of the server operating system - Red Hat Enterprise Linux 7 (RHEL 7 from now on),  and that I cannot install any software as root. That limits me to local installs in my home directory.

To get undercurl working we need a terminal which supports it, and software which will use it.

Enabling undercurl

Background

To get started we need to talk about Terminals, and a bit of terminal history. Way back terminals were monochrome (no color yet!) freestanding, simple computers running a fixed ROM program made by various manufacturers to various standards. They connected to a central server using a dedicated serial line each, and supported displaying and typing characters. Pretty quickly various manufacturers implemented features like colors, bold, italic, more characters, etc. by means of special codes that do not printed to the screen but turn on and off the features. Even graphic support arrived, e.g. the Tektronix 4010.

With all that variety came a bunch of different features, novel bugs, and programmers soon grew very, very, tired with supporting all these different terminals. Each manufacturer had their individual quirks as well as standards they supported. To address this, software libraries were created where the programmer writes software to talk to the library, and the library queries the terminal for its feature set, and converts the program instructions into a stream of character codes for that specific terminal.

The most common software library used these days is ncurses, and its corresponding database of terminal capabilities is terminfo.

Testing your terminal for the undercurl feature

Undercurl, first implemented by kitty terminal (AFAIK), requires your terminal software to understand and draw the the curly line via specific feature codes, and luckily there are now a reasonable number of choices. For Windows, however, the choice is currently very limited as the Windows ConPTY subsystem used by most shells strips undercurl commands. Cygwin's mintty works, as does WezTerm using its internal SSH mode to a *nix host. Alacritty on Windows, and WezTerm using external SSH do NOT support it due to the underlying Microsoft ConPTY limitation. (This ConPTY limitation may have been fixed in August 2023, however a brief read of the Github issue and the PR is currently beyond my ability to analyze, also no idea how long until it is rolled into production builds.) Linux and OSX terminals basically all seem to support it now.

 

First, test your terminal for undercurl support. For example, with the Bash shell, SSH to the target server, and then run the following command at the command prompt:

printf "\e[4:3mhello world\e[0m\n"

The code \e[4:3m starts the curly underline, and the code \e[0m resets everything to normal, stopping it. If you have undercurl support you will see this:

 

If you do not see this then your client terminal program (e.g. Cygwin mintty, Alacritty, etc.) does not support undercurl and you need to find either another client terminal program or fix your current one.

Secondly, drop your SSH connection and configure your terminal to report its name correctly - for Cygwin's mintty this is set by right clicking the top of the window and selecting mintty:

 

Now you are ready for the next steps!

Neovim

Getting a modern Neovim release running on RHEL 7 has not been easy, however by downloading the binary release tarball from Github and unpacking it into a directory in my home directory I could get started. 

However, I immediately ran into a block - Neovim has been compiled on top of more modern system libraries than the ones available from RHEL 7. Particularly problematic can be the foundational library for all C and C++ programs of this age, glibc - the most common alternative today, MUSL, will not be released for another six years!

I have had experience with this issue before when supporting either very old or newer proprietary CAD tools on Linux (and Solaris before that), and believe me it is a tedious process.

Step one, prepare your environment:

1. Create ~/.local/bin and ~/.local/lib directories
2. Set environment variables in .bashrc (or equivalent) so any programs you run will look in these directories first:
• export PATH=${HOME}/.local/bin:${PATH}
• export LD_LIBRARY_PATH=${HOME}/.local/lib 
• export GIT_EXEC_PATH=${HOME}/local/git-2.27/opt/rh/rh-git227/root/usr/libexec/git-core/ [I had to add this later to get git-2.27 running, but I have put this here to save you time]
  

 Step two, iterate running nvim and downloading missing libraries:

1. Execute nvim
2. Note error message about a missing library or binary
   
3. Search for missing library from RHEL 7 compatible CentOS SCLo repository or the project homepage
   
4. Unpack the .rpm into a folder in my home directory
• e.g. rpm2cpio ../sclo-git25-git-core-2.5.5-1.2.el7.x86_64.rpm | cpio -idmv
  
5. Symlink libraries and binaries into .local/bin or .local/lib 
• e.g. cp --symbolic-link ${HOME}/local/git-2.27/opt/rh/rh-git227/root/usr/bin/* ${HOME/.local/bin/
6. ... repeat until it works
   

Eventually I ended up installing the following (your list may well be different!):

• nvim-linux64.tar.gz - Neovim stable
• rh-git227-git-core-2.27.0-3.el7.x86_64.rpm - for lazy.vim plugin manager
  
• httpd24-libcurl-7.61.1-2.el7.x86_64.rpm - for git-2.27
• httpd24-libnghttp2-1.7.1-7.el7.x86_64.rpm - for git-2.27
• ripgrep-13.0.0-x86_64-unknown-linux-musl.tar.gz - for telescope.nvim plugin
• fd-v8.7.0-x86_64-unknown-linux-musl.tar.gz - for telescope.nvim plugin

Neovim undercurl support

So now load up Neovim and test undercurl support with the following two commands:

:hi Foo guifg=Red guibg=Blue gui=undercurl guisp=Green

:hi Foo

and check the output:

But ... that is not a curly underline! It is a straight underline! What is happening??

Well, the printf test statement forces the server to send the characters exactly as typed in the statement, which the terminal software recognizes and draws the curly underline below "Hello World" correctly.  However Neovim is built on the ncurses library which looks up the terminal features by searching the terminfo data for the terminal named in the environment variable TERM. On RHEL 7 the database of terminals, terminfo, does not have ANY terminals listed that support undercurl. Why would it, as it was written before the undercurl feature existed. The upshot of this is that when you load Neovim ncurses finds no undercurl support and falls back to the available feature, a straight underline, instead.

The excellent Neovim documentation on this topic specifies a way around this problem by installing a local modern terminfo database into your home directory:

curl -LO https://invisible-island.net/datafiles/current/terminfo.src.gz
gunzip terminfo.src.gz
tic terminfo.src

However if you try this tic, the terminfo compiler, will generate maybe a hundred errors of the form:

"terminfo.src", line 12153, col 29, terminal 'wy60': unknown capability 'kF3'

... and Neovim will still not generate undercurl in the test.

The problem is that the old tic program does not know about many new Terminal capabilities and as a result it can't build a terminfo database to include them ... or can it? Luckily such a problem was forseen and accomodated by this flag:

 -x   Treat unknown capabilities as user-defined.

So running:

tic -x terminfo.src

and retesting Neovim finally shows undercurl working properly: