This post is the first of a series of articles refloating some old notes about enterprise software, SAP security and previous research that was never published or socialized, as well as introducing some new findings along the way. We’ll touch on SAP archive files, Path Traversals and in-archive digital signature pitfalls to help you be your best.
Introduction
Targeted attacks against system administrators are known to be practices some bad actors use when attempting to gain access to and compromise high-value environments. As system administrators use of a wide range of tools to manage the environments under their reach, an interesting entry point for such targeted attacks is through those tools and the way they are used. In this blogpost I’ll share a couple of stories of my past journey researching and understanding SAP’s archive file format, the tools available to operate them, and some of the findings that were derived from that learning process.
First, why care about this?
Why bother reading about old stories and already fixed issues? Primarily because very recently SAP undergone a long-time effort to renew some support system and that involves moving to the distribution of digitally signed SAP Notes, which we’ll touch later on this article, but relates to this old experiences. We think that understanding how the different mechanisms works and how them can be attacked or bypassed help to take better and more educated decisions when assessing and planning security measures.
In addition, long after my research on the file format was published at Troopers 2016, on March 2018 SAP updated the behavior of some of the tools, which we’ll see and discuss later on this article. But I never got back to provide an update on what it means to the original research, so this is the opportunity to do it.
Also, high-profile vulnerabilities in WinRAR disclosed and found being exploited in-the-wild recently reminded me of the journey I went through on the SAR archive files, and the similar risks the users are exposed to. The wide-spread use of general-purpose archive files cannot be probably compared to the reach of SAP BASIS and security administrators, but still makes them a pretty interesting target. Nevertheless, archive files are the second more prominent file type used in malware attacks via email, and the SAP world is probably not exempt from that.
SAP Archive Files and the SAPCAR Tool
In the SAP universe, distribution of software packages, components, patches and upgrades (called “SAP Notes”), and other type of artifacts is done via archive files, in a similar way than many other software vendors. The main reasons are containment and easiness of use, by distributing a small number of files, and resource usage, by compressing data within the archive file.
Contrary to some other software vendors that make use of standard or well-known archive formats and tools (zip, gzip, 7zip, etc.), and in the same way that for other elements, SAP has come up with a proprietary file format and tool for handling archive files. The file format is called SAR while the archiving and extraction tool is called SAPCAR.
SAPCAR is a stand-alone and command-line program available in all SAP Netweaver, HANA and more current S4/HANA environments. The program can be found by default located in the directories defined as search paths on the operative system level. Its use is very simple and includes functionality covering the creation and manipulation of archive files, as well as obviously extraction of files from them. The following snippet show some of the common operations that can be performed using the SAPCAR program:
$ echo "This is a text file" > text-file.txt$ ./sapcar -cvf archive-file.sar text-file.txt a text-file.txt $ ls archive-file.sar archive-file.sar $ ./sapcar -tvf archive-file.sar SAPCAR: processing archive archive-file.sar (version 2.01) -rwxrwxrwx 20 07 Nov 2019 10:23 text-file.txt $ ./sapcar -xvf archive-file.sar SAPCAR: processing archive archive-file.sar (version 2.01) x text-file.txt SAPCAR: 1 file(s) extracted
Common operations of the SAPCAR tool
More documentation about the SAPCAR program and how to use it can be found in the official SAP Help pages.
It’s also worth remarking that there’s an older file format, denoted CAR, that was used in early versions of SAPCAR. I was unable to find the exact dates or time period but looks like a very ancient format that might be dated at least 15 years ago. While the SAPCAR program is backwards compatible and still can open and extract files from CAR archives (which is quite interesting from a security perspective as well, some 15 years-old bugs might be there!) nowadays it’s difficult to find archive files published in that format and SAP stopped its use while ago.
The SAP Archive File Format
I began the analysis of the file format by creating a set of simple and as small as possible archive files. By looking at the files produced by the SAPCAR tool, we were able to quickly confirm that the compression algorithm in use by the file format were SAP´s proprietary LZH and LZC algorithms. These algorithms are well-known, as they are used across several network protocols to compress traffic. An open source implementation was made available at some point, which facilitated incorporating it in our pysap library (as well other similar tools) for crafting and dissecting network protocols, in pysapcompress module. The compression algorithm have had its own issues in the past, but is not the focus of this article, so let’s get back to the file format details.
A high-level view of the file format and its parts can be observed in pysap’s documentation. There’re some minor nuances and small differences between the two known versions of the file format, 2.00 and 2.01 (I suspect that CAR was probably versioned internally as 1).
A hexadecimal view of the small archive file created in the previous section has the following look:
$ xxd archive-file.sar 00000000: 4341 5220 322e 3031 5247 ff81 0000 1400 CAR 2.01RG...... 00000010: 0000 0000 0000 0000 0000 3861 c45d 0000 ..........8a.].. 00000020: 0000 0000 0000 0000 0e00 7465 7874 2d66 ..........text-f 00000030: 696c 652e 7478 7400 4544 1e00 0000 1400 ile.txt.ED...... 00000040: 0000 121f 9d02 7b21 1999 c50a 4094 a850 ......{!....@..P 00000050: 925a 51a2 9096 9993 ca05 0000 4d9d d682 .ZQ.........M...
Content of a SAR archive file
Lately, and thanks to a BIZEC session held at Troopers 2018, I got to learn that the file format was somehow published and documented as part SAP’s NW-VSI (Virus Scan Interface) documentation. Moreover, a reference implementation was published as part of the VIS 2.0 SDK, which unfortunately doesn’t seem to be publicly available nowadays. This helped augment our knowledge about the file format as well as to refine and enhance our experimental custom implementation of it.
Initial Findings Exploring the SAPCAR Program
After getting an initial understanding of the SAR file format and SAPCAR, I looked for logical issues affecting the operation of archive files. Naturally, the more interesting operation from an attacker’ standpoint is the extraction of files from the archive.
I looked at some of the more common issues affecting archive files, and got some results that included the following findings:
CVE-2016-5845Denial of service via invalid file names, as theSAPCARprogram was not properly validating if certain characters were included in the file name while extracting files.CVE-2016-5847Race condition on permission change, as theSAPCARprogram was subject of aTOCTOUissue by extracting a file to the file system and then in a second operation setting the permissions defined in the archive file.
We were just scratching the surface, and the results were not interesting in the sense that either the impact was little to minimal or the conditions required for an effective attack to work had a very low likelihood. However, we decided to report the issues to the vendor, and were promptly patched as SAP Security Notes 2312905 and 2327384.
Traversing Archive File Paths
During the process, I looked several times at the program documentation and in-screen help. In particular, the following statement from SAPCAR’s help was interesting:
other options: [..] -P : use absolute path-names (use carefully) [..]
Excerpt of the SAPCAR tool help
So, this means that the SAPCAR tool can create archive files and use absolute paths for the files within it by providing the -P option. What about the extraction of those archive files? The following statement was even more interesting and provided the answer:
[..] using absolute pathnames: If you create an archive with absolute pathnames the files will be extracted with exactly these pathnames! SAPCAR does not cut the first slash like the UNIX tool tar. [..]
Excerpt of the SAPCAR tool help
If you’re following this from an offensive security background, there’re some attack vectors easy to spot on these sentences. Basically, an adversary can construct an archive file including files with names that contain full paths, and those would be honored by SAPCAR during the extraction. For example, one can create an archive file containing file names such as “/home/sapadm/.ssh/authorized_keys” and if extracted with a user with enough privileges SAPCAR will overwrite the original files.
Constructing archive files with absolute paths requires using SAPCAR with the aforementioned -P option flag, or better, crafting the archive manually which can also allow the use of any arbitrary character. In order to do so, during the research we added a script to pysap, called pysapcar. The usage is straightforward and it follows the same options than the original SAPCAR tool, but by means of the /n option it’s possible to use any arbitrary file name when creating an archive file or appending files to an existing one. Another way would be to directly play with pysap’s API and reconstruct the archive files after tampering with the file names. Using this we managed to create archive files that contained “../../” characters, enabling for “path traversing”.
An existing archive file can be “infected” by appending files to it that when extracted without proper care will result in files potentially being overwritten in the file system. Combining this with other type of vulnerabilities, such as an improper validation of TLS certificates, can allow for interesting Man-in-the-Middle attacks such as the one shared at Troopers 2016 and affecting older versions of the SAP Download Manager tool (see SAP Security Notes 2233617 and 2235412).
Long after this attack vector was presented at my Troopers’ 2016 talk Deep-dive into SAP archive file formats, in March 2018, SAP released a new version of SAPCAR that removed this default behavior. The details can be found in SAP Security Note 2597543, but basically SAPCAR now strips all trailing paths from the file names and validates that there’re no absolute ones in use before extraction.
A Short Tale about Memory Corruption
Another track followed was the use of fuzzing techniques over the SAPCAR tool, aiming at identifying potential vulnerabilities in the parsing routines. As I was dealing with a closed-source binary and source-code level instrumentation wasn’t possible, the choice was Google’s hongfuzz fuzzer due to some successful and happy previous experiences with it and its high performance under this scenario.
After a long run without too much optimization and a very large number of crashes found, along with Maximiliano Vidal we managed to identify a heap-based buffer overflow issue affecting the handling of block types unknown to the program. The issue was reported to SAP, tracked as CVE-2017-8852 and fixed in SAP Security Note 2441560 with the release of a new SAPCAR version.
It’s interesting to mention that the SAP Security Note denotes the issue as a “Potential Denial of Service (DoS) in SAPCAR”. Maximiliano managed to prove that under certain scenarios that’s not exactly true and shared some awesome technical details to back that claim in he’s “SAPCAR Heap Buffer Overflow: From crash to exploit” blog post.
Digital Signature to the Rescue
Either due to logical issues such as the mishandling of absolute/relative paths or implementation issues such as the memory corruption vulnerabilities early mentioned, it’s clear that users need to have a way to verify the validity and authenticity of archive files. Especially when dealing with software packages for sensitive systems as SAP, it’s hard to believe that users and administrators will follow the path of every single software package they download and extract, and even though vulnerabilities might appear as shown.
The solution that SAP proposed to increase the security and allow for validation of archive files was a digital signature scheme. The scheme was introduced in 2011 in SAPCAR version 7.20 patch level 2. It was not until 2016 that the feature got more traction and for example was documented in SAP Note 2178665.
The cryptographic primitives in use for generating and validating the digital signatures are provided by SAP’s standards cryptographic libraries, the CommonCryptoLib and previously the so called SAPCRYPTOLIB. Each signed archive file contains an additional file named SIGNATURE.SMF that acts as a detached manifest file. It contains the hashes and signature of the files included in the archive. The default algorithms used are SHA256 for hashes and PKCS7 for signature. The following is an excerpt of a valid manifest file:
SAP-MANIFEST Version: 1.0 Hash: SHA256 Signature: PKCS7-TSTAMP Body: Digest | Name-Length | Name f212e04bca96925a5cd424d9f8b2733533e1d800b38021bf332ea092d0be4c6f 000d Changelog.txt 9a07479720f9c40bd672e16cded8af322afd84eda40d18279d9d108bc0ed6163 0009 LEGAL.TXT ae113a023d4b93ca1dfe1c9e0d1bdf4d37ac0a6cbc3fd6077f8cd36756dcdb08 000b LICENSE.TXT [..] -----BEGIN SIGNATURE----- MIINTgYJKoZIhvcNAQcCoIINPzCCDTsCAQExDzANBglghkgBZQMEAgEFADALBgkq hkiG9w0BBwGgggQDMIID/zCCAuegAwIBAgIBCjANBgkqhkiG9w0BAQsFADBMMQsw CQYDVQQGEwJERTEfMB0GA1UEChMWU0FQIFRydXN0IENvbW11bml0eSBJSTEcMBoG [..] -----END SIGNATURE-----
Manifest content of a signed SAR archive file
The scheme is based on a Public Key Infrastructure maintained by SAP, and the issuer’s certificates are embedded in the binaries. Certificate revocation is supported by means of checking local CRL files that need to be previously downloaded, although is an optional feature that need to be explicitly called as a command-line option when using SAPCAR.
It’s worth mentioning that signature validation is not enforced by default on SAPCAR and it need to be explicitly enabled via the -V option. This means that extracting files from an archive file with invalid or non-existent signatures is the regular behavior. Care need to be taken and users are advised to always enforce signature validation using the -V option when manipulating SAR files.
Another interesting take on this signature scheme is that while it’s welcomed as a mechanism to reduce the attack surface, it keeps the door open to some potential security vulnerabilities. Memory corruption or logic issues such as the described before might still be possible due to the need of the extraction program to not only parse the archive file header to locate the manifest file but also decompress the content in order to obtain the signature file. Only after having extracted the manifest that contain hashes and signatures is that the authenticity of the archive file can be verified.
How not to Validate in-archive Digital Signatures
As mentioned at the beginning of the article, SAP started an initiative to increase the security of their software distribution processes and renew some of the delivery infrastructure. This involved replacing some backbone components and systems that are key to almost every ABAP-based system, as regularly those handle the download and installation of patches and update packages in an automated fashion. After January 2020, the new infrastructure only allows using fixes and packages that are digitally signed.
In order to implement these new mechanisms, SAP made available some guidance and a set of patches for their Netweaver servers that updates download and patching mechanisms such as the SAP Note Assistant (SNOTE) program or the SAP Solution Manager platform.
After having researched the archive files and the signature mechanism, I jump straight to reviewing the changes on SAP Note Assistant. The process that the updated version performed in order to verify the signature of the uploaded SAR files was, according to the documentation, as follows:
- The SAP Note file is copied into an application server directory for temporary files (
$(DIR_TRANS)/tmp) (e.g. /usr/sap/trans/tmp). - Checking for signature type (
SAP software) and the signature issuer (SAP Trust Community).
It was interesting that these steps were implemented by SNOTE in a single operation, by providing the -xvV flags to the SAPCAR tool to extract the files. This means that if using the early implementation of digital signatures in SNOTE, a tampered or “infected” archive file will be processed even if the signature was invalid, and files will be extracted to the temporary folder. Mixing this issue with the previous behavior of the SAPCAR tool of honoring absolute and relative paths resulted in a pretty bad combination, as the SNOTE program and the entire ABAP server runs with high operative system privileges on any SAP system.
Fortunately, this was also promptly fixed by SAP in the Security Note 2546220 and fixes were incorporated in updated versions of the SNOTE patches and technical documentation pieces. The updated version makes sure to first validate the signature of the archive file, before extracting any file.
As a final note, it’s important to mention that the SNOTE tool doesn’t perform any check nor enforcement on the version of SAPCAR available in the system. Additionally, the documentation only states that SAPCAR version 7.20 patch level 2 is required. System administrator should make sure they are running the latest SAPCAR version available.
Final Notes
Building secure software and managing software in a secure way requires not only will but a deep knowledge about how that software works. We celebrate the advance to more secure mechanisms to ensure trustworthiness in the distribution of software package for sensitive systems as SAP’s and enforcing digital signature on the archive files is a move in the right direction.
Although painful to BASIS and SAP security administrators, we recommend not only updating their infrastructure to follow this change but also make sure SAPCAR and CommonCryptoLib are in their latest versions across all systems in the landscape. While generally stand-alone components do not receive enough attention and might be overlooked from a vulnerability management perspective, them can leave doors open for security issues and attack vectors that reduce the overall security posture, as demonstrated across several research published and summarized in this article.
If you have questions, comments or feedback about any of these topics, reach out to me on Twitter at @martingalloar or email our Innovation Labs using labs@secureauth.com!
Continue reading
SecureAuth Innovation Labs Sheds Light on Protecting Credentials in SAP HANA: The Client Secure User Store
SAP HANA SAML Assertion Improper Validation Vulnerability
