I recently had the opportunity to speak at San Francisco Security Bsides with Sam Stelfox. We decided to investigate a topic which is very important to us. At Pwnie Express a lot of the work we do falls into the category of “Platform Development”. For our package repositories we rely on the Kali package repos and decided to investigate that set of tools in more depth.

Penetration testing professionals often depend on a complex and largely undocumented ecosystem of tools created by a wide variety of individuals. If you are staking your professional identity on freely available security tools it is crucial that you ask a few questions of those tools:

• Is it stable and reliable?
• What language is the tool written in?
• Is the tool properly licensed to be used or modified as needed?
• Is the project being actively maintained and developed?
• Is the source code publicly available and being managed through some sort of revision control?
• Are the tools safe to use?

In our research we combed through 369 different tools and asked these questions of them to try to evaluate the state of the tools that are available to us as well as to users of our products as well as Kali Linux in general.

Research Results

In general the tools were not in a bad state. The median project length was 2.3 years with the longest running active tool being developed for over 17 years. Those numbers are positive, but there were also a large number of tools which had never been updated since their initial release.

The openness of the source and version management was not as promising. Here was the breakdown for that:

This was quite surprising when we considered how important this sort of technology is for our workflow. This was a clear sign to us that many of these tools were not being developed with any sort of software engineering best practices.

The licensing situation was also not ideal. GPLv2/GPLv3 took the lead followed by MIT and BSD. However the number of unlicensed tools was much higher than we would have liked.

The long tale of other licenses speaks to a need for more consensus on how open software should be licensed to both protect the intellectual property of the developers but also allow for people to freely modify and update the code to meet their own needs.

Another common problem we encountered was “bit rot”. This occurs when a project is left unmaintained for an extended period of time. Dependencies can change or be deprecated, processor architectures can change or APIs might vanish or change. All these can contribute to a tool that was working ceasing to function. This is generally not hard to fix, but only if the code is available in such a way as to encourage people who are using that particular tool to submit fixes or contact the maintainer directly.

Recommendations

We had a few general recommendations for anyone releasing an open source security tool or script to follow:

• Licenses: License your tool int he appropriate way. We suggest TL;DR Legal as a place to start understanding the implications of different licenses.
• Version Control: Use public version control such at Github to host your code and maintain a free place to host documentation and stimulate collaboration.
• Code Quality: Take the time to learn coding best-practices for the language you choose to use. Well written code is much easier to maintain over time.
• Documentation: Document your project for users as well as other coders who may need to modify or work on your code.
• Output: Generate outputs that are both human readable and consumable by other scripts and tools. For GUI tools make sure you have an export option into a standard format such as XML or CSV.

Our full slides and research data are available here. Hopefully Bsides will also post a video of our talk soon!

If you are in the Bay area come say “Hi” at our RSA Booth (#2513).