Jsunpack Website Database Optimizations

I just published some new optimizations for the jsunpack.jeek.org database. This should dramatically improve the performance of the website.

Contact me if you need anything related to this update. I removed the search functionality (and associated RSS feeds) because that was one of the most performance intensive features on the database. While this type of function is still possible, I think I'll either have to limit terms that may be searched for or build a separate index structure so that it is better optimized.

[edit] In case you are curious, the jsunpack database has 186,459 submissions and 686,232 evalated scripts and URLs since 2010-01-29 14:17:36. This year was pretty active for web exploits. The majority of the submissions to jsunpack were publicly released in one of the RSS feeds with 167,356 submissions over this year. The opposite of that were the entries where users wished those submissions to remain private totaling 19,107 submissions.

Importing jsunpackn.py as a python library

In case you were wondering, it *IS* possible to import jsunpack-n from your own python programs. The benefits include:
1) greater control of the options (that you'd normally specify on the command line or in the configuration file)
2) control of the output in their native data structures (no more $ ./jsunpackn.py file|grep -E "suspicious|malicious" >> readme.log)

Check out the new exampleImport.py script, now available in the project's google code svn site.

If you review the main code for the exampleImport.py script, you will see that it defines a function that allows you to directly pass JavaScript... for example:
main('eval("var a=123;");')

Then you can see these loops in the main function:

        for type,hash,data in js.rooturl[url].files:
            print 'file              type=%s, hash=%s, data=%d bytes' % (type,hash,len(data))
        for printable,impact,msg in js.rooturl[url].msg:
            print 'output message    printable=%d, impact=%d, msg=%s' % (printable,impact,msg)

The point of this post is to show you that you can directly access these files and perform decoding using jsunpackn.py as a python library. If you need examples for how to operate on more than just JavaScript as the exampleImport.py shows, then try looking at the main function of jsunpackn.py, and you'll see how I process the various different types of input.

Cheers,

Jsunpack through a proxy

I have a special guest blog from http://twitter.com/malc0de today. He submitted some proxy suggestions for jsunpack-n, which I've added to svn. Here is a description of the features. Thanks for the great new feature ideas, keep them coming! also check out his site at http://malc0de.com/ if you haven't already.

---

In todays threatscape the ability to forward requests through proxy servers can come in handy in certain situations. Generally speaking a majority of drive-by exploit kits in the wild have logic built in that does not allow for duplicate requests originating from the same IP address. Since I primarily use jsunpack-n to investigate potentially malicious domains and am one of the many jsunpack-n users out there I thought it would be useful if jsunpack supported proxies. At first diving into the 1300+ lines of python was intimidating but after a few hours I had a working prototype that I sent to Blake who later modified (improved) and added to the svn repository.

Once you have checked out a new copy (https://code.google.com/p/jsunpack-n/source/checkout) you will notice two additional options:

-p PROXY, --proxy=PROXY
- use a random proxy from this list (comma separated)
-P CURRENTPROXY, --currentproxy=CURRENTPROXY
- define a single proxy

The first option -p allows you to specify a comma delimited list of proxies of which jsunpack will randomly pick one.

Example
./jsunpackn.py -u "www.google.com" -p 189.3.47.146:3128,187.49.68.11:8080,187.45.175.66:3128

URL fetch www.google.com
[fetch config] random proxy 187.45.175.66:3128
[fetch config] currentproxy 187.45.175.66:3128

The second option -P allows you to define a single proxy.

Example
./jsunpackn.py -u "www.google.com" -P 187.45.175.66:3128
URL fetch www.google.com
[fetch config] currentproxy 187.45.175.66:3128

For your convenience a perl script called getCurrentProxies.pl can be found in the tools directory. This script interfaces with a popular website named malwaregroup.com to retrieve an updated list of active proxies. The IP’s are printed out in a comma delimited format accepted by the new -p option.

Example

./tools/getCurrentProxies.pl
193.255.184.210:3128,193.110.187.209:3128,193.105.240.32:8080,190.200.151.23:8080,189.84.116.88:3128,189.3.177.146:8080,189.3.150.32:3128,189.3.47.146:3128,187.49.68.11:8080,187.45.175.66:3128,187.23.145.82:3128,187.0.80.180:3128,180.149.49.114:8080

I enjoyed working on this project with Blake and would like to take this opportunity to encourage others to contribute their ideas. Jsunpack has come a long way since it was first introduced and continues to remain one of my favorite tools.

---

I need your encrypted PDF files

I just released some major improvements to jsunpackn today in version 0.3.2c. The main new feature is the ability to parse encrypted PDF documents. One problem though, the output isn't quite perfect. For example, see one of these encrypted PDFs where jsunpackn extracted JavaScript:

http://jsunpack.jeek.org/dec/go?report=2ed7fde3fbc8d7c2857bfd69878f78e3b008518e
http://jsunpack.jeek.org/dec/go?report=1873b5faccc1574ce065f6528e85e64464e4b09c

Note, how certain characters in the output are not quite right, therefore the JavaScript decoding doesn't go beyond the first stage. If you find any PDF files with the string "/Encrypt" in them, please let me know, I'd like to test them to see if I can improve the parsing further. In the process of supporting the /Encrypt tag, I was able to build a testbed and was able to detect 20 new PDFs in my testcases with the new parsing!

I would like to thank Dave Touretzky (from the Computer Science Department and the Center for the Neural Basis of Cognition at Carnegie Mellon University) for posting details of encrypted PDFs within his Gallery of Adobe Remedies.

Friday Link Trifecta

Here are some cool things I ran into this week.

koto's Ultimate String override
koto gave a presentation to a polish owasp meeting describing how to detect and evade jsunpack. He also presents some ways to fix those evasions in Ultimate toString override. Great work!

Paul Makowski's Blog
In this post, Paul uses his hacked sshd (it logs attempted usernames and passwords) in order to track down the tools used against his server. I was investigating some similar tools, so it was great to see this. One of the interesting tools Paul found was a Mac OSX IRC bot. They all seem to have some connection to "trance". I even found some new files "trance.pdf", which isn't really a PDF if you were wondering :)

SWF Disassembler Plug-in for IDA Pro [PDF]
This looks like a great plugin for IDA. If you want to try it with some live samples from jsunpack, monitor this rss feed.

Jsunpack-n update 0.3.2b Custom Spidermonkey and Google Code Subversion

I now distribute my own modifications to Spidermonkey with the project. This allows you to easily compile it with my preferred modifications. If you want to see what modifications I make, read the INSTALL.spidermonkey or INSTALL.spidermonkey.shellcode files.

You can now get the source code for the project from https://code.google.com/p/jsunpack-n/ (instead of my website).

Updates 2010-05-20 version 0.3.2b

1) added INSTALL.spidermonkey.shellcode instructions. This adds improved shellcode detection.
2) updated jsunpack class options structure. New options will always use file contents instead of filenames (where possible). Also, rules are now part of the options structure.
3) socket defaulttimeout now part of jsunpack class (it was global before). If you import jsunpack, make sure to set a timeout on your own.
4) you can use jsunpack.version to get the current version string
5) new performance option (-f "fasteval") for disabling non-critical features in favor of performance
6) fixed a bug in redoevaltime option affecting performance of malicious scripts
7) fixed a pdf parsing bug for /Page related to testcase samples/pdf-numPages.file

Three days ... of bug hunting (and a new release) I blame python

This is the worst bug related to the project that I've encountered so far and I just spent a few days trying to find what went wrong... Well, I just found and fixed it!

I'd highly recommend NOT using version 0.3.2 because of this bug, version 0.3.2a has only a few new features but YOU SHOULD UPGRADE immediately if you use 0.3.2. Version 0.3.2 is the only version affected by this bug and it prevents jsunpack-n timeouts from killing the SpiderMonkey process (if the script never terminates).

Updates 2010-05-05 version 0.3.2a
1) implemented this.numPages, getPageNthWord, and getPageNumWords
2) fixed bug (python 2.6 only) for log_ips functionality
3) fixed REALLY BAD performance bug related to python dynamic types. This is why type checking is a good thing and why python sucks at it. Most of my python bugs are because I redefine another variable, wipe out a loop variable, or otherwise use a bad type.