Friday, 17 October 2014

OWTF 1.0 "Lionheart": UI and Database

REMINDER: We just released OWTF 1.0 "Lionheart", Please try it and give us feedback!

NOTE: This blog post is a guestpost by Bharadwaj 'tunnelshade' Machiraju, who devised and implemented the UI and Database idea from conception to implementation, full props to you, my friend :)

And with that, a big welcome and THANK YOU to Bharadwaj!


OWASP OWTF - User Interface and Database support


How OWTF used to be?



OWTF initially was a CLI program, which produced an interactive html report. Though OWTF was highly configurable, its usability was limited to huge configuration files.


console_view.png


What was done during this project?



The project had one main goal, i.e build an interface so as to control all aspects of OWTF. This involved more than a few challenges:


  • Refactoring the codebase to make use of databases.
  • Building a RESTful api to make the interface interactive.
  • Creating a web interface using the REST api.
  • Extending control on plugin execution (or worker processes as we cal it).


The technology stack which was finalized for this project:


  • Tornado (for interface and api servers)
  • Postgresql (for database)
  • SQLAlchemy (for ORM)


How does OWTF look now?



After the completion, OWTF must be launched from the command line and then everything can be done from the web interface. The following screenshots will take you through a tour, but for detailed explainations, a visit to our user docs is required (http://docs.owtf.org/en/latest/usage.html).


target_manager.png
Target Manager


plugin_launcher.png
Plugin Launcher


rated_target_report.png
Target Report
plugin_report.png
Plugin Report


Screen Shot 2014-09-16 at 4.38.27 am.png
Transaction Log


worker_manager_1.png
Worker Manager


worklist_manager_2.png

Worklist Manager


Resources!





Wednesday, 15 October 2014

OWTF 1.0 "Lionheart": Brucon 5x5 video, slides and more

REMINDER: We just released OWTF 1.0 "Lionheart", Please try it and give us feedback!

Just a quick note to say that the materials used by the OWTF Crew during the Brucon 5x5 presentations are now online:

Slides here:
https://speakerdeck.com/tunnelshade/brucon-2014-5by5-owasp-owtf

OWTF 1.0 "Lionheart" - Brucon 5x5 Video:



Talk structure and higher resolution demos

    1. (From minute: 0) Introduction to OWTF and discussion of the Web UI, REST API and DB by Bharadwaj Machiraju (higher resolution: OWTF UI Demo)
    2. (From minute: ~15:50) Discussion of WAF Bypasser AND OWTF Botnet mode by Marios Kourtesis (higher resolution: WAF bypasser fuzzing DemoWAF Bypasser detect allowed characters)
    3. (From minute: ~24:50) OWTF re-architecture by Alessandro Fanio Gonz├ílez

    On top of this, Bharadwaj became the Brucon 2014 Lightening talk winner with his awesome talk about Flashbang, which he developed for Cure53. The talk is now public and you can watch it here:


    Congratulations Bharadwaj, you rock! ;)




    Friday, 10 October 2014

    OWTF 1.0 "Lionheart": Automated Rankings

    REMINDER: We just released OWTF 1.0 "Lionheart", Please try it and give us feedback!

    NOTE: This blog post is a guestpost by Tao 'depierre' Sauvage, who authored one of the most successful GSoC 2014 projects for OWASP OWTF this year: OWASP OWTF: Automated Rankings

    Helicopter view:
    Ever had to test 30 URLs in 5 days and wondered where to start? OWTF will now take the MAX severity from ALL tools run against EACH target and tell you where :)


    And with that, a big THANK YOU and welcome to Tao! :)

    Introduction


    Thanks to GSoC, I had the opportunity to work on the OWASP - OWTF project. My task consisted in implementing an automated ranking system but first of all, let us have a quick overview of OWTF.

    As you surely know, because you are reading this blog, OWTF is a framework that helps user -- might be a security expert as well as an unsavvy but curious person -- in security assessments. It will take care of the unpleasant part of the job and automatically generates an interactive report containing all the information for the selected plugins.

    The powerful feature here is the interactive report. In a few words, instead of having a report that cannot be modified (like Skipfish or W3AF for instance), OWTF will take into account the user's actions. For instance, it might be interesting to add a screenshot for the plugin XY to clearly show the SQL injection that was found, and, you can!

    no_ranking.png


    But before GSoC began, the user had to manually evaluate the security risks for each plugin. If you had an assessment concerning 30 different websites it would take a lot of times. Therefore, the need showed up by itself: OWTF needed to pre-evaluate the security risks of its plugins.

    The ranking system


    By the end of GSoC, the automated ranking system has been completed and integrated to OWTF, a good news for its users.

    It has been developed following the simple rules below:

    1. OWTF's ranking scale would be Unranked/No risk, Informational, Low , Medium, High and Critical risk (6 different values).
    2. OWTF cannot automatically rank the outputs higher than high.
    3. The automated rankings will be highlighted as such.
    4. The user will be able to confirm/override the ranking.
    5. If the ranking has been confirmed/overridden, the highlight is removed.



    The first rule is kind of obvious, it is based on the most common scales that are used by any security tools.

    rankings.png

    6 available rankings


    The second rule is more interesting. We, OWASP contributors, decided that only the user is able to correctly estimate a critical risk because they are the only one aware of all parameters, such as the application context.



    mixed_risks2.png
    Highlight of the Critical ranking
    Let's say that tool XY found an SQL injection on target AB. According to most of vulnerability scores, such discovery is estimated really hazardous. On the other hand, OWTF cannot know what the database contains and if its information is critical or not. Therefore, instead of yelling the critical big red flag, it ranks the discovery as high and let the user decide whether it deserve a higher or lower ranking.

    zoom_medium.png
    Difference between an automated ranking and a confirmed one

    The three last rules have been chosen according to OWTF's philosophy, which is to have interactive reports. Therefore the automated ranking system would have its rankings more transparent than the ones of the user. That way, a quick view of the report will allow them to see what has been found and who found it.mixed_risks3.png
    Example of OWTF with its new automated ranking system

    As a visual result, OWTF’s automated ranking system will save a lot of time for the users. On the report above, a fraction of second is needed to understand what security aspect should be reviewed first.

    Current development state


    At the time of writing, OWTF's automated ranking system supports a couple of plugins and can rank at most 103 different ones. The following table describes which plugins are ranked:
    Supported tools
    Number of possible corresponding plugins
    Arachni
    1 plugin (OWTF-WVS-001)
    DirBuster
    12 plugins
    Metasploit
    85 plugins
    OWASP
    1 plugin (OWTF-CM-008)
    robots.txt
    1 plugin (OWTF-IG-001)
    Skipfish
    1 plugin (OWTF-WVS-006)
    W3AF
    1 plugin (OWTF-WVS-004)
    Wapiti
    1 plugin (OWTF-WVS-003)
    Supported plugins that can be ranked by OWTF

    Even though 103 seems a big number, a lot of plugins still has to be supported. The automated ranking system is in its early-stage development and will keep growing time after time in order to support each and every OWTF's plugins.

    Reusable


    During the development phase, I decided to export the automated ranking system in a standalone library that I baptized ptp (Pentester's Tools Parser). It means that OWTF's ranking system is reusable by anyone and to be honest, it is quite easy to embed ptp in your own project.
    If you are curious about developing a similar tool as OWTF but you don't want to bother ranking the discoveries, then have a look at ptp's documentation (linked in the resources section).

    Resources


    Before I let you go back to your activities, here are some useful links that would give you more information on the topic:


    If you made it this far, don't miss out Tao 'depierre' Sauvage's personal blog here!

    Tuesday, 7 October 2014

    Deadline=Oct 15th: Get credits, help OWASP, meet OWASP Winter Code Sprint, plz RT!

    NOTE: OWTF 1.0 "Lionheart" has been released!, please help us get it straight reporting bugs here.

    There is an awesome OWASP initiative this winter called OWASP Winter Code Sprint (OWCS).

    In essence, if you are a university student this lets you to contribute to a participating OWASP project in exchange for university credits and other perks.

    The full list of participating OWASP projects and ideas can be found here.

    Important deadlines:
    Please have a look at the OWCS page for more information and apply for your favourite project today! :).


    If any of the following seem interesting to you, please contact me ASAP on name.surname@owasp.org -minor thinking required ;)-
    NOTE: Yes, these are the ideas that are FREE for the 2nd round! :)

    Sunday, 5 October 2014

    OWTF 1.0 "Lionheart" released!

    UPDATE: 10/10/2014 - Added link to OWTF 1.0 "Lionheart": Automated Rankings

    OWTF 1.0 "Lionheart" (beta) is dedicated to everybody that helped make this challenging release happen, in particular to the courage of all this people, who overcame their sweat, blood and tears to make OWTF the amazing tool it is now, to all of you, thank you!

    We would like to take this opportunity to thank all contributors, mentors, everybody who sent us cool ideas, feedback or reported bugs, you all played a role in making OWTF be what is is now, many thanks to you all!.

    Some links:

    OWTF 1.0 “Lionheart” (beta) is our biggest release ever, this contains many cool projects implemented by many , so, in no particular order, here is a quick overview of the new major features! :)


    NOTE: For a more detailed explanation please see OWTF 1.0 "Lionheart": Automated Rankings

    When testing a bunch of web applications, OWTF will append the ranking associated with each target for an easy quick overview.



    More information can be retrieved when accessing to one specific target report. Here, OWTF uses labels and color codes to help the user reviewing the results.mixed_risks3.png

    Sexy Web UI + REST API + Database by Bharadwaj Machiraju






  • OWTF now features a Web UI from where you can run scans, plugins, etc.
  • RESTful API to send commands to OWTF!
  • Highly performant postgres database






  • Interactively send HTTP requests to a target
  • Record and play Zest scripts
  • Pass interesting HTTP requests to ZAP using the ZAP API



  • WAF bypasser by Marios Kourtesis

    Detecting blocked characters and searching for bypasses:
    Fuzzing with XSS payloads:



    OWASP OWTF - State Improvements by Viyat Bhalodia








    Improved Plug-n-hack support by Punga Cornel

    Plug-n-Hack is a proposed Mozilla standard.
    Now OWTF supports Plug-n-Hackv2, which allows you to intercept, change and fuzz client side messages!


    configuration page (nightly version).png


    configuration commands to injected probe.png



    monitor events message.png


    Online passive scanner, Boilerplate templates and Flexible Mappings by Anirudh Anand



    • Online Boilerplate templates: Easy to copy-paste stuff into your pentest reports (i.e. mitigation, etc.)
    • Passive Online scanner: Try some of the OWTF passive tests without installing anything! :)
    • Flexible mappings: OWTF will now let you view plugins by OWASP Top 10, OWASP Testing Guide v4, OWASP Testing Guide v3 and even NIST! :)

    demo.jpg

    Thursday, 25 September 2014

    OWTF 1.0 "Lionheart" to be presented @Brucon

    Why wait? Download OWTF 1.0 "Lionheart" now! ;)

    Just a quick note that the OWTF Crew will be presenting part of what is coming on OWTF 1.0 "Lionheart" during the Brucon 5x5 presentations:

    When: Friday September 26, 2014 13:00 - 15:00 
    Where: 5 La Trappe (Novotel Ghent) - Brucon, Ghent, Belgium, Europe :)

    OWTF talks
    Enjoy! :)

    Saturday, 13 September 2014

    OWTF 1.0 "Lionheart": Call for testers + GSoC Poll

    Call for testers

    OWTF 1.0 "Lionheart" is inminent, PLEASE help us:
    1) Testing the bleeding edge branch here: https://github.com/owtf/owtf/tree/lions_2014
    2) Reporting bugs here: https://github.com/owtf/owtf/issues 

    -other options: tutorials, demos, documentation, ideas, suggestions, bug fixes and any other form of contribution you can think of :)-

    How to get started:

    OWASP OWTF GSoC 2014 Student participation poll


    In a similar fashion to what we did last year, in the scientific spirit of “observe and record” a poll was run among GSoC participants that submitted or showed interest to participate in the GSoC for OWASP OWTF.

    Poll format



    The poll was based on the following questions:

    CASE 1) IF you submitted something for OWTF this year, could you please
    answer the following question?

    What made you submit a proposal for OWASP OWTF? (i.e. as opposed to
    other OWASP projects and/or organisations)

    CASE 2) IF you decided *NOT* to submit for OWTF this year, could you
    please answer the following question?

    What made you NOT submit a proposal for OWASP OWTF?

    Poll answers - CASE 1) Students who submitted


    NOTE: I have only redacted what could identify the student and/or project.

    Student 1 - OWTF is written in Python, which is a one of prog. languages I love and is all about Infosec. OWTF is all about automating boring, but rewarding work in pentesting. Lastly, there is heavy active development in the project compared to others.



    Student 2 - I have successfully submitted a Proposal to OWASP OWTF *only* because of your support. Initially when I contacted you, I was thinking that I am too late to do a GSoC. My primary aim was to do a *good project* under a really helpful *Mentor* so that I can increase my skills under his guidance. Luckily for me, I got a chance to talk with you and I must say that made a significant difference. After started talking to you, within a week, I was confident that I can do a GSoC (or at-least a good project) with you. That motivated me to submit a proposal for OWTF. -redacted-. Thank you so much for helping me out.

    Also even if I didn't get into GSoC, I am planning to complete the project which I have started. I am hoping to start working on it from -redacted-. I love to work, with you as my mentor. Also, I need your valuable suggestion on my project -redacted- so that I can make it a good project (since you are very experienced, I hope you can give some tips).

    As tunnelshade said : *You are a project leader which every project must have ;)  *. That's one statement I completely agree.



    Student 3 - Because I am familiar with OWTF, and its because a raising project that seems have a brilliant future



    Student 4 - Case 1) I submitted to OWASP OWTF because I met my interests: a project related to security, networking, Python or C language. I saw OWASP ZAP 2 or 3 days before OWTF, I was impressed, that an open source project has such great success and is used by a lot of pen-testers, security specialists, programmers. So, I decided to search more about OWASP and its projects, this way I found OWTF and an important step to begin working on proposal was fast and clear response from Abraham Aranguren, thank you Abraham! Before my research for proposal, I knew basic concepts about proxy servers, present technologies in information security, thus OWTF project became an incentive to study more about. Thank you very much OWASP team, for all that you are doing!



    Student 5 - What made me submit a proposal was that the project aligned with my skills and helped me grow. Plus the mentors were more supportive than any other org I had interacted with :)



    Student 6 - I submitted a project proposal for OWTF because I have been part of OWTF from last -redacted- and my major aim from the beginning was to pick one project (and it was OWTF) and work for it.Honestly, initially when I decided working on OWTF, it was completely a random decision . I just wanted a project in python with few contributors. I asked around, in OWASP mailing list, and gradually Samantha and Fabio suggested me this one. And there I came in contact with Abraham Aranguren, that was the most amazing thing that happened to me. His continuous guidance, motivation and quick responses made me stick with the OWTF project.
    I am glad I made that choice :)



    Student 7 - Well, I guess this is not the right time for me to answer this question. But to be honest, the reply that I got from you(Compared to other OWASP mentors) after I dropped my first mail in the OWASP mailing list was my motivation to submit proposal to OWTF. Though there are a lot to tell, lot to work on and a way to go but still...



    Student 8 - I have a passion for security ever since I have been into it. I started working more on it when I joined my college for undergraduate course. Since then, I have had a good platform to do more research on it. I started playing CTFs as member of team -redacted-. I have great interest in web application security and reverse engineering obfuscated codes.

    What I always wanted when I played CTFs was task automation, especially in attack-defence ( service based ) CTFs involving real-time attacks. Python was the best language I found for this, which has heavy modules support (including third party) and, simplicity and efficiency in data handling. I use it for automating blob vulnerabilities over the network against all insecure opponents.

    I also wanted to do work on a security project and I believe that doing GSoC would be the best immediate way to actively work on an existing project. In the long run, I would like to keep contributing to Open Source projects outside GSoC too.

    Taking all these into consideration, OWTF turned out to be the security project I want to work on. This could be the chance of working on a big Open Source Project. When I started working on it, I came to know more technical details and got really interested and confidant. This motivated me in totally focusing totally on OWASP OWTF.

    The mentors in this project are sincerely motivating us at every stage and are happy to help. I am sure that they will be a great source of motivation for all those who work with them.

    I am doing GSoC just for gaining experience, it is definitely going to amp up my skills and confidence. But my long term intention is to work on the project beyond GSoC and be a part of it for long enough. And I am interested in this project.
    If at all I would have not chosen to work on OWTF, it would be just because I was not lucky enough to notice how relevant this project is to what I do.



    Student 9 - Before the GSoC organizations list was released, I was thinking about applying
    for a HoneyNet project. When I learned that, this year, this organization was
    not selected, I changed my plan.
    First of all, I have downloaded the GSoC organizations list and searched for
    the Python and security keywords.
    Why these keywords? Well, I have been using the Python language for a couple of
    years now and it is the language, along with the C one, that I feel the more
    confident about. Then security because I am fond of security since a long time
    now.
    After applying the filter, several organizations appeared.
    Among them, the TOR foundation and the OWASP organization. While looking at
    the two organizations’ projects list, I found OWTF, which gathered both
    criteria.
    Furthermore, I have spent the last six months working in the security service
    of -redacted- where I discovered what the work of pentesters was
    about. I have learned about the deadlines during which the pentesters had to
    accomplish their work and I was surprised about how short they were.
    As a conclusion, I have found a great interest in the OWTF which aims to
    accomplish the information gathering step that pentesters have to accomplish
    before doing further investigations. That way, the pentesters will be more
    efficient while doing their work.
    All together, I have contacted the mentor of the OWASP OWTF project, Abraham
    Aranguren, in order to get started on the project.
    In order to sum up my answer, I have applied to work on the OWASP OWTF project
    because:
    • It is written in Python
    • It is about security
    • It aims to solve a problem I feel concerned about
    PS: I am not sure if it is related but the project name if OWTF like “Oh WTF!”,
    which sounds awesome!



    Student 10 - I have submit a proposal to OWTF because of the following reasons:
    - I Like this project.
    - I had contribute to OWTF in the past and this makes me to like this project even more.
    - The community R0cKs!
    - I am using this program in my work (internship), so I want to help to make it even better.

    I didn't choose another project from OWASP because I am not as familiar as OWTF and also i wanted to focus 100% to one project for higher chances of acceptance.

    Poll answers - CASE 2) Students who did NOT submit

    NOTE: I have only redacted what could identify the student and/or project.

    Student 1 - I'm on the second case. I couldn't submit a proposal this year because I didn't have time to write it (because of studies, work, etc.) and I won't be able to work in a GSoC project during the summer. Despite of that, I found you emails really encouraging, and I think it is a good way of motivating the GSoC candidates, so keep doing it.



    Student 2 - I decided not to submit for OWTF this year. I had initially chosen 3 organizations to submit a gsoc proposal (OWTF being one of them). But due to shortage of time, I decided to focus on only 1, to increase my chances of getting selected.



    Student 3 - well i was very interested to learn pentesting but there are many resources on internet for learning(actually confusing) but owtf combines the best tools so obviously we are going to learn the best ones and gives us a chance to  excel in this field without wasting time  and it is written in PYTHON



    Student 4 - Sorry I wasn't able to submit! For your poll, I am definitely a case 2 - I got other offers for work over the summer and decided I wanted to spend my time working for a company this summer instead of doing GSoC. I would still like to contribute to OWTF however! If no one picks up the boilerplate/scripting project for GSoC, is there any way I can still work on it? The topic still really interests me, and I would love to put work into on my own time if I can. Let me know what you think! :)



    Student 5 - I am part of case 2 where I couldn't submit the proposal for OWASP OWTF this year. I was determined to submit my proposal but due to lack of time since I am new and got to know about the program late, I couldn't submit the proposal. I thank you for helping me out and supporting me even in the 11 th hour which most wouldn't have done.