1.13.2010

A new type of security testing...

I am not sure if this is really a new idea, it probably isn't, but I haven't seen anything in the tubes about it and it is a concept that I have been slowly rolling around my brainpan while working on the ESAPI.

When we write unit tests for ESAPI, what we are doing is giving the API inputs and looking for the correct outputs, or looking for an exception. This is nothing new, this is what unit testing is really all about - but it is all based around business functionality.

My idea is to take that idea, of testing business functionality and apply the same philosophy to a new set of test cases.

Traditionally, there are 3 main types of assessments of an application to uncover security flaws:
  1. Vulnerability Scanning: This is usually the first step in an application assessment. It generally consists of an automated scanner, such as Nessus. If the assessment is being done by someone competent, it will also include a manual review of the site. The problem with vulnerability scanners is they are really good at finding the really obvious flaws, but they have no context to the application. The only way to do a full assessment of an application is to sit down in front of it and try everything you can to break it by hand.
  2. Static Analysis: There are some great static analysis tools out there that are great at finding common coding errors that lead to security flaws. For Java there are things like FindBugs and PMD. There are also services online that will scan the code in your repository for security related code issues. These can find some bugs that would otherwise be very difficult to find, but require a lot of tuning to limit the number of false positives that are reported.
  3. Manual Code Review: If you come into an existing application that already has an existing codebase, this can be one of the most daunting tasks you will ever encounter. Especially if you come into a codebase that has had a lot of hands in the pot (so to speak) with a lot of different coding philosophies. This process is exactly what it sounds like, sitting down in front of the code and analyzing every line of it looking for coding errors and bugs. To date, this is the most effective form of assessment, but it is rare for everything to be caught unless this process is implemented by a very good developer or architect from day 1 of coding.

What I am proposing is using the concept of automated integration testing in the runtime by writing code to test your code for security vulnerabilities.

Imagine the following situation. You have a simple guestbook web application. The application consists of a single JSP file, a servlet and a facade that sits in front of a DAO. Let's illustrate this with some pseudo-java below.

Value Objects to represent the data
public class Entry implements Serializable, Comparable<Entry> {
   private long id;
   private String message;
   private String fromName;
   private String fromEmail;

   // ... declare getters and setters
}

Data Access Layer
public interface GuestbookDAO {
   List<Entry> getEntries() throws DAOException;
   void saveEntry(Entry entry) throws DAOException;
}

public class MySQLGuestbookDAO implements GuestbookDAO {
   public List<Entry> getEntries() throws DAOException {
      Connection con = null;
      try {
         List<Entry> entries = new ArrayList<Entry>();
         con = JDBCHelper.getConnection();
         Statement st = con.createStatement( "select * from entries" );
         ResultSet rs = con.executeStatement();
         while ( rs.next() ) {
            entries.add( new Entry( rs.getLong(1), rs.getString(2), rs.getString(3), rs.getString(4));
         }
      } catch (Throwable t) {
         throw new DAOException(t);
      } finally {
         if ( con != null ) con.close();
      }
   }

   public void saveEntry(Entry entry) throws DAOException {
      Connection con = null;
      try {
         con = JDBCHelper.getConnection();
         PreparedStatement st = con.createPreparedStatement( ... );
         // ... standard jdbc code
      } catch (Throwable t) {
         throw new DAOException(t);
      } finally {
         if ( con != null ) con.close();
      }
   }
}

public class GuestbookDAOFactory {
   public static GuestbookDAO getDAO() {
      return new MySQLGuestbookDAO();
   }
}

Facade
public class GuestbookFacade {
   private GuestbookFacade() { /* Singleton */ }
   private static final GuestbookFacade myInstance = new GuestbookFacade();
   public static GuestbookFacade getInstance() { return myInstance; }
   private static final Logger log = Logger.getLogger( GuestbookFacade.class );

   private final GuestbookDAO dao = GuestbookDAOFactory.getDAO();

   public List<Entry> getEntries() {
      List<Entry> out = new ArrayList<Entry>();
      try {
         out = dao.getEntries();
      } catch (DAOException e) {
         log.error(e);
      }
      return out;
   }

   public void saveEntry(String message, String fromName, String fromEmail) {
      Entry entry = new Entry( message, fromName, fromEmail );
      try {
         dao.saveEntry(entry);
      } catch (DAOException e) {
         log.error("Unable to create entry: " + message + ", from: " + fromName + "<" + fromEmail + ">", e);
      }
   }
}

Controller
public class GuestbookServlet extends HttpServlet {
   public void doPost(HttpServletRequest req, HttpServletResponse resp) {
      String action = request.getParameter( "action" );
      if ( "SAVE".equals( action ) ) {
         String message = request.getParameter( "message" );
         String fromUser = request.getParameter( "fromUser" );
         String fromEmail = request.getParameter( "fromEmail" );
 
         GuestbookFacade.getInstance().saveEntry( message, fromUser, fromEmail );
      }

      List<Entry> entries = GuestbookFacade.getInstance().getEntries();
      request.setAttribute( "entries", entries );
   }
}

View
<html>
<head><title>Guestbook</title></head>
<body>
   <%
      List<Entry> entries = request.getAttribute( "entries" );
      for ( Entry e : entries ) {
   %>
   <div class="entry"><%= e.getMessage() %> - From <a href="mailto:<%= e.getFromEmail(); %>"><%= e.getFromUser() %></a>
   <%
      }
   %>
</body>
</html>

This is a very basic application (that is full of security flaws I might add, but we will ignore that little tidbit for now and let our test catch them) that we will now write an example SecurityTestCase for.

public class SecurityTestGuestbook extends TestCase {
   private MockHttpServletRequest request;
   private MockHttpServletResponse response;
   private HttpServlet servlet = new GuestbookServlet();

   public void setup() {
      request = new MockHttpServletRequest();
      response = new MockHttpServletResponse();
   }

   public testSQLInjection() throws Exception {
      final String[] injections = new String[] { "' or 2=2--" };
      request.initialize();
      response.initialize();
      request.setParameter( "action", "SAVE" );
      request.setParameter( "fromUser", "Beef" );
      request.setParameter( "fromEmail", "email@domain.com" );
      for ( String test : injections ) {
         request.setParameter( "message", test );
         try {
            servlet.doPost( request, response );
            fail("No exception thrown - SQL Injection Possible" );
         } catch (Throwable t) {
            // Success
         }
      }
   }
}

This is just a single quick example, but it illustrates the point nicely. We will leave verifying functionality works to our standard unit tests, but write another test suite specifically to test your security controls. You should probably have more granular tests that just testSQLInjection - maybe testRequestSQLInjectionMessage, testRequestPersistentXSSMessage, etc.

I would love to hear what people think about this as a development concept, especially those who work in TDD and Agile environments. I would also be interested to hear from people who have used some of the macro based front end testing tools to see if they have created similar view test suites.

5 comments:

  1. I couldn't agree more! If you consider that test cases are based upon use cases, then there is also the latitude to consider test cases written upon 'abuse cases'. This is, I think, what you are describing here.

    I've long been a proponent of not just testing what is *supposed* to happen, but explicitly testing for things that are supposed *not* to happen. Your suggestion here is a concrete example of this philosophy.

    What would be great is if we could move some of the common test cases into something like ESAPI... so we're providing developers with the tools to mitigate vulnerabilities and provide assurance of that mitigation.

    ReplyDelete
  2. @Dave
    I think it is important that the ESAPI test cases test the interfaces of the API so that they can be used to test "your" ESAPI implementations as well as what we ship with ESAPI.

    However, I don't think that is enough. This doesn't touch on specific use cases or as you say "abuse cases" (I dig it BTW) Testing only the security controls themselves does not ensure that your codebase is using those security controls and more importantly doesn't test that your business logic is using them correctly.

    An example is that while ESAPI tests the functionality of the Encoder. canonicalize() method, if you are storing a canonicalized string in your database without validating it, the ESAPI tests will not catch this issue, and storing a canonicalized value is often more dangerous than storing an encoded value.

    I think your point is valid, and you are right on with what I am describing here, but I think that it needs to go beyond just the ESAPI and into the realm of the application code itself. I am currently working on ways that this could be accomplished without requiring the developer to write a ton of test code every time they create a class, but generifying that process is difficult and subject to the same types of problems as static code analysis and vulnerability scanning.

    ReplyDelete
  3. I don't think this idea is good. I think this idea is **NECESSARY** if we want to provide a deeper level of assurance for different ESAPI implementations.

    Nice work as usually, Beef!

    ReplyDelete
  4. I'm a bit late but... I believe this approach dramatically outperforms more classic tool-centric DAST approaches over time. the key is over time. I think most people shy away because they're driven only by audit/compliance or because of the perceived cost of the spin-up time required to get good coverage on the canon of, say, OWASP security guidance. This is a shame.

    I was speaking with someone from my OWASP chapter and a few guys from our quality practice and suggested the following: let's build 'testing legos' for a critical mass of the OWASP Security Testing Guide into a suite of Selenium scripts that people can use 'out of the box'. If we successfully hit, say, 33-50% of the guide, people might view the proper security testing approach as less onerous and might make that initial jump from DAST. They'd also have a clear path to transition test-case maintenance to their QA groups.

    We've already done an analysis of the testing guide for ability to automate, now we just need to start prototyping the toolkit. Anyone want to help?

    ReplyDelete
  5. A similar approach was suggested by Stephen in OWASP AppSec EU 2006.
    http://www.owasp.org/images/6/62/OWASPAppSecEU2006_SecurityTestingthruAutomatedSWTests.ppt

    I have also included this in my trainings for developers for some time now. But I'm not sure how many of them follow it or even bother to go that extra length. Also the in-house **security experts** in most companies would not be able to help unless they have some kind of understanding for code (which most of them do not !!!).

    But this is a very good (and yes ... *NECESSARY*) approach and I believe including such tests will help in identifying security bugs early in the Dev lifecycle.

    I don't want to sound too negative but what is the probability that the devs would do this considering the stringent deadlines under which they are working?

    Any way ... this is an excellent idea :)

    ReplyDelete