A new type of security testing...

I am not sure if this is really a new idea, it probably isn't, but I haven't seen anything in the tubes about it and it is a concept that I have been slowly rolling around my brainpan while working on the ESAPI.

When we write unit tests for ESAPI, what we are doing is giving the API inputs and looking for the correct outputs, or looking for an exception. This is nothing new, this is what unit testing is really all about - but it is all based around business functionality.

My idea is to take that idea, of testing business functionality and apply the same philosophy to a new set of test cases.

Traditionally, there are 3 main types of assessments of an application to uncover security flaws:

1. Vulnerability Scanning: This is usually the first step in an application assessment. It generally consists of an automated scanner, such as Nessus. If the assessment is being done by someone competent, it will also include a manual review of the site. The problem with vulnerability scanners is they are really good at finding the really obvious flaws, but they have no context to the application. The only way to do a full assessment of an application is to sit down in front of it and try everything you can to break it by hand.
2. Static Analysis: There are some great static analysis tools out there that are great at finding common coding errors that lead to security flaws. For Java there are things like FindBugs and PMD. There are also services online that will scan the code in your repository for security related code issues. These can find some bugs that would otherwise be very difficult to find, but require a lot of tuning to limit the number of false positives that are reported.
3. Manual Code Review: If you come into an existing application that already has an existing codebase, this can be one of the most daunting tasks you will ever encounter. Especially if you come into a codebase that has had a lot of hands in the pot (so to speak) with a lot of different coding philosophies. This process is exactly what it sounds like, sitting down in front of the code and analyzing every line of it looking for coding errors and bugs. To date, this is the most effective form of assessment, but it is rare for everything to be caught unless this process is implemented by a very good developer or architect from day 1 of coding.

What I am proposing is using the concept of automated integration testing in the runtime by writing code to test your code for security vulnerabilities.

Imagine the following situation. You have a simple guestbook web application. The application consists of a single JSP file, a servlet and a facade that sits in front of a DAO. Let's illustrate this with some pseudo-java below.

Value Objects to represent the data

public class Entry implements Serializable, Comparable<Entry> {
   private long id;
   private String message;
   private String fromName;
   private String fromEmail;

   // ... declare getters and setters
}

Data Access Layer

public interface GuestbookDAO {
   List<Entry> getEntries() throws DAOException;
   void saveEntry(Entry entry) throws DAOException;
}

public class MySQLGuestbookDAO implements GuestbookDAO {
   public List<Entry> getEntries() throws DAOException {
      Connection con = null;
      try {
         List<Entry> entries = new ArrayList<Entry>();
         con = JDBCHelper.getConnection();
         Statement st = con.createStatement( "select * from entries" );
         ResultSet rs = con.executeStatement();
         while ( rs.next() ) {
            entries.add( new Entry( rs.getLong(1), rs.getString(2), rs.getString(3), rs.getString(4));
         }
      } catch (Throwable t) {
         throw new DAOException(t);
      } finally {
         if ( con != null ) con.close();
      }
   }

   public void saveEntry(Entry entry) throws DAOException {
      Connection con = null;
      try {
         con = JDBCHelper.getConnection();
         PreparedStatement st = con.createPreparedStatement( ... );
         // ... standard jdbc code
      } catch (Throwable t) {
         throw new DAOException(t);
      } finally {
         if ( con != null ) con.close();
      }
   }
}

public class GuestbookDAOFactory {
   public static GuestbookDAO getDAO() {
      return new MySQLGuestbookDAO();
   }
}

Facade

public class GuestbookFacade {
   private GuestbookFacade() { /* Singleton */ }
   private static final GuestbookFacade myInstance = new GuestbookFacade();
   public static GuestbookFacade getInstance() { return myInstance; }
   private static final Logger log = Logger.getLogger( GuestbookFacade.class );

   private final GuestbookDAO dao = GuestbookDAOFactory.getDAO();

   public List<Entry> getEntries() {
      List<Entry> out = new ArrayList<Entry>();
      try {
         out = dao.getEntries();
      } catch (DAOException e) {
         log.error(e);
      }
      return out;
   }

   public void saveEntry(String message, String fromName, String fromEmail) {
      Entry entry = new Entry( message, fromName, fromEmail );
      try {
         dao.saveEntry(entry);
      } catch (DAOException e) {
         log.error("Unable to create entry: " + message + ", from: " + fromName + "<" + fromEmail + ">", e);
      }
   }
}

Controller

public class GuestbookServlet extends HttpServlet {
   public void doPost(HttpServletRequest req, HttpServletResponse resp) {
      String action = request.getParameter( "action" );
      if ( "SAVE".equals( action ) ) {
         String message = request.getParameter( "message" );
         String fromUser = request.getParameter( "fromUser" );
         String fromEmail = request.getParameter( "fromEmail" );
 
         GuestbookFacade.getInstance().saveEntry( message, fromUser, fromEmail );
      }

      List<Entry> entries = GuestbookFacade.getInstance().getEntries();
      request.setAttribute( "entries", entries );
   }
}

View

<html>
<head><title>Guestbook</title></head>
<body>
   <%
      List<Entry> entries = request.getAttribute( "entries" );
      for ( Entry e : entries ) {
   %>
   <div class="entry"><%= e.getMessage() %> - From <a href="mailto:<%= e.getFromEmail(); %>"><%= e.getFromUser() %></a>
   <%
      }
   %>
</body>
</html>

This is a very basic application (that is full of security flaws I might add, but we will ignore that little tidbit for now and let our test catch them) that we will now write an example SecurityTestCase for.

public class SecurityTestGuestbook extends TestCase {
   private MockHttpServletRequest request;
   private MockHttpServletResponse response;
   private HttpServlet servlet = new GuestbookServlet();

   public void setup() {
      request = new MockHttpServletRequest();
      response = new MockHttpServletResponse();
   }

   public testSQLInjection() throws Exception {
      final String[] injections = new String[] { "' or 2=2--" };
      request.initialize();
      response.initialize();
      request.setParameter( "action", "SAVE" );
      request.setParameter( "fromUser", "Beef" );
      request.setParameter( "fromEmail", "email@domain.com" );
      for ( String test : injections ) {
         request.setParameter( "message", test );
         try {
            servlet.doPost( request, response );
            fail("No exception thrown - SQL Injection Possible" );
         } catch (Throwable t) {
            // Success
         }
      }
   }
}

This is just a single quick example, but it illustrates the point nicely. We will leave verifying functionality works to our standard unit tests, but write another test suite specifically to test your security controls. You should probably have more granular tests that just testSQLInjection - maybe testRequestSQLInjectionMessage, testRequestPersistentXSSMessage, etc.

I would love to hear what people think about this as a development concept, especially those who work in TDD and Agile environments. I would also be interested to hear from people who have used some of the macro based front end testing tools to see if they have created similar view test suites.