Tuesday, April 14, 2015

ESAPI4CF v2 code is in GitHub master

Yup, I am still working on ESAPI4CF and now the initial code for verson 2 has been pushed to GitHub master.  I have not made a release yet as the code still needs work and that's where the community can help pitch in.

Be sure to check it out here: https://github.com/damonmiller/esapi4cf/

A few things about this version:
  • updated to follow the ESAPI-java v2.1.0.
  • full script components - no more hideous tags.
  • aims for compatibility with CF10+, latest Railo, and latest Lucee.
  • overhauled configuration to make it simpler to implement.
That last point is big!  Personally I hated trying to implement ESAPI4CF v1.x into my projects for 2 main reasons:
  1. Trying to secure the ESAPI.properties file and keep it separate per CF app.
  2. Having to overwrite/extend the Authenticator and AccessController just so ESAPI could talk to my DB.
So I took my own personal complaints and addressed them.
  1. ESAPI.properties is gone and you can now configure ESAPI by passing a struct into init().  This makes it simple to configure ESAPI different for each of your CF applications.
  2. Introducation of the ESAPIAdaptor interface.  This is not in esapi-java at all so this veers a bit - maybe they'll like this idea and include it in future versions.  So all of the file-based interaction built-in to ESAPI has been moved into the default Adaptor and is still used by ESAPI by default.  You can very easily roll you own Adaptor and tell ESAPI to use it, via the new ESAPI init(), and this is how ESAPI will talk to your DB, BeanFactory, whatever you use.  The Adaptor is meant to be part of "your" application so it can extend your architecture, it merely needs to implement the ESAPI Adaptor interface and that's it.
Anyway, this version is not done and is definitely in need of more hands to help finish it so I could use any help.

Areas needed for help:
  1. I have not updated any of the documentation as of yet, not even the readme file, so it is all still for v1.x.
  2. I have only been testing against CF11 and Lucee (was Railo until Lucee came out).  I would like it to be tested under CF10 and Railo as well.
  3. I have not finished the AccessController yet as it is a mess to implement in java v2.1.0 so this will take some time and patience.  Then once its working it needs all its file-based code moved into the Adapter as well.
  4. Anything that seems overly complicated to implement in CF.  Let's make this version easy to use!
  5. Everything else!  Still lots of unit tests to get passing.
Also, I will point out that I have been slowly implementing v2 into a production app already.  I do it cause I know the ESAPI code very well and can fix any issues that comes up quickly and then go back and fix it in the project.  But a warning to anyone else wanting to use it in a live project right now - it is not finished, it is buggy, so use at your own risk, read the license around liability cause you assume it all :)

With that said, if you do decide to use it in a live app that is great for the project as it will get the real world exposure and testing that it needs.  Just don't forget to commit your bug fixes back to your fork and submit a pull request - it will only make the project better for all!

Thank you to those who continue to support this project.

Thursday, October 23, 2014

Don't use ESAPI encoders in ColdFusion/Railo!

We came across this issue awhile ago at my place of work and I never posted anything about it.  I recently saw Jason Dean's notes from CFSummt about using ESAPI4CF for CF8/9 and it reminded me that I should point this out.

I do not recommend using the encoders in ESAPI or ESAPI4CF on ColdFusion 8.  This version of CF Server includes ESAPI v1.4.4 which had a documented concurrency issue around the encoders.  The encoder library was rewritten for ESAPI v2.x so I believe this was addressed.

We personally experienced this concurrency issue under ColdFusion 8 at my place of work.  The issue can arise is high volume applications and causes the thread to hang.  The only way we could find to recover from this hung thread was a CF server restart.

In the end, we ended up switching all of our encoder logic over to the OWASP Java Encoder project and have not had any issues with it.

So my recommendation - don't use ESAPI or ESAPI4CF encoders for CF8.

While we are on the subject I wanted to bring up the use of the ESAPI encoders for any CF/Railo version and yes, I am including the native encodeFor* functions added in CF10.  I say again, don't use them!  They are performance hogs!

While I was testing the Java Encoder alternative I did some performance testing comparing the ESAPI encoders to the Java Encoders.  Huge difference!  Now I will make it known that the Java Encoders are not as strict on HTML encoding everything outside of the low ASCII range, in fact it encodes very few characters.  But I put my trust in this project (which is an active project unlike ESAPI) that they know what they are doing and only encode what is necessary to ensure a safe browsing experience.  With that said, the numbers do not lie.  The Java Encoders were way faster than the ESAPI encoders in the majority of cases.

Keep in mind that the native encodeFor* functions added in CF10 and Railo are simply using the ESAPI encoders under the hood.  I performance tested against the native encoders as well and came away with the same results as testing directly against the ESAPI encoders - they are SLOW!  I am quite sure it is due to the excessive amount of characters it is attempting to encode OR that the library has not been maintained in quite awhile as no one is improving it.

Regardless of the reason, my recommendation on ESAPI encoders overall - don't use them!  Choose the OWASP Java Encoder project as an alternative.

UPDATE 2015-03-30: The ESAPI for CFML project will be including a setting to allow you to use the Java Encoder project instead of the ESAPI encoders in the 2.x version which is still in development.  This will give you the flexibility to choose which encoder solution works best for your project.

Wednesday, April 2, 2014

My thoughts on the inevitable ESAPI demotion

So I have taken a few days to think about the upcoming demotion of ESAPI as an OWASP flagship project - see here.  Now I am not sure if my thoughts on this are based from what I read in the announcement and email threads or if I came to this thought on my own.  Regardless, I'm not looking for credit, just wanted to express my thoughts and how this may effect ESAPI4CF.

I feel ESAPI is still a viable project for sure.  It packages the tools to secure your web application in a nice and tidy bundle so everything is at your disposal.  It integrates some of the modules, particularly the Authenticator/CurrentUser with the Logger and each module uses the Logger therefore logging not only the security message, but the user data along with it - this is one of my favorite parts of it.

However, perhaps ESAPI bit off more than it could chew.  Each module of ESAPI tends to have a different focus from authentication, access control, encoding, encryption, logging, etc.  Each of these areas tends to have a different knowledge expert.  Security is far too important to be addressing all of these areas as a "jack of all trades, master of none".  The masters are the ones that need to be involved with each module.

I did see the idea somewhere in that news of splitting the viable ESAPI modules off into their own OWASP projects.  I believe the term that was used was "salvaging" ESAPI - possibly applicable.  I see this as a great idea!  There is already the Java Encoder and Java HTML Sanitizer projects and these could both easily be used in place of the ESAPI Encoder and the ESAPI Validator is/getValidSafeHTML() pieces respectively.  This, I know, was mentioned in the threads somewhere.

However, let's take a another look here.  I agree that breaking the ESAPI modules off into their own projects, with each their own leads, contributors, knowledge experts, is a great idea.  They will each get the focus they well deserve.  But let's not disregard ESAPI itself.  It brings all the security tools together and I don't think that should change.  What I am saying is, instead of the ESAPI project creating each module as part of its project, change ESAPI to include other projects as part of its library.  ESAPI could then focus on integrating the modules and providing the pieces which are not in individual projects.  It could be the glue that brings the OWASP projects together.

ESAPI essentially does this already with the Validator is/getValidSafeHTML() methods which leverage AntiSamy under the hood.  So replacing this one should be fairly straightforward - rip out AntiSamy and replace with the HTML Sanitizer.

As for the Java Encoder, the underlying classes to the ESAPI Encoder module could all be dropped and the DefaultEncoder guts replaced with calls to the Java Encoder.  Easy enough!

In fact, ESAPI was architected to provide "DEFAULT" implementations of the interfaces, so you don't even have to wait for the project to do this - create your own implementation doing exactly this!

Encryption is a big one and I did see it mentioned how this should become its own OWASP project and I wholeheartedly agree with that statement.  That is a complex beast best left to the masters. I welcome seeing this as its own project if it is not already (in case I missed that announcement).

Now there are modules like the Authenticator and AccessController that I don't really see being their own projects which is also fine.  I could be wrong, perhaps there is justification for them being a separate project but I'll leave that decision to those smarter than myself.  Even without their own project, these could still be key components to the ESAPI project and the integration with the other modules.

Anyway, just my thoughts on the announcement.  Personally, I feel that I may not wait for a decision or direction.  I will most likely move forward with changing ESAPI4CF's Encoder module and Validator is/getValidSafeHTML() methods over to the Java Encoder/Sanitzer projects in the next development cycle.  Why?  Cause I think its a great idea and will only help make the ESAPI4CF project better.

Saturday, March 29, 2014

Off-the-Wall Security: ESAPI No Longer an OWASP Flagship Project

Off-the-Wall Security: ESAPI No Longer an OWASP Flagship Project: I read the news today oh, boy… By now, you’ve probably heard about several of the OWASP board members (and perhaps, some bored members...

Tuesday, November 26, 2013

ESAPI4CF v1.1.0 released!

A new minor version of ESAPI4CF is now available!  I decided to increment the minor version this go around because there were several significant changes included so it seemed appropriate.

The full release notes can be found here: https://github.com/damonmiller/esapi4cf/releases/tag/v1.1.0a.

The ESAPI4CF API documentation and tutorials have also been updated: http://damonmiller.github.io/esapi4cf/.

First thing I'd like to point out is that I finally took the time to learn a little more about GitHub and found out how to tag and define releases.  So v1.1 marks the first release that shows under "Releases" on the GitHub project.  Sorry, but you probably don't want v1.0.x anyway since there is some good stuff in this release including some important bug fixes.  Version history won't be an issue moving forward now that I "git" it. :)

Now only 1 line to init ESAPI

Previously, you had to init the main ESAPI component then make a separate call to tell ESAPI where you resources folder was located.  I've combined this into a single call so you can now pass the resources folder location into the ESAPI init.  The 2 lines for init was annoying me so I imagine others would feel the same.

You can still make the 2 line init in v1.1 and it will continue to work as before or use setResourceDirectory() if you need to change the location later on.

Encoder is now ESAPI4J dependent

Not sure why I didn't do this sooner.  I just think it never clicked before.  I think I looked at so many articles on people using the encoders from ESAPI4J enough times where it finally just occurred to me, why am I not doing that?  ESAPI4CF is dependent on ESAPI4J for several things, why not the encoders?  Let Java do the heavy lifting and we get a small performance improvement per encoder call.  Win-win if I do say so.

JSESSIONID cookie now has HttpOnly and Secure set

This one is in the ESAPI specification but ESAPI4CF was just not doing it correctly until now.  Railo and CF10 can do this and CF9 can set HttpOnly but if you wanted Secure in CF9 or either attribute in CF8 you were out of luck previously unless you knew to use Java to override the cookie.  Now ESAPI4CF takes care of this for you and it does it without any additional calls.  It is part of the initial request/response registration using setCurrentHTTP() so no changes required on your part to take advantage of what should be a requirement in any modern day web application.

Password strength now fails if it matches the accountName

This was an addition to the ESAPI v2 specification but I added it now because it is an important one to have.  Your password cannot be the same as your accountName - enough said.

Internationalization support for isValidNumber/getValidNumber

I am big on I18N support (probably because this is one of my main responsibilities at work) and ESAPI4J severely lacks in this department.  This is the first of several changes I will be making to ESAPI4CF which deviate from ESAPI4J around I18N support - the other changes will involve resource bundling for translations.  I added an additional required argument to the isValidNumber and getValidNumber methods to provide a number format instance to use to parse the input.  This allows you to pass in localized numbers that can be parsed and you will get back a numeric object.  The date validator methods already supported a format argument which provides the identical functionality - why this was never added for the number validators is beyond me.

What I really like about the "format" argument approach for both date and number validators is you can use the default Java formatter instances if you choose or if you are like me and prefer ICU4J, you can pass those formatter instances as well and it works just the same.  Well it actually works better because ICU4J is just better. :)

The other number validators, double, integer, etc, were not altered.  If you are dealing with localized numbers you should call the getValidNumber validator first to get back a numeric object then call the appropriate numeric validator for your scenario.  Just FYI, the getValidNumber method already does call getValidDouble after it parses the number so a separate call is not necessary.

SafeFile now supports all the Java File methods

I believe this just got missed the first go around.  SafeFile is supposed to be a CF representation of the ESAPI4J SafeFile which extends java.io.File but the native Java File method wrappers were never implemented.  These now all exist.


So those are the high points of this release.  There are some smaller items as well plus several important bug fixes so check out the release notes for the details or better yet, just download ESAPI4CF and check it out yourself.  I am trying to make ESAPI4CF as easy to understand and implement as possible but it takes community feedback to get there.  Help is always appreciated!  Thanks!

Tuesday, October 29, 2013

Using OData complex filters in REST and CFML

Lately I have been researching into how to build REST web services.  I have been using Taffy to make this all happen which has been a great experience overall. I have always been a strong advocate of following standards whenever possible so I wanted to do so as well with my REST web services.  Through all my searching I stumbled upon this document of RESTful Best Practices - very good read, I highly recommend it.  One of the areas I really wanted to understand how to do right was around filtering in REST.  The Best Practices document mentioned using OData for complex filtering.  Once I saw this was possible I knew I had to have it.  Luckily for me there is a OData4J project but unlucky for me was the fact that no matter how much I searched I could not find anyone who used OData4J in CFML.  Now I am not saying that articles on this do not exist but I was not able to find any.  If anyone knows of any please pass them on to me so I can compare notes.

So this post is all about how to get OData4J working in CFML, specially just one method in OData4J - the method that parses the OData style filters into something we can easily use and therefore turn into usable SQL.  Be aware that for my purposes to get this working I only needed it to work in Railo (4.1) so I have not tested this in any ColdFusion Server versions.  I do not see any reason why this would not work but some minor tweaks may be necessary if it does not.

REST filter

So for example let's say for instance you pass the below to your REST web service.

https://mydomain.com/REST/People/?filter=startswith(firstName, 'd') and isActive eq 1

How do we go about turning that into usable SQL?  Let's take a look...

Download OData4J

First thing you will need is the jar.  The download is available from the http://odata4j.org/ website.  Extract the zip.  The only jar you need for the purpose of this feature is odata4j-0.7.0-nojpabundle.jar.  Add the jar to your WEB-INF/lib and restart CFML.

Make the call

As I mentioned there is only one method in OData4J we are interested in for the purposes of parsing these complex filters.  The method is located here: org.odata4j.producer.resources.OptionsQueryParser#parseFilter() and only takes 1 string argument which is the complex filter used in the REST call.
So with that we can very easily make a call to this method.

createObject("java", "org.odata4j.producer.resources.OptionsQueryParser").parseFilter(javaCast("string", "myColumn eq 'abc'"))

One thing I will point out is that you cannot pass the parseFilter method an empty string.  It does not like this and will throw an error which isn't very informative.  Why they could not easily handle an empty value and just return null is beyond me.  Anyway, that stumped me for awhile so I'll save you the trouble - wrap this call in a condition checking the filter length first.

What do we get for our troubles?

The return we get for this call is not SQL - that would be too easy and make a very short write up.  What we get is a Java object of class EqExpression.  At this point of implementing anything new we would typically resort to documentation which is found here: http://odata4j.org/v/0.7/javadoc/.  Let me tell you these are the most useless docs I have read in a long time.  Yes all of the classes and methods link to each other which is great but it lacks any text explaining what anything does.  Completely useless!  So we resort to our second best method of exploring new libraries in ColdFusion... writeDump()!  And there was a lot of that going on to figure this one out.

This object has two methods that give us the data we need: getLHS() and getRHS().  I am guessing those mean LHS - left-hand side and RHS - right-hand side.  I could not find anything that explained what these meant.  Exploring the getLHS() method gives us a getPropertyName() method which returns us 'myColumn' and exploring the getRHS() method gives us a getValue() method which returns us 'abc'.  Cool, very simply we can use that to write our SQL.  But honestly, this wasn't very complex and if that's all you need out of your REST filters, you may as well just skip OData.  So let's put this to some good use.

createObject("java", "org.odata4j.producer.resources.OptionsQueryParser").parseFilter(javaCast("string", "myColumn eq 'abc' and myColumn2 ne 'xyz'"))

Okay this doesn't look too much harder than the last example. Well when you look at what you get back you will see just how wrong you are.  This time you get back a Java object of class AndExpression.  This object still has both getLHS() and and getRHS() but now they each return a EqExpression object.  Now if we added yet another 'and' to the above filter or used the 'substringof' in the filter, things will just keep getting more and more complicated.

What this level of complexity calls for is a recursive function.  This will allow the function to call itself whenever it encounters an object which contains other objects.  So let's jump into it.

var filter = "myColumn eq 'abc' and myColumn2 ne 'xyz'";
var parsedFilter = createObject("java", "org.odata4j.producer.resources.OptionsQueryParser").parseFilter(javaCast("string", filter));
var parsedSQL = expressionToSQL(parsedFilter);
variables.operatorsMap = {
 "EqExpression": "=",
 "NeExpression": "!=",
 "GtExpression": ">",
 "GeExpression": ">=",
 "LtExpression": "<",
 "LeExpression": "<=",
 "AndExpression": "AND",
 "OrExpression": "OR",
 "NotExpression": "NOT"

function expressionToSQL(required filter) {
 // this function does not handle everything yet:
 // works: and, or, eq, ne, lt, le, gt, ge, startswith, endswith, substringof
 // not working: paranthesis, not, arithmetic operators, no methods except noted above
 var sql = createObject("java", "java.lang.StringBuilder").init();
 var params = {};
 var type = filter.toString();

 if (listFind("EqExpression,NeExpression,GtExpression,GeExpression,LtExpression,LeExpression", type)) {
  if (filter.getLHS().toString() == "EntitySimpleProperty") {
   var columnName = filter.getLHS().getPropertyName();
   sql.append(columnName & variables.operatorsMap[type] & ":" & columnName);
   params[columnName] = filter.getRHS().getValue();
  else {
   // ideally you want to log a warning here and skip this object

   throw(message="Could not convert expression to SQL.", detail="Type '" & filter.getLHS().toString() & "' unaccounted for.");
 else if (listFind("StartsWithMethodCallExpression,EndsWithMethodCallExpression,SubstringOfMethodCallExpression", type)) {
  var columnName = filter.getTarget().getPropertyName();
  sql.append(columnName & " LIKE :" & columnName);
  if (type == "StartsWithMethodCallExpression") {
   // startswith converts to 'LIKE value%'
   params[columnName] = filter.getValue().getValue() & "%";
  else if (type == "EndsWithMethodCallExpression") {
   // endswith converts to 'LIKE %value'
   params[columnName] = "%" & filter.getValue().getValue();
  else if (type == "SubstringOfMethodCallExpression") {
   // substringof converts to 'LIKE %value%'
   params[columnName] = "%" & filter.getValue().getValue() & "%";
 else if (listFind("AndExpression,OrExpression", type)) {
  // recursively call method passing LHS object
  var lResult = expressionToSQL(filter.getLHS());
  // add returned SQL to our SQL
  // merge parameters together

  // recursively call method passing RHS object
  var rResult = expressionToSQL(filter.getRHS());
  // add returned SQL to our SQL with proper operator
  sql.append(" " & variables.operatorsMap[type] & " " & rResult.sql);
  // merge parameters together
 else {
  // ideally you want to log a warning here and skip this object
  throw(message="Could not convert expression to SQL.", detail="Type '" & type & "' unaccounted for.");

 return {
  "sql": sql.toString(),
  "parameters": params

Now as you can see from the comments in the function not all operators and methods available in OData are accounted for. For my purposes this function suits my needs. I am sure in the future I will want some if not all of the other features of the OData filter but for now this will do. This is also a great starting point for anyone needing this feature and you can build on this to finish out the other OData features.

You will also notice that I split the generated SQL from the values.  This makes it very easy to pass this onto your Query() object or in ORMExecuteQuery() while keeping SQL injection prevention in mind, especially with an interface as exposed to the outside world as a REST web service will be.  You may want to also add validation to this function, perhaps pass a second argument with a list of allowed columns to perform filtering on.  This serves two purposes 1) ensures no one can break the query by passing invalid columns and 2) restricts the columns that can be filtered in case you have some you do not want to be filtered on like large text.

So back to our original example.

https://mydomain.com/REST/People/?filter=startswith(firstName, 'd') and isActive eq 1

Passing through the above functions this will give you back the below:

With this return data, assuming we assigned the result to a variable named 'parsed', we can call ORMExecuteQuery("FROM People WHERE " & parsed.sql, parsed.parameters) and will give you a successful query run.

So that's it!  This should be enough to get you started on your way to more complex filters in REST web services.  If anyone gets additional operators, methods, or other parts of the conversion working and wants to share, please feel free.

UPDATE: This is now a project on GitHub available here: https://github.com/damonmiller/odata4cf.  Please check it out for the latest working version of this code.  Thanks!

Saturday, August 31, 2013

ESAPI4CF site is live and tutorials are in the works

Just wanted to write a quick note letting those followers know that there is now a GitHub.io site for ESAPI4CF. It is available at http://damonmiller.github.io/esapi4cf/.

Included on this site is some basic overview of ESAPI4CF, JavaDoc style API docs, and the start of tutorials to walk you through each module. The tutorials are still being worked on but the basic setup is now completed so please take a look and let me know any feedback you have.

As always, looking for any feedback and/or contributors if your are interested.