Tuesday, November 26, 2013

ESAPI4CF v1.1.0 released!

A new minor version of ESAPI4CF is now available!  I decided to increment the minor version this go around because there were several significant changes included so it seemed appropriate.

The full release notes can be found here: https://github.com/damonmiller/esapi4cf/releases/tag/v1.1.0a.

The ESAPI4CF API documentation and tutorials have also been updated: http://damonmiller.github.io/esapi4cf/.

First thing I'd like to point out is that I finally took the time to learn a little more about GitHub and found out how to tag and define releases.  So v1.1 marks the first release that shows under "Releases" on the GitHub project.  Sorry, but you probably don't want v1.0.x anyway since there is some good stuff in this release including some important bug fixes.  Version history won't be an issue moving forward now that I "git" it. :)

Now only 1 line to init ESAPI

Previously, you had to init the main ESAPI component then make a separate call to tell ESAPI where you resources folder was located.  I've combined this into a single call so you can now pass the resources folder location into the ESAPI init.  The 2 lines for init was annoying me so I imagine others would feel the same.

You can still make the 2 line init in v1.1 and it will continue to work as before or use setResourceDirectory() if you need to change the location later on.

Encoder is now ESAPI4J dependent

Not sure why I didn't do this sooner.  I just think it never clicked before.  I think I looked at so many articles on people using the encoders from ESAPI4J enough times where it finally just occurred to me, why am I not doing that?  ESAPI4CF is dependent on ESAPI4J for several things, why not the encoders?  Let Java do the heavy lifting and we get a small performance improvement per encoder call.  Win-win if I do say so.

JSESSIONID cookie now has HttpOnly and Secure set

This one is in the ESAPI specification but ESAPI4CF was just not doing it correctly until now.  Railo and CF10 can do this and CF9 can set HttpOnly but if you wanted Secure in CF9 or either attribute in CF8 you were out of luck previously unless you knew to use Java to override the cookie.  Now ESAPI4CF takes care of this for you and it does it without any additional calls.  It is part of the initial request/response registration using setCurrentHTTP() so no changes required on your part to take advantage of what should be a requirement in any modern day web application.

Password strength now fails if it matches the accountName

This was an addition to the ESAPI v2 specification but I added it now because it is an important one to have.  Your password cannot be the same as your accountName - enough said.

Internationalization support for isValidNumber/getValidNumber

I am big on I18N support (probably because this is one of my main responsibilities at work) and ESAPI4J severely lacks in this department.  This is the first of several changes I will be making to ESAPI4CF which deviate from ESAPI4J around I18N support - the other changes will involve resource bundling for translations.  I added an additional required argument to the isValidNumber and getValidNumber methods to provide a number format instance to use to parse the input.  This allows you to pass in localized numbers that can be parsed and you will get back a numeric object.  The date validator methods already supported a format argument which provides the identical functionality - why this was never added for the number validators is beyond me.

What I really like about the "format" argument approach for both date and number validators is you can use the default Java formatter instances if you choose or if you are like me and prefer ICU4J, you can pass those formatter instances as well and it works just the same.  Well it actually works better because ICU4J is just better. :)

The other number validators, double, integer, etc, were not altered.  If you are dealing with localized numbers you should call the getValidNumber validator first to get back a numeric object then call the appropriate numeric validator for your scenario.  Just FYI, the getValidNumber method already does call getValidDouble after it parses the number so a separate call is not necessary.

SafeFile now supports all the Java File methods

I believe this just got missed the first go around.  SafeFile is supposed to be a CF representation of the ESAPI4J SafeFile which extends java.io.File but the native Java File method wrappers were never implemented.  These now all exist.


So those are the high points of this release.  There are some smaller items as well plus several important bug fixes so check out the release notes for the details or better yet, just download ESAPI4CF and check it out yourself.  I am trying to make ESAPI4CF as easy to understand and implement as possible but it takes community feedback to get there.  Help is always appreciated!  Thanks!

Tuesday, October 29, 2013

Using OData complex filters in REST and CFML

Lately I have been researching into how to build REST web services.  I have been using Taffy to make this all happen which has been a great experience overall. I have always been a strong advocate of following standards whenever possible so I wanted to do so as well with my REST web services.  Through all my searching I stumbled upon this document of RESTful Best Practices - very good read, I highly recommend it.  One of the areas I really wanted to understand how to do right was around filtering in REST.  The Best Practices document mentioned using OData for complex filtering.  Once I saw this was possible I knew I had to have it.  Luckily for me there is a OData4J project but unlucky for me was the fact that no matter how much I searched I could not find anyone who used OData4J in CFML.  Now I am not saying that articles on this do not exist but I was not able to find any.  If anyone knows of any please pass them on to me so I can compare notes.

So this post is all about how to get OData4J working in CFML, specially just one method in OData4J - the method that parses the OData style filters into something we can easily use and therefore turn into usable SQL.  Be aware that for my purposes to get this working I only needed it to work in Railo (4.1) so I have not tested this in any ColdFusion Server versions.  I do not see any reason why this would not work but some minor tweaks may be necessary if it does not.

REST filter

So for example let's say for instance you pass the below to your REST web service.

https://mydomain.com/REST/People/?filter=startswith(firstName, 'd') and isActive eq 1

How do we go about turning that into usable SQL?  Let's take a look...

Download OData4J

First thing you will need is the jar.  The download is available from the http://odata4j.org/ website.  Extract the zip.  The only jar you need for the purpose of this feature is odata4j-0.7.0-nojpabundle.jar.  Add the jar to your WEB-INF/lib and restart CFML.

Make the call

As I mentioned there is only one method in OData4J we are interested in for the purposes of parsing these complex filters.  The method is located here: org.odata4j.producer.resources.OptionsQueryParser#parseFilter() and only takes 1 string argument which is the complex filter used in the REST call.
So with that we can very easily make a call to this method.

createObject("java", "org.odata4j.producer.resources.OptionsQueryParser").parseFilter(javaCast("string", "myColumn eq 'abc'"))

One thing I will point out is that you cannot pass the parseFilter method an empty string.  It does not like this and will throw an error which isn't very informative.  Why they could not easily handle an empty value and just return null is beyond me.  Anyway, that stumped me for awhile so I'll save you the trouble - wrap this call in a condition checking the filter length first.

What do we get for our troubles?

The return we get for this call is not SQL - that would be too easy and make a very short write up.  What we get is a Java object of class EqExpression.  At this point of implementing anything new we would typically resort to documentation which is found here: http://odata4j.org/v/0.7/javadoc/.  Let me tell you these are the most useless docs I have read in a long time.  Yes all of the classes and methods link to each other which is great but it lacks any text explaining what anything does.  Completely useless!  So we resort to our second best method of exploring new libraries in ColdFusion... writeDump()!  And there was a lot of that going on to figure this one out.

This object has two methods that give us the data we need: getLHS() and getRHS().  I am guessing those mean LHS - left-hand side and RHS - right-hand side.  I could not find anything that explained what these meant.  Exploring the getLHS() method gives us a getPropertyName() method which returns us 'myColumn' and exploring the getRHS() method gives us a getValue() method which returns us 'abc'.  Cool, very simply we can use that to write our SQL.  But honestly, this wasn't very complex and if that's all you need out of your REST filters, you may as well just skip OData.  So let's put this to some good use.

createObject("java", "org.odata4j.producer.resources.OptionsQueryParser").parseFilter(javaCast("string", "myColumn eq 'abc' and myColumn2 ne 'xyz'"))

Okay this doesn't look too much harder than the last example. Well when you look at what you get back you will see just how wrong you are.  This time you get back a Java object of class AndExpression.  This object still has both getLHS() and and getRHS() but now they each return a EqExpression object.  Now if we added yet another 'and' to the above filter or used the 'substringof' in the filter, things will just keep getting more and more complicated.

What this level of complexity calls for is a recursive function.  This will allow the function to call itself whenever it encounters an object which contains other objects.  So let's jump into it.

var filter = "myColumn eq 'abc' and myColumn2 ne 'xyz'";
var parsedFilter = createObject("java", "org.odata4j.producer.resources.OptionsQueryParser").parseFilter(javaCast("string", filter));
var parsedSQL = expressionToSQL(parsedFilter);
variables.operatorsMap = {
 "EqExpression": "=",
 "NeExpression": "!=",
 "GtExpression": ">",
 "GeExpression": ">=",
 "LtExpression": "<",
 "LeExpression": "<=",
 "AndExpression": "AND",
 "OrExpression": "OR",
 "NotExpression": "NOT"

function expressionToSQL(required filter) {
 // this function does not handle everything yet:
 // works: and, or, eq, ne, lt, le, gt, ge, startswith, endswith, substringof
 // not working: paranthesis, not, arithmetic operators, no methods except noted above
 var sql = createObject("java", "java.lang.StringBuilder").init();
 var params = {};
 var type = filter.toString();

 if (listFind("EqExpression,NeExpression,GtExpression,GeExpression,LtExpression,LeExpression", type)) {
  if (filter.getLHS().toString() == "EntitySimpleProperty") {
   var columnName = filter.getLHS().getPropertyName();
   sql.append(columnName & variables.operatorsMap[type] & ":" & columnName);
   params[columnName] = filter.getRHS().getValue();
  else {
   // ideally you want to log a warning here and skip this object

   throw(message="Could not convert expression to SQL.", detail="Type '" & filter.getLHS().toString() & "' unaccounted for.");
 else if (listFind("StartsWithMethodCallExpression,EndsWithMethodCallExpression,SubstringOfMethodCallExpression", type)) {
  var columnName = filter.getTarget().getPropertyName();
  sql.append(columnName & " LIKE :" & columnName);
  if (type == "StartsWithMethodCallExpression") {
   // startswith converts to 'LIKE value%'
   params[columnName] = filter.getValue().getValue() & "%";
  else if (type == "EndsWithMethodCallExpression") {
   // endswith converts to 'LIKE %value'
   params[columnName] = "%" & filter.getValue().getValue();
  else if (type == "SubstringOfMethodCallExpression") {
   // substringof converts to 'LIKE %value%'
   params[columnName] = "%" & filter.getValue().getValue() & "%";
 else if (listFind("AndExpression,OrExpression", type)) {
  // recursively call method passing LHS object
  var lResult = expressionToSQL(filter.getLHS());
  // add returned SQL to our SQL
  // merge parameters together

  // recursively call method passing RHS object
  var rResult = expressionToSQL(filter.getRHS());
  // add returned SQL to our SQL with proper operator
  sql.append(" " & variables.operatorsMap[type] & " " & rResult.sql);
  // merge parameters together
 else {
  // ideally you want to log a warning here and skip this object
  throw(message="Could not convert expression to SQL.", detail="Type '" & type & "' unaccounted for.");

 return {
  "sql": sql.toString(),
  "parameters": params

Now as you can see from the comments in the function not all operators and methods available in OData are accounted for. For my purposes this function suits my needs. I am sure in the future I will want some if not all of the other features of the OData filter but for now this will do. This is also a great starting point for anyone needing this feature and you can build on this to finish out the other OData features.

You will also notice that I split the generated SQL from the values.  This makes it very easy to pass this onto your Query() object or in ORMExecuteQuery() while keeping SQL injection prevention in mind, especially with an interface as exposed to the outside world as a REST web service will be.  You may want to also add validation to this function, perhaps pass a second argument with a list of allowed columns to perform filtering on.  This serves two purposes 1) ensures no one can break the query by passing invalid columns and 2) restricts the columns that can be filtered in case you have some you do not want to be filtered on like large text.

So back to our original example.

https://mydomain.com/REST/People/?filter=startswith(firstName, 'd') and isActive eq 1

Passing through the above functions this will give you back the below:

With this return data, assuming we assigned the result to a variable named 'parsed', we can call ORMExecuteQuery("FROM People WHERE " & parsed.sql, parsed.parameters) and will give you a successful query run.

So that's it!  This should be enough to get you started on your way to more complex filters in REST web services.  If anyone gets additional operators, methods, or other parts of the conversion working and wants to share, please feel free.

UPDATE: This is now a project on GitHub available here: https://github.com/damonmiller/odata4cf.  Please check it out for the latest working version of this code.  Thanks!

Saturday, August 31, 2013

ESAPI4CF site is live and tutorials are in the works

Just wanted to write a quick note letting those followers know that there is now a GitHub.io site for ESAPI4CF. It is available at http://damonmiller.github.io/esapi4cf/.

Included on this site is some basic overview of ESAPI4CF, JavaDoc style API docs, and the start of tutorials to walk you through each module. The tutorials are still being worked on but the basic setup is now completed so please take a look and let me know any feedback you have.

As always, looking for any feedback and/or contributors if your are interested.

Monday, March 4, 2013

CFESAPI is now ESAPI4CF and now includes docs!

Yes, the ESAPI for ColdFusion/CFML project is still alive!  I haven't posted about it in awhile for 2 reasons.  One, I have not had much time to put towards it but I am trying harder now.  And two, I was always horrible at blogging so finding the time to write something up was not a priority. Anyway, I made some time for both and I have a few things to go over.

I decided to rename CFESAPI to ESAPI4CF.  I thought this just made more sense.  The official OWASP project is defined as "ESAPI for ColdFusion/CFML" and well, enough said.  The GitHub repository was renamed and I created a bogus project under the old "cfesapi" repo with links to get to the updated repository.

I am aware that ColdFusion Server now includes the esapi.jar since the 8.0.1 patch and the 9.0.1 patch (and is now part of 9.0.2) along with version 10.  Railo 4 also includes the jar.  So since the jar is now so popular with the CFML engines, I no longer include it with the library.  In fact, the library requirements have changed so that you must be on a CFML version which includes the esapi.jar.  Again, makes more sense and that seems to be my thing in this post.

So back to the GitHub repo, I have been committing against the development branch once and awhile and I decided to move development to master.  I think this code is pretty stable and it also includes documentation...first time for everything!  You will notice that I did a bit of reorganization.  I moved the main source into a subfolder so that I could add additional folders for apirefs, swingset, and libs.  I also moved the unit tests out of the main source folder and into its own.  This will allow you to better know which folders to deploy to production and which to not (hint: only the main source should go to production).  The lib folder has your additional jar dependencies needed to run ESAPI4CF.  These still need to go into your /WEB-INF/lib/ folder as always.

So as for what has changed, well the addition of documentation is big!  This is probably what I get asked about the most. The apiref has your JavaDoc-like references to the core library.  I have been working to port the ESAPI SwingSet into ESAPI4CF.  This is not done yet but I was excited to get something new out and felt it time to make it happen.  The SwingSet has a lot of explanation, sample usage, and labs.  It is not all working yet as this is a work in progress but I do hope that it helps out a lot.

The last thing I should note and this is very important so pay attention.  The previous master code attempted to support ESAPI 2.0-something or other.  This became increasingly difficult with the expectation to support back to ColdFusion 8.  I know, CF8 is not supported by Adobe anymore but I made that decision awhile back when it still was.  Since I was the majority of the way complete with this work when support was dropped, I didn't want to scrap what I had so I decided to just finish it.  Anyway, I decided that since CF8 was my oldest version to support, that ESAPI4CF would support the same esapi.jar version added to CF8 with the patch, ESAPI-1.4.4.  Some code tweaks had to be made in order to run on CF9's ESAPI-2.0_rc10 version but they were minor. Regardless, back when I made this decision I was going to release an ESAPI4CF-1.4.4 for CF8+, ESAPI4CF-2.0_rc10 for CF9+, and ESAPI4CF-2.0.1 for CF10+.  But since this took so long and Adobe has mentioned CF11 coming only a year after CF10 (we'll see), I decided that next I will tackle whatever the latest version currently is in an attempt to play catch up.  So I will for sure be skipping the ESAPI4CF-2.0_rc10 for CF9+ and jumping to whatever the latest is when I decide to start on that.  Not guaranteeing I will ready to begin while CF10 is still the latest.  Obviously, I still want 1.4.4 completed so that something is out there and I will continue to work on that version to get it feature complete and address bugs.

So anyway, that's my brain dump this time around.  I still have hope that I can get some contributors to ESAPI4CF.  Maybe now that there is documentation others may be more willing to help out.  Just let me know.

UPDATE: Someone pointed out the lack of a link to the GitHub project so here it is: http://damonmiller.github.com/esapi4cf/.  Thanks Tony!