Hacking OpenAM – An Open Response to Radovan Semancik

I have been working with Sun, Oracle and ForgeRock products for some time now and am always looking for new and interesting topics that pertain to theirs and other open source identity products.  When Google alerted me to the following blog posting, I just couldn’t resist:

Hacking OpenAM, Level: Nightmare

Radovan Semancik | February 25, 2015

There were two things in the alert that caught my attention.  The first was the title and the obvious implications that it contained and the second is the author of the blog and the fact that he’s associated with Evolveum, a ForgeRock OpenIDM competitor.

The identity community is relatively small and I have read many of Radovan’s postings in the past.  We share a few of the same mailing lists and I have seen his questions/comments come up in those forums from time to time.  I have never met Radovan in person, but I believe we are probably more alike than different.  We share a common lineage; both being successful Sun identity integrators.  We both agree that open source identity is preferable to closed source solutions.  And it seems that we both share many of the same concerns over Internet privacy.  So when I saw this posting, I had to find out what Radovan had discovered that I must have missed over the past 15 years in working with these products.  After reading his blog posting, however, I do not share his same concerns nor do I come to the same conclusions. In addition, there are several inaccuracies in the blog that could easily be misinterpreted and are being used to spread fear, uncertainty, and doubt around OpenAM.

What follows are my responses to each of Radovan’s concerns regarding OpenAM. These are based on my experiences of working with the product for over 15 years and as Radovan aptly said, “your mileage may vary.”

In the blog Radovan comments “OpenAM is formally Java 6. Which is a problem in itself. Java 6 does not have any public updates for almost two years.”

ForgeRock is not stuck with Java 6.  In fact, OpenAM 12 supports Java 7 and Java 8.  I have personally worked for governmental agencies that simply cannot upgrade their Java version for one reason or another.  ForgeRock must make their products both forward looking as well as backward compatible in order to support their vast customer base.

In the blog Radovan comments “OpenAM also does not have any documents describing the system architecture from a developers point of view.”


I agree with Radovan that early versions of the documentation were limited.  As with any startup, documentation is one of the things that suffers during the initial phases, but over the past couple of years, this has flipped.  Due to the efforts of the ForgeRock documentation team I now find most of my questions answered in the ForgeRock documentation.  In addition, ForgeRock is a commercial open source company, so they do not make all high value documents publicly available.  This is part of the ForgeRock value proposition for subscription customers.

In the blog Radovan comments “OpenAM is huge. It consists of approx. 2 million lines of source code. It is also quite complicated. There is some component structure. But it does not make much sense on the first sight.”


I believe that Radovan is confusing the open source trunk with commercial open source product.  Simply put, ForgeRock does not include all code from the trunk in the OpenAM commercial offering.  As an example the extensions directory, which is not part of the product, has almost 1000 Java files in it.

More importantly, you need to be careful in attempting to judge functionality, quality, and security based solely on the number of lines of code in any product.  When I worked at AT&T, I was part of a development team responsible for way more than 2M lines of code.  My personal area of responsibility was directly related to approximately 250K lines of code that I knew inside and out.  A sales rep could ask me a question regarding a particular feature or issue and I could envision the file, module, and even where in the code the question pertained (other developers can relate to this).  Oh, and this code was rock solid.

In the blog Radovan comments that the “bulk of the OpenAM code is still efficiently Java 1.4 or even older.”


Is this really a concern?  During the initial stages of my career as a software developer, my mentor beat into my head the following mantra:

If it ain’t broke, don’t fix it!

I didn’t always agree with my mentor, but I was reminded of this lesson each time I introduced bugs into code that I was simply trying to make better.  Almost 25 years later this motto has stuck with me but over time I have modified it to be:

If it ain’t broke, don’t fix it, unless there is a damn good reason to do so!

It has been my experience that ForgeRock follows a mantra similar to my modified version.  When they decide to refactor the code, they do so based on customer or market demand not just because there are newer ways to do it.  If the old way works, performance is not limited, and security is not endangered, then why change it.   Based on my experience with closed-source vendors, this is exactly what they do; their source code, however, is hidden so you don’t know how old it really is.

A final thought on refactoring.  ForgeRock has refactored the Entitlements Engine and the Secure Token Service (both pretty mammoth projects) all while fixing bugs, responding to RFEs, and implementing new market-driven features such as:

• Adaptive Authentication
• New XUI Interface
• REST STS
• WS-TRUST STS
• OpenID Connect
• OAuth 2.0
• Core Token Service

In my opinion, ForgeRock product development is focused on the right areas.

In the blog Radovan comments “OpenAM is in fact (at least) two somehow separate products. There is “AM” part and “FM” part.”


From what I understand, ForgeRock intentionally keeps the federation code independent. This was done so that administrators could easily create and export a “Fedlet” which is essentially a small web application that provides a customer with the code they need to implement SAML in a non-SAML application.  In short, keeping it separate allows for sharing between the OpenAM core services and providing session independent federation capability.  Keeping federation independent has also made it possible to leverage the functionality in other products such as OpenIG.

In the blog Radovan comments “OpenAM debugging is a pain. It is almost uncontrollable, it floods log files with useless data and the little pieces of useful information are lost in it.“


There are several places that you can look in order to debug OpenAM issues and where you look depends mostly on how you have implemented the product.

• Debug Files – OpenAM debug files contain messages output by the developer in order to debug code. This includes a timestamp, the ID of the thread that called the Debug API, the message recorded by the code (error, info, warning), and optionally, a Java stack trace.  As a default, the verbosity level is low, but you can increase the verbosity to see additional messages.
• OpenAM Log Files – OpenAM log files contain operational information for the OpenAM components. They are not designed for debugging purposes, but they may shed additional light in the debugging process. As a default, the verbosity level is low, but you can increase the verbosity to see additional messages.
• Java Container Log Files – The Java container hosting the OpenAM application will also contain log files that may assist in the debugging process.  These log files contain general connection request/response for all connectivity to/from OpenAM.
• Policy Agent Log Files – Policy Agents also generate log messages when used in an OpenAM implementation.  These log files may be stored on the server hosting the Policy Agent or on OpenAM, itself (or both).

I will agree with Radovan’s comments that this can be intimidating at first, but as with most enterprise products, knowing where to look and how to interpret the results is as much of an art as it is a science.  For someone new to OpenAM, debugging can be complex.  For skilled OpenAM customers, integrators, and ForgeRock staff, the debug logs yield a goldmine of valuable information that often assists in the rapid diagnosis of a problem.

Note:  Debugging the source code is the realm of experienced developers and ForgeRock does not expect their customers to diagnose product issues.

For those who stick strictly to the open source version, the learning curve can be steep and they have to rely on the open source community for answers (but hey, what do you want for free).  ForgeRock customers, however, will most likely have taken some training on the product to know where to look and what to look for.  In the event that they need to work with ForgeRock’s 24×7 global support desk, then they will most likely be asked to capture these files (as well as configuration information) in order to submit a ticket to ForgeRock.

In the blog Radovan comments that the “OpenAM is still using obsolete technologies such as JAX-RPC. JAX-RPC is a really bad API.” He then goes on to recommend Apache CXF and states “it takes only a handful of lines of code to do. But not in OpenAM.”

Ironically, OpenAM 12 has a modern REST STS along with a WS-TRUST Apache CXF based implementation (exactly what Radovan recommends).  ForgeRock began migrating away from JAX-RPC towards REST-based web services as early as version 11.0.  Now with OpenAM 12, ForgeRock has a modern (fully documented) REST STS along with a WS-TRUST Apache CXF based implementation (exactly what Radovan recommends).

ForgeRock’s commitment to REST is so strong, in fact, that they have invested heavily in the ForgeRock Common REST (CREST) Framework and API – which is used across all of their products.  They are the only vendor that I am aware of that provides REST interfaces across all products in their IAM stack.  This doesn’t mean, however, that ForgeRock can simply eliminate JAX-RPC functionality from the product.  They must continue to support JAX-RPC to maintain backwards compatibility for existing customers that are utilizing this functionality.

In the blog Radovan comments “OpenAM originated between 1998 and 2002. And the better part of the code is stuck in that time as well.”


In general, Radovan focuses on very specific things he does not like in OpenAM, but ignores all the innovations and enhancements that have been implemented since Sun Microsystems.  As mentioned earlier, ForgeRock has continuously refactored, rewritten, and added several major new features to OpenAM.

“ForgeRock also has a mandatory code review process for every code modification. I have experienced that process first-hand when we were cooperating on OpenICF. This process heavily impacts efficiency and that was one of the reasons why we have separated from OpenICF project.”

I understand how in today’s Agile focused world there is the tendency to shy away from old school concepts such as design reviews and code reviews.  I understand the concerns about how they “take forever” and “cost a lot of money”, but consider the actual cost of a bug getting out the door and into a customer’s environment.  The cost is born by both the vendor and the customer but ultimately it is the vendor who incurs a loss of trust, reputation, and ultimately customers.  Call me old school, but I will opt for code reviews every time – especially when my customer’s security is on the line.

Note:  there is an interesting debate on the effectiveness of code reviews on Slashdot.

Conclusion

So, while I respect Radovan’s opinions, I don’t share them and apparently neither do many of the rather large companies and DOD entities that have implemented OpenAM in their own environments.  The DOD is pretty extensive when it comes to product reviews and I have worked with several Fortune 500 companies that have had their hands all up in the code – and still choose to use it.  I have worked with companies that elect to have a minimal IAM implementation team (and rely on ForgeRock for total support) to those that have a team of developers building in and around their IAM solution.  I have seen some pretty impressive integrations between OpenAM log files, debug files, and the actual source code using tools such as Splunk.  And while you don’t need to go to the extent that I have seen some companies go in getting into the code, knowing that you could if you wanted to is a nice thing to have in your back pocket.  That is the benefit of open source code and one of the benefits of working with ForgeRock in general.

I can remember working on an implementation for one rather large IAM vendor where we spent more than three months waiting for a patch.  Every status meeting with the customer became more and more uncomfortable as we waited for the vendor to respond.  With ForgeRock software, I have the opportunity to look into the code and put in my own temporary patch if necessary.  I can even submit the patch to ForgeRock and if they agree with the change (once it has gone through the code review), my patch can then be shared with others and become supported by ForgeRock.

It is the best of both worlds, it is commercial open source!