m i k e b r e v o o r t

The last two weeks have been a blur. Yesterday I returned home from a trip back to the east coast with Jeanie and Ephraim. We set out to show Ephraim as much as we could shove into four days. Prior to our trip though I was in Memphis and before that I was ill and remodeling my sister in-laws bathroom.

The trip to Memphis was work related. I really liked the city, but my trip was tarnished by a lopsided standoff with a concrete pillar in a parking garage and my rental dodge caravan. Claim still pending…

On Wednesday there was a power outage at the Memphis International Airport that wreaked havoc on flight schedules. I arrived home with only a few hours to unpack, pack and return to the airport for a red-eye to Philly.

While in Philly we stayed with my brother Joe in Prospect Park, PA… basically South Philly with a pizza or sandwich shops every third store. All the Colorado “New York” pizza shops should be ashamed of themselved. Pizza in Philly was still only ~$8 for a large pie. Most Colorado shops are almost double that. We got 3 large pizzas with toppings for less that $30. The last time I got Pizza in Colorado I paid nearly $40 for 2 “large” pizzas with toppings. Can someone please explain this to me??

On Friday we headed to New York City and rendezvoused with my brother Tom who was kind enough to treat us to lunch in Manhattan a few blocks from Marvel “… the fashion district, home of the tiny over priced portion.”

The next day we headed to Washington DC where we hooked up with my other brother Ken who treated us to dinner at a Mexican tapas-like restaurant where Ken and Ephraim ordered grasshopper tacos. No joke, there was a big pile of grasshopper wrapped in a corn tortilla! Missed photo opportunity for sure. Here are some other pictures…

On Sunday we attended the grand opening of Cambridge Sound Studios of the Newtown Square, PA since Joe did a painting for the studio. Awesome party. Great food and they even had Mummer!

Monday we hit the Jersey Shore before heading to the airport. Ocean City:

The final meal of the trip was a chicken cheese steak. Amoroso role, fried peppers, onions, provolone and hot sauce.

Mmmm Herrs Old Bay Potato Chips.

It was a great trip, but a long two weeks. I haven’t been on my bike since the CCTT two weeks ago and before that was the CCTT the week before. Not good spring training campaigh by any means…

We landed in Philly this morning at 5:15 AM, and I’m hanging out with Joe at his house. We’re here through the weekend planning to see family and show Ephraim NYC, DC and the mighty Atlantic Ocean. I spent a lot of time in airports yesterday, finishing up a trip to Memphis and the power was out at the Memphis airport through mid-afternoon. I flew into DIA at 5PM, drove home, unpacked, repacked, ate at Qdoba, and headed back to the airport.

I haven’t been on my bike in over a week now and missed this week’s CCTT…

Somehow I though the winter race in Leadville was going to be my last race in the snow this year… afraid not. Tonight was Cherry Creek Time Trial #2. Attendance was lite and I welcomed the rock star parking as we enjoyed temperatures in the mid 30’s, a strong wind and persistent snow. I wish I had pictures to share. Maybe some will surface from Erik or Shawn. My time was a few seconds slower than last week and both were short of stellar. I can’t wait until we have some decent weather.

Last Saturday I did my first team time trial with Chris, Shawn and Erik at Haystack TT near boulder. I suffered like a dog and was dragged around the course by Chris and Erik. Shawn meanwhile was busy trying to stick the old dropped chain to cart wheel maneuver.

Approaching the line… ouch

Congrats to Marc V for his win at Fairhill last weekend! And good luck to Chris who just headed down to AZ for the Arizona Trail 300 “race”. If all goes well, Chris will be the first single speed (rigid no less) finisher of the AZT.

I recently had the opportunity to do a fresh installation of Atlassian Jira and was faced again with how to integrate Jira with LDAP (in this case Active Directory). Jira does support simple authentication integration with LDAP via OSUser but requires that each LDAP user exist in Jira first. So if an LDAP user needs access to Jira, I have to go to Jira and create a new corresponding user record with the same user id as LDAP (sAMAccountName in this case). Ideally I would like Jira to synchronize LDAP users based on some query filter.

Luckily Jira has a perfect way to address this “opportunity”: Services. Jira Services are asynchronous scheduled units of work configured to run within Jira. I created a new service using JNDI called LDAPUserSyncService that can periodically query LDAP user objects based on configurable filter and create new Jira users for new users it finds in LDAP. I also added the ability to put a subset of users in a group called “internal-users” for users whose email matched a particular domain.

Its pretty simple. When the service runs, it queries LDAP, iterates through each user checking if the user already exists in LDAP, if not it created the user and puts it in the jira-users group and then checks if the email address contains myco.com (fictitious domain for this example) and if so put the user in the internal group as well. I just realized as I’m writing this though that I didn’t do anything to handle paging or results if the number of users is greater than the max returned by LDAP. Oh well, maybe another time…

public class LDAPUserSyncService extends AbstractService {

    private static org.apache.log4j.Category log =
            org.apache.log4j.Category.getInstance(LDAPUserSyncService.class);

    public static String INITCTX = "com.sun.jndi.ldap.LdapCtxFactory";
    public static String GROUP_INTERNAL =  "Internal-Users";
    public static String LDAP_USER_ID_ATTRIBUTE = "sAMAccountName";

    public void run() {
        try {
            log.debug("Running com.myco.jira.service.LDAPUserSyncService");
            String LDAP_HOST = getProperty("LDAP_HOST");
            String LDAP_USER = getProperty("LDAP_USER");
            String LDAP_PASSWORD = getProperty("LDAP_PASSWORD");
            String LDAP_SEARCHBASE = getProperty("LDAP_SEARCHBASE");
            String LDAP_FILTER = getProperty("LDAP_FILTER");   

            Hashtable env = new Hashtable();
            env.put(Context.INITIAL_CONTEXT_FACTORY, INITCTX);
            env.put(Context.PROVIDER_URL, LDAP_HOST);
            env.put(Context.SECURITY_AUTHENTICATION, "simple");
            env.put(Context.SECURITY_PRINCIPAL, LDAP_USER);
            env.put(Context.SECURITY_CREDENTIALS, LDAP_PASSWORD);

            DirContext ctx = new InitialDirContext(env);

            SearchControls constraints = new SearchControls();
            constraints.setSearchScope(SearchControls.SUBTREE_SCOPE);

            NamingEnumeration results = ctx.search(LDAP_SEARCHBASE, LDAP_FILTER, constraints);

            while(results != null && results.hasMore()) {
                SearchResult sr = (SearchResult) results.next();
                Attributes attrs = sr.getAttributes();
                log.debug("Processing " + attrs.get("dn"));

                // need sn, givenName, mail
                String first = (attrs.get("givenName") != null) ? (String) attrs.get("givenName").get() : null;
                String last = (attrs.get("sn") != null) ? (String) attrs.get("sn").get() : null;
                String mail = (attrs.get("mail") != null) ? (String) attrs.get("mail").get() : null;
                String userId = (attrs.get(LDAP_USER_ID_ATTRIBUTE) != null) ? (String) attrs.get(LDAP_USER_ID_ATTRIBUTE).get() : null;

                if(first != null && last != null && mail != null && userId != null) {

                    // check if user is in Jira
                    if(!userExists(userId.toLowerCase())) {

                        // create user, if a myco company user then add to internal group
                        if(createJiraUser(userId, first, last, mail) != null) {
                            if(mail.contains("myco.com"))
                                addUserGroup(userId, GROUP_INTERNAL);
                        } else {
                            log.error("LDAP User " + userId + " could not be created");
                        }
                    }
                }
            }
        } catch (NamingException nex) {
            log.error("Caught exception trying to Synch LDAP Users: " + nex.toString());
        } catch (ObjectConfigurationException oce) {
            log.error("Caught exception trying to Synch LDAP Users.  Configuration setup failed: " + oce.toString());
        }
    }

    private User createJiraUser(String userId, String fname, String lname, String email) {

        UserManager userMgr = UserManager.getInstance();
        User osUser = null;
        try {
            osUser = userMgr.createUser(userId.toLowerCase());
            osUser.setFullName(fname + " " + lname);
            osUser.setEmail(email);

            Group jiraUserGroup = userMgr.getGroup("jira-users");
            osUser.addToGroup(jiraUserGroup);
            osUser.store();
            log.debug("Successfully added LDAP user "+ userId);
        } catch (ImmutableException e) {
            log.error("Error creating User in Jira: " + userId, e);
        } catch (DuplicateEntityException e) {
            log.error("Error creating User in Jira: " + userId, e);
        } catch (EntityNotFoundException e) {
            log.error("Could not find group jira-users.  Error creating User in Jira: " + userId, e);
        }

        // to be sure the user was created, return the reloaded user from Jira
        return osUser;

    }

    private boolean userExists(String userId) {
        UserManager userMgr = UserManager.getInstance();
        boolean exists = false;
        try {
            if(userMgr.getUser(userId) != null)
                exists = true;
        } catch (EntityNotFoundException e) {
            exists = false;
        }

        return exists;
    }

    private void addUserGroup(String userId, String groupName){
        UserManager userMgr = UserManager.getInstance();

        try {
            User osUser = userMgr.getUser(userId);

            Group group = userMgr.getGroup(groupName);
            osUser.addToGroup(group);
            osUser.store();
            log.debug("Successfully added LDAP user " + userId + " to group: " + groupName);
        } catch (ImmutableException e) {
            log.error("Error adding User to acis-users group: " + userId, e);
        } catch (EntityNotFoundException e) {
            log.error("Could not find group acis-users.  Error adding User to acis-users group:  " + userId, e);
        }
    }

    public ObjectConfiguration getObjectConfiguration() throws ObjectConfigurationException {
        return getObjectConfiguration("LDAPUSERSYNCSERVICE", "com/myco/jira/service/LDAPUserSyncService.xml", null);
    }
}

To configure a service within Jira navigate to Services in the System section of the Administration interface:

Here I’ve added the service you can see the parameters that can be customized for the service:

The parameters are configured in an XML file that is referenced by the getObjectConfiguration() method in the service code above:

<debugservice id="LDAPUserSyncService">
    <description>
    </description>
    <properties>

        <property>
            <key>LDAP_HOST</key>
            <name>LDAP Host (format: ldap://myco.com:389)</name>
            <type>string</type>
        </property>
        <property>
            <key>LDAP_USER</key>
            <name>LDAP User DN</name>
            <type>string</type>
        </property>
                <property>
            <key>LDAP_PASSWORD</key>
            <name>LDAP Password)</name>
            <type>string</type>
        </property>
        <property>
            <key>LDAP_SEARCHBASE</key>
            <name>LDAP Search Base DN</name>
            <type>string</type>
        </property>
        <property>
            <key>LDAP_FILTER</key>
            <name>LDAP Search Filter</name>
            <type>string</type>
        </property>

    </properties>
</debugservice>

And that’s it. New users are automatically created in Jira within 30 minutes of being created in LDAP. Ooo Rah.

I’d love to see stats on what percentage of all blog posts are an apology for not posting often enough. I have to assume the number is lower than I may expect since all the apologizing slackers (myself included), well, don’t post often enough. Anyway, that’s my passive aggressive apology.

I’ve been incredibly busy and have since transitioned back into consulting, taking a full time position with Avalon Consulting. I’m excited about my new role at Avalon, and the impact I’ll have there. I’ve recently had the opportunity to do some more work with Jira and set up a new Subversion environment. I should have some stuff to share there soon as well as pick back up on Grails.

On the cycling front, the first Cherry Creek time trial is Wednesday. I’m excited to see where my fitness is compared to last year at the same time. I’m about 13 lbs lighter if that’s any indication…

Hopefully more soon.

Tonight I headed down to the Springs for the first meeting of the Colorado Springs Open Source Software Meetup Group. I learned about the group from this post on the Atlassian developer blog. Though it was a bit slow tonight, overall it was a good event and I think it will develop into a solid user group. My only concern is that the topic is too broad to successfully satisfy everyone’s interest. Tonight’s speaker was Kirstan Vandersluis who demoed the open source data integration platform XAware. There seems to be a lot of momentum in the commercially backed open source BI/EAI space lately with others including Pentaho and Apatar.

I walked away tonight with a free license of Intellij IDEA so thanks to Dave Booth from Jetbrains for the free swag! Definitely my favorite IDE, and I was actually in need of a license! And thanks TEKsystems for the food and the other sponsors as well!

I’m scheduled to present on Grails in May so if you’re in the area feel free to come heckle! Wow, four sentences ending in exclamation points (make that five)!

Saturday night marked my return to mountain bike racing or rather mountain biking period, as it was my first time back on a mountain bike since the crash. Bill, Dave, Chris, Marni and I headed up to Ski Cooper, just outside of Leadville for the Tennessee Pass Night Jam. It was soooo much fun. Temperatures were in the mid teens for the 7PM start and Chris loaned me an awesome light that I named “White Heat”.

The new team had a good showing with Chris winning overall, Dave finishing 3rd in Single Speed and I landed a 2nd place in Single Speed!

Here are a few pictures from the night, when my camera was warm enough to operate that is.

The Nordic Center

Getting Ready

Marni at the Finish

Bill, Chris and I “cooling” down

The Food

Round Two of my quest to cluster the Grails Searchable Plugin index with Terracotta! In my first post I outlined my intent to use Terracotta to cluster the Grails Searchable Plugin (Lucene + Compass) using the Lucene RAMDirectory approach. Incidentally, my first attempt failed miserably.

As a quick confidence booster before setting out again, I successfully clustered the Grails Searchable Plugin using a JDBCDirectory store for the index. Since my last attempt I received some guidance from the guys at Terracotta and learned that Shay Banon is working with Terrocotta to develop a Compass module for Terracotta. Good stuff… check out the details of that here.

The first hurdle I needed to overcome was the classloader error:

Classloader name not set, instances defined from this
loader not supported in Terracotta
(loader: org.codehaus.groovy.grails.cli.support.GrailsRootLoader)

Before an object can be shared by Terracotta, the ClassLoader that loaded the object’s class has to be known to Terracotta. For core java and supported frameworks/container Terracotta takes care of registering relevant classloaders. For the GrailsRootLoader I can do this one of two ways:

1. hack the grails source where the class loader is instantiated, registering it with Terracotta
2. this can be converted into a config module that will inject the proper byte code at runtime so as to avoid any code tweaking.

The guys at Terracotta offered to help with #2 if I could get #1 working.

So I checked out the Grail trunk from SVN, and added these lines after the GrailsRootLoader is created:

 // create loader and execute main class
GrailsRootLoader loader = new GrailsRootLoader(lc);
((com.tc.object.loaders.NamedClassLoader) loader).__tc_setClassLoaderName("Grail Classloader");
com.tc.object.bytecode.hook.impl.ClassProcessorHelper.registerGlobalLoader((com.tc.object.loaders.NamedClassLoader) loader);

I built the Grails jars and replaced them in my grails-1.0-final-SNAPSHOT/dist directory. That resolved my classloader error! Next I went through about 100 iterations of root and lock configuration for my tc-config.xml as I dug and spun my way around the Compass source. I finally ended up with this for my tc-config.xml (keep in mind this can surely be optimized and is pretty brute force):

<?xml version="1.0" encoding="UTF-8" ?>

<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
  <system>
    <configuration-model>development</configuration-model>
  </system>

  <servers>
    <server name="localhost" />
  </servers>

  <clients>
    <logs>%d/client-logs-%h</logs>
    <dso>

       <debugging>
        <runtime-logging>
          <lock-debug>true</lock-debug>
        </runtime-logging>
        <runtime-output-options>
          <full-stack>true</full-stack>
        </runtime-output-options>
      </debugging>

    </dso>
    <modules>
      <module name="clustered-lucene-2.0.0" version="2.5.0"/>
    </modules>
  </clients>

  <application>
    <dso>
      <instrumented-classes>
        <include>
          <class-expression>org.compass.core.lucene.engine..*</class-expression>
        </include>
        <include>
          <class-expression>org.apache.lucene..*</class-expression>
        </include>
      </instrumented-classes>
      <roots>
        <root>
          <field-name>org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore.ramIndexes</field-name>
        </root>
      </roots>
      <locks>
	<named-lock>
          <method-expression>* org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.*(..)</method-expression>
          <lock-level>write</lock-level>
          <lock-name>theLockName</lock-name>
        </named-lock>
        </locks>
     </dso>
  </application>
</tc:tc-config>

It worked! I was able to test with the Terracotta server running, and two Grails apps running in separate JVMs that all of the Domain class data that the first JVM indexed, could be searched by the second JVM and they both returned identical results. I also verified the existence of the objects under the root in the Terracotta Admin GUI. And finally, I shut both Grails app JVMs down and brought them back up and my index was still available.

What’s next? Work to get a config module for Terracotta and optimize my root and locking configuration. I’m very interested in what will come of the promised Compass/Terracotta integration. Also Clustering Grails is a much taller mountain. I was told that past attempts to cluster Groovy with Terracotta had not been too successful. For now I’m happy just to have climbed my little hill…

In my last post I mentioned that one of the options I was considering for clustering the Grails Searchable plugin, though not the preferred option, was to use the JDBCDirectory implementation of Lucene. I had this working in just 15 minutes and though it’s probably not the optimal approach, it is by far the simplest.

Just to set the stage, I’m running Grails RC3, Groovy 1.5.1, JDK 1.6, grails-searchable-0.4-SNAPSHOT and MySQL 5.0.45. I will configure the Compass to use the JdbcDirectory connection instead of file or ram and test that it can be updated and read by two instances of my Grails application.

First I created compass.cfg.xml in grails-app/conf (just needs to be somewhere in the classpath) with jdbc connection details:

<compass-core-config xmlns="http://www.opensymphony.com/compass/schema/core-config"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.opensymphony.com/compass/schema/core-config
          http://www.opensymphony.com/compass/schema/compass-core-config.xsd">
   <compass name="default">
     <connection>
         <jdbc dialect="org.apache.lucene.store.jdbc.dialect.MySQLDialect">
             <dataSourceProvider>
                 <driverManager url="jdbc:mysql://localhost/search?emulateLocators=true"
                                      username="root" password="password"
                                      driverClass="com.mysql.jdbc.Driver" />
             </dataSourceProvider>
             <fileEntries>
                 <fileEntry name="__default__">
                     <indexInput bufferSize="4096" />
                     <indexOutput bufferSize="4096" />
                 </fileEntry>
             </fileEntries>
         </jdbc>
     </connection>
   </compass>
</compass-core-config>

One thing I missed at first was the dialect setting in the jdbc tag. Make sure you select the correct dialect for your database. I used this as a reference.
In SearchableConfiguration.groovy I set compassConnection to null

String compassConnection = null

I started my grails app and hit just one more snag, I got a “packet too large” error while it was building the index. Turns out the default max_allowed_packet in MySQL on the server is 1M and since Lucene stores parts of the index in BLOBs I needed to up the max. I found this article and set the max_allowed_packet setting to 16M. I did this in my.ini:

[mysqld]
max_allowed_packet=16M

That was it! I tested by running two instances of my Grails application simultaneously and I modified one of my Domain class instances and executed a search on the other instance and within a few seconds, the new value was returned in my search.

The last thing I considered was configuring a Local Directory Cache in Compass. This seemed like it could offer some good optimization though it was unclear from the documentation how much control you have over the size of the cache, what gets cached, the TTL, etc. Also, I soon realized that this was introduced in the latest version of Compass (1.2) and isn’t the version that is bundled in the Searchable plugin.

I’m on a quest to figure out how to cluster the Grails Searchable plugin that’s based on Lucene/Compass across multiple nodes with as little intrusion and in a way that’s as turnkey as possible. I’m in immediate need of a solution, and I think it would be a good contribution to the Searchable plugin.

To start here’s what I’m considering:

1. Manage a local index per node. Probably disable mirrorChanges via Compass::GPS and rebuild the index on someinterval. This is not very desirable since it would eliminate the Compass:GPS capability, and there just must be a more elegant way…
2. Use the JDBCDirectory implementation of Lucene and store the index in a sql database. This option is viable but has obvious performance implications that somewhat defeat the purpose. The configuration would be fairly clean using the native Compass XML config. This maybe a good fall back option to do some A/B testing against.
3. Use Terracotta and RAMDirectory to handle synchronization of the index across nodes. This has a lot of promise and will be the path I head down first. Though I have no experience whatsoever with Terracotta

Another possibility is to look at Compass’ new support for Gigaspaces. Thanks Marcos for passing that on. This sounds interesting but “feels” fairly heavy compared to the others, though I know nothing about GigaSpaces or Coherence so I’m just talking out of my arse at this point!

I made my first attempt with Terracotta and Compass today. I dug deep enough into Compass to find that it manages the RAMDirectory objects in this class: org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore and manages each subindex RAMDirectory in a java.util.HashMap called ramIndexes. The first issue I had to overcome was that HashMap isn’t a synchronized data structure, and I was having problems configuring the locking in Terracotta for ramIdexes. I kept getting UnlockedSharedObjectException exceptions not matter what I tried.

After a little help from the Terracotta users list I ended up with a named lock. My tc-config.xml looked like this:

<?xml version="1.0" encoding="UTF-8" ?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
  <system>
    <configuration-model>development</configuration-model>
  </system>

  <servers>
    <server name="localhost" />
  </servers>

  <clients>
    <logs>%d/client-logs-%h</logs>
    <dso>
      <debugging>
        <runtime-logging>
          <lock-debug>true</lock-debug>
        </runtime-logging>
        <runtime-output-options>
          <full-stack>true</full-stack>
        </runtime-output-options>
      </debugging>
    </dso>
    <modules>
      <module name="clustered-lucene-2.0.0" version="2.5.0"/>
    </modules>
  </clients>

  <application>
    <dso>
      <instrumented-classes>
        <include>
          <class-expression>org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore</class-expression>
        </include>
      </instrumented-classes>
      <roots>
        <root>
          <field-name>org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore.ramIndexes</field-name>
        </root>
      </roots>
      <locks>
	<named-lock>
          <method-expression>* org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore.*(..)</method-expression>
          <lock-level>write</lock-level>
          <lock-name>theLockName</lock-name>
        </named-lock>
      </locks>
     </dso>
  </application>
</tc:tc-config>

That at least got me past the UnlockSharedObjectException, but tonight I hit a bigger roadblock. Terracotta is choking on Grails custom ClassLoader. I did a preliminary search on the error message below but came up short. I followed up on the Terracotta users’ list but I thought I’d share my progress and my intentions here in case anyone out there has any ideas…

Error creating bean with name 'compassGps': Cannot resolve reference to
bean'compass' while setting bean property 'compass'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'compass': FactoryBean threw exception on object creation;
nested exception is java.lang.IllegalStateException: Classloader name not set,
instances defined from this loader not supported in Terracotta (loader:
org.codehaus.groovy.grails.cli.support.GrailsRootLoader):
java.lang.IllegalStateException: Classloader name not set, instances
defined from this loader not supported in Terracotta (loader:
org.codehaus.groovy.grails.cli.support.GrailsRootLoader)

Stay tuned…

Update (Jan 29, 2008): See part 2 here