I walked away tonight with a free license of Intellij IDEA so thanks to Dave Booth from Jetbrains for the free swag! Definitely my favorite IDE, and I was actually in need of a license! And thanks TEKsystems for the food and the other sponsors as well!

I’m scheduled to present on Grails in May so if you’re in the area feel free to come heckle! Wow, four sentences ending in exclamation points (make that five)!

As a quick confidence booster before setting out again, I successfully clustered the Grails Searchable Plugin using a JDBCDirectory store for the index. Since my last attempt I received some guidance from the guys at Terracotta and learned that Shay Banon is working with Terrocotta to develop a Compass module for Terracotta. Good stuff… check out the details of that here.

The first hurdle I needed to overcome was the classloader error:

Classloader name not set, instances defined from this
loader not supported in Terracotta
(loader: org.codehaus.groovy.grails.cli.support.GrailsRootLoader)

Before an object can be shared by Terracotta, the ClassLoader that loaded the object’s class has to be known to Terracotta. For core java and supported frameworks/container Terracotta takes care of registering relevant classloaders. For the GrailsRootLoader I can do this one of two ways:

1. hack the grails source where the class loader is instantiated, registering it with Terracotta
2. this can be converted into a config module that will inject the proper byte code at runtime so as to avoid any code tweaking.

The guys at Terracotta offered to help with #2 if I could get #1 working.

So I checked out the Grail trunk from SVN, and added these lines after the GrailsRootLoader is created:

 // create loader and execute main class
GrailsRootLoader loader = new GrailsRootLoader(lc);
((com.tc.object.loaders.NamedClassLoader) loader).__tc_setClassLoaderName("Grail Classloader");
com.tc.object.bytecode.hook.impl.ClassProcessorHelper.registerGlobalLoader((com.tc.object.loaders.NamedClassLoader) loader);

I built the Grails jars and replaced them in my grails-1.0-final-SNAPSHOT/dist directory. That resolved my classloader error! Next I went through about 100 iterations of root and lock configuration for my tc-config.xml as I dug and spun my way around the Compass source. I finally ended up with this for my tc-config.xml (keep in mind this can surely be optimized and is pretty brute force):

<?xml version="1.0" encoding="UTF-8" ?>

<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
  <system>
    <configuration-model>development</configuration-model>
  </system>

  <servers>
    <server name="localhost" />
  </servers>

  <clients>
    <logs>%d/client-logs-%h</logs>
    <dso>

       <debugging>
        <runtime-logging>
          <lock-debug>true</lock-debug>
        </runtime-logging>
        <runtime-output-options>
          <full-stack>true</full-stack>
        </runtime-output-options>
      </debugging>

    </dso>
    <modules>
      <module name="clustered-lucene-2.0.0" version="2.5.0"/>
    </modules>
  </clients>

  <application>
    <dso>
      <instrumented-classes>
        <include>
          <class-expression>org.compass.core.lucene.engine..*</class-expression>
        </include>
        <include>
          <class-expression>org.apache.lucene..*</class-expression>
        </include>
      </instrumented-classes>
      <roots>
        <root>
          <field-name>org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore.ramIndexes</field-name>
        </root>
      </roots>
      <locks>
	<named-lock>
          <method-expression>* org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.*(..)</method-expression>
          <lock-level>write</lock-level>
          <lock-name>theLockName</lock-name>
        </named-lock>
        </locks>
     </dso>
  </application>
</tc:tc-config>

It worked! I was able to test with the Terracotta server running, and two Grails apps running in separate JVMs that all of the Domain class data that the first JVM indexed, could be searched by the second JVM and they both returned identical results. I also verified the existence of the objects under the root in the Terracotta Admin GUI. And finally, I shut both Grails app JVMs down and brought them back up and my index was still available.

What’s next? Work to get a config module for Terracotta and optimize my root and locking configuration. I’m very interested in what will come of the promised Compass/Terracotta integration. Also Clustering Grails is a much taller mountain. I was told that past attempts to cluster Groovy with Terracotta had not been too successful. For now I’m happy just to have climbed my little hill…

Just to set the stage, I’m running Grails RC3, Groovy 1.5.1, JDK 1.6, grails-searchable-0.4-SNAPSHOT and MySQL 5.0.45. I will configure the Compass to use the JdbcDirectory connection instead of file or ram and test that it can be updated and read by two instances of my Grails application.

First I created compass.cfg.xml in grails-app/conf (just needs to be somewhere in the classpath) with jdbc connection details:

<compass-core-config xmlns="http://www.opensymphony.com/compass/schema/core-config"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.opensymphony.com/compass/schema/core-config
          http://www.opensymphony.com/compass/schema/compass-core-config.xsd">
   <compass name="default">
     <connection>
         <jdbc dialect="org.apache.lucene.store.jdbc.dialect.MySQLDialect">
             <dataSourceProvider>
                 <driverManager url="jdbc:mysql://localhost/search?emulateLocators=true"
                                      username="root" password="password"
                                      driverClass="com.mysql.jdbc.Driver" />
             </dataSourceProvider>
             <fileEntries>
                 <fileEntry name="__default__">
                     <indexInput bufferSize="4096" />
                     <indexOutput bufferSize="4096" />
                 </fileEntry>
             </fileEntries>
         </jdbc>
     </connection>
   </compass>
</compass-core-config>

One thing I missed at first was the dialect setting in the jdbc tag. Make sure you select the correct dialect for your database. I used this as a reference.
In SearchableConfiguration.groovy I set compassConnection to null

String compassConnection = null

I started my grails app and hit just one more snag, I got a “packet too large” error while it was building the index. Turns out the default max_allowed_packet in MySQL on the server is 1M and since Lucene stores parts of the index in BLOBs I needed to up the max. I found this article and set the max_allowed_packet setting to 16M. I did this in my.ini:

[mysqld]
max_allowed_packet=16M

That was it! I tested by running two instances of my Grails application simultaneously and I modified one of my Domain class instances and executed a search on the other instance and within a few seconds, the new value was returned in my search.

The last thing I considered was configuring a Local Directory Cache in Compass. This seemed like it could offer some good optimization though it was unclear from the documentation how much control you have over the size of the cache, what gets cached, the TTL, etc. Also, I soon realized that this was introduced in the latest version of Compass (1.2) and isn’t the version that is bundled in the Searchable plugin.

To start here’s what I’m considering:

1. Manage a local index per node. Probably disable mirrorChanges via Compass::GPS and rebuild the index on someinterval. This is not very desirable since it would eliminate the Compass:GPS capability, and there just must be a more elegant way…
2. Use the JDBCDirectory implementation of Lucene and store the index in a sql database. This option is viable but has obvious performance implications that somewhat defeat the purpose. The configuration would be fairly clean using the native Compass XML config. This maybe a good fall back option to do some A/B testing against.
3. Use Terracotta and RAMDirectory to handle synchronization of the index across nodes. This has a lot of promise and will be the path I head down first. Though I have no experience whatsoever with Terracotta

Another possibility is to look at Compass’ new support for Gigaspaces. Thanks Marcos for passing that on. This sounds interesting but “feels” fairly heavy compared to the others, though I know nothing about GigaSpaces or Coherence so I’m just talking out of my arse at this point!

I made my first attempt with Terracotta and Compass today. I dug deep enough into Compass to find that it manages the RAMDirectory objects in this class: org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore and manages each subindex RAMDirectory in a java.util.HashMap called ramIndexes. The first issue I had to overcome was that HashMap isn’t a synchronized data structure, and I was having problems configuring the locking in Terracotta for ramIdexes. I kept getting UnlockedSharedObjectException exceptions not matter what I tried.

After a little help from the Terracotta users list I ended up with a named lock. My tc-config.xml looked like this:

<?xml version="1.0" encoding="UTF-8" ?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
  <system>
    <configuration-model>development</configuration-model>
  </system>

  <servers>
    <server name="localhost" />
  </servers>

  <clients>
    <logs>%d/client-logs-%h</logs>
    <dso>
      <debugging>
        <runtime-logging>
          <lock-debug>true</lock-debug>
        </runtime-logging>
        <runtime-output-options>
          <full-stack>true</full-stack>
        </runtime-output-options>
      </debugging>
    </dso>
    <modules>
      <module name="clustered-lucene-2.0.0" version="2.5.0"/>
    </modules>
  </clients>

  <application>
    <dso>
      <instrumented-classes>
        <include>
          <class-expression>org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore</class-expression>
        </include>
      </instrumented-classes>
      <roots>
        <root>
          <field-name>org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore.ramIndexes</field-name>
        </root>
      </roots>
      <locks>
	<named-lock>
          <method-expression>* org.compass.core.lucene.engine.store.RAMLuceneSearchEngineStore.*(..)</method-expression>
          <lock-level>write</lock-level>
          <lock-name>theLockName</lock-name>
        </named-lock>
      </locks>
     </dso>
  </application>
</tc:tc-config>

That at least got me past the UnlockSharedObjectException, but tonight I hit a bigger roadblock. Terracotta is choking on Grails custom ClassLoader. I did a preliminary search on the error message below but came up short. I followed up on the Terracotta users’ list but I thought I’d share my progress and my intentions here in case anyone out there has any ideas…

Error creating bean with name 'compassGps': Cannot resolve reference to
bean'compass' while setting bean property 'compass'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'compass': FactoryBean threw exception on object creation;
nested exception is java.lang.IllegalStateException: Classloader name not set,
instances defined from this loader not supported in Terracotta (loader:
org.codehaus.groovy.grails.cli.support.GrailsRootLoader):
java.lang.IllegalStateException: Classloader name not set, instances
defined from this loader not supported in Terracotta (loader:
org.codehaus.groovy.grails.cli.support.GrailsRootLoader)

Stay tuned…

Update (Jan 29, 2008): See part 2 here

Let me first say that I was off to find a free solution. If I had the luxury of spending a little cash at this point I would have used Bamboo. I’m a big fan of Atlassian products and love what they are doing to integrate Jira, Fisheye, Bamboo, Crucible, Clover and Confluence. The traceability of code to issues to change sets to builds and tests is really cool. Anyway I digress.

So I found Hudson, and I was pleasantly surprised how easy it was to set up. I’m running on Windows with Tomcat 6 and JDK1.6. I simply deployed the Hudson WAR, and I was off to the races. Hudson’s configuration is all done via a web interface and the interface itself is very clean, especially compared to Cruise Control.

In Hudson I created a new job, selected SVN in the source code management section and entered the URL to my Grails app in the repository. Hudson also lets you configure a repository browser (ViewSVN, Fisheye, WebSVN or Sventon). Since I was already running Tomcat, I chose Sventon and quickly downloaded and deployed the WAR for Sventon. I set Hudson to poll SVN for changes every minute and for the build itself I told Hudson to use Ant (you need to set up ANT first in Hudsson) and pointed to the Grails build.xml.

I created two new targets in the build.xml: ‘all’ and ‘deploy’. The all target chains the depended targets together. I had originally had Hudson call test, war, deploy and doc separately but each invoked a compile separately.

<target name="all" depends="war, test, deploy, doc"
  description="build, test, deploy and build the javadoc"/>

I also created a deploy target that takes the war and deploys to Tomcat and property to specify the grails_env.

Under Post-build Actions I set the following properties:

• Files to Archive = “**/*.war”
• Javadoc Directory = “GRAILS-APP/docs/gapi”
• Test Reports XMLs = “GRAILS-APP/test/reports/*.xml”

So in about an hour I had Hudson continually building, running my test cases, generating javadoc and deploying my WAR to Tomcat, and it was all relatively painless. Here’s what the job status screen looks like:

Going forward I’ll probably investigate using GANT instead of ANT, otherwise it’s serving the purpose.