In my last post I mentioned that one of the options I was considering for clustering the Grails Searchable plugin, though not the preferred option, was to use the JDBCDirectory implementation of Lucene. I had this working in just 15 minutes and though it’s probably not the optimal approach, it is by far the simplest.
Just to set the stage, I’m running Grails RC3, Groovy 1.5.1, JDK 1.6, grails-searchable-0.4-SNAPSHOT and MySQL 5.0.45. I will configure the Compass to use the JdbcDirectory connection instead of file or ram and test that it can be updated and read by two instances of my Grails application.
First I created compass.cfg.xml in grails-app/conf (just needs to be somewhere in the classpath) with jdbc connection details:
<compass-core-config xmlns="http://www.opensymphony.com/compass/schema/core-config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.opensymphony.com/compass/schema/core-config
http://www.opensymphony.com/compass/schema/compass-core-config.xsd">
<compass name="default">
<connection>
<jdbc dialect="org.apache.lucene.store.jdbc.dialect.MySQLDialect">
<dataSourceProvider>
<driverManager url="jdbc:mysql://localhost/search?emulateLocators=true"
username="root" password="password"
driverClass="com.mysql.jdbc.Driver" />
</dataSourceProvider>
<fileEntries>
<fileEntry name="__default__">
<indexInput bufferSize="4096" />
<indexOutput bufferSize="4096" />
</fileEntry>
</fileEntries>
</jdbc>
</connection>
</compass>
</compass-core-config>
One thing I missed at first was the dialect setting in the jdbc tag. Make sure you select the correct dialect for your database. I used this as a reference.
In SearchableConfiguration.groovy I set compassConnection to null
String compassConnection = null
I started my grails app and hit just one more snag, I got a “packet too large” error while it was building the index. Turns out the default max_allowed_packet in MySQL on the server is 1M and since Lucene stores parts of the index in BLOBs I needed to up the max. I found this article and set the max_allowed_packet setting to 16M. I did this in my.ini:
[mysqld] max_allowed_packet=16M
That was it! I tested by running two instances of my Grails application simultaneously and I modified one of my Domain class instances and executed a search on the other instance and within a few seconds, the new value was returned in my search.
The last thing I considered was configuring a Local Directory Cache in Compass. This seemed like it could offer some good optimization though it was unclear from the documentation how much control you have over the size of the cache, what gets cached, the TTL, etc. Also, I soon realized that this was introduced in the latest version of Compass (1.2) and isn’t the version that is bundled in the Searchable plugin.
Dude!
you are so freeking awesome!
This has _got_ to go into the plugin as a nice easy configuration option (ie without the Compass XML).
And since Grails already has a DataSource configuration, that’s the one it could default to.
What do you think about (in the SearchableConfiguration.groovy or Grails’ Config.groovy):
// default file based store
connection “file”
// file based store at given location
connection “file”, path: “/path/to/dir”
// use JDBC store with DataSource from DataSource.groovy
connection “jdbc”
// use JDBC store with specific options (mirroring Compass XML)
connection “jdbc”, dialect: “org.apache…” {
dataSourceProvider … //
fileEntries … //
}
Might need more brackets though
Cheers,
Maurice
Thanks Maurice. I agree this would be great to add into the plugin with nice configuration as long as you have the ability to use the XML if you need more granular control. I think that’s one of the strengths of the current plugin and this is a perfect example of that! Great work!
Let me collect my thoughts and I’ll email you my suggestions. I’m willing to help out on supporting the plugin too if you’re interested.
Mike