Configuring MySQL for table case-insensitivity

While trying to add a recent update to our systems, I noticed that our demo system wouldn't start, which was very strange considering I had no problems on my local test system. I eventually traced the problem down to MySQL not being able to find certain tables. This was because I had created the tables in a different case than the Quartz library was expecting. My local testing box is a Windows box whereas our demo and production systems run Linux, the difference being that my Windows MySQL installation has case-insensitivity on by default, whereas the Linux installations do not. In order to remedy this, I had to explicitly add a line to my my.ini / my.cnf configurations :

lower_case_table_names=1

You can determine the case-sensitivity of your own MySQL installation by logging in with the command line client or running the query browser and entering the following command :

SHOW VARIABLES LIKE 'lower_case%'

Integrating Quartz and Spring using persistent jobs

I've recently found myself having a lot of jobs running in the background of our Java applications, often requiring changes to the scheduling based on my boss' whims. Combine this with the fact that our system's API is now seeing heavy use, and having the triggers configured in the Spring application context (and thus requiring a restart in order to change them) just doesn't cut it any more. Thus, I now find myself needing persistent jobs. However, there's not a lot of documentation out there for dealing with Quartz' persistent job feature in Spring, so I'm going to hopefully provide some help. To start, you need to create the quartz tables in whatever database you're going to be using. There are scripts for creating these tables in the Quartz distribution available on their website for whatever database you may be using. For convenience, I've copied the MySQL Inno DB script here :

DROP TABLE IF EXISTS QRTZ_JOB_LISTENERS;
DROP TABLE IF EXISTS QRTZ_TRIGGER_LISTENERS;
DROP TABLE IF EXISTS QRTZ_FIRED_TRIGGERS;
DROP TABLE IF EXISTS QRTZ_PAUSED_TRIGGER_GRPS;
DROP TABLE IF EXISTS QRTZ_SCHEDULER_STATE;
DROP TABLE IF EXISTS QRTZ_LOCKS;
DROP TABLE IF EXISTS QRTZ_SIMPLE_TRIGGERS;
DROP TABLE IF EXISTS QRTZ_CRON_TRIGGERS;
DROP TABLE IF EXISTS QRTZ_BLOB_TRIGGERS;
DROP TABLE IF EXISTS QRTZ_TRIGGERS;
DROP TABLE IF EXISTS QRTZ_JOB_DETAILS;
DROP TABLE IF EXISTS QRTZ_CALENDARS;
CREATE TABLE QRTZ_JOB_DETAILS(
JOB_NAME VARCHAR(200) NOT NULL,
JOB_GROUP VARCHAR(200) NOT NULL,
DESCRIPTION VARCHAR(250) NULL,
JOB_CLASS_NAME VARCHAR(250) NOT NULL,
IS_DURABLE VARCHAR(1) NOT NULL,
IS_VOLATILE VARCHAR(1) NOT NULL,
IS_STATEFUL VARCHAR(1) NOT NULL,
REQUESTS_RECOVERY VARCHAR(1) NOT NULL,
JOB_DATA BLOB NULL,
PRIMARY KEY (JOB_NAME,JOB_GROUP))
TYPE=InnoDB;

CREATE TABLE QRTZ_JOB_LISTENERS (
JOB_NAME VARCHAR(200) NOT NULL,
JOB_GROUP VARCHAR(200) NOT NULL,
JOB_LISTENER VARCHAR(200) NOT NULL,
PRIMARY KEY (JOB_NAME,JOB_GROUP,JOB_LISTENER),
INDEX (JOB_NAME, JOB_GROUP),
FOREIGN KEY (JOB_NAME,JOB_GROUP)
REFERENCES QRTZ_JOB_DETAILS(JOB_NAME,JOB_GROUP))
TYPE=InnoDB;

CREATE TABLE QRTZ_TRIGGERS (
TRIGGER_NAME VARCHAR(200) NOT NULL,
TRIGGER_GROUP VARCHAR(200) NOT NULL,
JOB_NAME VARCHAR(200) NOT NULL,
JOB_GROUP VARCHAR(200) NOT NULL,
IS_VOLATILE VARCHAR(1) NOT NULL,
DESCRIPTION VARCHAR(250) NULL,
NEXT_FIRE_TIME BIGINT(13) NULL,
PREV_FIRE_TIME BIGINT(13) NULL,
PRIORITY INTEGER NULL,
TRIGGER_STATE VARCHAR(16) NOT NULL,
TRIGGER_TYPE VARCHAR(8) NOT NULL,
START_TIME BIGINT(13) NOT NULL,
END_TIME BIGINT(13) NULL,
CALENDAR_NAME VARCHAR(200) NULL,
MISFIRE_INSTR SMALLINT(2) NULL,
JOB_DATA BLOB NULL,
PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP),
INDEX (JOB_NAME, JOB_GROUP),
FOREIGN KEY (JOB_NAME,JOB_GROUP)
REFERENCES QRTZ_JOB_DETAILS(JOB_NAME,JOB_GROUP))
TYPE=InnoDB;

CREATE TABLE QRTZ_SIMPLE_TRIGGERS (
TRIGGER_NAME VARCHAR(200) NOT NULL,
TRIGGER_GROUP VARCHAR(200) NOT NULL,
REPEAT_COUNT BIGINT(7) NOT NULL,
REPEAT_INTERVAL BIGINT(12) NOT NULL,
TIMES_TRIGGERED BIGINT(7) NOT NULL,
PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP),
INDEX (TRIGGER_NAME, TRIGGER_GROUP),
FOREIGN KEY (TRIGGER_NAME,TRIGGER_GROUP)
REFERENCES QRTZ_TRIGGERS(TRIGGER_NAME,TRIGGER_GROUP))
TYPE=InnoDB;

CREATE TABLE QRTZ_CRON_TRIGGERS (
TRIGGER_NAME VARCHAR(200) NOT NULL,
TRIGGER_GROUP VARCHAR(200) NOT NULL,
CRON_EXPRESSION VARCHAR(120) NOT NULL,
TIME_ZONE_ID VARCHAR(80),
PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP),
INDEX (TRIGGER_NAME, TRIGGER_GROUP),
FOREIGN KEY (TRIGGER_NAME,TRIGGER_GROUP)
REFERENCES QRTZ_TRIGGERS(TRIGGER_NAME,TRIGGER_GROUP))
TYPE=InnoDB;

CREATE TABLE QRTZ_BLOB_TRIGGERS (
TRIGGER_NAME VARCHAR(200) NOT NULL,
TRIGGER_GROUP VARCHAR(200) NOT NULL,
BLOB_DATA BLOB NULL,
PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP),
INDEX (TRIGGER_NAME, TRIGGER_GROUP),
FOREIGN KEY (TRIGGER_NAME,TRIGGER_GROUP)
REFERENCES QRTZ_TRIGGERS(TRIGGER_NAME,TRIGGER_GROUP))
TYPE=InnoDB;

CREATE TABLE QRTZ_TRIGGER_LISTENERS (
TRIGGER_NAME VARCHAR(200) NOT NULL,
TRIGGER_GROUP VARCHAR(200) NOT NULL,
TRIGGER_LISTENER VARCHAR(200) NOT NULL,
PRIMARY KEY (TRIGGER_NAME,TRIGGER_GROUP,TRIGGER_LISTENER),
INDEX (TRIGGER_NAME, TRIGGER_GROUP),
FOREIGN KEY (TRIGGER_NAME,TRIGGER_GROUP)
REFERENCES QRTZ_TRIGGERS(TRIGGER_NAME,TRIGGER_GROUP))
TYPE=InnoDB;

CREATE TABLE QRTZ_CALENDARS (
CALENDAR_NAME VARCHAR(200) NOT NULL,
CALENDAR BLOB NOT NULL,
PRIMARY KEY (CALENDAR_NAME))
TYPE=InnoDB;

CREATE TABLE QRTZ_PAUSED_TRIGGER_GRPS (
TRIGGER_GROUP VARCHAR(200) NOT NULL,
PRIMARY KEY (TRIGGER_GROUP))
TYPE=InnoDB;

CREATE TABLE QRTZ_FIRED_TRIGGERS (
ENTRY_ID VARCHAR(95) NOT NULL,
TRIGGER_NAME VARCHAR(200) NOT NULL,
TRIGGER_GROUP VARCHAR(200) NOT NULL,
IS_VOLATILE VARCHAR(1) NOT NULL,
INSTANCE_NAME VARCHAR(200) NOT NULL,
FIRED_TIME BIGINT(13) NOT NULL,
PRIORITY INTEGER NOT NULL,
STATE VARCHAR(16) NOT NULL,
JOB_NAME VARCHAR(200) NULL,
JOB_GROUP VARCHAR(200) NULL,
IS_STATEFUL VARCHAR(1) NULL,
REQUESTS_RECOVERY VARCHAR(1) NULL,
PRIMARY KEY (ENTRY_ID))
TYPE=InnoDB;

CREATE TABLE QRTZ_SCHEDULER_STATE (
INSTANCE_NAME VARCHAR(200) NOT NULL,
LAST_CHECKIN_TIME BIGINT(13) NOT NULL,
CHECKIN_INTERVAL BIGINT(13) NOT NULL,
PRIMARY KEY (INSTANCE_NAME))
TYPE=InnoDB;

CREATE TABLE QRTZ_LOCKS (
LOCK_NAME VARCHAR(40) NOT NULL,
PRIMARY KEY (LOCK_NAME))
TYPE=InnoDB;

INSERT INTO QRTZ_LOCKS values('TRIGGER_ACCESS');
INSERT INTO QRTZ_LOCKS values('JOB_ACCESS');
INSERT INTO QRTZ_LOCKS values('CALENDAR_ACCESS');
INSERT INTO QRTZ_LOCKS values('STATE_ACCESS');
INSERT INTO QRTZ_LOCKS values('MISFIRE_ACCESS');
commit; 

Once this is done, you'll need to integrate Quartz with Spring. If you're using Maven 2, getting Quartz into your project is as simple as adding the following snippet to the <dependencies> section of your pom.xml configuration file.

<dependency>
  <groupId>opensymphony</groupId>
  <artifactId>quartz</artifactId>
  <version>1.6.0</version>
  <scope>provided</scope>
</dependency>

Note that you're going to need Quartz version 1.6.0 to work with Spring 2.5.3 or greater, otherwise you can use Quartz 1.5.x. After you've got Quartz into your project, you'll need to define a Scheduler that accesses your database to schedule your jobs. Fortunately, Spring happens to come with a handy FactoryBean for defining Quartz schedulers. I've used the following :

<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
  <property name="jobFactory">
    <bean class="org.springframework.scheduling.quartz.SpringBeanJobFactory"/>
  </property>
  <property name="dataSource" ref="mainDataSource" />
  <property name="transactionManager" ref="mainDataSourceTransactionManager" />
  <property name="quartzProperties">
    <util:properties location="/WEB-INF/config/quartz.properties"/>
  </property>
  <property name="applicationContextSchedulerContextKey" value="applicationContext"/>
  <property name="waitForJobsToCompleteOnShutdown" value="true" />
</bean>

The above Spring beans configuration snippet requires some explanation, so let's go through it.

• The property 'jobFactory' is a Spring implementation of the Quartz JobFactory interface. When the scheduler encounters a trigger in the database that's to be fired, it loads the corresponding JobDetails from the database, loads it into a JobDetail object, places that JobDetail object into a TriggerFiredBundle, and passes the TriggerFiredBundle to the JobFactory interface to obtain a Job that's to be run. Now, you might think that because of the name of Spring's JobFactory implementation, it's going to do something with a bean defined in the Spring ApplicationContext, right ? Wrong. What this implementation does is get the Job class provided by JobDetail.getClass(), instantiate a new instance of it with a default no-arg constructor, possibly populate it with properties, and then run the Job. Which sucks, and is kind of useless, because no Jobs I need can do so without getting something from Spring's ApplicationContext. More on this later.


• The 'dataSource' property should be the javax.sql.DataSource used by your application to connect to whatever database you're using that stores the persistent Jobs in the tables used by Quartz.


• The 'transactionManager' property should be set to a org.springframework.transaction.PlatformTransactionManager implementation that's used to demarcate transactions in the database. If you don't have one for your dataSource already defined, then it's fortunate that Spring comes with one that you can define and use, like so :
  

  
  <bean id="mainDataSourceTransactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager" p:dataSource-ref="mainDataSource" />
  

  
  


• The 'quartzProperties' property should be configured with a java.util.Properties instance containing configuration values for Quartz. A sample is posted here for you :
  

  
  org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool
  org.quartz.threadPool.threadCount=5
  org.quartz.threadPool.threadPriority=4
  
  org.quartz.jobStore.tablePrefix=qrtz_
  org.quartz.jobStore.isClustered=false
  org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
  

  
  This sample contains a few simple properties that should be enough to get you started. There are numerous others that you should look into in the Quartz documentation. The first three properties are for configuring Quartz' thread-pooling capabilities. The latter three are for configuring the database (and access thereto-) that Quartz uses for storing persistent jobs. The table prefix is used to prefix Quartz' tables so that they don't conflict with any pre-existing tables in your database. The 'isClustered' property is used to determine whether Quartz is acting in a cluster. Clustered Quartz configurations are beyond the scope of this article. The 'driverDelegateClass' property is used to determine the class Quartz will use for dealing with your specific dialect of SQL (ie MySQL, MS-SQL, PostgreSQL, etc).


• The 'applicationContextSchedulerContextKey' property sets a key that's used to access the Spring ApplicationContext in the JobExecutionContext given to a Job implementation at run time after the SpringBeanJobFactory has populated it for you. This is important as this property does not have a default value and will not put the ApplicationContext into the JobExecutionContext for you unless you specify one.


• The 'waitForJobsToCompleteOnShutdown' property specifies, well, I think you can figure that one out.

At this point in time, you've got the tables which Quartz requires for persistent jobs defined (and presumably accessible) in your database, you've got the Quartz libraries present in your classpath, and you've got Quartz configured within Spring so that Quartz will run and check the database for jobs to be run. However, we're still missing a few more important pieces of information.

1. JobDetails stored in the database describing jobs to be run


2. Triggers defined in the database for describing when JobDetails will be executed

For the rest of this article, lets assume that you've set your Quartz table prefix as 'qrtz_'. In your database, lets look at the definition of the 'qrtz_job_details' table. It includes columns for defining job_name, job_group, description, job_class_name, is_durable, is_volatile, is_stateful, requests_recovery, and job_data.

The 'job_name' field is, as you've almost certainly guessed, the name of the job that you want to define. The job must also have a 'job_group' specified because triggers can be used to signal the execution of entire groups of jobs at once, not just single jobs. The two of these fields together form a unique key for the table. If you really don't give a shit about grouping jobs, you can just use Quartz' default group name, which is 'DEFAULT'. You also need to set a 'job_class_name'. This value is a fully qualified Java class name that must have a default, no-arg constructor (ie it's a bean, according to the Java Beans specification). This class will get instantiated at run time by the SpringBeanJobFactory and run as a Quartz job. Note that this class, obviously, must implement the org.quartz.Job interface. A decription of the job can be put in the 'description' field if you want to have a log-friendly message available to you. The is_durable, is_volatile, is_stateful, and requests_recovery fields are all (essentially) boolean fields that Quartz defines as being single characters. These fields can have the values 'Y' or 'N'. See the Quartz API for org.quartz.JobDetail for further elaboration on the meaning of these fields. The 'job_data' field is a BLOB object that's used to serialize the 'jobDataMap' (java.util.Map) associated with the JobDetail. In order to populate this field, you'll obviously have to write some JDBC code.

The 'qrtz_job_details' table is, obviously, where you store the information for the org.quartz.Jobs you want to run. Anytime you need to define a new job, you'll need to insert a new row into this table, properly configured of course and paying particular attention to the 'job_class_name' field.

Ok, so assuming you've got a couple rows in this table (ie defined a couple of jobs), now you'll need to schedule them so that, at some point, they'll actually run. This is where triggers come in. Now, in Quartz, there are a few different methods of triggering off Jobs, by far the most common being org.quartz.SimpleTriggers (qrtz_simple_triggers table) and org.quartz.CronTriggers (qrtz_cron_triggers table). SimpleTriggers have their use, but our company uses some pretty complex scheduling at times, so I'm going to do an example of using the cron triggers.

The 'qrtz_cron_triggers' table is pretty simple, it only has four fields : 'trigger_name', 'trigger_group', 'cron_expression' and 'time_zone_id', all of which should be pretty indicative of their content. The 'time_zone_id' field should be a valid id for a java.util.TimeZone, ie 'America/Denver'. See the documentation for that class for more on valid IDs.

Once you've set up a trigger with a valid name, group (ie DEFAULT), cron expression and TimeZone ID, you're pretty much ready to go. When Quartz starts up with your application, it'll consult the 'qrtz_triggers' table and ensure that entries exist for all of your corresponding cron / simple / blob triggers. It'll then proceed to fire off triggers and the Scheduler will pass off jobs to the SpringBeanJobFactory for execution. Now, if you haven't figured it out already, if you've got a lot of jobs, so far from what this article has shown you, you'll have to create a Quartz Job implementation for each job you want to run off, and even then, it'll have to be pretty simple, since you'll have to be able to instantiate it with a no-arg constructor, and so far you've got no way of injecting it with anything from Spring, aside from simple properties that are defined by the 'schedulerContextAsMap' property on the Spring SchedulerFactoryBean. So, what I did was create a class that uses its JobDetail#name as a convention for getting a bean from the application context, assumes it's an org.quartz.Job implementation, and executes it. The code is attached :

public class SpringBeanDelegatingJob implements Job {
 
 private static final Log LOGGER = LogFactory.getLog(SpringBeanDelegatingJob.class);
 
 public static final String APPLICATION_CONTEXT_KEY = "applicationContext";

 @SuppressWarnings("unchecked")
 public void execute(JobExecutionContext arg0) throws JobExecutionException {

  JobDetail jobDetail = arg0.getJobDetail();
  
  String beanName = substringBefore(jobDetail.getName(), "Detail");
  
  if (LOGGER.isInfoEnabled()) {
   LOGGER.info ("Running SpringBeanDelegatingJob - Job Name ["+jobDetail.getName()+"], Group Name ["+jobDetail.getGroup()+"]");
   LOGGER.info ("Delegating to bean ["+beanName+"]");
  }
  
  ApplicationContext applicationContext = null;
  
  try {
   applicationContext = (ApplicationContext) arg0.getScheduler().getContext().get(APPLICATION_CONTEXT_KEY);
  } catch (SchedulerException e2) {
   throw new JobExecutionException("Holy fuck, there was some kind of god-damned problem with the fucking Scheduler", e2);
  }
  
  Job bean = null;
  
  try {
   bean = (Job) applicationContext.getBean (beanName, Job.class);
  } catch (BeansException e1) {
   throw new JobExecutionException("Unable to retrieve target bean that is to be used as a job source", e1);
  }
  
  bean.execute (arg0);
  
  return;
 }

}

Now, this makes the persisted jobs a lot more capable : we can now run org.quartz.Job beans that are declared in our Spring ApplicationContext. What if we want to be able to run arbitrary methods on arbitrary beans as jobs ? Well, we can write an adapter class for that too, quite similar (though not as functional as) Spring's MethodInvokingJobDetailFactoryBean :

public class SpringBeanMethodInvokingJob implements InitializingBean, Job {
 
 private Object targetBean;
 private String targetMethod;

 //Constructors

 public SpringBeanMethodInvokingJob() {
  super();
 }
 
 //Behaviour Methods

 public void execute(JobExecutionContext arg0) throws JobExecutionException {
  
  Method method = null;
  
  try {
   method = targetBean.getClass().getMethod(targetMethod);
  } catch (Exception e) {
   throw new JobExecutionException("Unable to get targetMethod ["+targetMethod+
    "] on bean with class ["+targetBean.getClass().getName()+"]");
  }
  
  try {
   method.invoke(targetBean);
  } catch (Exception e) {
   throw new JobExecutionException("Unable to invoke method ["+method.getName()+"] on bean ["+targetBean.toString()+"]");
  }
  
  return; //done
 }

 public void afterPropertiesSet() throws Exception {
  Assert.notNull(targetBean, "'targetBean' cannot be null");
  Assert.isTrue(isNotBlank(targetMethod), "'targetMethod' cannot be blank");
 }

 //Property Accessors
 
 @Required
 public final void setTargetBean(Object targetBean) {
  this.targetBean = targetBean;
 }

 @Required
 public final void setTargetMethod(String targetMethod) {
  this.targetMethod = targetMethod;
 }

}

Note that the class above is implements org.quartz.Job, not org.quartz.JobDetail as is the product of MethodInvokingJobDetailFactoryBean. I hope this article has been useful for anybody reading it, feel free to comment on it.

The greatest MySQL companion ever

I just recently found out about the 'show processlist' command in MySQL that lets you get a resultset of all the processes MySQL is currently using and what query they're running. Today I was working in top monitoring the performance of our servers as I restarted our production server. MySQL popped up briefly on the top list of running processes and it occurred to me, "Wouldn't it be great of there was something like this for MySQL?". So while I was waiting, I popped open google, and ran a query for a MySQL and top. To my surprise, up came mytop. It's literally top for MySQL and it's so ridiculously useful you wouldn't believe it. Even better, it's got a .deb made for it and it's in the Ubuntu repositories, so installing it is as simple as running :

apt-get install mytop

It's a good day today :)

Setting a user's password in MySQL

I've recently found the need to change a bunch of passwords in MySQL. The command to do it is as follows :

SET PASSWORD FOR 'myuser'@'%.wherever.com' = PASSWORD('newpass');

You can also read the documentation on the MySQL website.

Finding out what MySQL is doing

I was recently updating our production system when I noticed that our main MySQL database was going unusually hard. On a quad-core processor system, it was using up three processors simaltaneously. This prompted me to find out what MySQL is doing, and after being unable to read the logs, I did a Google search and ran across this useful little command :

show processlist

This runs a query (can be run from either the command line or the query browser) and shows you a summary of all the processes that MySQL is currently using.