Branching out

I recently stumbled across a glowing review of the Wicket framework that made me want to research it and try it out. The author's main argument for Wicket was that it's pretty much just HTML and Wicket, as opposed to the combination of Spring, Spring Webflow, JSP, and tons of other stuff thrown into the mix. Unfortunately I really don't have the time to mess around with it right now, but I'll try to be back on this topic later.

*EDIT* : To add to the list of things I should really learn, Seam's going on the pile. It looks very cool and it's made by Gavin King, the maker of Hibernate. (I know I've bitched about how bad the Hibernate documentation is on numerous occasions, but really, I can't blame him since I hate doing documentation just about as much as if not more than the next guy)

Better idea for multi-versioned webapps

A while ago at our company, a request was made to me to create a separate instance of one of our webapplications for a select group of customers. In doing this, I had to create a separate instance of the backing database and copy over much of a preexisting version of the application's database which was simply configuration data.

It occurred to me today while working on another part of the application that I really should have separated the configuration from the data. The hard part about that is keeping security intermingled between data (for audit trailing) and keeping security intermingled with the configuration data (as part of configuration). I'm still searching for an answer to that one.

Excel and its annoying quirks

Excel has this annoying quirk of propagating unwanted hyperlinks in the background of spreadsheets, and in our organization, it gets to the point where it infests every cell of the sheet. But, I did find a fix (sorta). Press Alt+F11 to go into the VB editor in Excel, go to Insert -> Module, and paste this in:

Sub RemoveHyperlinks()

'Remove all hyperlinks from the active sheet
ActiveSheet.Hyperlinks.Delete

End Sub

Save it, return to the worksheet, press Alt+F8 to open the macros menu (Tools -> Macro -> Macros otherwise) and run the Macro of the same name as the subroutine above. This should delete all your hyperlinks. Unfortunately, this will most likely remove all other special formatting in the cells as well. If you have a really large worksheet, it will most likely cause Excel (XP and earlier at least) to crash, so you should really save your work immediately before you do this (or before you add the Macro to the sheet).

Taking a REST

Lately I've started to get back into research mode in the hopes of making my development cycles shorter, especially since the end of a large sprint is in view. With that in mind, I find myself being continually reminded of the increasing demand for REST-ful applications (ie web applications that make use of the REST HTTP method as opposed to POST or GET). I'm finding it's at least something I shoud be familiar with as a web developer, even if I don't use it.
*EDIT* : Look up UML with Rational Rose

Spring Batch Processing

No, it's not batch processing that takes place in the springtime (although that does happen at our company). It's actually a new subproject of the Spring Framework meant to provide a convenient , common framework for batch processing, and it might be just the thing I need to help me out. Oddly, I found out about it while I was searching for the documentation for the Spring-binding package.

Quirks with Maven2 and m2eclipse

If you're using m2eclipse (which I've found to be quite useful), you should be aware that it doesn't always check the settings.xml file in the local repository or in the maven home dir, and so that pesky command-that-you're-trying-to-get-working-that-should-be-working-after-you've -updated-settings.xml-but-still-isn't-working may have to be run from the command line using 'mvn' rather than from within eclipse.

Logs and RSSH

I've learned the hard why today why logs are uberuseful (once again) and that RSSH is a pain to debug if you're having trouble with it. For future reference, check '/var/log/messages' as the first thing you do when having trouble with SSH, especially RSSH.

For those not in the know, RSSH stands for Restricted Secure SHell, and is very useful if you want to create jails for people who have to transfer files between their boxes and your server. However, it's very hard to debug because when you log in, you don't get any messages, you either get the "This is a an account restricted by RSSH" message or you get the ever so friendly and useful "Connection closed" message. As it turned out, the passwords for several of my users had expired today and this created a huge issue when they weren't able to upload batch files to our server. After spending just over an hour finding that the problem was related to SSH login, I received a very helpful message from one of the people who operates our server showing me a grepped slice of the /var/log/messages log saying that the password had expired for several of the users : coincidentally, the only users that I had tried troubleshooting the problem with.
The point of all this: check '/var/log/messages'.

*EDIT:* Today is officially known as Bad Monday for me.

Fucking Hibernate

God f*ck*ng dammit, I'm so freaking tired of fucking Hibernate's shitty ass documentation wasting my time:

In the event that you ever have to query in HQL on a component and that component contains ManyToOne elements (ie entities, rather than strings or numbers), you have to query on properties of that element rather than the element itself and have hibernate query for the ID.

...which is lame, considering Hibernate is so good about that kind of thing everywhere else.

Maven ... and why you should use it (2, not 1)

For the longest time during my developer's journey of discovery while learning how to build web applications, I've come across references to Maven and how it's a build system like Ant, but supposedly better. I've also heard untold numbers of references to it being frustrating and very difficult to use, largely due to shitty documentation. It was for this reason that I avoided learning it and just stuck with my current build process, which is just using Ant, exporting everything with eclipse, then manually uploading things to the server. Or in the case of dealing with a library I've made that I share among various web apps, compiling everything, going through Eclipse's JAR export wizard and copying files around. No more.

I must have came into Maven at the right time because I've found there to be lots of documentation (or at least a sufficient amount for myself) to learn it and get going with it. I've also converted my library from just using Ant to using Maven to build it, test it, package it and deploy it to a central server within my company that will now be the new Maven repository for the library. From now on, my applications can retrieve it from a central repository for me automatically instead of me having to manually export the library, copy it around, and maintain various versions of it in CVS. And I'm thrilled that this is going to make my life easier.

I did however run into a few quirks when getting Maven going. They're as follows:
1) You should really align your project to have the same layout structure as the one recommended by the Maven documentation. It'll get things going faster and you'll have far fewer build problems.

2) If you use Java (1.)5 or greater, you may need to manually insert configuration settings in order to get your (tests in my case, and possibly other) code to compile properly since Maven strives to achieve compatibility by default and will likely try to compile your code for older VMs such as 1.3. To configure Maven to use 1.5 (or whatever) by default, put the following snippets in your pom.xml file :

<project ... >
<!--
|
|
-->

<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
</plugins>
</build>

<!--
|
|
-->
</project>

3) Deploying your library (or other code) to a Maven repository is wicked easy, especially since Maven doesn't require any special software to set one up: a Maven repository is just a folder containing libraries and metadata with a prescribed folder layout and naming convention. The code for deploying to a server is as simple as this :

<distributionManagement>
<!-- This repository depends on the following settings.xml file being in the local maven2 repository
and a .ssh folder being created under ~ (or %HOMEPATH%) in Windows:
<settings xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/settings-1.0.0.xsd">

<servers>
<server>
<id>[REPOSITORY NAME]</id>
<username>[REPOSITORY USER]</username>
<password>[REPOSITORY PASSWORD]</password>
</server>
</servers>

</settings>
-->
<repository>
<id>[REPOSITORY NAME]</id>
<name>My Maven2 Repository</name>
<url>scp://mybox.mydomain.com/path/to/repository</url>
</repository>
</distributionManagement>

Maven has been pretty simple to set up and use so far, once you've read the documentation and you have at least some idea of what you're doing. I also recommend getting the m2eclipse plugin if you're going to be using Maven with Eclipse. My next steps are going to be getting Maven to compile my XMLBeans for me and deploying those as well.

The head is venturing out of the ass

For the longest time, I've had my head up my ass: I've been content to build my projects with ant and the (very few) ant targets I've really needed: reports, xmlbeans and hibernate. (And fiddling around with jspc to precompile my JSPs). But now I've found that using Xdoclet2 for Hibernate is no longer acceptable since a certain bug has forced me away from it, and now I'm finding that it's time to move on to real build systems like Maven2. I've been scared away from it before by the fact that I've heard that it's exceedingly poorly documented and that it's been a total pain and has load of incompatibilities, but apparently it's been getting better and now I'm forced to learn it in the hopes of saving myself some trouble. I'm also going to have to learn EJB3 Annotations and their Hibernate extensions, but that's another matter. Let's see how it goes.

My list of stuff to learn

Frequently I find that there are dozens of technologies I want to learn, but really just don't have the time. As a list to myself (and anybody else who might care) of the stuff that I want to learn, but just haven't had the time, here's a short enumerated list:

• FreeMarker
• Scheme
• Groovy
• Grails
• Ruby
• Rails
• Python
• JPA (Java Persistence API)
• EJB Annotations (primarily for Hibernate)
• XSLT

And that's just the stuff I can remember off the top of my head as I write this. I'm sure that I'll add more later.

Reading blogs pays off

I've recently been trying to read more and more developer blogs in the hopes of improving myself as a developer. One of the blogs I came across is Brian Burridge's, and he recently had a good post about a little utility called Denim. It's a storyboarding/diagramming tool that was originally meant for web developers to quickly model their sites, but I see no reason why it can't be used for user interfaces in general. The program's made in Java so not unexpectedly) its Swing UI is ugly as shit, but it's so functional it's insane. This is how I'd make an interface (though I'd try to pretty mine up), but they got the basics right: functionality before beauty. If you're a serious developer, I think you should at least give it a shot and play around with it. It's free, open-source, and made at an academic institution. BTW, I jacked the latter link right from his page to save anybody reading this some time, not to take away from his blog, which I think you should check out.

The difference between links and forms ... at least for Spring

This post is really just a mental note to myself: the difference between links and forms is that links do not submit form variables. (Duh!) Therefore, you can't bind varibles in forms to Form Backing Objects in spring if you're going to use a link to do the transition rather than a form button.

(Web)Harvesting the web

I was recently assigned a task whereby I had to obtain data from the website of one of our commercial services, but the website is completely Web 1.0 and has no APIs of any kind that
we could hook into. To give you a bit of context, the site performs transactions on our behalf and we need information about those transactions. As my boss saw it, there were only two solutions really available :
1) Have somebody sit at a computer and download the transactions every 15 minutes
2) Have a computer set up to run a screen macro to log in and download the transactions file every 15 minutes (better, but still no where near ideal)

I think without even realizing it, my boss gave me the idea for option three. He continually mentioned the idea of screen-scraping the page to retrieve the information. Traditionally this means taking a visual representation of something and extracting information from what's essentially a picture. After doing some reading, I interpreted his suggestions to mean that I should find a way to do a web-scrape on the page (subtle difference). After doing some research, I found the perfect library for doing it in Java.

It's a project called WebHarvest, and it's fairly simple yet really powerful. You start off by writing a Scraper configuration in XML, load it in code, run the scraper, and it will store your data for you in the Scraper to retrieve when need it. The library itself works by doing either POSTs or GETs to a page, taking the response data, (almost certainly) doing a transform to convert the HTML to well-formed XHTML, and then running XPath queries and regular expressions on the result to get the data you need (ie rows of a table). It's incredibly powerful and if you need a solution where you want to automate logging into a website and retrieving data, then this is a great way to do it.

UTF-8 ... and why you should use it, part 2

I've sadly discovered that my previous post was lacking some bits of information. For starters,
when using tomcat, you should place this directive ( URIEncoding="UTF-8" ) in your connector definition. Then, you should use this attribute ( accept-charset="utf-8" ) with all your forms; this isn't really necessary, just good for practice. Then, you should ensure that you have a filter defined within your web.xml document to coerce incoming requests into UTF-8 format. The easy way of doing this if you're using Spring is to place the following definitions into your web.xml document :


<filter>
<filter-name>charsetFilter</filter-name>
<filter-class>
org.springframework.web.filter.CharacterEncodingFilter
</filter-class>
<init-param>
<param-name>encoding</param-name>
<param-value>UTF-8</param-value>
</init-param>
</filter>

<filter-mapping>
<filter-name>charsetFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>

This will filter all your incoming requests for you. *Edit* : As one additional side note, if you're moving an existing database to UTF-8 (which is fairly likely since new production databases aren't started that often), then you'll also need to run an ALTER DATABASE statement on the database in MySQL to set its default character set to UTF-8.

Lousy documentation at its finest

This post's going to have to be short because I desperately want to get out of work to go play poker. The short version: I just spent the last three hours in a rage at MySQL because, for a new person following the documentation on their site, it's hard to realize that despite example after example of calling a stored *FUNCTION* with CALL [function name](); syntax, said syntax applies only to stored *PROCEDURES* and you should call a stored function like so : SELECT [function name]();

Fuck, sometimes I hate my job as a developer and I wish I could just go work at a bar, like certain friends. And it's been one of those days.

Intricacies of Hibernate

If you're a serious developer / software engineer, then you know how important it is to keep a system you're building as modular as possible for all the standard reasons: separation of concerns, testing, etc . To this end, I've been developing my systems at work to use Data Access Object interfaces, rather than giving them direct exposure to hibernate. One of the benefits of this is that it abstracts my business and presentation layers from my persistence layer. One of the detractors is that it makes it harder to do certain other things:

1. Maintain efficiency when performing database operations
2. Use Hibernate effectively when loading / storing objects.

Case in point: for the last two days I've been trying to figure out why my detached objects keep having exceptions thrown regarding dirty collection references when I try to update them. As it turns out, the Session.update() function is really best used within the context of a transaction, where you modify your business objects within the transaction, so the persistent objects never become detached (this last portion is a lot more meaningful if you're familiar with Hibernate parlance) However, in a well modularized Web application, your business, presentation and persistence layers are detached so one never knows that your business objects are being persisted with Hibernate, and as such you must 'detach' your business objects from their sessions. Updating them in a new session won't work if you want to use Session.update. Instead, you need to use Session.merge . The subtle difference between these two is that .update checks things like dirty member collections and caching, whereas .merge verbatim overwrites the existing state of your business objects' members in the database. It took me two days to realize this and I'm very pissed off. I really wish it had been mentioned in the Hibernate documentation.

I still don't know enough, but I know something

For quite a while I've wanted an easier way of using Jasper Reports in my web applications. My first go around with Jasper Reports involved implementing custom servlets for Jasper reports and using (what I now realize is) a giant kludge to add to and maintain those reports and compile and run them on the fly. I've clearly learned a lot since that first try. My second go around with Jasper Reports involved precompiling them with an Ant task and using report views in Spring after I moved to the Spring framework. This was a lot better than what I had been doing, but I still made mistakes when I didn't quite know the best way to get parameters into my report.

Now I'm about to start with round three. This round started with an epiphane the other day: why not use a tag library to insert a Jasper Report into a page rather than dealing with entire views ? This would surely be far more compact and would even look far better. After googling for several hours and finding nothing even close to satisfactory in this regard, I decided to make my own tag library. I whipped up the tag library tutorial on Sun's website, and within a couple of hours, I had the very beginnings of a rudimentary tag library going, and I had successfully embedded a Jasper report within a page of one of my web apps. (It was a piece of shit test report embedded within a div, but it was more than enough for proof-of-concept).

This short snippet:
<div style="overflow: auto;">
<jr:report printerName="${printerName}" reportName="test" reportParameters="${sessionScope.mymap}"/>
</div>

Showed just how powerful and useful tag libraries can be. Combining this with Spring's dependency injection made things a lot easier too. I thought to myself: this could be really useful. And I was really surprised that nobody else had done this already. In light of my epiphane, I think I'm going to start up a source forge project for this, probably call it JasperTags. Who knows, it might even be useful. But there's a bunch of other stuff I need to learn first, like Maven 2, JIRA, and numerous other little useful developer tools so I can at least put on the facade of being a competent open source developer.

RAD, Spring, and thoomp!

Ok, so only the first two of the title trio have anything to do with each other. The latter's just the sound an air cannon makes and I thought it went well here. Moving on:


One of the best things about the Spring Framework for Java is that thanks to the fact that it's an Inversion of Control (IoC) container, you can very easily add to projects and update them with new code. The latest thing I did with one of my Spring-based projects was to add notifiers when certain batches are uploaded to the system. The first notifier I did was an email notifier, so nothing special. But immediately after I finished implementing it, my "Wouldn't it be cool if ..." sense kicked in and I considered implementing a notifier for MSN Messenger so that certain people in our organization could receive instant notification for batch uploads. After two minutes of google searching, I came upon the JML (Java MSN Messenger Library) posted on SourceForge.net . After a few minutes of looking at the example code posted with the project (still fairly sparse), I came up with the following to inject an ApplicationListener into my Spring context:

<bean id="msnMessenger" class="net.sf.jml.impl.MsnMessengerFactory" factory-method="createMsnMessenger" init-method="login" destroy-method="logout">
<constructor-arg value="my-email@hotmail.com"/>
<constructor-arg value="mypass"/>

<property name="logIncoming" value="true"/>
<property name="logOutgoing" value="true"/>
</bean>

<bean id="msnMessengerNotifier" class="my.package.ApplicationListenerImpl">
<constructor-arg ref="msnMessenger"/>

<property name="recipients">
<map>
<entry key="email_of_our_batch_processing_guy@ourcompany.com" value="His Name"/>
</map>
</property>
</bean>

...and the relevant implementation code :

this.msnMessenger.sendText(Email.parseStr(emailAddress), messageString);

...and in under 20 minutes I had an instant messaging notifier implemented thanks to the ease of Spring and the JML library. Please go see that site if you need instant messaging in Java with MSN Messenger, it's actually a great little project and they deserve to be supported. It seems however, that the one little caveat of this (and it's not even JML's fault) is that the notifier and the person receiving the notification must have each other on their contact lists.

I've lost all meaning for the term "day off"

I have. Really. I'm sitting around researching screen scrapers (because it'll be horribly useful for work, sadly) and while Googling and reading blogs, I've come across a very useful looking tool called Firebug (http://getfirebug.com/) that lets you inspect and modify a page's DOM in real time. The original thread that prompted this was of course searching for existing methods of web scraping. I found a pretty good looking thread on this guy's blog which I can't wait to try out at work on certain pages that I probably can't mention in a personal blog. Suffice it to say that the content of that blog entry has me convinced, now more than ever, that Firefox is hands down the best browser ever.

<edit> This page is super useful for learning screen scraping with Ruby </edit>