Composing XML in C# 2.0 or later

When testing some custom XML serialization I was writing for a project, I ran across a quirk of the XmlWriter created by XmlWriter#Create() : The writer that gets produced by this method doesn't actually write out XML to whatever stream or file you've given it in the Create method until Close() is called on the generated writer. This means that if you've been using the XmlWriter along with a using() statement, you're fine, but if you haven't been using it, as I have in the unit tests I've been trying to write, you're going to get some unexpected results. Something to pay attention to.

Visual Studio (2010?) Debugging Gotchas - Part I

I recently ran across a problem where I was trying to debug some code and the Visual Studio debugger wouldn't stop in my class. It would stop in the test class from which I was debugging, but not the class I really wanted to debug. I looked on the breakpoint and noticed that it was transparent, so I hovered over it to view the tooltip, and the tooltip informed me that my class had a DebuggerStepThroughAttribute assigned to it. My first thought was "what ? that's bullshit", and then it occurred to me that I hadn't looked at the rest of the (partial) class which was defined in another file that was generated from an XML schema by xsd.exe. Sure enough, xsd.exe places DebuggerStepThroughAttribute attributes on the classes it generates. Why is this default behaviour ? This was an annoying as hell bug that took time out of my day that's better devoted to other things, like being productive so that I don't get shit from my boss. It also took more time than I'd care to admit to, largely due to my inexperience with Visual Studio and .NET. I've had yet to see anything like this coming from a Java / scripting background.

Tricky styling issues in WPF

For about the last 8 or 9 hours (spread across two days) I've been troubleshooting a styling issue in WPF. I have a list of items that I'm displaying in a ListBox that have an 'enabled' property, which is an enum with values OFF and ON. Understand that this type was generated from an XML schema which is very tightly controlled, hence why the type is not a boolean (as would make sense with a name like 'enabled'). I've been trying to create a DataTemplate for the ListBoxItems so that they'll each have an icon associated with them depending on whether they're enabled or not. My original list looked like this :

<ListBox x:Name="ProcedureList" ItemsSource="{Binding Path=procedure}" HorizontalAlignment="Stretch" VerticalAlignment="Stretch">
  <ListBox.Resources>
    <Style TargetType="{x:Type ListBoxItem}">
      <EventSetter Event="MouseDoubleClick" Handler="WeldProcedure_MouseDoubleClick" />
      <Setter Property="HorizontalContentAlignment" Value="Stretch" />
    </Style>
    <Style TargetType="{x:Type Image}">
      <Style.Triggers>
        <DataTrigger Binding="{Binding}" Value="OFF">
          <Setter Property="Image.Source" Value="Resources/Error_16x16_72.png"/>
        </DataTrigger>
        <DataTrigger Binding="{Binding}" Value="ON">
          <Setter Property="Image.Source" Value="Resources/Success_16x16_72.png"/>
        </DataTrigger>
      </Style.Triggers>
    </Style>
  </ListBox.Resources>
  <ListBox.ItemTemplate>
    <DataTemplate DataType="{x:Type model:WeldingProcedure}">
      <DockPanel HorizontalAlignment="Stretch" LastChildFill="True">
        <Image DockPanel.Dock="Right" Height="16" Width="16" DataContext="{Binding Path=enabled}" ToolTip="Disabled"/>
        <TextBlock DockPanel.Dock="Left" VerticalAlignment="Center" TextAlignment="Left" Text="{Binding Path=procedureName}"/>
      </DockPanel>
    </DataTemplate>
  </ListBox.ItemTemplate>
</ListBox>

However, the images in the DataTemplate were not getting styled with the images as they should have been. As it turns out, if you're referencing default styles in Templates as above, the style resolution doesn't go outside of the template, so the default styling for all Image elements as above needs to go *inside* the DataTemplate resources :

<ListBox x:Name="ProcedureList" ItemsSource="{Binding Path=procedure}" HorizontalAlignment="Stretch" VerticalAlignment="Stretch">
  <ListBox.Resources>
    <Style TargetType="{x:Type ListBoxItem}">
      <EventSetter Event="MouseDoubleClick" Handler="WeldProcedure_MouseDoubleClick" />
      <Setter Property="HorizontalContentAlignment" Value="Stretch" />
    </Style>
  </ListBox.Resources>
  <ListBox.ItemTemplate>
    <DataTemplate DataType="{x:Type model:WeldingProcedure}">
      <DataTemplate.Resources>
        <Style TargetType="{x:Type Image}">
          <Style.Triggers>
            <DataTrigger Binding="{Binding}" Value="OFF">
              <Setter Property="Image.Source" Value="Resources/Error_16x16_72.png"/>
            </DataTrigger>
            <DataTrigger Binding="{Binding}" Value="ON">
              <Setter Property="Image.Source" Value="Resources/Success_16x16_72.png"/>
            </DataTrigger>
          </Style.Triggers>
        </Style>
      </DataTemplate.Resources>
      <DockPanel HorizontalAlignment="Stretch" LastChildFill="True">
        <Image DockPanel.Dock="Right" Height="16" Width="16" DataContext="{Binding Path=enabled}" ToolTip="Disabled"/>
        <TextBlock DockPanel.Dock="Left" VerticalAlignment="Center" TextAlignment="Left" Text="{Binding Path=procedureName}"/>
      </DockPanel>
    </DataTemplate>
  </ListBox.ItemTemplate>
</ListBox>

Generating XML objects from a schema in .NET

In my last job, I frequently made use of the JAXB library provided with the Java SDK for object de/serialization. I chose to generate my schemata from objects back then because I started off with the business objects and I knew how I wanted to serialize them to XML. Now I have the reverse situation, and I'm using a new language. As it turns out, generating classes from XML schema is dead easy : use the xsd.exe tool. It comes with .NET.

Useful GIMP tricks - Batch file conversion

Now that I've started having to do a lot more documenting for my job, I've found myself importing a lot of pictures taken as steps in tutorials and for providing figures. Since the camera gives us a nice, high-res version that's not really suitable for documentation, I have to scale the images down so that they're in a nice, readable size. When your tutorial has 30 steps, and an image (or even two or three) is required for *each step*, this can mean a lot of manual clicking, dragging, opening and saving in an image editor. Fortunately, GIMP has a plugin called Davids Batch Processor that allows you to easily do fairly simple transforms to file and output them in the format (that's most likely) of your choice. For the simple tasks of rotating, resizing (scaling) and exporting the images in a new format, this plugin's great. There's a tutorial on the site and the plugin itself has already proven to be exceedingly useful for me. I hope it will be for you too.

Subtleties of Perl - Reading files

I've recently begun a new job, and with it has come a whole new segment of the software development universe. The new job uses a lot of Perl, C, Bash and various other languages to get stuff done. Today, I ran afoul of a Perl idiosyncrasy that's worth making a note of, because I'm sure I'll stumble across this problem again and I'm going to need to refer to this in the future. I should also note that I'm writing this as I'm waiting for a significantly large file to parse.

We have large log files that we parse on a daily basis to extract summary information from them about mechanical systems. We read the files, and then output a summary on a secondly basis, one line at a time. Recently, I ran afoul of Perl's file reading mechanisms. When reading files in Perl, there's any number of ways to do so, and it turns out that for the longest time, we've been using the wrong one. Previously, we had been using :

foreach my $line_of_log (<LOG>)
{
    // DO STUFF WITH $line_of_log
}

We thought that this was reading the file in one line of the log file at a time, processing it, and then moving on. What it was actually doing was reading (or "slurping") the whole file into memory, and giving us an array of strings, which we processed one line at a time. After 10 minutes of cursory Googling, I ran across a tutorial which presented this :

while (<LOG>)
{
    my $line_of_log = $_;
    // DO STUFF WITH $line_of_log
}

The 'while' version of the file read actually does what we thought we where doing all along: reading one line from the file, and then doing stuff with it. The difference between the two methods is that in the 'foreach' version, the entire input file gets read into memory, whereas in the 'while' version, only a single line gets read into memory at any given time. As it turns out, another difference is that reading in a 7 MB file resulted in Perl grabbing 34 MB of memory with the 'foreach' version, but only 2.2 MB with the 'while' version. That's an ENTIRE ORDER OF MAGNITUDE in difference!. This also makes a huge difference when running Perl on memory-limited systems, as we are.

Tweaking visual studio

I've recently begun using Visual Studio 2010 Beta 2 more and more as I have increasing amounts of work that require .NET. In order to keep things nicely tabbed the both I and my boss like it, there's a setting that can be set according to instructions founds here : http://mhinze.com/tabs-whitespace-visual-studio/

Using the MySQL EXPLAIN statement

Recently I had been having trouble with queries on a certain table in a system that I've been maintaining. I had gone through just about every excuse for why queries on the table could be performing so slowly : the machine was slow (DB running on a VM), the webserver was slow (also running on a VM), I wasn't using the native libraries (webserver was Tomcat), I had other processes running in the background (I didn't). Then I ran across a tip on a forum suggesting usage of the MySQL EXPLAIN statement. I had known all about it for the longest time, but it never occurred to me to actually use it (I think I'm that good at writing queries, turns out : I'm wrong). After using the EXPLAIN statement, I found out that the query processor was using a suboptimal query plan which utilized an index I had added with the intent of improving performance (the index had a fairly high arity, so choosing to use it was sketchy at best in the first place). Most DBMSs should have a similar functionality built in. I think that'll be the first place I go in future.

Ubuntu 9.04 servers slow to login

Recently our production system's servers have been getting progressively slower, and it has finally gotten to the point where it's merited my full attention to remedy the situation. In my research on the problem, one of the things I came across was how incredibly slow the logins were (among other problems). After watching top during numerous logins, I saw that console-kit-daemon was going horribly slow and shooting the sshd CPU usage through the roof. After some time googling for issues related to console-kit-daemon, I found that many people were getting errors in their /var/log/daemon.log file regarding console-kit-daemon being unable to initialize policykit. After installing policykit, my logins to my production servers are now lightning fast

Keeping your servers up to date

As I've previously mentioned on this blog, our company uses Ubuntu for our servers. I won't re-iterate the reasons why in this post, you can search this blog using an Ubuntu tag if you're interested in them. Our servers have uptimes of months and months, and as a result, the clocks on the machines tend to drift over time, which has inspired me to start using NTP to keep the clocks synchronized. A quick Google search yielded the desired results. In order to manually synchronize with NTP, you can run the following commands :

sudo /etc/network/if-up.d/ntpdate

sudo ntpdate pool.ntp.org

Obviously, entering this command every day or every week gets tiresome and stupid, so you can get cron to schedule this daily for you by running the following commands :

echo "sudo ntpdate ntp.ubuntu.com" >> /etc/cron.daily/ntpdate
chmod 755 /etc/cron.daily/ntpdate

Tomcat keystore : too many open files - Continued

Further to my last post, it seems there was another, larger underlaying problem that was causing the exceptionally high number of connections to our servers. One of our clients "required" (I use that term loosely because they really didn't the information) extra information for transactions that was not included in the optimized change metadata that they had been instructed to query for in order to update their transactions. So instead of just querying for the update metadata, they would do that, and then they'd query for each individual transaction. Assume we page our metadata at 100 transactions / page. For example, if we were to have a batch of 800 transactions submitted by the client, instead of making 800 / 100 = 8 calls to our server to update the transactions in the batch, they'd have 8 + (800 * 1) = 808 calls to update their system. They were effectively launching a Denial of Service attack on our servers each time they wanted to update a batch of transactions. Needless to say I consulted with them on the issue and updated our change metadata to include the information they "need" (which they've already got in their system), and they've updated their system to take the number of requests down to the proper level to update their transactions. So let this be a lesson to anybody reading this blog post that has to develop systems that deal with external clients : ensure your clients fully understand the purpose and intent of all the features of our system before they start developing for it and using it.

Too many open sockets error in Tomcat 5.5

Our company recently started a new project. Our primary client is the first one to use it, and use they have. I've suspected they've been putting an unusually high load on our servers, and tonight it was confirmed when our server stopped handling requests, refusing them and then generating "Too many open files" errors in the Tomcat log files referring to my Tomcat SSL keystore. After doing some brief research on the error, I've discovered that this can happen when the Tomcat element in server.xml is configured with too few 'maxThreads' and a too low an 'acceptCount'. I've since tripled the number of threads and acceptable connections. Hopefully this will resolve things.

XML namespace error with Spring WS

Today I ran into a strange error trying to access a webservice via Spring WS (Spring Web Services). The exception looked like the following :

org.springframework.ws.soap.saaj.SaajSoapEnvelopeException: Could not access envelope: Unable to create envelope from given source: ; nested exception is com.sun.xml.messaging.saaj.SOAPExceptionImpl: Unable to create envelope from given source: 
Caused by: 
com.sun.xml.messaging.saaj.SOAPExceptionImpl: Unable to create envelope from given source: 
 at com.sun.xml.messaging.saaj.soap.EnvelopeFactory.createEnvelope(EnvelopeFactory.java:95)
 at com.sun.xml.messaging.saaj.soap.ver1_1.SOAPPart1_1Impl.createEnvelopeFromSource(SOAPPart1_1Impl.java:51)
 at com.sun.xml.messaging.saaj.soap.SOAPPartImpl.getEnvelope(SOAPPartImpl.java:106)
 at org.springframework.ws.soap.saaj.Saaj13Implementation.getEnvelope(Saaj13Implementation.java:145)
 at org.springframework.ws.soap.saaj.SaajSoapMessage.getEnvelope(SaajSoapMessage.java:84)
 at org.springframework.ws.soap.AbstractSoapMessage.getSoapHeader(AbstractSoapMessage.java:42)
 at org.springframework.ws.soap.server.SoapMessageDispatcher.handleRequest(SoapMessageDispatcher.java:91)
 at org.springframework.ws.server.MessageDispatcher.dispatch(MessageDispatcher.java:189)
 at org.springframework.ws.server.MessageDispatcher.receive(MessageDispatcher.java:166)
 at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handle(WebServiceMessageReceiverObjectSupport.java:78)
 at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:60)
 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:819)
 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:754)
 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:399)
 at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:364)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
 at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
 at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
 at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
 at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
 at java.lang.Thread.run(Unknown Source)
Caused by: javax.xml.transform.TransformerException: org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
 at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)
 at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)
 at com.sun.xml.messaging.saaj.util.transform.EfficientStreamingTransformer.transform(EfficientStreamingTransformer.java:371)
 at com.sun.xml.messaging.saaj.soap.EnvelopeFactory.createEnvelope(EnvelopeFactory.java:83)
 ... 30 more
Caused by: org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
 at com.sun.org.apache.xerces.internal.dom.AttrNSImpl.setName(Unknown Source)
 at com.sun.org.apache.xerces.internal.dom.AttrNSImpl.(Unknown Source)
 at com.sun.org.apache.xerces.internal.dom.CoreDocumentImpl.createAttributeNS(Unknown Source)
 at com.sun.xml.messaging.saaj.soap.SOAPDocumentImpl.createAttributeNS(SOAPDocumentImpl.java:142)
 at com.sun.org.apache.xerces.internal.dom.ElementImpl.setAttributeNS(Unknown Source)
 at com.sun.xml.messaging.saaj.soap.impl.ElementImpl.setAttributeNS(ElementImpl.java:1190)
 at com.sun.org.apache.xalan.internal.xsltc.trax.SAX2DOM.startElement(Unknown Source)
 at com.sun.org.apache.xml.internal.serializer.ToXMLSAXHandler.closeStartTag(Unknown Source)
 at com.sun.org.apache.xml.internal.serializer.ToSAXHandler.flushPending(Unknown Source)
 at com.sun.org.apache.xml.internal.serializer.ToXMLSAXHandler.startElement(Unknown Source)
 at org.xml.sax.helpers.XMLFilterImpl.startElement(Unknown Source)
 at com.sun.xml.messaging.saaj.util.RejectDoctypeSaxFilter.startElement(RejectDoctypeSaxFilter.java:157)
 at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(Unknown Source)
 at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
 at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
 at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
 at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
 at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
 at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
 at org.xml.sax.helpers.XMLFilterImpl.parse(Unknown Source)
 at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(Unknown Source)
 ... 34 more

As it turns out, the Java 6 JDK comes with its own integrated versions of Apache Xerces and Apache Xalan. The problem is that the integrated versions are old. Updating your project's POM to ensure versions of Apache Xerces >= 2.8.1 and Apache Xalan >= 2.7.0 are in your classpath will remedy the problem.

Some helpful troubleshooting tips for SFTP jails

Further to my previous post on SFTP jails, I've run across a few issues. When logging in via command line after first setting up the jail, you may encounter immediate disconnects with the response "Connection Closed". One possible cause of this is missing files, specifically "/lib64/ld-linux-x86-64.so.2" under the jail. There are other files that can possibly be missing, but that's a biggie if you're running a 64-bit kernel. If you're running WinSCP, you may also encounter the error message "Is the host running a SFTP server?"

A quick note on intercepting annotations with Spring AOP

Spring AOP is incredibly powerful. One of the most powerful features of the framework is that it lets you intercept methods that are decorated with particular annotations using the following AOP expression :

@annotation(my.package.MyAnnotation)

This can be used in place of :

execution(public * my.package.MyService.serviceMethod(*))

...or in combination with it. But one important point : the annotation being intercepted must be in service method implementations (ie concrete class implementations), not on the service method interface declarations! Otherwise Spring AOP will not be able to read and intercept the annotated methods / classes

Setting up Microsoft Outlook 2007 with Microsoft Exchange

...via RPC over HTTPS :

1. Ensure you have your root certificate in your trusted authority store in the Microsoft Management Console (mmc.exe)
2. Ensure you have your personal certificate in your personal key store in the Microsoft Management Console (mmc.exe)
3. Ensure you've got the correct server settings

Setting up a VPN connection to an Astaro security applicance on Mac OS X

So I finally got a working VPN configuration to be able to connect my shiny new MacBook Pro to our VPN on our production system which uses an Astaro firewall appliance. Turns out it's fairly easy too :

• Go to your Astaro firewall's user profile page, ie login as a user, and download the key / OpenVPN configuration package for your router. The great thing about these appliances is that they provide the package for you right from the device


• Download Tunnelblick. It's an (apparently very easy to use) GUI front end for OpenVPN on Mac OS X. The latest version (as of the time of this writing) is 3.0 beta 10. Tunnelblick is hosted on Google Code. The installation's even easier. Load up the disk image, and drag the sole application icon into your applications folder.


• Open the VPN package you downloaded from your Astaro firewall appliance, and copy everything in it to the ~/Library/openvpn folder created by the Tunnelblick installation


• Tunnelblick should automatically detect the additional files and provide a listing for the Astaro configuration in its connection list. Once you've got the item in the list, just click on it and it should automagically connect to the Astaro firewall appliance and your VPN connection should be working.

Oh, and make sure you've actually got Tunnelblick started, or else you won't be able to connect ;)

Playing with my new Mac

I recently managed to convince my employer to provide me with a MacBook Pro so that I could start developing in-house iPhone applications for the company. I'm not yet at the point where I'm writing applications, but I am installing tons of stuff on the laptop so that I can use it as my mobile computer for when I'm on the road for work. It certainly has its quirks that I have to get used to over running everything in Windows. This will be a very brief post, but suffice it to say that I've learned a few things :

1) When it comes to Maven, ensure that JAVA_HOME is set in your environment variables.
2) Make sure you're choosing the right JVM. MacOSX comes with a bunch of default JVMs installed, and you can configure them through the Java control panel in Applications -> Utilities

More Hibernate glitches, part 2

I had previously written on an Expected positional parameter glitch in Hibernate when joining between entities using OneToOne and PrimaryKey + ForeignKey columns. It turns out, Hibernate has this problem no matter what field you query on the joined entity if you're quering by another entity. A way around this is to use the ID property of the joined entities and add another association to your HQL / Criteria query.

ie So if you have Person and Address having a OneToOne relationship

public class Person {
  @OneToOne
  private Address address;
}

... and you want to query for a Person by Address, you would probably write something like :

from Person p where p.address = ?

... in HQL. Well, this is where you start running into the Expected positional parameter glitch. Instead, write your query like this :

from Person p where p.address.id = ?

... and pass in a long / int / whatever type you use for an identifier for Person / Address instead and this will work around the glitch until it can be fixed.

Stupid MySQL tricks

Recently, I found that our secondary server's MySQL partition was starting to get full. After running du -h --max-depth=1 several times recursively, I found that the source was the MySQL binary logs. As it turns out, MySQL comes with a very handy command for purging all log files. If you run :

show master logs

, you'll be given a list of the logs that the MySQL server is currently using. You can clear out old log files (some of which can be very large) by running the command

purge master logs to 'log-name-here -from-display'

. This will remove the old log files from the hard disk and save up some potentially much needed space.