Resolving slow SSH login times

At work, like most other organizations, we have a Linux server that we access via SSH. However, lately, I've found my use of it skyrocketing in order to test software, and log-ins and file copying have been very slow. In my efforts to find out why, I found out that sshd has in its configuration file a setting called 'UseDNS'. The default is 'yes', so even if this setting is commented out, it will try to perform reverse DNS lookups on the IP addresses of users logging in. This would be a bad thing for us, especially since we have this box locked down and unable to contact any DNS servers. After disabling DNS lookups, my login and file copy time dropped to near 0. I hope this helps somebody.

*EDIT:* I've also encountered this problem on the BeagleBone Black boards which I've recently acquired for use in a project.

Two bash functions that will make your software development life 10% easier

... if you use SSH to move between machines a lot. Which I do. I have several machines that I work on regularly, and a centralized development machine shared with others that I frequently have to exchange files with. The following two functions are the most useful two bash functions that you can put in your .bashrc file if you use SSH as much as I do :

function ssh-create-keys {
ssh-keygen -t rsa
}

function ssh-setup-on {
cat ~/.ssh/id_rsa.pub | ssh $1 "cat - >>.ssh/authorized_keys"
}

The first sets up an SSH key file for you in your home directory, This key file will be used to identify you on other machines, and can be used for various purposes on your local machine as well. The second function logs you into the machine determined by the username@anothermachine given as the first argument to the function, and adds your public key to the list of authorized keys for that machine.

Once you've run the first function, and run the second function to setup your public key on an account on another machine, you'll now be able to use SSH to log in and copy files freely with the account on the other machine without having to enter a password. While this is incredibly convenience, it also comes with a caveat : it's dangerous. If somebody other than yourself gains physical access to your machine and can log on as you, (or use your already logged on account), they can move to those same machines freely and perform possibly malicious actions, as you. Keep that in mind.

Copying files in linux

Lately I've been doing a lot of embedded development with Linux, and copying files between systems has been a bit of a pain. Fortunately, a combination of RSync and SSH solved my problems, with a command that lets me copy files from a directory on one system to a directory on another system, recursively with symlink preservation (and even duplication !) :

rsync -azuv -e ssh  user@systemaddress:~/path/to/dir/* .

WTF ?! Windows (as of Vista) no longer supports multiple different monitors!? WHY ????

Apparently, as of Windows Vista and later, Windows no longer supports multiple video cards that don't use the same driver!! Why ? I use a Radeon and a Matrox QID to drive 6 displays on my development box, and this just entirely fucks me over. At best, I can only use four of my displays now, and that's only going to be when Matrox gets off their lazy ass and releases a Windows 7 driver for their QID LP PCIe video cards. I'm so disappointed with what's a complete step backward for Microsoft. Epic fail, Microsoft.

Seeing where Perl looks for its modules on a system

Because we have very limited space on some embedded devices that we use that run Perl, we can only store a very few modules on these systems. This is a consequence of the fact that we run Busybox on these things, so by definition everything on these boxes is limited, if present at all. Therefore, we have to check and see if a module is available before we can use it in our code. Fortunately, there's a quick one-liner to see where Perl is looking for modules on a system :

perl -e'print join "\n", @INC'

I got this off a forum post from somewhere, and I'd post it here if I could find the link again. My apologies to the author of that forum post if they ever happen to run across this blog.

Embarking on a Perl journey

A long time ago, in a job far, far away, I had to deal with some Perl. I learned just enough to get me by for the duration of the task at hand, and then pretty much forgot everything I had learned. Now, at my latest job, I'm having to deal extensively with legacy systems which have a considerable amount of logic written in Perl that needs to be either ported over to other languages (for various reasons) or updated and new things written because Perl is the only language that's both abstract enough and not too processor intensive to run on the embedded systems we deal with. Therefore, you're going to start seeing a lot more Perl posts on this blog.

Setting up Tomcat (5.5) on Ubuntu Server 8.10

I recently ran into some old quirks when provisioning a new server for our company's web applications on Ubuntu 8.10 (Intrepid Ibex). Because the manager apps are no longer installed by default, you need to add extra packages to the list to install when installing Tomcat :

sudo apt-get install -y tomcat5.5 tomcat5.5-admin tomcat5.5-webapps

If you're copying configuration over from a previous Tomcat / Ubuntu installation, you need to make sure the permissions on all the files you copy are set correctly. In most cases, you'll have to run :

chown -R tomcat55:adm [file and folder list here]

If you're securing the applications with a certificate, try to make sure it's valid for your location and ensure that you've set it properly in your server.xml configuration file. If you want useful logging, you'll also have to place a log4j.properties file in

$CATALINA_HOME/common/classes

Hope this helps.

Dumping just your schema with MySQL dump

A simple one-liner :

mysqldump -u root -p mydatabasename --no-data=true --add-drop-table=false > test_dump.sql

With this command, you'll be prompted for your root password. I got this from here. Simple

The Curious Case of Damned DataIntegrityViolationException

In one of the projects on which I contract, we recently started encountering a problem importing and parsing text record files into our system which previously had no problem. My first thought on hearing this was that the partner from which we obtain the files had changed the file format (again). Upon closer inspection, nothing had changed in the files. My next step was to try importing them into a development system and seeing what was going on. As it turned out, the application was catching Spring's DataIntegrityViolationException. I was floored as soon as I saw this because our application was supposed to be catching this exception behind one of the business interfaces and converting it to an internal exception which is used in business logic. After some more poking around to confirm what was really going on, I threw the problem into google, and on the second result was a post in the Spring forum made by a user having exactly the same problem I was.


To summarize their problem quickly, they were using a transaction manager, and they were intercepting their business methods (via interfaces) with Aspects, which we're also doing. The problem was this : as soon as the internal Aspects were applied to the business interface, this changed the ordering of advice applied to the interface implementor, so now the Hibernate session underneath was getting flushed later by the transaction manager, instead of in the business method where it had been flushed previously. The result of this was that now DataIntegrityViolationExceptions were being thrown outside of the interecepted method, instead of inside where it was expected. A manual session.flush() inside of a HibernateCallback within the business method fixed this :

 /**
  * @see AchPaymentNoticeOfChangeService#registerNoc(AchPaymentNoticeOfChange)
  */
 @Override
 public void registerNoc(final AchPaymentNoticeOfChange changeNotification) throws IllegalArgumentException, NoticeOfChangeAlreadyExistsException, Exception {
  try {
   getHibernateTemplate().execute(new HibernateCallback() {
    @Override
    public Object doInHibernate(Session session) throws HibernateException, SQLException {
     
     session.save (changeNotification);
     
     // flush the session to ensure that the database gets synchronized
     // the end of this call, rather than waiting for any transaction
     // managers to handle it and risk letting a DataIntegrityViolationException
     // occur outside of this method's handling
     session.flush();
     
     return null;
    }
   });
  } catch (DataIntegrityViolationException dive) {
   throw new NoticeOfChangeAlreadyExistsException("A notice of change already exists for payment ["+changeNotification.getAchPayment().getId()+"]", dive, changeNotification);
  } catch (Exception e) {
   throw e;
  }
 }

I hope this post finds somebody else who runs into this problem.

Composing XML in C# 2.0 or later

When testing some custom XML serialization I was writing for a project, I ran across a quirk of the XmlWriter created by XmlWriter#Create() : The writer that gets produced by this method doesn't actually write out XML to whatever stream or file you've given it in the Create method until Close() is called on the generated writer. This means that if you've been using the XmlWriter along with a using() statement, you're fine, but if you haven't been using it, as I have in the unit tests I've been trying to write, you're going to get some unexpected results. Something to pay attention to.

Visual Studio (2010?) Debugging Gotchas - Part I

I recently ran across a problem where I was trying to debug some code and the Visual Studio debugger wouldn't stop in my class. It would stop in the test class from which I was debugging, but not the class I really wanted to debug. I looked on the breakpoint and noticed that it was transparent, so I hovered over it to view the tooltip, and the tooltip informed me that my class had a DebuggerStepThroughAttribute assigned to it. My first thought was "what ? that's bullshit", and then it occurred to me that I hadn't looked at the rest of the (partial) class which was defined in another file that was generated from an XML schema by xsd.exe. Sure enough, xsd.exe places DebuggerStepThroughAttribute attributes on the classes it generates. Why is this default behaviour ? This was an annoying as hell bug that took time out of my day that's better devoted to other things, like being productive so that I don't get shit from my boss. It also took more time than I'd care to admit to, largely due to my inexperience with Visual Studio and .NET. I've had yet to see anything like this coming from a Java / scripting background.

Tricky styling issues in WPF

For about the last 8 or 9 hours (spread across two days) I've been troubleshooting a styling issue in WPF. I have a list of items that I'm displaying in a ListBox that have an 'enabled' property, which is an enum with values OFF and ON. Understand that this type was generated from an XML schema which is very tightly controlled, hence why the type is not a boolean (as would make sense with a name like 'enabled'). I've been trying to create a DataTemplate for the ListBoxItems so that they'll each have an icon associated with them depending on whether they're enabled or not. My original list looked like this :

<ListBox x:Name="ProcedureList" ItemsSource="{Binding Path=procedure}" HorizontalAlignment="Stretch" VerticalAlignment="Stretch">
  <ListBox.Resources>
    <Style TargetType="{x:Type ListBoxItem}">
      <EventSetter Event="MouseDoubleClick" Handler="WeldProcedure_MouseDoubleClick" />
      <Setter Property="HorizontalContentAlignment" Value="Stretch" />
    </Style>
    <Style TargetType="{x:Type Image}">
      <Style.Triggers>
        <DataTrigger Binding="{Binding}" Value="OFF">
          <Setter Property="Image.Source" Value="Resources/Error_16x16_72.png"/>
        </DataTrigger>
        <DataTrigger Binding="{Binding}" Value="ON">
          <Setter Property="Image.Source" Value="Resources/Success_16x16_72.png"/>
        </DataTrigger>
      </Style.Triggers>
    </Style>
  </ListBox.Resources>
  <ListBox.ItemTemplate>
    <DataTemplate DataType="{x:Type model:WeldingProcedure}">
      <DockPanel HorizontalAlignment="Stretch" LastChildFill="True">
        <Image DockPanel.Dock="Right" Height="16" Width="16" DataContext="{Binding Path=enabled}" ToolTip="Disabled"/>
        <TextBlock DockPanel.Dock="Left" VerticalAlignment="Center" TextAlignment="Left" Text="{Binding Path=procedureName}"/>
      </DockPanel>
    </DataTemplate>
  </ListBox.ItemTemplate>
</ListBox>

However, the images in the DataTemplate were not getting styled with the images as they should have been. As it turns out, if you're referencing default styles in Templates as above, the style resolution doesn't go outside of the template, so the default styling for all Image elements as above needs to go *inside* the DataTemplate resources :

<ListBox x:Name="ProcedureList" ItemsSource="{Binding Path=procedure}" HorizontalAlignment="Stretch" VerticalAlignment="Stretch">
  <ListBox.Resources>
    <Style TargetType="{x:Type ListBoxItem}">
      <EventSetter Event="MouseDoubleClick" Handler="WeldProcedure_MouseDoubleClick" />
      <Setter Property="HorizontalContentAlignment" Value="Stretch" />
    </Style>
  </ListBox.Resources>
  <ListBox.ItemTemplate>
    <DataTemplate DataType="{x:Type model:WeldingProcedure}">
      <DataTemplate.Resources>
        <Style TargetType="{x:Type Image}">
          <Style.Triggers>
            <DataTrigger Binding="{Binding}" Value="OFF">
              <Setter Property="Image.Source" Value="Resources/Error_16x16_72.png"/>
            </DataTrigger>
            <DataTrigger Binding="{Binding}" Value="ON">
              <Setter Property="Image.Source" Value="Resources/Success_16x16_72.png"/>
            </DataTrigger>
          </Style.Triggers>
        </Style>
      </DataTemplate.Resources>
      <DockPanel HorizontalAlignment="Stretch" LastChildFill="True">
        <Image DockPanel.Dock="Right" Height="16" Width="16" DataContext="{Binding Path=enabled}" ToolTip="Disabled"/>
        <TextBlock DockPanel.Dock="Left" VerticalAlignment="Center" TextAlignment="Left" Text="{Binding Path=procedureName}"/>
      </DockPanel>
    </DataTemplate>
  </ListBox.ItemTemplate>
</ListBox>

Generating XML objects from a schema in .NET

In my last job, I frequently made use of the JAXB library provided with the Java SDK for object de/serialization. I chose to generate my schemata from objects back then because I started off with the business objects and I knew how I wanted to serialize them to XML. Now I have the reverse situation, and I'm using a new language. As it turns out, generating classes from XML schema is dead easy : use the xsd.exe tool. It comes with .NET.

Useful GIMP tricks - Batch file conversion

Now that I've started having to do a lot more documenting for my job, I've found myself importing a lot of pictures taken as steps in tutorials and for providing figures. Since the camera gives us a nice, high-res version that's not really suitable for documentation, I have to scale the images down so that they're in a nice, readable size. When your tutorial has 30 steps, and an image (or even two or three) is required for *each step*, this can mean a lot of manual clicking, dragging, opening and saving in an image editor. Fortunately, GIMP has a plugin called Davids Batch Processor that allows you to easily do fairly simple transforms to file and output them in the format (that's most likely) of your choice. For the simple tasks of rotating, resizing (scaling) and exporting the images in a new format, this plugin's great. There's a tutorial on the site and the plugin itself has already proven to be exceedingly useful for me. I hope it will be for you too.

Subtleties of Perl - Reading files

I've recently begun a new job, and with it has come a whole new segment of the software development universe. The new job uses a lot of Perl, C, Bash and various other languages to get stuff done. Today, I ran afoul of a Perl idiosyncrasy that's worth making a note of, because I'm sure I'll stumble across this problem again and I'm going to need to refer to this in the future. I should also note that I'm writing this as I'm waiting for a significantly large file to parse.

We have large log files that we parse on a daily basis to extract summary information from them about mechanical systems. We read the files, and then output a summary on a secondly basis, one line at a time. Recently, I ran afoul of Perl's file reading mechanisms. When reading files in Perl, there's any number of ways to do so, and it turns out that for the longest time, we've been using the wrong one. Previously, we had been using :

foreach my $line_of_log (<LOG>)
{
    // DO STUFF WITH $line_of_log
}

We thought that this was reading the file in one line of the log file at a time, processing it, and then moving on. What it was actually doing was reading (or "slurping") the whole file into memory, and giving us an array of strings, which we processed one line at a time. After 10 minutes of cursory Googling, I ran across a tutorial which presented this :

while (<LOG>)
{
    my $line_of_log = $_;
    // DO STUFF WITH $line_of_log
}

The 'while' version of the file read actually does what we thought we where doing all along: reading one line from the file, and then doing stuff with it. The difference between the two methods is that in the 'foreach' version, the entire input file gets read into memory, whereas in the 'while' version, only a single line gets read into memory at any given time. As it turns out, another difference is that reading in a 7 MB file resulted in Perl grabbing 34 MB of memory with the 'foreach' version, but only 2.2 MB with the 'while' version. That's an ENTIRE ORDER OF MAGNITUDE in difference!. This also makes a huge difference when running Perl on memory-limited systems, as we are.

Tweaking visual studio

I've recently begun using Visual Studio 2010 Beta 2 more and more as I have increasing amounts of work that require .NET. In order to keep things nicely tabbed the both I and my boss like it, there's a setting that can be set according to instructions founds here : http://mhinze.com/tabs-whitespace-visual-studio/

Using the MySQL EXPLAIN statement

Recently I had been having trouble with queries on a certain table in a system that I've been maintaining. I had gone through just about every excuse for why queries on the table could be performing so slowly : the machine was slow (DB running on a VM), the webserver was slow (also running on a VM), I wasn't using the native libraries (webserver was Tomcat), I had other processes running in the background (I didn't). Then I ran across a tip on a forum suggesting usage of the MySQL EXPLAIN statement. I had known all about it for the longest time, but it never occurred to me to actually use it (I think I'm that good at writing queries, turns out : I'm wrong). After using the EXPLAIN statement, I found out that the query processor was using a suboptimal query plan which utilized an index I had added with the intent of improving performance (the index had a fairly high arity, so choosing to use it was sketchy at best in the first place). Most DBMSs should have a similar functionality built in. I think that'll be the first place I go in future.

Ubuntu 9.04 servers slow to login

Recently our production system's servers have been getting progressively slower, and it has finally gotten to the point where it's merited my full attention to remedy the situation. In my research on the problem, one of the things I came across was how incredibly slow the logins were (among other problems). After watching top during numerous logins, I saw that console-kit-daemon was going horribly slow and shooting the sshd CPU usage through the roof. After some time googling for issues related to console-kit-daemon, I found that many people were getting errors in their /var/log/daemon.log file regarding console-kit-daemon being unable to initialize policykit. After installing policykit, my logins to my production servers are now lightning fast

Keeping your servers up to date

As I've previously mentioned on this blog, our company uses Ubuntu for our servers. I won't re-iterate the reasons why in this post, you can search this blog using an Ubuntu tag if you're interested in them. Our servers have uptimes of months and months, and as a result, the clocks on the machines tend to drift over time, which has inspired me to start using NTP to keep the clocks synchronized. A quick Google search yielded the desired results. In order to manually synchronize with NTP, you can run the following commands :

sudo /etc/network/if-up.d/ntpdate

sudo ntpdate pool.ntp.org

Obviously, entering this command every day or every week gets tiresome and stupid, so you can get cron to schedule this daily for you by running the following commands :

echo "sudo ntpdate ntp.ubuntu.com" >> /etc/cron.daily/ntpdate
chmod 755 /etc/cron.daily/ntpdate

Tomcat keystore : too many open files - Continued

Further to my last post, it seems there was another, larger underlaying problem that was causing the exceptionally high number of connections to our servers. One of our clients "required" (I use that term loosely because they really didn't the information) extra information for transactions that was not included in the optimized change metadata that they had been instructed to query for in order to update their transactions. So instead of just querying for the update metadata, they would do that, and then they'd query for each individual transaction. Assume we page our metadata at 100 transactions / page. For example, if we were to have a batch of 800 transactions submitted by the client, instead of making 800 / 100 = 8 calls to our server to update the transactions in the batch, they'd have 8 + (800 * 1) = 808 calls to update their system. They were effectively launching a Denial of Service attack on our servers each time they wanted to update a batch of transactions. Needless to say I consulted with them on the issue and updated our change metadata to include the information they "need" (which they've already got in their system), and they've updated their system to take the number of requests down to the proper level to update their transactions. So let this be a lesson to anybody reading this blog post that has to develop systems that deal with external clients : ensure your clients fully understand the purpose and intent of all the features of our system before they start developing for it and using it.