Bootstrapping a VSTS Linux build agent for Docker containers

1. Create an Ubuntu 17.04 VM in Azure
2. Install dotnetcore using the instructions here.
3. Install the Docker apt-get package using the instructions here.
4. Download and run the VSTS agent Docker image and run it as a Docker container itself using the instructions here, e.g. sudo docker run -e VSTS_ACCOUNT=myvstsaccountname -e VSTS_TOKEN=abcdtokenherefromportal -e VSTS_POOL="Docker Build Agents" -e VSTS_AGENT="myagentname" -it -v /var/run/docker.sock:/var/run/docker.sock microsoft/vsts-agent

Undefined reference to `log' when compiling on Ubuntu 13.10 with gcc

After I upgraded to Ubuntu 13.10, I inexplicably started getting errors in a build that had been perfect for a very long time. It turned out, there was a significant change in the linker and a bug introduced. The fix is described beautifully on this blog post, but for convenience sake:

Add '-Wl,--no-as-needed' to your LDFLAGS

Installing NTP on an Ubuntu server

NTP is used to synchronized time between machines. You can read the Ubuntu HOWTO here.

Installing a TFTP server using xinetd on Ubuntu

I recently had need to redo an old server that had been running an ancient Gentoo installation for which there was no longer a software upgrade path, so I chose to install Ubuntu on it. Part of the requirements of the server were that it runs a TFTP server for hosting files for configuring embedded devices. I had previously found an article on how to setup TFTP through inetd, but I could find it, so I'm cobbling this tutorial together from various sources.

1. # apt-get install xinetd tftpd
2. Ensure that the following lines are in the /etc/services file:
1. tftp     69/tcp
2. tftp     69/udp
3. Open the /etc/xinetd.d/tftp file (create it if it doesn't exist) and ensure that it contains the following:

   # default: off
   # description: The tftp server serves files using the Trivial File Transfer \
   #    Protocol.  The tftp protocol is often used to boot diskless \
   #    workstations, download configuration files to network-aware printers, \
   #    and to start the installation process for some operating systems.
   service tftp
   {
       socket_type     = dgram
       protocol        = udp
       wait            = yes
       user            = root
       server          = /usr/sbin/in.tftpd
       server_args     = -s /tftpboot
       disable         = no
   }
   

4. # /etc/init.d/xinetd restart

5. Test the server:# tftp localhost

   tftp> get hello.txt
   Received 23 bytes in 0.1 seconds
   tftp> quit 

You should now be good to go. The (abridged) instructions for this were retrieved from this article.

How to fix the ugly ass Ubuntu boot splash OR Damn, Ubuntu, you ugly

TL;WR: http://news.softpedia.com/news/How-to-Fix-the-Big-and-Ugly-Plymouth-Logo-in-Ubuntu-10-04-140810.shtml

If you've had Ubuntu for a while and you've got an nVidia video card, chances are you've got the ugly ass default boot screen. Fortunately, you can fix this. You just need to follow the instructions above. I hope you're comfortable with the command line.

Running multiple Tomcat 6 instances in Ubuntu, the quick and dirty way

Here's a quick list for running multiple Tomcat 6 instances on Ubuntu 10.04 :

• Copy the /etc/init.d/tomcat script with a new name in the same directory


• Update the NAME variable in the startup script with a name for the new instance that you want to run.


• Copy /usr/share/$(old)NAME to the (new) NAME you've just created, along with /var/lib/$(old)NAME and /etc/default/$(old)NAME


• Edit the server.xml file under /var/lib/$(new)NAME/conf/ and change all the ports (ie for shutdown, and all your Connectors) so that they don't conflict with the old instance


• Run /etc/init.d/$(new tomcat script name) start

You should now have a running, fully functional instance of Tomcat on the server, using a different port.

Setting up Tomcat (5.5) on Ubuntu Server 8.10

I recently ran into some old quirks when provisioning a new server for our company's web applications on Ubuntu 8.10 (Intrepid Ibex). Because the manager apps are no longer installed by default, you need to add extra packages to the list to install when installing Tomcat :

sudo apt-get install -y tomcat5.5 tomcat5.5-admin tomcat5.5-webapps

If you're copying configuration over from a previous Tomcat / Ubuntu installation, you need to make sure the permissions on all the files you copy are set correctly. In most cases, you'll have to run :

chown -R tomcat55:adm [file and folder list here]

If you're securing the applications with a certificate, try to make sure it's valid for your location and ensure that you've set it properly in your server.xml configuration file. If you want useful logging, you'll also have to place a log4j.properties file in

$CATALINA_HOME/common/classes

Hope this helps.

Keeping your servers up to date

As I've previously mentioned on this blog, our company uses Ubuntu for our servers. I won't re-iterate the reasons why in this post, you can search this blog using an Ubuntu tag if you're interested in them. Our servers have uptimes of months and months, and as a result, the clocks on the machines tend to drift over time, which has inspired me to start using NTP to keep the clocks synchronized. A quick Google search yielded the desired results. In order to manually synchronize with NTP, you can run the following commands :

sudo /etc/network/if-up.d/ntpdate

sudo ntpdate pool.ntp.org

Obviously, entering this command every day or every week gets tiresome and stupid, so you can get cron to schedule this daily for you by running the following commands :

echo "sudo ntpdate ntp.ubuntu.com" >> /etc/cron.daily/ntpdate
chmod 755 /etc/cron.daily/ntpdate

Some helpful troubleshooting tips for SFTP jails

Further to my previous post on SFTP jails, I've run across a few issues. When logging in via command line after first setting up the jail, you may encounter immediate disconnects with the response "Connection Closed". One possible cause of this is missing files, specifically "/lib64/ld-linux-x86-64.so.2" under the jail. There are other files that can possibly be missing, but that's a biggie if you're running a 64-bit kernel. If you're running WinSCP, you may also encounter the error message "Is the host running a SFTP server?"

Allowing external root access to your MySQL database

...is recommended against in the MySQL documentation, but it can be oh so handy for running scripts and adding databases, tables and views when you can't (or don't want to) log into that remote system on the command line to update it. However, when you try it (at least on a fresh Ubuntu install), you may be greeted with an Error 2003. This happens when your system does not allow root external access. To enable external access, you can run this query (as root) on your server from the command line (must be localhost) :

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY ('myrootpassword');

* EDIT * : The above statement does not necessarily copy all privileges over to the new entry for 'root'@'%' in the mysql.user table. A better way to accomplish full external root access is the following :

DELETE FROM `mysql`.`user` WHERE User = 'root' AND Host = '%';
INSERT INTO `mysql`.`user` (Host, User, Password, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Reload_priv, Shutdown_priv, Process_priv, File_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Show_db_priv, Super_priv, Create_tmp_table_priv, Lock_tables_priv, Execute_priv, Repl_slave_priv, Repl_client_priv, Create_view_priv, Show_view_priv, Create_routine_priv, Alter_routine_priv, Create_user_priv, ssl_type, ssl_cipher, x509_issuer, x509_subject, max_questions, max_updates, max_connections, max_user_connections) (SELECT '%', User, Password, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Reload_priv, Shutdown_priv, Process_priv, File_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Show_db_priv, Super_priv, Create_tmp_table_priv, Lock_tables_priv, Execute_priv, Repl_slave_priv, Repl_client_priv, Create_view_priv, Show_view_priv, Create_routine_priv, Alter_routine_priv, Create_user_priv, ssl_type, ssl_cipher, x509_issuer, x509_subject, max_questions, max_updates, max_connections, max_user_connections FROM `mysql`.`user` WHERE User = 'root' AND Host = 'localhost');
FLUSH PRIVILEGES;

Setting up MySQL server on Ubuntu

...is really easy. Run the command :

apt-get install -y mysql-server

Once that's done, you'll need to perform some additional configuration and setup the root password. My added configuration involves making table name comparisons all lower case (convenient for queries) and making the server run entirely in utf8. In order to configure the server as such, you'll need to make the following changes :

Add the following lines to the [mysqld] section of the my.cnf configuration :

default-character-set=utf8
default-collation=utf8_general_ci
lower_case_table_names=1

Add the following line to the [mysql] section of my.cnf (note that this is not the same as the [mysqld] section, the difference being the 'd' on the end) :

default-character-set=utf8

In order to have the Ubuntu MySQL installation properly run on external hosts, you'll have to comment out the line that looks like :

bind-address     127.0.0.1

After you've configured the my.cnf file, you'll have to stop the server, and restart it with --skip-grant-tables enabled so that you can have free, unfettered access to the system with privileges disabled. This is required so that you can set the initial root password. Run the following commands as root :

/etc/init.d/mysql stop
mysqld_safe --skip-grant-tables --skip-networking &

The latter command will start the MySQL server with privileges disabled.
Then log in with the command

mysql -u root -h localhost

Once logged in, run the command :

UPDATE `mysql`.`user` SET Password = PASSWORD('thepassword') WHERE User = 'root';

This will set the user password to 'thepassword' (though I recommend you don't use that exact password). Once you've set the root password, exit mysql and run the command

/etc/init.d/mysql restart

This will restart the server as usual, so that you'll no longer have unlimited privileges (and neither will anyone else).

Setting up an SSH Server on Ubuntu

Ubuntu is usually pretty good about coming pre-configured with everything you need for your system, but one thing I've found that it's lacking (at least the server distribution is, anyway) is a pre-setup SSH server. Fortunately though, it's quite easy to set one up. All you need to do is issue the command:

apt-get install openssh-server

...as root. If you get an error to the effect of :

Package does not exist

...then you'll have to run :

apt-get update

...to update your system's packages first.

A moment of developer clarity and zen

I had a moment today where I was just letting my mind wander. It was one of those moments where you have a brilliant idea that just seems so common sense that you can only ask yourself "Why the hell didn't I think of that before ?" This moment of clarity happened to be concerning my horribly slow laptop, with it's horribly old ATA 100, 5400 rpm hard drive. Then it occurred to me : just like with video encoding that I had setup months ago, I could setup a RAM disk to speed up Eclipse by storing my workspace on the RAM disk to speed up access to all the (ridiculous number of) temp files that Eclipse uses. On Ubuntu Linux, it turns out this was ridiculously easy : http://ubuntuforums.org/showthread.php?t=182764

The really amazing part was just how much this sped things up : it took my project build time from 6 mins, 30 seconds down to 35 seconds. And those are real numbers (plus or minus a couple seconds, due to slight variations in repeated builds.)

Pure FTP on Ubuntu

Pure FTP is a great FTP server and I love it because it's ridiculously fast, reliable and works well for our company. The only problem is that it's tricky to configure on Ubuntu because just like with numerous other programs, the people at Ubuntu, in their infinite wisdom, felt the need to fuck with the program's default way of doing things. Specifically, in Ubuntu, command line arguments for starting the FTP service are done through individual files for each command line option you want the service to be started with, with values for the options placed inside each of the files. This is a stupid way of doing things, but that's not the topic of this post.

The problem I've been having lately is that I recently deployed new projects within our production network, and these new projects required access to the FTP server, and aren't on the same box as the server, as is the sole other project that has been using the FTP server. I'd been getting problems trying to connect to the server from any LAN box, but not any external boxes nor the machine itself (localhost). The error I got back was "421 Service not available". I googled around for hours and found nothing useful, until I started realizing that other people were getting 421 errors when their PureFTP instance was misconfigured, but with different messages, and then it got me thinking that maybe my instance was somehow misconfigured.

I re-read the documentation for PureFTP and after an hour or so, it hit me that the server does reverse lookups to resolve fully qualified names, and such resolution doesn't work properly on our network (for good reasons that I'm not going to go into). After disabling reverse DNS resolution with the -H startup option (`echo "yes" >> DontResolve` in the configuration directory in Ubuntu), the problem went away.

I hope this helps anybody else who runs into the same problem.

Tomcat quirks

In recent weeks, we've had problems with our company's servers going down because of power problems at our colocation provider. As a result, the fact that Tomcat hadn't been properly configured to boot at startup had really burned us. Consequentially, I've been spending the past few days troubleshooting why the servers wouldn't startup, and ensuring that they do come up properly at boot time in future. The following are some of the things I've run across.

• 
  The JAVA_HOME and JRE_HOME variables must be set. If they're not, then Tomcat won't start, at least on Ubuntu Gutsy Gibbon Server. This is a problem I encountered with the default installation of JIRA on one of our production servers.
  


• 
  When using EhCache with Hibernate, the tmp directories don't get deleted, regardless of whether or not Tomcat is shutdown properly. This will block the container from starting up again, so these temp files must be deleted. I encountered this problem with my own applications.
  

Fixing package retrieval quirks in Ubuntu Gutsy Gibbon Server

In the default installation of Ubuntu Server (Gutsy Gibbon), the default source of packages is the CD-ROM drive. That's fine if you make the assumption that there will always be somebody physically present in order to maintain the servers, but for most server's these days, that's a fallacious assumption. The really bad part about this default, is that there's no way I know of at the time of this writing to override this on the update command. In our company, we have our servers hosted off-site like so many other companies, so everything must be done remotely. Therefore, the CD-ROM as a source for Ubuntu packages is a bad idea. To remedy this, you have to go into the apt-get configuration stored at

/etc/apt/sources.list

, and you'll have to find a line similar to :

deb cdrom:[Ubuntu-Server 7.10 _Gutsy Gibbon_ - Release i386 (20071016)]/ gutsy main restricted

Comment out this line, try your update again and you should be good to go.

Determining which version of Ubuntu you're running

At our company, we use Ubuntu to run our servers. Sure, there's other distributions with potentially better performance, but none so easy to setup, and that was the deciding factor in choosing which distro was going to run our systems because I had to set it up, and since I'm both developer and system administrator (*very* small company), I really don't have time to fuck around and I need to make things as easy on myself as possible. To that end, I've decided to start writing tutorials on everything I learn how to do, and I needed to make sure I note which version of Ubuntu / Linux I'm using. To that end, in order to determine which version of Ubuntu you're running, run this :

cat /etc/issue

I learned of this little trick from here.

*Edit (2008-10-29)*
A much better command for determining the current Ubuntu version you're running :

lsb_release -a

Setting up your own domain in (Ubuntu) Linux

...is easier than you actually think. I managed to get a working domain name for our company's servers going using tutorials and information collected from here, here, here and here.
I'd love to be able to write a coherent guide based off of the information from these pages, but I'm just too busy at work to do it at the moment.

A Quick note on SFTP jails

SFTP jails are uber useful and I've blogged about them before, but one crucial piece of information that I've been missing has been where it all started. This thread on the Ubuntu forums was the base for all of the work that I've done and it has made it super easy to setup jails here at work, which is a blessing due to the frequency with which we move files and make changes. I'm finding more and more that I owe an increasing amount of thanks to the people and posts on forums, both Ubuntu and others, so thanks.