Parsing Fail2ban logs

Fail2ban is a ssh (and other programs) log scanner that can block connections from IP addresses that enter incorrect credentials too many times. It’s very handy if you have an OpenSSH server connected to the Internet, as it will likely get heavy traffic from people or bots trying log in. Once you have Fail2ban installed, it’s pretty interesting to check out the logs. You can see the number of times each IP address has been blocked by using this command:

awk '($(NF-1) = /Ban/){print $NF}' /var/log/fail2ban.log | sort | uniq -c | sort -nr

That will get you the recent bans, but Fail2ban rotates the logs (every 24 hours, I think), so here is a command that will get you a more complete list:

sudo cp /var/log/fail2ban.log* .; sudo gunzip fail2ban.log.*.gz;\
for l in $(ls fail2ban.log*); do cat $l >> fail2bantotals; done;\
sudo awk '($(NF-1) = /Ban/){print $NF}' fail2bantots | sort | uniq -c | sort -nr

That command copies all the Fail2ban logs to your current directory, unzips the compressed ones, dumps the text into a common file (fail2bantotals) then parses the collection. You should get something like this:

54 75.127.65.186
36 178.137.194.52
23 94.77.193.239
23 210.107.122.209
21 69.172.133.250
 5 61.148.75.130
 3 74.122.227.71
 3 61.43.190.165
 3 61.155.178.242
 2 91.205.189.15
 2 211.169.65.3
 2 200.160.6.5
 1 101.79.68.141

Decrease energy usage with cpufrequtils

One of the benefits of using Linux is how easy it is to decrease your computer’s power consumption. There are a few ways to do this, but one of my favorites is to use cpufrequtils and the cpufreq-set command. With this utility, you can choose which frequency your CPU runs at. Further, you can choose the frequency for each core of your CPU, as most modern processors have multiple cores.

cpufreq-set comes with some default governors, or settings that choose the frequency for you. Although you can specify exactly what frequency your processor runs at (e.g. 1407 MHz), I prefer to let a governor choose for me. If you’re looking to save energy and lower your power consumption, you will want to check out the powersave governor. First, make sure you have cpufrequtils installed (sudo apt-get install cpufrequtils for Debian and Debian derivatives, like Ubuntu and Mint). Then, enter the following command in a terminal to make use of that governor:

sudo cpufreq-set -c 0 -g powersave

This will tell your computer to use the powersave governor on the first core (0) of your processor. If you have multiple cores, you will want to do this for each core:

sudo cpufreq-set -c 1 -g powersave
sudo cpufreq-set -c 2 -g powersave

The -c flag specifies the core to change. Using cpufreq-set from cpufrequtils can make your computer more energy efficient while running linux and even give your Linux laptop longer battery life.

A note to remember: using the powersave governor can make your computer slower at doing certain tasks, so if you need to do some CPU-intensive tasks then you might want to switch to the ondemand governor for those:

sudo cpufreq-set -c 0 -g ondemand

for each core.

I also assigned an alias for each of these commands in my bash config:

alias powersave='sudo cpufreq-set -c 0 -g powersave && sudo cpufreq-set -c 1 -g powersave'

Then a simple powersave command is available.

Parsing XML with PHP

Parsing XML using PHP is pretty easy. PHP has an extension called SimpleXML that “provides a very simple and easily usable toolset to convert XML to an object that can be processed with normal property selectors and array iterators.” Check out the example below to see how it works. The PHP code is first with the sample XML below it.

<?php
    $xml = simplexml_load_file("http://news.ycombinator.com/rss");

    $items = $xml->channel->item;

    for($i = 0; $i < count($items); $i++) { 
        $title = $items[$i]->title;
        $url = $items[$i]->link;
        echo "<a href='" . $url . "'>" . $title . "</a>";
    }
?>

<rss version="2.0">
    <channel>
        <title>Hacker News</title>
        <link>http://news.ycombinator.com/</link>
        <description>Links for the intellectually curious, ranked by readers.</description>
        <item>
            <title>Sphinxtr: Creating a Portable PhD Thesis</title>
            <link>http://jterrace.github.com/sphinxtr/singlehtml/index.html</link>
            <comments>http://news.ycombinator.com/item?id=4609769</comments>
            <description>
                <![CDATA[<a href="http://news.ycombinator.com/item?id=4609769">Comments</a>]]>
            </description>
            </item>
    </channel>
</rss>

Customizing the Openbox Menu

There are two ways you can make changes to the openbox menu. You can edit the menu.xml file, which is usually located with the other openbox configuration files (in ~/.config/openbox/ on Ubuntu 12.04).

There’s an easier, GUI-based way – using a program called obmenu. Once you have obmenu installed (sudo apt-get install obmenu), simply type obmenu in a terminal. A window like will come up and allow you to change the menu that pops up with a right click on the openbox desktop.

You can find some pretty handy openbox menu.xml files on the Internet, also. I personally like the one that comes with a GNU/Linux distribution called crunchbang: Default #!crunchbang openbox menu.xml

Create a passwordless ssh key

If you get tired of typing in a password when you connect to a remote machine with ssh, you can create an ssh key without a password to ease the process. Although this is not as secure as a key with a password, it is quicker. To create the key, type

ssh-keygen

This will create two keys, a private and a public key. Next you should copy the public key on the remote machine.

ssh-copy-id -i ~/.ssh/id_rsa.pub remotehost.com

Now you will be able to connect passwordlessly(!) to the machine by issuing this command:

ssh youruser@remotehost.com

where youruser is your username at the remote host.

Just remember that this does remove a security barrier (the entering of a password) for anyone that can gain access to your local machine.

Using x11vnc with Ubuntu

There might come a time when you want to connect to a remote ubuntu machine, whether for your own use or to assist someone else with a problem. An easy (relatively) way to do this is to use x11vnc as a VNC server. I’ll show you how to get it set up for Ubuntu (I’m using Ubuntu 12.04 for both machines, the remote one and the one I am on).

To get started, install x11vnc on the remote machine. You can do this via the terminal emulator and ssh:

sudo apt-get install x11vnc

Then you should set up a password:

x11vnc --storepasswd

Next you can start the VNC server:

x11vnc -geometry 1366x768 -rfbauth .vnc/passwd

The previous command started the server with a display resolution of 1366x768 (my local machine’s native resolution) and used the password you set by referencing the file passwd in the .vnc directory.

To connect to the machine, you will need to know the remote machine’s IP address. You can find this by searching for “my ip” on Google from the remote machine. It gets a little more complicated in that you need the VNC (port 5900) traffic going to that remote machine. This may involve changing a setting on the remote network’s router. I use Remmina to connect to remote machines using VNC.

Host multiple domains on one server with Apache

Apache allows you to host multiple domain names from a single IP address using Virtual Hosts. I’ll show you how to edit Apaches sites-enabled file to do this. First, open the 000-default file in /etc/apache2/sites-enabled/. It should look like this:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost

    DocumentRoot /var/www
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog ${APACHE_LOG_DIR}/access.log combined

    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>

</VirtualHost>

We will want to add the following section to the bottom:

<VirtualHost *:80>
DocumentRoot /var/www/directoryWithSiteFiles
ServerName domainname.com
<Directory />
    Options FollowSymLinks
    AllowOverride None
</Directory>
<Directory /var/www/directoryWithSiteFiles>
    Options Indexes FollowSymLinks MultiViews ExecCGI Includes
    AllowOverride None
                Order allow,deny
                Allow from all
        </Directory>
# Other directives here
</VirtualHost>

replacing domain.com with your domain name and directoryWithSiteFiles with the directory that holds the files you want hosted at that domain. Notice that this is for the naked domain domain.com. To also host www.domain.com, you will need another section added on like this:

<VirtualHost *:80>
DocumentRoot /var/www/directoryWithSiteFiles
ServerName www.domainname.com
<Directory />
    Options FollowSymLinks
    AllowOverride None
</Directory>
<Directory /var/www/directoryWithSiteFiles>
    Options Indexes FollowSymLinks MultiViews ExecCGI Includes
    AllowOverride None
                Order allow,deny
                Allow from all
        </Directory>
# Other directives here
</VirtualHost>

You can do this for multiple sites (I have three on my server right now). Remember: you will have to change your DNS server to point to the IP of this server, as well.

MySQL fails to start

I recently ran into an issue where the MySQL database service would not start on my server. After searching around a bit (and being led astray a few times), I found the culprit: not enough RAM.

It turns out that my set up (Ubuntu 12.04 with 256 MB RAM) did not have a enough memory to run a wordpress install and the MySQL database required for it. These specs are fairly common for VPS’s like mine and this should help you realize what is going wrong.

If you get ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) when you run sudo mysql start or start: Job failed to start when you run sudo /etc/init.d/mysql start then you should check the amount of RAM your machine/instance has available. You just might be running low.

As for solving this issue, you can try killing unneeded services. My instance had plymouthd running, a daemon for the Plymouth graphical boot that comes with Ubuntu. So, with a headless install, I did not need this and killed it. You can also temporarily kill services (like Apache) just to get your MySQL off the ground, but beware: you might run out of memory again soon.

Also, VPS’s like mine are even more vulnerable to this because many do not have any swap space allotted. So once your RAM is exhausted, you’re in trouble.

Confine SSH user to home directory with rbash

You can use rbash to limit a user to their own home directory (they cannot cd to other directories, or write to them). To do this, you will want to edit /etc/passwd and change the user’s shell:

test:x:1001:1001:,,,:/home/test:/bin/bash

should change to

test:x:1001:1001:,,,:/home/test:/bin/rbash

I did this on Ubuntu 12.04, so if your system varies from that then you might have additional steps.