Retooling

I haven't blogged for a long time, but I've decided that I'm going to try to write again, at least about technical stuff.

My plan was to blog about the projects I've been working on lately, the main one being the setup of the latest version of Kolab with the systems we already have at work, but I'll do that on the next days.

Today I'm just going to make a list of the tools I use on a daily basis and my plans to start using additional ones in the near future.

Shells, Terminals and Text Editors

I do almost all my work on Z Shell sessions running inside tmux; for terminal emulation I use gnome-terminal on X, VX ConnectBot on Android systems and iTerm2 on Mac OS X.

For text editing I've been using Vim for a long time (even on Mobile devices) and while I'm aware I don't know half of the things it can do, what I know is good enough for my day to day needs.

In the past I also used Emacs as a programming editor and my main tool to write HTML, SGML and XML, but since I haven't really needed an IDE for a long time and I mainly use Lightweight Markup Languages I haven't used it for a long time (I briefly tried to use Org mode, but for some reason I ended up leaving it).

Documentation formats and tools

Since a long time ago I've been an advocate of Lightweight Markup Languages; I started to use LaTeX and Lout, then moved to SGML/XML formats (LinuxDoc and DocBook) and finally moved to plain text based formats.

I started using Wiki formats (parsewiki) and soon moved to reStructuredText; I also use other markup languages like Markdown (for this blog, aka ikiwiki) and tried MultiMarkdown to replace reStructuredText for general use, but as I never liked Markdown syntax I didn't liked an extended version of it.

While I've been using ReStructuredText for a long time, I recently found Asciidoctor and the Asciidoc format and I guess I'll be using it instead of rst whenever I can (I still need to try the slide backends and conversions to ODT, but if that works I guess I'll write all my new documents using Asciidoc).

Programming languages

I'm not a developer, but I read and patch a lot of free software code written on a lot of different programming languages (I wouldn't be able to write whole programs on most of them, but thanks to Stack Overflow I'm usually able to fix what I need).

Anyway, I'm able to program in some languages; I write a lot of shell scripts and I go for Python and C when I need something more complicated.

On the near future I plan to read about javascript programming and nodejs (I'll probably need it at work) and I already started looking at Haskell (I guess it was time to learn about functional programming and after reading about it, it looks like haskell is the way to go for me).

Version Control

For a long time I've been a Subversion user, at least for my own projects, but seems that everything has moved to git now and I finally started to use it (I even opened a github account) and plan to move all my personal subversion repositories at home and at work to git, including the move of all my debian packages from svn-buildpackage to git-buildpackage.

Further Reading

With the previous plans in mind, I've started reading a couple of interesting books:

• Learn You a Haskell by Miran Lipovača (http://learnyouahaskell.com/)
• Pro Git written by Scott Chacon and Ben Straub (http://git-scm.com/book/en/v2)

Now I just need to get enough time to finish reading them ... ;)

Posted Sun 15 Feb 2015 10:24:41 CET

Static website generators

The last month I was supposed to work on a OpenStack related project, but for administrative reasons it has been delayed and I've tried to do small tasks to be able to finish them quickly and start the work on the main project when the issues get solved.

As the delay has been longer than expected last Wednesday I've realized than on the last weeks I did a lot of small system administration tasks:

• With a co-worker I started to work on a GNU/Linux version of our firewall based on Shorewall to handle the rules and conntrackd and keepalived to make it highly available (I had to stop my work on the Debian GNU/kFreeBSD based firewall a long time ago, and this summer the old firewalls' hardware started to fail, so a migration from Linux to Linux makes sense now, as it will be faster and a future migration will be simpler, as we will have a cleaner set of rules and better documentation),
• I installed and configured an instance of a web based File Exchange server (F*EX),
• I installed and configured an instance of a pastebin clone,
• I installed and configured an instance of ProFTPD that works only as a SFTP server using virtual users (without shell access),
• I also installed an instance of a web based event management system called indico that is being used to manage a conference and probably will be used for other events in the future,
• I installed and patched some plugins in our Trac servers,
• I tried a groupware system called SOGo that we will probable deploy in a week or two,
• And updated and fixed configurations of some other services,

With all the changes I did I noticed that I had to do something with our Intranet server; it is just a reverse proxy for a lot of different web services and its main page was one static HTML page with links to them, nothing else.

In the long term maybe we will replace it with something based on Drupal or Lifeay, but for now I just wanted something to be able to organize the links and provide some information about the services for the new users without having to write HTML (I really like Agile Documentation Tools that let me focus on the content and forget about the markup), and started to look at some of them.

My first idea was to use ikiwiki, as it has all the features I was looking for: I can use Markdown or reStructuredText to write the contents, the source pages are easily handled on a Version Control System, it supports the use of templates for the HTML, etc., but it seemed to me that using ikiwiki was like killing flies with a cannon (that's a Spanish say, I guess it's easy to understand it in English, ¿no?) and I decided to review other tools to build static web sites.

To make a long story short, I selected some tools that met my requirements and looked nice on their demo sites; after my first review I thought that Hyde was going to be my bet, as it uses technologies I'm already familiar with, but after trying it I saw that I was going to have a problem with documentation (the current Hyde version lacks it) and it was going to be more complicated that using ikiwiki.

Before giving up I decided to review simpler tools, just in case, and after looking some of them I ended up using poole, a simple python script (the source is just one file and it only requires python-markdown to work).

Before moving to the content I tried to adapt a couple of free themes to be used by the tool, but I didn't liked the result, so I went back to the plain style provided by the tool and added a logo and a background.

With that simple look and feel I started to work with the content, splitting it into eight markdown files and a python macro to include a file that has all the links used on the site.

While trying to make the main page look good I noticed how little I know about CSS, but using search engines I was able to build a two column block into the main page and publish the contents and with the help of some CSS enabled co-workers I changed the look and feel of the site in about 30 minutes.

In summary, if you want a really simple website, you know a little bit of python and don't want to spend much time learning how to use a website generator then Poole is a good option. If you want something more complex I still think that ikiwiki is a good option, but YMMV.

Posted Sat 01 Oct 2011 23:49:39 CEST

Encrypting a Debian GNU/Linux installation (take 3)

After my followup to the Tuesday post I've received some additional comments and I'm writing this entry to close the subject... ;)

One of the comments was from Gunnar to tell me that the followup setup was the same provided by the automatic partitioner of the Debian Installer since 2007.

I was unaware of that because until some weeks ago I never tried to install a system with encryption support and when I did it on my laptop I used the manual setup because I wanted to keep the MacOS X partitions.

Anyway my followup blog entry made sense anyway, as I just wanted to comment my thoughts about the advantages and disadvantages of each partitioning schema.

I also received a couple of messages proposing the use of three layers to keep the flexibility of the original setup and the simplicity of the second; the setup is as follows:

• Layer 1: use LVM on a physical volume,
• Layer 2: create a logical volume and format it as an encrypted volume,
• Layer 3: use LVM on top of the encrypted logical volume and put there the file systems that you want encrypted.

With the LVM at the lower level you get the advantages of my setup (mix encrypted and unencrypted partitions, the crypted volume can use multiple physical volumes, etc.) and the advantages of the second setup (only one key for all the encrypted file systems).

I believe that this setup is a little too much for a laptop, but can be a good option if you need encrypted file systems on a server.

Posted Thu 26 Feb 2009 08:30:16 CET

Encrypting a Debian GNU/Linux installation (followup)

Yesterday I received a mail message from a Debian user called Ekrem Erdem about my previous post, proposing a different partitioning schema that I found interesting.

The basic idea is to swap the order of the technologies, that is, use LVM on top of an encrypted partition instead of encrypting logical volumes.

I never thought about this schema because I always use LVM on servers and that is one of the fist things I setup (just after software RAID-1, if the machine has two hard drives); when I was evaluating how to setup my system for encryption I started with the LVM setup and never looked back.

The advantage of this setup is that there is only one pass phrase (the one used to unlock the encrypted partition, sda4 in my case), eliminating the need of derived keys (i. e. my swap setup) or key files (I use them to mount snapshots of the encrypted partition non interactively).

On the negative side I believe that this setup looses some flexibility:

• On my original model crypted and unencrypted partitions can coexist on the same volume group, while the new setup requires a different volume group for unencrypted volumes.

• If the user wants to have multiple partitions each one can use a different pass phrase or key file.

• If a logical volume is expanded through multiple physical volumes the new setup requires a key for each physical volume, while the original setup only needs one key.

Anyway if the plan is to encrypt all the file systems on a laptop the proposed setup is simpler and, IMHO, as safe as my configuration (remember that my keys are related).

I'm not going to change my setup now (it works great), but I'll probably try this one in the future if I need an encrypted setup on a different machine.

Posted Wed 25 Feb 2009 00:42:30 CET

Encrypting a Debian GNU/Linux installation on a MacBook

A couple of weeks ago I updated my Debian Sid setup on the MacBook to use disk encryption; this post is to document what I did for later reference.

The system was configured for dual booting Debian or Mac OS X using refit and grub2 as documented on the Debian Wiki; I don't use the Mac OS X system much, but I left it there to be able to test things and be able to answer questions of Mac OS X users when I have to.

The Debian installation was done using two primary partitions, one for swap (I used a partition to be able to suspend to disk without troubles) and an ext3 file system used as the root file system.

The plan was to use the Debian Installer to do the disk setup and recover the Sid installation from a backup once the encrypted setup was working OK.

Backup for later recovery

My first step was to install all the needed packages on the original system; basically I verified that I had the lvm2 and cryptsetup packages installed.

The second step was to backup the root file system; to do it I changed to run level 1 and copied the files to an external USB disk using rsync.

My third step was to boot into Mac OS X to reduce the space assigned to it; I had a lot of free space that I didn't plan to use with Mac OS X and I thought that this was the best occasion to reassign it to the Debian file system.

Encrypted Lenny installation

Now the machine was ready for the installer. As I formatted the system a couple of weeks ago I used a daily build of the Lenny Debian Installer, now that Lenny is out I would have used the official version.

I booted the installer and on the partition disk step I selected the manual method; I left sda1 and sda2 as they were (the Mac OS X installation uses them) and set up sda3 and sda4 as follows:

• sda3: 256 MB, use as ext3, mount point: /boot
• sda4: 86 GB, use as physical volume for LVM

Note that I decided to put /boot on a plain ext3 partition to be able to use grub2 as the boot loader (if we put the kernel on an LVM logical volume we need to use lilo as the boot loader).

Once sda4 was adjusted as LVM I entered on the LVM setup and created a LVM Volume Group (VG) with the name debian, using sda4 as the physical volume.

Once the VG was defined I created a couple of Logical Volumes (LV):

• root: 82 GB
• swap: 2 GB

I left some space unallocated to be able to create LVM snapshots (I use them to do backups, I'll post about it on the next days).

Once the LV were ready I finished with the LVM setup and went back to the partitioner to configure the Logical Volumes:

• debian-root: use as physicals volume for encryption
• debian-swap: use as pascal volume for encryption, encryption key: random

Once both encrypted volumes were ready I entered on the Configure the encrypted volumes menu and the installer formatted the volumes for encryption and asked for the debian-root pass phrase.

Back on the main partitioning menu I set up the debian-root_crypt encrypted volume:

• debian-root_crypt: use as ext3, mount point: /.

I didn't need to touch the debian-swap_crypt, it was configured automatically as swap because I choose a random encryption key.

At this point I was finished with the partitioning; to finish I installed a minimal system and rebooted to try the system.

As I had changed the disk layout I had to re-sync the partition tables from refit; once that was done I was able to boot from the newly installed system.

Setting up suspend to disk

I was using s2disk to suspend the system; to test if it still worked with the new setup I installed the uswsusp package and adjusted the resume device on the /etc/uswsusp.conf to /dev/mapper/debian-swap_crypt.

After my first try I noticed that the resume step failed with the encrypted swap partition because it was using a random key, which means that the swap contents are unrecoverable after a reboot.

Looking at the cryptsetup documentation I found that the solution was to use a derived key for the swap partition instead of a random one.

The command sequence was as follows:

# disable swap
swapoff -a
# close encrypted volume
cryptsetup luksClose debian-swap_crypt
# change the swap partition setup on the /etc/crypttab file
sed -e -i 's%^debian-swap.*%debian-swap_crypt /dev/mapper/debian-swap debian-root_crypt cipher=aes-cbc-essiv:sha256,size=256,swap,hash=sha256,keyscript=/lib/cryptsetup/scripts/decrypt_derived,swap%' /etc/crypttab
# open the encrypted volumes with the new setup
/etc/init.d/cryptdisks start
# enable swap
swapon -a
# update the initrd image
update-initramfs -u

After executing all those commands the suspend to disk system worked as expected.

Recovering the original system

If I were going to reinstall the system completely I would have finished here, but in my case I wanted to recover my original system setup (except the minimal changes required to use the encrypted passions, of course).

To recover my old installation I backed up some files (/etc/fstab, /etc/crypttab, /etc/uswsusp.conf and the current /boot contents to be able to boot in case of failure with my old kernel) from the current installation, after that I recovered all the files from the initial backup (except the ones just saved) using rsync again and regenerated the initrd images of my old kernels:

update-initramfs -u -k all

After that I rebooted and everything worked as on my original installation (except for the disk encryption, of course).

Posted Sun 22 Feb 2009 00:11:52 CET