Containerizing my transcript search app
Until recently, my transcript search web app was running (at https://transcripts.kaashif.co.uk, check it out) in a tmux session, with a PostgreSQL server running on the same machine in the usual way, as a daemon.
The web app knows nothing about its dependency on the database, this information is not recorded anywhere except in the code itself. And the database knows nothing about the web app. This isn't a huge problem except the database has a config of its own which isn't recorded anywhere in the source repo. If you try to get the web app to work with a misconfigured database, it won't work, of course.
Wouldn't it be nice if all of that configuration were in one place? And if the services all restarted themselves if they failed? And if you could migrate the entire blob of interconnected web apps and databases to a different machine with a single command?
That's where Docker comes in!
HP PA-RISC Assembly Crash Course
Since I have access to a machine that has the PA-RISC architecture, I thought I'd compile some test programs and see what sort of assembly code produced. Some highlights:
- A neat way to manage the stack pointer (and one surprise)
- Every instruction seems to be shorthand for
- Completers - a weird way of giving switches to your instructions
PA-RISC is considerably less popular than x86, MIPS, PowerPC, even SPARC. And being a RISC architecture means that humans hardly ever wrote assembly for it themselves. Most of the time, programmers probably never even gave (past tense since PA-RISC is dead) their binaries a second glance. Or really even any kind of look.
Well, that's about to change! The first program we'll look at is, of course, hello world.
Reviving a HP PA-RISC server
A while ago, I got my hands on a beast of a machine, a 7U HP L3000 (rp5470) PA-RISC server. These were released in the year 2000 and came with up to 16GB (whoa) of RAM and up to 4 CPUs.
The best site for information on PA-RISC machines is, no doubt, OpenPA.net, and they have a fantastic page on my machine.
This is the story of how I managed to install Gentoo GNU/Linux on this classic UNIX server.
Using PostgreSQL to search transcripts
Remember my transcript search engine, https://transcripts.kaashif.co.uk?
I'm not a database expert, but even I realised that spending hours and hours trying to optimise my homebrew database transcript search engine was a waste of time. For no reason at all other than to try it out, I went with ElasticSearch. The astute reader will notice that ElasticSearch is meant for "big" data. My collection of transcripts tops out at few dozen megabytes at most - this is most certainly not big (or even medium-sized) data, really.
So after getting some real-world experience with SQL databases (at a real company) and taking an in-depth database algorithms course at university, I decided to convert my web app to use a PostgreSQL database to store the transcripts, searching with bog-standard SQL queries.
There were a couple of neat tricks I used to speed things up, too.
Register windows: a cool feature of SPARC
Everyone's studied x86 assembly (just objdump any program on your PC...) and maybe even some ARM or MIPS in a class somewhere, but there are a few features that exist in some CPUs that don't exist at all in any of these designs.
I'm talking about register windows! When you call a function on SPARC, the new function just magically gets its own registers neatly separated into input registers, output registers and local registers. You're allowed to mess up your local registers as much as you want and the CPU does all of the saving and swapping for you.
No more weird arbitrary calling conventions about r10 and r11 being caller-saved, rax being return, rqb being Cthulhu-saved, rpqwuqew being quantum entangled with r554 on Tuesdays...
Porting OpenJK to sparc64
It's a little known fact that there is actually no way in C or C++ to do an unaligned access without invoking undefined behaviour. It's true! Read it yourself here:
A pointer to an object type may be converted to a pointer to a different object type. If the resulting pointer is not correctly aligned [...] for the referenced type, the behavior is undefined.
C11 (n1570) 22.214.171.124 p7
Sadly, the authors of many programs ignore this and rely on it working. Which it does, on x86, with little performance impact in most cases. On some architectures, like MIPS and PowerPC, unaligned access instructions exist but are slow. But on SPARC...unaligned access is impossible and leads to this:
$ openjk.sparc64 Bus error (core dumped)
Solving these issues with OpenJK is very difficult, especially considering Jedi Knight was never meant to run on SPARC (or indeed OpenBSD, but that's less of an issue).
Playing with LDoms, OpenBSD and Solaris
A few weeks ago, I got my hands on a Sun T2000 server. It's got an UltraSparc T1 CPU, 32 threads, 32 GB of memory, a Sun XVR-300 GPU and what sounds like a huge jet engine mounted at the front.
It's a great machine (although maybe not as a workstation...), and there are a few things unique to SPARC that I've really been looking forward to playing around with. Mostly LDoms (logical domains - Sun's virtualization technology), OpenBSD on a beefy sparc64 (compared to my older UltraSparcs anyway) and Solaris (just as a curiosity).
Using vmm(4) to target old OpenBSD releases
This server (the very one you are reading this post on), at the time of writing this post, runs OpenBSD 6.1-stable. It's fully patched and updated and everything, so it's a perfectly fine OS to run. But the VPS has limited memory and disk space, and the CPU isn't very fast, so compiling large projects on it, especially Haskell ones, is impractical.
This post describes a way to build fully-functional, dynamically linked (so you get all those security updates, super important for a public-facing web service), native Haskell binaries with cabal-install, Stack and...OpenBSD's (kind of) new native hypervisor, vmm.
Reviving a Sun Ultra 5 workstation
I recently got an old Sun Ultra 5 working. It wasn't too difficult, but I needed to dig up a few old serial cables...
It already had SunOS 5.8 installed, but I put OpenBSD 6.1 on it, since I need a modern OS to actually do anything with it.
Sorting a ton of mail
Migrating mail servers is a tricky business, especially when one server doesn't have IMAP set up. The easiest way is to download all the mail and reupload it to the new mail server.
This seems simple enough, but I ran into problems. After all, I was going from an IMAP server somewhere to a maildir (no IMAP sync tool supports mbox for some reason) to an mbox through procmail to a directory of mboxes. Not trivial.
Moving to my own email server
There I was, a loyal user of http://mail.zoho.com, when I decided to download all of my emails. For archival purposes, you know.
So I fired up mbsync, set everything up, let 'er rip, but after only about 10,000 emails downloaded, I got this error:
IMAP command 'AUTHENTICATE PLAIN <authdata>' returned an error: NO [ALERT] Your account is currently not accessible via IMAP due to excessive usage. Kindly try after some time. *** IMAP ALERT *** Your account is currently not accessible via IMAP due to excessive usage. Kindly try after some time.
What sort of...anyway, this was unacceptable, so I decided to set up my web server as a mail server.
Before I get my Sun Ultra 5 working and can write something about that, I thought I'd go through all the hardware I'm using right now and the OSes I'm running on them. Spoilers: it's all OpenBSD and Debian.
Cutting down memory usage of a Haskell web app
You may have heard of the Star Trek script search tool at http://scriptsearch.dxdy.name. I'm writing a similar thing for Stargate. The difference is, of course, is that my tool will be running on some crappy Amazon AWS t2.nano with no RAM.
The first prototype was written in Python, but the parsing code was always written in Haskell (Parsec is great). I decided to move everything into Haskell so there wouldn't be this redundancy of parsing in Haskell then serializing to disk then reading it from Python...one codebase for everything would be much simpler.
I wrote up a quick Haskell version, but there was one small problem when I tried to use it:
$ ./stargate-search Setting phasers to stun... (port 5000) (ctrl-c to quit) Killed
That's right, the OOM killer had to step in and put my app down like the sloppy wasteful piece of junk it was.
How could I fix this?
Trying out DragonflyBSD
So I was setting up a laptop I had just picked up, when it came to deciding what OS to install on it. Obviously, I'd probably end up installing some Linux and maybe also OpenBSD (hard drives are huge nowadays, I could fit hundreds of OSs on there). While I had tried out FreeBSD and NetBSD, DragonflyBSD had never been on my radar.
It still isn't, really, but I thought I'd try it out on an old laptop, just to see what it was like. It went pretty well, but there were a few oddities and one or two kind of weird design choices.
Playing around with distcc
Today, I decided to install Gentoo on a spare machine I had lying
around, since I was bored. Obviously, the first issue I ran into was
emerge x11-base/xorg-server was taking a really long time to
run since the Xorg server is a pretty bloated program. Then
firefox was taking forever too.
One solution (for Firefox anyway) was to use the provided binary
packages, this meant
firefox-bin for Firefox. But this means I
abandon all of the nice features (for me, that means USE flags) that
Gentoo offers. If I'm going to download a load of binaries that I
can't customize, why not just install Debian?
So the solution is to speed up compilation. That means putting more
CPU cores to work. But my poor old ThinkPad only has 2 cores! This is
distcc comes in.
Backing up PGP private keys
There are hundreds of blog posts about backing up your PGP keys around on the internet. Most of them just say something like: put a passphrase on it, keep it on a USB stick, a CD, a floppy disk, or something like that. These are all very useful ways to back important stuff up - in fact, I just restored a backup of my GPG keys from a CD after deleting them by accident. These mediums are, however, not going to survive for many decades like paper can. And if you store to a writeable medium like a rewritable CD, DVD, USB stick or floppy, there is still the danger of you trying to restore then accidentally deleting your backup. Or, more likely, you write over it without realising many months later that you wrote a Linux distro or movie to that DVD that had your only PGP key backup.
Making a list of the websites of people on nixers.net
I wanted to make a list of the websites of the people on the website http://nixers.net, and I decided to solve it not by asking people to tell me what their sites were called, but by scraping the forum.
I didn't scrape the whole forum, I just scraped one topic on the forum that I created a few years ago: https://nixers.net/showthread.php?tid=1547
Sharing /home between OpenBSD and Debian
Some people reading this might be thinking: "hey, it's really easy to do this, why is he writing an article on this?". You are partially right, this should be really easy, but there are some weird things that happened while I set this up that I feel should have been written down somewhere, so my fear that I was completely borking my system would have been assuaged.
Converting my blog to frog
What is frog?
Frog is a static website generator written in Racket. It does the same sort of thing as Jekyll, Hakyll and other software like that, some of which I've used in the past.
Hacking StumpWM with Common Lisp
Before a few weeks ago, I was always one of those people who said that Lisp isn't useful, it's not type-safe, it's not pure, Haskell is better etc etc ad nauseam. All of that may be true for writing some sorts of programs, but Lisp (well, Common Lisp anyway) provides something a lot more pervasive.
What does pervasive mean? Well, right now, I'm controlling my window
How to get a list of processes on OpenBSD (in C)
Is it portable?
First off, the information in this post definitely doesn't apply to Linux (as it has a completely different way of doing things) and may or may not apply to other BSDs (I see that NetBSD and FreeBSD both have similar, maybe identical, kvm(3) interfaces). There certainly isn't anything in POSIX to make this standard. The only real reason
Iodine is cool
I know that sometimes, I've bene stuck in an airport or in a coffee shop without internet. That's annoying in and of itself, but it's even more annoying when there's a WiFi hotspot nearby, but it requires you to pay £4/hour or something crazy like that.
You can still connect to the network, it's just that whatever URL you
Rainbow brackets in Emacs
You know something that really annoys me? When I'm writing some Racket, Clojure, or any other Lispy language, and my editor won't cooperate. Emacs is far, far, better than most other editors for this sort of thing, mostly due to paredit-mode and SLIME (and geiser-mode, and clojure-mode, and evil-mode, and...), but there's still one problem I hadn't solved until recently.
OpenShift vs Heroku: Haskell
There are a few Platform as a Service (PaaS) services out there, and the most famous is probably Heroku. I know I've had people come up to me and suggest using Heroku for the next big thing they're planning. There is a problem with Heroku: it's not free (libre). That didn't stop me from at least trying it out to see what all the fuss was about.
Quaternions, spinors and rotations
Earlier, I was trying to find something I could talk about at my school's maths society. It had to be something exciting, useful, or at least beautiful in some way. I really wanted to do something on quaternions and vectors, because it seemed fun. The problem came when I realised I had to do something more substantial than stand there and explain something that boring. Then I saw this quote:
ShareLaTeX on OpenBSD
The other day, I was trying to access http://sharelatex.com at school, and it didn't really work, probably due to a combination of Internet Explorer and possibly an overzealous filter that could have been blocking something. That's what I thought, anyway, until I tried it on Chrome and it still didn't work. Odd. The best solution was obviously to set up my own ShareLaTeX instance on my server.
Switching to Mercurial
UPDATE: I now use Mercurial on the client side (i.e. everything I do
hg) and Git on the server-side. It just makes it
easier to mirror to Gitorious, GitLab, etc. You can view all the
repos at http://kaashif.co.uk/code.
Earlier in the year, I was getting curious about version control
Pkgsrc on Slackware
Back in the day, I used to use Slackware. It was the best distro around and all the cool kids used it. Nowadays, it's rather different: a much lower proportion of people use Slackware. Despite the efforts of Eric and Patrick (and whoever else), Slackware isn't really all that popular. It's still a solid distro, though. There is one problem
Earlier today, I was discussing operating systems and came onto the subject of ease of installation. Which OS had the easiest installer? The obvious answers would tend towards OSs with GUI installers, but is that really easy? Sure, it could be familiar, but there's a lot more you have to do with GUI installers compared to, say, OpenBSD's
I've heard a lot of things about Racket (well really, things about many different Lisp dialects), and most of them were good. Recently, I decided to try to decide between Haskell and a Lisp once and for all. I wanted to go for a Lisp-1, since they keep functions and values in the same namespace, which is how it should be. Eventually, after
Emacs is great
I've seen many completely stupid articles where people furiously circlejerk over how Vim is the best and nothing will ever come close, but it's rare that I see anyone write much about Emacs, probably because fewer people use it (or maybe Emacs attracts a different demographic). It's rare I see an article like
Recently, I've been trying to understand the ins and outs of CVS in order to be able to contribute to OpenBSD without messing up anything. I have sent a few patches to ports@, but anything complex was beyond my abilities until recently.
Recently, I've been trying to get away from pre-packaged file sharing solutions (e.g. FreeNAS) and trying to set up the services they provide from scratch. While I obviously won't be able to write a web GUI or create a whole distro, that simply isn't necessary. What is necessary is setting up a file share and appropriate read/write permissions.
Haskell isn't difficult
How I share a file, simply
Earlier, I saw this article claiming to describe how to share a file "simply" by running Python's web server module (with either Python 2 or 3). While that may be easy, it's not simple, and certainly not fast.
The inspiration for this came from this blog post, where the author describes how he uses his computer. While he does use CRUX, a GNU/Linux distro and I use OpenBSD, our workflows are actually surprisingly similar, which can, in part, be attributed to the
Installing OpenBSD on a T61
I'm sure lots of people (dozens, perhaps) have installed OpenBSD on ThinkPad T61s of some description, but with the recent release of OpenBSD 5.5, lots of documentation has become (or already was, and now is even more so) obsolete, like this article
On all of my computers, I like being efficient. That means eliminating everything which uses all that precious CPU time and using applications which are very customisable and configurable. These sorts of applications tend to be text-based, which is, in my eyes, a good thing, since they'll show you the information you need with a minimum of
Ideas for a project
There is always a lot of buzz around the idea of "learning to program". While I think it's very important that children learn logical thinking and problem solving, I also realise that the majority of children, and people in general, would probably not benefit from a very language specific, rote learning based, generally old-fashioned approach to
How this blog works
UPDATE: I now use Hakyll. See http://kaashif.co.uk/about for more. Also, 100% of the information in this blog post is now wrong or outdated.
When I decided I wanted to write a blog, I had to come up with some way of writing posts (in a markup format which isn't HTML), and serving them somehow.
How not to run a website
After a few months of running a website, there are a few things I have realised about how I ran my server when it was first delivered, and how I run it now. The changes have been, for the most part, for the better. Needless to say, when I first started out, I was clueless, overeager and far too ambitious with my plans for "the next Facebook" or something
Creating a GNU/Linux distro
UPDATE: The project died, it went nowhere.
On nixers.net, the IRC channel of which I spend a bit of time in, there has been a bit of a stir as the community tried to decide on a project to commit to. The idea of creating a distro of some OS came up. A few people wanted a BSD-based distro, but it was decided that the Linux kernel was
When I first started programming, I barely had any idea of what constituted a good text editor, or why I'd want to use some old, texty editor from the 90s which didn't even have most of the features I took for granted in the IDE I was using at the time. Maybe this had something to do with one of my first languages being Java, which is widely considered an IDE language, but I went through the
Over the years, I've used a few different OSes and desktop environments, and the one I use currently is portable to many operating systems, mostly due to the efforts of the writers of i3 over at i3wm.org, but also the standards-compliance of POSIX, meaning that my shell scripts (which you can find here) work on all
Functors in Haskell
Whenever you hear something about Haskell, chances are it sounds arcane and involves lots of complicated and intimidating mathematical language. Well, the truth is that all this talk of 'endofunctors' and 'monoids' is really unnecessary, if the concept of functors is explained using a simple analogy.
Updating DNS records
When running a website on a residential connection, a problem one might run into is the dynamic IP address usually assigned by one's ISP. There are a few dynamic DNS services which basically let you have a subdomain (e.g. mydomain.example.com) and let you update it to point to your IP address whenever it changes. At one time, your IP might be 10.0.0.1, and your domain correctly
What's a monoid?
If you have spent time on programming or technology boards like /g/ or /r/programming, chances are you might have heard the word "monad" thrown around a lot. You may have even heard the oft misquoted phrase "A monad is a monoid in the class of endofunctors" intended to be a joke, or to scare programmers away from scary functional languages like Haskell. The truth is that monads aren't
Writing Unix manual pages
There are a few very important things that everyone involved in software (particularly free software) can do to help out. The most important is to file detailed and helpful bug reports, so the developers working on your favourite program can get the problem fixed. Since it is not very hard to write a bug report, and projects generally have their own bug report guidelines, I won't
Reviving an old ThinkPad
While I did have some old hardware lying around, I had never committed to actually getting that hardware usable. By that, I mean I had never tried to browse the web, read emails and that sort of day-to-day stuff on anything older than a few years. To see if it were really possible, I decided to buy an old ThinkPad (a 760EL from 1995) and see if I could get it working. Before I
Using GNU Stow
stow is a cool little Perl script which basically just creates and deletes
symlinks. That sounds pointless, but let me explain with an example. Let's say
you want to install a program using the usual
make install, which probably
installs into /usr/local, which means it's separated from the rest of your
system, which is managed with a "real" package manager. Unless you're using a
For a few years now, I have been using Vim to edit config files, program in C, Python, even Lisp (people apparently think that Vim isn't the best for programming in Lisp). This isn't because I took a side in the so-called "editor wars", it's just because it came preinstalled on the first GNU/Linux system I used, Debian. Over time,
Introduction to C
This tutorial is designed for those who have programmed before, perhaps in a higher level language like Python or Ruby. It's not too hard to understand for those who are completely inexperienced, but some knowledge of functions, data structures and pointers might help. Most of the low-level stuff will be new to high level programmers, however.
Email is the Future
Often, people look at me oddly when I suggest that they email me something. "Why can't I just send it to you on Facebook or Skype?" they say. Well, it doesn't have to be those two media of communication, but it's usually something like that. When I say often, I also mean that this has only happened on two occasions, so bear that in mind as I make things up about the types of people
How to Wipe a Disk
This article is not only about disk wiping, it will hopefully teach you something about using some GNU command line tools . This tutorial was written on my ThinkPad, which runs Debian, so the output should be pretty similar to what you'd get on Ubuntu, Mint or any other Debian- or Ubuntu-based systems. Basically, if you're using something
Why I Use FreeBSD
Installing packages from source
Recently, I had SSHed into one of Debian Stable virtual machines I was using as a file server. The main services I was running were FTP, an HTTP server with a directory listing and a Samba server, with shares set up for a few users on my
What is LaTeX?
You probably have not heard of LaTeX before now. If you have, then it is likely that you have no idea what LaTeX is, save for a vague feeling that it relates to documents in some way. By the end of this short post, you will not only know what LaTeX is, but be able to understand why people use it, and what its advantages are. While you probably won't switch to LaTeX immediately, you might
Imagine you're a person on some sort of device, using the internet. You see all of these websites and what do you ask? "How can I set up a web server?", of course. If you did not ask that question, then this guide is not for you. Anyway, down to business. You will need:
How to use GPG
Why am I writing this?
I have looked up "how to use gpg" so many times, on so many websites, and have found every guide to be focused on something I don't use or worded in such a way that I get confused and revoke all of my keys (that hasn't actually happened...yet). I thought I'd whip up a quick guide