kaashif's blog: Computers, with some mathematics on the sideurn:https-kaashif-co-uk:-index.html2024-01-03T00:00:00Zbinfmt_misc: The magic behind Linux/Windows interopurn:https-kaashif-co-uk:-2024-01-03-binfmt-misc-the-magic-behind-linux-windows-interop2024-01-03T00:00:00Z2024-01-03T00:00:00ZKaashif Hymabaccus
<p>I was running something in WSL, as you do, then I thought about it for a
second. When I'm doing this in WSL:</p>
<pre><code>$ clip.exe < file.txt
</code></pre>
<p>How does that actually work? It turns out this is done using <code>/init</code> which is
two things:</p>
<ol>
<li><p>PID 1, it's the init system, the parent of all processes in WSL.</p></li>
<li><p>An "interpreter" for Windows executables. When you run <code>clip.exe</code>, that's
the actual Windows binary you're running directly. This works via the
<code>binfmt_misc</code> mechanism of Linux, which allows you to register runners for any
binary with specific magic bytes.</p></li>
</ol>
<p><code>/init</code> is a bit hard to get at since it's a closed source component of WSL. We
can get some idea of how it might work by looking at (1) <a href="https://learn.microsoft.com/en-us/archive/blogs/wsl/windows-and-ubuntu-interoperability">a Microsoft blog post
describing how this works at a high
level</a>
and (2) <a href="https://github.com/xilun/cbwin">cbwin, an open source implementation of
this</a>.</p>
<p>We can also do fun things, like make Java jars directly executable without
needing to run them with <code>java -jar</code>. But beware - if you have "fully
executable" jars with scripts embedded at the start (like <a href="https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/deployment-install.html">the ones Spring Boot
makes</a>),
<code>binfmt_misc</code> can't possibly be able to tell that they're jars.</p>
<p>But <code>java -jar</code> still works on them! Weird. Here are the questions we want to
answer:</p>
<ul>
<li><p>What happens when you run a "normal" Linux executable? What about a shell script?</p></li>
<li><p>How does Linux tell that clip.exe is a Windows executable, and how does it
run from inside Linux?</p></li>
<li><p>How can Java tell that a shell script with some binary junk at the bottom is
really a jar, but the Linux kernel (via <code>binfmt_misc</code>) can't?!</p></li>
</ul>
<p>Answers are below.</p>
<!--more-->
<h1>What happens when you run a normal executable?</h1>
<p>Let's take <code>cat</code> as an example. From inside your shell, you execute:</p>
<pre><code>$ cat file.txt
</code></pre>
<p>Your shell will probably find <code>cat</code> in the <code>PATH</code> and use the
<a href="https://man7.org/linux/man-pages/man2/execve.2.html">execve</a> system call to
execute that file.</p>
<p>This is not mysterious at all. You can see the source code of <code>execve</code> here:
<a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/exec.c?id=HEAD#n2030">https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/exec.c?id=HEAD#n2030</a>.</p>
<p>This blog post isn't supposed to be a deep dive into <code>execve</code>, the point is that
<code>execve</code> executes executables.</p>
<h1>What about a shell script?</h1>
<p>Believe it or not, also <code>execve</code>! <code>execve</code> reads the first two bytes of the
given file, if they're <code>#!</code>, then the file gets executed in the way we're all
familiar with.</p>
<p>If <code>file.txt</code> is given to <code>execve</code> with these contents:</p>
<pre><code>#!/bin/sh
echo Hello
</code></pre>
<p>then <code>execve</code> will run <code>/bin/sh file.txt</code>, and we go back to the first case: a
normal executable.</p>
<p>So far, so good, everyone should be familiar with this. The interesting part
comes next.</p>
<h1>What is binfmt_misc?</h1>
<p><code>binfmt_misc</code> is documented very well here:
<a href="https://docs.kernel.org/admin-guide/binfmt-misc.html">https://docs.kernel.org/admin-guide/binfmt-misc.html</a>. At a high level,
<code>binfmt_misc</code> is a feature of the Linux kernel that allows you to specify a
rule matching either a filename suffix or magic bytes at an offset in the file,
and an executable to use to run that file, similar to how a shell script is
run.</p>
<p>For example, to match the <code>.txt</code> extension and cat the text file when "run",
you could run:</p>
<pre><code>$ sudo sh -c 'echo ":cattxt:E::txt::/bin/cat:" > /proc/sys/fs/binfmt_misc/register'
$ vim file.txt
$ chmod +x file.txt
$ ./file.txt
this is my file
it has content
hello
</code></pre>
<p>This isn't very useful. The next part is more interesting.</p>
<h1>How does WSL tell clip.exe is a Windows executable?</h1>
<p>Let's look at <code>clip.exe</code>:</p>
<pre><code>$ vim /mnt/c/Windows/system32/clip.exe
</code></pre>
<p>Right at the start, you'll see the characters "MZ" - these are the first two
bytes of any .exe file on DOS or Windows (and the initials of Mark Zbikowski).</p>
<pre><code>MZ<90>^@^C^@^@^@^D^@^@^@ÿÿ^@^@¸^@^@^@^@^@^@^@@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@è^@^@^@^N^_º^N^@´ Í!¸^ALÍ!This program cannot be run in DOS mode.^M^M
...
</code></pre>
<p>Let's look at the <code>binfmt_misc</code> registrations (this example only works in WSL,
of course):</p>
<pre><code>$ ls /proc/sys/fs/binfmt_misc/
WSLInterop register status
</code></pre>
<p>It's too easy!</p>
<pre><code>$ cat /proc/sys/fs/binfmt_misc/WSLInterop
enabled
interpreter /init
flags: PF
offset 0
magic 4d5a
</code></pre>
<p>And 4d5a is hex for "MZ". So when you <code>execve</code> a Windows executable like
<code>clip.exe</code>, Linux will invoke <code>/init</code> to run <code>clip.exe</code>. The magic is thus
inside <code>/init</code>.</p>
<p><code>/init</code> is not open source. <a href="https://learn.microsoft.com/en-us/archive/blogs/wsl/windows-and-ubuntu-interoperability">The blog post linked above has some
hints</a>
and I encourage you to read it.</p>
<p>There's also <a href="https://github.com/ionescu007/lxss">https://github.com/ionescu007/lxss</a> which contains some
interesting proofs of concept for interacting across the Windows/Linux
boundary.</p>
<h2>How do fully executable jars work?</h2>
<p>The interesting part about these is that they don't involve <code>binfmt_misc</code> at
all, instead they use a different trick.</p>
<p>Go to <a href="https://start.spring.io/">https://start.spring.io/</a> and generate the example project. Add this
section to the <code>build.gradle</code> to generate the "fully executable" jar:</p>
<pre><code>bootJar {
launchScript()
}
</code></pre>
<p>Run <code>./gradlew build</code> to build the project. You get two jars:</p>
<pre><code>$ ls build/libs/
demo-0.0.1-SNAPSHOT-plain.jar demo-0.0.1-SNAPSHOT.jar
</code></pre>
<p>The first jar is not executable and has no main. The second jar is, with either
<code>java -jar</code> or directly:</p>
<pre><code>$ java -jar build/libs/demo-0.0.1-SNAPSHOT.jar
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v3.2.1)
...
^C
$ build/libs/demo-0.0.1-SNAPSHOT.jar
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v3.2.1)
</code></pre>
<p>But what gives, there was no <code>binfmt_misc</code> for Java jars?! The trick here is
that the jar isn't a jar, it's a shell script:</p>
<pre><code>$ less build/libs/demo-0.0.1-SNAPSHOT.jar
#!/bin/bash
...
<shell script>
...
exit 0
<what looks like binary data>
</code></pre>
<p>The binary data after the <code>exit 0</code> is the jar. This is clever: when run
directly, the shell script re-invokes the jar itself (the shell script itself!)
with <code>java -jar</code>.</p>
<p>You can verify the binary data is a jar by looking at the magic bytes:</p>
<pre><code>...
*)
echo "Usage: $0 {start|stop|force-stop|restart|force-reload|status|run}"; exit 1;
esac
exit 0
PK^C^D^T^@^H^H^H^@
...
</code></pre>
<p><code>PK^C^D</code> is exactly the magic byte string for a zip archive. A jar file is just
a zip file with special contents.</p>
<p>This explains how directly invoking the jar executes it without involving
<code>binfmt_misc</code>.</p>
<h2>How does <code>java -jar</code> execute a jar with text at the start?</h2>
<p><code>java</code> isn't doing anything clever here, it just treats the jar as any other
zip file - we can even extract the "fully executable" jar with <code>unzip</code>:</p>
<pre><code>$ unzip build/libs/demo-0.0.1-SNAPSHOT.jar
Archive: build/libs/demo-0.0.1-SNAPSHOT.jar
creating: META-INF/
inflating: META-INF/MANIFEST.MF
...
</code></pre>
<p>The cleverness here is in the zip file format itself, see
<a href="https://en.wikipedia.org/wiki/ZIP_(file_format)">https://en.wikipedia.org/wiki/ZIP_(file_format)</a>. A tool that reads a zip file
must scan for the central directory data structure signature (some magic bytes)
and read it from there. This means that we are allowed to have whatever
preamble we want at the start of the file, including executable code, commonly
used for self-extracting archives (e.g. an .exe you can run or open with your
archive viewer).</p>
<p>This jar isn't self-extracting, but it is kind of self-running. I think it's
a neat trick.</p>
<h2>Conclusion: why we can't use <code>binfmt_misc</code> for jars</h2>
<p>It's pretty common for fully executable jars to not have a <code>.jar</code> extension,
since the whole point of being fully executable is that it's like a "normal"
executable. This means we can't use <code>binfmt_misc</code>'s extension matching.</p>
<p>We can't use the magic byte matching either since:</p>
<ol>
<li><p>Jars are just zip files, they don't have any unique magic bytes! <code>#!</code> is at
the start (which we can't and shouldn't hijack), and <code>PK</code> appears later, but
we can't hijack that either, those are the zip file magic bytes and not all zip
files are jars.</p></li>
<li><p>Even if there were jar specific magic bytes, we don't know the offset! The
shell script at the start can be any length.</p></li>
</ol>
<p>So <code>binfmt_misc</code> is useful for running files with a specific extension, magic
bytes at a specific offset (e.g. Windows executables!) but jars don't have any
of those.</p>
<h2>Final verdict on <code>binfmt_misc</code></h2>
<p><code>binfmt_misc</code> doesn't really seem incredibly useful if you ask me. One cool use
case is registering QEMU as a handler for ARM executables while on an x86
machine, then you can run those binaries as if they were native. That doesn't
seem like a real use case to me.</p>
<p>The WSL interop use case actually seems the most compelling to me, but is that
a reason to have a whole kernel thing? I don't know.</p>
Differences in backwards incompatibility between Rust and C++urn:https-kaashif-co-uk:-2024-01-02-differences-in-backwards-incompatibility-between-rust-and-c2024-01-02T00:00:00Z2024-01-02T00:00:00ZKaashif Hymabaccus
<p>Why is the following change to a Rust struct backwards incompatible?</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="k">struct</span> <span class="nc">S</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="o">+</span><span class="w"> </span><span class="n">y</span>: <span class="kt">i8</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
</pre></div>
<p>And why is the following change to a C++ struct backwards incompatible?</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">S</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="o">+</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">y</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
</pre></div>
<p>The answers are different and may surprise you. Rust provides <em>fewer</em> compile
guarantees about structs by default and <em>more</em> guarantees in code interacting
with structs than C++.</p>
<p>C++ is batshit crazy as always, providing guarantees no-one cares about while
allowing you to invoke UB by accident.</p>
<p>Worth thinking about when deciding whether you need to bump the major version
number of your Rust crate.</p>
<!--more-->
<h2>Why the C++ change is backwards incompatible</h2>
<p>Let's start with the easy one. The C++ standard defines a standard layout struct as:</p>
<blockquote><p>If you could write it in C, it's a standard layout class.</p>
<p>-- <a href="https://en.cppreference.com/w/cpp/language/classes#Standard-layout_class">https://en.cppreference.com/w/cpp/language/classes#Standard-layout_class</a></p></blockquote>
<p>Okay, that's not what it says - there are classes that are only expressible in
C++ that are "standard layout" (e.g. using inheritance), but anything you could
write in C is standard layout.</p>
<p>Standard layout classes have several guarantees:
<a href="https://en.cppreference.com/w/cpp/language/data_members#Standard-layout">https://en.cppreference.com/w/cpp/language/data_members#Standard-layout</a> but
most importantly:</p>
<blockquote><p>A pointer to an object of standard-layout class type can be reinterpret_cast
to pointer to its first non-static non-bitfield data member</p></blockquote>
<p>So given the struct:</p>
<div class="highlight"><pre><span></span><span class="k">struct</span><span class="w"> </span><span class="nc">S</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
</pre></div>
<p>the C++ standard guarantees that this code will work as expected:</p>
<div class="highlight"><pre><span></span><span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">S</span><span class="w"> </span><span class="n">s</span><span class="p">{</span><span class="mi">1</span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">*</span><span class="k">reinterpret_cast</span><span class="o"><</span><span class="kt">int</span><span class="o">*></span><span class="p">(</span><span class="o">&</span><span class="n">s</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>If we make the change above, this code compiles but is now undefined behaviour:</p>
<div class="highlight"><pre><span></span><span class="k">struct</span><span class="w"> </span><span class="nc">S</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">y</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">S</span><span class="w"> </span><span class="n">s</span><span class="p">{</span><span class="mi">1</span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">*</span><span class="k">reinterpret_cast</span><span class="o"><</span><span class="kt">int</span><span class="o">*></span><span class="p">(</span><span class="o">&</span><span class="n">s</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p><code>S*</code> can be cast to a <code>char*</code> but not an <code>int*</code>! That's now UB.</p>
<p>What will likely actually happen on a little endian machine is that it will
still print 1. We're reading the 0x01 at the start, and the rest of the struct
is probably zeroes. But it's not guaranteed to be - the C++ standard says
nothing about the presence or content of the padding in structs. It could be
all 0xff!</p>
<p>This does start to "matter" on big endian machines (quotes because no-one
cares), where the change is observable.</p>
<p>On a big endian machine where <code>sizeof(int) == 4</code>, the first struct may look like:</p>
<pre><code>00 00 00 01
</code></pre>
<p>where reading the first 4 bytes gives an int with value 1, while the second
struct looks like:</p>
<pre><code>01 [00 00 00] 00 00 00 00
</code></pre>
<p>The [bytes] are padding bytes to make sure the int starts at an address
divisible by 4.</p>
<p>Reading the first 4 bytes of this gives a completely different value:
0x01000000.</p>
<p>Two notes:</p>
<ol>
<li><p>Struct members are guaranteed to be initialized to zero if the struct is
partially initialized.</p></li>
<li><p>The contents of padding bytes aren't guaranteed to be 0x00.</p></li>
</ol>
<p>Overall, this is as expected. The surprising part comes next.</p>
<h2>Why the Rust change is backwards incompatible</h2>
<p>Let's suppose you swapped two elements in a struct:</p>
<div class="highlight"><pre><span></span><span class="k">struct</span> <span class="nc">S</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">y</span>: <span class="kt">i8</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>to</p>
<div class="highlight"><pre><span></span><span class="k">struct</span> <span class="nc">S</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">y</span>: <span class="kt">i8</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This is totally backwards compatible if you're using safe Rust! Rust makes
almost no guarantees about the layout of a struct in memory! Unlike C and C++,
Rust doesn't even guarantee the <em>order of elements in memory matches the order
in the struct declaration</em>.</p>
<p>It's true!</p>
<p>From <a href="https://doc.rust-lang.org/reference/type-layout.html">https://doc.rust-lang.org/reference/type-layout.html</a></p>
<blockquote><p>The only data layout guarantees made by this representation are those
required for soundness. They are:</p>
<ol>
<li><p>The fields are properly aligned.</p></li>
<li><p>The fields do not overlap.</p></li>
<li><p>The alignment of the type is at least the maximum alignment of its fields.</p></li>
</ol>
</blockquote>
<p>There is no guarantee about order or pointer conversions.</p>
<p>If you scroll down in that page, you'll notice Rust has <code>#[repr(C)]</code> which
gives you the same layout as C, if you want some stronger guarantees.</p>
<p>But why is our first change above still backwards incompatible? This code
works:</p>
<div class="highlight"><pre><span></span><span class="k">struct</span> <span class="nc">S</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">my_s</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">S</span><span class="p">{</span><span class="w"> </span><span class="n">x</span>: <span class="mi">1</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">my_s</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">S</span><span class="p">{</span><span class="n">x</span><span class="p">}</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>but this code doesn't compile:</p>
<div class="highlight"><pre><span></span><span class="k">struct</span> <span class="nc">S</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">y</span>: <span class="kt">i8</span>
<span class="p">}</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">my_s</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">S</span><span class="p">{</span><span class="w"> </span><span class="n">x</span>: <span class="mi">1</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">my_s</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">S</span><span class="p">{</span><span class="n">x</span><span class="p">}</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>because the initialization is missing <code>y</code>, and the pattern match is missing <code>y</code>.</p>
<p>This is also totally unsurprising.</p>
<h2>Conclusion</h2>
<p>The moral here is:</p>
<ol>
<li><p>C and C++'s standard layout provides some guarantees, but not really that many.</p></li>
<li><p>Rust's default struct memory layout provides <em>almost no</em> guarantees. But
your Rust code is still safe, since the compiler can make sure you don't
accidentally assume things about structs.</p></li>
</ol>
<p>Technically, Rust might write winning lottery ticket numbers as padding at the
start of your structs. Make sure to check.</p>
<h2>Addendum: sizeof</h2>
<p>One naive answer is that adding fields is backwards incompatible because the
<code>sizeof</code> the struct changes.</p>
<p>This isn't exactly right since nothing in the C or C++ standards guarantees the
sizeof a struct is the same even between compilations of the same program. This
is trivially true because e.g. long may be 4 or 8 bytes depending on the
platform and C implementation (long is always 4 bytes on Windows and is 8 bytes
on 64 bit Linux, usually (yes, I know, the standard doesn't say anything about
"bytes" in relation to int and long, only value ranges)).</p>
<p>So the struct sizeof could change arbitrarily depending on the time of day you
compiled the program - you can't rely on that anyway.</p>
<p>Same with Rust:</p>
<blockquote><p>In general, the size of a type is not stable across compilations</p>
<p>-- <a href="https://doc.rust-lang.org/std/mem/fn.size_of.html">https://doc.rust-lang.org/std/mem/fn.size_of.html</a></p></blockquote>
<p>Funnily enough, Rust guarantees that the <code>std::mem::size_of</code> a <code>#[repr(C)]</code> struct
is stable if all members are also <code>#[repr(C)]</code>. So that guarantee is actually
stronger than the C standard's guarantee.</p>
<p>This is unfair because GCC and Clang do guarantee a lot about struct layouts,
it's just the C standard that doesn't. <code>rustc</code> and "the Rust language" are hard
to separate because there's only really one real implementation that people use
and no standard.</p>
<p>Comparing things unfairly is fun though.</p>
How large are the arbitrage opportunities in Eve Online?urn:https-kaashif-co-uk:-2023-09-23-how-large-are-the-arbitrage-opportunities-in-eve-online2023-09-23T00:00:00Z2023-09-23T00:00:00ZKaashif Hymabaccus
<p>I just noticed that we're now past the 10th anniversary of my first blog post,
which I made on 2013-08-11! Maybe I'll write a retrospective. Moving on.</p>
<p>Recently, a friend suggested I'd be interested in EVE Online. For those not
familiar, it's an MMO with lots of stuff but in particular, it has a player
driven market economy. Prices are driven by market forces. Players place buy
and sell orders for various items, other players fill those orders, market
prices move over time.</p>
<p>It's exciting! CCP (the developers of EVE) even <a href="https://www.fastcompany.com/3024392/meet-the-alan-greenspan-of-virtual-currency-in-eve-online">employ
economists</a>
to help manage the in game money supply and inflation. Very exciting!</p>
<p>With any market the question is: how can I make the most money as quickly as
possible for the least effort?</p>
<p>In real life, the answer may be to get a job. In EVE, I thought the answer
<em>might</em> be to find and exploit arbitrages: market mispricings where someone is
selling for a low price and someone else is buying for a high price.</p>
<p>Websites like <a href="https://www.eve-trading.net/">eve-trading.net</a> exist but don't
let us answer the following questions:</p>
<ul>
<li><p>Given an initial investment and fixed cargo space, which opportunity has the
highest return (%) per jump? How does that vary with available capital?
Warren Buffett famously said that having a small amount of money to invest is
the trick to making good returns. Having $100B in cash is a curse - most "good"
opportunities are simply too small.</p></li>
<li><p>Has the size of arbitrage opportunities changed over time?</p></li>
<li><p>Historically, where (in which systems) are the best opportunities?</p></li>
</ul>
<p>These questions can be answered by downloading these datasets:</p>
<ul>
<li><p>Market data from <a href="https://data.everef.net/market-orders/">https://data.everef.net/market-orders/</a></p></li>
<li><p>Static data (e.g. about jump routes and how much cargo space items take up)
from <a href="https://wiki.eveuniversity.org/Static_Data_Export">https://wiki.eveuniversity.org/Static_Data_Export</a></p></li>
</ul>
<p>And doing some analysis. That's what I do in this post.</p>
<!--more-->
<h1>Getting the datasets</h1>
<p>The market data from <a href="https://data.everef.net/market-orders/">https://data.everef.net/market-orders/</a> is easily
available, just download the files you want from your browser. For bulk
downloads, follow the instructions
<a href="https://docs.everef.net/datasets/downloading-datasets.html">here</a> to use wget
to download everything you want.</p>
<p>The static datasets don't require bulk download, just download them from your
browser, you'll only need one file per dataset.</p>
<h1>Setting up our analysis</h1>
<p>I just used a Jupyter notebook and pandas to go through this dataset. The 2023
dataset up to the day of posting this is under 200 GiB, so it's not huge and
can easily be analyzed on a laptop.</p>
<p>I also used the networkx package to create the EVE universe graph and calculate
shortest paths.</p>
<p>The Jupyter notebook I'll be screenshotting graphs from is available in full from
<a href="https://github.com/kaashif/eve-arbitrage-finder">https://github.com/kaashif/eve-arbitrage-finder</a> so I won't include the full
code in this blog post. See there if you want to run the code.</p>
<p>Initially we'll just do all of the analysis for a single day, later we'll graph
these over time.</p>
<h1>Finding arbitrages</h1>
<p>The conditions for an arbitrage are simple. We're looking for a sell order and
a buy order where:</p>
<ol>
<li><p>The sell order has a lower price than the buy order, so we can buy from the
seller low and sell to a buyer high.</p></li>
<li><p>The sell order has a higher available quantity than the minimum quantity
for the buy order.</p></li>
</ol>
<p>The code to find these is incredibly simple:</p>
<div class="highlight"><pre><span></span><span class="k">if</span> <span class="n">sell_price</span> <span class="o"><</span> <span class="n">buy_price</span> <span class="ow">and</span> <span class="nb">int</span><span class="p">(</span><span class="n">sell</span><span class="p">[</span><span class="s2">"volume_remain"</span><span class="p">])</span> <span class="o">></span> <span class="nb">int</span><span class="p">(</span><span class="n">buy</span><span class="p">[</span><span class="s2">"min_volume"</span><span class="p">]):</span>
<span class="c1"># arbitrage found!</span>
<span class="n">arbitrages</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">sell</span><span class="p">,</span> <span class="n">buy</span><span class="p">))</span>
</pre></div>
<p>And now we have our list of sell/buy pairs that give us some profit if we
manage to execute. I know I ignored taxes, but let's just ignore those for this
post.</p>
<h1>Building the jump graph</h1>
<p>In EVE, each system is connected by stargates, which you have to jump between.
CCP provides a list of connected systems in the static data export linked
above.</p>
<p>We can construct the universe graph:</p>
<div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">bz2</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s2">"mapSolarSystemJumps.csv.bz2"</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="s2">"r"</span><span class="p">)</span> <span class="k">as</span> <span class="n">data_csv</span><span class="p">:</span>
<span class="n">route_contents</span> <span class="o">=</span> <span class="n">data_csv</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">route_contents_f</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">(</span><span class="nb">bytes</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="n">route_contents</span><span class="p">,</span> <span class="s2">"utf-8"</span><span class="p">))</span>
<span class="n">route_reader</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">DictReader</span><span class="p">(</span><span class="n">route_contents_f</span><span class="p">)</span>
<span class="n">G</span> <span class="o">=</span> <span class="n">nx</span><span class="o">.</span><span class="n">Graph</span><span class="p">()</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">route_reader</span><span class="p">:</span>
<span class="n">G</span><span class="o">.</span><span class="n">add_edge</span><span class="p">(</span><span class="n">row</span><span class="p">[</span><span class="s2">"fromSolarSystemID"</span><span class="p">],</span> <span class="n">row</span><span class="p">[</span><span class="s2">"toSolarSystemID"</span><span class="p">])</span>
</pre></div>
<p>Then to calculate the number of jumps between two points (solar system IDs):</p>
<div class="highlight"><pre><span></span><span class="nb">len</span><span class="p">(</span><span class="n">nx</span><span class="o">.</span><span class="n">shortest_path</span><span class="p">(</span><span class="n">G</span><span class="p">,</span> <span class="n">arb</span><span class="p">[</span><span class="s2">"from_system"</span><span class="p">],</span> <span class="n">arb</span><span class="p">[</span><span class="s2">"to_system"</span><span class="p">]))</span>
</pre></div>
<p>Very simple and fast - the universe graph isn't actually that big or dense.</p>
<h1>How much do these opportunities return?</h1>
<p>We would expect smaller initial investments to be able to return more: there
are probably opportunities to double or triple 1M ISK but few to do the same
with 100M.</p>
<p>We can graph the best return % in each snapshot (over the course of a day) in
2023 with 1M, 10M, 100M, and 1B in initial capital.</p>
<p><img src="/static/eve2.png" alt="" /></p>
<p>Holy shit, 1000% return for each arbitrage? The picture changes when we take
into account jump distances and instead graph the return per jump for each
opportunity:</p>
<p><img src="/static/eve1.png" alt="" /></p>
<p>But the concept stays the same - having more capital means lower returns. The
returns just look a little less crazy now.</p>
<p>This graph takes into account compounding. i.e. a 100% return per jump with 2
jumps means you double your money twice for a 300% return overall. A simple
division would mean the return is 300% overall, divide by two jumps gives 150%
return per jump. We actually want to take the nth root of one plus the return,
where n is the number of jumps.</p>
<h1>Are these opportunities really risk free?</h1>
<p>No. Hauling anything valuable through low or null sec systems is always
dangerous. People camp gates and will kill you. That's the main risk.</p>
<p>The other risk is that the buy order you want to sell into will disappear or
get corrected before you get there, leaving you holding the bag. This risk can
be detected and mitigated. If the sell order you're buying from is mispriced
(price significantly lower than the regional average) but the buy order isn't,
you're probably fine. If the buy order does disappear, you can sell to another
averagely priced order and still make the money.</p>
<p>If the buy order is mispriced too high, then there is risk.</p>
<p>Given historical market data, we can detect arbitrages and calculate
probabilities of whether the buy order will disappear by checking whether it
did - either someone fulfilled the order or it was corrected. Then we can
assign an expected profit to each opportunity based on those probabilities.</p>
<p>I didn't implement this given there are (I think) better ways to make money,
but it is interesting to think about.</p>
<p>If botting were allowed in EVE, no doubt all these opportunities would be gone.
Or maybe people would regularly go full <a href="https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html">Knight
Capital</a>
and create even more opportunities. Hard to say.</p>
<h1>Conclusion</h1>
<p>I'm pretty sure mining, PvP, doing raids, whatever, probably earns more money
with much less analysis. This isn't the point. The analysis is the point!</p>
<p>I didn't answer all of the questions at the start.</p>
<p>Some things I want to implement in future:</p>
<ul>
<li><p>Return per jump, taking into account your current position. A great
opportunity near Jita isn't that great if you're 20 jumps from Jita.</p></li>
<li><p>Analysis of how fast a player can make money over time using a particular
strategy. i.e. backtesting and simulating trading. Is this a waste of time
long term?</p></li>
<li><p>Simulating other strategies e.g. making markets by placing buy and sell
orders for the same item in the same place. Getting good enough at that may
have higher returns for less risk.</p></li>
<li><p>Adding route security preferences and cargo size limits.</p></li>
<li><p>Determining whether the buy or sell side of the arbitrage is mispriced. If
the sell is mispriced, you just have to buy at the low price and can sell at
your leisure. If the buy is mispriced, you have to sprint from sell to buy -
if it's gone (cancelled, amended) when you get there, you might just be
fucked. I have to know whether to be nervous or not!</p></li>
</ul>
<p>There's a lot of other stuff I'd like to know.</p>
<p>The code I'm using is also really slow, I haven't put any effort into
optimising or profiling anything. The data could be re-encoded from <code>.csv.bz2</code>
(compressed text) files into Parquet or Avro files. Bzip2 is notoriously slow
too, that can likely be improved.</p>
<p>The best route would probably be to convert all the files to a better file
format (e.g. Parquet) then load everything and analyze it using e.g. Dask.</p>
<p>And jump distances could likely be pre-calculated.</p>
<p>Next time, I'll write a bit more about the technical details of analyzing a
multi terabyte dataset with only 32 GB of memory, and try to get the analysis
much faster so I can iterate. Right now I'm waiting minutes for a single day's
worth of analysis.</p>
Valuing converters in Sidereal Confluenceurn:https-kaashif-co-uk:-2023-08-08-valuing-converters-in-sidereal-confluence2023-08-08T00:00:00Z2023-08-08T00:00:00ZKaashif Hymabaccus
<p>(or: How I learnt to stop worrying and love opportunity cost)</p>
<p>I'm a fan of the board game <a href="https://boardgamegeek.com/boardgame/202426/sidereal-confluence">Sidereal
Confluence</a>. It's a game where:</p>
<ul>
<li><p>There are resources: small cubes, large cubes, ultratech (octagons)</p></li>
<li><p>You have converters that change sets of cubes into others (e.g. 2 white
and one blue cube into 3 black cubes)</p></li>
<li><p>Your goal at the end of the game is to have the most victory points (VPs).
Some converters make VPs, during the game you can research techs that give
you VPs, and all resources you've accumulated convert to VPs at set rates.</p></li>
</ul>
<p>The meat of the game is trading - players can trade anything, with any terms.
The only rule is that trades are binding - you <em>must</em> honour your agreements.
This leads to simple agreements like "I'll give you one white cube for one
green cube", where the trade is obviously fair. You also sometimes get harder
to value trades, like "This turn you give me one small white cube and next turn
I'll give you one large black cube".</p>
<p>The rules have a guideline - three small cubes are worth two large cubes, which
is worth the same as an ultratech. This helps with valuations.</p>
<p>One of my friends executed a strategy I found interesting:</p>
<ol>
<li><p>He played the Eni Et, a race which has special converters. For example, a
usual converter might take 3 small cubes and give 4 small cubes, for a ratio of
4/3, but an Eni Et converter might have a ratio of 2 or 3.</p></li>
<li><p>The catch is the Eni Et can't use their special converters and must trade them
to other people who can.</p></li>
<li><p>He sold them permanently early on, valuing a 2 to 4 or 8 to 13 converter
pretty highly. After all, they'll make dozens of cubes for you over the game,
right?</p></li>
<li><p>He won. Did we pay too much for his converters?</p></li>
</ol>
<p>Questions I want to answer quantitatively:</p>
<ul>
<li><p>A simple one: if someone gives me a cube this turn in exchange for some
number of cubes next turn, how many cubes should they demand?</p></li>
<li><p>A harder one: when buying an Eni Et converter permanently, how many cubes
should I pay?</p></li>
</ul>
<!--more-->
<h2>How to value future trades</h2>
<p>Someone on BoardGameGeek has already done the hard work <a href="https://boardgamegeek.com/thread/2992334/equivalent-fair-trades-future-resources-colonies-v">in this
post</a>.
I don't know exactly how they did it, but they calculated average converter
rates and color correction rates (the rate at which players trade
non-convertible for convertible cubes). The reason that's difficult is that
converters change as technologies are researched, and when exactly techs are
done depends on the players, and which techs are drawn from the deck.</p>
<p>From those values, we can calculate the value of 1 small cube at the end of the
game. See that forum post for the details, but the final results that I'll use
are the values of 1 small cube at the end of the game on each turn:</p>
<table>
<thead>
<tr>
<th> Turn </th>
<th> Endgame Value per Cube </th>
</tr>
</thead>
<tbody>
<tr>
<td> 1 </td>
<td> 4.64 </td>
</tr>
<tr>
<td> 2 </td>
<td> 3.80 </td>
</tr>
<tr>
<td> 3 </td>
<td> 3.08 </td>
</tr>
<tr>
<td> 4 </td>
<td> 2.45 </td>
</tr>
<tr>
<td> 5 </td>
<td> 1.91 </td>
</tr>
<tr>
<td> 6 </td>
<td> 1.45 </td>
</tr>
</tbody>
</table>
<p>This means that on turn 1, on average (with average converters and trades), a
single small cube will turn into 4.64 small cubes after all 6 turns are over.</p>
<p>We can now answer our first question. Assuming we want to keep our endgame
value the same and we're paying 1 small cube on turn 1, we should demand 4.64
cubes of endgame value back on turn 2. On turn 2, each small cube is only worth
3.8 at endgame, so we need 4.64/3.8 = 1.22 small cubes back on turn 2.</p>
<p>We can make a table of the number of cubes we should demand on turn X+1 for a
single cube on turn X.</p>
<table>
<thead>
<tr>
<th> Turn </th>
<th> # cubes to ask for next turn </th>
</tr>
</thead>
<tbody>
<tr>
<td> 1 </td>
<td> 1.2198 </td>
</tr>
<tr>
<td> 2 </td>
<td> 1.2355 </td>
</tr>
<tr>
<td> 3 </td>
<td> 1.2558 </td>
</tr>
<tr>
<td> 4 </td>
<td> 1.2815 </td>
</tr>
<tr>
<td> 5 </td>
<td> 1.3194 </td>
</tr>
<tr>
<td> 6 </td>
<td> 1.4500 </td>
</tr>
</tbody>
</table>
<p>One obvious fact is that the rate increases as the game goes on. This is
because converters get better as the game goes on and techs get invented. If
the best converter has a 1.5 ratio and someone invents a 1 to 10 converter,
obviously you should start asking for more cubes in future trades for next turn
- people will be able to pay.</p>
<p>Some conclusions:</p>
<ul>
<li><p>I've played the Im'Dril Nomads a couple of times and they're very dependent
on future trades. Usually people settle on a rate of something like 1 small
cube this turn for 1 large cube next turn. This is a 1.5 rate, which is
better than the "average" rate, strictly speaking.</p></li>
<li><p>These are rules of thumb. If, like the Im'Dril, future trades are crucial to
run your high effiency converters, it's fine to give out favourable rates.
These calculations assume you make average trades and run an average
converter - almost no situations are average.</p></li>
<li><p>People will not generally agree to trades more than 1 turn in the future.
Situations change too drastically - techs come out, new converters are
invented, colonies are colonized. So more involved yield curve calculations are
pointless except maybe if you're playing with bond traders.</p></li>
</ul>
<h2>Valuing Eni Et converters</h2>
<p>Let's say the Eni Et are selling their 2 small cube to 4 small cube converter
on turn 1. If you run it on all 6 turns, you'll make 12 small cubes.</p>
<p>The Eni Et will try to pitch that to you. It's really worth 12 cubes they'll
say, I'll give you a good price - only 4 cubes! Maybe they'll hold an auction
and get a bid of 5. It'll pay for itself quickly and let you do those hard
techs they'll claim.</p>
<p>Let's make some assumptions:</p>
<ol>
<li><p>You manage to run the converter every turn.</p></li>
<li><p>The converter you forego running is an average converter for that turn.</p></li>
</ol>
<p>The average converter efficiency on each turn is:</p>
<table>
<thead>
<tr>
<th> Turn </th>
<th> Average Converter Efficiency </th>
</tr>
</thead>
<tbody>
<tr>
<td> 1 </td>
<td> 1.3 </td>
</tr>
<tr>
<td> 2 </td>
<td> 1.32 </td>
</tr>
<tr>
<td> 3 </td>
<td> 1.3 </td>
</tr>
<tr>
<td> 4 </td>
<td> 1.38 </td>
</tr>
<tr>
<td> 5 </td>
<td> 1.41 </td>
</tr>
<tr>
<td> 6 </td>
<td> 1.5 </td>
</tr>
</tbody>
</table>
<p>If you run the 2 to 4 converter and forego running an average converter, the
number of extra cubes produced as a result of buying the converter is 4 - 2 *
average converter efficiency. The cost of foregoing the average converter is
the opportunity cost, the cost the Eni Et scammers want you to overlook.</p>
<p>We then multiply the extra cubes produced on each turn by the endgame value of
a cube on that turn and sum all future extra cube production to find the value
at endgame of a 2 to 4 converter.</p>
<table>
<thead>
<tr>
<th> Turn </th>
<th> Endgame Value per Cube </th>
<th> Avg Efficiency </th>
<th> Extra Cubes </th>
<th> Endgame Value </th>
<th> Converter endgame value </th>
<th> # cubes </th>
</tr>
</thead>
<tbody>
<tr>
<td> 1 </td>
<td> 4.64 </td>
<td> 1.3 </td>
<td> 1.4 </td>
<td> 6.50 </td>
<td> 22.73 </td>
<td> 4.90 </td>
</tr>
<tr>
<td> 2 </td>
<td> 3.80 </td>
<td> 1.32 </td>
<td> 1.36 </td>
<td> 5.17 </td>
<td> 16.23 </td>
<td> 4.27 </td>
</tr>
<tr>
<td> 3 </td>
<td> 3.08 </td>
<td> 1.3 </td>
<td> 1.4 </td>
<td> 4.31 </td>
<td> 11.06 </td>
<td> 3.59 </td>
</tr>
<tr>
<td> 4 </td>
<td> 2.45 </td>
<td> 1.38 </td>
<td> 1.24 </td>
<td> 3.04 </td>
<td> 6.75 </td>
<td> 2.75 </td>
</tr>
<tr>
<td> 5 </td>
<td> 1.91 </td>
<td> 1.41 </td>
<td> 1.18 </td>
<td> 2.26 </td>
<td> 3.71 </td>
<td> 1.94 </td>
</tr>
<tr>
<td> 6 </td>
<td> 1.45 </td>
<td> 1.5 </td>
<td> 1 </td>
<td> 1.45 </td>
<td> 1.45 </td>
<td> 1.00 </td>
</tr>
</tbody>
</table>
<p>By endgame the value of a 2 to 4 converter should be obvious. If everyone has a
2 to 3 converter as standard, then you should obviously pay at most 1 cube for
a 2 to 4 converter.</p>
<p>The values on the previous turns are much less obvious, at least to me.</p>
<p>On turn 1 you should only pay 4.9 cubes for a 2 to 4 converter that you'll run
every turn, even though it's going to produce 24 cubes from 12 you put in.</p>
<p>It gets even worse when you realise you won't always run the converter. If you
run the converter on only 4/6 turns, you should only be paying 4/6 * 4.9 = 3.27
cubes for it.</p>
<p>I distinctly remember these converters being sold for far more than that, and
not always being run.</p>
<h2>Conclusions</h2>
<p>Opportunity cost matters.</p>
<p>People pitching complex transactions as "really good deals" are probably
tricking you.</p>
<p>A converter that you always run is like a 6 turn bond. You put in the cubes to
run the converter once (buy the bond at par), collect fixed payments every turn
(coupon payments), and once the 6 turns are over, you keep the cubes you put in
(you get the principal back).</p>
<p>This kind of means that when the Eni Et sell you a converter, you're not buying
a bond, you're buying the <em>option</em> to buy the bond, you don't get the converter
<em>with</em> the initial investment on it already. Well, if you negotiate well you
might.</p>
<p>The converter valuation was a discounted cash flow analysis of a security.</p>
<p>Normalizing cube values to endgame values is exactly the same thing as
calculating present values, where the present is the end of the game. A cash
flow further in the past is worth more, just as a cash flow further into the
future is worth less.</p>
<p>Cubes aren't granular enough for every trade to be fair. Those decimal places
getting shaved off in every trade are important. People very often trade 2
small cubes for one large, just because trading 1.5 small cubes isn't possible.
If you're on the winning side of those trades, you'll be very happy.</p>
<p>The same is true for future trades: if you have the choice between running a
1.3 converter and loaning a cube to someone at a 1.5 rate - loan them the cube!
They'll likely think it's a good deal if it lets them run a converter.</p>
<p>Even worse: if you have a pile of cubes waiting for a tech you can't do right
away, don't leave it in a pile earning 0%, loan it out! All of these
calculations are based on the "average" cube making 30% to 50% per turn!</p>
<p>I wish there were more data on Sidereal Confluence and how pros play it! I mean
computer-readable transaction data, full game logs, that kind of thing. But
people in real life aren't going to fill out logs and the game is much worse
online, so I'm not sure that's ever going to happen.</p>
<p>Unless we get AI to play it or something.</p>
Is implementing alloca(3) in C really impossible?urn:https-kaashif-co-uk:-2023-07-13-is-implementing-alloca-3-in-c-really-impossible2023-07-13T00:00:00Z2023-07-13T00:00:00ZKaashif Hymabaccus
<p><code>alloca</code> is a function provided by several C libraries (e.g. glibc) that lets
you allocate a block of memory that will be freed when the calling function
exits. It's usually done by allocating memory on the stack.</p>
<p>But here are a couple of questions:</p>
<ul>
<li><p>No C standard or POSIX standard mentions <code>alloca</code>, so what "should" it really
do?</p></li>
<li><p>Given that no C standard mentions the stack, is it even possible to implement
<code>alloca</code> in C, or do you need assembly to move the stack pointer?</p></li>
<li><p>Given that compiling code with <code>-fomit-frame-pointer</code> usually results in
addresses being expressed as relative to the stack pointer rather than the
frame pointer, is it safe to move the stack pointer ourselves?</p></li>
</ul>
<p>TL;DR: The answer is that you need special support from the compiler to
implement <code>alloca</code> and you can't do it yourself, in C or assembly.</p>
<!--more-->
<h1>What should <code>alloca</code> do?</h1>
<p>There's no standard to refer to, so let's look at the man pages. From Linux:
<a href="https://man7.org/linux/man-pages/man3/alloca.3.html">https://man7.org/linux/man-pages/man3/alloca.3.html</a></p>
<blockquote><p>The alloca() function allocates size bytes of space in the stack
frame of the caller. This temporary space is automatically freed
when the function that called alloca() returns to its caller.</p></blockquote>
<p>From OpenBSD: <a href="https://man.openbsd.org/alloca">https://man.openbsd.org/alloca</a></p>
<blockquote><p>The alloca() function allocates size bytes of space in the stack frame of
the caller. This temporary space is automatically freed on return.</p></blockquote>
<p>gnulib says something interesting:
<a href="https://www.gnu.org/software/gnulib/manual/html_node/alloca.html">https://www.gnu.org/software/gnulib/manual/html_node/alloca.html</a></p>
<blockquote><p>The alloca module provides for a function alloca which allocates memory on
the stack, where the system allows it. A memory block allocated with alloca
exists only until the function that calls alloca returns or exits abruptly.</p>
<p>There are a few systems where this is not possible: HP-UX systems, and some
other platforms when the C++ compiler is used. On these platforms the alloca
module provides a malloc based emulation. This emulation will not free a
memory block immediately when the calling function returns, but rather will
wait until the next alloca call from a function with the same or a shorter
stack length. Thus, in some cases, a few memory blocks will be kept although
they are not needed any more.</p></blockquote>
<p>That's weird, OpenBSD and Linux both support HP PA-RISC but never said anything
about a stack based alloca being impossible. That must be a quirk of HP-UX i.e.
the OS rather than the hardware. I don't really have an explanation for that,
since HP-UX actually does have alloca, and it does mention the stack:
<a href="https://www.unix.com/man-page/hpux/3C/alloca/">https://www.unix.com/man-page/hpux/3C/alloca/</a></p>
<blockquote><p>Allocates space from the stack of the caller</p></blockquote>
<p>Note that the stack on PA-RISC grows the opposite way to x86.</p>
<p>So alloca is consistent across a lot of Unices - allocates memory on the stack
and frees it when the calling function exits.</p>
<p>Why should it be difficult to implement in C? Let's try it.</p>
<h1>Can <code>alloca</code> be implemented in C?</h1>
<p>No C standard mentions the stack, so if we take the stack-based definition of
alloca seriously, it's completely impossible to implement in
standards-compliant, platform-independent, compiler-independent C.</p>
<p>Let's forget about the stack.</p>
<p>Our goal is to be able to do this:</p>
<div class="highlight"><pre><span></span><span class="kt">void</span><span class="w"> </span><span class="nf">f</span><span class="p">(</span><span class="kt">size_t</span><span class="w"> </span><span class="n">size</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="o">*</span><span class="w"> </span><span class="n">arr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">alloca</span><span class="p">(</span><span class="n">size</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ...</span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>and for arr to be automatically freed at the end of f, that's it.</p>
<p>If we restrict ourselves to GCC and clang, and allow
<code>__attribute__((cleanup))</code>
<a href="https://gcc.gnu.org/onlinedocs/gcc/Common-Variable-Attributes.html">https://gcc.gnu.org/onlinedocs/gcc/Common-Variable-Attributes.html</a>, we can
kind of do it:</p>
<div class="highlight"><pre><span></span><span class="kt">void</span><span class="w"> </span><span class="nf">free_memory</span><span class="p">(</span><span class="kt">void</span><span class="o">*</span><span class="w"> </span><span class="n">ptr</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">free</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="kt">void</span><span class="o">**</span><span class="p">)</span><span class="n">ptr</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kt">void</span><span class="w"> </span><span class="nf">f</span><span class="p">(</span><span class="kt">size_t</span><span class="w"> </span><span class="n">size</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">__attribute__</span><span class="p">((</span><span class="n">cleanup</span><span class="p">(</span><span class="n">free_memory</span><span class="p">)))</span><span class="w"> </span><span class="kt">char</span><span class="o">*</span><span class="w"> </span><span class="n">arr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">malloc</span><span class="p">(</span><span class="n">size</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ...</span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This is okay, but isn't exactly what we're looking for. We can define some
macros to make it nicer, but (1) this is on the heap, (2) we'd still need to
add something to the start of our declarations to get this to work.</p>
<p>I don't think there's any way to get this to work.</p>
<p>Variable length arrays (VLAs) let us allocate arrays of variable length on the
stack, but that is done through the compiler doing magic. Let's at least try to
avoid that for the moment.</p>
<h1>Can <code>alloca</code> be implemented in x86 assembly?</h1>
<p>We can't implement alloca as a normal function, since functions get their own
stacks. Using the System V x86_64 calling convention, rsp is callee preserved
- we're not allowed to change the size of the calling function's stack.</p>
<p>Let's do it using inline assembly:</p>
<div class="highlight"><pre><span></span><span class="cp">#define grow_stack(s) asm("sub rsp, " #s)</span>
</pre></div>
<p>This doesn't work because although it does grow the stack, and we can store
things there, we aren't restoring the original stack pointer before we
return.</p>
<p>This means code as simple as this will segfault:</p>
<div class="highlight"><pre><span></span><span class="cp">#define grow_stack(s) asm("sub rsp, " #s)</span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">grow_stack</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>The compiler knows nothing about our stack pointer manipulation:</p>
<div class="highlight"><pre><span></span><span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">size_t</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">grow_stack</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"%zu"</span><span class="p">,</span><span class="w"> </span><span class="n">s</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Although we grew the stack, when we get to the printf, the compiler still
believes <code>s</code> is at the top of the stack, printing s will print garbage if the
compiler is using offsets from rsp. It might work otherwise. This depends on
the optimization level, which is a hint that we shouldn't be doing this.</p>
<h1>How is alloca actually implemented?</h1>
<p>We need help from the compiler. The compiler needs to know when we call alloca
so that it can adjust all references to the stack pointer after the "call" to
alloca.</p>
<p>The glibc "implementation" of alloca is:</p>
<div class="highlight"><pre><span></span><span class="cp">#ifdef __GNUC__</span>
<span class="cp"># define alloca(size) __builtin_alloca (size)</span>
<span class="cp">#endif </span><span class="cm">/* GCC. */</span><span class="cp"></span>
</pre></div>
<p>But in gnulib (the GNU portability library, intended to work with lots of
compilers), there's an actual implementation of alloca in C. It works very
differently and has different semantics to the normal alloca:
<a href="https://github.com/coreutils/gnulib/blob/master/lib/alloca.c">https://github.com/coreutils/gnulib/blob/master/lib/alloca.c</a></p>
<div class="highlight"><pre><span></span><span class="cm">/* (Mostly) portable implementation -- D A Gwyn</span>
<span class="cm"> This implementation of the PWB library alloca function,</span>
<span class="cm"> which is used to allocate space off the run-time stack so</span>
<span class="cm"> that it is automatically reclaimed upon procedure exit,</span>
<span class="cm"> was inspired by discussions with J. Q. Johnson of Cornell.</span>
<span class="cm"> J.Otto Tennant <jot@cray.com> contributed the Cray support.</span>
<span class="cm"> There are some preprocessor constants that can</span>
<span class="cm"> be defined when compiling for your specific system, for</span>
<span class="cm"> improved efficiency; however, the defaults should be okay.</span>
<span class="cm"> The general concept of this implementation is to keep</span>
<span class="cm"> track of all alloca-allocated blocks, and reclaim any</span>
<span class="cm"> that are found to be deeper in the stack than the current</span>
<span class="cm"> invocation. This heuristic does not reclaim storage as</span>
<span class="cm"> soon as it becomes invalid, but it will do so eventually.</span>
<span class="cm"> As a special case, alloca(0) reclaims storage without</span>
<span class="cm"> allocating any. It is a good idea to use alloca(0) in</span>
<span class="cm"> your main control loop, etc. to force garbage collection. */</span><span class="w"></span>
</pre></div>
<p>This doesn't reclaim on caller exit, it reclaims garbage on the next call to
alloca. If you read through the implementation, you'll also notice it uses
malloc and free, not the stack.</p>
<p>If GNU couldn't make alloca without cheating work, then we can't either.</p>
<h1>Final word</h1>
<p>There's a page here about the advantages of alloca:
<a href="https://www.gnu.org/software/libc/manual/html_node/Advantages-of-Alloca.html">https://www.gnu.org/software/libc/manual/html_node/Advantages-of-Alloca.html</a></p>
<p>But I don't buy it - the unavoidable downside of alloca is that if you try to
allocate too much memory, it'll fail. And it's very hard to tell if it failed,
you don't just get back NULL like with malloc, your program will likely crash
in some way.</p>
<p>This why every alloca man page warns against its use. From the OpenBSD man page:</p>
<blockquote><p>The alloca() function is unsafe because it cannot ensure that the pointer
returned points to a valid and usable block of memory. The allocation made
may exceed the bounds of the stack, or even go further into other objects in
memory, and alloca() cannot determine such an error. Avoid alloca() with
large unbounded allocations.</p></blockquote>
<p>Don't use alloca.</p>
Booting the 1994 Dr Dobb's 386BSD 1.0 CDurn:https-kaashif-co-uk:-2023-06-19-booting-the-1994-dr-dobb-s-386bsd-1-0-cd2023-06-19T00:00:00Z2023-06-19T00:00:00ZKaashif Hymabaccus
<p>386BSD 1.0 was released in 1994 on a CD in an issue of Dr Dobb's Journal. There
are guides on the internet on how to boot 386BSD 1.0 in QEMU, like
<a href="http://gunkies.org/wiki/Talk:Installing_386BSD_1.0_on_Qemu">http://gunkies.org/wiki/Talk:Installing_386BSD_1.0_on_Qemu</a> but I don't think
there are any guides on how to boot it like someone in 1994 would've booted it,
from a real MS-DOS installation.</p>
<p>Rather funnily, 386BSD is listed as "theoretically bootable" here:
<a href="https://gunkies.org/wiki/386BSD_1.0">https://gunkies.org/wiki/386BSD_1.0</a>. And there's a post on WinWorld saying
"Personally I have no idea how to boot it (honestly don't ask)" with no
elaboration:
<a href="https://forum.winworldpc.com/discussion/13240/offer-386bsd-reference-cd-rom">https://forum.winworldpc.com/discussion/13240/offer-386bsd-reference-cd-rom</a>.</p>
<p>It's time to put theory into practice and work out how to boot this OS.
There are a couple of things I want to try:</p>
<ol>
<li>DOSBox - maybe it'll work?</li>
<li>QEMU with MS-DOS 6.22</li>
<li>The instructions from gunkies.org</li>
</ol>
<p>You can download the CD image here: <a href="https://archive.org/details/386BSD1.0">https://archive.org/details/386BSD1.0</a> and
follow along.</p>
<p>Also, RIP Bill Jolitz.</p>
<!--more-->
<h2>Poking around the CD image</h2>
<p>Download that ISO above, then mount it:</p>
<pre><code>$ sudo mount 386BSD-1.0.iso /mnt
$ ls /mnt
386bsd arch cd etc nbsd setup1.ex_ SOFTSUB.TXT vbrun200.dl_
386bsd.ddb b CONTRIB.TXT INFO.TXT RELEASE.TXT setup.exe tmp
386bsd.small bin COPYRGHT.TXT install root setupkit.dl_ usr
a boot.exe dev mnt sbin setup.lst var
</code></pre>
<p>A few things to notice:</p>
<ul>
<li>There are some EXEs around. We're probably supposed to run boot.exe from DOS:</li>
</ul>
<pre><code> $ file boot.exe
boot.exe: MS-DOS executable
</code></pre>
<ul>
<li>And setup.exe from Windows:</li>
</ul>
<pre><code> $ file setup.exe
setup.exe: MS-DOS executable, NE for MS Windows 3.x (EXE)
</code></pre>
<ul>
<li>There are some BSD kernels. We can tell because they're executables and have
a bunch of kernel-looking strings in them:</li>
</ul>
<pre><code> $ file 386bsd
386bsd: a.out little-endian 32-bit demand paged pure executable not stripped
$ strings 386bsd
...
/sbin/init
...
%s: blkdev %d too big, not configured.
%s: blkdev %d already used by %s, not configured.
devif: config %s blkdev
%s: chrdev %d too big, not configured.
</code></pre>
<p> These look like the messages you get when booting e.g. OpenBSD, which is
a descendent of 386BSD.</p>
<ul>
<li>There don't seem to be any instructions on the CD itself. We're probably
supposed to read the magazine.</li>
</ul>
<h2>Reading the instructions</h2>
<p>I cannot find any instructions anywhere. This section is a placeholder. If I
find the issue of Dr Dobb's Jorunal or the instructions that came with the CD,
I'll make another post.</p>
<h2>Trying to boot it from DOSBox</h2>
<p>First, install DOSBox:</p>
<pre><code>$ sudo apt install dosbox
</code></pre>
<p>You need to change some settings in your DOSBox config which, for me, lives at
<code>~/.dosbox/dosbox-0.74-3.conf</code>. This is because <code>boot.exe</code> has a faulty (or
maybe it's not faulty) Windows detection routine which aborts when you have
certain DOSBox settings. A failed boot looks like this:</p>
<pre><code>Z:\> imgmount E ~/tmp/3.iso -t iso
Z:\> E:
E:\> boot 386bsd
Text 466944
Data 20480
Start 0xfe000000
Cannot run from Windows DOS Shell
</code></pre>
<p>If you make sure the <code>[dos]</code> section looks like this:</p>
<pre><code>[dos]
# xms: Enable XMS support.
# ems: Enable EMS support.
# umb: Enable UMB support.
# keyboardlayout: Language code of the keyboard layout (or none).
xms=false
ems=false
umb=false
keyboardlayout=auto
</code></pre>
<p>Then you can get a bit further:</p>
<pre><code>Text 466944
Data 20480
Start 0xfe000000
can't open emm
386BSD Release 1.0 by William & Lynne Jolitz [1.0.22 10/27/94 15:32]
</code></pre>
<p>After this we see some concerning probing messages:</p>
<pre><code>probing for wd port 1f0
probing for fd port 3f0
</code></pre>
<p>And these probes don't find anything. It seems like the hardware DOSBox
presents to the 386BSD kernel is too fake and too different from what's
supported, meaning we can't boot.</p>
<pre><code>panic: cannot mount root
press key to boot/dump
</code></pre>
<p>I don't think I'll be able to get this to work.</p>
<h2>Running setup.exe</h2>
<p>This is interesting, there's a setup.exe that does something mysterious. I
installed Windows 3.11 into an MS-DOS 6.22 VM using QEMU and ran setup.exe to
see what would happen.</p>
<p>A setup wizard runs and installs a bunch of manuals! This is what it looks
like:</p>
<p><img src="/static/386bsd1.png" alt="Manuals!" />
<img src="/static/386bsd2.png" alt="Manuals!" /></p>
<p>Kind of cool, but this doesn't actually help us run 386BSD.</p>
<h2>Attempt 2: MS-DOS 6.22 in QEMU</h2>
<p>I got a bit further with this but spoiler alert: I didn't actually manage to
boot to a shell with this either.</p>
<p>Install QEMU:</p>
<pre><code>$ sudo apt install qemu-system-x86
</code></pre>
<p>Get MS-DOS 6.22 install disk images: <a href="https://winworldpc.com/product/ms-dos/622">https://winworldpc.com/product/ms-dos/622</a>.</p>
<p>Make a disk image:</p>
<pre><code>$ qemu-img create msdos.disk 2G
</code></pre>
<p>2G is the maximum MS-DOS partition size.</p>
<p>Boot the first floppy and follow the instructions:</p>
<pre><code>$ qemu-system-i386 -hda msdos.disk -fda disk1.img
</code></pre>
<p>When asked to swap the floppy, press CTRL-ALT-2, you'll see the QEMU monitor
prompt:</p>
<pre><code>QEMU 6.2.0 monitor - type 'help' for more information
(qemu)
</code></pre>
<p>Run this to change the floppy:</p>
<pre><code>(qemu) change floppy0 disk2.img
</code></pre>
<p>And press CTRL-ALT-1 to switch back to the MS-DOS installer. Continue until
MS-DOS is installed.</p>
<p>Get the MS-DOS CD Extensions:
<a href="https://winworldpc.com/product/ms-cd-extensions-msc/125">https://winworldpc.com/product/ms-cd-extensions-msc/125</a> and install them by
running setup.exe. This is so you can use your CD drive.</p>
<pre><code>$ qemu-system-i386 -hda msdos.disk -fda mscdex.img
</code></pre>
<p>In DOS:</p>
<pre><code>C:\> A:
A:\> install
</code></pre>
<p>Then it'll hang for a few minutes, but that's fine. Remove the floppy and
reboot.</p>
<p>Now start the VM with the CD inserted:</p>
<pre><code>$ qemu-system-i386 -hda msdos.disk -cdrom 3.iso
</code></pre>
<p>You'll see the CD-ROM driver stuff on boot:</p>
<pre><code>Booting from Hard Disk...
Starting MS-DOS...
HIMEM is testing extended memory...done.
----------------------------------------------------------------
| E-IDE/ATAPI CD-ROM device driver, Ver 1.25 |
| Copyright (C) LG Electronics Inc. 1997. All rights reserved. |
----------------------------------------------------------------
Unit 0: QEMU QEMU DVD-ROM Product Rev.: 2.5+
Transfer Mode : Programmed I/O
C:\>C:\DOS\SMARTDRV.EXE /X
MSCDEX Version 2.23
Copyright (C) Microsoft Corp. 1986-1993. All rights reserved.
Drive D: = Driver MSCD000 unit 0
C:\>
</code></pre>
<p>There's a problem, the CD drivers and stuff take up too much conventional
memory, and we can't boot 386BSD:</p>
<pre><code>D:\>boot 386bsd
boot: need 23632 more bytes of conventional memory
boot: cannot allocate enough DOS program memory - reduce DOS size
</code></pre>
<p>The problem is that a whopping 101K of conventional memory is filled:</p>
<pre><code>D:\>mem
Memory Type Total = Used + Free
---------------- ------- ------- -------
Conventional 639K 101K 538K
Upper 0K 0K 0K
Reserved 0K 0K 0K
Extended (XMS) 64,512K 2,112K 62,400K
---------------- ------- ------- -------
Total memory 65,151K 2,213K 62,938K
Total under 1 MB 639K 101K 538K
</code></pre>
<p>We can fix this by using EMM386 to move some stuff out of conventional memory.
Make config.sys look like this:</p>
<pre><code>DEVICE=C:\DOS\HIMEM.SYS
DEVICE=C:\DOS\EMM386.EXE NOEMS
DOS=HIGH,UMB
FILES=30
LASTDRIVE=Z
DEVICEHIGH=C:\CDROM\GSCDROM.SYS /D:MSCD000 /
</code></pre>
<p>The key is that the CD driver and as much of DOS as possible needs to live in
upper memory. This results in much less used conventional memory:</p>
<pre><code>Memory Type Total = Used + Free
---------------- ------- ------- -------
Conventional 640K 24K 616K
Upper 99K 81K 18K
Reserved 384K 384K 0K
Extended (XMS) 64,413K 2,353K 62,060K
---------------- ------- ------- -------
Total memory 65,536K 2,842K 62,694K
Total under 1 MB 739K 105K 634K
Largest executable program size 616K (630,864 bytes)
Largest free upper memory block 18K (18,048 bytes)
MS-DOS is resident in the high memory area.
</code></pre>
<p>And now we can boot 386BSD:</p>
<pre><code>D:\>boot 386bsd
Text 466944
Data 20480
Start 0xfe000000
Warning: Too little RAM memory, running in degraded mode.
panic: pmap_ptalloc: kernel pmap
press key to boot/dump
</code></pre>
<p>We need to boot QEMU with more memory (8M should be plenty):</p>
<pre><code>$ qemu-system-i386 -hda msdos.disk -cdrom 3.iso -m 8
</code></pre>
<p>Trying to boot again:</p>
<pre><code>D:\>boot 386bsd
Text 466944
Data 20480
Start 0xfe000000
386BSD Release 1.0 by William & Lynne Jolitz. [1.0.22 10/27/94 15:32]
Copyright (c) 1989-1994 William F. Jolitz. All rights reserved.
clk: irq0
pc: pc0 <color> port 60 irq1
aux: port 310 irq12
wd: wd0 <QEMU HARDDISK> port 1f0 irq14
fd: fd0: port 3f0 irq6 drq2
com: com1: fifo port 3f8 irq4
lpt: lpt0 port 378 irq7
npx: npx: irq13
mcd: port 300 irq10
wd0: cannot find label (no disk label)
panic: cannot mount root
press key to boot/dump
</code></pre>
<p>Now we're getting somewhere - 386BSD has scanned and detected the hard disk,
although there's no BSD disklabel.</p>
<h2>Trying the Gunkies instructions</h2>
<p>At this point I'm curious as to whether the Gunkies instructions work:
<a href="http://gunkies.org/wiki/Talk:Installing_386BSD_1.0_on_Qemu">http://gunkies.org/wiki/Talk:Installing_386BSD_1.0_on_Qemu</a>.</p>
<p>And they do!</p>
<pre><code>$ qemu-system-i386 -fda ddbboot.img -hda disk -hdb 3.iso -m 8
A:\>boot 386bsd.ddb wd1d
Text 335872
Data 114688
Start 0xfe000000
can't open emm
386BSD Release 1.0 by William & Lynne Jolitz. [1.0.21 10/27/94 14:23]
Copyright (c) 1989-1994 William F. Jolitz. All rights reserved.
clk: irq0
pc: pc0 <color> port 60 irq1
aux: port 310 irq12
wd: wd0 <QEMU HARDDISK> wd1 <QEMU HARDDISK> port 1f0 irq14
fd: fd0: 1.44M port 3f0 irq6 drq2
com: com1: fifo port 3f8 irq4
lpt: lpt0 port 378 irq7
npx: npx: irq13
mcd: port 300 irq10
erase ^?, kill ^U, intr ^C
</code></pre>
<p>But the install hangs when partitioning the disk, when we try to create the
backup superblocks:</p>
<pre><code># ./install
...
super-block backups (for fsck -b #) at:
32, 16224, 32416, 48608, 64800, 80992, 97184,
</code></pre>
<p>Then it just hangs.</p>
<h2>Conclusion</h2>
<p>Getting 386BSD 1.0 working is much more difficult than 0.1 and 0.2, which are
much more well studied and patched.</p>
<p>There may be a problem with newer versions of QEMU, specifically SeaBIOS. It
looks like all of the guides which end in success involve compiling e.g. QEMU
0.12.3 and using the PC BIOS included with that version.</p>
<p>If I do something non-trivial on 386BSD like compiling a program or running an
FTP server, I'll make another post. Right now, I've almost got the installer to
work, which is a somewhat unsatisfying place to stop.</p>
Adding keyword arguments to Java with annotation processingurn:https-kaashif-co-uk:-2023-03-27-adding-keyword-arguments-to-java-with-annotation-processing2023-03-27T00:00:00Z2023-03-27T00:00:00ZKaashif Hymabaccus
<p>Java is a language missing a lot of features. One of those missing features is
keyword arguments. By that, I mean something that lets you call functions like
this:</p>
<div class="highlight"><pre><span></span><span class="n">my_function</span><span class="p">(</span><span class="n">x</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">y</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">z</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
</pre></div>
<p>Or even:</p>
<div class="highlight"><pre><span></span><span class="n">my_function</span><span class="p">(</span><span class="n">z</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">x</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">y</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
</pre></div>
<p>That is, arguments that are named, can be reordered, and are non-optional at
compile time. You might quibble: Python doesn't have compile time. But you can
run mypy to check types and if you're missing a required keyword argument,
mypy will fail.</p>
<p>Let's limit our scope to constructors, and aim that if given code like this:</p>
<div class="highlight"><pre><span></span><span class="kn">package</span> <span class="nn">org.example</span><span class="p">;</span>
<span class="nd">@ReorderableStrictBuilder</span>
<span class="kd">public</span> <span class="kd">record</span> <span class="nc">MyBuiltClass</span><span class="p">(</span><span class="n">String</span> <span class="n">first</span><span class="p">,</span> <span class="n">String</span> <span class="n">second</span><span class="p">,</span> <span class="n">String</span> <span class="n">third</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span>
</pre></div>
<p>We want to be able to construct an object something like this:</p>
<div class="highlight"><pre><span></span><span class="c1">// Named arguments</span>
<span class="kd">var</span> <span class="n">x</span> <span class="o">=</span> <span class="n">Builder</span><span class="p">.</span><span class="na">create</span><span class="p">().</span><span class="na">setFirst</span><span class="p">(</span><span class="s">"1"</span><span class="p">).</span><span class="na">setSecond</span><span class="p">(</span><span class="s">"2"</span><span class="p">).</span><span class="na">setThird</span><span class="p">(</span><span class="s">"3"</span><span class="p">).</span><span class="na">build</span><span class="p">();</span>
<span class="c1">// Reorderable</span>
<span class="kd">var</span> <span class="n">x</span> <span class="o">=</span> <span class="n">Builder</span><span class="p">.</span><span class="na">create</span><span class="p">().</span><span class="na">setSecond</span><span class="p">(</span><span class="s">"2"</span><span class="p">).</span><span class="na">setThird</span><span class="p">(</span><span class="s">"3"</span><span class="p">).</span><span class="na">setFirst</span><span class="p">(</span><span class="s">"1"</span><span class="p">).</span><span class="na">build</span><span class="p">();</span>
<span class="c1">// Compile time error if you miss out any arguments - this shouldn't compile.</span>
<span class="kd">var</span> <span class="n">x</span> <span class="o">=</span> <span class="n">Builder</span><span class="p">.</span><span class="na">create</span><span class="p">().</span><span class="na">setSecond</span><span class="p">(</span><span class="s">"2"</span><span class="p">).</span><span class="na">setThird</span><span class="p">(</span><span class="s">"3"</span><span class="p">).</span><span class="na">build</span><span class="p">();</span>
</pre></div>
<p>Is that even possible? I tried to find out. First, let's look at some solutions
I don't like.</p>
<!--more-->
<h1>Errors at runtime - the worst kind of "builder"</h1>
<p>The way the builder pattern is usually done is pretty bad. Someone said one
time that design patterns are just evidence that the language isn't powerful or
expressive enough for the programmer's ideas. Any structure or regularity in
the code is repetition, and could be eliminated with a powerful enough language
or macro system.</p>
<p>I don't know if that's true, but the builder pattern is usually obviously
unsafe or unergonomic. Here's an unsafe or just plainly incorrect example: JAXB
code generation.</p>
<p>JAXB is a way to (among other things) generate Java classes from XSD schemas.
In an XSD schema you can mark fields as required or optional. JAXB classes can
be used in a pseudo-buildery way like so:</p>
<div class="highlight"><pre><span></span><span class="kd">var</span> <span class="n">x</span> <span class="o">=</span> <span class="k">new</span> <span class="n">MyClass</span><span class="p">().</span><span class="na">withX</span><span class="p">(</span><span class="mi">1</span><span class="p">).</span><span class="na">withZ</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
</pre></div>
<p>But there's a problem - there's no compile-time enforcement of required
parameters. Meaning your schema can have all required types, but you can just
do <code>new MyClass()</code> and even serialize it. No-one will complain, except at
runtime, <em>maybe</em>. If you're lucky.</p>
<p>Terrible, terrible, we obviously don't want to just delegate all checks to
runtime.</p>
<h1>The usual builder pattern</h1>
<p>I haven't done much research on this, but from various tutorials online e.g.
<a href="https://www.digitalocean.com/community/tutorials/builder-design-pattern-in-java">this one from
DigitalOcean</a>,
you can see something typical probably.</p>
<p>The required arguments are in the constructor:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="nf">ComputerBuilder</span><span class="p">(</span><span class="n">String</span> <span class="n">hdd</span><span class="p">,</span> <span class="n">String</span> <span class="n">ram</span><span class="p">){</span>
<span class="k">this</span><span class="p">.</span><span class="na">HDD</span><span class="o">=</span><span class="n">hdd</span><span class="p">;</span>
<span class="k">this</span><span class="p">.</span><span class="na">RAM</span><span class="o">=</span><span class="n">ram</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>You might think this is fine even for a small number of arguments, but they
have the same type! You can swap them around by mistake, and it's hard to tell
at the call site. Your IDE probably can't help you except by simulating keyword
arguments.</p>
<p>You type something like:</p>
<div class="highlight"><pre><span></span><span class="kd">var</span> <span class="n">x</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ComputerBuilder</span><span class="p">(</span><span class="n">ram</span><span class="p">,</span> <span class="n">hdd</span><span class="p">);</span>
</pre></div>
<p>and you can't tell it's wrong looking at the file or in the diff. Your IDE (if
it's IntelliJ, at least) might annotate this as:</p>
<div class="highlight"><pre><span></span><span class="kd">var</span> <span class="n">x</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ComputerBuilder</span><span class="p">(</span><span class="n">hdd</span><span class="p">:</span> <span class="n">ram</span><span class="p">,</span> <span class="n">ram</span><span class="p">:</span> <span class="n">hdd</span><span class="p">);</span>
</pre></div>
<p>and you can tell something's up. Fine, but that's showcasing a glaring
deficiency in the language that the IDE papers over.</p>
<p>One solution which isn't naming the arguments is to name the types better,
perhaps have an <code>Hdd</code> class and <code>Ram</code> class. Maybe a good idea, but it's a lot
of work and introduces a lot of code. A topic for another blog post.</p>
<p>This kind of builder pattern is out - it's not even really a builder pattern,
the core is just a plain old Java constructor.</p>
<h1>Staged builder</h1>
<p>There's another solution which will get us named arguments with a compile time
error if we miss anything required - staged builders.</p>
<p>A great example is <a href="https://immutables.github.io/immutable.html#staged-builder">https://immutables.github.io/immutable.html#staged-builder</a>
which can generate staged builder code for you. The way it works is that each
builder method returns a new builder type with only one method that lets you
set the next required field.</p>
<p>Immutables has a great example:</p>
<div class="highlight"><pre><span></span><span class="c1">// under the hood</span>
<span class="kd">public</span> <span class="kd">final</span> <span class="kd">class</span> <span class="nc">ImmutablePerson</span> <span class="kd">implements</span> <span class="n">Person</span> <span class="p">{</span>
<span class="p">...</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="n">NameBuildStage</span> <span class="nf">builder</span><span class="p">()</span> <span class="p">{</span> <span class="p">...</span> <span class="p">}</span>
<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">NameBuildStage</span> <span class="p">{</span> <span class="n">AgeBuildStage</span> <span class="nf">name</span><span class="p">(</span><span class="n">String</span> <span class="n">name</span><span class="p">);</span> <span class="p">}</span>
<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">AgeBuildStage</span> <span class="p">{</span> <span class="n">IsEmployedBuildStage</span> <span class="nf">age</span><span class="p">(</span><span class="kt">int</span> <span class="n">age</span><span class="p">);</span> <span class="p">}</span>
<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">IsEmployedBuildStage</span> <span class="p">{</span> <span class="n">BuildFinal</span> <span class="nf">isEmployed</span><span class="p">(</span><span class="kt">boolean</span> <span class="n">isEmployed</span><span class="p">);</span> <span class="p">}</span>
<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">BuildFinal</span> <span class="p">{</span> <span class="n">ImmutablePerson</span> <span class="nf">build</span><span class="p">();</span> <span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>And they can generate that code for you with an annotation. This is fine, the
errors are at compile time and you have to name all of your arguments. Maybe
this is good enough, but we do lose the reorderability of the arguments. And
there are a lot of extra types around.</p>
<p>Let's file that one away and move on.</p>
<h1>My attempt at a builder annotation</h1>
<p>I saw <a href="https://stackoverflow.com/questions/1638722/how-to-improve-the-builder-pattern/17896170#17896170">this StackOverflow
answer</a>
and it was something I hadn't seen before in Java (I haven't seen all that much
Java code). It kind of reminded me of template metaprogramming in C++. The
example, as written, is this:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">test</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// Compile Error!</span>
<span class="n">Complex</span> <span class="n">c1</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Complex</span><span class="p">(</span><span class="n">Complex</span><span class="p">.</span><span class="na">Builder</span><span class="p">.</span><span class="na">create</span><span class="p">().</span><span class="na">setFirst</span><span class="p">(</span><span class="s">"1"</span><span class="p">).</span><span class="na">setSecond</span><span class="p">(</span><span class="s">"2"</span><span class="p">));</span>
<span class="c1">// Compile Error!</span>
<span class="n">Complex</span> <span class="n">c2</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Complex</span><span class="p">(</span><span class="n">Complex</span><span class="p">.</span><span class="na">Builder</span><span class="p">.</span><span class="na">create</span><span class="p">().</span><span class="na">setFirst</span><span class="p">(</span><span class="s">"1"</span><span class="p">).</span><span class="na">setThird</span><span class="p">(</span><span class="s">"3"</span><span class="p">));</span>
<span class="c1">// Works!, all params supplied.</span>
<span class="n">Complex</span> <span class="n">c3</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Complex</span><span class="p">(</span><span class="n">Complex</span><span class="p">.</span><span class="na">Builder</span><span class="p">.</span><span class="na">create</span><span class="p">().</span><span class="na">setFirst</span><span class="p">(</span><span class="s">"1"</span><span class="p">).</span><span class="na">setSecond</span><span class="p">(</span><span class="s">"2"</span><span class="p">).</span><span class="na">setThird</span><span class="p">(</span><span class="s">"3"</span><span class="p">));</span>
<span class="p">}</span>
</pre></div>
<p>The interface isn't exactly what we wanted, but it's close. The trick is to
simulate a non-type boolean template parameter to keep track of which values
have been set:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">Builder</span><span class="o"><</span><span class="n">Has1</span><span class="p">,</span><span class="n">Has2</span><span class="p">,</span><span class="n">Has3</span><span class="o">></span> <span class="p">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">False</span> <span class="p">{}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">True</span> <span class="p">{}</span>
<span class="kd">private</span> <span class="nf">Builder</span><span class="p">()</span> <span class="p">{}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="n">Builder</span><span class="o"><</span><span class="n">False</span><span class="p">,</span><span class="n">False</span><span class="p">,</span><span class="n">False</span><span class="o">></span> <span class="nf">create</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="n">Builder</span><span class="o"><></span><span class="p">();</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="n">Builder</span><span class="o"><</span><span class="n">True</span><span class="p">,</span><span class="n">Has2</span><span class="p">,</span><span class="n">Has3</span><span class="o">></span> <span class="nf">setFirst</span><span class="p">(</span><span class="n">String</span> <span class="n">first</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="na">first</span> <span class="o">=</span> <span class="n">first</span><span class="p">;</span>
<span class="k">return</span> <span class="p">(</span><span class="n">Builder</span><span class="o"><</span><span class="n">True</span><span class="p">,</span><span class="n">Has2</span><span class="p">,</span><span class="n">Has3</span><span class="o">></span><span class="p">)</span><span class="k">this</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</pre></div>
<p>Then we only expose a constructor for the final result that takes the builder
with all type parameters set to true:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="nf">Complex</span><span class="p">(</span><span class="n">Builder</span><span class="o"><</span><span class="n">True</span><span class="p">,</span><span class="n">True</span><span class="p">,</span><span class="n">True</span><span class="o">></span> <span class="n">builder</span><span class="p">)</span> <span class="p">{</span>
<span class="n">first</span> <span class="o">=</span> <span class="n">builder</span><span class="p">.</span><span class="na">first</span><span class="p">;</span>
<span class="n">second</span> <span class="o">=</span> <span class="n">builder</span><span class="p">.</span><span class="na">second</span><span class="p">;</span>
<span class="n">third</span> <span class="o">=</span> <span class="n">builder</span><span class="p">.</span><span class="na">third</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>I thought that was pretty clever. Ideally I'd want something like:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">Builder</span><span class="o"><</span><span class="n">Has1</span><span class="p">,</span><span class="n">Has2</span><span class="p">,</span><span class="n">Has3</span><span class="o">></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">Has1</span> <span class="o">&&</span> <span class="n">Has2</span> <span class="o">&&</span> <span class="n">Has3</span><span class="p">)</span>
<span class="kd">public</span> <span class="n">Complex</span> <span class="nf">build</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="n">Complex</span><span class="p">(</span><span class="n">first</span><span class="p">,</span> <span class="n">second</span><span class="p">,</span> <span class="n">third</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>i.e. the <code>build</code> method only gets exposed after all fields are populated, but
Java can't do that kind of thing. C++ can with e.g. <code>enable_if</code>.</p>
<p>I wrote an annotation processor here:
<a href="https://github.com/kaashif/java-keyword-args/">https://github.com/kaashif/java-keyword-args/</a> which given:</p>
<div class="highlight"><pre><span></span><span class="nd">@ReorderableStrictBuilder</span>
<span class="kd">public</span> <span class="kd">record</span> <span class="nc">MyBuiltClass</span><span class="p">(</span><span class="n">String</span> <span class="n">first</span><span class="p">,</span> <span class="n">String</span> <span class="n">second</span><span class="p">,</span> <span class="n">String</span> <span class="n">third</span><span class="p">)</span> <span class="p">{</span>
<span class="p">}</span>
</pre></div>
<p>generates this builder code:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">HasFirst</span><span class="p">,</span> <span class="n">HasSecond</span><span class="p">,</span> <span class="n">HasThird</span><span class="o">></span> <span class="p">{</span>
<span class="kd">private</span> <span class="nf">MyBuiltClassBuilder</span><span class="p">()</span> <span class="p">{}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">True</span> <span class="p">{}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">False</span> <span class="p">{}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="n">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">False</span><span class="p">,</span> <span class="n">False</span><span class="p">,</span> <span class="n">False</span><span class="o">></span> <span class="nf">create</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="n">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">False</span><span class="p">,</span> <span class="n">False</span><span class="p">,</span> <span class="n">False</span><span class="o">></span><span class="p">();</span>
<span class="p">}</span>
<span class="kd">private</span> <span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">String</span> <span class="n">first</span><span class="p">;</span>
<span class="kd">public</span> <span class="n">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">True</span><span class="p">,</span> <span class="n">HasSecond</span><span class="p">,</span> <span class="n">HasThird</span><span class="o">></span> <span class="nf">setFirst</span><span class="p">(</span><span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">String</span> <span class="n">arg</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="na">first</span> <span class="o">=</span> <span class="n">arg</span><span class="p">;</span>
<span class="k">return</span> <span class="p">(</span><span class="n">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">True</span><span class="p">,</span> <span class="n">HasSecond</span><span class="p">,</span> <span class="n">HasThird</span><span class="o">></span><span class="p">)</span> <span class="k">this</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">private</span> <span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">String</span> <span class="n">second</span><span class="p">;</span>
<span class="kd">public</span> <span class="n">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">HasFirst</span><span class="p">,</span> <span class="n">True</span><span class="p">,</span> <span class="n">HasThird</span><span class="o">></span> <span class="nf">setSecond</span><span class="p">(</span><span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">String</span> <span class="n">arg</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="na">second</span> <span class="o">=</span> <span class="n">arg</span><span class="p">;</span>
<span class="k">return</span> <span class="p">(</span><span class="n">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">HasFirst</span><span class="p">,</span> <span class="n">True</span><span class="p">,</span> <span class="n">HasThird</span><span class="o">></span><span class="p">)</span> <span class="k">this</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">private</span> <span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">String</span> <span class="n">third</span><span class="p">;</span>
<span class="kd">public</span> <span class="n">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">HasFirst</span><span class="p">,</span> <span class="n">HasSecond</span><span class="p">,</span> <span class="n">True</span><span class="o">></span> <span class="nf">setThird</span><span class="p">(</span><span class="n">java</span><span class="p">.</span><span class="na">lang</span><span class="p">.</span><span class="na">String</span> <span class="n">arg</span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="na">third</span> <span class="o">=</span> <span class="n">arg</span><span class="p">;</span>
<span class="k">return</span> <span class="p">(</span><span class="n">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">HasFirst</span><span class="p">,</span> <span class="n">HasSecond</span><span class="p">,</span> <span class="n">True</span><span class="o">></span><span class="p">)</span> <span class="k">this</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="n">MyBuiltClass</span> <span class="nf">build</span><span class="p">(</span><span class="n">MyBuiltClassBuilder</span><span class="o"><</span><span class="n">True</span><span class="p">,</span> <span class="n">True</span><span class="p">,</span> <span class="n">True</span><span class="o">></span> <span class="n">builder</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="n">MyBuiltClass</span><span class="p">(</span><span class="n">builder</span><span class="p">.</span><span class="na">first</span><span class="p">,</span> <span class="n">builder</span><span class="p">.</span><span class="na">second</span><span class="p">,</span> <span class="n">builder</span><span class="p">.</span><span class="na">third</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Which seems to give us everything we wanted, except that the API is a little ugly:</p>
<div class="highlight"><pre><span></span><span class="n">MyBuiltClass</span> <span class="n">c1</span> <span class="o">=</span> <span class="n">MyBuiltClassBuilder</span><span class="p">.</span><span class="na">build</span><span class="p">(</span><span class="n">MyBuiltClassBuilder</span><span class="p">.</span><span class="na">create</span><span class="p">().</span><span class="na">setFirst</span><span class="p">(</span><span class="s">"1"</span><span class="p">).</span><span class="na">setSecond</span><span class="p">(</span><span class="s">"2"</span><span class="p">).</span><span class="na">setThird</span><span class="p">(</span><span class="s">"3"</span><span class="p">));</span>
</pre></div>
<p>It's really not that bad of a tradeoff given that we can enforce at
compile-time that all setters must be called at least once.</p>
<p>The worst part of this process was using JavaPoet to generate this code. For
those unfamiliar, Java annotation processors output text - they're barely a
level above C preprocessor macros. JavaPoet is a library that helps with that.
It was easier to just generate the text myself.</p>
<h1>The billion dollar elephant in the room</h1>
<p>Oh, we have compile-time enforced required parameters do we? WRONG! You can
still do something like <code>builder.setFirst(null)</code> and no-one can stop you! This
is sad.</p>
<p>No builder pattern will save you from null in Java. Switch to Kotlin.</p>
<h1>Conclusion</h1>
<p>I think you should probably just use Immutables - it's a great library with a
great staged builder annotation. I think that's the best off-the-shelf solution
even if it does mean we lose reorderability of arguments.</p>
<p>Writing an annotation processor in Java isn't that painful actually. Take a
look at the code! <a href="https://github.com/kaashif/java-keyword-args/">https://github.com/kaashif/java-keyword-args/</a> This is my
first time doing anything like this in Java, it was fun to learn about.</p>
The problem with using splice(2) for a faster cat(1)urn:https-kaashif-co-uk:-2023-03-12-the-problem-with-using-splice-2-for-a-faster-cat-12023-03-12T00:00:00Z2023-03-12T00:00:00ZKaashif Hymabaccus
<p>A few weeks ago, I was reading a Hacker News post about a clipboard manager. I
can't remember which one exactly, but an example is
<a href="https://github.com/Keruspe/GPaste">gpaste</a> - they let you have a clipboard
history, view that history, persist things to disk if you want, and so on.</p>
<p>One comment caught my eye: it asked why clipboard managers didn't use the
<a href="https://man7.org/linux/man-pages/man2/splice.2.html">splice(2) syscall</a>. After
all, splice allows copying the contents of a file descriptor to a pipe without
any copies between userspace and kernelspace.</p>
<p>Indeed, replacing a read-write combo with splice does yield massive performance
gains, and we can benchmark that. That got me thinking: why don't other tools
use splice too, like cat? What are the performance gains? Are there any edge
cases where it doesn't work? How can we profile this?</p>
<p>There are blog posts from a while ago lamenting the lack of usage of splice,
e.g. <a href="https://endler.dev/2018/fastcat/">https://endler.dev/2018/fastcat/</a> and interestingly enough, things may
have changed since 2018 (specifically, in 2021), giving us new reasons to avoid
splice.</p>
<p>The conclusion is basically that splice isn't generic enough, the details are
pretty interesting.</p>
<!--more-->
<h1>What's our performance metric?</h1>
<p>The basic question we're trying to answer is how fast can a program take a
filename and write the contents to stdout? We're measuring performance in bits
per second.</p>
<p>One important point is that we want to benchmark with the kernel read cache
warmed, i.e. we run the benchmarks a few times until the number settles down.
This is important because the only difference between any of our methods will
be a memory-to-memory copy, which is always going to be multiple times slower
than a disk-to-memory read, even with DMA.</p>
<p>Warming the read cache means everything is memory-to-memory and differences in
how we do that will show up.</p>
<p>I'll create a file with 10,000M of zeroes and benchmark cat using pv as follows:</p>
<pre><code>$ dd if=/dev/zero of=10g_zero bs=1M count=10000
$ cat 10g_zero | pv > /dev/null
...
$ !! # Repeat to warm cache
9.77GiB 0:00:02 [4.72GiB/s] [ <=> ]
</code></pre>
<p>So 4.72GiB/s is the number to beat!</p>
<h1>read-write implementation</h1>
<p>This is the dumb way you'd write a file to stdout. Make a buffer, open the
file, read it out in chunks, and write those chunks to stdout. The only thing
to tune here is really the buffer size I think. 32k seems to get the best
performance on my machine.</p>
<p>Here's the code, no error handling:</p>
<div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><unistd.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdlib.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><fcntl.h></span><span class="cp"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="o">*</span><span class="w"> </span><span class="n">argv</span><span class="p">[])</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">size_t</span><span class="w"> </span><span class="n">buf_size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">32</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">1024</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">malloc</span><span class="p">(</span><span class="n">buf_size</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">fname</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">fd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">open</span><span class="p">(</span><span class="n">fname</span><span class="p">,</span><span class="w"> </span><span class="n">O_RDONLY</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">ssize_t</span><span class="w"> </span><span class="n">bytes_read</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">read</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span><span class="w"> </span><span class="n">buf</span><span class="p">,</span><span class="w"> </span><span class="n">buf_size</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">bytes_read</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">EXIT_SUCCESS</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">write</span><span class="p">(</span><span class="n">STDOUT_FILENO</span><span class="p">,</span><span class="w"> </span><span class="n">buf</span><span class="p">,</span><span class="w"> </span><span class="n">bytes_read</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>I called this <code>slow.c</code>. Here's the benchmark:</p>
<pre><code>$ ./slow 10g_zero | pv > /dev/null
9.77GiB 0:00:01 [7.38GiB/s] [ <=> ]
</code></pre>
<p>So that's actually faster than cat already. 7.38 GiB/s vs 4.72 GiB/s. But this
is doing unnecessary memory-to-memory copies from kernelspace to userspace on
read, then from userspace to kernelspace on write. Our ideal solution would
just move (not even copy) pages from the file to stdout, with all buffers owned
by the kernel.</p>
<h1>splice implementation</h1>
<p>The splice implementation is a bit more complex, but not much. Looking at the
man page for splice with <code>man 2 splice</code>, we can see the description:</p>
<pre><code>splice() moves data between two file descriptors without copying between kernel address
space and user address space. It transfers up to len bytes of data from the file de-
scriptor fd_in to the file descriptor fd_out, where one of the file descriptors must
refer to a pipe.
</code></pre>
<p>Here's my code for my splice-based cat:</p>
<div class="highlight"><pre><span></span><span class="cp">#define _GNU_SOURCE</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><unistd.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdlib.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><fcntl.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><errno.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><string.h></span><span class="cp"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">argv</span><span class="p">[])</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">size_t</span><span class="w"> </span><span class="n">buf_size</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">16</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">1024</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">fname</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">fd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">open</span><span class="p">(</span><span class="n">fname</span><span class="p">,</span><span class="w"> </span><span class="n">O_RDONLY</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">off64_t</span><span class="w"> </span><span class="n">offset</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">ssize_t</span><span class="w"> </span><span class="n">bytes_spliced</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">splice</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">offset</span><span class="p">,</span><span class="w"> </span><span class="n">STDOUT_FILENO</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="n">buf_size</span><span class="p">,</span><span class="w"> </span><span class="n">SPLICE_F_MOVE</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">SPLICE_F_MORE</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">bytes_spliced</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">EXIT_SUCCESS</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">bytes_spliced</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span><span class="w"> </span><span class="s">"%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">strerror</span><span class="p">(</span><span class="n">errno</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">bytes_spliced</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>I called this <code>fast.c</code>.</p>
<p>Some notes about this:</p>
<ul>
<li><p><code>#define _GNU_SOURCE</code> gives us access to splice, which is a non-standard
(where the standard is POSIX) extension to <code>fcntl.h</code>. This is one reason
splice probably isn't used more widely - it's not portable.</p></li>
<li><p>The flag <code>SPLICE_F_MOVE</code> is literally a no-op, it used to be a hint to the
kernel to move pages where possible, but now does literally nothing. I added
it because I do want a move, but I know it does nothing.</p></li>
<li><p><code>SPLICE_F_MORE</code> is a hint saying more data is coming in a future splice. It's
true for most splices in our case (all but the last). Not sure how useful it
is outside of socket programming, where it's sometimes not obvious to the
kernel that more data is coming.</p></li>
</ul>
<p>Enough with the notes! Let's see some performance numbers!</p>
<pre><code>$ ./fast 10g_zero | pv > /dev/null
9.77GiB 0:00:00 [26.8GiB/s] [ <=> ]
</code></pre>
<p>Whoa, holy shit, 26.8 GiB/s? That's more than 5.6x as fast as cat! This
warrants some further investigation.</p>
<h1>Profiling, fast and slow</h1>
<p>This section title is a reference to "Thinking, Fast and Slow" by Daniel
Kahneman, which I haven't read.</p>
<p><code>fast</code> is so fast I feel like we have to look into it to make sure nothing
weird is going on.</p>
<p>We can use <code>perf</code> to profile our programs and see where we're spending time.
You can install it by installing the <code>linux-tools</code> version specific to your
kernel version. I'm on Ubuntu so I needed to do:</p>
<pre><code>$ sudo apt install linux-tools-5.19.0-32-generic
</code></pre>
<p>Let's look at cat first. Here's the command to run your program and record
performance in <code>perf.data</code>:</p>
<pre><code>$ sudo perf record -- cat ../10g_zero > /dev/null
</code></pre>
<p>Why sudo? Without sudo, perf says something about kernel symbols and symbol map
restrictions if you're not root, so I just run everything here as root. Sue me.
It's not like we're running untrusted code here!</p>
<p>To generate a breakdown with the percentage of time spent in each function:</p>
<pre><code>$ sudo perf report
</code></pre>
<p>For the above case, the report looks like:</p>
<pre><code>Overhead Command Shared Object Symbol
75.43% cat [kernel.kallsyms] [k] copy_user_generic_string
3.22% cat [kernel.kallsyms] [k] filemap_read
2.75% cat [kernel.kallsyms] [k] filemap_get_read_batch
</code></pre>
<p>Then a bunch of negligible <1% stuff.</p>
<p>The function <code>copy_user_generic_string</code> copies to/from userspace. It's clear
that's what's taking the vast majority of time. The perf report for <code>slow</code>
looks the same:</p>
<pre><code>Overhead Command Shared Object Symbol
70.53% slow [kernel.kallsyms] [k] copy_user_generic_string
3.86% slow [kernel.kallsyms] [k] filemap_read
3.82% slow [kernel.kallsyms] [k] filemap_get_read_batch
</code></pre>
<p>This is as expected. Let's look at the perf report for <code>fast</code>:</p>
<pre><code>$ sudo perf record ../fast ../10g_zero > /dev/null
Invalid argument
</code></pre>
<p>Oh, that's because at least one of the input and output have to be a pipe and
in this case, both are files. Let's just throw a cat in there:</p>
<pre><code>$ sudo perf record ../fast ../10g_zero | cat > /dev/null
Invalid argument
</code></pre>
<p>Huh? What? This is annoying, maybe perf does something dodgy to stdout so we
can't splice to it? Let's try making perf output to a file:</p>
<pre><code>$ sudo perf record -o perf.out -- ../fast ../10g_zero | cat > /dev/null
</code></pre>
<p>That finally works. What an ordeal. The report looks like this:</p>
<pre><code>Overhead Command Shared Object Symbol
60.55% fast [kernel.kallsyms] [k] mutex_spin_on_owner ▒
7.86% fast [kernel.kallsyms] [k] filemap_get_read_batch ▒
2.95% fast [kernel.kallsyms] [k] copy_page_to_iter ▒
2.86% fast [kernel.kallsyms] [k] __mutex_lock.constprop.0 ▒
2.47% fast [kernel.kallsyms] [k] copy_user_generic_string
</code></pre>
<p>Notice how little time we're spending copying pages between user and kernel.
It's clear that the stories of increased performance are true.</p>
<h1>The final straw: why splice isn't more widely used</h1>
<p>Our journey has led us to a few reasons why splice isn't used more widely:</p>
<ul>
<li><p>Not portable: this is kind of a non-reason because everyone just uses Linux,
but maybe someone cares about this.</p></li>
<li><p>Not general: you can't splice between files and files (you can just use
sendfile for that anyway), or sockets and sockets, you need to have a pipe at
one of the ends of the splice. This means file-to-file operations like
<code>cat f1 f2 f3 > f4</code> are impossible with splice.</p></li>
<li><p>Not universally supported: not all filesystems actually let you splice
to/from them. It's possible to try a fast implementation and fall back to a
slow one if we're on a non-splice filesystem, but that adds complexity for
little gain.</p></li>
</ul>
<p>And here's the kicker IMO: there still are bugs. Here's one, you still can't
splice from <code>/dev/zero</code> to a pipe:</p>
<pre><code>$ ./fast /dev/zero | pv > /dev/null
Invalid argument
</code></pre>
<p>Here's a thread on the kernel mailing list about that:
<a href="https://lore.kernel.org/all/202105071116.638258236E@keescook/t/">https://lore.kernel.org/all/202105071116.638258236E@keescook/t/</a>. It's
slightly unfair to call this a bug since it was intentional - the death of
generic splice was a planned affair:</p>
<blockquote><p>The general loss of generic splice read/write is known.</p></blockquote>
<p>The ultimate reason for this <code>/dev/zero</code> funkiness is that there's no real
demand for it to work, I guess. Instead of directly using <code>/dev/zero</code>, I used
actual zero files.</p>
<h1>Conclusion</h1>
<p>My advice is to use splice where you can, but keep in mind its drawbacks and
lack of generality. If you control the types of fds passed in and the
filesystem, then you can really go crazy and experience almost zero-copy file
copies.</p>
<p>But if you're writing a general tool in the vein of cat or tee, it's probably
best to stay away from splice unless you really handle all of the weird cases.</p>
Searching for Planet X with the Z3 solverurn:https-kaashif-co-uk:-2023-02-26-searching-for-planet-x-with-the-z3-solver2023-02-26T00:00:00Z2023-02-26T00:00:00ZKaashif Hymabaccus
<p>A few weeks ago, I played the board game "The Search for Planet X". The premise
is that you have a circular board divided into 12 sectors, each containing one
object. That object could be an asteroid, a gas cloud, and so on, but most
importantly, it could be Planet X. Which object is in each space is hidden at
the start of the game and you're racing your opponents to discover Planet X by
scanning sectors and deducing information using a set of rules like "an
asteroid is always adjacent to another asteroid". The winner is the first
player to correctly guess the location of Planet X and the two adjacent
objects.</p>
<p>The full rules can be found here: <a href="https://foxtrotgames.com/planetx/">https://foxtrotgames.com/planetx/</a>.</p>
<p>I don't find these kinds of games very fun, but it did get me thinking: what's
the best strategy? How many possible boards are there, and how hard is this game?</p>
<p>This meant I had to write a program to:</p>
<ol>
<li>Generate all possible boards</li>
<li>Come up with various strategies to pick the best action</li>
<li>See how good those strategies are</li>
</ol>
<p>The source code is here: <a href="https://github.com/kaashif/search-for-planet-x-solver">https://github.com/kaashif/search-for-planet-x-solver</a>.</p>
<!--more-->
<h2>Step 1: Generating all possible boards</h2>
<p>My first attempt at doing this was to randomly assign locations for comets,
then the asteroids, then the gas clouds, and so on, and just run that process
for a long while. Not very intelligent.</p>
<p>A better way is to use the <a href="https://github.com/Z3Prover/z3">Z3 theorem prover</a>,
encode the logic rules describing the possible boards, and generate all models
satisfying those rules.</p>
<p>This is actually pretty easy, Z3 has a set of very ergonomic Python bindings.</p>
<p>To encode the board state for the standard board with 12 sectors, we can create
12 variables $X_i$, all integers ranging from 1 to 6 (representing the 6 types
of object), and apply the constraints from the rulebook. For example, the rule
saying that an asteroid must be adjacent to another asteroid can be encoded as:</p>
<div class="highlight"><pre><span></span><span class="n">z3</span><span class="o">.</span><span class="n">Implies</span><span class="p">(</span>
<span class="n">X</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">==</span> <span class="n">ObjectType</span><span class="o">.</span><span class="n">ASTEROID</span><span class="o">.</span><span class="n">value</span><span class="p">,</span>
<span class="n">z3</span><span class="o">.</span><span class="n">Or</span><span class="p">(</span>
<span class="n">X</span><span class="p">[</span><span class="n">prev_sector</span><span class="p">(</span><span class="n">i</span><span class="p">)]</span> <span class="o">==</span> <span class="n">ObjectType</span><span class="o">.</span><span class="n">ASTEROID</span><span class="o">.</span><span class="n">value</span><span class="p">,</span>
<span class="n">X</span><span class="p">[</span><span class="n">next_sector</span><span class="p">(</span><span class="n">i</span><span class="p">)]</span> <span class="o">==</span> <span class="n">ObjectType</span><span class="o">.</span><span class="n">ASTEROID</span><span class="o">.</span><span class="n">value</span>
<span class="p">)</span>
<span class="p">)</span>
</pre></div>
<p>Once we have all of the constraints, we can generate a model i.e. a set of
values for the variables satisfying the constraints.</p>
<div class="highlight"><pre><span></span><span class="n">solver</span> <span class="o">=</span> <span class="n">z3</span><span class="o">.</span><span class="n">Solver</span><span class="p">()</span>
<span class="n">solver</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="o">*</span><span class="n">constraints</span><span class="p">)</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">solver</span><span class="o">.</span><span class="n">model</span><span class="p">()</span>
</pre></div>
<p>But how do we get the other models? The trick is to save that model and add a
new constraint saying that we only want new models, i.e. at least one variable
is different from the model we've just seen. Then we can just generate models
until the constraint set is unsatisfiable.</p>
<div class="highlight"><pre><span></span><span class="k">while</span> <span class="n">solver</span><span class="o">.</span><span class="n">check</span><span class="p">()</span> <span class="o">==</span> <span class="n">z3</span><span class="o">.</span><span class="n">sat</span><span class="p">:</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">solver</span><span class="o">.</span><span class="n">model</span><span class="p">()</span>
<span class="n">models</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">model</span><span class="p">)</span>
<span class="c1"># At least one of the variables is different</span>
<span class="n">solver</span><span class="o">.</span><span class="n">add</span><span class="p">(</span>
<span class="n">z3</span><span class="o">.</span><span class="n">Or</span><span class="p">([</span>
<span class="n">x</span> <span class="o">!=</span> <span class="n">model</span><span class="p">[</span><span class="n">x</span><span class="p">]</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">X</span>
<span class="p">])</span>
<span class="p">)</span>
</pre></div>
<p>Using this method, I get 4446 possible boards. There may also be an extra
constraint present in any real game that it's always possible to know the
location of Planet X. The reason that's sometimes not possible is that Planet X
shows up in scans as "empty", and there are two "truly empty" sectors. It's
possible to work out if a sector is truly empty most of the time, since gas
clouds must be adjacent to a truly empty sector - if there are three empty
sectors and only one of them is away from a gas cloud, that's Planet X.</p>
<p>It's unclear to me whether that constraint really exists, so I didn't use it.</p>
<h2>Step 2: Picking the best actions</h2>
<p>There are a few types of action possible:</p>
<ul>
<li><p>You can survey a range of sectors for a particular kind of object and get
back the number of objects of that type in your range, this takes less time
the wider the search - it costs 3 days for a survey of 4, 5, or 6 sectors.</p></li>
<li><p>You can target a sector and know what's there - that takes 4 days.</p></li>
</ul>
<p>Those two are easy to implement, since the set of possibilities is limited and
well-defined. Target is just bad and never a good action given its time cost,
as we'll see later, so I didn't implement it.</p>
<ul>
<li><p>You can research a topic, getting a clue in the form of a new logic rule,
e.g. "Planet X is next to a comet". I didn't implement this because I don't
have the full set of research clues. The best I have is some clues I
extracted from a set of PDFs I found on BoardGameGeek, but clues are very
particular to the board - the creators don't give clues that narrow things
down too much given the board. e.g. you won't get "Planet X is next to a
comet" when the comets are next to each other.</p>
<p>I did have a fun time writing a script that uses <code>pdfminer</code> and some regexes
to extract the clues from those PDFs, but there were only 50 boards in that
set - not a big enough sample to be able to derive how research clues depend on
the board.</p></li>
</ul>
<h2>What's the best action?</h2>
<p>Intuitively, you might say something like you want the most information for the
time taken. It turns out that has a rigorous definition: the self-information
of an event $x$ is $-log_b(p(x))$ where $b$ is any base we choose. If we choose
$b=2$, we measure information in Shannons where 1 Shannon is the information
content of 1 bit. I'll just say "bit" from now on.</p>
<p>We're trying to guess the board, and we can reasonably say that each
possibility for a board has equal probability.</p>
<p>I claim (without proof, and I'm not going to prove it because I'm not
completely sure it's true) that the best action to take is the one with the
largest expected self-information. The expected information of a random
variable $X$ is called its entropy and is denoted by $\mathcal{H}(X)$.</p>
<p>This makes some sense because you'd obviously never take an action with a
guaranteed outcome since that tells you nothing. The entropy in that case is
zero. On the other end of the spectrum, knowing which way a 50/50 choice went
tells you much less than knowing which way a uniform choice among a billion
choices went. Intuitively, playing the lottery isn't expected to have a
surprising outcome so if there's one universe where you win and a million where
you lose, playing the lottery isn't a great way to narrow down which universe
you're in.</p>
<p>Now we're in good shape, our algorithm is:</p>
<ol>
<li>Generate all boards.</li>
<li>Pick the action $A$ which maximises $\mathcal{H}(A)$ and execute it.</li>
<li>Filter down the set of possible boards given the result.</li>
<li>Continue until we know where Planet X is and what its two adjacent objects are.</li>
</ol>
<h2>Step 3: How does it perform?</h2>
<p>Without doing any research and just doing surveys, we find Planet X in ~30 days
on average. That's after running a few thousand simulations.</p>
<p>It's nice to get some confirmation that some players' intuitions are good: the
best action in terms of entropy per day is to make the widest possible search
for asteroids. This makes sense since there are four asteroids, there are a
wide range of possible answers - you can learn a lot.</p>
<p>Another confirmed intuition is that research is really good. Although I didn't
implement research actions in our algorithm, we can calculate the expected
information gain from some research clues by hand. Some clues give on the order
of 2 or 3 bits of information per day compared to the best survey with an
entropy of less than 1 bit per day.</p>
<p>It may be possible to kind of guess which research topics are usually better
than others using our small sample of 50 sets of rules. If we knew the full set
of rules for all boards, we could:</p>
<ol>
<li><p>Know when to pick certain research projects given our current information</p></li>
<li><p>Deduce which board we're on from the clues - the clues are part of the board.</p></li>
</ol>
<p>(1) is in keeping with the spirit of the game. (2) seems like cheating. In any
case, doing an immediate research, a survey, then another research is sure to
be a good move. Intuitively it seems like research projects for more numerous
objects are better - the asteroid research projects in particular may be a
strong starter.</p>
<h2>Conclusions</h2>
<p>We don't really have enough information (haha) to reliably test the performance
of our algorithm against the Planet X web app "AI". That's because we don't
have enough information on the strongest move: research.</p>
<p>Given that, there's only one possible next step: write a program to scrape the
web app for all research projects, boards, and bot moves.</p>
<p>The bot moves are pre-determined, so given those we know whether we can beat
the bot or not on that board. I think it's probably not fair to use advance
knowledge of the bot's moves to deduce information about the board - the bot's
moves take no time for us and would tell us something, so those moves would be
infinitely good. That also wouldn't work against a real player, unless we knew
exactly what their algorithm is (an easy countermeasure: randomly pick a
slightly suboptimal move).</p>
<p>I'm thinking of using something like <a href="https://playwright.dev/">https://playwright.dev/</a>, which I've
already successfully used to auto-play some online puzzle games.</p>
Why does Mockito need JVM bytecode generation?urn:https-kaashif-co-uk:-2023-01-23-why-does-mockito-need-jvm-bytecode-generation2023-01-23T00:00:00Z2023-01-23T00:00:00ZKaashif Hymabaccus
<p>Mockito is a pretty popular Java mocking library. It lets you write code like this:</p>
<div class="highlight"><pre><span></span><span class="n">MyClass</span> <span class="n">mockObject</span> <span class="o">=</span> <span class="n">mock</span><span class="p">(</span><span class="n">MyClass</span><span class="p">.</span><span class="na">class</span><span class="p">);</span>
<span class="n">when</span><span class="p">(</span><span class="n">mockObject</span><span class="p">.</span><span class="na">myMethod</span><span class="p">(</span><span class="mi">1</span><span class="p">)).</span><span class="na">thenReturn</span><span class="p">(</span><span class="s">"one"</span><span class="p">);</span>
</pre></div>
<p>Which is pretty cool, even if it's a bit magic. It's not really that magical, conceptually - Mockito simply intercepts method calls and keeps track of which methods have been called globally, and with what arguments. The call to <code>.thenReturn</code> effectively writes to global state, so that the next call to <code>mockObject.myMethod(1)</code> will have the right behaviour.</p>
<p>My question is simple: Mockito uses bytecode generation libraries (cglib or bytebuddy) to construct the proxies - why do we need to go to those lengths? Can't we get by with something more mundane, meaning either in the standard library or higher level (where I consider JVM bytecode to be low level)?</p>
<!--more-->
<h2>Trying to implement Mockito.mock without anything fancy</h2>
<p>The most magic method is mock, of course. That's what takes the class that we want to mock and makes an object which does all of the Mockito magic.</p>
<p>We could read through all of the code here: <a href="https://github.com/mockito/mockito/tree/f48d794ad14982a134fd14dd2aef03477b699dc6/src/main/java/org/mockito/internal/creation/bytebuddy">https://github.com/mockito/mockito/tree/f48d794ad14982a134fd14dd2aef03477b699dc6/src/main/java/org/mockito/internal/creation/bytebuddy</a> and try to understand what it's doing line by line, but that's no fun. It seems like it'd be more fun to try to implement <code>Mockito.mock</code> without all of this bytecode generation voodoo and see where we run into trouble.</p>
<p>We can use <a href="https://docs.oracle.com/javase/7/docs/api/java/lang/reflect/Proxy.html">https://docs.oracle.com/javase/7/docs/api/java/lang/reflect/Proxy.html</a> to do the heavy lifting.</p>
<p>Let's call our mocker class <code>Mocker</code>, we need three methods: <code>mock</code>, which will take a class and give us a mock object; <code>when</code>, which will take any object and return a <code>Mocker</code> that we can call some expectation-setting methods on; and <code>thenReturn</code>, which allows us to set expectations.</p>
<p><code>when</code> is really just to make things read a bit nicer, and so we can match the Mockito example.</p>
<p>Mocker might look like:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">java.lang.reflect.InvocationHandler</span><span class="p">;</span>
<span class="kn">import</span> <span class="nn">java.lang.reflect.Proxy</span><span class="p">;</span>
<span class="kn">import</span> <span class="nn">java.util.HashMap</span><span class="p">;</span>
<span class="kn">import</span> <span class="nn">java.util.Map</span><span class="p">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Mocker</span> <span class="p">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="n">Map</span><span class="o"><</span><span class="n">CallKey</span><span class="p">,</span> <span class="n">Object</span><span class="o">></span> <span class="n">callKeyToReturnValue</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashMap</span><span class="o"><></span><span class="p">();</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="n">T</span> <span class="nf">mock</span><span class="p">(</span><span class="n">Class</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="n">classToMock</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">final</span> <span class="n">InvocationHandler</span> <span class="n">handler</span> <span class="o">=</span> <span class="k">new</span> <span class="n">MockInvocationHandler</span><span class="p">(</span><span class="n">callKeyToReturnValue</span><span class="p">);</span>
<span class="k">return</span> <span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="n">Proxy</span><span class="p">.</span><span class="na">newProxyInstance</span><span class="p">(</span>
<span class="n">classToMock</span><span class="p">.</span><span class="na">getClassLoader</span><span class="p">(),</span>
<span class="k">new</span> <span class="n">Class</span><span class="o">[]</span><span class="p">{</span> <span class="n">classToMock</span> <span class="p">},</span>
<span class="n">handler</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="n">Mocker</span> <span class="nf">when</span><span class="p">(</span><span class="n">Object</span> <span class="n">mockReturnValue</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="n">Mocker</span><span class="p">();</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">thenReturn</span><span class="p">(</span><span class="n">Object</span> <span class="n">value</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">final</span> <span class="n">CallKey</span> <span class="n">callKey</span> <span class="o">=</span> <span class="n">MockInvocationHandler</span><span class="p">.</span><span class="na">lastInvocation</span><span class="p">;</span>
<span class="n">callKeyToReturnValue</span><span class="p">.</span><span class="na">put</span><span class="p">(</span><span class="n">callKey</span><span class="p">,</span> <span class="n">value</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Where CallKey is defined as:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">java.util.Arrays</span><span class="p">;</span>
<span class="kn">import</span> <span class="nn">java.util.Objects</span><span class="p">;</span>
<span class="kd">public</span> <span class="kd">record</span> <span class="nc">CallKey</span><span class="p">(</span><span class="n">String</span> <span class="n">methodName</span><span class="p">,</span> <span class="n">Object</span><span class="o">[]</span> <span class="n">args</span><span class="p">)</span> <span class="p">{</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">equals</span><span class="p">(</span><span class="n">Object</span> <span class="n">o</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="k">this</span> <span class="o">==</span> <span class="n">o</span><span class="p">)</span> <span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">o</span> <span class="o">==</span> <span class="kc">null</span> <span class="o">||</span> <span class="n">getClass</span><span class="p">()</span> <span class="o">!=</span> <span class="n">o</span><span class="p">.</span><span class="na">getClass</span><span class="p">())</span> <span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="n">CallKey</span> <span class="n">callKey</span> <span class="o">=</span> <span class="p">(</span><span class="n">CallKey</span><span class="p">)</span> <span class="n">o</span><span class="p">;</span>
<span class="k">return</span> <span class="n">Objects</span><span class="p">.</span><span class="na">equals</span><span class="p">(</span><span class="n">methodName</span><span class="p">,</span> <span class="n">callKey</span><span class="p">.</span><span class="na">methodName</span><span class="p">)</span> <span class="o">&&</span> <span class="n">Arrays</span><span class="p">.</span><span class="na">equals</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">callKey</span><span class="p">.</span><span class="na">args</span><span class="p">);</span>
<span class="p">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">int</span> <span class="nf">hashCode</span><span class="p">()</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">result</span> <span class="o">=</span> <span class="n">Objects</span><span class="p">.</span><span class="na">hash</span><span class="p">(</span><span class="n">methodName</span><span class="p">);</span>
<span class="n">result</span> <span class="o">=</span> <span class="mi">31</span> <span class="o">*</span> <span class="n">result</span> <span class="o">+</span> <span class="n">Arrays</span><span class="p">.</span><span class="na">hashCode</span><span class="p">(</span><span class="n">args</span><span class="p">);</span>
<span class="k">return</span> <span class="n">result</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>(Thank you, IntelliJ, for generating these methods for me!)</p>
<p>CallKey represents a method call as a method name and a list of arguments. This is how we set expectations - we just map CallKeys to the expected values. More complex behaviours would require more work - executing some action on an invocation, for example.</p>
<p>We need to override equals and hashCode because we want two arg arrays with different identity but equal elements to compare as equal.</p>
<p>MockInvocationHandler is where we fetch expected values:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">java.lang.reflect.InvocationHandler</span><span class="p">;</span>
<span class="kn">import</span> <span class="nn">java.lang.reflect.Method</span><span class="p">;</span>
<span class="kn">import</span> <span class="nn">java.util.Map</span><span class="p">;</span>
<span class="kd">public</span> <span class="kd">record</span> <span class="nc">MockInvocationHandler</span><span class="p">(</span><span class="n">Map</span><span class="o"><</span><span class="n">CallKey</span><span class="p">,</span> <span class="n">Object</span><span class="o">></span> <span class="n">callKeyToExpectedValue</span><span class="p">)</span> <span class="kd">implements</span> <span class="n">InvocationHandler</span> <span class="p">{</span>
<span class="kd">static</span> <span class="n">CallKey</span> <span class="n">lastInvocation</span><span class="p">;</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="n">Object</span> <span class="nf">invoke</span><span class="p">(</span><span class="n">Object</span> <span class="n">proxy</span><span class="p">,</span> <span class="n">Method</span> <span class="n">method</span><span class="p">,</span> <span class="n">Object</span><span class="o">[]</span> <span class="n">args</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">final</span> <span class="kd">var</span> <span class="n">callKey</span> <span class="o">=</span> <span class="k">new</span> <span class="n">CallKey</span><span class="p">(</span><span class="n">method</span><span class="p">.</span><span class="na">getName</span><span class="p">(),</span> <span class="n">args</span><span class="p">);</span>
<span class="n">lastInvocation</span> <span class="o">=</span> <span class="n">callKey</span><span class="p">;</span>
<span class="k">return</span> <span class="n">callKeyToExpectedValue</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="n">callKey</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>When we invoke a method on the mock object, we return the expected value if one exists, otherwise null.</p>
<p>Yes, I know, it would be great to just error out if there's no expectation set. That's what GoogleTest does (in C++), and I wish Mockito did that. Anyway, I went with returning null here because it's simpler. Sue me.</p>
<p>Let's try it out! We can define a simple class to use for our example:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyClass</span> <span class="p">{</span>
<span class="kd">public</span> <span class="n">String</span> <span class="nf">myMethod</span><span class="p">(</span><span class="kt">int</span> <span class="n">i</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">String</span><span class="p">.</span><span class="na">valueOf</span><span class="p">(</span><span class="n">i</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>And let's write a simple main method:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Main</span> <span class="p">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="p">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// mock object</span>
<span class="kd">final</span> <span class="n">MyClass</span> <span class="n">mockObject</span> <span class="o">=</span> <span class="n">Mocker</span><span class="p">.</span><span class="na">mock</span><span class="p">(</span><span class="n">MyClass</span><span class="p">.</span><span class="na">class</span><span class="p">);</span>
<span class="c1">// set an expectation (different from the real implementation!)</span>
<span class="n">Mocker</span><span class="p">.</span><span class="na">when</span><span class="p">(</span><span class="n">mockObject</span><span class="p">.</span><span class="na">myMethod</span><span class="p">(</span><span class="mi">1</span><span class="p">)).</span><span class="na">thenReturn</span><span class="p">(</span><span class="s">"one"</span><span class="p">);</span>
<span class="c1">// should print "one"</span>
<span class="n">System</span><span class="p">.</span><span class="na">out</span><span class="p">.</span><span class="na">println</span><span class="p">(</span><span class="n">mockObject</span><span class="p">.</span><span class="na">myMethod</span><span class="p">(</span><span class="mi">1</span><span class="p">));</span>
<span class="c1">// should print null - we didn't set any expectation</span>
<span class="n">System</span><span class="p">.</span><span class="na">out</span><span class="p">.</span><span class="na">println</span><span class="p">(</span><span class="n">mockObject</span><span class="p">.</span><span class="na">myMethod</span><span class="p">(</span><span class="mi">2</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>And let's run it!</p>
<pre><code>Exception in thread "main" java.lang.IllegalArgumentException: MyClass is not an interface
at java.base/java.lang.reflect.Proxy$ProxyBuilder.validateProxyInterfaces(Proxy.java:706)
at java.base/java.lang.reflect.Proxy$ProxyBuilder.<init>(Proxy.java:648)
at java.base/java.lang.reflect.Proxy$ProxyBuilder.<init>(Proxy.java:656)
at java.base/java.lang.reflect.Proxy.lambda$getProxyConstructor$0(Proxy.java:429)
at java.base/jdk.internal.loader.AbstractClassLoaderValue$Memoizer.get(AbstractClassLoaderValue.java:329)
at java.base/jdk.internal.loader.AbstractClassLoaderValue.computeIfAbsent(AbstractClassLoaderValue.java:205)
at java.base/java.lang.reflect.Proxy.getProxyConstructor(Proxy.java:427)
at java.base/java.lang.reflect.Proxy.newProxyInstance(Proxy.java:1037)
at Mocker.mock(Mocker.java:11)
at Main.main(Main.java:10)
</code></pre>
<p>And there's the problem - java.lang.reflect.Proxy only works on interfaces, not classes. This isn't so bad, we can still test out our implementation by making MyClass an interface, then we get the expected output:</p>
<pre><code>one
null
</code></pre>
<p>This sucks, we had to edit our code to be mockable. You don't have to do that with Mockito.</p>
<h2>How do you mock a class, not an interface?</h2>
<p>In our example, if we wanted to avoid Mockito, we'd have to make an interface MyInterface which is implemented by MyClass, and mock MyInterface - annoying! We really just want to have MyClass and mock it. The magic of Mockito is that you don't have to add all these extra interfaces to your code just for the purposes of testing.</p>
<p>Mockito does this by effectively re-implementing java.lang.reflect.Proxy, but making it work for classes as well interfaces.</p>
<p>If you skim over <a href="https://github.com/mockito/mockito/blob/f48d794ad14982a134fd14dd2aef03477b699dc6/src/main/java/org/mockito/internal/creation/bytebuddy/SubclassBytecodeGenerator.java#L126">https://github.com/mockito/mockito/blob/f48d794ad14982a134fd14dd2aef03477b699dc6/src/main/java/org/mockito/internal/creation/bytebuddy/SubclassBytecodeGenerator.java#L126</a> you effectively see what you'd do if you were writing a mock class by hand, written as bytecode generating code using ByteBuddy.</p>
<h2>Conclusions</h2>
<p>Mockito needs ByteBuddy because Java lacks macros, meaning we need to resort to insane bytecode generation hacks just to generate proxies for classes. Maybe that's not the conclusion you were expecting, but them's the facts.</p>
<p>Clojure doesn't need this nonsense because it has macros and can generate code in a sane way.</p>
<p>Rust also has macros good enough for a great mocking library: <a href="https://docs.rs/mockall/latest/mockall/">https://docs.rs/mockall/latest/mockall/</a> and what would the equivalent of JVM bytecode generation be in a compile-to-native language, anyway? My first thought would be assembly, but that doesn't seem like it makes any sense.</p>
<p>In C++, there really isn't any mocking library as good as Mockito. This is because there are certain things a library can never do. gmock will never let you mock a class with non-virtual methods and use that polymorphically in place of real instances - it's just fundamentally impossible. The gmock cookbook suggests templatizing your code, which is terrible: <a href="https://github.com/google/googletest/blob/main/docs/gmock_cook_book.md#mocking-non-virtual-methods-mockingnonvirtualmethods">https://github.com/google/googletest/blob/main/docs/gmock_cook_book.md#mocking-non-virtual-methods-mockingnonvirtualmethods</a>.</p>
<p>Anyway, use Mockito, it's great! Ignore the bytecode wizard behind the curtain, he's on your side!</p>
Java doesn't really get immutabilityurn:https-kaashif-co-uk:-2022-10-23-java-doesn-t-really-get-immutability2022-10-23T00:00:00Z2022-10-23T00:00:00ZKaashif Hymabaccus
<p>This is a post from the perspective of a new Java programmer, so it is 100%
likely that the concerns here are well-known and already addressed. Or at
least discussed.</p>
<p>Java, as a language, doesn't get (understand) immutability and "delivers" it in
a way that grants almost none of the benefits of immutability in other
languages, like C++ or Rust. I picked those examples to show that the lesson
was learnt a long time ago (C++) and the lesson is still valid and a good idea
(Rust).</p>
<p>Java can, in some sense, be forgiven of its crimes because it's a pretty old
language and is stuck with backwards compatibility. But that doesn't mean it
doesn't commit those crimes.</p>
<p>The primary benefit of immutability is that the programmer knows that value
cannot be changed, so they no longer need to think about what would happen if
it did.</p>
<p>Java doesn't give you that and worse, it pretends that it does. Let's look at
some examples of Java lies and deception.</p>
<!--more-->
<h2>final</h2>
<p>final is basically, from my perspective, useless. It protects against
reassignment, which isn't even nearly the most common type of mutability.</p>
<div class="highlight"><pre><span></span><span class="kd">final</span> <span class="n">List</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">myList</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><></span><span class="p">();</span>
<span class="n">myList</span><span class="p">.</span><span class="na">add</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="n">myList</span><span class="p">.</span><span class="na">add</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
</pre></div>
<p>I mean, no-one even pretends to think <code>final</code> is supposed to stop this, but
this kind of mutation is 99% of all mutation, and final doesn't stop it, so
what's the point of <code>final</code>?</p>
<p>There's also a rule that lambdas can't capture variables that aren't final or
effectively final because of possible race conditions. That's fine, but you can
capture a mutable object just fine, so that restriction seems to be completely
pointless.</p>
<p>An example continuing on from before. This isn't fine because i changes:</p>
<div class="highlight"><pre><span></span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">Supplier</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">badCapture</span> <span class="o">=</span> <span class="p">()</span> <span class="o">-></span> <span class="n">myList</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="n">i</span><span class="p">);</span>
<span class="n">i</span><span class="o">++</span><span class="p">;</span>
</pre></div>
<p>Giving the error:</p>
<pre><code>error: local variables referenced from a lambda expression must be final or effectively final
</code></pre>
<p>But this, despite being exactly the same thing, is fine:</p>
<div class="highlight"><pre><span></span><span class="kd">class</span> <span class="nc">MyInteger</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">incr</span><span class="p">()</span> <span class="p">{</span>
<span class="n">i</span><span class="o">++</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="n">i</span> <span class="o">=</span> <span class="k">new</span> <span class="n">MyInteger</span><span class="p">();</span>
<span class="n">Supplier</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">badCapture</span> <span class="o">=</span> <span class="p">()</span> <span class="o">-></span> <span class="n">myList</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="n">i</span><span class="p">.</span><span class="na">i</span><span class="p">);</span>
<span class="n">i</span><span class="p">.</span><span class="na">incr</span><span class="p">();</span>
</pre></div>
<p>Now tell me, does that make any sense? If the rationale of banning captures of
non-final variables is that they might change, what's the rationale of allowing
captures of other variables that change in slightly different ways?</p>
<p>Whatever it is (and I'm sure there is one, somewhere), it adds to the list of
reasons to avoid Java. Either the language is so hobbled that true safety
measures <em>can't</em> be implemented, or everyone thinks this is fine.</p>
<h2>Immutable data structures</h2>
<p>But (I hear you say) in my example I used an ArrayList, which is mutable! Why not
use an immutable list instead? Sure, let's give that a try:</p>
<div class="highlight"><pre><span></span><span class="kd">final</span> <span class="n">List</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">myList</span> <span class="o">=</span> <span class="n">List</span><span class="p">.</span><span class="na">of</span><span class="p">();</span>
<span class="n">myList</span><span class="p">.</span><span class="na">add</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="n">myList</span><span class="p">.</span><span class="na">add</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span>
</pre></div>
<p>This compiles, but when I run it:</p>
<pre><code>Exception in thread "main" java.lang.UnsupportedOperationException
at java.base/java.util.ImmutableCollections.uoe(ImmutableCollections.java:142)
at java.base/java.util.ImmutableCollections$AbstractImmutableCollection.add(ImmutableCollections.java:147)
at Main.main(example.java:9)
</code></pre>
<p>Oh, but I just used it wrong, right? NO! Java is wrong! This dreadful
"implementation" of immutable data structures violates the basic tenets of
programming. You can no longer treat a List as a List, because secretly it
might be immutable and fail at runtime!</p>
<p>This means if you're calling a library function:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kt">int</span> <span class="nf">doACalculation</span><span class="p">(</span><span class="n">List</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">values</span><span class="p">);</span>
</pre></div>
<p>You can't tell from the type signature whether it's safe to pass an immutable
list in, despite the fact that would implement the interface <code>List<Integer></code>.
You haven't been saved by Java's immutable lists, they've just created
a whole different problem.</p>
<p>The name for the principle Java egregiously violates here is the Liskov
substitution principle:
<a href="https://en.wikipedia.org/wiki/Liskov_substitution_principle">https://en.wikipedia.org/wiki/Liskov_substitution_principle</a>:</p>
<blockquote><p>It is based on the concept of "substitutability" - a principle in
object-oriented programming stating that an object (such as a class) may be
replaced by a sub-object (such as a class that extends the first class)
without breaking the program.</p></blockquote>
<p>Java breaks this because you can't replace a <code>List<T></code> with any other <code>List<T></code>
in all situations. If you mutate, you can only take mutable lists. If you
don't, you can take any list.</p>
<p>This is ridiculous because mutable operations are an obvious superset of
immutable operations. Immutable and mutable lists can both be read, but only
mutable ones can be written.</p>
<p>Mutable lists should be an extension of immutable lists!</p>
<p>Kotlin kind of tried to fix this by making immutable collections the default
(with an interface actually not supporting writing), and mutable ones an
extension of that, but the pain is still there.</p>
<p>Kotlin is still missing decent language level immutability controls (like C++'s
const), probably due to some bullshit (but entirely reasonable) concern about
Java interop.</p>
<h2>Conclusion</h2>
<p>Please do not put up with the incredible farce that is Java immutablity. If at
all possible use some language that made a serious attempt to fix some of
Java's problems, like Kotlin or (if at all possible, again) something descended
from languages that already solved this problem, e.g. Rust. It's more likely
that you can move to Kotlin if you're using Java, so do that. It's really easy.</p>
<p>I can't say with any authority if Kotlin is really the best we can do to fix
the mistakes of Java, but it tries.</p>
<p>This isn't a very optimistic ending.</p>
Don't hide things from people reading your codeurn:https-kaashif-co-uk:-2022-10-16-don-t-hide-things-from-people-reading-your-code2022-10-16T00:00:00Z2022-10-16T00:00:00ZKaashif Hymabaccus
<p>People write code that relies on all sorts of implicit or obfuscated knowledge.
In the worst case, people write code that requires any caller to read through
the entire source to work out how to use it or what it does.</p>
<p>What confuses me is that people often seem to do this intentionally, it's like
they want to require omniscient knowledge of the codebase for anyone wanting to
call or write tests for their code.</p>
<p>I can hear you saying that's ridiculous, and telling me to ask literally anyone
whether they think they have the entire codebase in their heads: they'll
definitely say no.</p>
<p>Everyone will <em>say</em> fitting a huge codebase into their mental working memory is
impossible, but actions speak louder than words. Many people (I see it all the
time) constantly choose programming patterns and idioms that only make sense if
you think everyone coming after you will have read, digested, and memorised all
of the code. There are a few really important ones:</p>
<ul>
<li>Using nullability</li>
<li>Using mutable state</li>
<li>Using global (or static) variables</li>
</ul>
<p>If you ever have to familiarise yourself with a codebase, or ever misremember
anything, you should try to encourage authors to avoid these as much as
possible. But people don't! They love to make things hard for reviewers and
future generations of code readers.</p>
<p>In the same breath as complaining about something a coworker has written,
people will go on to make the same faulty assumptions and obfuscate their code,
perhaps in a slightly different but materially equivalent way.</p>
<p>Let's look at what the problems are. How to convince your coworkers to stop is
left as an exercise for the reader.</p>
<!--more-->
<h2>Nullability</h2>
<p>This is a Java-focused blog post, but this equally well applies to any language
where things can be null - C++ has null pointers and <code>std::optional</code>, Python
has <code>None</code> that people like to pass around. Even Rust often has people passing
around <code>None</code> <code>std::option</code>s and going to great pains to handle that as
unsafely as possible.</p>
<p>It's well known that <code>null</code> was itself a mistake in Java, and this is one of
the reasons I agree with that. Here's some code that a programmer trying to
confuse callers might write:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="n">Pet</span> <span class="nf">getPet</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">hasPet</span><span class="p">())</span> <span class="p">{</span>
<span class="c1">// Means there's no pet</span>
<span class="k">return</span> <span class="kc">null</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">hasDog</span><span class="p">())</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">getDog</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">hasCat</span><span class="p">())</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">getCat</span><span class="p">();</span>
<span class="p">}</span>
<span class="c1">// Means the pet type is unsupported by this function</span>
<span class="k">return</span> <span class="kc">null</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>That seems reasonable. It even has comments. But the intention of this code is
<em>only</em> encoded in comments, not in the null values, which mean different
things but are the same to any caller. Suppose later the class is extended:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kt">void</span> <span class="nf">adoptPet</span><span class="p">(</span><span class="n">Snake</span> <span class="n">snake</span><span class="p">);</span>
</pre></div>
<p>And a caller writes code expecting their snake pet to come back from <code>getPet</code>:</p>
<div class="highlight"><pre><span></span><span class="n">petOwner</span><span class="p">.</span><span class="na">adoptPet</span><span class="p">(</span><span class="k">new</span> <span class="n">Snake</span><span class="p">());</span>
<span class="c1">// ...</span>
<span class="n">Pet</span> <span class="n">pet</span> <span class="o">=</span> <span class="n">petOwner</span><span class="p">.</span><span class="na">getPet</span><span class="p">();</span>
</pre></div>
<p>Obviously pet is null, but why? The original author of the getPet method assumed several things:</p>
<ol>
<li><p>Future contributors would know what null meant and know where to look for
the true meaning of null.</p></li>
<li><p>No-one would ever forget to update <code>getPet</code> to work for more than cats and dogs.</p></li>
</ol>
<p>Solving (2) would be a different blog post entirely (sum types and enforced
exhaustive checking), but (1) has made diagnosing the problem unnecessarily
difficult - why is the value null? Comments don't appear in compiled Java
classes, if this quirk isn't in the PetOwner documentation, it may be
impossible to diagnose.</p>
<p>The solution is to avoid implicitly assigning meaning to null. You won't even
remember what you meant in 2 years when getPet gives you back null. If there's
an error, please please please don't just return null, throw an informative
exception. And no - logging and returning null is not as good as an exception.</p>
<p>This applies in exactly the same way to <code>Optional.empty()</code>. Just because you're
being explicit that you can return a meaningless empty <code>Optional</code> doesn't mean
that's actually much better for readability than returning null.</p>
<p>Generic nullability (e.g. null reference or Optional without any context) is
never good in my opinion. Here are some alternatives:</p>
<ul>
<li>In Rust, prefer Result (which can include a message) over Option</li>
<li>In Java, prefer exceptions over null and Optional</li>
<li>In Haskell, prefer Either (which can include a message) over Maybe</li>
<li>In Python, prefer exceptions over None</li>
</ul>
<p>I hate generic nulls with a passion. If you see a null, it always means context
has been thrown away! Don't assume that future programmers will remember what
your null means or even be able to read your code to work it out.</p>
<p>A common theme in this post is that requiring callers to read through your
source code to be able to use your code is bad - they won't do it. The caller
should be violently and non-optionally confronted with relevant information
through type signatures or exceptions.</p>
<h2>Mutability</h2>
<p>Java doesn't have support for immutability at the language level. This was in
my opinion a huge mistake - C++ had <code>const</code> when Java was being designed! Even
worse, some Java programmers believe that <code>final</code> is a substitute for
immutability. It's not.</p>
<p>If an object is immutable, you only have to look at where the constructor was
called to know the state of the object.</p>
<p>If an object is mutable, you have to read all code referencing that object,
plus have full knowledge of the order that code is going to be called in. That
requires full knowledge of the codebase and gets impossible to manage after a
certain point.</p>
<p>Here's the interface of an object that relies on mutability:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Executor</span> <span class="p">{</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">setParameter</span><span class="p">(</span><span class="n">String</span> <span class="n">name</span><span class="p">,</span> <span class="n">String</span> <span class="n">value</span><span class="p">);</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">execute</span><span class="p">();</span>
<span class="p">}</span>
</pre></div>
<p>The caller will set some parameters, then execute. Fine, how hard could that
be? Let's suppose this is the caller:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">FileManager</span> <span class="p">{</span>
<span class="kd">final</span> <span class="n">Executor</span> <span class="n">executor</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Executor</span><span class="p">();</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">sendFile</span><span class="p">(</span><span class="n">File</span> <span class="n">file</span><span class="p">)</span> <span class="p">{</span>
<span class="n">executor</span><span class="p">.</span><span class="na">setParameter</span><span class="p">(</span><span class="s">"filename"</span><span class="p">,</span> <span class="n">file</span><span class="p">.</span><span class="na">name</span><span class="p">());</span>
<span class="n">executor</span><span class="p">.</span><span class="na">execute</span><span class="p">();</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">deleteFile</span><span class="p">(</span><span class="n">File</span> <span class="n">file</span><span class="p">)</span> <span class="p">{</span>
<span class="n">executor</span><span class="p">.</span><span class="na">setParameter</span><span class="p">(</span><span class="s">"deletion"</span><span class="p">,</span> <span class="s">"true"</span><span class="p">);</span>
<span class="n">executor</span><span class="p">.</span><span class="na">execute</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>There's an obvious problem here - when we call deleteFile, we don't set the
filename. Is that fine? Maybe, if we always call sendFile for a given file
then we delete it right after.</p>
<p>The problem with the Executor is that it holds state, and that state changes.
When you're reading the code and you want to know what <code>executor</code> is, or what
will happen when you call <code>executor.execute()</code>, you don't just have to read
that line of code, you have to read <em>all lines of code that could possibly be
executed before it</em>.</p>
<p>(Note: final doesn't save us here and does almost nothing to help - Java has no
language level immutability enforcement)</p>
<p>The solution here is to change the interface of Executor to avoid requiring
mutation:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Executor</span> <span class="p">{</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">execute</span><span class="p">(</span><span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="p">,</span> <span class="n">String</span><span class="o">></span> <span class="n">parameters</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>Yes, you now have to provide a parameter map every time you call execute. But
this means that the outcome of execute should only depend on one line of code:
the line where you call execute.</p>
<p>This isn't exactly true because Java will still allow Executor to mutate
itself, and there's no way to stop that. In C++, you can stop that by making
execute const:</p>
<div class="highlight"><pre><span></span><span class="k">class</span><span class="w"> </span><span class="nc">Executor</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="k">public</span><span class="o">:</span><span class="w"></span>
<span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="n">execute</span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">map</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">>&</span><span class="p">)</span><span class="w"> </span><span class="k">const</span><span class="p">;</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
</pre></div>
<p>const gives us the guarantee that execute won't mutate its argument or itself.
That means we can free ourselves from needing to hold the entire codebase in
our minds just to work out what this one line does.</p>
<p>Rust has immutability by default, so this problem is solved by default there.
You can unsolve it by making things mutable if you want, but that's
discouraged.</p>
<p>Although Java doesn't allow you to make those kinds of guarantees at the
language level, teams who enforce immutability (except where needed e.g. for
high performance) will have an easier time understanding isolated lines of
code.</p>
<p>In particular, this is a godsend for pull request reviews - you no longer need
to build up a mental model of the temporal ordering of everything that happens
before every line of code. You can skip around and know no state is changing in
the lines you skip over, which is helpful since looking at diffs inherently
involves skipping over many lines.</p>
<h1>Global variables</h1>
<p>Singleton instances. Global variables. Static variables. These are all faces of
the same demon, which is already well known to be evil. Testing is made
particularly hard when the global variables and implicit dependencies are
widespread, and that's the angle I'm coming at this from.</p>
<p>Suppose we have a database connection we want to share throughout our
application. You can get it through a static method:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">DatabaseConnection</span> <span class="p">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="n">DatabaseConnection</span> <span class="nf">getInstance</span><span class="p">();</span>
<span class="p">}</span>
</pre></div>
<p>And we have another class that already exists and has a lot of code:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Server</span> <span class="p">{</span>
<span class="kd">public</span> <span class="nf">Server</span><span class="p">(</span><span class="n">OtherServiceClient</span> <span class="n">client</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">serveRequest</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">cleanUp</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Now let's suppose we write some unit tests:</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">ServerTest</span> <span class="p">{</span>
<span class="nd">@Test</span>
<span class="kt">void</span> <span class="nf">testServeRequest</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="n">server</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Server</span><span class="p">(</span><span class="n">mock</span><span class="p">(</span><span class="n">OtherServiceClient</span><span class="p">.</span><span class="na">class</span><span class="p">));</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Has anything gone wrong here? Is this unit test isolated? We don't know <em>unless
we read the code under test</em>. There may or may not be a hidden, undeclared
dependency of Server on the database, and we'd need to mock that for our unit
tests.</p>
<p>In the worst case, a test author might write a unit test, happen to run it in
the right environment, and think they've isolated their test. In reality,
they're writing and cleaning out a real database somewhere.</p>
<p>I thought by this point everyone knew that sharing global state was bad, but in
Java land, somehow "Singleton" isn't a four-letter word like it should be.</p>
<p>If mutability is banned (which outlaws "setter injection" (which is utterly
disgusting)) and global variables are banned (which outlaws implicit
dependencies on shared state), the only route left is passing dependencies in
as part of the constructor. i.e.</p>
<div class="highlight"><pre><span></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Server</span> <span class="p">{</span>
<span class="kd">public</span> <span class="nf">Server</span><span class="p">(</span><span class="n">OtherServiceClient</span> <span class="n">client</span><span class="p">,</span> <span class="n">DatabaseConnection</span> <span class="n">connection</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</pre></div>
<p>There's no reason there can't still be a single DatabaseConnection, the only
change here is that we're explicitly declaring a dependency on
DatabaseConnection, and now there's no way a test author can accidentally
forget to mock a database, they must explicitly provide a database.</p>
<h1>Conclusion</h1>
<p>I consider these truths self-evident and will broach no criticism. Mutability
hurts readability. Nulls hurt readability and usability. Globals hurt
readability and testability. The most effective way to avoid these is to use a
language where those problems have been solved - use languages that enforce
immutability, allow a wide range of meaningful non-null return values
(exceptions are fine, I guess), and never use global state!</p>
<p>Just say no to mutable state! It confuses you (even if you don't think it does)
and definitely confuses your code reviewers.</p>
Why object to the death of Venice?urn:https-kaashif-co-uk:-2022-08-23-why-object-to-the-death-of-venice2022-08-23T00:00:00Z2022-08-23T00:00:00ZKaashif Hymabaccus
<p>While in Venice, I picked up a book, <em>If Venice Dies</em> by Salvatore Settis. The
main thrust of the book is that tourism is bad, Venice has died or is on the
cusp of death, and changes need to be made to remedy the situation.</p>
<p>This is the kind of book that only works if you already agree with its premise,
and read it as outrage porn rather than as a well-motivated, well-explained
argument. That might seem a bit uncharitable, but the author makes a lot of
claims that are only backed up by vibes and impassioned rhetorical questions
like "Wouldn't that be a tragedy?" rather than any kind of reasoning.</p>
<p>There are two ways to look at each of the claims:</p>
<ol>
<li><p>The author wants to enforce their vision of what Venice should be, and came
up with various justifications post hoc to make it seem like it's in everyone's
interests.</p></li>
<li><p>The author actually believes their own arguments, which largely amount to
fluff, vibes, and vaguely authoritarian policy prescriptions.</p></li>
</ol>
<p>We can go through the book and see if (1) the sinister explanation or (2) the
naive explanation fits better.</p>
<p>There's of course the third explanation: that the author doesn't believe in any
of it and just wrote the book for a quick buck, which we'll ignore. Settis is a
lifelong archaelogist and art historian, so it's at least credible that he does
believe in the conclusions of the book.</p>
<p>Anyway, let's look at some of the unmotivated fluff. I don't expect this blog
post to be particularly entertaining or anything more than a rant.</p>
<!--more-->
<h1>Sometimes cities die</h1>
<p>The first chapter is about the death of Athens, how it devolved into a barely
populated village and its history was lost:</p>
<blockquote><p>Yet nothing could be further from the truth: when Michael Choniates, who
hailed from Constantinople, was appointed archbishop of Athens in the late
twelfth century, he was astonished by the ignorance of the Athenians, who
were unaware of their city's former glories, and weren't able to tell foreign
visitors about their still intact temples, nor could they point out the
places where Socrates, Plato, and Aristotle had preached their doctrines.</p>
<p>[...]</p>
<p>When Athens was conquered by the Ottoman Turks in 1456-and the
Parthenon-Church was turned into a mosque-the city even lost its name. What
remained was a wretched village with a few huts scattered among the ruins,
while the local population, which had been reduced to a few thousand, had
started to call the city Satines-or Sethines-a bastardization that Rome was
never subjected to.</p></blockquote>
<p>This stuff is kind of sad. But it's hard to see what the problem is or what the
solution is - are there things missing from the Italian school curriculum that
the author wishes to add? Should schools in Venice teach more Venetian history?
These questions aren't asked or answered, so it's hard to understand the
solution the author is proposing.</p>
<p>I think the point of the chapter is that the author thinks some cities look
nice and he wants them to stay around after he dies:</p>
<blockquote><p>No: on the contrary, we the living should nurture beauty on a daily basis if
we want some of it to survive, so that we may enjoy it and ensure its
survival after our death.</p></blockquote>
<p>He uses the word "we" instead of "I" for some reason, presumably as a device to
get us to agree with him.</p>
<p>There's no real argument or policy proposal made in this chapter, it appears to
just be here to set the tone - cities die. I don't think Venice is at risk of
dying so I consider this chapter to be irrelevant at best and an attempt at
emotional manipulation at worst.</p>
<h1>Population decline and tourism are bad</h1>
<p>The tiny island of Venice used to have a population much larger than it does
right now, peaking at maybe 150,000 a few hundred years ago, declining to
50,000 today. Why is this a problem? The author doesn't explain.</p>
<blockquote><p>Although the extant demographic data is less reliable, the plague of 1348
proved to be equally devastating, after which it's estimated that the
population dropped from about 120,000 to 58,000: just a little over today's
figures. Yet starting in the 1970s, a new kind of plague broke out in Venice.</p></blockquote>
<p>Settis apparently thinks population decline due to people moving away or not
being born is in some way comparable to people literally dying of a plague.
That wouldn't be such a problem if there were some other argument about why
population decline is bad, but there isn't! The analogy of plague is used again
and again without any sense of proportion.</p>
<p>Even worse, apparently tourism has "devastated" Venice like a "bomb" and has
"annihilated" it. Someone should tell the tourists that the city has been
destroyed and there's nothing to look at.</p>
<p>Why is it so bad that the population of Venice has migrated to the mainland and
prefers to have more space, perhaps a bigger house and a garden, and so on?
Settis also thinks that a tourism monoculture is harmful but doesn't explain why:</p>
<blockquote><p>A tourist monoculture now dominates a city which banishes its native citizens
and shackles the survival of those who remain to their willingness to serve.</p></blockquote>
<p>This looks like it's about how Settis views it as beneath him to serve others
and assumes we share his opinion. Unless he's a subsistence farmer, Settis is
"shackled" to serving others too, which he doesn't seem to realise. There's also
a diatribe about how we must return Venice's people to Venice when evidently
they don't even want to live there.</p>
<p>He wants Venice's population to increase because he thinks it'd be cool, I guess.</p>
<p>By the way, a tourist monoculture might be bad (see: the pandemic) but it's by
far the most productive industry Venice can offer. Trying to shoehorn anything
else into Venice would reduce its prosperity greatly.</p>
<h1>Big cities are bad</h1>
<p>The author decries megalopolises and talks about how every city seems destined
to become some kind of megacity like Tokyo or Shanghai.</p>
<p>I did enjoy how Settis mentioned Trantor from Asimov's <em>Foundation</em> series,
it's one of my favourites. It makes sense that Settis would have read it,
considering it's about the fall of a civilisation and the abandonment of a
city.</p>
<blockquote><p>A perverse continuity has established itself between the megalopolises and
the shanty towns. Thus, was Isaac Asimov's ecumenopolis-the city-planet of
Trantor featured in the science fiction Foundation series which numbered 40
billion inhabitants-a nightmare or a prophecy?</p></blockquote>
<p>I don't know, I think it's neither, maybe it's a dream. This chapter has one of
the worse examples of the repeated rhetorical question form of "argument":</p>
<blockquote><p>Yet do we really want to think of this phenomenon as inevitable and assume it
will conquer the world, supplanting all other urban forms? Or would it
instead be worthwhile to keep other alternatives in mind when we think of the
city of the future, analyzing its characteristics and effects on history and
the notion of livability in our present times? Do we wish to nurture or
destroy the multiplicity and diversity of urban forms?</p></blockquote>
<p>Reading this is hilarious if you come up with different answers to the author.
I consider large cities very livable, I'm even moving to New York.</p>
<p>There's another whole chapter about how skyscrapers are bad because the author
thinks they're ugly and stupid. I agree that the government subsidising
skyscraper construction as a vanity project is stupid, but that's not at all
the angle Settis takes.</p>
<blockquote><p>The acropolis of skyscrapers dominates the historic city from its heights,
having situated itself there in a commanding position that calculatedly
relegates the historic city to the sidelines.</p></blockquote>
<p>He feels that the historic city, which hasn't been destroyed, is being
humiliated in some way, and that's enough for him to ban them. Great. There
can't be any supporting evidence for this since it's inherently subjective.</p>
<p>There's this bit which may be some kind of projection of Settis's own
insecurities onto Venice:</p>
<blockquote><p>But could Venice eventually be surrounded by a ring of skyscrapers, as
envisioned in Aqualta, or will its historic center be dwarfed by a single
high-rise, turning the former into an old dwarf who gets stared down by a
young, muscled giant?</p></blockquote>
<p>Is Settis really the old dwarf? What a bizarre analogy. The single high-rise
he's talking about here is a tower on the mainland, in a derelict industrial
zone, not the island itself. You'd barely be able to see it from the island.</p>
<h1>Money matters, or does it?</h1>
<p>This is really where the book starts to take a turn. Previously Settis limited
himself to talking about how the city doesn't conform to his desires, and how
he finds tall buildings ugly (except if old), but now he starts making economic
statements.</p>
<blockquote><p>We should counter the approximations of self-styled appraisers with the
seriously pondered reflections of others on the true value of cultural
heritage. One need only cross the Alps and head into France. A report
entitled The Economy of the Immaterial: The Growth of Tomorrow, which was
drawn up by Maurice Levy and Jean-Pierre Jouyet, reflects on immaterial
values (meaning priceless ones) as the basis of all future growth.</p>
<p>[...]</p>
<p>The report was commissioned by the French Ministry of the Economy in 2006
under the presidency of Jacques Chirac and concluded that immaterial values
are "concealing a huge potential for growth, which can stimulate the French
economy by generating hundreds of thousands of jobs, while simultaneously
preserving others that would otherwise be put at risk."</p></blockquote>
<p>So immaterial values are good because they can stimulate the economy and create
jobs. Isn't that <em>exactly</em> what's going on right now? The immaterial value of
Venice is creating jobs and income! But Settis condemns that, he only approves
of certain kinds of jobs and income, I guess.</p>
<p>Or should we just ignore that report? Literally the next paragraph says:</p>
<blockquote><p>Yet Venice is now threatened by what John Maynard Keynes once termed the
"parody of an accountant's nightmare," in other words the abject, prejudiced
view that everything should have a price tag, or better yet, that money is
the only thing that matters:</p></blockquote>
<p>This is pretty much incoherent. Immutable values will produce lots of money,
except money doesn't matter. Sure, but I don't see what we should take away
from that.</p>
<h1>Policy</h1>
<p>It goes on and on, there is a ton of fluff about how Venice has been humiliated
and degraded by having copies of it made, and how theme parks have gondolas in
them, but the real meat of the book comes when Settis makes some policy
recommendations.</p>
<h2>No development for tourism</h2>
<p>Settis says he wants what's best for Venice's destiny, but that seems to be
code for conforming to what he deems acceptable, and he doesn't want tourism:</p>
<blockquote><p>Venice must know how to creatively construct its own destiny, tailoring each
change it makes according to the best possible future for its citizens, and
not what the tourists or real estate agencies want.</p></blockquote>
<p>This implies that tourism isn't best for Venice, but never backs that up.</p>
<h2>Ban development completely, actually</h2>
<p>Settis supports plans to ban development in a belt around cities' historic
limits due to some idea of the importance of the meeting of city and
countryside:</p>
<blockquote><p>It's time to "limit the endless expansion of suburban sprawl by returning
cities to their margins," as Zanardi has written, while at the same time
"soldering the historic center to its periphery" and reestablishing the
connection between the city and its citizens.</p></blockquote>
<p>Luckily this proposal can't apply to Venice since it has been naturally limited
by the lagoon.</p>
<h2>Enshrine a right to the city</h2>
<p>He supports a right to the city, similar to the one Brazil recognised, which:</p>
<blockquote><p>guarantee[s] the right to sustainable cities, understood as the right to
urban land, housing, environmental sanitation, urban infrastructure,
transportation and public services, to work and leisure for current and
future generations; democratic administration by means of participation of
the population and of the representative associations of the various segments
of the community in the formulation, execution and monitoring of urban
development projects, plans and programs.</p></blockquote>
<p>This is incredibly vague but is presented as Settis as a panacea. It's nothing
but a tool to block any and all development. Settis claims he doesn't want to
block all progress, but only advocates laws and "rights" that'll enable all
progress to be blocked.</p>
<p>The wishy-washiness of this part of the book is exemplified:</p>
<blockquote><p>Even in Venice, the health of democracy is determined by how successful
citizens prove in defending their rights, which include the common good and
the social functions of property.</p></blockquote>
<p>The social functions of property! What does that actually mean? Who gets to
decide what the social functions of property are? Its owners or others? People
voting to use the coercive power of the state to restrict what property owners
can do is not moral unless the owner's plans will materially harm them. Having
to look at a building on the horizon is not harmful in the way that a polluting
factory is.</p>
<h2>Tourism is bad</h2>
<p>Yes, this same proposal to have the government distort things away from tourism
crops up again:</p>
<blockquote><p>In Venice's case, the work available to local residents can't be restricted to
the tourist monoculture, but has to be worthy of the immense civic capital
the city has accumulated over centuries. The social function of property,
regardless of its ownership, can't be solely determined by increasing real
estate value while decimating the local population and condemning the city to
die. It must nurture creative and productive enterprises, repopulate the city
with young people, and loosen the tourist monoculture's stranglehold.</p></blockquote>
<p>The work isn't restricted to the tourist monoculture. Anyone is free to do
whatever they want. But if they don't generate enough income to pay the bills,
they'll have to leave.</p>
<p>The only solution that will eliminate the tourist monoculture is for the
government to subsidise other industries and displace tourist-oriented
industries. Why is that necessary? Because tourism is not worthy of Venice!
Catering to tourists is unworthy of Venetians! This is enough to justify taking
money from productive Italians and using it to subsidise Venetians to do
something other than serve tourists.</p>
<h2>An architectural code of ethics</h2>
<p>Settis wants architects to have to take a Vitruvian Oath, like the Hippocratic
Oath, wherein architects swear not to design ugly buildings. Yes, I'm serious.</p>
<blockquote><p>We could easily adopt every single one of the professional requirements that
Vitruvius lists in his book and compile them into a "Vitruvian Oath," turning
it into the perfect equivalent of the Hippocratic Oath.</p>
<p>[...]</p>
<p>If those who build in Venice knew how to marry practice and theory, no
architectural design would so flagrantly ignore the physical conditions and
construction practices unique to that city.</p></blockquote>
<p>The various elements of the proposed oath are extremely vague and impossible to
criticise, except in how vague they are.</p>
<h2>We should encourage people to live in Venice</h2>
<p>Right at the end of the book, we get a very confused chapter which describes
immense corruption in the MOSE project to protect Venice from flooding, then
proceeds to advocate a range of subsidies and tax breaks the government should
deploy to revive Venice:</p>
<blockquote><p>In Venice's case, this new pact will have to begin from a strong sense of
commitment to spur politicians and public institutions to adopt a more
creative outlook toward the city, to bring the historic city back to life and
gear it toward the future, the means to create a new kind of politics to stem
the perverse logic causing the exodus of citizens, and to encourage the young
to remain via strong incentives such as tax breaks. It would also mean
curbing the rampant proliferation of second homes and the transformation of
buildings into nothing more than hotels. It would mean encouraging
manufacturing and private enterprise as well as generating opportunities for
a wider range of creative jobs. It would mean reunifying the historic city,
lagoon, and mainland by differentiating their functions, making more
agricultural land available and investing in new fisheries, reutilizing old,
vacant buildings, incentivizing research, launching new professional training
schemes and apprenticeships and investing in universities, chiefly by making
it affordable for students to actually live in the city. It would mean
developing new models, analyzing situations, evaluating options, and
emphasizing initiatives of a higher caliber (like the universities and the
Biennale) and not just enslaving the city to "uncontrollable market forces."
It would mean enshrining the right to the city and the common good as our
first priority.</p></blockquote>
<p>Why is manufacturing specifically singled out as a "good" kind of enterprise?
Why are second homes bad? Why are hotels bad? Why are creative jobs good? Who
is going to re-utilize the abandoned buildings, when they have been abandoned
due to being useless? Why is it inherently good for students to live in Venice?</p>
<p>This list of proposals is unmotivated and unexplained.</p>
<h1>Conclusion</h1>
<p>Settis wants the government to ban stuff he doesn't like (tourism, tall
buildings, subways) and subsidise stuff he does like (creatives, people living
in Venice). He finds tourism degrading and would prefer Venice be a military
(yes, he explicitly uses the word "military") and economic powerhouse like it
was a thousand years ago.</p>
<p>As usual with these kinds of people, they get a raft of academics from their
bubble to rant and rave about how great their book is, ignoring that it only
works if you already agree with the conclusions.</p>
<p>The only motivations given are hyperbolic, borderline insane analogies about
how skyscrapers are muscled young men overlooking dwarves, and how someone
moving out of Venice is equivalent to being killed by a plague.</p>
<p>Truly crazy stuff. I guess I bought the book, so the author wins in the end.</p>
We sent the worst YCombinator application possibleurn:https-kaashif-co-uk:-2022-08-02-we-sent-the-worst-ycombinator-application-possible2022-08-02T00:00:00Z2022-08-02T00:00:00ZKaashif Hymabaccus
<p>There are many mistakes we made when halfheartedly trying to get
funding for our startup. The worst was that we didn't actually have a
business at all - we had no users yet due to not having regulatory
approval for our financial product, and no practical plan to get that
other than sending applications and hoping. The second worst was that
our funding applications were really bad in many ways.</p>
<p>Not having a business is obviously a bigger problem than anything
else, but being unable to convince investors is a big deal too,
especially if you're <em>really</em> bad at it.</p>
<p>Here's a list of mistakes we made in our YCombinator application.</p>
<!--more-->
<h2>Preamble: How to avoid these mistakes</h2>
<p>I watched this video:
<a href="https://www.ycombinator.com/library/6t-how-to-apply-and-succeed-at-y-combinator">https://www.ycombinator.com/library/6t-how-to-apply-and-succeed-at-y-combinator</a>
but then we made all the mistakes and did the opposite of that advice.</p>
<p>I don't know what happened in my cofounder's case, but I think my
excuse is that I was too focused on writing software rather than
building a business. It should be as easy as building a real business,
then laying out the facts so that YC can understand what you're doing.</p>
<p>I'll try to avoid much more commentary on how to succeed, since I
didn't. Instead, let's focus on how to fail, which I do know something
about.</p>
<h2>Poor spelling and grammar</h2>
<p>This should go without saying, but apparently not. My co-founder, a
non-native speaker, wrote the application and gave it to me to
proofread. I'm a native English speaker, but I was writing code on a
36 hour sleep cycle revolving around exhaustion, so I didn't spot many
of the mistakes.</p>
<p>I spotted some of the mistakes, but this just led to unproductive
arguments. For example: we had "we've done X in USA". It should be
"the USA" or, more naturally, "the US". We talked about this mistake,
but I hit my cofounder with the old "trust me I'm right" and brushed
aside any request for explanation.</p>
<p>Our final application still had the mistake.</p>
<p>I think there's something deeper going on here - if there isn't a deep
level of trust between cofounders, you can't get much done. Even if we
both had the same level of English proficiency, something else
would've gone wrong down the road and produced a suboptimal outcome.</p>
<p>This is why firms exist in the first place, it's easier for incentives
to be aligned and for people to trust each other within a firm. A firm
without trust may as well not exist.</p>
<h2>Stretching the definition of user</h2>
<p>An example: how many active users did we have? None, since we didn't
have regulatory approval yet. But we put down 114 users, since
hundreds of people had registered their interest and thus..."used" our
website, right? I mean that's obvious bullshittery, don't do that.</p>
<p>The worst part is that we started to treat "interest" as a key
performance indicator (KPI). We got excited and thought we were
accomplishing something when thousands of people came to our site and
signed up.</p>
<p>We hadn't done anything! We shouldn't have believed our own lies about
user numbers. There's only one kind of user that matters, and that's
one that's <em>giving you money</em>, because revenue and profit are the only
real KPIs.</p>
<h2>Embellishing progress</h2>
<p>We had contacted several regulators about approval of our product. We
had something that seemed to be workable. That was good. But we hadn't
received approval from anyone yet. Indeed, we hadn't received
<em>replies</em> from any regulators yet at the time of sending the
application.</p>
<p>We said we were in "talks" with regulators, and wrote a long list,
including regulators we hadn't contacted yet but we "surely" would. We
of course didn't indicate that we hadn't contacted all regulators in
our list.</p>
<p>We also claimed to have a minimal viable product (MVP), and that was
true from a technical perspective. But from a regulatory perspective
(the important one), it was unclear that our methods of sending money
would be allowed. So it wasn't viable at all. We still claimed to have
an MVP. This was obviously false, since we didn't have any paying
users.</p>
<h2>Delusional projections</h2>
<p>An obsession with projections many years out is always a bad sign, and
we didn't have that. We had to come up with some numbers for the "How
much could you make?" question for the YC application, and there were
some facts (the size of the market, average fees) but the "possible"
market share we picked was completely made up.</p>
<p>We also made up a future growth rate (modest, we thought) and
"demonstrated" that we'd be making billions within 5 years.</p>
<p>We knew the projections were unrealistic, but then something funny
happened: we started to believe them just a bit. The act of projecting
that we'd be billionaires made us think it'd be true, to a tiny
extent. It's almost literally like writing "I'll be a billionaire"
into a cell in Excel - meaningless in reality, but it may have some
psychological effect.</p>
<p>This made us confident (for no reason) in our projected billions
despite being pre-revenue. I can't help but think this kind of
delusion has to hurt an application, it sure was obvious in how we
phrased our predictions.</p>
<h2>Complete humour incompatibility</h2>
<p>This is important. I can laugh at something with my friends and say
"that's fucking hilarious, can you believe that?" My cofounder and I
never did that.</p>
<p>This made the "tell us something surprising or amusing" question very
difficult.</p>
<p>I made a suggestion - it's literally cheaper to fly to some countries
with a suitcase full of cash than it is to send money
electronically. And it'll even be faster! And there are flights on
weekends, but sometimes no bank transfers! Crazy, right?</p>
<p>We couldn't find a single thing we both found funny, and ended up
writing that we'd both started and quit (different) degree programs.</p>
<p>Not being able to agree on something interesting or funny was in
itself interesting to me. I wasn't able to run a company with someone
I hadn't shared a real laugh with, despite not thinking that was
particularly important beforehand.</p>
<p>A good lesson for me, I think.</p>
<h2>Conclusion</h2>
<p>Being friends with your cofounder is far, far, far more important than
literally anything else. Doesn't matter what skills either of you
have, whether they complement each other, whether you have the right
backgrounds, etc.</p>
<p>The company can't survive if the cofounders don't like and trust each
other. That means the founders should've known each other for a while
and ideally have built something non-trivial together, so that they
can trust in each other's abilities.</p>
<p>I think YC's
<a href="https://www.ycombinator.com/cofounder-matching">co-founder matching</a>
is very stupid if you have <em>any</em> other choices. It should be a last
desperate resort when you've asked <em>everyone</em> you know, and everyone
they know too. It probably won't work and in the worst case, you only
realise it's not working after months of work and slowly building
distrust.</p>
What does it mean for someone to "deserve" success?urn:https-kaashif-co-uk:-2022-07-31-what-does-it-mean-for-someone-to-deserve-success2022-07-31T00:00:00Z2022-07-31T00:00:00ZKaashif Hymabaccus
<p>I recently read a couple of blog posts about deserving success, and I
found them very interesting, mostly because of what they tell me about
the people writing them.</p>
<p>I have some thoughts on these posts but reading them back, I think
they're nonsense. I'll post them anyway. Here are the posts:</p>
<ul>
<li><p>One from someone who hates the word "deserve" because they believe
luck plays a much larger part in success than people want to
believe: <a href="https://moontowermeta.com/my-personal-trigger/">https://moontowermeta.com/my-personal-trigger/</a></p></li>
<li><p>A follow up from the same author on why talking about "deserving"
things makes their skin crawl:
<a href="https://moontowermeta.com/why-deserve-makes-my-skin-crawl/">https://moontowermeta.com/why-deserve-makes-my-skin-crawl/</a></p></li>
<li><p>And a series of blog posts where the author argues they don't
deserve their success, full socialism wouldn't work, and that the
way to make things better doesn't involve taxing him more:
<a href="https://russroberts.medium.com/do-i-deserve-what-i-have-part-i-6553091dd85c">https://russroberts.medium.com/do-i-deserve-what-i-have-part-i-6553091dd85c</a></p></li>
</ul>
<p>The first two are written by an options trader and the last by an
economist, both wealthy. Both seem to feel it's obvious that they
don't "deserve" their success, but neither seem to actually attempt to
define whatever it is they're talking about.</p>
<p>I don't think it's possible to define "deserve" in a way that matches
our intuitions but also means those two people don't deserve their
success at all.</p>
<p>I'll try to define "deserve" but it'll probably go really badly.</p>
<!--more-->
<h2>Did you deserve to be born?</h2>
<p>Russ Roberts (the economist) seems to take it for granted that you
can't deserve something if you had to be lucky to get it:</p>
<blockquote><p>I work pretty hard at what I do. But do I deserve credit for
perseverance or grit? Or are they just another part of my genetic
inheritance? So hard to say.</p>
<p>So do I deserve the life I have?</p>
<p>Of course not. I am so lucky.</p></blockquote>
<p>He also considers it part of his "luck" that he was born to loving
parents in a rich country. Putting these two together, you had to be
lucky to be born at all, and therefore can't deserve anything. I don't
think that's what he actually believes though.</p>
<p>The Moontower guy says something even more extreme:</p>
<blockquote><p>At a society level, appreciating the role of chance is ultimately
about empathy. It's the recognition that you could have hatched from
an egg anywhere in the world in any time in history. Our policies
should not amplify the extremes of cosmic dice but instead balance
them.</p></blockquote>
<p>Other people were/will be born, but the chance that I "could have"
been them is exactly zero for any reasonable definition of "I" and
"them". The only way this makes sense is if he believes that before a
fetus becomes conscious, there's a random selection process among some
set of possible consciousnesses and one "wins".</p>
<p>That may be a useful device for thinking up rules for a society (see
the
<a href="https://en.wikipedia.org/wiki/Original_position">veil of ignorance</a> -
the idea that you should construct rules for a society without
knowledge of where you'd end up in that society), mainly because doing
that ensures that everyone will accept the rules as "fair".</p>
<p>But it's just a thought experiment, there aren't actually (as far as
we know) free-floating consciousnesses that end up randomly embedded
in babies. It's bizarre to me to state that you could have been
another person, because I couldn't have been! What does that actually
mean? I don't think he'd be able to even explain that if pressed.</p>
<p>Both blog post authors say things that lead me to believe they can't
think anyone deserves anything at all. I think the Moontower guy
really does think he believes that, but he may hesitate to apply his
principle to the bitter logical end.</p>
<h2>People who "obviously" deserve things</h2>
<p>I generally have an intuition about what the results should be and try
to work out what my principles really are based on that. I'll try to
answer some questions using (1) my intuition and (2) the "cosmic dice"
idea where no-one deserves anything.</p>
<p>I'll use the answers to (1) to try to work out what I mean by
"deserve".</p>
<p>I am constructing a strawman out of (2) and will intentionally make
the answers ridiculous.</p>
<h3>Do rich people deserve success?</h3>
<ol>
<li><p>It depends on how they became rich. Stealing? No. The lottery?
No. Producing something valuable to others? Yes. Trading stocks?
Maybe, if they actually had reasons behind their trades and those
reasons were correct.</p></li>
<li><p>No, they were lucky to be born as a person with those abilities or
in that circumstance. Next.</p></li>
</ol>
<p>That's too vague and not distasteful enough on either side to be
interesting.</p>
<h3>Did the architect of the Holocaust, Reinhard Heydrich, deserve assassination?</h3>
<ol>
<li><p>Yes, he did something terrible and deserved to die.</p></li>
<li><p>No, he was unlucky enough to be born as a person predisposed to
genocide, and didn't deserve assassination.</p></li>
</ol>
<p>Note that this is different from saying that he shouldn't be
assassinated, it's just saying that he doesn't deserve it. I don't
know if that makes sense or not, or when I should start talking about
"cosmic dice".</p>
<h3>Do welfare recipients deserve the money?</h3>
<ol>
<li><p>Maybe, depending on whether they really need it or not. A single
mother who's just been walked out on with kids to feed?
Yes. Someone gaming the system and living in subsidised housing
when they don't need it? No.</p></li>
<li><p>No, because, uh...I'm actually finding it pretty hard to decide
what strawman to place here. "They don't deserve welfare" is
basically synonymous with "don't give them welfare", and both
authors clearly think we <em>should</em> support the poor. But you still
have to get lucky enough to live in a country with a welfare state
to get welfare, so presumably they don't think recipients deserve
it, but that we should still do it.</p></li>
</ol>
<h3>Do poor people deserve to be poor?</h3>
<ol>
<li><p>It depends. Lied to a lot of people and got sued for everything?
You probably deserve poverty. Murdered people and now you can't get
a job? I think that's deserved. Born into poverty and abuse? That
seems undeserved.</p></li>
<li><p>This one's easy since they explicitly state it - no, poor people
don't deserve to be poor. Even ones that are there due to their own
actions are really there due to the initial conditions of their
lives, which were out of their control.</p></li>
</ol>
<h2>Cheaters don't deserve to win</h2>
<p>People in the real world think others deserve things more or less
depending on how much their actions affected the results, and
crucially, <strong>whether they played by the rules</strong>.</p>
<p>Cheats never deserve anything. Even if they spent a lot of time and
effort honing their cheating skills.</p>
<p>Here are some examples:</p>
<ul>
<li><p>People who cheat at board games don't deserve to win. Opinions about
this may vary depending on which rules are the rules we're going
by. The cheater has broken the game rules, but not the law. But
their win was entirely due to their actions, and there might've been
little or no luck involved. They still cheated.</p></li>
<li><p>Bank robbers don't deserve the money they steal. This is despite
going to all that trouble to plan and execute a heist, which is a
lot of work. If they don't "get lucky" and just planned really
well, don't they deserve the money? Well no, because they're
breaking a lot of rules.</p></li>
</ul>
<p>That helps us find more people who don't deserve things, but that
doesn't separate me from a cosmic dice believer who thinks no-one
deserves anything.</p>
<h2>Getting some desert</h2>
<p>Deserving things is a spectrum. The two authors claim they don't
deserve their success, but that's not a useful way to think about the
world and doesn't match anyone's intuition. I don't even think it
matches their intuitions.</p>
<p>I think a lot of people (maybe the authors included) would agree with
the following statements:</p>
<ul>
<li><p>People who earn income by working deserve it more than those who
inherited wealth.</p></li>
<li><p>The guilty deserve punishment more than the innocent.</p></li>
</ul>
<p>This is all possible without making the absolute statement "murderers
deserve punishment" or "successful traders deserve success", which the
authors must disagree with.</p>
<p>Desert is a half open interval ranging from zero to 100%. We can all
think of people who don't deserve things at all and people who deserve
things more than others, but so much is an accident of birth it's
impossible to think of anything that is absolutely 100% deserved.</p>
<h2>Summary</h2>
<p>I don't know if any of that made sense, but I'll still go on saying
that I deserve X or that others deserve Y, as long as they played by
the rules and didn't just gamble their way to success.</p>
<p>That's a definition of what it means to deserve, not a value
judgement, by the way.</p>
I'm moving to New York, here's whyurn:https-kaashif-co-uk:-2022-06-05-i-m-moving-to-new-york-here-s-why2022-06-05T00:00:00Z2022-06-05T00:00:00ZKaashif Hymabaccus
<p>No-one reads this blog, so this is a safe place to make a public
announcement I don't really want anyone to see.</p>
<p>I'll be moving to New York in a few months when my visa goes through.</p>
<p>There are many reasons I'm moving to New York, the main ones are:</p>
<ul>
<li><p>More money</p></li>
<li><p>Looking to run away from my startup failure</p></li>
<li><p>And the main one, I'm looking for something cool to do because
I'm bored.</p></li>
</ul>
<p>Read on for some elaboration.</p>
<!-- more -->
<p>Here's the elaboration: \$\$\$\$\$\$\$\$\$\$\$\$. I think that about sums it up.</p>
<p>But seriously, here are some reasons I want to get out of the UK and
into the US.</p>
<h2>More money</h2>
<p>Software engineers are paid significantly more in the US than in the
UK, especially in hubs like San Francisco and New York. It's not even
close. The best career move any software engineer can make is moving
to the US.</p>
<p>Salaries in Europe are generally disgraceful. My salary isn't bad, but
there is no reason I should be leaving hundreds of thousands of
dollars on the table to stay in the UK. The percentage raise from
switching jobs would be significantly less than the raise from just
moving to the US within the same company.</p>
<p>Maybe if I had a family I'd stay here, but I don't. I think that's the
only case where millions or billions of dollars in lifetime earnings
is a price people would actually pay.</p>
<h2>Running away from my problems</h2>
<p>I tried to be a solo startup founder for a bit, all my ideas were
ultimately stupid and I went about evaluating them wrong way (see
earlier blog posts on this).</p>
<p>I then tried to co-found a payments startup with someone I
communicated with mainly online, but that didn't work out since we
just weren't close enough to really trust each other or be able to
support each other. That meant as things got worse, neither of us
really had anyone to really open up to. Or, at least, I didn't.</p>
<p>After that didn't work out, I went back to my old job with the express
purpose of moving to the US.</p>
<p>The problem I'm trying to fix is that it feels like I took a huge leap
into the unknown (founding a startup), cracked my skull against a wall I
should've seen (bad solo strategy, wrong co-founder), and was about to
fall back into the same old job doing the same old stuff, living in
the same old place.</p>
<p>The only way I can convince myself my life is going in an interesting
direction (i.e. is worth living) is to do something drastic.</p>
<h2>I'm bored</h2>
<p>Is this really what life is? Just going around doing the same types of
things until you die? The least I could do is change it up, a quarter
of a century in more or less one place is enough.</p>
<p>I know people who are gearing up to buy a house and settle down. I
could buy a house, but that feels more like tying a noose around my
neck - tying up all of my capital in a single asset I don't even
really need or want.</p>
<p>Once you have a house, that's it, you're tied down. The asset isn't
particularly liquid, so there's a barrier to escape.</p>
<p>Settling down sounds like something a corpse with lead weights tied to
it does at the bottom of a lake. People with partners and families see
things differently, no doubt, but I don't have those.</p>
<h2>I'm a practising neoliberal</h2>
<p>It boils down to this. My religion, my ideology is liberalism. I truly
believe that immigration and the price system allow workers to be
allocated where they will be the most productive, and that workers
actually do follow these signals.</p>
<p>In order to practise what I preach, I have to look at what the labor
market is telling me and act accordingly.</p>
<h2>Other options I considered</h2>
<p>I considered going to work for CERN, who I did interview with several
times while working at my job the first time around. Ultimately I gave
up on that once I ran the numbers and saw the effect on my lifetime
earnings. CERN pays well, but not as well as some companies in the US,
or even Switzerland.</p>
<p>I considered trying another startup, but unfortunately that felt like
more of the same. Founding a startup doesn't feel great a lot of the
time, and having the right co-founder for mutual support (and/or
shared delusion) is extremely valuable. And there's a huge pool of
investors and co-founders I haven't tapped, across the pond in the
US. I think the kind of borderline delusional conviction a startup
founder needs is more common in the US. It's something I used to have
and could build up again.</p>
<p>I also considered <em>working</em> at a startup, but that would've delayed my
move to the US by at least a year. You need a year at the company to
L1B visa into the US, and my 2 years at Bloomberg still count despite
the gap. And I would've probably jumped at an okay looking offer just
to have the company go bankrupt a few months later.</p>
<h2>Conclusion</h2>
<p><a href="https://www.youtube.com/watch?v=EEjq8ZoyXuQ">https://www.youtube.com/watch?v=EEjq8ZoyXuQ</a></p>
LeetCode on a Z80 CPU from 1976urn:https-kaashif-co-uk:-2022-02-10-leetcode-on-a-z80-cpu-from-19762022-02-10T00:00:00Z2022-02-10T00:00:00ZKaashif Hymabaccus
<p>A while ago, I soldered together a Z80 homebrew computer and ran some
programs on it, then
<a href="https://kaashif.co.uk/2019/07/10/my-first-homebrew-computer/">wrote a blog post about it</a>.</p>
<p>I did end up running some toy programs and some BASIC games like Super
Star Trek, but that got me thinking - how hard would it be to solve a
real algorithmic problem on it?</p>
<p>It turns out it wasn't hard at all, thanks to
<a href="https://z88dk.org/site/">the Z80 development kit (z88dk)</a> and the
<a href="https://github.com/RC2014Z80/RC2014/tree/master/BASIC-Programs/hexload">hexload BASIC program</a>
that allows running arbitrary binaries from BASIC, without needing to
reprogram the ROM. That's very handy since I don't have a ROM
programmer!</p>
<p>Comparing compiled Z80 assembly to modern x86 assembly for the same C
program is pretty cool - the Z80 is a slightly extended 8080
instruction set, so you might expect the programs to look pretty
similar, but x86 is really a whole different beast.</p>
<!--more-->
<h2>Step 0: Building the computer</h2>
<p>Step 0 is building the computer itself - look at that blog post I
linked above for some tips, but I really recommend checking out the
<a href="https://rc2014.co.uk/">RC2014 website</a> for the latest advice.</p>
<p>The computer has 8K ROM, 32K RAM, runs at 7.3728MHz and communicates
over serial at 115,200 baud. It's not exactly a speed demon, but it's
surprising what such a "slow" computer can do - that's still 7 million
clock cycles per second.</p>
<h2>Step 1: Writing the program</h2>
<p>The problem I chose was the eternal classic
<a href="https://leetcode.com/problems/two-sum/">Two Sum</a>. The problem is as
follows:</p>
<blockquote><p>Given an array of integers nums and an integer target, return
indices of the two numbers such that they add up to target.</p>
<p>You may assume that each input would have exactly one solution, and
you may not use the same element twice.</p>
<p>You can return the answer in any order.</p></blockquote>
<p>This is a really simple problem with an easy O(n<sup>2</sup>) solution, and a
slightly less easy O(n) solution. The quadratic solution is to loop
over all pairs and check them: this requires no extra data structures
and indeed no extra libraries, it's just looping over an array.</p>
<p>The linear solution is to construct a hash set of the numbers in the
array then loop over the array, checking if the required summand to
get to the target result is in the array.</p>
<p>The linear solution isn't all that complicated, but involves a lot of
extra stuff (there's no C standard library hash set). Here's my
quadratic solution:</p>
<div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdlib.h></span><span class="cp"></span>
<span class="kt">void</span><span class="w"> </span><span class="nf">two_sum</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">target</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">length</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="o">*</span><span class="n">nums</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">length</span><span class="mi">-1</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">length</span><span class="p">;</span><span class="w"> </span><span class="n">j</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">nums</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">nums</span><span class="p">[</span><span class="n">j</span><span class="p">]</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">target</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"%d, %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">j</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"twosum</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">target</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">scanf</span><span class="p">(</span><span class="s">"%d"</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">target</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">length</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">scanf</span><span class="p">(</span><span class="s">"%d"</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">length</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="o">*</span><span class="n">nums</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">calloc</span><span class="p">(</span><span class="n">length</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="kt">int</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">length</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">scanf</span><span class="p">(</span><span class="s">"%d"</span><span class="p">,</span><span class="w"> </span><span class="n">nums</span><span class="o">+</span><span class="n">i</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">two_sum</span><span class="p">(</span><span class="n">target</span><span class="p">,</span><span class="w"> </span><span class="n">length</span><span class="p">,</span><span class="w"> </span><span class="n">nums</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">free</span><span class="p">(</span><span class="n">nums</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>The first thing to notice is that I'm just using C standard library
functions like printf and calloc - z88dk lets you write "normal" C
code almost as if you're writing code for a PC. The difference is that
the z88dk standard library is mostly hand-written Z80 assembly, with
over 250k lines of code.</p>
<p>Take a look at
<a href="https://github.com/z88dk/z88dk/tree/master/libsrc">the z88dk source on GitHub</a>,
it's crazy the amount of work that's gone into this. They claim it's
the largest repo of Z80 assembler online, and I believe them.</p>
<p>That means we literally can just compile the same program for Z80 and
x86 and compare them!</p>
<h2>Step 2: Compiling the program</h2>
<p>Once you install z88dk, you'll have the zcc compiler. Make sure to run
<code>make install</code> to install the compiler to <code>/usr/local</code>, and you can
invoke it with <code>zcc</code>. Full instructions are here:
<a href="https://github.com/z88dk/z88dk/wiki/installation">https://github.com/z88dk/z88dk/wiki/installation</a></p>
<p>Assuming the source code above is in <code>twosum.c</code>, you can compile it
with this:</p>
<pre><code>$ zcc +rc2014 -subtype=basic -clib=new twosum.c -o twosum -create-app --c-code-in-asm
</code></pre>
<p>This uses the "new" C library. I couldn't get the recommended command
from the hexload instructions page to work, which used
<code>-clib=sdcc_iy</code>, it complained of a missing binary called <code>zsdcpp</code> -
presumably a C preprocessor. Even after symlinking a C preprocessor to
have that name, I was still getting errors, so I gave up and used the
command above, which worked great.</p>
<p>This outputs a few .bin files and an .ihx file - we are interested in
the .ihx file.</p>
<h2>Step 3: Uploading the program to the computer</h2>
<p>I have an RC2014 running Microsoft BASIC from 1978 and a serial
connection. There's no way to run an HTTP server on my PC and download
the files to the RC2014 or copy the file from a disk - there isn't
even any non-volatile storage except the ROM, and certainly no
network.</p>
<p>This is where hexload comes in:
<a href="https://github.com/RC2014Z80/RC2014/tree/master/BASIC-Programs/hexload">https://github.com/RC2014Z80/RC2014/tree/master/BASIC-Programs/hexload</a>.</p>
<p>Hexload is a program that allows you to transfer binaries encoded in
Intel HEX format (which is just hex encoded binary data with some
metadata and other stuff, see the gritty details
<a href="https://en.wikipedia.org/wiki/Intel_HEX">here</a>) over a serial line to
a computer running Microsoft BASIC, then run them.</p>
<p>The key mechanism behind this is that, after we write the binary to
memory, BASIC has a command <code>USR(x)</code> that tells the CPU to jump to an
address with a user program and start executing. That is, we boot to
BASIC, load the program into memory with hexload, then run</p>
<pre><code>PRINT USR(0)
</code></pre>
<p>to run our program. This is pretty exciting. Full instructions are on
the hexload page linked above, including the slowprint.py program
needed to make sure the program isn't printed too fast (which would
cause characters to be dropped).</p>
<p>First, I connect the serial cable, power on the RC2014, then connect
with screen to my USB FTDI serial cable (sudo since I couldn't be
bothered to fix permissions on the device file):</p>
<pre><code>$ sudo screen /dev/ttyUSB0 115200
</code></pre>
<p>Then we press the reset button on the clock board or on the backplane
to show the boot prompt:</p>
<pre><code>Z80 SBC By Grant Searle
Memory top?
</code></pre>
<p>According to the hexload instructions, I put in 35071:</p>
<pre><code>Z80 SBC By Grant Searle
Memory top? 35071
Z80 BASIC Ver 4.7b
Copyright (C) 1978 by Microsoft
1916 Bytes free
Ok
</code></pre>
<p>Now we're ready to send a BASIC program over the wire. For that we can
use
<a href="https://github.com/RC2014Z80/RC2014/blob/master/BASIC-Programs/hexload/slowprint.py">slowprint.py</a>.</p>
<p>First we send hexload over:</p>
<pre><code>$ python3 slowprint.py < hexload.bas | sudo tee /dev/ttyUSB0
</code></pre>
<p>I use tee so we can see if there are any discrepancies between what's
sent and received. A few times one or two characters were missed and
there was an SN Error in BASIC. If you get that, reset the RC2014 and
try again, it'll probably work next time.</p>
<p>Once that finishes, you'l be greeted with this:</p>
<pre><code>Loading Data
Start Address: 8900
End Address: 89D7
USR(0) -> HexLoad
HEX LOADER by Filippo Bergamasco & feilipu for z88dk
:
</code></pre>
<p>And now we can start sending over our two sum program. Run this:</p>
<pre><code>$ python3 slowprint.py < twosum.ihx | sudo tee /dev/ttyUSB0
</code></pre>
<p>The upload may take a while, but after it's done, the program should
automatically start executing and we can input our example (the first
example from LeetCode):</p>
<pre><code>twosum
9
4
2
7
11
15
0, 1
0
Ok
</code></pre>
<p>That is, the target was 9 and the input array was [2,7,11,15]. We got
0, 1, which is correct - indices 0 and 1 are 2 and 7 which do add up
to 9.</p>
<h2>Z80 vs x86</h2>
<p>That's great, but what does the machine code look like compared to the
devil we know - x86? And how big is it?</p>
<p>For comparison, I compiled twosum.c on my PC, optimizing binary size,
with glibc and gcc 11:</p>
<pre><code>$ cc twosum.c -g -Os -o twosum.x86
$ objdump -M intel -S twosum.x86
</code></pre>
<p>This prints the assembly we get from compiling twosum.c on a PC. The
most interesting part to me is the two_sum function, since it should
be rather similar - a few for loops, a branch, and a function
call. How big could the differences be? I won't paste both binaries
here in full, but there are a few interesting points.</p>
<h3>Using the stack is horrific</h3>
<p>On Z80, only push and pop can read or write directly to the stack. You
may wonder why we can't just do:</p>
<pre><code>ld hl, (sp)
</code></pre>
<p>to load whatever's at the stack pointer into the hl register, and the
answer is that you just can't - the Z80 had to maintain binary
compatibility with the Intel 8080 (an 8-bit CPU) and didn't have that
much freedom in how it could extend the ld instruction. That's why,
when we want to read something from the stack into hl, we end up doing
this:</p>
<pre><code>pop hl
push hl
</code></pre>
<p>which does a whole lot of reading and writing to the stack just to
achieve the effect of reading the top of the stack into
hl. Performance-focused Z80 programmers wrote assembler by hand and
avoided doing things like this. One might even say that C was too slow
since compilers weren't advanced enough to beat the best assembler
programmers.</p>
<p>The x86 code didn't even use the stack at all - we have a full 16
8-byte registers available, all general purpose. There are even more
non-general purpose registers available like the stack pointer, base
pointer, instruction pointer, various SIMD registers, etc.</p>
<p>This is in comparison to the Z80 which only really has 4 16-bit
general purpose registers, and not all of them can be used with all
instructions.</p>
<p>As an example, say we want to compile this statement in C:</p>
<div class="highlight"><pre><span></span><span class="kt">int</span><span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>(part of the for loop in the two_sum function)</p>
<p>In x86 it's simple:</p>
<pre><code>lea r9,[rax+0x1]
</code></pre>
<p>We say we're keeping j in r9 and i in rax, we write rax+1 to
r9. Easy.</p>
<p>With the Z80 we're severely constrained in how many registers there
are, and we have to keep i and j both on the stack.</p>
<pre><code>pop hl
push hl
inc hl
push hl
</code></pre>
<p>So i was at the top of the stack. we set hl to i with the pop/push
dance, increment hl, and push hl to the stack, so we're keeping j at
the top of the stack.</p>
<p>I can see how all these extra registers make life easier.</p>
<h3>Optimising for size hurts speed, a lot</h3>
<p>This may seem obvious, but on the Z80 this is true to an extreme. When
you have a limited space, like 32K of memory, you have to work hard to
fit BASIC and whatever your program is into memory. That means it's
often required to call a function where x86 (with no concern about
space) can just inline.</p>
<p>One example is the l_gint function, which you can see here:
<a href="https://github.com/z88dk/z88dk/blob/87003c95c1f3d9be8d4704beff94010159989ec2/libsrc/_DEVELOPMENT/l/sccz80/9-common/l_gint.asm">https://github.com/z88dk/z88dk/blob/87003c95c1f3d9be8d4704beff94010159989ec2/libsrc/_DEVELOPMENT/l/sccz80/9-common/l_gint.asm</a>. It's
so trivial as to be ridiculous, it's just</p>
<pre><code>l_gint:
ld a,(hl+)
ld h,(hl)
ld l,a
ret
</code></pre>
<p>This just does some loads and increments hl. Why a function? Because
this sequence of operations is really common, and it makes for a much
smaller binary if these operations are called as a function rather
than inlined. This is absolutely required, as the Z80's primitive
instruction set already makes it incredibly difficult to get anything
done in a reasonable number of instructions.</p>
<p>As an example, take this line of C:</p>
<div class="highlight"><pre><span></span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">nums</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">nums</span><span class="p">[</span><span class="n">j</span><span class="p">]</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">target</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
</pre></div>
<p>This is just reading from some memory locations, adding, comparing and
perhaps jumping. Easy. Yes, on x86:</p>
<pre><code>mov r8d,DWORD PTR [rdx+rcx*4]
add r8d,DWORD PTR [rdx+rax*4]
cmp r8d,edi
jne 1221 <two_sum+0x18>
</code></pre>
<p>We read the two values from memory, indexing into the array with the
lovely CISC mov and add instructions, and jump past the printf if the
sum isn't the target.</p>
<p>On Z80, it's a complete nightmare, and would be even worse without
these l_gint-style functions to reduce the size:</p>
<pre><code> ld hl,6 ;const
call l_gintspsp ;
ld hl,4 ;const
add hl,sp
call l_gint ;
add hl,hl
pop de
add hl,de
ld e,(hl)
inc hl
ld d,(hl)
push de
ld hl,8 ;const
call l_gintspsp ;
ld hl,4 ;const
add hl,sp
call l_gint ;
add hl,hl
pop de
add hl,de
call l_gint ;
pop de
add hl,de
ex de,hl
ld hl,10 ;const
add hl,sp
call l_gint ;
call l_eq
jp nc,i_8
</code></pre>
<p>All these function calls are slow, even if they save space. By the
way, Z80 assembly is a complete disaster to write by hand. x86
assembly is much nicer in comparison.</p>
<p>It's important to note: the space optimizations work very well and
although the code looks terrible for the Z80, it's actually much
smaller:</p>
<pre><code>$ du -h twosum.bin
8.0K twosum.bin
$ du -h twosum.x86
20K twosum.x86
</code></pre>
<p>That's 8K for the Z80 code and 20K for the x86 code.</p>
<h2>Conclusion</h2>
<p>It is incredible that anyone ever wrote software for the Z80 in pure
assembler. Hats off to Bill Gates and Paul Allen for actually starting
a company based on software written for the Intel 8080, in assembler.</p>
<p>Like holy shit, Altair BASIC fit into 4K of memory and was an actual
programming language kids could write programs in!</p>
<p>I could have maybe stomached writing software in assembler once more
advanced 16-bit CPUs like the 8086 or 286 came around, but I'm afraid
that writing real programs for the Z80 or even worse, 6502, seems
really tedious.</p>
<p>I'm in shock that the much more powerful Motorola 68000 didn't become
the standard.</p>
<p>Anyway, we've come a long way in 50 years, maybe in another 50 people
will wonder how we coped with Python and Java.</p>
Don't stick to what you're good at - my startup catastrophesurn:https-kaashif-co-uk:-2022-02-01-don-t-stick-to-what-you-re-good-at--my-startup-catastrophes2022-02-01T00:00:00Z2022-02-01T00:00:00ZKaashif Hymabaccus
<p>The conventional wisdom is that if you're starting a business you
should do something you're good at or know something about. If you're
a software engineer, you should probably be the guy running
engineering. If you're in sales, you should probably be the sales
guy. I mean, that makes sense.</p>
<p>This is going to sound stupid and obvious, but people have a strong
bias towards doing things they enjoy and are good at. And when you're
doing those things it feels good and even worse, it feels
<em>productive</em>.</p>
<p>Why is that bad? Because feeling productive can have almost no
correlation to <em>being</em> productive.</p>
<p>I quit my job to do things that felt more productive. Spoiler alert:
they were dismal failures and produced nothing except some bitter
lessons! Isn't that fun?</p>
<p>I'm going to dissect some of my failed businesses:</p>
<ol>
<li><p>I tried to re-sell NordVPN activation keys, passing on some of the
bulk discount. What could go wrong - people already buy NordVPN and
this is just NordVPN <em>but cheaper</em>, right? Not quite, which was a
rather painful lesson.</p></li>
<li><p>I co-founded a remittance service (think TransferWise, Western
Union) that was faster, cheaper, and aimed mainly at transfers from
the UK to under-served African countries. We had people lining up to
use it, we had all of the tech in place to actually provide the
service, it was 10x cheaper, 10x faster, worked on weekends, and
the Financial Conduct Authority would've never approved it. Oh.</p></li>
</ol>
<p>Why did I keep (or even start) working on them? Because writing
software <em>felt good</em>. It felt productive. But it wasn't! I'll go
through what I think the warning signs were and how you (and I) can
avoid making the same mistakes.</p>
<!--more-->
<h2>Warning sign 0: I hadn't talked to users before building software</h2>
<p>When starting out with my VPN resale business, the first thing I did
was build the website and set up payments with Stripe. That seems
reasonable, doesn't it? I thought that people would love to buy a
cheaper NordVPN plan. After all, NordVPN has lots of customers, so if
I sell literally the same product but cheaper, I'll sell more. The
only problem is getting the word out, right?</p>
<p>No, there's something missing from that plan: actually talking to
potential users and seeing if they want it! After building the
website, I did talk to several potential users and saw some gigantic
red flags that I didn't recognise. Here are some people I talked to:</p>
<ol>
<li><p>Myself! I already have a VPN, but I bought like a 5 year
subscription to another provider. I didn't even want to buy my own
product since I already had a VPN and didn't need another.</p></li>
<li><p>Friends: my friends with VPNs weren't interested since
they...already had VPNs, and my friends without didn't care. But I
pitched my idea to them and they said "That sounds great, the
potential market is huge!" "NordVPN is great, I hear they make a
lot of money!" but none of them bought anything from me.</p></li>
<li><p>People on Twitter: I ran a few ads on Twitter and some people
retweeted, trying to alert NordVPN to the scam I was running, since
it's obviously impossible that I could be selling for <em>cheaper than
buying directly</em>. No-one bought anything, even after I contacted
them and explained how I was doing it (NordVPN gives discounts to
resellers who buy in bulk).</p></li>
<li><p>People on Reddit: I was banned from some subs for shilling my
product. I tried to be subtle about it but I got caught! Whoops.</p></li>
</ol>
<p>I never encountered anyone who actually wanted to buy the product,
despite encountering many people who literally said to me that they
"would" buy it. You should at least find one user <em>before</em> you build
the product! And it is crucial that they agree to pay you, and then
actually <em>do</em> pay you at some point, likely after you build the
product.</p>
<p>I stuck with this VPN reselling idea because I enjoyed building it. I
enjoyed building a web app, I enjoyed integrating with NordVPN's API,
I enjoyed integrating with Stripe and PayPal. I should've tried to
find users and got them to buy something before I spent all this time
building software.</p>
<p>Building software felt productive, but didn't produce anything
valuable. In fact, the value was negative since I spent weeks doing
work that was ultimately just thrown away. If I had just spoken to
people and really tried to find customers before building anything,
I'd have avoided that wasted time.</p>
<p>Taking a day to speak to people and avoid two weeks of waste is an
insanely high level of productivity. I knew talking to users was
important, but I guess I didn't really get it until I felt that
sinking feeling of despair when I realised I had wasted weeks of
effort.</p>
<p>Is there a magic way I could've realised I was wasting my time?
Yes. It turns out I just had to believe my own eyes.</p>
<h2>Warning sign 1: I didn't believe my KPI</h2>
<p>KPI means Key Performance Indicator.</p>
<p>For a startup where you're selling something (a service, a product),
the only KPI that really makes sense is revenue. You can count users,
hits, downloads, etc, all you want, but those don't mean anything
unless they <em>pay you</em>. If they're not paying you, you aren't
validating the core hypothesis which is that people will pay you for
something!</p>
<p>You might think it's enough for people to say "I would pay for that"
but it's not! They're lying! Read this book:
<a href="http://momtestbook.com/">http://momtestbook.com/</a>. Your KPI is revenue, your experiments
should be simple to evaluate: success means revenue goes up, failure
means it doesn't. Vague non-committal compliments should not be taken
as a good sign.</p>
<p>Yes, that means you are failing whenever your revenue isn't
growing. It feels bad to realise this. It's easier to convince
yourself that you built this thing, and that's progress. That's what I
did: my eyes told me my KPI was zero, but I chose not to believe
them. Instead, I was measuring my progress by the amount of work I had
put into my minimal viable product (MVP). After all, your MVP needs to
scale, and it needs to work for complete strangers.</p>
<p>Wrong! So wrong! Don't measure inputs (time, code), measure outputs
(revenue)!</p>
<p>MVPs don't even have to involve code. I later realised that I could
just ask friends and family to pay me for a VPN plan via a Stripe
invoice, and I could send them the key. The first sale would've moved
the needle and been real progress - it would've increased the
KPI. After weeks of intensive coding, I had that epiphany and just
started to try to sell by just speaking to people.</p>
<p>I asked several people to buy a plan and they all said no. Not in as
many words: some people changed the topic, deflected, claimed they'd
buy at an unspecified future date, and so on. These are all
no. Rejection hurts so I moved on to finishing the site, and
plastering Google, Reddit, and Twitter with ads. Thousands of people
went to my site, hundreds went to the checkout page, no-one bought
anything. Interaction with users led me to believe that everyone
thought it was some kind of scam, or they weren't actually interested.</p>
<p>From another perspective, I ran thousands of experiments, all
indicating my hypothesis that people would buy VPNs from some random
site just because it was cheaper was false. I even tried planting
trees with a portion of the revenue, but that didn't work either -
that's just a nonsense gimmick or marketing ploy that doesn't really
solve a valuable problem for anyone.</p>
<p>After revenue failed to budge, I eventually learnt the lesson that if
my KPI was telling me my idea wasn't working, I should move on. Again,
the reason it took me so long to realise this was that the novelty of
running ads, building a website, building a checkout, and so on,
hadn't worn off.</p>
<p>I just like building stuff and would do it for free. So that's what I
ended up doing inadvertently.</p>
<p>This warning relates to market risk - the risk that the market won't
need or buy your product. The product itself was simple enough, the
MVP was trivial (speaking to people and getting them to buy
something). Even the website wasn't that hard to build, although it
wasn't minimal. I could easily build the product but no-one wanted it.</p>
<p>There's another kind of risk - the risk that one can't even deliver
the product. That's what I learnt about in my next attempt.</p>
<h2>Warning sign 2: High barriers to entry and product risk</h2>
<p>I moved on. While using YCombinator's Startup School forum, I met a
few possible co-founders, but only one had a problem he was solving
which I personally had. I had this problem and never even considered
that I could be the one to solve it. This sense that I could actually
improve my own life in a valuable way was intoxicating. My problem was
sending money to Africa cheaply and quickly.</p>
<p>I'm a dual citizen of the UK and Mauritius (a lovely little island in
the Indian Ocean). Sending money via banks incurs foreign exchange
fees and vaguely defined fees for whatever it is that banks do (it
turns out they do a lot). Fintech startups like Wise didn't offer
transfers to Mauritius, and the ones that did (e.g. Remitly) offered
pretty bad FX rates. Money sent on Friday would arrive the next week
despite the fact that domestic bank transfers did work on weekends.</p>
<p>This was a problem I had personally. It was a high value problem since
a lot of people send a lot of money back home, and do it regularly. We
spoke to over a hundred people (friends, family, and relatives) who
not only wanted a solution, but were willing to trust us with their
money if we could deliver. This is all great!</p>
<p>Next step: we spent a few months building the product. AAH NO! The KPI
is still at zero, we're supposed to get people to pay for
something. Why didn't we, despite the lesson I ostensibly learnt
earlier?</p>
<p>One word: regulation.</p>
<p>Consumer fintech is extremely regulated, especially in the UK. It
would have been literally illegal for us to start charging people and
moving their money abroad without registering with the Financial
Conduct Authority. To register, we had to prove that we had the
ability to build and deliver the product. They weren't concerned about
market risk, they were concerned about product risk.</p>
<p>We focused on the tech, building an Android app, a website, a backend
integrating with Plaid and Visa, bank APIs, all kinds of stuff. It all
looked good, it seemed technically possible to initiate a transfer and
actually send money quickly, using something resembling a digital
<a href="https://en.wikipedia.org/wiki/Hawala">hawala</a> system.</p>
<p>We realised too late, after months of work, that the FCA would never
approve such a product. Not one built by us, a couple of guys with no
experience in regulatory compliance and no idea what we actually had
to do to comply with the regulations. We <em>thought</em> we knew what we had
to do to comply, but as it turns out, we didn't.</p>
<p>What went wrong?</p>
<ol>
<li><p>Talking to users felt good but didn't actually move our KPI. You
can breathe only when that number goes up. Talking to users is a
basic requirement that we fulfilled, but it's not enough.</p></li>
<li><p>Building the software felt good and we forgot about our KPI. Is it
really this easy to build a working fintech service? It sure is,
technology sure is great! We must be doing well, look at all of
this code!</p></li>
<li><p>We never considered, not for a second, that we <em>couldn't</em> build the
product. We knew we had the technical ability (and were right), but
we had a big case of the Dunning-Kruger effect - being a computer
whiz doesn't actually grant you magic powers to convince the
regulator that you're totally not building a money laundering
service.</p></li>
</ol>
<p>It's tempting to blame the regulator, but there are reasons for these
regulations. We were really killed by our lack of appreciation for
product risk. We failed to seriously consider issues outside our
expertise that meant we just couldn't deliver the product.</p>
<p>It's probably for the best. I once saw a video of Michael Seibel
(co-founder of justin.tv, later called Twitch) and he said that once
you mess up with people's money, that's it. You get the Wikipedia
page with "your company messed up big time" and maybe even broke the
law. Googling your name brings up the fact that you can't be trusted
with customers' money. That's permanent, it doesn't go away, and is an
outcome that we avoided at least.</p>
<h2>Conclusion</h2>
<p>There's a clear trajectory here and clear lessons learnt, so these
experiences weren't all bad. The key points are startlingly simple, to
the point that I feel like an idiot even writing this stuff down.</p>
<p>Here's the checklist:</p>
<ul>
<li>Talk to users</li>
<li>Get them to pay you</li>
<li>Don't break the law</li>
</ul>
<p>Here are some things that you shouldn't do unless ABSOLUTELY
necessary:</p>
<ul>
<li>Build software</li>
<li>Run ads</li>
<li>Spend money</li>
</ul>
<p>Note that this stuff seems somewhat easier if you have a B2B startup,
since the number of users to speak to is usually much smaller, and
they are much more willing to pay you if your product is valuable to
them. I haven't personally done that, maybe I'll try that at some
point.</p>
<p>I console myself with something I saw in a lecture from Steve Jobs (I
know, I know): you can't fake scar tissue. I've got a lot of very real
(figurative) scar tissue now, and I got it all crawling through the
razor-blade riddled crawlspace that is the real world.</p>
<p>Hunt people down and solve their problems!</p>
Why doesn't GCC do this "easy" NRVO optimization?urn:https-kaashif-co-uk:-2022-01-25-why-doesn-t-gcc-do-this-easy-nrvo-optimization2022-01-25T00:00:00Z2022-01-25T00:00:00ZKaashif Hymabaccus
<p>Since C++17, us C++ programmers have rejoiced in the fact that when we
return something from a function, the standard now guarantees that the
value won't be copied. This is known as return value optimization
(RVO) or copy/move elision, and happens in cases like this:</p>
<div class="highlight"><pre><span></span><span class="n">MyType</span><span class="w"> </span><span class="nf">myfunc</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ...</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">MyType</span><span class="p">{</span><span class="n">arg1</span><span class="p">,</span><span class="w"> </span><span class="n">arg2</span><span class="p">};</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>That is, <code>MyType</code> is constructed once and never copied or moved.</p>
<p>But some compilers still don't perform RVO in some cases. It turns out
this is because RVO refers only to when you return unnamed
values. <em>Named</em> RVO is apparently <em>not</em> considered RVO by the
standard's definition. Named means something like:</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="n">MyType</span><span class="w"> </span><span class="n">x</span><span class="p">{};</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="n">do_something</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>And gcc (11.2) doesn't always perform NRVO, even if it "obviously"
can. Why? Do other compilers do better? I tried to find out.</p>
<!--more-->
<h2>An example that works</h2>
<p>Let's look at a code sample to see what the problem isn't, i.e. an
example where NRVO does happen. We have this code:</p>
<div class="highlight"><pre><span></span><span class="kt">void</span><span class="w"> </span><span class="nf">doSomething</span><span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="p">);</span><span class="w"></span>
<span class="k">struct</span><span class="w"> </span><span class="nc">MyType</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">buffer</span><span class="p">[</span><span class="mi">100000</span><span class="p">];</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
<span class="n">MyType</span><span class="w"> </span><span class="nf">wantNrvo</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">MyType</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="n">buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'\0'</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">auto</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">wantNrvo</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">doSomething</span><span class="p">(</span><span class="n">x</span><span class="p">.</span><span class="n">buffer</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>And we are interested in what the <code>wantNrvo</code> function compiles down to
with <code>-O3</code>, we want gcc to give it its best shot. That's why we add
the <code>doSomething</code> function, to stop x being optimised away. We could
have instead added <code>volatile</code> to x, but I hardly ever see that in real
code - that would just confuse us. A function call is more realistic.</p>
<p>Anyway, the generated assembly is:</p>
<div class="highlight"><pre><span></span><span class="nf">wantNrvo</span><span class="p">():</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">BYTE</span><span class="w"> </span><span class="no">PTR</span><span class="w"> </span><span class="p">[</span><span class="no">rdi</span><span class="p">],</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">rax</span><span class="p">,</span><span class="w"> </span><span class="no">rdi</span><span class="w"></span>
<span class="w"> </span><span class="nf">ret</span><span class="w"></span>
<span class="nl">main:</span><span class="w"></span>
<span class="w"> </span><span class="nf">sub</span><span class="w"> </span><span class="no">rsp</span><span class="p">,</span><span class="w"> </span><span class="mi">100008</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">rdi</span><span class="p">,</span><span class="w"> </span><span class="no">rsp</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">BYTE</span><span class="w"> </span><span class="no">PTR</span><span class="w"> </span><span class="p">[</span><span class="no">rsp</span><span class="p">],</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="nf">call</span><span class="w"> </span><span class="no">doSomething</span><span class="p">(</span><span class="no">char</span><span class="p">*)</span><span class="w"></span>
<span class="w"> </span><span class="nf">xor</span><span class="w"> </span><span class="no">eax</span><span class="p">,</span><span class="w"> </span><span class="no">eax</span><span class="w"></span>
<span class="w"> </span><span class="nf">add</span><span class="w"> </span><span class="no">rsp</span><span class="p">,</span><span class="w"> </span><span class="mi">100008</span><span class="w"></span>
<span class="w"> </span><span class="nf">ret</span><span class="w"></span>
</pre></div>
<p>We can see there's obviously no copying of the huge 100,000-byte
buffer there, since there's no memcpy (or equivalent) anywhere. That's
what we're looking for.</p>
<p>It's interesting to note that NRVO means the caller has to make some
space for the object to be returned on the stack <em>before</em> the function
is even run, then the object is built in the space given.</p>
<p>The function is being passed the location to build the object as
rdi. Thus, RVO can be viewed as a rewriting of our function to take a
pointer to a block of memory big enough for MyType, and using
placement new.</p>
<p>That might look like this:</p>
<div class="highlight"><pre><span></span><span class="kt">void</span><span class="w"> </span><span class="nf">wantNrvo</span><span class="p">(</span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">memory</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">MyType</span><span class="w"> </span><span class="o">*</span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="p">(</span><span class="n">memory</span><span class="p">)</span><span class="w"> </span><span class="n">MyType</span><span class="p">{};</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span><span class="o">-></span><span class="n">buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'\0'</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>The point is that there is one single block of memory where the return
value is constructed, and this is allocated by the caller. That seems
fine: the caller knows the size of the block and can allocate it -
what could go wrong?</p>
<h2>What goes wrong</h2>
<p>Sometimes it's hard to tell which object out of multiple objects is
going to be returned, meaning we don't know which object to construct
in the return value area.</p>
<p>That is kind of non-obvious, so here's an example:</p>
<div class="highlight"><pre><span></span><span class="n">MyType</span><span class="w"> </span><span class="nf">wantNrvo</span><span class="p">(</span><span class="kt">bool</span><span class="w"> </span><span class="n">test</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">MyType</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">MyType</span><span class="w"> </span><span class="n">y</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="n">buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'\0'</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">test</span><span class="w"> </span><span class="o">?</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">y</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>There is obviously no way to know at compile time which of x or y we
should construct in the return value area. Using our knowledge of this
program, you might think: hey, why don't we just check <code>test</code> and
decide which of x and y to construct in the return value? Alas, gcc,
even with <code>-O3</code>, isn't that smart. This is the machine code produced:</p>
<div class="highlight"><pre><span></span><span class="nf">wantNrvo</span><span class="p">(</span><span class="no">bool</span><span class="p">):</span><span class="w"></span>
<span class="w"> </span><span class="nf">sub</span><span class="w"> </span><span class="no">rsp</span><span class="p">,</span><span class="w"> </span><span class="mi">200008</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">r9d</span><span class="p">,</span><span class="w"> </span><span class="no">esi</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">edx</span><span class="p">,</span><span class="w"> </span><span class="mi">100000</span><span class="w"></span>
<span class="w"> </span><span class="nf">lea</span><span class="w"> </span><span class="no">rax</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="no">rsp</span><span class="err">+</span><span class="mi">100000</span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="nf">test</span><span class="w"> </span><span class="no">r9b</span><span class="p">,</span><span class="w"> </span><span class="no">r9b</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">rsi</span><span class="p">,</span><span class="w"> </span><span class="no">rsp</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">BYTE</span><span class="w"> </span><span class="no">PTR</span><span class="w"> </span><span class="p">[</span><span class="no">rsp</span><span class="p">],</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="nf">cmove</span><span class="w"> </span><span class="no">rsi</span><span class="p">,</span><span class="w"> </span><span class="no">rax</span><span class="w"></span>
<span class="w"> </span><span class="nf">call</span><span class="w"> </span><span class="no">memcpy</span><span class="w"></span>
<span class="w"> </span><span class="nf">add</span><span class="w"> </span><span class="no">rsp</span><span class="p">,</span><span class="w"> </span><span class="mi">200008</span><span class="w"></span>
<span class="w"> </span><span class="nf">ret</span><span class="w"></span>
</pre></div>
<p>It's there, plain as day: memcpy! The huge buffer is being copied!</p>
<h2>It gets worse!</h2>
<p>There are other, even more obvious cases where gcc fails. Maybe the
problem is that we're construction two objects, then picking one to
return. Maybe that's confusing for some reason. We can attempt to
rewrite the code:</p>
<div class="highlight"><pre><span></span><span class="n">MyType</span><span class="w"> </span><span class="nf">wantNrvo</span><span class="p">(</span><span class="kt">bool</span><span class="w"> </span><span class="n">test</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">test</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">MyType</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="n">buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="sc">'\0'</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">x</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">MyType</span><span class="w"> </span><span class="n">y</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">y</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This code is starting to get a bit contrived. But let's see what that
compiles to:</p>
<div class="highlight"><pre><span></span><span class="n">wantNrvo</span><span class="p">(</span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">[</span><span class="n">clone</span><span class="w"> </span><span class="p">.</span><span class="n">part</span><span class="mf">.0</span><span class="p">]</span><span class="o">:</span><span class="w"></span>
<span class="w"> </span><span class="n">sub</span><span class="w"> </span><span class="n">rsp</span><span class="p">,</span><span class="w"> </span><span class="mi">100008</span><span class="w"></span>
<span class="w"> </span><span class="n">mov</span><span class="w"> </span><span class="n">edx</span><span class="p">,</span><span class="w"> </span><span class="mi">100000</span><span class="w"></span>
<span class="w"> </span><span class="n">mov</span><span class="w"> </span><span class="n">rsi</span><span class="p">,</span><span class="w"> </span><span class="n">rsp</span><span class="w"></span>
<span class="w"> </span><span class="n">call</span><span class="w"> </span><span class="n">memcpy</span><span class="w"></span>
<span class="w"> </span><span class="n">add</span><span class="w"> </span><span class="n">rsp</span><span class="p">,</span><span class="w"> </span><span class="mi">100008</span><span class="w"></span>
<span class="w"> </span><span class="n">ret</span><span class="w"></span>
<span class="n">wantNrvo</span><span class="p">(</span><span class="kt">bool</span><span class="p">)</span><span class="o">:</span><span class="w"></span>
<span class="w"> </span><span class="n">push</span><span class="w"> </span><span class="n">r12</span><span class="w"></span>
<span class="w"> </span><span class="n">mov</span><span class="w"> </span><span class="n">r12</span><span class="p">,</span><span class="w"> </span><span class="n">rdi</span><span class="w"></span>
<span class="w"> </span><span class="n">test</span><span class="w"> </span><span class="n">sil</span><span class="p">,</span><span class="w"> </span><span class="n">sil</span><span class="w"></span>
<span class="w"> </span><span class="n">je</span><span class="w"> </span><span class="p">.</span><span class="n">L5</span><span class="w"></span>
<span class="w"> </span><span class="n">mov</span><span class="w"> </span><span class="n">rax</span><span class="p">,</span><span class="w"> </span><span class="n">r12</span><span class="w"></span>
<span class="w"> </span><span class="n">mov</span><span class="w"> </span><span class="n">BYTE</span><span class="w"> </span><span class="n">PTR</span><span class="w"> </span><span class="p">[</span><span class="n">rdi</span><span class="p">],</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="n">pop</span><span class="w"> </span><span class="n">r12</span><span class="w"></span>
<span class="w"> </span><span class="n">ret</span><span class="w"></span>
<span class="p">.</span><span class="n">L5</span><span class="o">:</span><span class="w"></span>
<span class="w"> </span><span class="n">call</span><span class="w"> </span><span class="n">wantNrvo</span><span class="p">(</span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">[</span><span class="n">clone</span><span class="w"> </span><span class="p">.</span><span class="n">part</span><span class="mf">.0</span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="n">mov</span><span class="w"> </span><span class="n">rax</span><span class="p">,</span><span class="w"> </span><span class="n">r12</span><span class="w"></span>
<span class="w"> </span><span class="n">pop</span><span class="w"> </span><span class="n">r12</span><span class="w"></span>
<span class="w"> </span><span class="n">ret</span><span class="w"></span>
</pre></div>
<p>Now this is bizarre! There are two branches of this function:</p>
<ul>
<li><p>The test = true branch which uses NRVO and writes directly to the
block of memory pointed to by rdi.</p></li>
<li><p>The test = false branch which allocates new memory on the stack, and
copies the new, just-allocated memory into the area pointed to by
rdi with memcpy.</p></li>
</ul>
<p>Surely this is just optimized badly because we don't actually call it,
right? Don't be so sure, when we call it it gets inlined, but <em>still
uses memcpy</em>:</p>
<div class="highlight"><pre><span></span><span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">auto</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">wantNrvo</span><span class="p">(</span><span class="nb">false</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">doSomething</span><span class="p">(</span><span class="n">x</span><span class="p">.</span><span class="n">buffer</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="nl">main:</span><span class="w"></span>
<span class="w"> </span><span class="nf">sub</span><span class="w"> </span><span class="no">rsp</span><span class="p">,</span><span class="w"> </span><span class="mi">200008</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">edx</span><span class="p">,</span><span class="w"> </span><span class="mi">100000</span><span class="w"></span>
<span class="w"> </span><span class="nf">lea</span><span class="w"> </span><span class="no">rsi</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="no">rsp</span><span class="err">+</span><span class="mi">100000</span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">rdi</span><span class="p">,</span><span class="w"> </span><span class="no">rsp</span><span class="w"></span>
<span class="w"> </span><span class="nf">call</span><span class="w"> </span><span class="no">memcpy</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">rdi</span><span class="p">,</span><span class="w"> </span><span class="no">rsp</span><span class="w"></span>
<span class="w"> </span><span class="nf">call</span><span class="w"> </span><span class="no">doSomething</span><span class="p">(</span><span class="no">char</span><span class="p">*)</span><span class="w"></span>
<span class="w"> </span><span class="nf">xor</span><span class="w"> </span><span class="no">eax</span><span class="p">,</span><span class="w"> </span><span class="no">eax</span><span class="w"></span>
<span class="w"> </span><span class="nf">add</span><span class="w"> </span><span class="no">rsp</span><span class="p">,</span><span class="w"> </span><span class="mi">200008</span><span class="w"></span>
<span class="w"> </span><span class="nf">ret</span><span class="w"></span>
</pre></div>
<h2>Why, God? Why?</h2>
<p>I am not a compiler author. I have never written a compiler and am
much dumber than those behind gcc's NRVO.</p>
<p>But NRVO is pretty important, and I expect it sometimes! That's backed
up by this comment on this gcc bug:
<a href="https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51571">https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51571</a></p>
<blockquote><p>I would like to strongly oppose the notion that this "just a missed
optimisation and not very critical, really".</p>
<p>NRVO is not just "an optimisation". It's actually one that is
explcitly permitted to change observable behaviour of the program
and it's extremely powerful.</p>
<p>And it it <em>required</em> for performant C++. Just try to return a
std::vector by value to see the importance of this
optimisation. This is not missed optimisation. This is premature
pessimisation.</p>
<p>You could just as well stop all optimisation work for the C++
frontend until this is implemented, because any other optimisation
effords are dwarfed by the overhead when NRVO is expected by the
developer but not applied.</p>
<p>Please make this a top priority. Every C++ program will benefit both
in text size and in runtime performance - dramatically.</p>
<p>-- Marc Mutz</p></blockquote>
<p>But then again, there are often hyperbolic, borderline rude comments
left on projects saying "make this a top priority", so who knows?</p>
<h2>Let's see clang's business card</h2>
<p>Let's see clang's machine code, does it do any better? Yes!</p>
<p>(using clang 13.0.0)</p>
<div class="highlight"><pre><span></span><span class="nf">wantNrvo</span><span class="p">(</span><span class="no">bool</span><span class="p">):</span><span class="w"> </span><span class="c1"># @wantNrvo(bool)</span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">rax</span><span class="p">,</span><span class="w"> </span><span class="no">rdi</span><span class="w"></span>
<span class="w"> </span><span class="nf">test</span><span class="w"> </span><span class="no">esi</span><span class="p">,</span><span class="w"> </span><span class="no">esi</span><span class="w"></span>
<span class="w"> </span><span class="nf">je</span><span class="w"> </span><span class="no">.LBB0_2</span><span class="w"></span>
<span class="w"> </span><span class="nf">mov</span><span class="w"> </span><span class="no">byte</span><span class="w"> </span><span class="no">ptr</span><span class="w"> </span><span class="p">[</span><span class="no">rax</span><span class="p">],</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="nl">.LBB0_2:</span><span class="w"></span>
<span class="w"> </span><span class="nf">ret</span><span class="w"></span>
</pre></div>
<p>This is exactly the perfect machine code we want. Give us some memory
in rdi, collapse those branches down to writing or not writing a nul
byte to it. Easy, no memcpy!</p>
<h2>Lessons to learn</h2>
<p>Although you might see some text on cppreference.com saying:</p>
<blockquote><p>Return value optimization is mandatory and no longer considered as
copy elision; see above. (since C++17)</p>
<p>-- <a href="https://en.cppreference.com/w/cpp/language/copy_elision">https://en.cppreference.com/w/cpp/language/copy_elision</a></p></blockquote>
<p>This doesn't refer to NRVO! Assuming that NRVO will happen can be very
dangerous, resulting in copies of huge objects, and that hurts
performance a lot.</p>
<p>Try as hard as possible to return prvalues to take advantage of
C++17's guaranteed RVO.</p>
<p>It's a dog-eat-RVO world out there. Be careful.</p>
Representing complex numbers exactly on a computerurn:https-kaashif-co-uk:-2020-06-03-representing-complex-numbers-exactly-on-a-computer2020-06-03T00:00:00Z2020-06-03T00:00:00ZKaashif Hymabaccus
<p>There are a lot of ways to represent numerical values on a computer,
you've got the various fixed-size integer types and floating point
types, but you also have arbitrary precision arithmetic, where the
size of the number is limited only by the memory of the machine.</p>
<p>To represent the real numbers, $\mathbb{R}$, programmers often choose
floating point numbers. But of course, floating point is terrible:
multiplication isn't associative or commutative, neither is addition,
you don't really have reciprocals, and so on. Useless for exact
arithmetic of the kind you need when doing algebra.</p>
<p>This blog post is about a way to represent cyclotomic fields, which
provide (among other things):</p>
<ul>
<li><p>All rational numbers</p></li>
<li><p>$\sqrt{m}$ for any integer $m$</p></li>
<li><p>All of the field axioms: multiplication and addition with proper
associativity, commutativity, and invertibility</p></li>
<li><p>Enough numbers to represent any finite group</p></li>
</ul>
<p>They're still a bit tricky to represent on a computer though, as
you'll see.</p>
<!--more-->
<h2>What is a cyclotomic field?</h2>
<p>I refer to $K$ as a cyclotomic field if there exists some $n > 0$ such
that $K = \mathbb{Q}(\zeta_n)$. That is, $K$ is the rational numbers
with one primitive nth root of unity added. Essentially, that means
that we have $\mathbb{Q}$ plus all solutions to the equation $x^n - 1
= 0$, and we take the closure under all field operations (addition,
multiplication, inversion).</p>
<p>Why is this a field? The proof of this requires a bit of mathematical
knowledge, but it's because our definition of $K$ was a bit non
rigorous. In fact, $K$ is the splitting field of $x^n - 1$ over
$\mathbb{Q}$. There's a decent amount of Galois theory machinery you
need to build up before it becomes clear why $K$ should exist, but
essentially you just factor $x^n - 1$ into irreducible factors and,
at each stage, add a root to $K_i$ by constructing $K_{i+1} =
K_i[x]/(f_i(x))$. The details aren't as important as the result,
which is that $K$ exists and is a field.</p>
<p>At this point, we can prove that we have all of the field
axioms. Great! We also have all roots of rationals by the
Kronecker-Weber theorem (non-trivial to prove, but a nice
result). What do I mean by "we have enough numbers to represent any
finite group"?</p>
<p>If you read my
<a href="/2020/04/19/decomposing-representations-a-slice-of-computational-group-theory/">other blog post</a>
or know a thing or two about representation theory, you'll know what a
representation is. The cool thing is that you can <em>represent</em> any
finite group with coefficients in some cyclotomic field $K$.</p>
<p>More specifically, if $G$ is a finite group with elements whose order
have least common multiple $m$, then if $K$ contains the $m$th roots
of unity, we can realise all representations of $G$ using coefficients
from $K$. This is a theorem due to Brauer which I saw in a textbook on
representation theory by Serre.</p>
<h2>How do you represent one in a computer?</h2>
<p>The answer to this is simple, you just represent it the same way as
you do on paper - as a sum of powers of $\zeta_n$ with coefficients
from $\mathbb{Q}$. So if we have $z \in K$, we can represent an
element as:</p>
<p>$$z = \sum_{i=0}^{n-1} q_i \zeta_n^i$$</p>
<p>where $q_i \in \mathbb{Q}$. So we can just store a vector of the
coefficients $q_i$, right?</p>
<p>WRONG! If you did this, the representations of elements would not be
unique, so you wouldn't be able to check equality. The problem is that
${ \zeta_n^i : 0 \leq i < n }$ isn't a basis for $K$ as a
$\mathbb{Q}$-vector space. This is obvious if you factor the
polynomial defining $K$ a bit:</p>
<p>$$\frac{x^n - 1}{x-1} = 1 + x + \ldots + x^{n-1}$$</p>
<p>So we have a linear dependency: $-1 = x + \ldots + x^{n-1}$. We can
we can try to fix this by seeing if ${ \zeta_n^i : 0 < i < n }$ is
a basis. Since we got rid of the constant term, that's it, this is a
basis now, right?</p>
<p>WRONG! Well, wrong in all except one case. If $n=p$ prime, then this
actually is a basis. This is because all of the roots are primitive,
$p$ is coprime to all smaller numbers (except 1), and thus
$K/\mathbb{Q}$ is a degree $p-1$ extension i.e. a dimension $p-1$
vector space over $\mathbb{Q}$. That's not exactly rigorous, but you
get the idea.</p>
<p>From this point, we assume $n=p$ prime, so this basis is actually a
basis. Otherwise we have to choose a better basis and things get
complicated.</p>
<h2>Operating in the field</h2>
<p>It's clear how to compute $z + z'$ and $-z$, you just add the list of
coefficients in $\mathbb{Q}$. $zz'$ isn't much more difficult, you
just write out the sums and multiply them:</p>
<p>$$
\begin{align}
zz' &= \left(\sum_{i=1}^{p-1} q_i \zeta_p^i\right)\left(\sum_{i=1}^{p-1} q_i' \zeta_p^i\right) \\
&= \sum_{i, j} q_i q_j' \zeta_p^{i + j} \\
\end{align}
$$</p>
<p>After computing this, you just reduce all of the $i + j$ modulo $p$,
and get rid of any constant terms produced using the $-1 = x +
\ldots + x^{p-1}$ relation.</p>
<p>The only kind of tricky operation is multiplicative inversion, that
is, computing $1/z$. At this point we must rely on some Galois
theory. The Galois group $G = \text{Gal}(K/\mathbb{Q})$ is, in this
case, the group $\mathbb{Z}_p^\times$ of multiplicative units of the
integers modulo $p$, or the additive group of the integers modulo
$p-1$.</p>
<p>$G$ is generated by any nontrivial field automorphism. We can
determine a field automorphism by saying where $\zeta_p$ goes, and it
can go to any other nontrivial root. Since $p$ is prime, all of the
maps given by $\zeta_p \mapsto \zeta_p^i$ for $1 \leq i \leq p-1$
are invertible field automorphisms. Let's just pick $f \in G$ to be
$\zeta_p \mapsto \zeta_p^2$, although we could pick any nontrivial
image.</p>
<p>If we just multiply all elements in the $G$-orbit of $z$, then we get
something that's fixed by all of $G$, so must live in
$\mathbb{Q}$. Let $g \in G$, then:</p>
<p>$$g\left(\prod_{h \in G}h(z)\right) = \prod_{h \in G}gh(z) = \prod_{h \in G}h(z)$$</p>
<p>where the last equality is just a relabelling of the elements of $G$.</p>
<p>Thus $\prod_{i=0}^{p-2}f^i(z) = q \in \mathbb{Q}$. And now we have
a formula for the inverse of $z$, since:</p>
<p>$$q^{-1}\prod_{i=0}^{p-2}f^i(z) = q^{-1}q = 1$$</p>
<p>But also:</p>
<p>$$q^{-1}\prod_{i=0}^{p-2}f^i(z) = z\left(q^{-1}\prod_{i=1}^{p-2}f^i(z)\right)$$</p>
<p>So finally, in terms of only taking reciprocals of rationals,
multiplication, and addition:</p>
<p>$$z^{-1} = q^{-1}\prod_{i=1}^{p-2}f^i(z)$$</p>
<p>Nice! Now we have all of the field axioms.</p>
<h2>Other things to explore</h2>
<p>How do we represent the list of coefficients? Sparsely with a hash
map, densely with an array, maybe sorted with respect to the exponents
using a heap?</p>
<p>And how can this work for non-prime $n$? Like I said earlier, the key
is to pick a good basis. The obvious one is ${ \zeta_n^i : 0 < i <
n, \text{gcd}(n, i) = 1}$, but there is a much nicer basis we can
choose, see if you can work it out. I definitely couldn't, but I'll
write another blog post about it at some point.</p>
<p>There is a lot of existing work in this field (haha) out there, but
particular thanks go to the authors of the
<a href="https://www.gap-system.org/">GAP computer algebra system</a> for
providing an open source implementation of cyclotomic fields.</p>
Writing a parser for a function call is surprisingly hardurn:https-kaashif-co-uk:-2020-05-17-writing-a-parser-for-a-function-call-is-surprisingly-hard2020-05-17T00:00:00Z2020-05-17T00:00:00ZKaashif Hymabaccus
<p>I was recently trying to get to grips with the
<a href="https://www.gap-system.org/Manuals/doc/ref/chap4.html">GAP programming language</a>. For
those not familiar, it's the programming language for the GAP
computational algebra system. It has tons of algorithms implemented
for group theory, representation theory, algebraic number theory, and
so on. I was thinking about implementing a TypeScript-style transpiler
so I could program with some types, and the first step is to parse the
syntax.</p>
<p>To get the most elegant parser, I went for a parser written in Haskell
using
<a href="https://www.gap-system.org/Manuals/doc/ref/chap4.html">Parsec</a>, which
is an elegant library for LL(1) parsers.</p>
<p>The first problem I ran into was that GAP supports several function
call syntaxes:</p>
<div class="highlight"><pre><span></span><span class="nv">f</span><span class="p">(</span><span class="nv">x</span><span class="o">,</span> <span class="nv">y</span><span class="o">,</span> <span class="nv">z</span><span class="p">)</span><span class="o">;</span> <span class="c1"># positional</span>
<span class="nv">g</span><span class="p">(</span><span class="nv">p</span> <span class="o">:=</span> <span class="nv">1</span><span class="o">,</span> <span class="nv">q</span> <span class="o">:=</span> <span class="nv">2</span><span class="p">)</span><span class="o">;</span> <span class="c1"># named-ish parameters</span>
<span class="nv">h</span><span class="p">(</span><span class="nv">x</span><span class="o">,</span> <span class="nv">y</span><span class="o">,</span> <span class="nv">z</span> : <span class="nv">p</span> <span class="o">:=</span> <span class="nv">1</span><span class="p">)</span><span class="o">;</span> <span class="c1"># mixed</span>
</pre></div>
<p>This is surprisingly non-trivial to parse in general! The path is
fraught with infinite recursion, ambiguities, and backtracking.</p>
<!--more-->
<h2>First problem: infinite recursion</h2>
<p>First, we need to understand how Parsec works, it's an LL(1)
parser. That means it can be used to parse context-free languages, it
tries to perform a leftmost derivation, with 1 token of
lookahead. It's easier to explain this with an example. I'll use the
notation of
<a href="https://en.wikipedia.org/wiki/Formal_grammar">production rules for formal languages</a>.</p>
<p>We want to say that a function call is some expression, with a
parenthesis delimited argument list which is how we tell we're calling
something. So we have things like <code>f(x)</code> but also <code>f(x)(y)</code> which is
calling the result of <code>f(x)</code> with the argument <code>y</code>. Using start symbol
$E$ (for "expression"), the function call grammar looks like:</p>
<p>$$
\begin{align}
E &\to E(A) \\
E &\to L \\
A &\to E, A \\
A &\to E
\end{align}
$$</p>
<p>where $L$ can be any literal, and $A$ represents an argument list.</p>
<p>We'd write this in Parsec as something like:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">Expr</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Variable</span><span class="w"> </span><span class="kt">String</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">FunctionCall</span><span class="w"> </span><span class="kt">Expr</span><span class="w"> </span><span class="p">[</span><span class="kt">Expr</span><span class="p">]</span><span class="w"></span>
<span class="nf">expression</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">functionCall</span><span class="w"> </span><span class="o"><|></span><span class="w"> </span><span class="p">(</span><span class="kt">Variable</span><span class="w"> </span><span class="o"><$></span><span class="w"> </span><span class="n">identifier</span><span class="p">)</span><span class="w"></span>
<span class="nf">functionCall</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">FunctionCall</span><span class="w"> </span><span class="o"><$></span><span class="w"> </span><span class="n">expression</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="p">(</span><span class="n">parens</span><span class="w"> </span><span class="o">$</span><span class="w"> </span><span class="n">commaSep</span><span class="w"> </span><span class="n">expression</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>where we're parsing a string into the data type <code>Expr</code>.</p>
<p>The parameters to <code>FuncCall</code> are: the expression we're calling, and
the argument list. This is a cut down example, so I say we only have
variables as the literals.</p>
<p>The problem comes when we try to run this on a string:</p>
<div class="highlight"><pre><span></span><span class="nf">parse</span><span class="w"> </span><span class="n">expression</span><span class="w"> </span><span class="s">""</span><span class="w"> </span><span class="s">"f(x)"</span><span class="w"></span>
</pre></div>
<p>This will just hang at 100% CPU usage. Why? Because the definition of
an expression is immediately left-recursive. Parsec is LL(1) so tries
to recurse as far as possible on the left when parsing, and we can
just keep going: try to parse an expression, the first step is to
check if it's a function call. To check if it's a function call, we
need to parse an expression, the first step to that is check if it's a
function call, and so on.</p>
<p>What's the solution? We can cleverly transform the grammar to avoid
the immediate left recursion.</p>
<p>When we actually see a function expression, what do we see? A
non-function call expression on the far left, then a lot of argument
lists. As production rules:</p>
<p>$$
\begin{align}
E &\to LM \\
E &\to L \\
M &\to M(A) \\
M &\to (A)
\end{align}
$$</p>
<p>where $A$ is the same as before. In Parsec:</p>
<div class="highlight"><pre><span></span><span class="nf">expression</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">functionCall</span><span class="w"> </span><span class="o"><|></span><span class="w"> </span><span class="p">(</span><span class="kt">Variable</span><span class="w"> </span><span class="o"><$></span><span class="w"> </span><span class="n">identifier</span><span class="p">)</span><span class="w"></span>
<span class="nf">functionCall</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">foldl'</span><span class="w"> </span><span class="kt">FuncCall</span><span class="w"> </span><span class="o"><$></span><span class="w"> </span><span class="p">(</span><span class="kt">Variable</span><span class="w"> </span><span class="o"><$></span><span class="w"> </span><span class="n">identifier</span><span class="p">)</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="n">many1</span><span class="w"> </span><span class="n">argList</span><span class="w"></span>
<span class="w"> </span><span class="kr">where</span><span class="w"> </span><span class="n">argList</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">parens</span><span class="w"> </span><span class="o">$</span><span class="w"> </span><span class="n">commaSep</span><span class="w"> </span><span class="n">expression</span><span class="w"></span>
</pre></div>
<p>Some explanation: if we have a string "f(x)(y)" this is parsed as:</p>
<ul>
<li><code>Variable "f"</code> when we see the <code>f</code></li>
<li><code>FuncCall (Variable "f") [Variable "x"]</code> when we see the argument
list <code>(x)</code>.</li>
<li><code>FuncCall (FuncCall (Variable "f") [Variable "x"]) [Variable "y"]</code>
when we see the argument list <code>(y)</code>.</li>
</ul>
<p>Perfect! This avoids the infinite recursion. Next problem!</p>
<h2>Horrific named argument grammar</h2>
<p>To implement the keyword arguments, we need to make a few changes to
the <code>argList</code> parser. The obvious thing to do is just implement the
alternatives in a big list:</p>
<div class="highlight"><pre><span></span><span class="nf">argList</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Parser</span><span class="w"> </span><span class="p">([</span><span class="kt">Expr</span><span class="p">],</span><span class="w"> </span><span class="p">[(</span><span class="kt">String</span><span class="p">,</span><span class="w"> </span><span class="kt">Expr</span><span class="p">)])</span><span class="w"></span>
<span class="nf">argList</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">parens</span><span class="w"> </span><span class="o">$</span><span class="w"> </span><span class="kr">do</span><span class="w"></span>
<span class="w"> </span><span class="n">args</span><span class="w"> </span><span class="ow"><-</span><span class="w"> </span><span class="n">commaSep</span><span class="w"> </span><span class="n">expression</span><span class="w"></span>
<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">length</span><span class="w"> </span><span class="n">args</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">colon</span><span class="w"></span>
<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">return</span><span class="w"> </span><span class="nb">()</span><span class="w"></span>
<span class="w"> </span><span class="n">named</span><span class="w"> </span><span class="ow"><-</span><span class="w"> </span><span class="n">commaSep</span><span class="w"> </span><span class="n">namedArg</span><span class="w"></span>
<span class="w"> </span><span class="kr">where</span><span class="w"> </span><span class="n">namedArg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(,)</span><span class="w"> </span><span class="o"><$></span><span class="w"> </span><span class="n">identifier</span><span class="w"> </span><span class="o"><*</span><span class="w"> </span><span class="n">reservedOp</span><span class="w"> </span><span class="s">":="</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="n">expression</span><span class="w"></span>
</pre></div>
<p>where <code>argList</code> now returns a tuple of the non-named arguments and the
named arguments.</p>
<p>But this doesn't work! How do you parse the following argument list:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">x</span> <span class="o">:=</span> <span class="nv">1</span><span class="p">)</span>
</pre></div>
<p>The "correct" parser would see this as a single named argument, but
our parser currently sees this as a variable x, then the colon
separating the non-named from named arguments, then an =. That's a
syntax error - the parser fails!</p>
<p>The solution is to try the cases in order, explicitly.</p>
<div class="highlight"><pre><span></span><span class="nf">argList</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Parser</span><span class="w"> </span><span class="p">([</span><span class="kt">Expr</span><span class="p">],</span><span class="w"> </span><span class="p">[(</span><span class="kt">String</span><span class="p">,</span><span class="w"> </span><span class="kt">Expr</span><span class="p">)])</span><span class="w"></span>
<span class="nf">argList</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">parens</span><span class="w"> </span><span class="o">$</span><span class="w"> </span><span class="n">try</span><span class="w"> </span><span class="n">justNamed</span><span class="w"> </span><span class="o"><|></span><span class="w"> </span><span class="n">try</span><span class="w"> </span><span class="n">both</span><span class="w"> </span><span class="o"><|></span><span class="w"> </span><span class="n">justNotNamed</span><span class="w"></span>
<span class="w"> </span><span class="kr">where</span><span class="w"></span>
<span class="w"> </span><span class="n">justNamed</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(,)</span><span class="w"> </span><span class="kt">[]</span><span class="w"> </span><span class="o">$</span><span class="w"> </span><span class="n">commaSep1</span><span class="w"> </span><span class="n">namedArg</span><span class="w"></span>
<span class="w"> </span><span class="n">namedArg</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(,)</span><span class="w"> </span><span class="o"><$></span><span class="w"> </span><span class="n">identifier</span><span class="w"> </span><span class="o"><*</span><span class="w"> </span><span class="n">reservedOp</span><span class="w"> </span><span class="s">":="</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="n">expression</span><span class="w"></span>
<span class="w"> </span><span class="n">justNotNamed</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(,)</span><span class="w"> </span><span class="o"><$></span><span class="w"> </span><span class="n">commaSep</span><span class="w"> </span><span class="n">expression</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="n">pure</span><span class="w"> </span><span class="kt">[]</span><span class="w"></span>
<span class="w"> </span><span class="n">both</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kr">do</span><span class="w"></span>
<span class="w"> </span><span class="n">notNamed</span><span class="w"> </span><span class="ow"><-</span><span class="w"> </span><span class="n">justNotNamed</span><span class="w"></span>
<span class="w"> </span><span class="n">colon</span><span class="w"></span>
<span class="w"> </span><span class="n">named</span><span class="w"> </span><span class="ow"><-</span><span class="w"> </span><span class="n">justNamed</span><span class="w"></span>
<span class="w"> </span><span class="n">return</span><span class="w"> </span><span class="p">(</span><span class="n">fst</span><span class="w"> </span><span class="n">notNamed</span><span class="p">,</span><span class="w"> </span><span class="n">snd</span><span class="w"> </span><span class="n">named</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>Ah, there we go.</p>
<h2>Conclusion</h2>
<p>Parsing is non-trivial, surprisingly. To see my code in action, check
out
<a href="https://github.com/kaashif/gap-parser/blob/master/src/Language/GAP/Parser.hs">the full source of my parser for GAP</a>
(not yet fully tested).</p>
<p>Please don't complain that the code in this post doesn't work, look at
that repo for the real working source!</p>
Decomposing representations: a slice of computational group theoryurn:https-kaashif-co-uk:-2020-04-19-decomposing-representations-a-slice-of-computational-group-theory2020-04-19T00:00:00Z2020-04-19T00:00:00ZKaashif Hymabaccus
<p>For my Master's degree, I (helped greatly by my supervisor)
implemented some algorithms and even invented some new algorithms to
decompose representations of finite groups. I wrote an extremely long
(well, relative to other things I've written) and technical thesis
about this, but I find myself increasingly unable to understand what
any of it means or why I even have a degree.</p>
<p>I thought being forced into a short-form blog post would help me
remember whatever it is I spent a few years studying to do. There are
some foundational questions:</p>
<ul>
<li>What is a group?</li>
<li>What is a representation?</li>
<li>What is a decomposition of a representation?</li>
</ul>
<p>And some more interesting questions, involving some computational
tricks relevant to a wider audience:</p>
<ul>
<li>Why is this useful?</li>
<li>How do you get a computer to do it?</li>
<li>How do you get a computer to do it, quickly?</li>
</ul>
<p>These are the questions I'll attempt to answer in this blog
post. It'll be fun!</p>
<!--more-->
<h2>What is a group?</h2>
<p>Any undergraduate mathematician should know this, but if you don't do
algebra and need a refresher,
<a href="https://en.wikipedia.org/wiki/Group_(mathematics">read the Wikipedia page</a>). For
non-mathematicians, a group is:</p>
<ul>
<li><p>A set $G$ (for example, the real numbers or the rational numbers)</p></li>
<li><p>With a binary operation $\ast$ (like addition or multiplication)
which takes two elements of $G$ and gives you another element of $G$</p></li>
<li><p>Such that $\ast$ is associative, meaning bracketing doesn't matter -
$(a \ast b) \ast c = a \ast (b \ast c)$. True for multiplication or
addition, but NOT for subtraction</p></li>
<li><p>Such that $G$ has an identity element with respect to $\ast$ - an
element $e$ such that applying it with $\ast$ does nothing, like
multiplying by 1 or adding 0</p></li>
<li><p>Such that elements of $G$ all have inverses with respect to $\ast$,
meaning any element $g \in G$ has another element $g^{-1} \in G$
such that combining them with $\ast$ gives the identity. For
addition, the inverse of $x$ is $-x$. For multiplication, it's
$\frac1{x}$.</p></li>
</ul>
<p>Examples of groups include the real numbers $\mathbb{R}$ with addition
and the non-zero rational numbers $\mathbb{Q}^\ast$ with
multiplication.</p>
<p>There are also finite examples, like the set ${+1, -1}$ with
multiplication, and the set of permutations of $n$ elements with
composition - $S_n$, the symmetric group on $n$ elements.</p>
<h2>What is a representation?</h2>
<p>A representation $\rho : G \to \text{GL}(V)$ of a group $G$ is a
homomorphism (a function respecting the group structure) from $G$ to
the group of linear automorphisms of a vector space $V$. You can
imagine "vector space" to mean $\mathbb{C}^n$ (the space of vectors
with $n$ entries in the complex numbers), since that's usually what
we're using.</p>
<p>"Linear automorphism" is a coordinate-independent way of saying
"invertible matrix". So you can rephrase the definition of a
representation as a map from a group $G$ to $n \times n$ matrices (the
$n$ is fixed for all elements), such that the group structure is
respected.</p>
<p>Sue me if you don't like it, but you'll never get your vector space
into a computer without picking a basis, meaning you are forced to
think of linear maps as matrices.</p>
<h2>What is a decomposition of a representation?</h2>
<p>An irreducible representation is a representation that doesn't have
any subrepresentations. What does that mean? We could go through the
definitions, but for complex representations of finite groups,
irreducible means indecomposable: you can't write the representation
as a direct sum of other representations. That is the same thing as
saying you can't simultaneously block diagonalise all $\rho(g)$ for $g
\in G$ such that the blocks are smaller than the whole matrix.</p>
<p>Now that the basics are out of the way, we can ask an interesting
question:</p>
<h2>Why is this useful?</h2>
<p>Irreducible representations come up in all sorts of places:</p>
<ul>
<li><p>Solving problems in quantum chemistry involving symmetries (oh,
hello $S_n$)</p></li>
<li><p>Solving completely computationally intractable optimization problems
by spotting interesting symmetries ($S_n$ again?), then reducing them</p></li>
<li><p>Turning groups into linear algebra, then studying them through how
their representations behave (this is just a vague statement
encompassing all of representation theory)</p></li>
</ul>
<p>Anything with symmetries probably has something to do with groups and
thus, something to do with representations.</p>
<p>I keep mentioning $S_n$, but the people who paid attention in algebra
might note that it is kind of misleading to talk about it as if it is
special. In fact, all finite groups appear as a subgroup of some
symmetric group due to
<a href="https://en.wikipedia.org/wiki/Cayley%27s_theorem">Cayley's theorem</a>. The
proof is actually fairly simple (for such a deep-seeming result) and I
encourage you to look at it. So really, saying $S_n$ pops up
everywhere in group theory is the same thing as saying groups pop up
everywhere in group theory - not surprising.</p>
<p>Sadly, physicists are interested in infinite groups and in particular,
Lie groups. You might have heard of the Lorentz group $\text{O}(1,3)$,
a certain special unitary group $\text{SU}(2)$, or some other Lie
groups - these pop up all the time in physics.</p>
<p>I say "sadly" because my algorithms are not directly applicable to
those cases, since they're too continuous to fit inside my poor
computer. It's very happy that there is such strong motivation to
study Lie groups coming from physics, but my work isn't really
relevant there.</p>
<h2>How do you get a computer to compute a decomposition?</h2>
<p>The first question to ask is: how do you do it by hand?</p>
<p>Let's say we have a finite group $G$ and a representation $\rho : G
\to \text{GL}(V)$ where $V$ is a finite dimensional complex vector
space (I'm skipping over something very important here, see if you can
guess what it is, or skip to the end for the answer).</p>
<p>If you <em>already</em> have the complete list of irreducible representations
of $G$ (up to isomorphism), $(\rho_i)$, you can just combine all
$\rho_i$ as blocks in all possible ways that give matrices of the
correct size. That is, you try to find some $i_j$ such that:</p>
<p>$$\rho(g) = \begin{pmatrix}
\rho_{i_1}(g) & & \\
& \ddots & \\
& & \rho_{i_m}(g)
\end{pmatrix}$$</p>
<p>Where anything not labelled above is zero. This is easy for some small
groups. For example, if you have the representation of ${\pm 1}$
given by:</p>
<p>$$\pm 1 \mapsto \begin{pmatrix}
1 & 0 \\
0 & \pm 1
\end{pmatrix}$$</p>
<p>It's easy to spot this is given by two $1 \times 1$ blocks. But that's
because we already know all of the irreducible representations (let's
shorten that to irrep, as is standard), there are only two. Even
better, the matrices are already given in the nicest basis
possible. It's not really obvious how to "observe" the structure of a
representation like this in general.</p>
<p>Here's the dirty trick. Define $\chi_\rho(g) :=
\text{trace}(\rho(g))$. $\chi_\rho$ is called the character of
$\rho$. It has the useful property that for a finite $G$, a finite
dimensional representation (meaning the matrices are finite size), and
coefficients in $\mathbb{C}$: $\rho$ is isomorphic to $\tau$ (as
representations) if and only if their characters are the same.</p>
<p>This gives rise to this simple algorithm:</p>
<ol>
<li><p>List all irreps of $G$ using e.g. Dixon's algorithm
(<code>IrreducibleRepresentationsDixon</code> in GAP)</p></li>
<li><p>Put them together in all possible ways adding up to the correct
matrix dimension.</p></li>
<li><p>Compute all of the characters - there will be exactly one matching
the character of $\rho$, that's your decomposition!</p></li>
</ol>
<p>This involves a lot of listing and trial and error - it is highly
inefficient and intractable for even small groups and small
representations.</p>
<p>Even ignoring efficiency, this tells you the decomposition <em>up to
isomorphism</em>. Telling you the basis change from $\rho$ to actually get
the smallest possible blocks is another problem. There's one algorithm
due to Serre to do that, with a full proof of correctness in his
textbook on Linear Representations of Finite Groups.</p>
<h2>How do you get a computer to compute it, quickly?</h2>
<p>I (and my supervisor) came up with a bag of tricks to speed up the
computation by taking advantage of some extra information that's
usually available.</p>
<h3>Trading space for speed</h3>
<p>$G$ is always isomorphic to a permutation group, but in the real
world, $G$ is usually actually <em>already</em> a permutation group. This
lets us specialise our algorithms using the vast array of tools
available for computing with permutation groups. The main trick is
that a good
<a href="https://en.wikipedia.org/wiki/Base_(group_theory">base and strong generating set</a>)
can be computed with the
<a href="https://en.wikipedia.org/wiki/Schreier%E2%80%93Sims_algorithm">Schreier-Sims algorithm</a>,
letting us sum over $G$ quickly (computing $\sum_g f(g)$ for certain
$f$). This lets us compute some of Serre's formulas much, much faster.</p>
<p>Let's say we have two representations $\rho$ and $\tau$ as earlier,
which are isomorphic but we don't know the basis change (the
isomorphism). $G$ is a permutation group, so summing over it is
fast. But how can we reduce finding an isomorphism to a sum over $G$?
It's quite a neat trick.</p>
<p>We're looking for a matrix $A$ such that $A^{-1} \tau(g) A = \rho(g)$
for all $g \in G$. the "dumb" way would be to forget the
representation structure and just try to simultaneously diagonalise
the $\tau(g)$ and $\rho(g)$. I spent a whole six months hammering away
before noticing the smart way, which is to notice that there's another
representation $\alpha$ defined by:</p>
<p>$$g \mapsto (X \mapsto \tau(g)X\rho(g^{-1}))$$</p>
<p>If we say that $\rho$ and $\tau$ map to $\text{GL}(V)$ then this is a
representation $G \to \text{GL}(\text{GL}(V))$. Crazy! The funny thing
is that this representation is actually $\tau \otimes \rho^\ast$, the
Kronecker product of $\tau$ and the conjugate transpose of $\rho$
(tensor product and dual representation, if you're still
coordinate-free). The key thing here is that $A$ is a suitable basis
change matrix if and only if $\alpha(g)A = A$ for all $g \in G$.</p>
<p>Even better, the projection from $\text{GL}(V)$ to the subspace of the
suitable $A$ is given by the map (a specialisation of a more general
theorem from Serre):</p>
<p>$$p = \sum_g \alpha(g)$$</p>
<p><em>This</em> is where the key speed gain is: we can use the summation trick
from before. The sacrifice is that the matrices $\alpha(g)$ are
absolutely huge. If $n$ is the dimension of $V$, this uses
$\text{O}(n<sup>4</sup>)$ space. Terrible unless you come up with another trick
to cut down the space usage.</p>
<h3>Be lazy where possible to get the space back</h3>
<p>We don't actually care what the matrices $\alpha(g)$ are, we just need
to know how to apply them to compute the image of the $p$ above, then
pick an invertible element from this image.</p>
<p>Even better, you can pick a random $B$ and $pB$ will almost always be
invertible in a rigorous sense: the determinant map $B \mapsto
\text{det}(pB)$ is polynomial, nonzero (we know $pB$ is invertible for
<em>some</em> $B$), so its zero set is measure zero. In practice, you can
just generate a random matrix and it will always work as a basis
change matrix.</p>
<p>So in the end we don't need to ever compute the matrices $\tau \otimes
\rho^\ast$ in full all at the same time, we can just compute:</p>
<p>$$A = pB = \sum_g (\tau(g) \otimes \rho^\ast(g))(B)$$</p>
<p>The tensor product has the neat property that $(A \otimes B)(C \otimes
D) = AC \otimes BD$. We can write the matrix $B$ as $\sum_{i,j}
B_{ij} e_i \otimes e_j$, that is we "vectorise" the matrix $B$ into
an $n^2$ length vector. Then the equation above simplifies to:</p>
<p>$$A = \sum_g \sum_{i,j} B_{ij} \tau(g)e_i \otimes
\rho^\ast(g)e_j$$</p>
<p>So we have ended up with a formula for $A$ (the basis change matrix)
which doesn't require the computation of any matrices with $n^4$
entries, and is amenable to the fast group summing method vaguely
mentioned earlier.</p>
<p>Since we still have to do matrix multiplications many times, this
whole algorithm is still at least $\text{O}(n^3)$ but using clever
tensor product tricks, we got it down from $\text{O}(n^4)$. Not bad!</p>
<p>One thing to mention is that the independence of the run time from
$|G|$ frees us to compute with huge groups. My algorithm can operate
on small degree representations of groups like $S_{10}$, with $10!$
elements (3.6 million elements).</p>
<p>Hopefully this has been some kind of insight into what computational
representation theory might involve. Take a look at the
<a href="https://github.com/gap-packages/RepnDecomp">source code</a> for the
gritty details.</p>
<h1>What did you gloss over?</h1>
<p>There's one huge thing I haven't explained. Have you guessed what it
is?</p>
<p>You can't represent $\mathbb{C}$ in a computer! Floating point is
useless for algebra! So how were we multiplying matrices and adding up
coefficients in $\mathbb{C}$? There's another trick: restrict your
attention to the
<a href="https://en.wikipedia.org/wiki/Cyclotomic_field">cyclotomic numbers</a>.</p>
<p>But why is that good enough? Is multiplication still constant time?
Was it ever constant time?</p>
<p>I'll answer these questions at some point.</p>
How to accidentally become a maintainer of a projecturn:https-kaashif-co-uk:-2020-04-15-how-to-accidentally-become-a-maintainer-of-a-project2020-04-15T00:00:00Z2020-04-15T00:00:00ZKaashif Hymabaccus
<p>I'm somehow one of the maintainers of
<a href="https://github.com/rss2email/rss2email">rss2email</a>, a popular Python
program for reading RSS feeds and sending feed updates to your email
address. I think I reached this point via an unusual route, so I
thought I'd write a little about how it happened.</p>
<p>For those not familiar, back in 2004, Aaron Swartz (yes, that one,
rest in peace) wrote a short Python script that would read an RSS feed
and send you emails. It had a few options, but was fairly simple. It
was a few hundred lines long.</p>
<p>After 16 years of features being added and being passed around
various maintainers (see
<a href="https://raw.githubusercontent.com/rss2email/rss2email/master/AUTHORS">here</a>
for a complete list of contributors), rss2email is still around, with
many more features, many more lines of code, and many (many) more
bugs.</p>
<p>How did I get involved?</p>
<!--more-->
<h2>How I got involved</h2>
<p>Back in 2017, when I was still studying for my Master's degree, I was
looking for open source projects to take part in to pad my CV, mainly
to impress interviewers for summer internships. This led to me looking
at the list of software I used and trying to contribute to all of
them, fixing any bugs that annoyed me, updating any packages that were
out of date, and so on.</p>
<p>At the time, OpenBSD (which I've always had a soft spot for), had a
severely outdated version of rss2email in its packages
repository: 2.70. At the time, the current version was 3.9. I sent an
email to ports@openbsd.org with some updates, expecting it to be
accepted without fanfare. Since I was already a user of the new
version, I didn't realise the sheer number of breaking changes
introduced, so I couldn't predict the
<a href="https://marc.info/?l=openbsd-ports&m=150335670213521&w=2">long discussion about how to upgrade</a>
that ensued.</p>
<p>Eventually, a consensus was reached that the changes were too
breaking, so a new package would have to be made. Even worse, the new
version had many more bugs than the old version, so existing users on
the ports@ mailing list questioned why they should even upgrade.</p>
<p>The OpenBSD package was shelved.</p>
<p>Around this time, @wking's fork was the most updated fork of
rss2email - there had been a Python 3 conversion, a test suite had
been added, lots of features. However, the fork wasn't very
active. Users would open issues, make PRs, but there weren't enough
maintainers and not enough manpower to keep the project going. In the
past, when this happened, another lone maintainer took over. This is
essentially what @wking did, contributing in a big way for many
years.</p>
<h2>The bus factor</h2>
<p>Python is the 2nd most famous example of a project which was run by a
BDFL, a benevolent dictator for life, Guido van Rossum. In 1994,
someone made
<a href="https://legacy.python.org/search/hypermail/python-1994q2/1040.html">this post on a newsgroup</a>:</p>
<blockquote><p>What if you saw this posted tommorrow.</p>
<p>Guido's unexpected death has come as a shock to us all. Disgruntled
members of the Tcl mob are suspected, but no smoking gun has been
found...</p>
<p>I just returned from a meeting in which the major objection to using
Python was its dependence on Guido. They wanted to know if Python
would survive if Guido disappeared. This is an important issue for
businesses that may be considering the use of Python in a product.</p>
<p>-- Michael McLay (mclay@eeel.nist.gov)</p></blockquote>
<p>This led to the popularity of the term "bus factor". A bus factor of n
indicates that n team members would have to "get hit by a bus" for the
project to stall. In our case a core team member had already passed
away, before most of us had started contributing: Aaron Swartz. So
the bus factor wasn't a joke - a low bus factor had already almost
stalled rss2email for good.</p>
<p>Enter @jsbackus, who, in 2018, decided to create the
<a href="https://github.com/rss2email">rss2email GitHub org</a> and add some
erstwhile contributors to the project, in the hopes of revitalising
it and increasing the bus factor.</p>
<p>Probably by accident, I was included in this effort. See
<a href="https://github.com/rss2email/rss2email/issues/19">here</a>. I guess the
strategy was to add anyone who had shown an interest in maintaining
the project. Here's the fateful comment by @Profpatsch:</p>
<blockquote><p>An update: Jeff has added @Ekleog, @kaashif, @Yannik and myself to
the rss2email organization, so we'll hopefully be able to
administrate the project with more manpower.</p>
<p>-- @Profpatsch</p></blockquote>
<p>And there you have it, I was a maintainer!</p>
<h2>What does maintaining a project mean, anyway?</h2>
<p>After a sizeable effort by the other maintainers to migrate Linux
distros to point to our repo, and to claim
<a href="https://pypi.org/project/rss2email/">the package on PyPI</a>, we were
all set to start writing PRs and fixing bugs! Off to the OpenBSD
mailing list to update to 3.9 (again), but this time with new and
improved links to the new repo.</p>
<p><a href="https://marc.info/?l=openbsd-ports&m=151285481902867&w=2">It didn't go too well</a>,
this package never made it. The objections can really be summarised by
some feedback I received from an OpenBSD user of the old rss2email:</p>
<blockquote><p>This is at least the second unreleased fix I noticed for newer
python3+ versions. This begs the question as to how this newer
version is better than what we already have in ports...</p>
<p>-- jca</p></blockquote>
<p>Version 3.9 was the last version released by @wking, from before the
effort to get all of the forks and maintainers under one roof to fix
all the bugs. No surprise that it was riddled with bugs then. Back to
the drawing board.</p>
<p>Eventually, a year later, we fixed a lot of bugs. The other
maintainers put a lot of work in to incorporate forks and many users'
personal bugfixes and we released our first version, 3.10.</p>
<p>This time I had a bit more success in
<a href="https://marc.info/?l=openbsd-ports&m=156736204030157&w=2">my effort to get the package added to ports</a>.
At some point, I managed to convince sthen@ (or he decided
independently of me) that rss2email didn't need a new package and
could just work as a major upgrade. This was because there was a
working upgrade path. So if you look at the
<a href="https://openports.se/mail/rss2email">OpenBSD port of rss2email</a>,
you'll see that:</p>
<ol>
<li>I am the maintainer</li>
<li>It's not a new package: it's a straight upgrade</li>
</ol>
<p>Amusingly, someone updated it from 2.70 to 2.71 one month before I
updated it to 3.10.</p>
<h2>Plans for the future</h2>
<p>Nowadays, when I get time to work on rss2email, I respond to users'
issues on GitHub, review PRs, merge PRs, fix bugs, and so on. It's not
glamorous, but it's honest work.</p>
<p>Our plan is simple:</p>
<ol>
<li>Fix bugs</li>
<li>Don't make users' lives difficult</li>
<li>Add new features</li>
</ol>
<p>In that order. If something's nonsense, sometimes you've got to break
it to fix it...</p>
<p>As always, contributors are welcome! Happy hacking!</p>
Electromagnetic weaponry for fun and profiturn:https-kaashif-co-uk:-2019-10-29-electromagnetic-weaponry-for-fun-and-profit2019-10-29T00:00:00Z2019-10-29T00:00:00ZKaashif Hymabaccus
<p>Anyone who has played Halo and fired the Gauss cannon on the Warthog
has experienced a strong desire to do it in real life. Usually this is
cut down by the idea that Gauss guns aren't real. But of course, they
are, and I've built a few. Feel free to replace all mentions of
"coilgun" with "Gauss superaccelerator" or "magnetic spaceship cannon"
if this will satisfy a childhood fantasy of yours.</p>
<p>There are two main types of coilgun I've built:</p>
<ul>
<li><p>The "standard" kind, which involves turning an electromagnet on,
attracting a ferromagnetic projectile down a barrel towards the
coil, then turning the magnet off once it reaches the centre of the
coil. This is a reluctance coilgun.</p></li>
<li><p>The other kind, which involves turning an electromagnet on and using
the sudden change in magnetic field to induce eddy currents in a
non-ferromagnetic projectile. The projectile must be an inductor of
some kind (e.g. a shorted coil) so that the eddy currents form a
magnetic field that <em>repels</em> the projectile <em>away</em> from the
coil. This is an inductance coilgun.</p></li>
</ul>
<p>I thought an induction coilgun would be easier to time, since we don't
have to quickly turn off the coil, but intractable problems led me to
turn my coilgun into a reluctance coilgun.</p>
<!--more-->
<h2>Why not a railgun?</h2>
<p>I actually am trying to build a railgun, but it is extremely obvious
to me why coilguns are more popular among hobbyists: you don't need
moving contacts for a coilgun! It's that simple!</p>
<p>Inductance or reluctance, a coilgun is not a railgun and
doesn't require any electrical contact between the projectile and the
coil.</p>
<h2>What do I need to build it myself?</h2>
<p>The bill of materials for a minimal version of this project is
actually quite...minimal. My very first coilgun was a low voltage (30V)
design, built with parts I had on hand. The voltage had to be low
because I had not yet acquired any high voltage diodes or
capacitors. All I had was:</p>
<ul>
<li><p>Thyristors salvaged from a power drill battery charger
(BT151-500R) - decent voltage capability but disappointingly low
non-repetitive on-state current capability. This is not strictly
necessary for a minimal design, so I won't use it right now.</p></li>
<li><p>Some enamelled magnet wire from a transformer from an old computer
power supply.</p></li>
<li><p>Some 200V 680uF capacitors from the same power supply. That voltage
tolerance is weird because it's below what would come from the mains
but really high for a "low voltage" project. Nothing special about
the capacitance, it was just what I had.</p></li>
<li><p>A 120W halogen lightbulb for current limiting.</p></li>
<li><p>Various light switches and wiring for the SPDT and DPDT switches you
will see later on.</p></li>
<li><p>PVC pipe. Get pipe as thin as possible, you want the projectile to
be light and fill the barrel, which is much easier if the barrel is
smaller. The fact that it doesn't conduct is important to avoid eddy
currents (see Lenz's law).</p></li>
</ul>
<p>My first Gauss rifle had this design:</p>
<p><img src="/static/gauss1.png" alt="baby's first gauss gun" /></p>
<p>The idea is that all switches are open to start off. Then you close
SW1 to charge the capacitor, open SW1 and close SW2 to
discharge. There is no timing or triggering circuitry involved, we
rely on the capacitor current petering out by the time our projectile
passes the coil (for a reluctance launcher) or we just don't care at
all (for an inductance launcher).</p>
<p>On a practical level, you wrap the coil around the pipe, put the
projectile somewhere in it, and experiment with different capacitances
and projectile starting positions until you get something that works
as well as possible. I did mention that I started out only using 30V,
so you get a stored energy (with two capacitors in parallel) of:</p>
<p>$$E = \frac12 C V<sup>2</sup> = \frac12 (1360 \mu \text{F}) (30 \text{V})<sup>2</sup>
= 0.612 \text{ joules}$$</p>
<p>Is that a lot? Who knows? At 100% efficiency (completely
preposterous), this would propel a 1 gram iron nail to:</p>
<p>$$v = \sqrt{\frac{2E}{m}} = \sqrt{\frac{2(0.612 \text{J})}{1
\text{ gram}}} = 35 \text{ms}^{-1}$$</p>
<p>Pretty decent, right? Wrong! So many things were wrong with this
initial design it's incredible. Here's a rundown so you don't have to
repeat my mistakes.</p>
<h3>Efficiency? What's that?</h3>
<p>Hobbyist coilguns peak in efficiency at about 1%. Even really good
hobbyists can only squeeze out a few percent with really well-designed,
multi-stage, low-friction, narrow barrel designs. Those of us building
with random pipes and scavenged components can only hope for fractions
of a percent.</p>
<p>The solution is to get bigger caps: higher voltages and
capacitances. Voltage is more important since it's the highest order
term in the equation. Too much capacitance isn't desirable since this
increases the time constant $RC$ of the circuit and will lead to
slower discharge.</p>
<h3>Safety? Never heard of it!</h3>
<p>If you accidentally close both switches, you blow the fuse in your
plug or, if you're unlucky, you trip your house's circuit breaker and
all of the lights shut off. Very dramatic and easily avoided if you
wire your switches differently or add some current limiting (in the
form of a bulb).</p>
<h3>Back EMF</h3>
<p>As I learnt (explosively), when you shut off current to an inductor
(e.g. by opening a switch), there is a large back EMF. Essentially, a
large voltage is induced in the reverse direction, damaging/destroying
any polarised components, like that electrolytic capacitor.</p>
<p>The reason for this is apparent if you look at the inductor equation:</p>
<p>$$V = L \frac{dI}{dt}$$</p>
<p>If you cut off the current suddenly, the rate of change is negative
infinity, and so is the voltage over the inductor. In the real world,
you don't really get an infinite back EMF, but it'll be more than
enough to fry your caps.</p>
<p>How does the current traverse the gap when you open your switch? The
inductor always wins and closes the gap with an arc, probably damaging
your switch too.</p>
<p>The solution is to add a flyback diode, this is a diode connected in
reverse parallel with the inductor, so the back EMF is conducted
through it and dissipates the energy stored in the inductor
safely. Diagrams will come later to make this clearer.</p>
<h2>A better design</h2>
<p>Some lessons learnt:</p>
<ul>
<li>Bigger caps</li>
<li>Bigger voltages</li>
<li>Bigger currents</li>
<li>...and also more safety I guess</li>
</ul>
<p>Sounds good. Here's what I did to apply these lessons:</p>
<ul>
<li><p>Take apart some servers to get some better caps. I managed to find
some 450V 470uF caps like this, good enough to charge directly from
the mains.</p></li>
<li><p>Replace SW2 with a high-current thyristor, so we can fire the gun
many times without needing to replace the switch. I ended up
choosing the TYN640RG which can handle 640V (more than enough) and a
peak current of ~500A, which should be fine.</p></li>
<li><p>Add a current-limiting lightbulb so even if the thyristor and switch
fail closed, it just safely turns on the lightbulb. I just found a
120W 240V halogen bulb in a drawer.</p></li>
<li><p>Add a flyback diode! I bought a reel of 1N5408G rectifying diodes
that can block 1000V and have a fairly small (1.1V) forward
voltage. They can also handle fairly large currents.</p></li>
</ul>
<p>Here's what my final single-stage design looks like in circuit form:</p>
<p><img src="/static/gauss2.png" alt="baby's second gauss gun" /></p>
<p>The idea of the circuit is the same, close SW1 to charge, open it,
then close SW2 to fire. The difference is that now, if you forget and
close everything, the lightbulb just turns on and everything remains
safe.</p>
<p>It goes without saying that you should use a ton of wire, otherwise
this will draw a huge current and destroy your thyristor. You could
also add a tiny resistor if you feel like it, but beware of a high
time constant slowing down your projectile, don't make R too large.</p>
<h2>Where to now?</h2>
<p>I would do some calculations and measurements to work out how good my
coilgun is, but there's no point: I already know the path to higher
efficiency is having more stages. And bigger voltages will mean faster
speeds and maybe something worth making a video about.</p>
<p>I'll also have learnt something about how to make a practical
high-speed switching circuit with a photodiode, IR LED and maybe IGBTs
instead of SCRs (thyristors), since IGBTs can be turned off.</p>
<p>Or maybe I'll make an induction ring launcher or something.</p>
Finding uses for neodymium magnetsurn:https-kaashif-co-uk:-2019-09-04-finding-uses-for-neodymium-magnets2019-09-04T00:00:00Z2019-09-04T00:00:00ZKaashif Hymabaccus
<p>I was decommissioning a few of my older computers and servers,
stripping them down for parts. The hard drives in them were mostly IDE
drives which had long stopped working. There are almost no useful
parts you can strip from a non-functional hard drive except maybe the
IDE connector and of course, the extremely strong rare-earth magnets.</p>
<p>But what could I do with them? Here are some things I did and
photographed:</p>
<ul>
<li>Book light that holds on with magnets</li>
<li>Converting a Star Trek combadge from using pins to magnets</li>
<li>Sticking an amplifier to the bottom of my desk</li>
</ul>
<!--more-->
<h2>Book light</h2>
<p>I read a lot of books, not all of them during the day. An easy
solution for reading books in bed is to have a lamp, but sometimes you
want a more portable solution. Who am I kidding, this is just a flimsy
excuse to use some magnets.</p>
<p>I got the battery pack from a small torch, a PCI card filler from a
Sun workstation, and a light from a torch built into a radio I got
from the Science Museum in London 15 years ago. After a few generous
applications of hot glue and solder, this is what I ended up with:</p>
<p><img src="/static/light_side.jpg" alt="Side View" /></p>
<p>In that picture, it's attached to a book using two magnets. One glued
to the light itself and the other inside the book. The idea is that
once you get past halfway in the book, you switch the side the light
is on. And there's no need to worry about the light falling out, the
magnets can lift several kilos.</p>
<p>Here's one of me reading Dune:</p>
<p><img src="/static/dune.jpg" alt="Dune" /></p>
<h2>Converting a combadge</h2>
<p>Here's the product I have at the moment:</p>
<p><img src="/static/orig_badge.jpg" alt="Original Combadge" /></p>
<p>I got it at Destination Star Trek in Birmingham a couple of years
ago. It's a good badge, but you have to actually poke holes in your
shirt to wear it. This is suboptimal. An improvement would be to have
one magnet glued to the badge and one you put inside your shirt.</p>
<p>I cut the pins and filed the stumps down, applied some hot glue and
the magnet and voila! A magnetic combadge!</p>
<p><img src="/static/magnet_badge.jpg" alt="Magnetic Combadge" /></p>
<p>You may wonder why I used hot glue instead of something else, like
superglue. The answer is that hot glue is very forgiving and not too
strong: you can easily remove it with a heat gun and some pliers.</p>
<h2>Below-desk amplifier</h2>
<p>Recently I've been trying to design and build some amplifiers for
various speakers. I have a class B amplifier in the works, but all of
the amplifiers I have managed to bias correctly and so on are class
A.</p>
<p>Currently, I'm using a class A common emitter amplifier with some 4
ohm speakers I found. A problem I had is that I kept knocking it off
my desk, I needed to either stick it down or put it under my
desk. Using magnets, I managed to do both:</p>
<p><img src="/static/amplifier.jpg" alt="Amplifier under desk" /></p>
<p>If you're interested, each speaker is driven by a 2N2222 transistor,
with just a base-collector bias resistor (not a very good or complex
design, I know, I know) and a 5 ohm 5 watt power resistor in series
with the speaker so we don't exceed the current limits of the
transistor. I did that once and my transistors let out their magic
smoke, I decided it was better to live with inefficiency than have
nothing at all. Some bigger transistors are coming in the post though.</p>
<p>I'm looking to design some more complex amplifiers and use them with
random speakers from eBay - some huge speakers go for very cheap with
no bids, it's very bizarre. Stay tuned. Or maybe watch the local news
for an article about a man killed by overpressure from dodgy speakers.</p>
<h2>Other magnet fun</h2>
<p>I've also repaired fridge magnets, made a screw holder, magnetised
some screwdrivers and more! But most of those applications are too
mundane to write a blog post about.</p>
Modding a Sun Ultra 45 fan moduleurn:https-kaashif-co-uk:-2019-08-14-modding-a-sun-ultra-45-fan-module2019-08-14T00:00:00Z2019-08-14T00:00:00ZKaashif Hymabaccus
<p>I have a Sun Ultra 45, the last and most powerful Sun SPARC
workstation. Even though mine doesn't have both of the two CPU slots
filled, the fans are still really loud. Is there a way to control the
fans? How could I slow them down?</p>
<p>I first tried to find a software solution on OpenBSD and later had to
resort to soldering and adding some resistors to the fans to slow them
down.</p>
<!--more-->
<h2>The Problem</h2>
<p>To understand the problem, you just need to know how many fans there
are. There's a fan on the CPU heatsink: this one is fairly quiet and I
have no problems with it. The loud fans are the three huge monster
fans between the motherboard and the hard drive/disc drive bays.</p>
<p>Here's a picture of the three fan module removed with various crap
around for scale:</p>
<p><img src="/static/fan1.jpg" alt="Fan Module" /></p>
<p>Those fans are huge, they take 12V DC and draw 1.3 Amps! That's a
power drain of:</p>
<p>$$3 \times 12 \text{V} \times 1.3 \text{A} = 46.8 \text{W}$$</p>
<p>50W just for fans? No wonder they sound like jet engines!</p>
<h2>Software fan control</h2>
<p>The first port of call is a software solution: can we use a fan
control program to slow the fans? The support for fan control on
OpenBSD is already sparse, even on amd64, so this was unlikely.</p>
<p>On amd64, there is generally good fan sensor support: you can view how
fast your fans are going. I tried to get this working on sparc64, but
couldn't, so I'd have to content myself with knowing my fans were
"loud".</p>
<p>If I couldn't control the fans in OpenBSD, could I control them in
Solaris? I took a closer look at the fan module to see where it
connected to the backplane:</p>
<p><img src="/static/fan2.jpg" alt="Fan Connector" /></p>
<p>Using a 12V bench power supply and some crocodile clips, I tested the
connections to make sure the wire colours were true to what I
knew. They were. Ground was black, 12V was yellow. The other two were
clearly fan speed control pins: pulse width modulation (full voltage
but pulsed at different speeds to control fan speed) and voltage
control (varying voltage to control speed). There are 3 blocks of 4
pins on the connector, one block for each fan. The top two pins are
ground and 12V, the bottom two are for speed control.</p>
<p>It seems to me like it might be possible to use PWM to control fan
speed, but not out of the box with OpenBSD. I didn't want to try
Solaris since every time an Oracle logo appears on my screen, a puppy
dies. Next!</p>
<h2>Hardware fan "control"</h2>
<p>There is one other way to reduce fan speed: reduce the voltage across
the fan. The current draw of the fan varies so ideally we'd have a
voltage regulator, but I didn't have any on hand.</p>
<p>A workable candidate would've been something like a 7806 regulator to
halve the fan voltage to 6V. The 78xx range of linear voltage
regulators can provide up to 1.5A of current, so that'd be more than
enough. How much is "enough" exactly? Modelling the fan as a resistor
(sue me):</p>
<p>$$R = \frac{12 \text{V}}{1.3 \text{A}} = 9.2 \Omega$$</p>
<p>So the current draw at 6V would be:</p>
<p>$$I = \frac{6 \text{V}}{9.2 \Omega} = 0.65 \text{A}$$</p>
<p>We could even get away with the low power versions of the 78xx
regulators at that sort of current. The issue is, of course, that I
don't have any spare voltage regulators lying around!</p>
<p>My other option was using some power resistors I had lying
around. From the earlier calculation, we know the fan is something
like a 10 ohm resistor. So to halve the speed I added one 10 ohm
resistor in series with each fan. Problem solved.</p>
<h2>Power dissipation</h2>
<p>Resistors get hot. How hot? To work this out, we need to know the
power being dissipated by the resistor. Voltage splits in proportion
to resistance, so we can say there are 6V across the resistor, so:</p>
<p>$$P = \frac{(6 \text{V})<sup>2</sup>}{10 \Omega} = 3.6 \text{W}$$</p>
<p>Kind of a lot for a resistor, certainly a 1/4 W resistor wouldn't
cope. But the resistors I had lying around were 10W, so no problem.</p>
<p>How do we deal with the heat? Well, we are powering fans, so I just
thought I could dangle the resistors in front of the fans and cut some
holes in the plastic so everything lines up nicely. If that goes badly
for me, I'll certainly write a post about it.</p>
<h2>Does it work?</h2>
<p>So far, yes. Here's a picture of the final fan module:</p>
<p><img src="/static/fan3.jpg" alt="Modded fan module" /></p>
<p>I had to cut away some of the plastic for the resistors to fit. A more
elegant solution was probably possible, but I'm not going to sell this
Ultra 45, so I'm fine with it being a bit damaged aesthetically.</p>
<p>Like I said, the dream solution would be to provide the fans with a
regulated 6V, perhaps with a 78xx regulator. As long as the fans work
ok with my hack job solution, I probably won't end up going that way
though.</p>
Hacking into a Sky routerurn:https-kaashif-co-uk:-2019-08-06-hacking-into-a-sky-router2019-08-06T00:00:00Z2019-08-06T00:00:00ZKaashif Hymabaccus
<p>Like everyone, I have a ton of old routers lying around. It pains me
to see these very useful computers go to waste, so I made it my
business to hack into all of mine and replace the firmware. Maybe the
title is a bit dramatic, but it's technically accurate.</p>
<p>My first target was an old Sky router, a Sagemcom F@ST2504n.</p>
<!--more-->
<h2>Which OS?</h2>
<p>The obvious choice is <a href="https://openwrt.org">OpenWRT</a> since they have a
page with
<a href="https://openwrt.org/toh/sagem/sagem_fast2504n_v.6">easy instructions</a>
to get started.</p>
<p>There is one issue though, the pictures on that page don't work! This
wasn't too much of an issue, since the pads on the PCB were all
labelled so I was able to solder without incident.</p>
<h2>Getting a console</h2>
<p>I mentioned soldering. This is needed because there is no way to
access the bootloader to flash the firmware without getting a
console. There isn't a serial port on the case, but there are pads on
the PCB conspicuously labelled VCC, Tx, Rx and GND. This is where you
can solder some wires and connect your favourite 3.3V TTL UART. I used
an FT232, but I'm sure others work too.</p>
<p>Interestingly, if you set your UART to 5V, you can actually power the
whole router from the serial port! I discovered this by accident, I
had my FT232 set to 5V and connected it without having power connected
to the router. Rather spookily, the router powered on without any
power. But this is possibly unsafe, don't do it! Who knows how much
current the router needs, it's best to use the barrel jack and a real
power supply!</p>
<p>Anyway, after changing to 3.3V, I got a console, here is what it looks
like:</p>
<pre><code>CFE version 5.14.7 for BCM96362 (32bit,SP,BE)
Build Date: Tue Mar 29 15:03:07 CST 2011 (zouchenbo@SZ01007.DONGGUAN.CN)
Copyright (C) 2005-2010 SAGEM Corporation.
HS Serial flash device: name MX25L64, id 0xc217 size 8192KB
Total Flash size: 8192K with 128 sectors
Chip ID: BCM6362B0, MIPS: 384MHz, DDR: 320MHz, Bus: 160MHz
Main Thread: TP0
Memory Test Passed
Total Memory: 67108864 bytes (64MB)
Boot Address: 0xb8000000
Board IP address : 192.168.1.1:ffffff00
Host IP address : 192.168.1.100
Gateway IP address :
Run from flash/host (f/h) : f
Default host run file name : vmlinux
Default host flash file name : bcm963xx_fs_kernel
Boot delay (0-9 seconds) : 1
Board Id (0-2) : F@ST2504n
Number of MAC Addresses (1-32) : 11
Base MAC Address : 7c:03:4c:ad:19:f6
PSI Size (1-64) KBytes : 40
Enable Backup PSI [0|1] : 0
System Log Size (0-256) KBytes : 0
Main Thread Number [0|1] : 0
*** Press any key to stop auto run (1 seconds) ***
Auto run second count down: 1
web info: Waiting for connection on socket 0.
CFE>
</code></pre>
<p>Some interesting info there, but we actually don't care about any of
it except the prompt telling us to stop the auto boot. Mash enter or
something and you'll end up at the CFE prompt.</p>
<p>I wasn't able to get minicom to work with this console, I couldn't
write anything, leading me to waste some time checking my soldering. I
switched to cu (from the cu package on Debian, it's in base on
OpenBSD), with the recommended 115200 8N1 settings and it worked as
above. I don't really know how to use minicom, so it was probably just
user error.</p>
<h2>Flashing the firmware</h2>
<p>From here, you can follow the instructions. For a TFTP server, I used
the one from the <code>tftpd-hpa</code> package on Debian. The one in the base
system on OpenBSD also works. Here is what I did, I erased the flash
then wrote the image from the TFTP server:</p>
<pre><code>CFE> e a
Erase all flash (except bootrom)? (y/n):y
...............................................................................................................................
Resetting board...HELO
CPUI
L1CI
HELO
CPUI
L1CI
DRAM
----
PHYS
ZQDN
PHYE
DINT
LASY
USYN
MSYN
LMBE
PASS
----
ZBSS
CODE
DATA
L12F
MAIN
CFE version 5.14.7 for BCM96362 (32bit,SP,BE)
Build Date: Tue Mar 29 15:03:07 CST 2011 (zouchenbo@SZ01007.DONGGUAN.CN)
Copyright (C) 2005-2010 SAGEM Corporation.
HS Serial flash device: name MX25L64, id 0xc217 size 8192KB
Total Flash size: 8192K with 128 sectors
Chip ID: BCM6362B0, MIPS: 384MHz, DDR: 320MHz, Bus: 160MHz
Main Thread: TP0
Memory Test Passed
Total Memory: 67108864 bytes (64MB)
Boot Address: 0xb8000000
** Flash image not found. **
Board IP address : 192.168.1.1:ffffff00
Host IP address : 192.168.1.100
Gateway IP address :
Run from flash/host (f/h) : f
Default host run file name : vmlinux
Default host flash file name : bcm963xx_fs_kernel
Boot delay (0-9 seconds) : 1
Board Id (0-2) : F@ST2504n
Number of MAC Addresses (1-32) : 11
Base MAC Address : 7c:03:4c:ad:19:f6
PSI Size (1-64) KBytes : 40
Enable Backup PSI [0|1] : 0
System Log Size (0-256) KBytes : 0
Main Thread Number [0|1] : 0
web info: Waiting for connection on socket 0.
CFE> f 192.168.1.4:/home/kaashif/firm.bin
Loading 192.168.1.4:/home/kaashif/firm.bin ...
Finished loading 3932164 bytes
Flashing root file system and kernel at 0xb8010000: .............................................................
Flashing File Tag....
*** Image flash done *** !
</code></pre>
<p>192.168.1.4 is the address of my laptop, I connected to the router
directly with an Ethernet cable. Now (after waiting for it to boot)
you end up at the OpenWRT shell:</p>
<pre><code>BusyBox v1.28.3 () built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 18.06.0-rc2, r7141-e4d0ee5af5
-----------------------------------------------------
=== WARNING! ======�================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:/# uname -a
Linux OpenWrt 4.9.111 #0 SMP Sat Jul 14 13:48:14 2018 mips GNU/Linux
</code></pre>
<p>Fantastic. That's one router set free, I'll do the rest (and not write
posts about them, since I imagine they'll all be the same as this one).</p>
<p>One last thing, here's a picture of the router in action, with the
serial terminal connected:</p>
<p><img src="/static/router.jpg" alt="" /></p>
My first homebrew computer!urn:https-kaashif-co-uk:-2019-07-10-my-first-homebrew-computer2019-07-10T00:00:00Z2019-07-10T00:00:00ZKaashif Hymabaccus
<p>I've always wanted to design and build my own homebrew computer. By
this, I mean buying some ICs and soldering together something like an
Apple I or a ZX Spectrum.</p>
<p>This post isn't going to be about my struggles designing a homebrew
PC, since I haven't done that. Instead, I bought a new Z80 homebrew
kit, the <a href="https://rc2014.co.uk/">RC 2014</a>. That's right, there is
still someone out there making these kits.</p>
<!--more-->
<h2>Why the Z80?</h2>
<p>There are a few reasons I picked a Z80 as my first homebrew. The first
is that you can buy a Z80 new from any reputable supplier: Zilog is
still in business and still makes them new! You can just log onto
DigiKey or whatever and add it to your basket.</p>
<p>The other is that you don't have to breadboard it: you can buy the
RC2014 kit I linked to earlier, you just have to solder it together. I
admit it's been a while since I've soldered anything but after getting
this kit working, I can feel a desire to get the stripboard out and
design some peripherals. So soldering is not a turn-off for me. This
means you could theoretically build an entire Z80 computer without
touching any wires (except the serial cable).</p>
<p>The other other reason is that there is an active community not just
around the Z80 but around the RC2014 kit specifically. There are tons
of modules people have designed and some you can even buy from
them. And the creator of the RC2014 has a few available including an
HDMI output, keyboard input, and more.</p>
<h2>The build</h2>
<p>I ordered online and I got my kit in the post. What did I get? A nice
set of instructions, all of the resistors, capacitors, buttons and
headers you need (no spares) and some PCBs. All as expected.</p>
<p>The
<a href="http://rc2014.co.uk/wp-content/uploads/2016/07/Full-Monty-Assembly-Guide.pdf">instructions</a>
are very easy to follow and even include some troubleshooting tips.</p>
<p>The soldering iron I use is a cheap Chinese clone of a Hakko. It's
temperature controlled and works with Hakko tips, so I would recommend
it. It only cost 30 GBP and served me well for the (admittedly very
small) amount of soldering I have done so far. No doubt if I get
serious I'll end up having to buy something better. I would link it,
but I am entirely unqualified to recommend soldering irons, so I
won't. That's not the point of this post anyway.</p>
<p>I put the reset button I was given on the clock board and also added
my own reset button (and 2k2 resistor) on the backplane. This is just
because I found the clock board button hard to press without feeling
like I was bending pins.</p>
<p>I chose not to use the barrel jack initially, instead opting to use
power over the FTDI cable. This meant I had to put the jumper on the
serial board to connect the 5V line from the FTDI cable.</p>
<p>And a side note: I didn't solder the right angle headers onto the ROM
board very well, so it "leans back" and touches the serial board. I
just used a folded up piece of card as a spacer to stop any shorts.</p>
<h2>Problems</h2>
<p>I connected my serial cable, and hit the reset button: nothing
happened. Yes, my serial settings were correct and the cable was in
the right way. Now what?</p>
<p>I don't have an oscilloscope capable of reading a 7 MHz signal, I just
have a crappy handheld one that does maybe 200 kHz. This meant that I
was restricted to the instructions for
<a href="https://rc2014.co.uk/troubleshooting/troubleshooting-with-a-multimeter/">troubleshooting with a multimeter</a>. Those
instructions are very comprehensive by the way and did end up solving
my problem.</p>
<p>I checked connectivity of all the lines: all good. No solder bridges
either. But then I noticed that the voltage from the 5V line to ground
was actually 3.3V. This should've been my first clue that my cheap
Chinese FTDI cable was junk. Instead, I soldered on the barrel jack,
moved the jumper so the board would take power from the jack instead
of the FTDI cable, and moved on. All 5V lines were actually 5V now, at
least.</p>
<p>The next step was building the signal tracer circuit on that
troubleshooting page. As I understand it, this is just a spin on a
<a href="https://en.wikipedia.org/wiki/Differentiator">differentiator circuit</a>,
since it gives a voltage only when the input voltage is
changing. i.e. the output is related in some way to the derivative of
the input voltage with respect to time.</p>
<p>Here is a photo of my breadboarded signal tracer:</p>
<p><img src="/static/tracer.jpg" alt="Signal Tracer" /></p>
<p>It worked like a charm and gave me some very useful info:</p>
<ul>
<li>There was a clock signal (good)</li>
<li>Activity on the address lines, so the CPU was running a program</li>
<li>Activity on the data lines, so the RAM and ROM probably worked (this
wasn't conclusive, but they actually did work, so I never needed to
check them in detail)</li>
</ul>
<p>And here was the kicker that took me an age to notice: there was
activity on the TX_Data pin of the serial communications controller
(the MC68B50CP, on the serial board) but <em>only if the FTDI cable was
not connected</em>. So the whole thing worked except the serial board!</p>
<p>I tried a different cable (a generic "Raspberry Pi debug cable",
really a USB to TTL serial adapter) to no avail. Distraught, I emailed
the RC2014-Z80 Google group and got some very good advice from the
designer of the RC2014 himself, Spencer Owen. One of the things he
wrote was that it was probably the serial cable not being sensitive
enough - I should lower the values of the resistors on the TX and RX
lines or just short them.</p>
<p>I shorted them and still my FTDI cable wasn't working! But then I
tried my other generic Chinese cable and it worked!</p>
<h2>The finished result</h2>
<p>Here it is:</p>
<p><img src="/static/rc2014.jpg" alt="RC2014" /></p>
<p>The resistor on the serial board is sticking out in an ugly way since
this was while I was still in the middle of trying out different
resistor values to see if I could get away with not shorting the
connctions. Right now I just have them shorted and the serial
controller gets a bit warm, which is probably not desirable.</p>
<h2>Future steps</h2>
<p>I look forward to writing some Z80 assembly programs, playing some
BASIC games (including my favourite: Star Trek!) and designing some
peripherals to use.</p>
<p>And if you're reading this, maybe you should check out the RC2014,
it's a great kit for beginners (like me).</p>
Containerizing my transcript search appurn:https-kaashif-co-uk:-2019-06-07-containerizing-my-transcript-search-app2019-06-07T00:00:00Z2019-06-07T00:00:00ZKaashif Hymabaccus
<p>Until recently, my transcript search web app was running (at
<a href="https://transcripts.kaashif.co.uk">https://transcripts.kaashif.co.uk</a>, check it out) in a tmux session,
with a PostgreSQL server running on the same machine in the usual
way, as a daemon.</p>
<p>The web app knows nothing about its dependency on the database, this
information is not recorded anywhere except in the code itself. And
the database knows nothing about the web app. This isn't a huge
problem except the database has a config of its own which isn't
recorded anywhere in the source repo. If you try to get the web app
to work with a misconfigured database, it won't work, of course.</p>
<p>Wouldn't it be nice if all of that configuration were in one place?
And if the services all restarted themselves if they failed? And if
you could migrate the entire blob of interconnected web apps and
databases to a different machine with a single command?</p>
<p>That's where Docker comes in!</p>
<!--more-->
<h2>What problem are we trying to solve?</h2>
<p>I was thinking of migrating VPS providers, and I realised that my
strategy of running my web app in a tmux window was poor. All of the
configs are spread out over the system, the database has been
configured to run a certain way and I have a certain way of running
the web app which isn't even recorded in any script anywhere. This
means someone trying to host the app on their own would have to do a
lot of guesswork or rely on me to tell them what to do.</p>
<p>There's also a lack of fault tolerance: if the web app crashes, I have
no idea and it certainly doesn't restart itself. If the database
crashes, I similarly have no idea. It would be nice if we could get
some auto-restarting behaviour.</p>
<h2>Which tools will we use?</h2>
<p>The solution I chose was
<a href="https://docs.docker.com/compose/">Docker Compose</a>. In the words of
that page I just linked:</p>
<blockquote><p>Compose is a tool for defining and running multi-container Docker
applications.</p></blockquote>
<p>In our case, one container is the web app and one the database. This
is a very common setup and is probably exactly the use case Docker
Compose was designed for. This is apparent if you go through the
examples in the docs.</p>
<p>For those who don't know what Docker is, here is a helpful
explanation:</p>
<blockquote><p>Enterprise Container Platform for High-Velocity Innovation: Securely
build, share and run any application, anywhere</p>
<p>-- <a href="https://www.docker.com/">https://www.docker.com/</a></p></blockquote>
<p>Just kidding, that's weapons-grade business nonsense. A container is
essentially a really lightweight copy of your machine where exactly
one program is running. It shares the kernel and hardware, but has its
own network and bundles its own filesystem (with all its
dependencies). <a href="https://opensource.com/resources/what-docker">Here</a> is
an actually good explanation.</p>
<h2>What does the end result look like?</h2>
<p>Rather than describing the long and tedious process of trying to get
everything to work and learning how to use Docker, let my just
show you the end result.</p>
<p>The git repo for my web app looks like:</p>
<pre><code>.
|-- build.sh
|-- cli
|-- data
|-- docker-compose.yml
|-- Dockerfile
|-- lib
|-- LICENSE
|-- make_pretty.sh
|-- README.md
|-- stack.yaml
|-- transcripts
|-- transcript-search.cabal
`-- web
</code></pre>
<p>Assuming you have the transcript parser built (just install
<a href="https://docs.haskellstack.org/en/stable/README/">Haskell Stack</a> and
run <code>stack build --copy-bins</code>, which will get the GHC compiler, all
dependencies, build and copy the binaries to the right place), we only
need to focus on:</p>
<ul>
<li><p><code>data</code>: contains the transcript data, ready to be loaded into the
SQL database</p></li>
<li><p><code>docker-compose.yml</code>: defines the relationships between the
containers and what the names of the images used are</p></li>
<li><p><code>Dockerfile</code>: defines our web app container</p></li>
</ul>
<h2>The data directory</h2>
<p>This is produced using <code>transcript-parse</code>, the Swiss Army Knife of
sci-fi TV show transcript parsers. A small niche, but a very important
one. There are only two files here:</p>
<ul>
<li><code>transcripts.tsv</code>, which is a tab-separated values file with the
entire transcript database. This is the biggest file, at 72 MB. Not
quite big data yet.</li>
</ul>
<p><code>load_data.sh</code>, which the database container will pick up (more on
this later) and use to load the TSV file.</p>
<h2>docker-compose.yml</h2>
<p>The docs for Docker Compose aren't bad, you should check them out if
you want to see some more examples. My code is essentially just
another example to add to the list. They actually don't have a Python
Flask web app as one, so maybe my code will be instructive.</p>
<p>Here is the entire file:</p>
<div class="highlight"><pre><span></span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'3.1'</span><span class="w"></span>
<span class="nt">services</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">web</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">depends_on</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span><span class="w"></span>
<span class="w"> </span><span class="nt">build</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">.</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"1234:8000"</span><span class="w"></span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">DB_HOST</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span><span class="w"></span>
<span class="w"> </span><span class="nt">DB_PASS</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">redacted</span><span class="p p-Indicator">]</span><span class="w"></span>
<span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">always</span><span class="w"></span>
<span class="w"> </span><span class="nt">postgres</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres:11</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"5432:5432"</span><span class="w"></span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">POSTGRES_USER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">transcripts</span><span class="w"></span>
<span class="w"> </span><span class="nt">POSTGRES_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">redacted</span><span class="p p-Indicator">]</span><span class="w"></span>
<span class="w"> </span><span class="nt">POSTGRES_DB</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">postgres</span><span class="w"></span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">./data:/docker-entrypoint-initdb.d</span><span class="w"></span>
<span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">always</span><span class="w"></span>
</pre></div>
<p>For a full reference of what all of these keywords mean, check out the
compose file reference: <a href="https://docs.docker.com/compose/compose-file/">https://docs.docker.com/compose/compose-file/</a>.</p>
<p>For now, I'll just focus on how it solves my problems.</p>
<ol>
<li><p>The full relationship between the services is defined in one
file. This is great, since it means there is no need to trawl
through the code to see where the database connections are made.</p></li>
<li><p><code>restart: always</code>! So if the web app crashes, it will just restart
and the database won't know. If the database crashes, it will
restart. Maybe the web app will crash too, but then it will restart
and at some point the system will be working again.</p></li>
<li><p><code>depends_on</code> means the services get started in the correct order:
database then web app. This means we don't have to care about
waiting for the server to come up in the web app code, we can just
assume it's always up. Docker Compose handles enforcing this
condition for us.</p></li>
</ol>
<p>A technical note: when the database container is started for the first
time, it mounts the <code>data</code> directory into its filesystem at a mount
point with a special name. The Postgres image has a script somewhere
that scans this directory looking for scripts to run. There is a lot
of complexity hidden inside the prebuilt Postgres image, but we do not
need to worry about any of this.</p>
<h2>Our custom container definition</h2>
<p>This is very simple, we just need something to run a Flask app. I
picked Gunicorn, a decent enough WSGI HTTP server for Python
apps. Here's the entire <code>Dockerfile</code>:</p>
<pre><code>FROM ubuntu:18.04
MAINTAINER Kaashif Hymabaccus "kaashif@kaashif.co.uk"
RUN apt-get update && \
apt-get install -y gunicorn3 python3-flask python3-psycopg2
COPY ./web /web
WORKDIR /web
EXPOSE 8000
ENTRYPOINT [ "gunicorn3" ]
CMD [ "-b", "0.0.0.0:8000", "app:app" ]
</code></pre>
<p>Building the container just involves copying the scripts and assorted
goodies (in the <code>web</code> directory) into the container. Running the
container just runs the web app.</p>
<h2>Getting this up and running</h2>
<pre><code>$ docker-compose start
</code></pre>
<p>No, really, it's that easy! This will build the containers and start
them in the right order for you. With full fault-tolerance, isolation
and so on. Feels too easy.</p>
<h2>Conclusion</h2>
<p>Docker is great and I hope to use it in more projects in the
future. Maybe at some point I'll make it big and have to delve into
the world of Kubernetes and Docker Swarm - orchestration on a larger
scale.</p>
<p>Until then, I'm happy with this small-scale success and I highly
encourage you to try containerizing your web apps.</p>
<p>Happy hacking!</p>
HP PA-RISC Assembly Crash Courseurn:https-kaashif-co-uk:-2019-04-18-hp-pa-risc-assembly-crash-course2019-04-18T00:00:00Z2019-04-18T00:00:00ZKaashif Hymabaccus
<p>Since I have access to a machine that has the PA-RISC architecture, I
thought I'd compile some test programs and see what sort of assembly
code produced. Some highlights:</p>
<ul>
<li>A neat way to manage the stack pointer (and one surprise)</li>
<li>Every instruction seems to be shorthand for <code>or</code></li>
<li>Completers - a weird way of giving switches to your instructions</li>
</ul>
<p>PA-RISC is considerably less popular than x86, MIPS, PowerPC, even
SPARC. And being a RISC architecture means that humans hardly ever
wrote assembly for it themselves. Most of the time, programmers
probably never even gave (past tense since PA-RISC is dead) their
binaries a second glance. Or really even any kind of look.</p>
<p>Well, that's about to change! The first program we'll look at is, of
course, hello world.</p>
<!--more-->
<h2>Hello, world!</h2>
<p>I wanted to compile, as a showcase, a program with some
nontrivialities. That means function calls, string literals and
non-leaf and leaf functions (functions that do/don't call other
functions). This will hopefully let us discover the quirks of PA-RISC
in a controlled environment. Here is the first one, which has a leaf
function call with no arguments and a function call with an argument:</p>
<div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">f</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"Hello, world!</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">f</span><span class="p">();</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>And here is the binary, compiled with <code>gcc -O0 -g test.c</code> and dumped
with <code>objdump -S</code>:</p>
<pre><code>000105a8 <f>:
#include <stdio.h>
int f() {
105a8: 08 03 02 41 copy r3,r1
105ac: 08 1e 02 43 copy sp,r3
105b0: 6f c1 00 80 stw,ma r1,40(sp)
return 0;
105b4: 34 1c 00 00 ldi 0,ret0
}
105b8: 34 7e 00 80 ldo 40(r3),sp
105bc: 4f c3 3f 81 ldw,mb -40(sp),r3
105c0: e8 40 c0 02 bv,n r0(rp)
000105c4 <main>:
int main() {
105c4: 6b c2 3f d9 stw rp,-14(sp)
105c8: 08 03 02 41 copy r3,r1
105cc: 08 1e 02 43 copy sp,r3
105d0: 6f c1 00 80 stw,ma r1,40(sp)
printf("Hello, world!\n");
105d4: 23 88 10 00 ldil L%10800,ret0
105d8: 37 9a 01 b0 ldo d8(ret0),r26
105dc: e8 5f 1a ed b,l 10358 <_end_init+0x14>,rp
105e0: 08 00 02 40 nop
return f();
105e4: e8 5f 1f 7d b,l 105a8 <f>,rp
105e8: 08 00 02 40 nop
}
105ec: 48 62 3f d9 ldw -14(r3),rp
105f0: 34 7e 00 80 ldo 40(r3),sp
105f4: 4f c3 3f 81 ldw,mb -40(sp),r3
105f8: e8 40 c0 02 bv,n r0(rp)
</code></pre>
<p>Only the relevant part is included. Now, we have to go through the
<a href="https://parisc.wiki.kernel.org/images-parisc/6/68/Pa11_acd.pdf">PA-RISC ISA Reference Manual</a>
and decipher what all of this means.</p>
<p>Note that there are some examples of C programs and resulting assembly
in that manual, but they aren't explained too much since the manual is
supposed to be a reference, not a beginner's guide. It's also a bit
long, over 400 pages.</p>
<p>Also, I have no idea how to get my hands on the C compilers and
assemblers they used, so I can't verify any of their examples. Moving
on.</p>
<h1>Registers</h1>
<p>All registers are 64 bits wide on PA-RISC 2.0 CPUs (like the one I
have).</p>
<p>If you recall
<a href="https://www.kaashif.co.uk/2018/08/11/register-windows-a-cool-feature-of-sparc/">my article about SPARC assembly</a>,
you'll notice that it's almost entirely about register windows and
related coolness. There is no such magic on PA-RISC, it is rather
similar to x86 in that respect. That is, there are just a number of
registers and you have to just remember what they're for.</p>
<p>Luckily, there are some helpful synonyms on page 28 of the
manual. Here are the important ones:</p>
<ul>
<li><p><code>ret0</code> is <code>r28</code>, the return value. This is set when a function wants
to return something, as we will see.</p></li>
<li><p><code>sp</code> is <code>r30</code>, the stack pointer. There is something weird about how
this is used in the above code which I'll go over later. Can you
guess what it is?</p></li>
<li><p><code>rp</code> is <code>r2</code>, the return link. This is the return link.</p></li>
</ul>
<p>Next comes the argument convention, which is a bit odd: <code>r26</code> is
<code>arg0</code>, <code>r25</code> is <code>arg1</code>, <code>r24</code> is <code>arg2</code>, <code>r23</code> is <code>arg3</code>. Yes, it's
numbered backwards for some unusual reason.</p>
<p>Now we can get started deciphering the code.</p>
<h1>Which way does your stack grow?</h1>
<p>On x86, the stack usually grows downwards. This means if you are at
address 10 and you need more space, you, by convention, decrease the
stack pointer (move it towards zero). The heap starts at the bottom
and grows up. It's the same way on SPARC, PowerPC, MIPS and so
on.</p>
<pre><code>memory addresses growing -->
+---------------------------------------------------------------+
| heap grows --> <-- stack grows |
+---------------------------------------------------------------+
</code></pre>
<p>On PA-RISC, somehow the convention is the opposite - you increase the
stack pointer to allocate more memory. The heap starts at the top and
grows down.</p>
<pre><code>memory addresses growing -->
+---------------------------------------------------------------+
| stack grows --> <-- heap grows |
+---------------------------------------------------------------+
</code></pre>
<p>This doesn't really matter and isn't a cool feature in any way. It's
an interesting difference from the norm, though.</p>
<h1>A leaf function</h1>
<p>Leaf functions are simple, since we don't have to worry about setting
up the registers for callees, we can just try our best to avoid
messing things up for the caller and we're good.</p>
<pre><code>int f() {
105a8: 08 03 02 41 copy r3,r1
105ac: 08 1e 02 43 copy sp,r3
105b0: 6f c1 00 80 stw,ma r1,40(sp)
</code></pre>
<p>This is us saving the stack pointer. While <code>copy</code> may seem
self-explanatory, it is actually a pseudo-operation, meaning the
hardware doesn't know about it. Instead, <code>copy x,y</code> is shorthand for
<code>or x,0,y</code>, which ors x with 0 and stores it in y.</p>
<p><code>stw,ma r1,40(sp)</code> stores the value of the register <code>r1</code> at
<code>sp+40</code>. Note that we have the x86-like memory address addition
syntax. We can't do multiplications, though, so there is no shortcut
to accessing arrays like on x86, where you can write <code>5*eax+2</code> into a
<code>mov</code> instruction. The <code>stw</code> instruction means "store word", fairly
self explanatory. But what does <code>,ma</code> mean?</p>
<p>In some PA-RISC instructions, there are two bits labeled <code>m</code> and
<code>a</code>. If you use the completer (what the <code>,ma</code> or <code>,mb</code> part is
called), then this sets them in certain ways. What exactly this means
varies for each instruction.</p>
<p>In our case, <code>,ma</code> means "modify after". This is referring to
modifying the base address before/after we calculate the
offset. Modify after means our offset is just the base, then we add
the displacement to the base (actually modifying the base
register). <code>,mb</code> or modify before computes the base + displacement and
uses this as both the final effective address and the value to write
into the base register.</p>
<p>There's a diagram on page 113 of the manual.</p>
<p>This might seem like a pain, but this is essentially designed to make
stack pointer manipulation a breeze: using modify before/after, the
stack pointer can manage itself!</p>
<p>In this case, <code>,ma</code> means the stack pointer is updated essentially
automatically after we save <code>r1</code>.</p>
<p>Next, we need to return:</p>
<pre><code>return 0;
105b4: 34 1c 00 00 ldi 0,ret0
</code></pre>
<p>Again, this seems self explanatory: load 0 into <code>ret0</code>, right? But no,
there is a little more going on here. The "instruction" <code>ldi i,r</code>
(load immediate) is actually a pseudo-operation that generates an
instruction <code>ldo i(0),r</code>. <code>ldo d(b),t</code> is the load offset instruction,
which calculates the offset given by the expression <code>d(b)</code> and loads
this into <code>t</code>.</p>
<p>In our case, <code>ldi 0,ret0</code> calculates the offset <code>0(0)</code>, which is 0,
and loads this into <code>ret0</code>. Due to the instruction encoding requiring
all instructions to be 32 bits long (a common design decision in RISC
architectures), the immediate <code>d</code> is limited to 14 bits in length.</p>
<pre><code> 105b8: 34 7e 00 80 ldo 40(r3),sp
105bc: 4f c3 3f 81 ldw,mb -40(sp),r3
105c0: e8 40 c0 02 bv,n r0(rp)
</code></pre>
<p>This loads <code>40+r3</code> into <code>sp</code>, then uses the <code>ldw,mb</code>
pseudo-instruction to pop a value off the stack (updating the stack
pointer appropriately) into <code>r3</code>. You'll notice that this is value we
saved earlier. This is because <code>r3</code> is callee-saved.</p>
<p>Also, <code>r1</code> is caller-saved, so we don't have to worry about restoring
it. That wasn't really that bad, right?</p>
<h1>The main course</h1>
<p>The main function showcases two features: calling a non-leaf function
(printf) and calling a leaf function (f). Here we go:</p>
<pre><code>000105c4 <main>:
int main() {
105c4: 6b c2 3f d9 stw rp,-14(sp)
105c8: 08 03 02 41 copy r3,r1
105cc: 08 1e 02 43 copy sp,r3
105d0: 6f c1 00 80 stw,ma r1,40(sp)
</code></pre>
<p>Again, we save <code>r3</code> and update the stack appropriately.</p>
<pre><code>printf("Hello, world!\n");
105d4: 23 88 10 00 ldil L%10800,ret0
105d8: 37 9a 01 b0 ldo d8(ret0),r26
105dc: e8 5f 1a ed b,l 10358 <_end_init+0x14>,rp
105e0: 08 00 02 40 nop
</code></pre>
<p>Here's the juicy bit. The string is stored in the data segment, so we
use the <code>ldil</code> instruction to "load immediate into left". This means
we load the immediate (some pointer into the data segment) into the
left part of the <code>ret0</code> register. The left part, in this case, is 32
bits long.</p>
<p>Next, we write the address of the string (imagine it's a <code>char *</code>) to
<code>r26</code>, which is <code>arg0</code>, the first argument of printf.</p>
<p>The branch and link <code>b,l</code> instruction branches (i.e. unconditionally
jumps to the address given) but also places the return point into the
register <code>rp</code>, the link register.</p>
<p>The delay slot is an instruction that is executed <em>before</em> the
branch/jump happens. In this case it's a <code>nop</code>, so nothing happens. But
there is more to this <code>nop</code> than meets the eye: it's a
pseudo-instruction! It really means <code>or 0,0,0</code>, which is a <code>nop</code> since
nothing is changed.</p>
<pre><code> return f();
105e4: e8 5f 1f 7d b,l 105a8 <f>,rp
105e8: 08 00 02 40 nop
}
</code></pre>
<p>Using the branch and link instruction, it's very easy to call <code>f</code>. It
sets <code>ret0</code>, so no need to set it ourselves. Now there's only one
thing left to do...</p>
<pre><code> 105ec: 48 62 3f d9 ldw -14(r3),rp
105f0: 34 7e 00 80 ldo 40(r3),sp
105f4: 4f c3 3f 81 ldw,mb -40(sp),r3
105f8: e8 40 c0 02 bv,n r0(rp)
</code></pre>
<p>We restore <code>r3</code> and <code>sp</code>, the only caller-saved registers! There is a
new instruction here, though, <code>bv</code>. This is a vectored branch, which
sounds interesting. In actual fact, <code>bv,n x(b)</code> just means that we
jump to <code>b</code> added to <code>x</code> left shifted by 3 bits.</p>
<p>That's a full program in PA-RISC assembly!</p>
<h1>Conclusions</h1>
<p>There are some commonalities with both x86 and SPARC.</p>
<p>SPARC:</p>
<ul>
<li>Link registers</li>
<li>Everything is a pseudo-instruction</li>
<li>Delay slots</li>
</ul>
<p>x86:</p>
<ul>
<li>Two operand instructions</li>
<li><code>immediate(register)</code> syntax, although no multiplications</li>
<li>Lots of arithmetic is done using instructions supposedly meant for
calculating addresses.</li>
</ul>
<p>Overally, I would say that PA-RISC isn't really that cool of an
architecture at first glance. It doesn't have anything extra exciting
like SPARC's register windows except completers maybe, but those are
more confusing than anything.</p>
<p>There's probably tons I've missed out, but I have a feeling that there
won't be hordes of HP aficionados chasing me down.</p>
Reviving an HP PA-RISC serverurn:https-kaashif-co-uk:-2019-04-13-reviving-an-hp-pa-risc-server2019-04-13T00:00:00Z2019-04-13T00:00:00ZKaashif Hymabaccus
<p>A while ago, I got my hands on a beast of a machine, a 7U HP L3000
(rp5470) PA-RISC server. These were released in the year 2000 and came
with up to 16GB (whoa) of RAM and up to 4 CPUs.</p>
<p>The best site for information on PA-RISC machines is, no doubt,
<a href="https://www.openpa.net/systems/hp_l1500_l3000-rp5430_rp5470.html">OpenPA.net</a>,
and they have a fantastic page on my machine.</p>
<p>This is the story of how I managed to install Gentoo GNU/Linux on this
classic UNIX server.</p>
<!--more-->
<p>OK, maybe classic is too strong a word, but it is a fairly unique
machine. I've written a few posts about SPARC machines and PA-RISC is
in the same vein - a RISC CPU architecture with machines and OS both
sold by a single vendor. In this case, the vendor is HP, the CPU is
has the PA-RISC architecture, and the OS is (or was) HP-UX.</p>
<p>I don't have any disks of HP-UX around and HP doesn't provide them on
their website. Oracle (!!) provides Solaris freely, but I had no such
luck with HP. Using OpenPA.net to work out which OSs were compatible
with my L3000, I eventually settled on Gentoo GNU/Linux.</p>
<h2>The Guardian Service Processor</h2>
<p>Similarly to most enterprise servers, there is a kind of service
processor that you can connect to the network and use to access the
console, administrate the server, etc, without powering it on.</p>
<p>HP calls it the Guardian Service Processor or GSP for short.</p>
<p>The first task for me was to reset the password. I had to open up the
server and press the GSP reset button. I then connected the GSP to the
network, determined its IP address, and was able to telnet in. This is
what it looks like:</p>
<pre><code>$ telnet gsp.gondolin
Trying 192.168.1.18...
Connected to gsp.gondolin.int.kaashif.co.uk.
Escape character is '^]'.
Service Processor login:
Service Processor password:
Hewlett-Packard Guardian Service Processor
(c) Copyright Hewlett-Packard Company 1999-2001. All Rights Reserved.
System Name: gsp
*************************************************************************
GSP ACCESS IS NOT SECURE
No GSP users are currently configured and remote access is enabled.
Set up a user with a password (see SO command)
OR
Disable all types of remote access (see EL and ER commands)
*************************************************************************
</code></pre>
<p>You can just hit enter twice to login without a user or password,
since those were reset.</p>
<h2>Booting from a CD</h2>
<p>Luckily, my server has a working CD drive, so I could download the
Gentoo minimal installation CD
(<a href="https://wiki.gentoo.org/wiki/Handbook:HPPA/Installation/Media">https://wiki.gentoo.org/wiki/Handbook:HPPA/Installation/Media</a>), pop
it in, and get started.</p>
<p>The first step is to power on the machine. To do this, I connected to
the console through telnet, CTRL-E then CF to activate console write
access, and CTRL-B to access the GSP prompt:</p>
<pre><code>[Read only - use ^Ecf for console write access.]
[bumped user - ]
Leaving Console Mode - you may lose write access.
When Console Mode returns, type ^Ecf to get console write access.
GSP Host Name: gsp
GSP>
</code></pre>
<p>Now you can type <code>he</code> for a help menu. The list of commands is:</p>
<pre><code>==== GSP Help ============================================(Administrator)===
AC : Alert display Configuration MS : Modem Status
AR : Automatic System Restart config. PC : Remote Power Control
CA : Configure asynch/serial ports PG : PaGing parameter setup
CL : Console Log- view console history PS : Power management module Status
CO : COnsole- return to console mode RS : Reset System through RST signal
CSP : Connect to remote Service Proc. SDM : Set Display Mode (hex or text)
DC : Default Configuration SE : SEssion- log into the system
DI : DIsconnect remote or LAN console SL : Show Logs (chassis code buffer)
EL : Enable/disable LAN/WEB access SO : Security options & access control
ER : Enable/disable Remote/modem SS : System Status of proc. modules
EX : Exit GSP and disconnect TC : Reset via Transfer of Control
HE : Display HElp for menu or command TE : TEll- send a msg. to other users
IT : Inactivity Timeout settings VFP : Virtual Front Panel display
LC : LAN configuration WHO : Display connected GSP users
LS : LAN Status XD : Diagnostics and/or Reset of GSP
MR : Modem Reset XU : Upgrade the GSP Firmware
====
(HE for main help, enter command name, or Q to quit)
</code></pre>
<p>The important command is PC, for remote power control. Turn the power
switch on, then execute the PC command to turn it on.</p>
<p>If you're doing this for the first time, you'll need to follow the
instructions in the Gentoo page to install Gentoo. That is out of the
scope of this post.</p>
<p>The interesting parts of the boot process are the hardware detection:</p>
<pre><code>Firmware Version 42.06
Duplex Console IO Dependent Code (IODC) revision 1
------------------------------------------------------------------------------
(c) Copyright 1995-2000, Hewlett-Packard Company, All rights reserved
------------------------------------------------------------------------------
Processor Speed State CoProcessor State Cache Size
Number State Inst Data
--------- -------- --------------------- ----------------- ------------
0 550 MHz Active Functional 512 KB 1 MB
Central Bus Speed (in MHz) : 133
Available Memory : 2097152 KB
Good Memory Required : 25000 KB
Primary boot path: 0/0/1/1.2
Alternate boot path: 0/0/2/0.2
Console path: 0/0/4/1.0
Keyboard path: 0/0/4/0.0
Processor is booting from first available device.
To discontinue, press any key within 10 seconds.
10 seconds expired.
Proceeding...
</code></pre>
<p>Then Linux starts booting. There are a ton of errors, but somehow it
boots up fine and gives me a login prompt.</p>
<h2>Can you actually use it for anything?</h2>
<p>It would be a really bad idea to use this server for anything
real. It's huge, it's heavy, it has a slow CPU. There's not really
anything special or revolutionary about the CPU architecture, as far
as I can tell. I haven't measured it, but it probably uses a few
thousand watts.</p>
<p>There aren't really any binary packages available for Gentoo, so you
have to compile everything, which is a huge time sink. Debian might be
a better choice in this regard.</p>
<p>Lets get onto the really important question:</p>
<h2>Why not OpenBSD?</h2>
<p>There's no support for hppa64! You may wonder, then, how did Linux get
support? HP helped out. They supplied documentation and code,
eventually leading to Debian and Gentoo being ported (and they both
still work on hppa, to this day!).</p>
<p>OpenBSD has support for most workstations, 32 bit and 64 bit running
in 32 bit mode. Server support is a bit lacking, but this is
understandable given the lack of hardware and interest.</p>
<h2>What does the machine code look like?</h2>
<p>I'm glad you asked. After snooping around the binaries, I objdumped
some interesting-looking ones. Here's <code>/bin/sh</code>:</p>
<pre><code>
/bin/sh: file format elf32-hppa-linux
Disassembly of section .init:
000112e4 <.init>:
112e4: 6b c2 3f d9 stw rp,-14(sp)
112e8: 6f c4 00 80 stw,ma r4,40(sp)
112ec: 6b d3 3f c1 stw r19,-20(sp)
112f0: e8 40 14 78 b,l 11d34 <_GLOBAL_OFFSET_TABLE_@@Base-0x19fa0>,rp
112f4: 08 00 02 40 nop
112f8: e8 4b 0b a0 b,l 278d0 <_GLOBAL_OFFSET_TABLE_@@Base-0x4404>,rp
112fc: 08 00 02 40 nop
11300: 4b c2 3f 59 ldw -54(sp),rp
11304: 08 04 02 53 copy r4,r19
11308: e8 40 c0 00 bv r0(rp)
1130c: 4f c4 3f 81 ldw,mb -40(sp),r4
Disassembly of section .text:
00011310 <.text>:
11310: 2b 60 00 00 addil L%0,dp,r1
11314: 48 35 06 50 ldw 328(r1),r21
11318: ea a0 c0 00 bv r0(r21)
1131c: 48 33 06 58 ldw 32c(r1),r19
11320: 2b 60 00 00 addil L%0,dp,r1
</code></pre>
<p>Wow, I don't recognise any of those instructions!</p>
<p>Expect a blog post in the near future (less than a year) explaining
some quirks of PA-RISC. I'm sure there's no shortage of weirdness and
oddities... Maybe there'll even be a cool feature or two not found in
modern processors.</p>
<p>If I ever get my hands on a working copy of HP-UX, expect a post about
that, too.</p>
Using PostgreSQL to search transcriptsurn:https-kaashif-co-uk:-2019-03-31-using-postgresql-to-search-transcripts2019-03-31T00:00:00Z2019-03-31T00:00:00ZKaashif Hymabaccus
<p>Remember my transcript search engine,
<a href="https://transcripts.kaashif.co.uk">https://transcripts.kaashif.co.uk</a>?</p>
<p>I'm not a database expert, but even I realised that spending hours and
hours trying to optimise my homebrew database transcript search engine
was a waste of time. For no reason at all other than to try it out, I
went with ElasticSearch. The astute reader will notice that
ElasticSearch is meant for "big" data. My collection of transcripts
tops out at few dozen megabytes at most - this is most certainly not
big (or even medium-sized) data, really.</p>
<p>So after getting some real-world experience with SQL databases (at a
real company) and taking an in-depth database algorithms course at
university, I decided to convert my web app to use a PostgreSQL
database to store the transcripts, searching with bog-standard SQL
queries.</p>
<p>There were a couple of neat tricks I used to speed things up, too.</p>
<!--more-->
<h2>The ultimate optimisation</h2>
<p>I wrote an article about how I
<a href="https://kaashif.co.uk/2017/06/28/cutting-down-memory-usage-of-a-haskell-web-app/">optimised my transcript parser</a>. Lots
of great tips there, but the greatest optimisation of them all is not
running the code at all. Why do we need any parsing done while the web
app is running, anyway? Of course, we don't - the transcripts are
static and unchanging, we can just pre-parse everything.</p>
<p>What happens now is the parser runs once and parses the transcripts
into a table with these columns:</p>
<pre><code>create table raw_transcripts (
series text,
season_number int,
episode_number int,
episode_title text,
scene_number int,
line_number int,
person text,
present text,
place text,
speech text);
</code></pre>
<p>The old data was nested: a series has episodes, an episode has scenes,
each scene has lines. Scenes have the same people present and take
place in a single place. So of course, there is a lot of redundancy
introduced when we de-nest the transcript structure like this.</p>
<p>The key thing to notice is that the data is so small that none of this
matters!</p>
<p>Instead of doing some parsing for each request, we now just do some
database calls. An upside to this is that the Haskell code only runs
once, offline, so it can be written for maximum readability rather
than speed.</p>
<p>Essentially, the whole of that optimisation article is obsolete.</p>
<h2>Expected workload</h2>
<p>We need to know what the expected workload is so that we can organise
our indices, primary keys, clustering, etc. This has a huge impact on
performance.</p>
<p>In a typical session with the transcript search engine, a user will
search for a fragment of speech, maybe with place, speaker, episode,
series or season specified. Indexing on speech is essentially
useless - a user will probably never want to search for a line of
speech using exact equality. Since this is the most important column,
indexing is basically useless for speeding up searches, we'll always
have to do a full table scan.</p>
<p>While the other columns like season number and episode number seem
mildly indexable, the reality is that a user will basically never use
these. No-one remembers a line by which season it was in, they usually
just remember some fragment of the line.</p>
<p>The most important optimisation here is related to the form the user
expects the results in: the line of speech <em>with some surrounding
context</em>.</p>
<h2>Context is for kings</h2>
<p>Our web app wants to make queries that look like:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">transcripts</span><span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">speech</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="o">%</span><span class="n">somestring</span><span class="o">%</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>As discussed before, this is always going to incur a full table scan,
indices can't help.</p>
<p>Then we want to get the context:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">transcripts</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">line_number</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">line_number</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="n">x</span><span class="o">+</span><span class="mi">2</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>But here is an opportunity to cluster the table. Why not order the
lines on-disk according to their line numbers? Then we can simply look
at the surrounding records, which have already probably been loaded
into memory with the matching record - databases load records into
memory in batches or pages.</p>
<p>To take advantage of this, I associate an ID to each line which is
globally unique and orders each line in chronological (as they appear
in each script) order.</p>
<p>An informed reader might notice that PostgreSQL doesn't really support
clustered indices. This is true, but we can cluster the table on an
index and, as long as the table <em>never changes</em> (this is true in our
case), the clustering will be maintained. Inasmuch as never
interacting with the order counts as "maintaining" it.</p>
<p>In PostgreSQL, this is easy, just add an <code>id</code> column as the <code>PRIMARY
KEY</code>, and you get indexing automatically. We can cluster the table
using <code>CLUSTER transcripts ON id</code> Since we are exclusively going to be
doing very small range queries (maybe for 5 records max), we might
want a hash index or something. But this never caused a performance
issue, so I left it as-is, with the default B-Tree index.</p>
<h2>End result</h2>
<p>I switched out the Haskell web app stuff with a bog-standard Python
web app written with Flask about a hundred lines long. I'd never want
to write a parser in Python and I didn't really want to write a web
app in Haskell, so this situation is much better.</p>
<p>Performance is through the roof compared to the old handrolled
searching code and even the ElasticSearch engine (I'm sure it's
fantastic for big data...).</p>
<p>What an effect a tiny bit of database knowhow can have!</p>
Register windows: a cool feature of SPARCurn:https-kaashif-co-uk:-2018-08-11-register-windows-a-cool-feature-of-sparc2018-08-11T00:00:00Z2018-08-11T00:00:00ZKaashif Hymabaccus
<p>Everyone's studied x86 assembly (just objdump any program on your
PC...) and maybe even some ARM or MIPS in a class somewhere, but there
are a few features that exist in some CPUs that don't exist at all in
any of these designs.</p>
<p>I'm talking about register windows! When you call a function on SPARC,
the new function just magically gets its own registers neatly
separated into input registers, output registers and local
registers. You're allowed to mess up your local registers as much as
you want and the CPU does all of the saving and swapping for you.</p>
<p>No more weird arbitrary calling conventions about r10 and r11 being
caller-saved, rax being return, rqb being Cthulhu-saved, rpqwuqew
being quantum entangled with r554 on Tuesdays...</p>
<!--more-->
<h2>History</h2>
<p>You could just go to the
<a href="https://en.wikipedia.org/wiki/Register_window">Wikipedia article</a>
about these, there is some good info there. The basic rundown is that
the idea of register windows originated with the Berkeley RISC design
back in the first half of the 80s, then they were implemented in a few
architectures of which SPARC is the only (barely) surviving example.</p>
<p>This post isn't supposed to be about history, it's supposed to be
about actual nitty-gritty assembly code, so let's get to it.</p>
<h2>How are the registers laid out?</h2>
<p>Each window consists of 8 input registers (i0 to i7), 8 local
registers (l0 to l7) and 8 output registers (o0 to o7).</p>
<p>There are also 8 global registers (g0 to g7) which are visible at all
times. g0 is actually just 0 all the time and there is some spooky
stuff going on with the others: g1 to g5 might change between caller
and callee, so can't be used to pass parameters. g6 and g7 are
reserved for OS use, so don't use them.</p>
<p>Then there's sp, the stack pointer, which is also global.</p>
<p>The registers are seen by functions like this:</p>
<pre><code>input
local
output input
local
output input
local
output ...
</code></pre>
<p>This may be a bit confusing. By this diagram, I mean that if you are a
function f and you call a function g, if g switches to the next
window, g will "see" different registers. It will see <em>your</em> output
registers as <em>its</em> input registers and it will have its own local and
output registers.</p>
<p>This means a callee can clobber the caller's output registers (e.g. to
return values), but cannot even see the caller's input and local
registers. In fact, no other function can see your local registers if
it's in a different register window.</p>
<p>This is nice because it's not like x86 where there is just a
convention on which registers are for what: this differs between
operating systems and no program really has to follow it. On SPARC
there is an easy and powerful way for functions to have their own
registers.</p>
<h2>An actual program!</h2>
<p>Let's write a program in C, here is the source code:</p>
<div class="highlight"><pre><span></span><span class="kt">int</span><span class="w"> </span><span class="nf">g</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">argv</span><span class="p">[])</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">g</span><span class="p">();</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Compile with debug symbols and objdump it (I use <code>-O1</code> because it gets
rid of a lot of writing to memory that is pointless and unnecessary):</p>
<pre><code>$ cc -g -O1 test.c
$ objdump -S a.out
</code></pre>
<p>There's a lot of output, here are the relevant bits:</p>
<pre><code>0000000000000580 <g>:
int g() {
580: 9c 03 bf 30 add %sp, -208, %sp
return 0;
}
584: 90 10 20 00 clr %o0
588: 81 c3 e0 08 retl
58c: 9c 23 bf 30 sub %sp, -208, %sp
0000000000000590 <main>:
int main(int argc, char *argv[]) {
590: 9d e3 bf 30 save %sp, -208, %sp
return g();
594: 40 08 01 7b call 200b80 <g@plt>
598: 01 00 00 00 nop
}
</code></pre>
<p>Let's analyze this bit by bit, starting with the main function.</p>
<h2>The <code>save</code> instruction</h2>
<p>On SPARC, much like x86, the stack grows downwards. So if we want to
grow the stack to give a new stack frame to our function, we want to
<em>subtract</em> from the stack. At the start of our main function, we want
to grow the stack by 208 bytes, so we want to subtract 208 from %sp.</p>
<p>We also want to move to a new register window, where we will be able
to see our input parameters: argc will be i0 and argv will be i1. And
of course, we'll have our own local and output registers.</p>
<p>This is exactly what this does:</p>
<pre><code> 590: 9d e3 bf 30 save %sp, -208, %sp
</code></pre>
<p>This is called "save" because it saves the previous register window,
making it inaccessible unless we go back to the previous window.</p>
<p>We have a stack frame and register window. Now what?</p>
<h2>Calling a function</h2>
<p>Now we call some memory address where g is. This is similar to x86,
it's really just a jump plus some convenience - you can return.</p>
<p>But where is the return address kept? On x86, it's kept on the
stack. On SPARC (and many other RISC architectures you may be familiar
with), it's stored in a link register.</p>
<p>When you call a function, the return address is written to <em>your</em> o7
register. So when the callee executes a save, it will be in its i7
register. No need to touch memory.</p>
<p>In fact, if you look it up in the SPARC V9 Manual, you'll find that
<code>call addr</code> is literally synonymous with <code>jmpl addr, %o7</code>. <code>jmpl</code>
means "jump and link" which writes the return address to the given
register (in this case o7) then jumps to <code>addr</code>.</p>
<h2>Why is there a nop? Delay slot!</h2>
<p>This is due to something weird on SPARC known as the delay slot. When
you do a branch, the branch doesn't happen right away, the CPU
actually executes the instruction <em>after</em> the jmp or call or whatever,
<em>then</em> branches. This means you can fill it with a useful instruction
or just whack a nop in there if it's too confusing.</p>
<h2>Inside g</h2>
<p>We call g, then g executes this:</p>
<pre><code> 580: 9c 03 bf 30 add %sp, -208, %sp
</code></pre>
<p>This reserves some space on the stack <em>without</em> switching to a new
register window. Notice that a new register window is not necessary
since we do barely anything in g. In particular, we call no other
functions, which means g is known as a "leaf" function.</p>
<p>Then we zero out the return value (we are returning 0):</p>
<pre><code> 584: 90 10 20 00 clr %o0
</code></pre>
<p>Notice that because we are not in another window, we just zero out our
o0 which is the same as our caller's o0.</p>
<p>Now we need to return and we have a choice to make when we look up
"return" in the manual: there are 2 return instructions, <code>ret</code> and
<code>retl</code>. <code>ret</code> is for returning from functions that have gone to a new
register window, it jumps to i7+8. <code>retl</code> is for leaf functions, it
jumps to o7+8 (instructions are 8 bytes long, fixed). We're a leaf, so
we use <code>retl</code>:</p>
<pre><code> 588: 81 c3 e0 08 retl
</code></pre>
<p>Remember the delay slot! Before the branch happens, we get 1
instruction to do something. Let's use it to throw away our stack
frame:</p>
<pre><code> 58c: 9c 23 bf 30 sub %sp, -208, %sp
</code></pre>
<p>Now the return value, 0 is in o0 and main can just leave it there and
do nothing more to return 0.</p>
<h2>Summary</h2>
<p>There are a few mildly interesting things in this post you may not
have seen before:</p>
<ul>
<li><p>Register windows: stopping all of that confusing callee/caller-saved
business</p></li>
<li><p>Leaf/non-leaf functions: you can still just not use register windows
if you don't need them</p></li>
<li><p>Link registers: if you only stick to x86, having a register for the
return address is a bit different</p></li>
<li><p>The delay slot: a quirk of SPARC, originating from the time when
pipelines were simple and this let you save some stalling when a
branch instruction comes along. Not really necessary nowadays
(speculative execution in particular lets the processor just guess
what's coming up). That's a whole 'nother blog post, though.</p></li>
</ul>
<h2>Credits</h2>
<p>All of this information comes in part from my own experimentation and
writing programs, but it all derives from the SPARC V9 Architecture
Manual in the end. Props to Sun for writing some good documentation.</p>
Porting OpenJK to sparc64urn:https-kaashif-co-uk:-2018-01-13-porting-openjk-to-sparc642018-01-13T00:00:00Z2018-01-13T00:00:00ZKaashif Hymabaccus
<p>It's a little known fact that there is actually no way in C or C++ to
do an unaligned access without invoking undefined behaviour. It's
true! Read it yourself
<a href="http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf">here</a>:</p>
<blockquote><p>A pointer to an object type may be converted to a pointer to a
different object type. If the resulting pointer is not correctly
aligned [...] for the referenced type, the behavior is undefined.</p>
<p><cite>C11 (n1570) 6.3.2.3 p7</cite></p></blockquote>
<p>Sadly, the authors of many programs ignore this and rely on it
working. Which it does, on x86, with little performance impact in most
cases. On some architectures, like MIPS and PowerPC, unaligned access
instructions exist but are slow. But on SPARC...unaligned access is
impossible and leads to this:</p>
<pre><code>$ openjk.sparc64
Bus error (core dumped)
</code></pre>
<p>Solving these issues with OpenJK is very difficult, especially
considering Jedi Knight was never meant to run on SPARC (or indeed
OpenBSD, but that's less of an issue).</p>
<!--more-->
<h2>Dealing with custom allocators</h2>
<p>The Jedi Knight engine has a custom allocator, which is a bit of a
pain considering it was written to RELY ON UNDEFINED BEHAVIOUR (read:
uses unaligned reads). Let's examine the first SIGBUS we hit:</p>
<div class="highlight"><pre><span></span><span class="n">ZoneTailFromHeader</span><span class="p">(</span><span class="n">pMemory</span><span class="p">)</span><span class="o">-></span><span class="n">iMagic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ZONE_MAGIC</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>This requires a bit of context. Currently, we are in the Z_Malloc
function, with the signature:</p>
<div class="highlight"><pre><span></span><span class="kt">void</span><span class="w"> </span><span class="o">*</span><span class="n">Z_Malloc</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">iSize</span><span class="p">,</span><span class="w"> </span><span class="n">memtag_t</span><span class="w"> </span><span class="n">eTag</span><span class="p">,</span><span class="w"> </span><span class="n">qboolean</span><span class="w"> </span><span class="n">bZeroit</span><span class="w"> </span><span class="cm">/* = qfalse */</span><span class="p">,</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">iUnusedAlign</span><span class="w"> </span><span class="cm">/* = 4 */</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>This is a replacement for <code>malloc</code>, which takes a size, a tag saying
what "sort" of memory this is, and a boolean saying whether to zero
the memory.</p>
<p><code>pMemory</code> is a pointer to a <code>zoneHeader_t</code>, which is this:</p>
<div class="highlight"><pre><span></span><span class="k">typedef</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">zoneHeader_s</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">iMagic</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">memtag_t</span><span class="w"> </span><span class="n">eTag</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">iSize</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">zoneHeader_s</span><span class="w"> </span><span class="o">*</span><span class="n">pNext</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">zoneHeader_s</span><span class="w"> </span><span class="o">*</span><span class="n">pPrev</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"> </span><span class="n">zoneHeader_t</span><span class="p">;</span><span class="w"></span>
<span class="k">typedef</span><span class="w"> </span><span class="k">struct</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">iMagic</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"> </span><span class="n">zoneTail_t</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>Why is this a problem? It's kind of subtle if you're not used to these
kinds of bugs. We know that <code>pMemory</code> and all its members are aligned
correctly: it was created earlier and initialised without any pointer
casting or anything like that. The bug reveals itself when we look at
<code>ZoneTailFromHeader</code>:</p>
<div class="highlight"><pre><span></span><span class="k">static</span><span class="w"> </span><span class="kr">inline</span><span class="w"> </span><span class="n">zoneTail_t</span><span class="w"> </span><span class="o">*</span><span class="nf">ZoneTailFromHeader</span><span class="p">(</span><span class="n">zoneHeader_t</span><span class="w"> </span><span class="o">*</span><span class="n">pHeader</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">(</span><span class="n">zoneTail_t</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="p">)</span><span class="n">pHeader</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">pHeader</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">pHeader</span><span class="o">-></span><span class="n">iSize</span><span class="w"> </span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>So when we do this:</p>
<div class="highlight"><pre><span></span><span class="n">ZoneTailFromHeader</span><span class="p">(</span><span class="n">pMemory</span><span class="p">)</span><span class="o">-></span><span class="n">iMagic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ZONE_MAGIC</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>What we're really doing is interpreting a pointer (<code>pHeader</code>) that was
just cast to <code>char *</code> and had some values added to it. Interpreting
<code>pHeader</code> as something (e.g. <code>int *</code> or <code>char *</code>) would actually be
fine: <code>malloc</code> gives us memory satisfying the strictest alignment
requirements and we just made <code>pHeader</code> with <code>malloc</code>, so that's OK.</p>
<p>The problem arises when we start adding stuff to <code>pHeader</code>. The
result, which we interpret as a <code>zoneTail_t</code>, might actually not lie
on any particular boundary. In this case, I was getting a SIGBUS when
the engine tries to allocate a block of size 11. Obviously that won't
end up on any boundary, so when we try to access an int on that
misaligned boundary (iMagic is an int):</p>
<div class="highlight"><pre><span></span><span class="n">ZoneTailFromHeader</span><span class="p">(</span><span class="n">pMemory</span><span class="p">)</span><span class="o">-></span><span class="n">iMagic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ZONE_MAGIC</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>We get a SIGBUS! Mystery solved.</p>
<h2>The fix</h2>
<p>The engine code in OpenJK (and the original code from Raven) is split
into multiplayer and singleplayer. The mechanics of the game work
differently in MP and there is no netcode in SP, so this makes some
kind of sense. However, there is a huge downside: bugs which are
noticed and fixed in one codebase might be overlooked in the other.</p>
<p>This isn't exactly what happened here, but it's close. In the SP code,
in the SP version of Z_Malloc, I saw the following line:</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="c1">// Add in tracking info and round to a longword... (ignore longword aligning now we're not using contiguous blocks)</span>
<span class="w"> </span><span class="c1">//</span>
<span class="c1">// int iRealSize = (iSize + sizeof(zoneHeader_t) + sizeof(zoneTail_t) + 3) & 0xfffffffc;</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">iRealSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">iSize</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">zoneHeader_t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">zoneTail_t</span><span class="p">));</span><span class="w"></span>
</pre></div>
<p>This is interesting! What this does is round iRealSize (the size of
the memory plus the zone metadata we add) up to the nearest multiple
of 4. This doesn't quite work as written, since the zone layout is
like this:</p>
<pre><code>+--------+----------------------------------------------+------+
| header | block of malloc'ed memory | tail |
+--------+----------------------------------------------+------+
</code></pre>
<p>The SIGBUS comes when we access the tail. The end of the header is
already aligned to a 4 byte boundary, just by looking at its
members. The problem is the bit in the middle, which pushes the tail
off of a 4 byte boundary.</p>
<p>We can solve this by rounding iSize:</p>
<div class="highlight"><pre><span></span><span class="c1">// Round size of allocation up to multiple of 4 bytes</span>
<span class="kt">int</span><span class="w"> </span><span class="n">iRoundedSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">iSize</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">3</span><span class="p">)</span><span class="w"> </span><span class="o">&</span><span class="w"> </span><span class="mh">0xfffffffc</span><span class="p">;</span><span class="w"></span>
<span class="kt">int</span><span class="w"> </span><span class="n">iRealSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">iRoundedSize</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">zoneHeader_t</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="n">zoneTail_t</span><span class="p">));</span><span class="w"></span>
</pre></div>
<p>Then replace <code>iSize</code> with <code>iRoundedSize</code> in the rest of the function.</p>
<h2>Did it work?</h2>
<p>Yes, kind of. There are a few other similar errors with the memory
allocator. We also have some "static" memory blocks, where the entire
zone layout is written into a struct, like:</p>
<div class="highlight"><pre><span></span><span class="cp">#define DEF_STATIC(_char) {ZONE_MAGIC,TAG_STATIC,2,NULL,NULL},{_char,'\0'},{ZONE_MAGIC}</span>
</pre></div>
<p>This was kind of hard to spot a priori, but after running the game
under gdb, the SIGBUSes were easier to track down.</p>
<p>The fix here is just to add more NULLs so the ZONE_MAGIC isn't
misaligned.</p>
<p>The game does run now. There were a heap of other similar errors
(mostly casting <code>char *</code> to <code>int *</code>), but nothing impossible to fix.</p>
<h2>Isn't this still a bit dodgy?</h2>
<p>Yes. I don't even see why the zone magic thing even exists. Was memory
corruption that big a deal? Maybe it was on the Gamecube or Xbox,
where this code also ran.</p>
Playing with LDoms, OpenBSD and Solarisurn:https-kaashif-co-uk:-2017-12-03-playing-with-ldoms-openbsd-and-solaris2017-12-03T00:00:00Z2017-12-03T00:00:00ZKaashif Hymabaccus
<p>A few weeks ago, I got my hands on a Sun T2000 server. It's got an
UltraSparc T1 CPU, 32 threads, 32 GB of memory, a Sun XVR-300 GPU and
what sounds like a huge jet engine mounted at the front.</p>
<p>It's a great machine (although maybe not as a workstation...), and
there are a few things unique to SPARC that I've really been looking
forward to playing around with. Mostly LDoms (logical domains - Sun's
virtualization technology), OpenBSD on a beefy sparc64 (compared to my
older UltraSparcs anyway) and Solaris (just as a curiosity).</p>
<!--more-->
<h2>Which OS to run on the bare metal?</h2>
<p>I have a few choices here, but it essentially boiled down to OpenBSD
or Solaris. There are a few factors to compare, but I settled on
OpenBSD, for a few reasons.</p>
<ol>
<li><p>Better software support. No software nowadays has Solaris in mind
and I have no clue where to get updates for free. Oracle makes it
impossible to download anything related to Solaris without having a
support contract. OpenBSD has regular updates (for free) and a
ports tree filled with up-to-date software.</p></li>
<li><p>I don't have any clue how to use Solaris and learning it is a waste
of time.</p></li>
<li><p>If there's a problem, I won't know where to ask for help. I can't
contact the devs and there isn't really a big Solaris community
online.</p></li>
</ol>
<p>And OpenBSD supports all the cool stuff I want to do fairly well
anyway, so picking it is a no-brainer.</p>
<p>Someone not acquainted with the state of affairs when using old Sun
hardware might ask why I'm not using some Linux distro. Surely there
must be one out there that does what I need, and isn't Linux more
popular and thus more well-supported than OpenBSD? Well there is a
distro that does LDoms...Oracle Linux.</p>
<p>I think I'll pass on that one. And I can't find anyone running Debian
as a primary domain on SPARC, so I guess you can't do it.</p>
<h2>Installing the primary OS</h2>
<p>This went so painlessly I don't even feel the need to write anything
about it. Just put the disk in and everything works as expected. I
guess you could netboot, it's not too difficult, but using an
install DVD is way easier.</p>
<h2>Setting up LDoms</h2>
<p>This is actually really easy too. OpenBSD supports LDoms remarkably
well, due to the efforts of Mark Kettenis (and others, probably). Take
a look at
<a href="https://www.tedunangst.com/flak/post/OpenBSD-on-a-Sun-T5120">this guide</a>
to see how I did it. With the T2000, there are a few differences, but
these are confined to the ILOM/ALOM.</p>
<p>Where Ted types <code>start /SYS</code>, I type <code>poweron</code>. He types <code>start
/SP/console</code>, I type <code>console</code>.</p>
<p>There is little real difference between the ALOM of the T2000 and the
ILOM of the T5120 as far as I can tell. <code>ldomctl</code> from OpenBSD can't
see the I/ALOM anyway, since that lives below OpenBoot.</p>
<p>I ended up with the following config:</p>
<pre><code>domain primary {
vcpu 16
memory 16G
}
domain openbsd {
vcpu 8
memory 8G
vdisk "/home/kaashif/vm/install62.iso"
vdisk "/home/kaashif/vm/openbsd.img"
vnet
}
domain solaris {
vcpu 8
memory 8G
vdisk "/home/kaashif/vm/install-solaris-10.iso"
vdisk "/home/kaashif/vm/solaris.img"
vnet
}
</code></pre>
<p>Where <code>solaris.img</code> and <code>openbsd.img</code> are files created with
<code>truncate</code>:</p>
<pre><code>$ gtruncate -s 10G openbsd.img
</code></pre>
<p>This creates a sparse file rather than actually filling some file with
zeroes.</p>
<h2>Networking</h2>
<p>For this, I just bridged vnet0, vnet1 and em0 (the interface connected
to my LAN):</p>
<pre><code># ifconfig bridge0 up add vnet0 add vnet1 add em0
</code></pre>
<p>This lets the LDoms speak to my LAN as if they were real. I haven't
got DHCP to work, but that hasn't been an issue.</p>
<h2>What to do with the LDoms?</h2>
<p>As of now, I haven't actually got the Solaris LDom to boot. I boot
from the install media, install, but then it refuses to boot from the
disk. Since this was just a toy, I might just abandon it.</p>
<p>In future, I might give the Debian SPARC port a spin, just to see if
it's not a steaming mess. Same goes for NetBSD. Expect a post or two
in the future about these.</p>
<h2>Dmesg!</h2>
<p>Here is the dmesg from before I sliced it up with LDoms:</p>
<pre><code>console is /virtual-devices@100/console@1
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2017 OpenBSD. All rights reserved. https://www.OpenBSD.org
OpenBSD 6.2-current (GENERIC.MP) #307: Wed Oct 11 15:17:26 MDT 2017
deraadt@sparc64.openbsd.org:/usr/src/sys/arch/sparc64/compile/GENERIC.MP
real mem = 34225520640 (32640MB)
avail mem = 33610276864 (32053MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root: SPARC Enterprise T2000
cpu0 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu1 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu2 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu3 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu4 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu5 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu6 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu7 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu8 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu9 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu10 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu11 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu12 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu13 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu14 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu15 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu16 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu17 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu18 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu19 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu20 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu21 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu22 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu23 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu24 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu25 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu26 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu27 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu28 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu29 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu30 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu31 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
vbus0 at mainbus0
"flashprom" at vbus0 not configured
cbus0 at vbus0
vldc0 at cbus0
vldcp0 at vldc0 chan 0x0: ivec 0x0, 0x1 channel "hvctl"
"ldom-primary" at vldc0 chan 0x1 not configured
"fmactl" at vldc0 chan 0x3 not configured
vldc1 at cbus0
"ldmfma" at vldc1 chan 0x4 not configured
vldc2 at cbus0
vldcp1 at vldc2 chan 0x14: ivec 0x28, 0x29 channel "spds"
"system-management" at vldc2 chan 0xd not configured
vcons0 at vbus0: ivec 0x111, console
vrtc0 at vbus0
"fma" at vbus0 not configured
"sunvts" at vbus0 not configured
"sunmc" at vbus0 not configured
"explorer" at vbus0 not configured
"led" at vbus0 not configured
"flashupdate" at vbus0 not configured
"ncp" at vbus0 not configured
vpci0 at mainbus0: bus 2 to 7, dvma map 80000000-ffffffff
pci0 at vpci0
ppb0 at pci0 dev 0 function 0 "PLX PEX 8532" rev 0xbc
pci1 at ppb0 bus 3
ppb1 at pci1 dev 1 function 0 "PLX PEX 8532" rev 0xbc
pci2 at ppb1 bus 4
em0 at pci2 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x795, address 00:14:4f:e1:c8:82
em1 at pci2 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x796, address 00:14:4f:e1:c8:83
ppb2 at pci1 dev 2 function 0 "PLX PEX 8532" rev 0xbc
pci3 at ppb2 bus 5
ppb3 at pci1 dev 8 function 0 "PLX PEX 8532" rev 0xbc: msi
pci4 at ppb3 bus 6
ppb4 at pci1 dev 9 function 0 "PLX PEX 8532" rev 0xbc
pci5 at ppb4 bus 7
mpi0 at pci5 dev 0 function 0 "Symbios Logic SAS1064E" rev 0x02: msi
mpi0: UNUSED, firmware 1.9.0.0
scsibus1 at mpi0: 63 targets
sd0 at scsibus1 targ 0 lun 0: <HITACHI, H101414SCSUN146G, SA25> SCSI3 0/direct fixed naa.5000cca00098ddfc
sd0: 140009MB, 512 bytes/sector, 286739329 sectors
vpci1 at mainbus0: bus 2 to 9, dvma map 80000000-ffffffff
pci6 at vpci1
ppb5 at pci6 dev 0 function 0 "PLX PEX 8532" rev 0xbc
pci7 at ppb5 bus 3
ppb6 at pci7 dev 1 function 0 "PLX PEX 8532" rev 0xbc
pci8 at ppb6 bus 4
ppb7 at pci8 dev 0 function 0 "Intel 41210 PCIE-PCIX" rev 0x09
pci9 at ppb7 bus 5
ebus0 at pci9 dev 2 function 0 "Acer Labs M1533 ISA" rev 0x00
com0 at ebus0 addr 3f8-3ff ivec 0x2: ns16550a, 16 byte fifo
ohci0 at pci9 dev 5 function 0 "Acer Labs M5237 USB" rev 0x03: ivec 0x7c1, version 1.0, legacy support
ohci1 at pci9 dev 6 function 0 "Acer Labs M5237 USB" rev 0x03: ivec 0x7c3, version 1.0, legacy support
pciide0 at pci9 dev 8 function 0 "Acer Labs M5229 UDMA IDE" rev 0xc4: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide0: using ivec 0x7c4 for native-PCI interrupt
atapiscsi0 at pciide0 channel 0 drive 0
scsibus2 at atapiscsi0: 2 targets
cd0 at scsibus2 targ 0 lun 0: <TEAC, DW-224SL-R, 1.0B> ATAPI 5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Acer Labs OHCI root hub" rev 1.00/1.00 addr 1
usb1 at ohci1: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Acer Labs OHCI root hub" rev 1.00/1.00 addr 1
ppb8 at pci8 dev 0 function 2 "Intel 41210 PCIE-PCIX" rev 0x09
pci10 at ppb8 bus 6
ppb9 at pci7 dev 2 function 0 "PLX PEX 8532" rev 0xbc
pci11 at ppb9 bus 7
em2 at pci11 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x7d6, address 00:14:4f:e1:c8:84
em3 at pci11 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x7d7, address 00:14:4f:e1:c8:85
ppb10 at pci7 dev 8 function 0 "PLX PEX 8532" rev 0xbc: msi
pci12 at ppb10 bus 8
radeondrm0 at pci12 dev 0 function 0 "ATI FireGL V3100" rev 0x80
drm0 at radeondrm0
radeondrm0: ivec 0x7d4
ppb11 at pci7 dev 9 function 0 "PLX PEX 8532" rev 0xbc: msi
pci13 at ppb11 bus 9
umass0 at uhub0 port 2 configuration 1 interface 0 "USB2.0 Flash Disk" rev 2.00/0.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus3 at umass0: 2 targets, initiator 0
sd1 at scsibus3 targ 1 lun 0: <USB2.0, Flash Disk, 2.60> SCSI2 0/direct removable serial.1221323400000000847F
sd1: 998MB, 512 bytes/sector, 2043904 sectors
uhub2 at uhub1 port 1 configuration 1 interface 0 "Atmel UHB124 hub" rev 1.10/3.00 addr 2
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
bootpath: /pci@780,0/pci@0,0/pci@9,0/scsi@0,0/disk@0,0
root on sd0a (8b6ac481aa65550f.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
WARNING: clock lost 6494 days -- CHECK AND RESET THE DATE!
BIOS signature incorrect 0 0
error: [drm:pid0:r100_cp_init_microcode] *ERROR* radeon_cp: Failed to load firmware "radeon-r300_cp"
error: [drm:pid0:r100_cp_init] *ERROR* Failed to load firmware!
drm:pid0:r300_startup *ERROR* failed initializing CP (-2).
drm:pid0:r300_init *ERROR* Disabling GPU acceleration
drm:pid0:radeon_bo_unpin *WARNING* 0x40027322390 unpin not necessary
radeondrm0: 1024x768, 8bpp
wsdisplay0 at radeondrm0 mux 1
wsdisplay0: screen 0 added (std, sun emulation)
</code></pre>
Using vmm(4) to target old OpenBSD releasesurn:https-kaashif-co-uk:-2017-09-10-using-vmm-4-to-target-old-openbsd-releases2017-09-10T00:00:00Z2017-09-10T00:00:00ZKaashif Hymabaccus
<p>This server (the very one you are reading this post on), at the time
of writing this post, runs OpenBSD 6.1-stable. It's fully patched and
updated and everything, so it's a perfectly fine OS to run. But the
VPS has limited memory and disk space, and the CPU isn't very fast, so
compiling large projects on it, <em>especially</em> Haskell ones, is
impractical.</p>
<p>This post describes a way to build fully-functional, dynamically
linked (so you get all those security updates, super important for a
public-facing web service), native Haskell binaries with
cabal-install, Stack and...OpenBSD's (kind of) new native hypervisor,
vmm.</p>
<!--more-->
<h2>Why can't I just do this on my local machine?</h2>
<p>Well, I don't have any machines running 6.1-release or 6.1-stable, so
that's right out. The reason I don't keep any around is that I need
them all on -current to test ports and random patches I see on
tech@. There have been a lot of major version bumps and not to mention
the switch to clang and probably some other security features that all
conspire to make 6.2-beta very binary incompatible with 6.1-stable.</p>
<h2>Act 1: The setup</h2>
<p>Very simple actually. I want to install 6.1 on a virtual machine with
4G of memory, 15G of disk (overkill, but disks are huge nowadays) and
a network interface.</p>
<p>First, start up vmd:</p>
<pre><code># rcctl enable vmd
# rcctl start vmd
vmd(ok)
</code></pre>
<p><code>cd</code> somewhere with more than 15G of free space (and get more than 4G
of memory).</p>
<pre><code># vmctl create disk.img -s 15G
</code></pre>
<p>Get a ramdisk to boot from. We want the amd64 6.1 one:</p>
<pre><code>$ ftp http://ftp.fr.openbsd.org/pub/OpenBSD/6.1/amd64/bsd.rd
</code></pre>
<p>Now spin up a VM (here we name it "vm61"):</p>
<pre><code># vmctl start vm61 -b bsd.rd -m 4G -d disk.img -i 1
</code></pre>
<h2>Intermission: Networking</h2>
<p>You do need some networking if you want to get anything useful
done. If you read the vmctl man page, you'll notice that vmctl's <code>-i</code>
option creates a network interface inside the VM which uses the vio(4)
driver and a tap(4) interface in your host machine. Since you want the
VM to access the real world, you probably want to bridge the tap and
your real interface.</p>
<p>My real interface is re0, my tap interface is tap0. I created a bridge
interface bridge0 as follows:</p>
<pre><code># cat > /etc/hostname.bridge0 <<EOF
add re0
add tap0
up
# sh /etc/netstart bridge0
</code></pre>
<p>Now the network is bridged and you can proceed as normal.</p>
<h2>Act 2: Building the binaries</h2>
<p>To get a console on the VM, run:</p>
<pre><code># vmctl console vm61
</code></pre>
<p>From here, you install as normal, using the vio interface to setup the
network and proceed with the install as usual (look up how to do this
elsewhere if you don't already know).</p>
<p>You might recognise the console program as cu. Accordingly, look at
the cu(1) man page for instructions on escape sequences to send files,
exit the console and so on.</p>
<p>Note: since this is essentially a throwaway VM I'm just going to use
to build Haskell binaries, I overrode the default partition layout and
just made one huge 14G root partition with wxallowed on and a 1G swap
partition. I don't know how much space exactly cabal, stack and so on
will take up (it's a lot though) and I know they need wxallowed. A
more elegant solution is possible where you allocate the bare minimum
space (15G is way too much), but this works well enough.</p>
<p>After installing, you have to boot from the disk. We need to kill and
start the VM again without the bsd.rd argument:</p>
<pre><code># vmctl stop vm61
# vmctl start vm61 -m 4G -d disk.img -i 1
</code></pre>
<p>You may also notice that the tap0 interface was destroyed when the VM
halted. So add it back to the bridge:</p>
<pre><code># ifconfig bridge0 add tap0
</code></pre>
<p>And now re-enter the console (or SSH in).</p>
<p>Here's what I did next (after adding <code>~/.cabal/bin</code> to PATH):</p>
<pre><code>$ doas pkg_add ghc cabal-install git
$ cabal install stack
< heaps of output >
$ stack setup
$ git clone git://github.com/kaashif/stargate-search.git
$ cd stargate-search
$ stack install
</code></pre>
<p>Surprisingly, the binaries installed to <code>~/.local/bin</code> Just Worked
(tm) on this server.</p>
<p>Perhaps that's not so surprising, since they're both OpenBSD 6.1 amd64
machines and all computers are the same nowadays.</p>
Reviving a Sun Ultra 5 workstationurn:https-kaashif-co-uk:-2017-08-10-reviving-a-sun-ultra-5-workstation2017-08-10T00:00:00Z2017-08-10T00:00:00ZKaashif Hymabaccus
<p>I recently got an old Sun Ultra 5 working. It wasn't too difficult,
but I needed to dig up a few old serial cables...</p>
<p>It already had SunOS 5.8 installed, but I put OpenBSD 6.1 on it, since
I need a modern OS to actually do anything with it.</p>
<!--more-->
<h2>The first boot</h2>
<p>The first step was obviously to just turn it on and see what
happened. This went well, it turned on, the CD drive and floppy drives
made some noises, then it stayed on. Of course, I didn't have access
to a Sun keyboard anymore, so this is basically all I could do at this
point.</p>
<p>I decided to connect a monitor to it and see what it said.</p>
<pre><code>No keyboard detected. Redirecting output to ttya
</code></pre>
<p>Then the screen went blank.</p>
<p>Of course, how could I forget! Sun workstations aren't piece of crap
i386 machines, they're right and proper workstations, designed to work
properly over serial ports.</p>
<p>I consulted the
<a href="https://www.openbsd.org/faq/faq7.html#SerCon">OpenBSD faq page</a> for
serial consoles. There was a bit about sparc64 machines:</p>
<pre><code>These machines are designed to be completely maintainable with a
serial console. Simply remove the keyboard from the machine, and
the system will run serial.
</code></pre>
<p>I don't have a keyboard, so this is perfect. The serial port on the
back of a Sun Ultra 5 is DB25. There is a DB9 port, but it's ttyb, so
isn't used automatically. I dug up a DB25 to DB9 adapter and a USB to
DB9 cable, connected it and hey presto, it worked!</p>
<p>I was able to boot into SunOS! I was greeted with a curses-type setup
screen, I was asked about the network (it failed to connect) and the
hostname. Then I booted into SunOS 5.8!</p>
<p>This was great, but since I had no network, I had to get a modern OS
on it, one I knew how to use.</p>
<h2>Getting OpenBSD install media</h2>
<p>There are a couple of options here:</p>
<ul>
<li>CD</li>
<li>Floppy</li>
<li>Network</li>
</ul>
<p>I tried to find a CD-R to no avail. I went to a nearby OpenBSD mirror
to download the
<a href="http://mirror.bytemark.co.uk/OpenBSD/6.1/sparc64/">install61.iso</a> for
sparc64, but since I could only find already-written CD-Rs (useless)
and CD-RW, I decided to give it a shot with a CD-RW. It didn't work:</p>
<pre><code>ok boot cdrom
Resetting ...
Sun Ultra 5/10 UPA/PCI (UltraSPARC-IIi 270MHz), No Keyboard
OpenBoot 3.11, 512 MB memory installed, Serial #1653024.
Ethernet address 8:0:20:19:39:20, Host ID: 80193920.
Rebooting with command: boot cdrom
Boot device: /p...@1f,0/p...@1,1/i...@3/cd...@2,0:f File and args:
Can't read disk label.
Can't open disk label package
Evaluating: boot cdrom
Can't open boot device
</code></pre>
<p>As I suspected, I couldn't get the drive to read a CD-RW.</p>
<p>Next up was floppy. I got the floppy61.fs from there and tried it out:</p>
<pre><code>ok boot floppy bsd
Resetting ...
Sun Ultra 5/10 UPA/PCI (UltraSPARC-IIi 270MHz), No Keyboard
OpenBoot 3.11, 512 MB memory installed, Serial #1653024.
Ethernet address 8:0:20:19:39:20, Host ID: 80193920.
Rebooting with command: boot floppy bsd
Boot device: /p...@1f,0/p...@1,1/e...@1/fdthree File and args: bsd
Bad magic number in disk label
Can't open disk label package
Evaluating: boot floppy bsd
</code></pre>
<p>So what option did I have left? miniroot61.fs, of course.</p>
<p>On my laptop:</p>
<pre><code>$ ftp http://mirror.bytemark.co.uk/OpenBSD/6.1/sparc64/miniroot61.fs
$ uuencode -o miniroot61.fs.uue miniroot61.fs miniroot61.fs
</code></pre>
<p>On the Ultra (using cu as a serial terminal):</p>
<pre><code># uudecode
</code></pre>
<p>Then I hit <code>~></code> which is the cu command to send a file. I choose
<code>miniroot61.fs.uue</code>, which is then uudecoded and placed into the file
<code>miniroot61.fs</code> where I ran uudecode.</p>
<p>Now I write it to the disk. Just in case it doesn't work, I decided to
write it to swap. This way, if it fails to boot, I haven't hosed my
system.</p>
<pre><code># dd if=miniroot61.fs of=/dev/rdsk/c0t0d0s1
</code></pre>
<p>Then enter the ok prompt:</p>
<pre><code># init 0
...some output...
ok boot disk:b
Resetting ...
Sun Ultra 5/10 UPA/PCI (UltraSPARC-IIi 270MHz), No Keyboard
OpenBoot 3.11, 512 MB memory installed, Serial #1653024.
Ethernet address 8:0:20:19:39:20, Host ID: 80193920.
Rebooting with command: boot disk:b bsd
Boot device: /pci@1f,0/pci@1,1/ide@3/disk@0,0:b File and args: bsd
OpenBSD IEEE 1275 Bootblock 1.4
..>> OpenBSD BOOT 1.9
open /pci@1f,0/pci@1,1/ide@3/disk@0,0:b/etc/random.seed: No such file or directory
Booting /pci@1f,0/pci@1,1/ide@3/disk@0,0:b/bsd
4045496@0x1000000+1352@0x13dbab8+3251904@0x1800000+942400@0x1b19ec0
symbols @ 0xfff62300 120 start=0x1000000
console is /pci@1f,0/pci@1,1/ebus@1/se@14,400000:a
</code></pre>
<p>From here I saw the usual OpenBSD installer output. The network was
detected without a hitch, the installer downloaded the sets and
everything worked perfectly.</p>
<p>It's almost a bit disappointing how well it works, I wanted to have to
solve some problems.</p>
<p>Anyway, now I have a real big-endian system to test my software on! Of
course I could have picked up an old macppc powerbook, but that would
be even less fun than this!</p>
<p>Here is the <code>sysctl hw</code>:</p>
<pre><code>hw.machine=sparc64
hw.model=SUNW,UltraSPARC-IIi (rev 1.3) @ 269.804 MHz
hw.ncpu=1
hw.byteorder=4321
hw.pagesize=8192
hw.disknames=wd0:0d5853015d2b2605,cd0:
hw.diskcount=2
hw.cpuspeed=269
hw.vendor=Sun
hw.product=SUNW,Ultra-5_10
hw.physmem=536870912
hw.usermem=536854528
hw.ncpufound=1
hw.allowpowerdown=1
</code></pre>
<p>And <a href="/static/dmesg.sun-ultra-5">here</a> is the dmesg.</p>
<p>I hope I can get my hands on a 440 MHz CPU at some point to give it an
upgrade, but it is already fast enough for daily use.</p>
Sorting a ton of mailurn:https-kaashif-co-uk:-2017-08-04-sorting-a-ton-of-mail2017-08-04T00:00:00Z2017-08-04T00:00:00ZKaashif Hymabaccus
<p>Migrating mail servers is a tricky business, especially when one
server doesn't have IMAP set up. The easiest way is to download all
the mail and reupload it to the new mail server.</p>
<p>This seems simple enough, but I ran into problems. After all, I was
going from an IMAP server somewhere to a maildir (no IMAP sync tool
supports mbox for some reason) to an mbox through procmail to a
directory of mboxes. Not trivial.</p>
<!--more-->
<h2>The Problem</h2>
<p>I am a user of Zoho's mail service. It serves me well but there are
usage limits. I set up my own mail server so I wouldn't have to deal
with there limits. But how do I get my mail from an IMAP server to a
non-IMAP server.</p>
<p>I'm obviously not going to setup IMAP, another public service which
increases my attack surface, just to migrate mail.</p>
<h2>The Solution Part 1: Downloading the mail</h2>
<p>This was supposed to be simple. I installed <code>isync</code> and wrote an
<code>.mbsyncrc</code> which would fetch mail and deliver to a maildir. But there
are usage limits, and I ran into them while fetching my 100k messages.</p>
<pre><code>IMAP command 'AUTHENTICATE PLAIN <authdata>' returned an error: NO [ALERT] Your account is currently not accessible via IMAP due to excessive usage. Kindly try after some time.
*** IMAP ALERT *** Your account is currently not accessible via IMAP due to excessive usage. Kindly try after some time.
</code></pre>
<p>Argh! So annoying. I noticed that I could try again after a few
minutes. So what if I slowed down <code>mbsync</code> to the point where it takes
longer to hit the usage limit than the time it takes for the limit to
reset. This turned out to be simple, I just added this line to my
<code>.mbsyncrc</code> account:</p>
<pre><code>PipelineDepth 1
</code></pre>
<p>This entirely disables pipelining, i.e. only one IMAP command can
happen concurrently whereas before, the limit was infinite.</p>
<h2>Converting to mbox</h2>
<p>So I have all my mail in <code>~/.mail</code>. But now what? My mailserver deals
in mboxes, not maildir.</p>
<p>I came across a Python script that converts a maildir to a mailbox:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/python</span>
<span class="c1"># -*- coding: utf-8 -*-</span>
<span class="kn">import</span> <span class="nn">mailbox</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">email</span>
<span class="n">mdir</span> <span class="o">=</span> <span class="n">mailbox</span><span class="o">.</span><span class="n">Maildir</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span> <span class="p">[</span><span class="o">-</span><span class="mi">2</span><span class="p">],</span> <span class="n">email</span><span class="o">.</span><span class="n">message_from_file</span><span class="p">)</span>
<span class="n">outfile</span> <span class="o">=</span> <span class="n">file</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="s1">'w'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">mdir_msg</span> <span class="ow">in</span> <span class="n">mdir</span><span class="p">:</span>
<span class="c1"># parse the message:</span>
<span class="n">msg</span> <span class="o">=</span> <span class="n">email</span><span class="o">.</span><span class="n">message_from_string</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">mdir_msg</span><span class="p">))</span>
<span class="n">outfile</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">msg</span><span class="p">))</span>
<span class="n">outfile</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
<span class="n">outfile</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
<p>It's so simple and speaks to the simplicity of the mbox format. So I
ran it:</p>
<div class="highlight"><pre><span></span>$ <span class="nb">cd</span> mail
$ <span class="k">for</span> d <span class="k">in</span> *<span class="p">;</span> <span class="k">do</span> python2.7 mailconv.py <span class="s2">"</span><span class="nv">$d</span><span class="s2">"</span> <span class="s2">"</span><span class="si">${</span><span class="nv">d</span><span class="si">}</span><span class="s2">.mbox"</span><span class="p">;</span> <span class="k">done</span>
</pre></div>
<p>It eventually completed and I was left with a mess of disjointed
mailboxes.</p>
<h2>Sorting</h2>
<p>Now I had to sort them. But of course, mails get sent one at a time,
so if my final intention is to pipe all of them through procmail to
apply my new filters, I had to smush them all together first.</p>
<p>Funnily enough, I came across a Python script that sorts mbox:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python2.7</span>
<span class="kn">from</span> <span class="nn">email.utils</span> <span class="kn">import</span> <span class="n">parsedate</span>
<span class="kn">import</span> <span class="nn">mailbox</span><span class="o">,</span> <span class="nn">sys</span>
<span class="k">def</span> <span class="nf">extract_date</span><span class="p">(</span><span class="n">email</span><span class="p">):</span>
<span class="n">date</span> <span class="o">=</span> <span class="n">email</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Date'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">parsedate</span><span class="p">(</span><span class="n">date</span><span class="p">)</span>
<span class="n">the_mailbox</span> <span class="o">=</span> <span class="n">mailbox</span><span class="o">.</span><span class="n">mbox</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">sorted_mails</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">the_mailbox</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">extract_date</span><span class="p">)</span>
<span class="n">the_mailbox</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="nb">enumerate</span><span class="p">(</span><span class="n">sorted_mails</span><span class="p">))</span>
<span class="n">the_mailbox</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
</pre></div>
<p>Using this script:</p>
<div class="highlight"><pre><span></span>$ cat *.mbox > all.mbox
$ python2.7 sort.py all.mbox
</pre></div>
<p>And there we have it, a sorted mbox! That script uses a <em>lot</em> of
memory and CPU, Python isn't the best language for this.</p>
<h2>Getting it to the mail server</h2>
<p>This wasn't too difficult. First I sent the blob:</p>
<div class="highlight"><pre><span></span>$ gzip all.mbox
$ scp all.mbox.gz mail.kaashif.co.uk:~/
</pre></div>
<p>Then, on the server, I used a nifty program called <code>formail</code> which
comes with procmail and applies a command to each mail in an mbox:</p>
<div class="highlight"><pre><span></span>$ gunzip all.mbox.gz
$ formail -ds procmail < all.mbox
</pre></div>
<p>And the mail was sorted as I specified and it's all there!</p>
Moving to my own email serverurn:https-kaashif-co-uk:-2017-08-03-moving-to-my-own-email-server2017-08-03T00:00:00Z2017-08-03T00:00:00ZKaashif Hymabaccus
<p>There I was, a loyal user of <a href="http://mail.zoho.com">http://mail.zoho.com</a>, when I decided to
download all of my emails. For archival purposes, you know.</p>
<p>So I fired up mbsync, set everything up, let 'er rip, but after only
about 10,000 emails downloaded, I got this error:</p>
<pre><code>IMAP command 'AUTHENTICATE PLAIN <authdata>' returned an error: NO [ALERT] Your account is currently not accessible via IMAP due to excessive usage. Kindly try after some time.
*** IMAP ALERT *** Your account is currently not accessible via IMAP due to excessive usage. Kindly try after some time.
</code></pre>
<p>What sort of...anyway, this was unacceptable, so I decided to set up
my web server as a mail server.</p>
<!--more-->
<h2>Prerequisites</h2>
<p>Everything you need to run a mail server is already installed on
OpenBSD. Of course, if you want to use DKIM (and you really do, or
your mail will be sent to spam or just not received by anyone), you
need to install <code>dkimproxy</code>.</p>
<div class="highlight"><pre><span></span>$ doas pkg_add dkimproxy
</pre></div>
<p>That's basically it.</p>
<h2>Setting up OpenSMTPD</h2>
<p>OpenSMTPD has a pf-inspired config file. That is to say, it's very
easy to grok.</p>
<h3>Step 1: Set up an SSL certificate</h3>
<p>This is very easy with <code>acme-client</code>, which comes with OpenBSD.</p>
<p>Your <code>/etc/acme-client.conf</code> should already have the letsencrypt
authorities in it, so you just need to add your domain to the bottom.</p>
<p>It's so easy, look at <code>man acme-client.conf</code>. For reference, here is
my domain entry:</p>
<pre><code>domain earendil.kaashif.co.uk {
alternative names { kaashif.co.uk git.kaashif.co.uk
www.kaashif.co.uk mail.kaashif.co.uk }
domain key "/etc/ssl/private/earendil.kaashif.co.uk.key"
domain certificate "/etc/ssl/earendil.kaashif.co.uk.crt"
domain full chain certificate "/etc/ssl/earendil.kaashif.co.uk.fullchain.pem"
sign with letsencrypt
}
</code></pre>
<p>Then you run <code>acme-client -vFAD earendil.kaashif.co.uk</code> as the man page
suggests and everything will Just Work, with your new certificates
ending up where you specified.</p>
<p>Put that command in <code>/etc/monthly.local</code> and it will run every month,
keeping everything valid and non-expired.</p>
<h3>Step 2: Set up your virtual users</h3>
<p>My mail server will be single user, so I just slapped this into
<code>/etc/mail/virtuals</code>:</p>
<pre><code>kaashif@mail.kaashif.co.uk kaashif
</code></pre>
<p>I don't even think you really need it, but in the future you may want
to add a webmaster email, or mailing lists for specific projects, etc,
so its good to have it.</p>
<p>Although you didn't add anything to <code>/etc/mail/aliases</code>, you should
still run <code>newaliases</code> just to generate the <code>aliases.db</code> that
OpenSMTPD uses.</p>
<h3>Step 3: /etc/mail/smtpd.conf</h3>
<p>This is the big one, where the magic happens.</p>
<p>Actually, this file is so simple, I can just dump it here and explain
it:</p>
<pre><code>pki mail.kaashif.co.uk certificate "/etc/ssl/earendil.kaashif.co.uk.crt"
pki mail.kaashif.co.uk key "/etc/ssl/private/earendil.kaashif.co.uk.key"
</code></pre>
<p>These are the letsencrypt certs you made earlier that smtpd will use.</p>
<pre><code>table aliases file:/etc/mail/aliases
table virtuals file:/etc/mail/virtuals
</code></pre>
<p>This declares the tables so you can use them later on in the file as
<code><aliases></code> and <code><virtuals></code>.</p>
<pre><code>listen on all
listen on all port 10028 tag DKIM
</code></pre>
<p>This requires some explanation. We listen on all interfaces on the
usual smtp ports for mail (this is where mailservers will send us
mail). But on port 10028, we listen for mail tagged DKIM. This is
where <code>dkimproxy</code> comes in.</p>
<p>When we send mail from this server, we want it signed with DKIM. So
the <code>dkimproxy_out</code> daemon listens on port 10027, we send it mail and
it sends it back (signed) on port 10028 and the session has a DKIM
tag.</p>
<pre><code>accept from any for domain "mail.kaashif.co.uk" virtual <virtuals> deliver to mda "/usr/local/bin/procmail"
</code></pre>
<p>Procmail is a well-known mail filtering agent. Explaining how to
configure it is out of the scope of this post, but basically you just
make a <code>~/.procmailrc</code> in the users' home and put some rules in
there. Where the mail ends up (mailbox, maildir, somewhere else) is
entirely up to Procmail, smtpd forgets about it after that.</p>
<pre><code>accept from local for local alias <aliases> deliver to mbox
</code></pre>
<p>We don't want to mix local and non-local mail, so all that local-only
daily output and insecurity output will end up in our normal mbox in
<code>/var/mail</code>. But of course, there's nothing stopping you from
configuring Procmail to put some non-local mail in <code>/var/mail</code> too.</p>
<pre><code>accept tagged DKIM for any relay
accept from local for any relay via smtp://127.0.0.1:10027
</code></pre>
<p>The only sessions that will ever be tagged DKIM are ones made by
<code>dkimproxy</code>. So it is safe to just relay them all. Any mail sent from
here is sent to <code>dkimproxy</code> on port 10027, to be signed and returned
later.</p>
<h2>Setting up DKIM and SPF</h2>
<h3>Step 1: Generating a key</h3>
<p>DKIM works by signing messages with a private key stored locally. Then
untrusting mail servers get the public key from a DNS TXT record and
check the signature. We don't have any keys yet.</p>
<div class="highlight"><pre><span></span>$ mkdir -p /etc/mail/dkim
$ <span class="nb">cd</span> /etc/mail/dkim
$ openssl genrsa -out private.key <span class="m">1024</span>
$ openssl rsa -in private.key -pubout -out public.key
</pre></div>
<p>Why only 1024 bits? Because that's all that fits in a single TXT
record, it's a bit incovenient otherwise.</p>
<p>Now, make those readable by dkimproxy and no-one else:</p>
<div class="highlight"><pre><span></span>$ chmod <span class="m">0600</span> *.key
$ chown _dkimproxy:_dkimproxy *.key
</pre></div>
<p>Now edit <code>/etc/dkimproxy_out.conf</code>'s <code>keyfile</code> line to point to the
private key you just made:</p>
<pre><code>keyfile /etc/mail/dkim/private.key
</code></pre>
<h3>Step 2: Putting everything into DNS</h3>
<p>This is important since people will have no idea how to verify your
sigs if they don't know your public key.</p>
<p>Make a record on <code>selector1._domainkey.your.domain.tld</code>, type TXT,
with the following content:</p>
<pre><code>v=DKIM1; k=rsa; p=<your public key>
</code></pre>
<p>When I say your public key, I mean go to your public key file, ignore
the header and footer lines, concatenate each line then paste it there.</p>
<p>Also take the opportunity to set up SPF. Add a TXT record to
<code>your.domain.tld</code>, with:</p>
<pre><code>v=spf1 a mx ip4:<your ip address> ~all
</code></pre>
<p>Where your IP address is your <em>static</em> IP. If you don't have a static
IP address, you'll probably forever be on every spam list
ever. Contact your ISP to get a static IP, or get a VPS.</p>
<h2>Final touches</h2>
<p>Now just start everything:</p>
<div class="highlight"><pre><span></span>$ rcctl <span class="nb">enable</span> dkimproxy_out
$ rcctl restart dkimproxy_out smtpd
</pre></div>
<p>And everything should work!</p>
<p>But now you wonder how I read mail? Easy, I SSH in and use
mutt. Couldn't be any simpler.</p>
<p>But how do I check it on my phone? Dude, I just told you! SSH and
mutt. I can use <code>mail(1)</code> if I really can't deal with a complicated
TUI.</p>
<p>To my surprise, everything worked. <a href="http://www.mail-tester.com">http://www.mail-tester.com</a> gave
me a clean bill of health and I found my mails were able to reach all
my friends and family.</p>
<p>And no arbitrary usage limits! I can download my many gigabytes of
mail as many times as I want!</p>
<p>But of course, I do have to still download them from Zoho to upload
them to my new server...</p>
Hardware Censusurn:https-kaashif-co-uk:-2017-08-02-hardware-census2017-08-02T00:00:00Z2017-08-02T00:00:00ZKaashif Hymabaccus
<p>Before I get my Sun Ultra 5 working and can write something about
that, I thought I'd go through all the hardware I'm using right now
and the OSes I'm running on them. Spoilers: it's all OpenBSD and Debian.</p>
<!--more-->
<p>I have accumulated quite a few laptops in various states of
disrepair. The working ones, I prefer to install OpenBSD on them. The
non-working ones become spare parts for the working ones. Occasionally
I see a great laptop on eBay for 99p with the comment "for
parts". Except the "problem" is that the memory is bad, or the HDD
died, or something easy to fix, so I snag it and get it working.</p>
<p>Enough with the backstory, let's see some dmesgs!</p>
<h3>thinkpad-t61</h3>
<pre><code>hw.machine=amd64
hw.model=Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHz
hw.ncpu=2
hw.byteorder=1234
hw.pagesize=4096
hw.disknames=cd0:,sd0:6ec4c14a8e9df601
hw.diskcount=2
hw.cpuspeed=2001
hw.setperf=100
hw.vendor=LENOVO
hw.product=765912G
hw.version=ThinkPad T61
hw.serialno=L3B4859
hw.uuid=c4ede381-495a-11cb-a14e-9b55a480804b
hw.physmem=1047199744
hw.usermem=1042952192
hw.ncpufound=2
hw.allowpowerdown=1
hw.perfpolicy=manual
</code></pre>
<p>This was my main laptop for quite a while and I still use it
occasionally. It's got enough RAM to get work done and a strong enough
CPU so you don't die waiting for things to compile. But while the
graphics are well supported (Intel GMA), they are very weak.</p>
<p>Now, 2/3 times it fails to boot with a bad memory error. So I moved
on. I still test a port or two on it occasionally.</p>
<p><a href="/static/dmesg.thinkpad-t61">Here</a> is the dmesg. It runs OpenBSD-current.</p>
<h3>thinkpad-r61</h3>
<p>This one runs Debian, since it has an Nvidia Quadro graphics card
(unaccelerated on OpenBSD). I suppose I could run NetBSD or FreeBSD on
it, but it works fine. Besides, an OS monoculture is bad, right?</p>
<p>It has 2GB of RAM, a decent enough CPU and the GPU is actually capable
of running some games at an OK speed. But when you have to turn the
graphics to low on 15 year old games, you get the feeling this laptop
wasn't designed for gaming.</p>
<p><a href="/static/dmesg.thinkpad-r61">Here</a> is the dmesg.</p>
<h3>thinkpad-760el</h3>
<pre><code>hw.machine = i386
hw.model = Intel Pentium (P54C) ("GenuineIntel" 586-class)
hw.ncpu = 1
hw.byteorder = 1234
hw.physmem = 33140736
hw.usermem = 32948224
hw.pagesize = 4096
hw.disknames = wd0,fd0
hw.diskcount = 2
</code></pre>
<p>This is the old one. Since I only use it as a serial terminal, it
really doesn't matter what OS I run on it. I run OpenBSD 3.3 on
it. This is because I tried to install NetBSD, but there were network
problems even though my card (see the dmesg if you want to know which
card) is supported. So I went the easy route and installed
OpenBSD 3.3. It installed from a single floppy, which is great. Even
OpenBSD 6.1 installs from a single floppy, so that hasn't changed at least.</p>
<p><a href="/static/dmesg.thinkpad-760el">Here</a>'s the dmesg.</p>
<h3>acer-vn7-591g</h3>
<pre><code>hw.machine=amd64
hw.model=Intel(R) Core(TM) i7-4720HQ CPU @ 2.60GHz
hw.ncpu=8
hw.byteorder=1234
hw.pagesize=4096
hw.disknames=sd0:050a15853dd14169,sd1:
hw.diskcount=2
hw.cpuspeed=2601
hw.setperf=99
hw.vendor=Acer
hw.product=Aspire VN7-591G
hw.version=V1.15
hw.serialno=NXMUVEK046514021B66600
hw.uuid=64980eb6-aa08-4cf3-80f6-ff6cd8648269
hw.physmem=12788166656
hw.usermem=12738244608
hw.ncpufound=8
hw.allowpowerdown=1
hw.perfpolicy=manual
</code></pre>
<p>This is the laptop I do most of my work on. It has an Intel GPU (the
one I use most of the time) and an Nvidia GPU (I only use this to boot
into Debian and play the occasional game of whatever).</p>
<p>I, of course, run OpenBSD-current on it. This is where I play with
ports and stuff.</p>
<p><a href="/static/dmesg.acer-vn7-591g">Here</a> is the dmesg.</p>
<p>I have a few more laptops, but since I never use them for anything,
listing them here would be pointless.</p>
<h3>ibm-system-x3455</h3>
<pre><code>hw.machine=amd64
hw.model=Dual-Core AMD Opteron(tm) Processor 2218
hw.ncpu=4
hw.byteorder=1234
hw.pagesize=4096
hw.disknames=wd0:a216e88897876460
hw.diskcount=1
hw.cpuspeed=2593
hw.vendor=IBM
hw.product=IBM System x3455-[7984W20]-
hw.serialno=KDYHXP7
hw.uuid=0b2a5418-7277-3bc8-a15a-85dc8760d5d2
hw.physmem=12867076096
hw.usermem=12867051520
hw.ncpufound=4
hw.allowpowerdown=1
</code></pre>
<p>I picked this up for cheap to be my build server, file server, etc. It
doesn't have many places to put HDDs, but that's not an issue since I
only have 2 hard drives to put in it, and I keep one disconnected from
it (long term backups like photos, videos, etc). I also keep off-site
backups and backups on other people's computers (also known as the
cloud).</p>
<p><a href="/static/dmesg.ibm-system-x3455">Here</a> is the dmesg, it's fairly
boring.</p>
<p>So there we have it, that's my in-use hardware.</p>
<p>Soon, I'll be adding a Sun Ultra 5 workstation to that. Finally, I'll
be able to see what specs it has (I forgot and don't have the
documents). I could open it up and look inside, but who knows how many
of the disks, sticks of RAM, etc actually work. Maybe the CPU doesn't
work.</p>
<p>Hopefully the next post won't be a post-mortem of the workstation.</p>
Cutting down memory usage of a Haskell web appurn:https-kaashif-co-uk:-2017-06-28-cutting-down-memory-usage-of-a-haskell-web-app2017-06-28T00:00:00Z2017-06-28T00:00:00ZKaashif Hymabaccus
<p>You may have heard of the Star Trek script search tool at
<a href="http://scriptsearch.dxdy.name">http://scriptsearch.dxdy.name</a>. I'm writing a similar thing for
Stargate. The difference is, of course, is that my tool will be
running on some crappy Amazon AWS t2.nano with no RAM.</p>
<p>The first prototype was written in Python, but the parsing code was
always written in Haskell (Parsec is great). I decided to move
everything into Haskell so there wouldn't be this redundancy of
parsing in Haskell then serializing to disk then reading it from
Python...one codebase for everything would be much simpler.</p>
<p>I wrote up a quick Haskell version, but there was one small problem
when I tried to use it:</p>
<pre><code>$ ./stargate-search
Setting phasers to stun... (port 5000) (ctrl-c to quit)
Killed
</code></pre>
<p>That's right, the OOM killer had to step in and put my app down like
the sloppy wasteful piece of junk it was.</p>
<p>How could I fix this?</p>
<!-- more -->
<h2>Profiling</h2>
<p>Obviously the first step when trying to fix code is trying to work out
what's wrong. I compiled my program with profiling options and ran
some stress tests.</p>
<div class="highlight"><pre><span></span>$ stack build --profile
$ stack <span class="nb">exec</span> -- stargate-search +RTS -p -s <span class="p">&</span>
$ ./tools/stress.sh
</pre></div>
<p>(All stress.sh does is shoot 25 requests to the server so we can see
how it runs)</p>
<p>The results were deeply troubling. Here are some highlights (bear in
mind, all this code does is search a few dozen MB of fairly structured
data).</p>
<pre><code>75,013,509,424 bytes allocated in the heap
5,751,785,424 bytes copied during GC
528,085,912 bytes maximum residency (17 sample(s))
5,809,384 bytes maximum slop
1523 MB total memory in use (0 MB lost due to fragmentation)
</code></pre>
<p>Wow, that's really something. 75 GB allocated? That seems like a lot,
but remember that the garbage collector in GHC is really good, we can
afford to allocate a ton as long as we get rid of it fairly quickly.</p>
<p>But the 528 MB residency is concerning. Surely all of that is during
the initial parse of the Stargate transcripts, right? I'd better take
a look at the graph...</p>
<p><img src="/static/profiles/df44891.png" alt="" /></p>
<p>Oh...I guess the peak residency <em>is</em> in the parsing stage, but it
doesn't go down much. Let's take a look at the cost centre breakdown
to see where the low-hanging fruit is.</p>
<pre><code>COST CENTRE MODULE SRC %time %alloc
match Web.Stargate.Main lib/Web/Stargate/Main.hs:95:1-87 71.3 56.4
satisfy Text.Parsec.Char Text/Parsec/Char.hs:(140,1)-(142,71) 6.0 13.0
string Text.Parsec.Char Text/Parsec/Char.hs:151:1-51 5.2 14.5
manyAccum.\.walk Text.Parsec.Prim Text/Parsec/Prim.hs:(607,9)-(612,41) 2.3 5.3
</code></pre>
<p>So this basically tells us that the vast majority of work is done in
the string matching code. Not a surprise. Let's take a look at what
that code is, exactly:</p>
<div class="highlight"><pre><span></span><span class="nf">match</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">String</span><span class="w"> </span><span class="ow">-></span><span class="w"> </span><span class="kt">String</span><span class="w"> </span><span class="ow">-></span><span class="w"> </span><span class="kt">Bool</span><span class="w"></span>
<span class="nf">match</span><span class="w"> </span><span class="n">query</span><span class="w"> </span><span class="n">body</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="p">(</span><span class="n">null</span><span class="w"> </span><span class="n">query</span><span class="p">)</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="p">((</span><span class="n">map</span><span class="w"> </span><span class="n">toUpper</span><span class="w"> </span><span class="n">query</span><span class="p">)</span><span class="w"> </span><span class="p">`</span><span class="n">isInfixOf</span><span class="p">`</span><span class="w"> </span><span class="p">(</span><span class="n">map</span><span class="w"> </span><span class="n">toUpper</span><span class="w"> </span><span class="n">body</span><span class="p">))</span><span class="w"></span>
</pre></div>
<p>So many things wrong with this...everyone always says premature
optimization is the devil, but I think I took it a bit too far in the
other direction.</p>
<h2>String is rubbish</h2>
<p>The obvious low-hanging fruit here is String. This is for several
reasons:</p>
<ul>
<li><p><code>type String = [Char]</code>: There is no secret optimization going on
here, you get all of the downsides of O(n) access with none of the
upsides of a linked list since we never do anything with the head,
where everything is O(1).</p></li>
<li><p><code>map toUpper query</code>: This is blatantly inefficient. Every time we
want to do a case-insensitive search, we map over the query but then
throw away the result? I guess the compiler might optimize this
away, but it would be smarter to upper case it before running the
match function a billion times.</p></li>
<li><p><code>isInfixOf</code> from <code>Data.List</code> is literally implemented like this:
<code>isInfixOf needle haystack = any (isPrefixOf needle) (tails haystack)</code>.
Sweet Jesus, that's the most naive implementation of anything I've
ever seen. I suppose that, since this is from <code>Data.List</code>, no
assumptions can be made about the data, so fair enough. But this is
another clue that I shouldn't be using <code>String</code>.</p></li>
</ul>
<p>Let's get on with fixing this. Essentially, I just imported
<code>Data.Text</code> qualified with <code>T</code> at the start of every file, replaced
<code>String</code> with <code>T.Text</code> everywhere, dealt with all of the type errors
and it Just Worked.</p>
<p>With no further optimization, here is the memory graph for the new
Text app:</p>
<p><img src="/static/profiles/d723356.png" alt="" /></p>
<p>A pretty big improvement, right? But it's still pretty slow.</p>
<h2>Highly hanging fruit</h2>
<p>Now that the low-hanging fruit was done with, let's move on. I thought
it would improve the program if I did away with all of the inefficient
list stuff and replaced it with <code>Data.Vector</code> where I could. After
all, I just build these things once when parsing then access them
millions of times: the cost is in the access, which is O(1) with
<code>Data.Vector</code>. I also replaced some hash tables with (key, value)
vectors, since 95% of the work is iteration, and all this hashtable
stuff is unnecessarily complex for that.</p>
<p>So in summary, I:</p>
<ul>
<li>Replaced <code>[a]</code> with <code>Vector a</code> where possible.</li>
<li>Replaced <code>HashMap a b</code> with <code>Vector (a,b)</code> where it made sense.</li>
</ul>
<p>This resulted in:</p>
<p><img src="/static/profiles/b51e1d8.png" alt="" /></p>
<p>Uh, that's a bit better I guess, but not really. The execution time,
in particular, is still really bad for such a simple app. The memory
usage I can forgive, since the data is about that big so we need to
keep it all in memory.</p>
<h2>Writing some imperative code for more speed</h2>
<p>At this point, I was just assuming that the RAM usage was OK (the
initial parsing thing happens only once, we can just ignore it) and I
set my sights on reducing the run time.</p>
<p>Looking at the search function, it's filled with folds and
stuff. People are always saying that the unpredictability of Haskell's
performance is due to laziness and inherent to the functional style,
since it's so far from the metal.</p>
<p>So why not write an imperative-style algorithm to search? We can do
this very easily using the <code>ST</code> monad. For those not in the know, this
is basically the <code>IO</code> monad, but you can escape it.</p>
<p>Essentially, you can have pointers, write and read to them, modify
stuff in memory, but still have the function you write be pure. This
is done by not allowing you to use the pointers or mutable values from
one <code>ST</code> monad in another one, so the mutability can't leak out of
your pure function.</p>
<p>For example, here's a Fibonacci function that runs in constant space:</p>
<div class="highlight"><pre><span></span><span class="nf">fibST</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Integer</span><span class="w"> </span><span class="ow">-></span><span class="w"> </span><span class="kt">Integer</span><span class="w"></span>
<span class="nf">fibST</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="ow">=</span><span class="w"></span>
<span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="n">n</span><span class="w"></span>
<span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="n">runST</span><span class="w"> </span><span class="o">$</span><span class="w"> </span><span class="kr">do</span><span class="w"></span>
<span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="ow"><-</span><span class="w"> </span><span class="n">newSTRef</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="ow"><-</span><span class="w"> </span><span class="n">newSTRef</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="n">fibST'</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">y</span><span class="w"></span>
<span class="w"> </span>
<span class="w"> </span><span class="kr">where</span><span class="w"> </span><span class="n">fibST'</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="kr">_</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">readSTRef</span><span class="w"> </span><span class="n">x</span><span class="w"></span>
<span class="w"> </span><span class="n">fibST'</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kr">do</span><span class="w"></span>
<span class="w"> </span><span class="n">x'</span><span class="w"> </span><span class="ow"><-</span><span class="w"> </span><span class="n">readSTRef</span><span class="w"> </span><span class="n">x</span><span class="w"></span>
<span class="w"> </span><span class="n">y'</span><span class="w"> </span><span class="ow"><-</span><span class="w"> </span><span class="n">readSTRef</span><span class="w"> </span><span class="n">y</span><span class="w"></span>
<span class="w"> </span><span class="n">writeSTRef</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">y'</span><span class="w"></span>
<span class="w"> </span><span class="n">writeSTRef</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">$!</span><span class="w"> </span><span class="n">x'</span><span class="o">+</span><span class="n">y'</span><span class="w"></span>
<span class="w"> </span><span class="n">fibST'</span><span class="w"> </span><span class="p">(</span><span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">y</span><span class="w"></span>
</pre></div>
<p>So I rewrote the searching function in this style. The resulting code
was much easier to read, but the performance, well:</p>
<p><img src="/static/profiles/4d80194.png" alt="" /></p>
<p>Yeah, that's right, it's <em>worse</em>! I guess GHC does a good job with all
of those folds and stuff, better than I can do.</p>
<h2>Remember to check all of your profiler's output</h2>
<p>I hadn't actually looked at the profiler output for the execution
time. What I saw when I did surprised me and made me facepalm a bit.</p>
<pre><code>COST CENTRE MODULE SRC %time %alloc
match Web.Stargate.Search lib/Web/Stargate/Search.hs:36:1-87 41.5 30.4
caseConvert Data.Text.Internal.Fusion.Common Data/Text/Internal/Fusion/Common.hs:(398,1)-(405,45) 15.4 0.0
upperMapping Data.Text.Internal.Fusion.CaseMapping Data/Text/Internal/Fusion/CaseMapping.hs:(16,1)-(219,53) 14.2 24.8
</code></pre>
<p>So it turns out 24.8% of the allocations and 29.6% of the runtime of
the program is spent <em>mapping text to upper case</em>. That's ridiculous!
How didn't I see this?</p>
<p>So I just added a field to the transcript data structure which
contained an uppercased version, I made <code>match</code> use this instead of
computing it itself every time and look at the result:</p>
<p><img src="/static/profiles/a1bf617.png" alt="" /></p>
<p>So now my program is faster. The memory usage is still bad, but that's
a one-time thing. I can solve that by just switching to Attoparsec or
some other actually performant parsing library.</p>
<h2>Conclusion</h2>
<p>Don't assume you know better than your tools: trust your profiler and
compiler to tell you where you're wrong!</p>
Trying out DragonflyBSDurn:https-kaashif-co-uk:-2017-06-22-trying-out-dragonflybsd2017-06-22T00:00:00Z2017-06-22T00:00:00ZKaashif Hymabaccus
<p>So I was setting up a laptop I had just picked up, when it came to
deciding what OS to install on it. Obviously, I'd probably end up
installing some Linux and maybe also OpenBSD (hard drives are huge
nowadays, I could fit hundreds of OSs on there). While I had tried out
FreeBSD and NetBSD, DragonflyBSD had never been on my radar.</p>
<p>It still isn't, really, but I thought I'd try it out on an old laptop,
just to see what it was like. It went pretty well, but there were a
few oddities and one or two kind of weird design choices.</p>
<!-- more -->
<h2>Getting installation media</h2>
<p>I expected this to be a breeze, since all OSs offer all sorts of
images. Dragonfly is no exception, on
<a href="https://www.dragonflybsd.org/download/">their downloads page</a>, they
offer a ISO for DVDs and a raw disk image, for USB disks. I went for
the latter.</p>
<div class="highlight"><pre><span></span>$ wget http://mirror-master.dragonflybsd.org/iso-images/dfly-x86_64-4.8.0_REL.img.bz2
$ bunzip2 dfly-x86_64-4.8.0_REL.img.bz2
$ sudo dd <span class="k">if</span><span class="o">=</span>dfly-x86_64-4.8.0_REL.img <span class="nv">of</span><span class="o">=</span>/dev/sdb
</pre></div>
<p>This should work, but when I booted it up on my laptop, the bootloader
screen came up, some kernel messages were printed, but then I got an
error saying da8s1a (the root partition of the USB disk) was not
found.</p>
<p>This was weird, so I typed the following:</p>
<pre><code>mountroot> ?
da8
da0s1
da0s2
</code></pre>
<p>Uh, so apparently the USB disk doesn't have any partitions. I was not
able to solve this problem, probably because all of my disks were
bad - when I wrote the image to each of 3 USB sticks, each of them was
different and none of them matched the actual image. I didn't manage
to overcome this, so I just used the DVD image:</p>
<div class="highlight"><pre><span></span>$ wget http://mirror-master.dragonflybsd.org/iso-images/dfly-x86_64-4.8.0_REL.iso.bz2
$ bunzip2 dfly-x86_64-4.8.0_REL.iso.bz2
$ sudo cdrecord <span class="nv">dev</span><span class="o">=</span>/dev/sr0 dfly-x86_64-4.8.0_REL.iso
</pre></div>
<p>Your device name may vary: the machine I'm doing this from is running
Gentoo Linux, so the CDRW drive is <code>/dev/sr0</code>.</p>
<p>This disk booted and brought me to the installation menu without a
hitch.</p>
<h2>Installing</h2>
<p>The menus were fairly pleasant and user-friendly. There's even a nice
dragonfly logo behind the ncurses menu.</p>
<p>I was prompted about whether I wanted UFS or HAMMER. I've heard a lot
of good things about HAMMER, so I went for it, mainly to see if it
would break. It didn't break. I don't plan to use DragonflyBSD too
much (since it's very close to FreeBSD, which I tried for a period of
time and later abandoned), so this is the most I can really say.</p>
<p>I noticed that the partition layout was simple, just <code>/</code> (taking up
90% of the space), <code>/boot</code>, swap and something called <code>/build</code> taking
up the rest of the space. I assume this is something to do with
building from source, since it contains a <code>usr.obj</code> directory, and
some other buildy stuff.</p>
<p>Something <em>very</em> strange was that I was not allowed to use any
non-alphanumeric characters in my passwords. This means no ;'[]"! and
so on. I cannot fathom why this is the case.</p>
<p>Also, the default shell is tcsh, which is just csh plus some
things. It goes without saying that there are
<a href="http://www.grymoire.com/unix/CshTop10.txt">a lot of reasons not to use csh</a>
(at least ten). So this is less than ideal. We can change this later,
so I wasn't too fussed about it at this point. But ugh, I hate csh.</p>
<p>Next, I was asked which network interface I wanted to configure. There
were a ton of them, including <code>faith0</code>, <code>sl0</code>, <code>ppp0</code>, but I knew I
was looking for <code>em0</code>, so there was no confusion here. For a beginner,
it might be a bit weird that these pseudo-devices are enabled by
default. I don't think many beginners install DragonflyBSD, though...</p>
<p>The rest of the installation (time zone, installing the bootloader,
etc) was problem-free, so I just had to reboot and get cracking.</p>
<h2>Using the system</h2>
<p>Obviously I won't be using it long-term but I just wanted to confirm I
was able to get everything set up with a minimum of pain.</p>
<p>The first order of business is to install X and some other stuff. I
settled on Xfce, since it's so hassle-free to set up.</p>
<p>I realised that sudo wasn't installed (neither is doas from OpenBSD),
so I tried <code>su</code>. I was prohibited from doing this, so I logged out and
back in as root to install sudo. It's very simple:</p>
<div class="highlight"><pre><span></span>$ pkg update
$ pkg install sudo
</pre></div>
<p>It already comes with package mirrors preconfigured and everything, so
this is all you need.</p>
<p>Then I logged back in as <code>kaashif</code> and installed Xfce:</p>
<div class="highlight"><pre><span></span>$ sudo pkg install xfce
</pre></div>
<p>After installing, I tried to run it with <code>startxfce4</code>, but I got
complaints that xinit wasn't installed. And neither was X, actually!
This is weird since I thought Xfce should depend on X.</p>
<div class="highlight"><pre><span></span>$ sudo pkg install xorg xinit
</pre></div>
<p>Surely now X would run! No, I get an error that <code>libssl.so</code> isn't
found. How is that possible? Anyway, I install libressl:</p>
<div class="highlight"><pre><span></span>$ sudo pkg install libressl
</pre></div>
<p>And try again, but now I get a different error:</p>
<pre><code>...
/etc/machine-id does not exist
abort
...
</code></pre>
<p>I thought dbus was supposed to make that for me. No matter, we can
solve that:</p>
<div class="highlight"><pre><span></span>$ dbus-uuidgen <span class="p">|</span> sudo tee /etc/machine-id
$ startxfce4
</pre></div>
<p>And voilà! It works! Except for the clit mouse...but I guess I can
make do with a USB mouse.</p>
<h2>Conclusion</h2>
<p>You know, that wasn't too bad, I remember struggling with video
drivers and monitor refresh rates and dreadful Xorg.conf
files...things have really improved not just on Linux, but on BSD,
too.</p>
<p>Not perfect, but I expected it to be a lot worse, given how few people
use it. I guess that's the result of being a fork of the most popular
BSD.</p>
Playing around with distccurn:https-kaashif-co-uk:-2017-03-20-playing-around-with-distcc2017-03-20T00:00:00Z2017-03-20T00:00:00ZKaashif Hymabaccus
<p>Today, I decided to install Gentoo on a spare machine I had lying
around, since I was bored. Obviously, the first issue I ran into was
that <code>emerge x11-base/xorg-server</code> was taking a really long time to
run since the Xorg server is a pretty bloated program. Then <code>emerge
firefox</code> was taking forever too.</p>
<p>One solution (for Firefox anyway) was to use the provided binary
packages, this meant <code>firefox-bin</code> for Firefox. But this means I
abandon all of the nice features (for me, that means USE flags) that
Gentoo offers. If I'm going to download a load of binaries that I
can't customize, why not just install Debian?</p>
<p>So the solution is to speed up compilation. That means putting more
CPU cores to work. But my poor old ThinkPad only has 2 cores! This is
where <code>distcc</code> comes in.</p>
<!--more-->
<h2>What is distcc?</h2>
<p>It's a way of distributing compilation jobs to build servers which are
supposed to have huge beefy CPUs. This saves a lot of time.</p>
<p>To explain how to set it up, I only need quote the <code>distcc</code> man page:</p>
<pre><code>1 For each machine, download distcc, unpack, and install.
2 On each of the servers, run distccd --daemon with --allow
options to restrict access.
3 Put the names of the servers in your environment:
$ export DISTCC_HOSTS='localhost red green blue'
4 Build!
$ make -j8 CC=distcc
</code></pre>
<p>This is an OK explanation of the general process, but the specifics
can get a bit more fiddly than that.</p>
<h2>My setup</h2>
<p>Because I didn't want to put too much effort into standardising the
OSs of my machines, my <del>pile of shitty laptops</del> build cluster runs a
whole host of different Linux distros. There's Debian, Slackware,
Ubuntu, Arch. I really wanted to use my OpenBSD server in here, but
sadly distcc doesn't abstract away the build server OS that much: the
files it sends just get compiled by the server's compiler then sent
back - nothing too fancy.</p>
<p>My instructions will focus on Debian, since my Xeon desktop and
dual-Opteron server both run Debian - all the other machines probably
contribute less than 10% of the total CPU power.</p>
<p>Installing and configuring on Gentoo should be basically the same if
you're using systemd.</p>
<h2>Installing</h2>
<p>The package and ebuild are both named <code>distcc</code>, so just use <code>apt</code> or
<code>emerge</code> as appropriate. Also install <code>distcc-pump</code>, for an extra
speed boost: it offloads even more of the build process to the server
(take a look at
<a href="https://opensource.googleblog.com/2008/08/distccs-pump-mode-new-design-for.html">this</a>).</p>
<h2>Configuring the client</h2>
<p>There are a few things that cause distcc to fail.</p>
<p>On the client (where the jobs get sent from), in <code>/etc/distcc/hosts</code>,
there may be a line reading <code>+zeroconf</code>. If you don't use Zeroconf,
just delete this line. It will likely cause distcc to fail to
distribute jobs even though it's supposed to realise you're not using
Zeroconf. Type the IP addresses or hostnames (if they will resolve
correctly) here, separated by spaces. For each IP, add <code>,cpp,lzo</code> to
the end, so that pump mode will work correctly and compression will be
used to send files.</p>
<p>So an example config might look like:</p>
<pre><code>192.168.0.32,cpp,lzo big-server,cpp,lzo my-desktop,cpp,lzo
</code></pre>
<p>Then go to <code>/etc/portage/make.conf</code> and add <code>distcc distcc-pump</code>
to the <code>FEATURES</code> variable (or create it if it doesn't exist).</p>
<p>Change <code>MAKE_OPTS</code> to <code>-jN -lM</code> where <code>N</code> is double the total number
of CPU cores available and <code>M</code> is however many jobs you want to run
locally (in my case, 2). We pick double the number of CPU cores since
the jobs are relatively small: it's possible the network will
bottleneck us if the number of jobs is too small and we have to wait
for many round trips for small batches of work.</p>
<h2>Configuring the server</h2>
<p>After installing distcc, you need to edit <code>/etc/default/distcc</code>.</p>
<p>Add your subnet to <code>ALLOWEDNETS</code>. Mine looks like:</p>
<div class="highlight"><pre><span></span><span class="nv">ALLOWEDNETS</span><span class="o">=</span><span class="s2">"192.168.0.0/24"</span>
</pre></div>
<p>Change <code>LISTENER</code> to your IP address. That is, the IP address the
client will use to acces the server. For me that's:</p>
<div class="highlight"><pre><span></span><span class="nv">LISTENER</span><span class="o">=</span><span class="s2">"192.168.0.83"</span>
</pre></div>
<p>And set <code>ZEROCONF</code> to <code>"false"</code>, just in case it messes something up.</p>
<p>Now, enable and start the service:</p>
<div class="highlight"><pre><span></span>$ systemctl <span class="nb">enable</span> distcc
$ systemctl start distcc
</pre></div>
<p>There is one small thing left to do. The platform on my Gentoo
installation is <code>x86_64-pc-linux-gnu</code>, so when it sends out a job, it
expects the compiler to be <code>x86_64-pc-linux-gnu-gcc</code>. This is not
actually the case on my Debian installation, where the compiler is
called <code>x86_64-linux-gnu-gcc</code>. Note that the <code>pc</code> is missing. This
makes no difference - it really is the same platform. But unless we
fix this, distcc will fail.</p>
<p>The solution (maybe a bit of a hack) is to just symlink all the
platform-specific stuff (ld, nm, gcov, gcc, g++ etc). Here's how I did
that:</p>
<div class="highlight"><pre><span></span>$ ls <span class="p">|</span> grep x86_64-linux <span class="p">|</span> <span class="k">while</span> <span class="nb">read</span> a<span class="p">;</span> <span class="k">do</span> sudo ln -sf <span class="nv">$a</span> <span class="si">${</span><span class="nv">a</span><span class="p">//linux/pc-linux</span><span class="si">}</span><span class="p">;</span> <span class="k">done</span>
</pre></div>
<p>This certainly works, maybe there's a better way to do it, though.</p>
<p>Since we are working in a single LAN, I assume you don't firewall
traffic between computers in your LAN (if you're its only user, for
example). If you do, you want to open port 3632 on your servers.</p>
<h2>The moment of truth</h2>
<p>Now, try to emerge a random package using <code>pump</code> and <code>distcc</code> as
follows:</p>
<div class="highlight"><pre><span></span>$ pump emerge firefox
</pre></div>
<p>This should transparently just use <code>distcc</code> and send some jobs over to
one or more of your servers.</p>
<p>Take a look at <code>/var/log/distcc.log</code> on one of your servers, it should
be filled with lines like this:</p>
<pre><code>distccd[14465] (dcc_job_summary) client: 192.168.0.82:36466 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:169ms x86_64-pc-linux-gnu-g++ main.cpp
</code></pre>
<p>My build time is 1/30 of what it was before, it's incredible. Of
course, what I save in time, I lose in electricity. So maybe next time
I'll just use a binary distro. But then, of course, I wouldn't be able
to make use of USE flags to stop 20 scripting languages getting pulled
in when I install <code>sudo</code>.</p>
<p>Happy compiling!</p>
Backing up PGP private keysurn:https-kaashif-co-uk:-2017-03-10-backing-up-pgp-private-keys2017-03-10T00:00:00Z2017-03-10T00:00:00ZKaashif Hymabaccus
<p>There are hundreds of blog posts about backing up your PGP keys around
on the internet. Most of them just say something like: put a
passphrase on it, keep it on a USB stick, a CD, a floppy disk, or
something like that. These are all very useful ways to back important
stuff up - in fact, I just restored a backup of my GPG keys from a CD
after deleting them by accident. These mediums are, however, not going
to survive for many decades like paper can. And if you store to a
writeable medium like a rewritable CD, DVD, USB stick or floppy, there
is still the danger of you trying to restore then accidentally
deleting your backup. Or, more likely, you write over it without
realising many months later that you wrote a Linux distro or movie to
that DVD that had your only PGP key backup.</p>
<!--more-->
<p>There are dozens of blog posts floating around about
<a href="http://www.jabberwocky.com/software/paperkey/">paperkey</a>, which is a
program intended to extract the important bits of a PGP key, output
this binary data in a pleasing human-readable format (basically a hex
dump), then allow you to print it out and have a reasonably short
thing you could recover your private PGP key from. The reason paperkey
has a use at all is that without it, the whole "private" key (which
includes a copy of the public key) would be ludicrously long and
impossible to type into a computer correctly. By this, I mean it will
be about 3000 bytes long. This is short for a computer, but very long
for a human.</p>
<p>There is still the question of how to recover this paper key. The
difficulty of this will be heavily dependent on how exactly you choose
to encode your private key before printing it out. Here are a few
methods of encoding binary data as something easily retrievable from a
printout.</p>
<h2>Getting started</h2>
<p>First, I'll generate a dummy key so I can walk you through the steps
of backing everything up.</p>
<div class="highlight"><pre><span></span>$ gpg --gen-key
gpg <span class="o">(</span>GnuPG<span class="o">)</span> <span class="m">2</span>.1.18<span class="p">;</span> Copyright <span class="o">(</span>C<span class="o">)</span> <span class="m">2017</span> Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Note: Use <span class="s2">"gpg --full-generate-key"</span> <span class="k">for</span> a full featured key generation dialog.
GnuPG needs to construct a user ID to identify your key.
Real name: Joe Bloggs
Email address: joe@bloggs.com
You selected this USER-ID:
<span class="s2">"Joe Bloggs <joe@bloggs.com>"</span>
Change <span class="o">(</span>N<span class="o">)</span>ame, <span class="o">(</span>E<span class="o">)</span>mail, or <span class="o">(</span>O<span class="o">)</span>kay/<span class="o">(</span>Q<span class="o">)</span>uit? o
...unimportant output...
pub rsa2048 <span class="m">2017</span>-03-11 <span class="o">[</span>SC<span class="o">]</span> <span class="o">[</span>expires: <span class="m">2019</span>-03-11<span class="o">]</span>
B39C0AB5ED8C797D25A559B7DDEC6B207C796781
B39C0AB5ED8C797D25A559B7DDEC6B207C796781
uid Joe Bloggs <joe@bloggs.com>
sub rsa2048 <span class="m">2017</span>-03-11 <span class="o">[</span>E<span class="o">]</span> <span class="o">[</span>expires: <span class="m">2019</span>-03-11<span class="o">]</span>
</pre></div>
<p>There are several choices at this stage. You can either just use GPG's
ASCII output and mangle it some way yourself, or you can use
paperkey.</p>
<h2>Using GPG's built in output</h2>
<p>Simply run:</p>
<div class="highlight"><pre><span></span>$ gpg -a --export-secret-key <span class="s2">"Joe Bloggs"</span>
-----BEGIN PGP PRIVATE KEY BLOCK-----
lQOYBFjDP+0BCACgxDb16nklo76aud9Fr0trxU2orqMA9JuDjNm4bkKoBcFkeEAe
cK4N36fpV7IWFGgYyNqd576EaC+83Trgv1YuDQ6m0dn4gFnWDN5OquTyvjcrU5/z
STIS38DcicuMKT1SUoGhX6/zBtghVH3dEDuqiwMgT8gelKt12kOy/ZnpTnBLgIEv
V2Y6zLjNqVoG8RXKeop+pyzRkIeKE667O7hPMez5ODKNnLtJDBTADxkxVyZx5ZAk
afmY74ChEotNBJNyDOri2KJ0TEeH0nKR3e7El5lZNeUzXP84y5DxQ/ufusKtTlFg
mIrtU3LurMw2Sdvkr0HqBrxFveeNMMRXaBbRABEBAAEAB/sGHtI01INeMnisLU1Z
us21QaKuPE/KVoWhIXicc94nxWhSad2PCx0lPBGJaaRHAOnhn6vq/Qqcwdanawi1
y7L9N9QJ981Dj6db5cuE1S64KxOwm5NoUK4OV+RgwQI1yNAj1S5INXteVjFeO3g7
NUYAPSCWV1M4DtLkPrX7F3qHjjx2nSrRD7H5umEAi7xjwTyc9CQtwLaWRf0MqMhj
h4AWqB6NED7OGz5pB0ExxEab6CgMqX4Qhmn5GadqlS5BdzUEIQ8iVNSiaKsw4pZE
+H9XvQTlq/iPekjVKG6F4Zh9dlPllAFjo3R7CHwwe+zgAen6kM8dilugBkBkSx7k
sl0RBADF6uBsa+Rn5tKw4rtU21ox4qwLiPBeircXdWa8TIFsuzW04SJ5TMkUswKR
QrdWeiMmVWjAOv8UTJBPNuswZR9hDpK84Ohtkn9l9p91Z4kNQmqOsPxUDk4++Vx0
q4TiTnDYk6MvfY9VjU8Avq62suV3aXs3Ikj/O/FTYIKRYUzNNQQAz/JCkEbVfZ4f
tvhLmk7SQsu3zn1eh8nRMj73VLJA2ZFL0ccmdjdHiqZIfYlMpASXberfpiZQ4q4x
YMtCdZ6Wcv7QV1FBeTFJyu/tM5dswXiy+pRvg3PFj8XR2XHAf00ZwXpva2ODgG6X
QFMSXO7xrmkQXkPYS9NUE9yyzBErAq0D/0lAApgs6UyMuiyMFRLDHc5lWYAHID6m
0FWOXSkhBrMTAPBfx8/W67+Gp+zJ0NufC4HsP8oJ1J9Uu/TY0HrmxyFuIZSGv/KI
+gFmjKjPoaUCAPvGNFbmRfLLK7wmz3n8RVTgl1Z8SXB/3cpxryTTEa+8ikhdv+cB
PVdfpVsMcM7cPYi0G0pvZSBCbG9nZ3MgPGpvZUBibG9nZ3MuY29tPokBVAQTAQgA
PhYhBLOcCrXtjHl9JaVZt93sayB8eWeBBQJYwz/tAhsDBQkDwmcABQsJCAcCBhUI
CQoLAgQWAgMBAh4BAheAAAoJEN3sayB8eWeB7WMH+wcwrQx/lvjRdCa43vmSReD8
dJN1jgkz6h9Q6jJCdwLVl8DXz7IO8vAJ7nbpDlHPOZ+t4FndrgbHw0kiwbIk9HUp
HueWxhf60cmOJtUnvKmVKnhie3jsd/T/W+N4DbNv1vO+cwtVFTC2KKhT0I5xCPd1
7q/L6zszpeBk4fUyrNN4vM3wwoNykmOENtEv9095Q99ccLxIYuhwMKaqTFV1XhNT
gPczshvyxfznReSAC5dRGLf/dG7tgoYEBtS7gc6ejOuMSfJe1b4Xp4hwY6QU3d/A
mTSguXEP3qqZAh5ZRMc2M723ACu906y4OWLPtP2Ip5DGWH5JWYrUR7hZ8yr6zyOd
A5gEWMM/7QEIAKjsUvglQZ00JPotnDGiGNE95ZGXnSn7bkOSWGHOdNkHW3Z7kmmZ
H/R619Xc8TNQ7rN6sQp2uz3wcvqYF5k5B+gG1st3RKii5tIwTykvgIn4ddoitn/h
agrvUjxJmh9pyS8HlhHfp12qnZ7Upyzl0DfHg8Qpht61Wr9/tbnUkuHTzEYOqag1
5rVBSKqB/yRaYG8XPOYN8pn5AUWdnyHxT0MBLbwfDeA5HuHYz2yx0QCZHjiJga3B
zPe570GvsE9spoDjYyN8cdpO4FG4NhMqwFwQMSWWH02/ouM+93XXwLnlFUaCwP6h
AdJfjbHMiMx5UFn/bY8MJi9Ia7GbA4dQQhMAEQEAAQAH/RnYmvNL5Ae1IElFLEZt
2mU9lsAZlhsD1QGyxSIl8Dv6w7RTwPm2S6zhFOAsn50t72/3wFntA8Y84aLVHZs8
niiSz0+vbops7mtPp/URxxWVNhcLw6e6ajrFFmySCGpxCa7P9tbCRT3wKpDQUcnt
WdgHB3K+tduinQF6/WezDkxOF5aTCMiocdmx68ufAmgDrOFhY2o13/i1ODaOwt1b
2izlWGHYoSCye4tADqXGr6zLENlwm5xilV1nn+XQSDKqTksLSarVebKUn0bJJpDx
lKbiMO2rTDkLL37V8XetREU/wiOREnu1XY1y4Y7ciKDR4fTXEHVVDRU6rpXun9PN
NZUEAMXX0UWJgmCtLoJaF8+toshE6o23UYllZyv4JCuVv2KANiq9IFKX0lnKnffa
CRivKTuRhD/Qfao7hK9TkRDfIhQTvRPp6xTjqkfe1BNE9+wco0bTa6ECGDCkZ9eV
mV8mKD+931kHIKL0XaXEAdadDXS0NJIb93DhO2hDEx/c78VXBADalDZV8ivldVV7
+GSehIoaXYIS7Mj+FImj4fTTeQeE3G/qD7Q5OchnBTDCqxHdnh9V0TJkhW0KcaLA
hnoR+rn8LLKXHRk01EWNPlpKp7Iky4//gGHEolfETDTtDksfMwEnmo4JUhkINgku
icA96d14pnFhZOykMVk8wYkbrgbXpQP/QsyHzvST5Xzk31qhAbIv5nUQHOCcQUrc
qf//U9OSk87/BFV+iL65ILFlKrpjG27rIfnxCO9s220FEkeXa9nio26PkpJdtKwg
4BeBIUmBozX+Z+qnwEilzwOt1pbdVIWjbsfCK83zeLGSv+7fcMaWf4UM+Bhi9zHj
LApKiZSl77lHJ4kBPAQYAQgAJhYhBLOcCrXtjHl9JaVZt93sayB8eWeBBQJYwz/t
AhsMBQkDwmcAAAoJEN3sayB8eWeBwI4H/2FZvwPtZRkUbBdZmrdgQZXYPE0Qi++8
JIT0PVAfF2oUGYPWMTks+G5FaoBaWC3cqkNU9pym63FTTNu84z1V54bXwJy8Czuu
9BWNxe4oEu4vKqqW/cTF4gXIruET5uywC+2RiO6XRlSlaR3pD8LgLYQ6LmZ/JdKB
eyj65AO7lnkaP6BrEhEF0dMrY5hR6afSoqp0DyhvBajUSLDnMvq9hl5MAIxEZ1Mc
Ix7sEC+9Pa0daKBXp+8t80Sqp+Qpj/Dk4LYs3IIMp95F51qxF2b0+eimpV7SWEdT
s7/Duz3ZCmtk87r2NJkw2UzY3mXL2i4tQYPiu8m8riFfJiUkPk8SF5s<span class="o">=</span>
<span class="o">=</span>bte1
-----END PGP PRIVATE KEY BLOCK-----
</pre></div>
<p>You could print that out and try to type it in to recover. That would
be impossible. You could also print it in a nice font and try to use
OCR to read it. That might work, but if the OCR software makes even a
tiny error, you'd be incredibly hard-pressed to find it. I do not
recommend doing this.</p>
<p>You could pipe this into xxd, which would turn it into a hex dump. OCR
software may have an easier time with only letters and numbers, but I
still don't recommend it.</p>
<p>If you do go for this method and try to restore from backup, here's
how you do that (after getting your paper key into text file form by
either typing or OCRing):</p>
<div class="highlight"><pre><span></span>$ gpg --import my-key.txt
</pre></div>
<h2>Cutting down the size using paperkey</h2>
<p>First, we install paperkey. You could go over the sources yourself to
verify it's not malware then compile it yourself, but I have a nominal
level of trust in the Debian package maintainers. Thus this is fine
for me:</p>
<div class="highlight"><pre><span></span>$ sudo apt install paperkey
</pre></div>
<p>Next, we export our secret key, pipe it through paperkey to make sure
we only end up with the secret bits, then pipe that into a
file. There are a couple of different ways to do this.</p>
<h2>Using paperkey's built-in base16 output</h2>
<p>This is perhaps the easiest way to proceed. Run the following:</p>
<div class="highlight"><pre><span></span>$ gpg --export-secret-key <span class="s2">"Joe Bloggs"</span> <span class="p">|</span> paperkey
<span class="c1"># Secret portions of key B39C0AB5ED8C797D25A559B7DDEC6B207C796781</span>
<span class="c1"># Base16 data extracted Sat Mar 11 00:27:46 2017</span>
<span class="c1"># Created with paperkey 1.3 by David Shaw</span>
<span class="c1">#</span>
<span class="c1"># File format:</span>
<span class="c1"># a) 1 octet: Version of the paperkey format (currently 0).</span>
<span class="c1"># b) 1 octet: OpenPGP key or subkey version (currently 4)</span>
<span class="c1"># c) n octets: Key fingerprint (20 octets for a version 4 key or subkey)</span>
<span class="c1"># d) 2 octets: 16-bit big endian length of the following secret data</span>
<span class="c1"># e) n octets: Secret data: a partial OpenPGP secret key or subkey packet as</span>
<span class="c1"># specified in RFC 4880, starting with the string-to-key usage</span>
<span class="c1"># octet and continuing until the end of the packet.</span>
<span class="c1"># Repeat fields b through e as needed to cover all subkeys.</span>
<span class="c1"># </span>
<span class="c1"># To recover a secret key without using the paperkey program, use the</span>
<span class="c1"># key fingerprint to match an existing public key packet with the</span>
<span class="c1"># corresponding secret data from the paper key. Next, append this secret</span>
<span class="c1"># data to the public key packet. Finally, switch the public key packet tag</span>
<span class="c1"># from 6 to 5 (14 to 7 for subkeys). This will recreate the original secret</span>
<span class="c1"># key or secret subkey packet. Repeat as needed for all public key or subkey</span>
<span class="c1"># packets in the public key. All other packets (user IDs, signatures, etc.)</span>
<span class="c1"># may simply be copied from the public key.</span>
<span class="c1">#</span>
<span class="c1"># Each base16 line ends with a CRC-24 of that line.</span>
<span class="c1"># The entire block of data ends with a CRC-24 of the entire block of data.</span>
<span class="m">1</span>: <span class="m">00</span> <span class="m">04</span> B3 9C 0A B5 ED 8C <span class="m">79</span> 7D <span class="m">25</span> A5 <span class="m">59</span> B7 DD EC 6B <span class="m">20</span> 7C <span class="m">79</span> <span class="m">67</span> <span class="m">81</span> F905AA
<span class="m">2</span>: <span class="m">02</span> 8B <span class="m">00</span> <span class="m">07</span> FB <span class="m">06</span> 1E D2 <span class="m">34</span> D4 <span class="m">83</span> 5E <span class="m">32</span> <span class="m">78</span> AC 2D 4D <span class="m">59</span> BA CD B5 <span class="m">41</span> 84698B
<span class="m">3</span>: A2 AE 3C 4F CA <span class="m">56</span> <span class="m">85</span> A1 <span class="m">21</span> <span class="m">78</span> 9C <span class="m">73</span> DE <span class="m">27</span> C5 <span class="m">68</span> <span class="m">52</span> <span class="m">69</span> DD 8F 0B 1D 5CBE5F
... lots more output ...
</pre></div>
<p>This is a pretty good printout to have: it has instructions on how to
recover, it has CRCs for every line and for the whole thing,
everything's in nicely formatted base16. There is a problem: it can't
be easily read by a machine.</p>
<p>So you could OCR it and feed it into paperkey again, but this is
error-prone. There are CRCs, so you can easily check if each line is
correct and guess the correct line (since the errors are not random,
OCR will mistake similar-looking letters for one another).</p>
<p>Here is how you would do that (again, after converting your paper
backup to a text file):</p>
<div class="highlight"><pre><span></span>$ paperkey --pubring pubring.gpg --secrets printout.txt --output secretkey.gpg
</pre></div>
<p>Where the pubring.gpg is the file located in <code>~/.gnupg</code> after you've
imported your public key. You can then import <code>secretkey.gpg</code> as usual.</p>
<h2>Converting to a machine-readable paper format</h2>
<p>You might want to convert to a QR code, barcode or some other
machine-readable image format. For this section, I'll assume you have
your secrets in a <code>secrets.bin</code> file, made as follows:</p>
<div class="highlight"><pre><span></span>$ gpg --export-secret-key <span class="s2">"Joe Bloggs"</span> <span class="p">|</span> paperkey --output-type raw --output secrets.bin
</pre></div>
<p>There are a few ways to convert this to a machine-readable image:
let's start off with the most well-known, QR codes.</p>
<h3>Encoding as a QR code with qrencode</h3>
<p>Install qrencode:</p>
<div class="highlight"><pre><span></span>$ sudo apt install qrencode
</pre></div>
<p>Now you can encode your secrets.bin as follows:</p>
<div class="highlight"><pre><span></span>$ qrencode --8bit --level<span class="o">=</span>M -o key.png < secrets.bin
</pre></div>
<p>It looks like this:</p>
<p><img src="/static/key.png" alt="" /></p>
<p>Note: the reason I used the error correction level of M instead of the
highest, H, is that the resulting image would be obnoxiously large and
hard to print out (except on a poster). Printing a large image on a
small sheet of paper would only defeat the whole purpose of error
correction: you would be introducing more errors (squashed image) in
order to reduce errors.</p>
<p>If you find that your key's image is way too big, turn the error
correction level to L (this is actually the default).</p>
<p>This is great, but if your key is huge (maybe your chosen key format
is just big), you might notice that the largest type of QR code, a
version 40 177x177 one, can only store 2953 bytes. This is a serious
problem since if your key is too big, you're simply SOL - there is no
workaround. We'll have to find a different format.</p>
<p>If this problem doesn't affect you, then I highly recommend this
method: it's really easy to restore a QR code backup with any QR code
reader.</p>
<h3>Encoding as a data matrix with dmtxwrite</h3>
<p>Install the program:</p>
<div class="highlight"><pre><span></span>$ sudo apt install dmtx-utils
</pre></div>
<p>Now, encode your secrets:</p>
<div class="highlight"><pre><span></span>$ dmtxwrite -e <span class="m">8</span> -f PNG -o key2.png < secrets.bin
</pre></div>
<p>You'll end up with this:</p>
<p><img src="/static/key2.png" alt="" /></p>
<p>This has fewer problems than QR codes, in my opinion, since there is
no limit to the size of the data you're encoding. It is just as easy
to restore the data, though, so really this is an almost purely
aesthetic choice if either will work for you. If your key is huge, use
this. If your key is not huge, it doesn't matter which you pick.</p>
<h3>Encoding with a linear barcode</h3>
<p>No, this is stupid. Do you know how long that barcode would have to
be? And half of it would need to be error correction stuff. And
folding the paper in half would probably triple the number of errors
you get.</p>
<h2>Conclusion</h2>
<p>I would personally slap my private key into a paper data matrix and
CD-R and call it a day. I keep a folded up copy of my private key in
my wallet, just in case. In case of what, I have no idea.</p>
<p>Just don't put your private key online (even if it has a passphrase),
that's just idiocy. Even if you're storing it with someone you trust,
this doesn't mean the network between you and them is also
trustworthy. The only type of offsite backup of this sort of sensitive
data you should be keeping is one that you physically transport
offsite yourself.</p>
Making a list of the websites of people on nixers.neturn:https-kaashif-co-uk:-2016-05-05-making-a-list-of-the-websites-of-people-on-nixers-net2016-05-05T00:00:00Z2016-05-05T00:00:00ZKaashif Hymabaccus
<p>I wanted to make a list of the websites of the people on the website
<a href="http://nixers.net">http://nixers.net</a>, and I decided to solve it not by asking people to
tell me what their sites were called, but by scraping the forum.</p>
<p>I didn't scrape the whole forum, I just scraped one topic on the forum
that I created a few years ago: <a href="https://nixers.net/showthread.php?tid=1547">https://nixers.net/showthread.php?tid=1547</a></p>
<!-- more -->
<p>There really isn't much to it, all I had to do was to fetch all of the
pages of the forum post and somehow go through them and retrieve all
of the URLs, then do some filtering.</p>
<p>Getting the pages wasn't very difficult, I just used my good old
friend cURL:</p>
<div class="highlight"><pre><span></span>$ curl https://nixers.net/showthread.php?tid<span class="o">=</span><span class="m">1547</span> > <span class="m">1</span>.html
$ <span class="k">for</span> i <span class="k">in</span> <span class="o">{</span><span class="m">2</span>..8<span class="o">}</span><span class="p">;</span> <span class="k">do</span> curl <span class="s2">"https://nixers.net/showthread.php?tid=1547&page=</span><span class="nv">$i</span><span class="s2">"</span> > <span class="si">${</span><span class="nv">i</span><span class="si">}</span>.html<span class="p">;</span> <span class="k">done</span>
</pre></div>
<p>So I have all of the pages in the current directory. All I need to do
now was do some text processing. As always, the first tool in my
toolbox when I need to do complicated string matching involving
regexes is Perl. Sure enough, there is a Perl module on CPAN for
this. It even comes with a script to make running it super easy.</p>
<p>The module is URI::Find and after CPANing that, the script it installs
is urlfind. The documentation for the script can be found
<a href="http://search.cpan.org/~mschwern/URI-Find-20140709/bin/urifind">here</a>.</p>
<p>What I need to do is find all the URLs, remove all of the ones that
aren't personal sites, remove all of the duplicates, then store the
result in a file. Psh, no problem.</p>
<div class="highlight"><pre><span></span>$ urifind -n * <span class="p">|</span> <span class="se">\</span>
grep -vE <span class="s1">'nixers.net|github|imgur|openbsd|tumblr'</span> <span class="p">|</span> <span class="se">\</span>
sed -e <span class="s1">'s/https/http/'</span> -e <span class="s1">'s,/$,,'</span> -e <span class="s1">'s,http://,,'</span> -e <span class="s1">'s,/.*$,,'</span> <span class="p">|</span> <span class="se">\</span>
sort <span class="p">|</span> uniq > sites
</pre></div>
<p>I also snuck in a sed command that removes trailing slashes and all of
the URI path stuff, just leaving us with the domain names, which is
all we really want, anyway.</p>
<p>This gave me a file that I then went through to remove anything the
command didn't get rid of. The list I got is reproduced faithfully
below:</p>
<pre><code>albertocg.com
andrew.harrison.nu
arcetera.moe
arcetera.party
b4dtr1p.tk
blog.neeasade.net
blog.xero.nu
bugsofberk.net
charliethe.ninja
code.xero.nu
elliottpardee.me
eyenx.ch
fontvir.us
git.b4dtr1p.tk
icetimux.com
jona.io
josm.xyz
kaashif.co.uk
literallyryan.weebly.com
lugm.org
neeasa.de
neeasade.net
nullball.nu
pluviophile.xyz
ports.brianctomlinson.com
pub.iotek.org
punkweb.co
purestench.blogspot.com
qoob.nu
quitter.se
redpanduh.com
rocx.rocks
s0lll0s.me
stenchforums.net
strangequark.tk
thevypr.com
u2620.net
venam.1.ai
wildefyr.net
www.brianctomlinson.com
www.dafont.com
www.letterheadfonts.com
www.unixcri.me
xcelq.org
xero.nu
xero.owns.us
</code></pre>
<p>So there you have it, a list of websites you might want to check
out. I'll also put this on my about page, in case this post gets
buried (by me, in the future).</p>
Sharing /home between OpenBSD and Debianurn:https-kaashif-co-uk:-2016-05-03-sharing-home-between-openbsd-and-debian2016-05-03T00:00:00Z2016-05-03T00:00:00ZKaashif Hymabaccus
<p>Some people reading this might be thinking: "hey, it's really easy to
do this, why is he writing an article on this?". You are partially
right, this should be really easy, but there are some weird things
that happened while I set this up that I feel should have been written
down somewhere, so my fear that I was completely borking my system
would have been assuaged.</p>
<!-- more -->
<p>Anyway, let's get onto the first thing I had to do:</p>
<h2>Resizing my Windows partition</h2>
<p>Yes, I have Windows on my laptop. Yes, Stallman wouldn't be too
impressed. But come on, occasionally (very occasionally nowadays), I
have to run a program that only has a Windows version. In any case, I
don't have to justify myself to you, dear reader.</p>
<p>Doing it from inside Windows turned out to be less than easy, since I
was unable to shrink the C partition even after disabling and cleaning
out the pagefile, some system restore points and a load of other files
the Disk Cleanup Utility removed. I don't know what sort of hidden,
immutable system files were around stopping me, but I decided to just
boot into Debian, forcibly shrink it, and worry about it later.</p>
<p>I booted into Debian, started up Gparted and resized the partition
without issue. I then created a new partition (around 5GB in size, I
don't really keep too much stuff in my /home, this was really too
big).</p>
<h2>What filesystem should I use?</h2>
<p>This is the next big issue: there is not that much overlap between the
filesystems supported by OpenBSD and Linux. Well, certainly every
filesystem OpenBSD supports is supported by Linux to some extent, but
there are a few problems with most of them.</p>
<h3>FAT32</h3>
<p>Really, I can't use FAT32. It's 2016, how can I be caught storing my
files on FAT?</p>
<p>Seriously, though, it has no journaling, it's slow, it's really crap
and my data would all be corrupted and irretrievable after 5 mins.</p>
<p>On the other hand, it is very well supported by both OSes, but no,
let's move on.</p>
<h3>NTFS</h3>
<p>This is the really perverted option. Using NTFS to store the home
partition of 2 different Unix-like OSes. This doesn't seem like too
bad of an option really, since it is a fairly modern filesystem
compared to FAT32.</p>
<p>But most programs are designed with Unix permissions and
ownership. This is possible to setup with NTFS, but it doesn't work
out of the box, which is what I'd prefer. Moving on.</p>
<h3>ReiserFS, XFS, BTRFS, ZFS, others</h3>
<p>OpenBSD has very limited (read: none) support for any of these.</p>
<h3>Just using the filesystems I already use</h3>
<p>Why should I go through all of this hassle when I could just format
the partition with ext4 or UFS and be done with it?</p>
<p>Well, rw support for UFS does exist under Linux, but very little
attention is given to the default ufs (the FreeBSD one, as far as I
know), let alone the ones where you have to add the option
ufstype=something (44bsd for OpenBSD's UFS). I really don't trust the
Linux support for UFS at all. Obviously there is no support for soft
updates or any of the SSD stuff OpenBSD has either.</p>
<p>ext4 support technically does partially exist under OpenBSD, since you
can mount an ext4 partition with all of the ext4-specific features
(journaling, some extended attributes, basically everything in
/etc/mke2fs.conf) disabled, but then you might as well be using ext2.</p>
<h3>The only choice remaining</h3>
<p>Yes, that's right, I decided that plain ext2 was the best for my
needs.</p>
<p>Now all I had to do was format it and change my /etc/fstabs to point
to the right place. A piece of cake, right? Not quite, I ran into some
compatibility issues.</p>
<h2>Partition table woes</h2>
<p>I had already formatted the disk as ext2 while I was booted into
Debian, so I expected to just have to copy over my files, change the
fstab and that would be that.</p>
<p>Since OpenBSD was my main OS, I had to boot into OpenBSD to copy that
home directory to the partition I had made (and formatted) in
Debian. The partition was /dev/sda7.</p>
<p>Now, for those of you who don't use BSD, you won't be familiar with
BSD disklabels. They come from a time when you weren't allowed more
than 4 partitions per disk due to the limitations of the MBR (this is
still true for a good number of computers, so I guess it isn't a
complete anachronism). On my laptop, I can have a lot more partitions
that that thanks to GPT: I can have up to 128 partitions, more than
anyone reasonable would need, so really, the disklabels just get in my
way, since they are basically just a way of cramming 16 partitions
into 1 MBR or GPT partition.</p>
<p>I had 7 GPT partitions already and 10 OpenBSD "partitions" in the
disklabel in OpenBSD's little area of my disk. This makes a total
of 17.</p>
<p>This shouldn't really be a problem, except due to the way OpenBSD's
partitioning system works, it relies entirely on the disklabel to
tell it where the partitions are on the disk, and since Debian
obviously has no reason to update the OpenBSD disklabel, there is no
record of the ext2 partition we can see.</p>
<p>But what does that have to do with there being 17 partitions? Well,
disklabels can only keep track of 16 partitions, so I have to get rid
of some of the partitions we already know about.</p>
<p>Anyway, the only place I could get the info to add the ext2 partition
was in the GPT, the very place the partitions are stored. OpenBSD
should really just look at this itself every so often:</p>
<pre><code>$ doas fdisk sd0
Disk: sd0 Usable LBA: 34 to 117231374 [117231408 Sectors]
#: type [ start: size ]
----------------------------------------------------------------------
0: EFI Sys [ 2048: 204800 ]
1: e3c9e316-0b5c-4db8-817d-f92df00215ae [ 206848: 262144 ]
2: DOS FAT-12 [ 468992: 68757504 ]
3: Win Recovery [ 95830016: 921600 ]
4: 4f68bce3-e8cd-4db1-96e7-fbcaf984b709 [ 96753664: 20475904 ]
5: OpenBSD [ 77418496: 18411520 ]
6: OpenBSD [ 69226496: 8192000 ]
</code></pre>
<p>Yeah, I gave the partition the type OpenBSD in the hopes that this
would help OpenBSD see it. No, that didn't work.</p>
<p>Anyway, the actual solution was really simple, I just had to edit the
disklabel and add a record of this partition:</p>
<div class="highlight"><pre><span></span>$ doas disklabel -e sd0
</pre></div>
<p>At that moment, my disklabel looked something like this:</p>
<pre><code># /dev/rsd0c:
type: SCSI
disk: SCSI disk
label: LITEON IT L8T-64
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 7297
total sectors: 117231408
boundstart: 77418496
boundend: 95830016
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
a: 350400 77418496 4.2BSD 2048 16384 1 # /
b: 536980 77768896 swap
c: 117231408 0 unused
d: 544256 78305888 4.2BSD 2048 16384 1 # /tmp
e: 648896 78850144 4.2BSD 2048 16384 1 # /var
f: 2029760 79499040 4.2BSD 2048 16384 1 # /usr
g: 1160512 81528800 4.2BSD 2048 16384 1 # /usr/X11R6
h: 4567424 82689312 4.2BSD 2048 16384 1 # /usr/local
i: 204800 2048 MSDOS
j: 262144 206848 unknown
k: 68757504 468992 MSDOS
l: 921600 95830016 unknown
m: 20475904 96753664 unknown
n: 2171776 87256736 4.2BSD 2048 16384 1 # /usr/src
o: 2811648 89428512 4.2BSD 2048 16384 1 # /usr/obj
p: 3589760 92240160 4.2BSD 2048 16384 1 # /home
</code></pre>
<p>I just made this up, but you get the idea, it's basically a complete
mess and I've exhausted the number of partitions I can have. I just
deleted partitions m and n, since I'll probably never need those on
separate partitions, and added a line with the fdisk info for my new
partition:</p>
<pre><code>m: 8192000 69226496 ext2fs
</code></pre>
<p>Then I added the line to my /etc/fstab:</p>
<pre><code>/dev/sd0m /home ext2fs rw,nodev,nosuid 1 2
</code></pre>
<p>And I deleted the old /home line, of course.</p>
<p>After copying the files, I rebooted and everything worked.</p>
<p>Or did it?</p>
<h2>panic: system on fire</h2>
<p>After booting into OpenBSD and using it for a bit, it worked, so I
decided to boot into Debian and see if it worked there. It did.</p>
<p>But then I tried to boot back into OpenBSD and got a rather scary
error:</p>
<pre><code>** /dev/rsd0m
BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE
/dev/rsd0m: BLOCK SIZE DETERMINED TO BE ZERO
</code></pre>
<p>The system could boot, though, but all of my files had mysteriously
vanished. Running <code>fsck_ext2fs</code> gave the same error, leading me to
believe something was actually wrong.</p>
<p>I booted back into Debian and forced a fsck of the partition, but it
came out clean.</p>
<p>The moral of the story is that Linux and OpenBSD sometimes do things
differently, and in this case it is harmless, since the filesystem
works correctly from both OSes, it's just that the number of backup
superblocks Debian decided to keep was different than the number
OpenBSD was expecting. I could look up the right way and the exact
options to use to have perfect interop, but it works as is, so I can
worry about that later.</p>
<p>All I needed to do was disable the fscking of the partition at boot in
the /etc/fstab on OpenBSD, and both OSes booted up fine.</p>
<h2>Conclusion</h2>
<p>Hopefully, someone at some point will benefit from this post, even if
it's only that someone looks at this and decides it's too much trouble.</p>
Converting my blog to frogurn:https-kaashif-co-uk:-2016-04-29-converting-my-blog-to-frog2016-04-29T00:00:00Z2016-04-29T00:00:00ZKaashif Hymabaccus
<h1>What is frog?</h1>
<p><a href="https://github.com/greghendershott/frog">Frog</a> is a static website
generator written in Racket. It does the same sort of thing as Jekyll,
Hakyll and other software like that, some of which I've used in the
past.</p>
<!-- more -->
<h1>What does "frog" mean?</h1>
<p>There's a simple answer to this on their README:</p>
<blockquote><p>Q: "Frog"?
A: Frozen blog.</p></blockquote>
<p>I'm not 100% on what that means exactly, but the program works great,
so who am I to say that their name doesn't make sense?</p>
<h1>What this blog post isn't</h1>
<p>They have some really great documentation on how to get started using
frog and I'm not trying to write a blog post replicating all of their
effort.</p>
<p>What I really want to focus on is how I converted my Hakyll website
into a frog site.</p>
<h1>Converting all of my posts into the frog format</h1>
<p>The bulk of the work was going to be converting all of the metadata I
had from the Hakyll/Jekyll style:</p>
<pre><code>---
title: My Post Title
date: 1970-01-01
comment: something witty
---
</code></pre>
<p>into the frog format, which doesn't have a comment field (I added that
on top of Hakyll's default config myself anyway), but does have a tags
field and looks like this:</p>
<p>My Post Title
1970-01-01</p>
<p>So the header at the top of every file had to change almost
completely. Let's think, what tool can we use to go through a load of
files and transform each line based on a series of regexes and maybe
something more complicated?</p>
<p>That's right, I wrote a Perl script!</p>
<p>The aim of this exercise is just to come up with a script where I can
do <code>./myscript.pl post</code> and it'll output the changed post, so that I
can then just use <code>sponge</code> to overwrite the post itself.</p>
<p>(If you don't know about sponge, it's a really cool program for when
you want to do something to a file in-place, you can't be bothered
with a temporary file, and the program you're using doesn't support in
place operations, install it! It's usually in the package <code>moreutils</code>
on most distros, or at least it is on Debian.</p>
<h1>Writing the script</h1>
<p>It's a pretty standard script to be honest, since this is what Perl is
built for, processing text. Also, there's a long tradition of
sysadmins whipping up quick Perl scripts to do things, since Perl is everywhere.</p>
<p>I wrote the following in <code>script.pl</code>.</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/perl</span>
<span class="k">use</span> <span class="nn">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">warnings</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$fname</span> <span class="o">=</span> <span class="nv">$ARGV</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="nb">open</span><span class="p">(</span><span class="k">my</span> <span class="nv">$file</span><span class="p">,</span> <span class="s">"<"</span><span class="p">,</span> <span class="nv">$fname</span><span class="p">);</span>
<span class="k">while</span> <span class="p">(</span><span class="k">my</span> <span class="nv">$line</span> <span class="o">=</span> <span class="sr"><$file></span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$line</span> <span class="o">=~</span><span class="sr"> /^---$/</span><span class="p">)</span> <span class="p">{</span>
<span class="k">next</span><span class="p">;</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span><span class="nv">$line</span> <span class="o">=~</span><span class="sr"> /^title:/</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$title</span> <span class="o">=</span> <span class="p">(</span><span class="nv">$line</span> <span class="o">=~</span> <span class="sr">s/title: //</span><span class="n">r</span><span class="p">);</span>
<span class="nv">$title</span><span class="err">"</span><span class="p">;</span>
<span class="k">next</span><span class="p">;</span>
<span class="p">}</span> <span class="k">elsif</span> <span class="p">(</span><span class="nv">$line</span> <span class="o">=~</span><span class="sr"> /^date:/</span><span class="p">)</span> <span class="p">{</span>
<span class="k">my</span> <span class="nv">$date</span> <span class="o">=</span> <span class="p">(</span><span class="nv">$line</span> <span class="o">=~</span> <span class="sr">s/date: //</span><span class="n">r</span> <span class="o">=~</span> <span class="sr">s/\R//</span><span class="n">r</span><span class="p">);</span>
<span class="k">next</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">print</span> <span class="nv">$line</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>I don't know if that warrants much explanation, really, it's an easy
to understand script. I made sure not to use $_, since invisible
variables really frighten me.</p>
<p>Anyway after that, I just typed something into the shell:</p>
<div class="highlight"><pre><span></span><span class="k">for</span> f <span class="k">in</span> *.md<span class="p">;</span> <span class="k">do</span>
./script.pl <span class="nv">$f</span> <span class="p">|</span> sponge <span class="nv">$f</span>
<span class="k">done</span>
</pre></div>
<p>And that was that, all of my posts were now frog.</p>
<h1>That's essentially it</h1>
<p>Everything else worked as I'd expect, I just needed to convert my
templates to frog's style, which is really Racket's style, and that
was very easy, since I kept barely any of the logic of my site in the
templates.</p>
<p>Well, uh, there's not much more to say. Really, using frog is <em>that</em>
easy, I'd really recommend you use it if your current blogging
platform is giving you trouble.</p>
<p>Happy frogging!</p>
Hacking StumpWM with Common Lispurn:https-kaashif-co-uk:-2015-06-28-hacking-stumpwm-with-common-lisp2015-06-28T00:00:00Z2015-06-28T00:00:00ZKaashif Hymabaccus
<p>Before a few weeks ago, I was always one of those people who said that
Lisp isn't useful, it's not type-safe, it's not pure, Haskell is
better etc etc ad nauseam. All of that may be true for writing some
sorts of programs, but Lisp (well, Common Lisp anyway) provides
something a lot more pervasive.</p>
<p>What does pervasive mean? Well, right now, I'm controlling my window</p>
<!-- more -->
<p>manager and browser through a Lisp REPL from Emacs, and it's a lot
more useful (and fun) than it sounds.</p>
<h2>Setting up Emacs</h2>
<p>You too can get in on this action very easily. First, install a Common
Lisp implementation: I recommend SBCL, usually available in repos as
"sbcl" (e.g. apt install sbcl). Next, type <code>M-x package-install RET
slime RET</code> into Emacs and you'll have already installed the Superior
Lisp Interaction Mode for Emacs. It's as good as it sounds, trust
me. Next, add the following to your Emacs init file to make sure slime
knows where to find Lisp:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="k">setq</span> <span class="nv">inferior-lisp-program</span> <span class="s">"sbcl"</span><span class="p">)</span>
</pre></div>
<p>As an aside, you may also want to install <code>slime-company</code>, the
completion backend for <code>company-mode</code> for SLIME. Without it,
<code>company-mode</code> completion doesn't really work for SLIME. If you do do
that, then you'll also want to add the following to your Emacs init
file:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">slime-setup</span> <span class="o">'</span><span class="p">(</span><span class="nv">slime-company</span><span class="p">))</span>
</pre></div>
<p>I assume you already have run <code>global-company-mode</code> (why wouldn't you),
but if not, just add <code>(global-company-mode)</code> to the above to turn it
on.</p>
<p>Also, you will <em>definitely</em> want to install <code>rainbow-delimiters</code> and
<code>paredit-mode</code>, they are essential to any Lisp programming
experience. They are, however, not impossible to do without and I
won't go over how to use them in this article. Do install them,
though, they are really cool.</p>
<h2>Quicklisp</h2>
<p>The de facto Common Lisp library installer is Quicklisp. You'll
definitely need it, and need SLIME set up to work with it. Here's how
to do that. First, download <code>quicklisp.lisp</code> and run it (this is
copied and pasted from <a href="http://www.quicklisp.org/beta/">http://www.quicklisp.org/beta/</a>):</p>
<pre><code>$ curl -O https://beta.quicklisp.org/quicklisp.lisp
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 49843 100 49843 0 0 33639 0 0:00:01 0:00:01 --:--:-- 50397
$ sbcl --load quicklisp.lisp
This is SBCL 1.0.42.52, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
==== quicklisp quickstart loaded ====
To continue, evaluate: (quicklisp-quickstart:install)
* (quicklisp-quickstart:install)
</code></pre>
<p>There will be more output, but unless it's something glaringly
errorful, you're good to go, just tell Quicklisp to add itself to your
.sbclrc and quit:</p>
<pre><code>* (ql:add-to-init-file)
I will append the following lines to #P"/Users/quicklisp/.sbclrc":
;;; The following lines added by ql:add-to-init-file:
#-quicklisp
(let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
(user-homedir-pathname))))
(when (probe-file quicklisp-init)
(load quicklisp-init)))
Press Enter to continue.
#P"/Users/quicklisp/.sbclrc"
* (quit)
$
</code></pre>
<p>And that's that, you don't need to worry about telling SBCL about any
library stuff again. You do, however, need to add the following to
your Emacs init file:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">load</span> <span class="p">(</span><span class="nv">expand-file-name</span> <span class="s">"~/quicklisp/slime-helper.el"</span><span class="p">))</span>
</pre></div>
<p>That will make sure SLIME plays nice with anything you do with
Quicklisp in the future (it will see all of the libraries you install,
completion will work etc).</p>
<h2>Some light Lisp hacking</h2>
<p>Now you're ready to get hacking with Lisp! Just to test out SLIME,
open a file and input the following:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">defun</span> <span class="nv">hello-world</span> <span class="p">()</span>
<span class="p">(</span><span class="nb">format</span> <span class="no">t</span> <span class="s">"Hello, world!"</span><span class="p">))</span>
</pre></div>
<p>Now, <code>M-x slime</code>. You'll see something like this:</p>
<pre><code>; SLIME 2015-02-19
CL-USER>
</code></pre>
<p>That's the fabled REPL everyone always talks about. You should still
be in the buffer with the <code>hello-world</code> function, so type C-c C-l to
load it, then C-c C-z to switch to the REPL it was loaded to. You
could just use C-x o, but you may have multiple buffers open and it
may be more convenient to switch right to the REPL.</p>
<p>Now you can execute the procedure you just wrote by typing
<code>(hello-world)</code> and hitting return. You should end up with something
like this:</p>
<pre><code>CL-USER> (hello-world)
Hello, world!
NIL
</code></pre>
<h2>Some of what makes Lisp special</h2>
<p>Well that wasn't exciting, you can do the same thing with Python and
inf-python, or Ruby and inf-ruby! Anyone can load code from a buffer
and play around with it, what makes Lisp special? Well, here's a quote
from some guy who works on space stuff:</p>
<blockquote><p>Debugging a program running on a $100M piece of hardware that is 100
million miles away is an interesting experience. Having a
read-eval-print loop running on the spacecraft proved invaluable in
finding and fixing the problem.</p></blockquote>
<p>That's actually a quote from
<a href="http://www.flownet.com/gat/jpl-lisp.html">a Lisper at JPL</a> talking
about how useful a REPL is for debugging.</p>
<p>Think about it: you're running a window manager, you want to change a
tiny bit in the configuration, but you don't want to restart the
window manager, that could take <em>seconds</em> if not <em>tens of
seconds</em>. You're already in Emacs, so wouldn't it be great if you
could control and modify the state and configuration of your WM while
it runs?</p>
<p>I certainly think so. There aren't millions of dollars on the line
here if your WM crashes, but you could definitely save some time.</p>
<p>Anyway, onto the reason the post exists: StumpWM.</p>
<h2>Getting StumpWM set up</h2>
<p>You installed Quicklisp earlier, and for good reason. You <em>could</em>
install StumpWM using the system package manager, but it tends not to
work out well. I couldn't even get StumpWM to start with Debian's
stumpwm package, because of errors involving the also installed
cl-asdf package. I assume it was out of date, or something wasn't
being loaded properly or maybe I'm just an idiot.</p>
<p>Anyway, to install StumpWM, open up SBCL and eval the following:</p>
<pre><code>$ sbcl
* (ql:quickload "stumpwm")
</code></pre>
<p>It will take care of all dependencies and everything for you. Best of
all, you don't need to fiddle with any manual loading of libraries,
since Quicklisp takes care of all of that for you.</p>
<p>Now, replace the last line of your .xinitrc with the following:</p>
<pre><code>exec sbcl --load /path/to/startstump
</code></pre>
<p>In that <code>startstump</code> script, place the following:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">require</span> <span class="ss">:stumpwm</span><span class="p">)</span>
<span class="p">(</span><span class="nv">stumpwm:stumpwm</span><span class="p">)</span>
</pre></div>
<p>SBCL already knows about all of the libraries Quicklisp installed, so
this will start StumpWM. Just one more thing: you want to be able to
debug it live, right? Add the following to your <code>.stumpwmrc</code> (well,
create it with the following content):</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">in-package</span> <span class="ss">:stumpwm</span><span class="p">)</span>
<span class="p">(</span><span class="nb">require</span> <span class="ss">:swank</span><span class="p">)</span>
<span class="p">(</span><span class="nv">swank-loader:init</span><span class="p">)</span>
<span class="p">(</span><span class="nv">swank:create-server</span> <span class="ss">:port</span> <span class="mi">4004</span>
<span class="ss">:style</span> <span class="nv">swank:*communication-style*</span>
<span class="ss">:dont-close</span> <span class="no">t</span><span class="p">)</span>
</pre></div>
<p>This won't work until you install the <code>swank</code> library:</p>
<pre><code>$ sbcl
* (ql:quickload "swank")
</code></pre>
<p>Going back to the <code>.stumpwmrc</code>, notice how the port is set to 4004? I
do that so that when you start SLIME in Emacs, there are no errors
because the default port is actually 4005. This ensures you can't mess
up your WM by accident while writing unrelated code.</p>
<p>OK, ready for the moment of truth? Kill your X session and run
<code>startx</code> and you should see a "Welcome to StumpWM message". If it
didn't work, chances are there are some errors in the TTY you started
X from. Kill X and look at them if something went wrong. Chances are
something went wrong before the swank server started, so you wouldn't
be able to use SLIME to fix those errors.</p>
<p>If everything worked, fantastic!</p>
<h2>A taste of what's possible</h2>
<p>So you're sitting around coding up the next Node.js webscale NoSQL
business synergy application when you notice something you want fixed
with your window manager. You want to fix it <em>right now</em> with minimal
hassle. No worries, you can do it from within Emacs!</p>
<p><code>M-x slime-connect</code>. When prompted for host, accept 127.0.0.1. When
prompted for port, put in 4004 (not 4005). You are now inside the live
Lisp image of your WM. Exciting, right? Why not see if you can really
control it?</p>
<pre><code>CL-USER> (require :stumpwm)
NIL
CL-USER> (stumpwm:select-window-by-number 1)
NIL
</code></pre>
<p>That should've switched to window number 1...so you are in control! Why
not rebind a key?</p>
<pre><code>CL-USER> (stumpwm:define-key stumpwm:*root-map* (stumpwm:kbd "u") "exec urxvt")
NIL
</code></pre>
<p>Try it: press your prefix key then "u" (by default, C-t u) and a
urxvt (replace with your favourite terminal) will spawn.</p>
<p>And you did it all without leaving Emacs or restarting your WM!</p>
<p>I hope this has opened up a whole new world of Lisp hacking for
you. For me, it was the gateway drug. I now dream about macros and s-expressions.</p>
<p>Happy hacking!</p>
How to get a list of processes on OpenBSD (in C)urn:https-kaashif-co-uk:-2015-06-18-how-to-get-a-list-of-processes-on-openbsd-in-c2015-06-18T00:00:00Z2015-06-18T00:00:00ZKaashif Hymabaccus
<h2>Is it portable?</h2>
<p>First off, the information in this post definitely doesn't apply to
Linux (as it has a completely different way of doing things) and may
or may not apply to other BSDs (I see that NetBSD and FreeBSD both
have similar, maybe identical, kvm(3) interfaces). There certainly
isn't anything in POSIX to make this standard. The only real reason</p>
<!-- more -->
<p>any UNIX-like OSes have this particular interface comes from the
kvm(3) man page in OpenBSD:</p>
<blockquote><p>The kvm interface was first introduced in SunOS. A considerable number
of programs have been developed that use this interface, making backward
compatibility highly desirable. In most respects, the Sun kvm interface
is consistent and clean. Accordingly, the generic portion of the
interface (i.e., kvm_open(), kvm_close(), kvm_read(), kvm_write(), and
kvm_nlist()) has been incorporated into the BSD interface. Indeed, many
kvm applications (i.e., debuggers and statistical monitors) use only this
subset of the interface.</p></blockquote>
<p>So even with that, only the "generic portion" of the interface is
"standardised" (although it's not <em>really</em> standardised, it's just de
facto). Hence, using kvm_openfiles(3) and the like has no reason to
work on any other OS the same way it does on OpenBSD.</p>
<h2>Actually writing some code</h2>
<p>The information about running processes is stored somewhere: the
kernel. Even in Linux, with procfs (/proc), the info all really comes
from the kernel.</p>
<p>On OpenBSD, you want to access the running system's kernel image. This
is done using kvm_openfiles(3), which takes a host of parameters
detailing the <em>file</em> you want to load the kernel image
from. Obviously, you don't want to load a file, but the running
kernel. To do this, just pass in NULL as the parameters that have
anything to do with files: the function will know this means you want
the running system:</p>
<div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><kvm.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><limits.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><sys/param.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><sys/sysctl.h></span><span class="cp"></span>
<span class="kt">int</span><span class="w"></span>
<span class="nf">main</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">errbuf</span><span class="p">[</span><span class="n">_POSIX2_LINE_MAX</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="n">kvm_t</span><span class="w"> </span><span class="o">*</span><span class="n">kernel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">kvm_openfiles</span><span class="p">(</span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="n">KVM_NO_FILES</span><span class="p">,</span><span class="w"> </span><span class="n">errbuf</span><span class="p">);</span><span class="w"></span>
</pre></div>
<p>You should really check if it's null or whatever, but for didactic
purposes, I'll just leave out all of the boring error handling - it's
obvious where the error string would be stored (hint - errbuf), so you
can handle that.</p>
<p>Next, you want to get a list of processes from that kernel image using
kvm_getprocs(3):</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">nentries</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">kinfo_proc</span><span class="w"> </span><span class="o">*</span><span class="n">kinfo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">kvm_getprocs</span><span class="p">(</span><span class="n">kernel</span><span class="p">,</span><span class="w"> </span><span class="n">KERN_PROC_ALL</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">kinfo_proc</span><span class="p">),</span><span class="w"> </span><span class="o">&</span><span class="n">nentries</span><span class="p">);</span><span class="w"></span>
</pre></div>
<p>Again, check if null, handle errors. The number of processes obtained
is stored in nentries. If you're wondering, the "0" in the arguments
to kvm_getprocs(3) actually doesn't matter - KERN_PROC_ALL is an
operation that doesn't take an argument. There are other useful
operations which do take an argument, so see kvm_getprocs(3) for info
on those.</p>
<p>Now, you obviously want to go through the processes and do something
with them: you know how many processes there are and you have the
pointer to the first one, so a simple for loop with a counter will
do. Why don't we just print the binary name for every process?</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">nentries</span><span class="p">;</span><span class="w"> </span><span class="o">++</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">kinfo</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">p_comm</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
</pre></div>
<p>Now that was easy. There are a lot of fields in the kinfo_proc struct,
and there actually isn't a man page for them, since the full
definition is available in <sys/sysctl.h>. Look there
(/usr/include/sys/sysctl.h) for info on information you can get.</p>
<p>Almost forgot: this is supposed to be a valid program, so return
something:</p>
<div class="highlight"><pre><span></span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<h2>The final code</h2>
<div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><kvm.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><limits.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><sys/param.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><sys/sysctl.h></span><span class="cp"></span>
<span class="kt">int</span><span class="w"></span>
<span class="nf">main</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="n">errbuf</span><span class="p">[</span><span class="n">_POSIX2_LINE_MAX</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="n">kvm_t</span><span class="w"> </span><span class="o">*</span><span class="n">kernel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">kvm_openfiles</span><span class="p">(</span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="n">KVM_NO_FILES</span><span class="p">,</span><span class="w"> </span><span class="n">errbuf</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">nentries</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">kinfo_proc</span><span class="w"> </span><span class="o">*</span><span class="n">kinfo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">kvm_getprocs</span><span class="p">(</span><span class="n">kernel</span><span class="p">,</span><span class="w"> </span><span class="n">KERN_PROC_ALL</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="k">sizeof</span><span class="p">(</span><span class="k">struct</span><span class="w"> </span><span class="nc">kinfo_proc</span><span class="p">),</span><span class="w"> </span><span class="o">&</span><span class="n">nentries</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">nentries</span><span class="p">;</span><span class="w"> </span><span class="o">++</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">kinfo</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">p_comm</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Compile that with <code>cc -lkvm main.c</code> and run it to get some output
(hopefully).</p>
<h2>See Also</h2>
<p>This short tutorial is basically just a gateway into the hundreds of
possible things you can do with processes in OpenBSD. The
documentation in the man pages is excellent, so don't hesitate to use
apropos(1) in your search for more knowledge. Also, you can just go to
the man page for kvm(3) and just work your way through the "SEE ALSO"
sections, playing with functions you find as you go.</p>
<p>Have fun!</p>
Iodine is coolurn:https-kaashif-co-uk:-2015-04-19-iodine-is-cool2015-04-19T00:00:00Z2015-04-19T00:00:00ZKaashif Hymabaccus
<p>I know that sometimes, I've bene stuck in an airport or in a coffee
shop without internet. That's annoying in and of itself, but it's even
more annoying when there's a WiFi hotspot nearby, but it requires you
to pay £4/hour or something crazy like that.</p>
<p>You can still connect to the network, it's just that whatever URL you</p>
<!-- more -->
<p>visit in your browser, you get redirected to the "pay for internet"
page. Obviously, you can't SSH tunnel somewhere, or use an ordinary
HTTP proxy, those things are all blocked.</p>
<p>In desperation, I would ping google.com, since that's what everyone
does to check internet connectivity. There, I noticed something:</p>
<pre><code>$ ping google.com
PING google.com (216.58.210.46) 56(84) bytes of data.
</code></pre>
<p>How did it know the IP of google.com? It must have resolved it using a
DNS server, it's not like I have a big list of sites in my /etc/hosts.</p>
<p>That means I can request information from a server, and get back
information from on any server on the internet. All it would take is
someone running a DNS server (or similar) putting some data in the
records it sends back and I'll have a way of communicating to the
outside... without even logging into the hotspot.</p>
<h2>Iodine - how to install and set up</h2>
<p>I was ready to write a solution myself, but it already
existed. Iodine! Here's how to set it up.</p>
<p>First, set up an A record pointing to your public server and an NS
record pointing to that A record. Basically set it up like the
<a href="http://code.kryo.se/iodine/README.html">readme</a> tells you to.</p>
<p>Now, on the server pointed to by the A record (assuming Debian):</p>
<pre><code>$ sudo apt install iodine
</code></pre>
<p>Obviously that's not going to do anything. On Debian, there is a
config file at /etc/default/iodine. Fire up your editor and open it,
and make the relevant variables look like this:</p>
<pre><code>IODINED_ARGS="-c -d tap0 192.168.233.1/24 t1.mydomain.com"
IODINED_PASSWORD="mypassword"
</code></pre>
<p>Where t1.mydomain.com is an NS record, like in the README. Put in
something secure for the password. Now, all you need to do is start it
up:</p>
<pre><code>$ sudo systemctl start iodined
</code></pre>
<p>And the server should be running. Check <code>systemctl status iodined</code> for
any errors if you're really paranoid.</p>
<h2>The client</h2>
<p>On the client, setup is just as easy. To make your life a bit easier,
why not make iodine a service? Install iodine and edit
/etc/systemd/system/iodineclient.service and put in the following:</p>
<pre><code>[Unit]
Description=Iodine DNS proxy
[Service]
ExecStart=/usr/sbin/iodine -fP mypassword t1.mydomain.com
[Install]
WantedBy=multi-user.target
</code></pre>
<p>Now, just <code>systemctl start iodineclient</code> and wait a while for it to
connect and you'll be golden. You'll get an IP of 192.168.233.2, if
you used the same settings I did. The server it's running from should
be at 192.168.233.1, so you can run whatever web proxy you want there,
and you'll be able to access it so long as you can query DNS servers
for t1.mydomain.com.</p>
<h2>Setting up Squid</h2>
<p>The whole point of this was to access the web from access points where
all I have is DNS, so I need a web proxy. I chose Squid. Setting it up
is <em>really</em> simple. On the server:</p>
<pre><code>$ sudo apt install squid
$ echo 'http_access allow localnet' | sudo tee -a /etc/squid/squid.conf
$ sudo systemctl enable squid
$ sudo systemctl start squid
</code></pre>
<p>I know there are a lot of other things you can do with Squid, but with
this, you can just fire up iodine and use 192.168.233.1 on port 3128
as your HTTP proxy and it'll be like you paid that £400/second to
access the internet, but your wallet will stay full.</p>
Rainbow brackets in Emacsurn:https-kaashif-co-uk:-2015-04-11-rainbow-brackets-in-emacs2015-04-11T00:00:00Z2015-04-11T00:00:00ZKaashif Hymabaccus
<p>You know something that really annoys me? When I'm writing some
Racket, Clojure, or any other Lispy language, and my editor won't
cooperate. Emacs is far, far, better than most other editors for this
sort of thing, mostly due to paredit-mode and SLIME (and geiser-mode,
and clojure-mode, and evil-mode, and...), but there's still one
problem I hadn't solved until recently.</p>
<!-- more -->
<p>Matching up sets of parentheses.</p>
<p>I know what you're thinking, a quick 2 second Google search would have
told me to install rainbow-delimiters, and I did, with
<code>M-x package-install RET rainbow-delimiters RET</code>. Then I added it to
my .emacs file:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">require</span> <span class="ss">'rainbow-delimiters</span><span class="p">)</span>
<span class="p">(</span><span class="nv">global-rainbow-delimiters-mode</span><span class="p">)</span>
</pre></div>
<p>And that was that. Or was it? Take a look at this:</p>
<p><img src="/static/paren1.png" alt="" /></p>
<p>How can you tell the colours apart when they're so...gray?</p>
<p>How could I fix that? All the colours look the same! I don't want have
to set all the colours manually, I just want to programatically make
all of the colours brighter and less drab and indistinguishable from
each other. Is that so much to ask?</p>
<p>It turns out that it's not, and there's an easy way to do that, just
<code>M-x package-install RET cl-lib RET</code> (although you might already have
it) and add the following to your .emacs (after requiring
rainbow-delimiters):</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nb">require</span> <span class="ss">'cl-lib</span><span class="p">)</span>
<span class="p">(</span><span class="nb">require</span> <span class="ss">'color</span><span class="p">)</span>
<span class="p">(</span><span class="nv">cl-loop</span>
<span class="nv">for</span> <span class="nv">index</span> <span class="nv">from</span> <span class="mi">1</span> <span class="nv">to</span> <span class="nv">rainbow-delimiters-max-face-count</span>
<span class="nb">do</span>
<span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nv">face</span> <span class="p">(</span><span class="nb">intern</span> <span class="p">(</span><span class="nb">format</span> <span class="s">"rainbow-delimiters-depth-%d-face"</span> <span class="nv">index</span><span class="p">))))</span>
<span class="p">(</span><span class="nv">cl-callf</span> <span class="nv">color-saturate-name</span> <span class="p">(</span><span class="nv">face-foreground</span> <span class="nv">face</span><span class="p">)</span> <span class="mi">30</span><span class="p">)))</span>
</pre></div>
<p>That increases the saturation on all the colours to the maximum, so
they look nice and bright.</p>
<p><img src="/static/paren2.png" alt="" /></p>
<p>Much better.</p>
OpenShift vs Heroku: Haskellurn:https-kaashif-co-uk:-2015-03-30-openshift-vs-heroku-haskell2015-03-30T00:00:00Z2015-03-30T00:00:00ZKaashif Hymabaccus
<p>There are a few Platform as a Service (PaaS) services out there, and
the most famous is probably Heroku. I know I've had people come up to
me and suggest using Heroku for the next big thing they're
planning. There is a problem with Heroku: it's not free (libre). That
didn't stop me from at least trying it out to see what all the fuss
was about.</p>
<!-- more -->
<h2>Deploying a Haskell app on Heroku</h2>
<p>Obviously, it wasn't as easy as just <code>git push heroku master</code>, or this
post wouldn't exist. No, it was more complicated than that.</p>
<p>I couldn't just push my Happstack app (a toy app, now hosted at
<a href="http://quenya.kaashif.co.uk">http://quenya.kaashif.co.uk</a>) to Heroku, so I had to do a bit of
searching around. The best way to do this was to create an app with
the buildpack set to either
<a href="https://github.com/begriffs/heroku-buildpack-ghc">https://github.com/begriffs/heroku-buildpack-ghc</a> or
<a href="https://github.com/mietek/haskell-on-heroku">https://github.com/mietek/haskell-on-heroku</a>. The latter was
apparently faster, so I tried that out.</p>
<p>Setting the buildpack was as easy as:</p>
<div class="highlight"><pre><span></span>$ heroku buildpack:set https://github.com/mietek/haskell-on-heroku.git -a quenya
</pre></div>
<p>Apparently, it <em>was</em> now just as easy as pushing. So I tried:</p>
<div class="highlight"><pre><span></span>$ git push heroku master
</pre></div>
<p>And it failed, with the following errors:</p>
<pre><code>*** ERROR: Cannot build sandbox directory
*** ERROR: Failed to deploy app
*** ERROR: Deploying buildpack only
</code></pre>
<p>There was some other output, too, but those were the important
lines. This wasn't supposed to happen. After looking around on the
tutorial page, I see that this error occurs when you're trying to
build using a one-off dyno. Maybe I could still make it work!</p>
<div class="highlight"><pre><span></span>$ heroku run -s 1X build
</pre></div>
<p>And that failed, telling me private storage was expected (an S3
bucket). Well, I should've expected that, I suppose, considering the
errors I got when the push failed are only supposed to occur when
trying to use private storage.</p>
<p>Well anyway, let's try the other buildpack.</p>
<h2>Trying the other buildpack</h2>
<p>I set the buildpack to the first one I looked at:</p>
<div class="highlight"><pre><span></span>$ heroku buildpack:set https://github.com/begriffs/heroku-buildpack-ghc -a quenya
</pre></div>
<p>That worked fine. Then, I pushed.</p>
<div class="highlight"><pre><span></span>$ git push heroku master
</pre></div>
<p>It seemed to be going well, GHC was downloaded, cabal-install was
downloaded, and it was about to run when suddenly:</p>
<pre><code>cabal: error while loading shared libraries: libgmp.so.3: cannot open shared object file: No such file or directory
</code></pre>
<p>And it was going so well! If I got this error on my machine, I'd just
do some dirty symlink trick to make cabal think it had the right
library, but I couldn't do that on Heroku, since I didn't have
permission to do anything.</p>
<p>Now that I think about it, this means that the haskell-on-heroku
buildpack would have probably failed too, even if I set up an S3
bucket for it.</p>
<p>Anyway, I'd had enough of Heroku and decided to try out OpenShift.</p>
<h2>Getting started with OpenShift</h2>
<p>I visited <a href="https://openshift.redhat.com">https://openshift.redhat.com</a> and signed up for a free
account. It was simple enough to make an application, I just had to
select a "cartridge" (basically a set of preinstalled libraries and
programs), add my SSH key, and pick a name.</p>
<p>There was already a Happstack cartridge which came with
happstack-server, blaze-html, and all the libraries I needed, bar a
few. Before pushing my app, I decided to install <code>rhc</code>, the command
line tool for OpenShift, like <code>heroku</code>.</p>
<p>It has some cool features, like <code>rhc port-forward</code>, which sets up a
tunnel so you can access the internal services (Redis, PostgreSQL etc)
from your own computer, on localhost. For example, if you run <code>rhc
port-forward</code>, you could access PostgreSQL on 127.0.0.1:5432. Cool,
right? I suppose Heroku must have something similar, but this isn't
supposed to be a balanced blog post.</p>
<p>Anyway, I sshed into the application server to check things out, and
it looked like a normal RHEL server, without anything weird or
virtual-looking (although it probably is virtual). I had a home
directory in /var/lib/openshift/, alongside a number of
others. <code>cabal</code> was actually installed, unlike on Heroku, so I didn't
need any "buildpacks": the build would take place exactly as it would
do on my computer.</p>
<p>All I had to do was <code>git push <really long uri></code> and it worked. All
that happened was that <code>cabal run <host> <port></code> was run (or something
to that effect), and it started the app. Simple, right?</p>
<h2>Moving a static site to OpenShift</h2>
<p>This is easy, although there isn't a "static site" cartridge. All you
have to do it pick a PHP cartridge, and Apache just serves the root
directory of the repository (with mod_php enabled). So I just
generated my blog, pushed to a new PHP app, and Apache served my
site. Easy.</p>
<h2>Conclusions</h2>
<p>OpenShift is good, use it instead of Heroku. I can't speak for the
paid options, but it's a lot easier to deploy Haskell on OpenShift.</p>
<p>Note: I have almost no experience in using either, except for what
I've written in this article.</p>
Quaternions, spinors and rotationsurn:https-kaashif-co-uk:-2014-12-26-quaternions-spinors-and-rotations2014-12-26T00:00:00Z2014-12-26T00:00:00ZKaashif Hymabaccus
<p>Earlier, I was trying to find something I could talk about at my
school's maths society. It had to be something exciting, useful, or at
least beautiful in some way. I really wanted to do something on
quaternions and vectors, because it seemed fun. The problem came when
I realised I had to do something more substantial than stand there and
explain something that boring. Then I saw this quote:</p>
<!-- more -->
<blockquote><p>No one fully understands spinors. Their algebra is formally
understood but their general significance is mysterious. In some
sense they describe the "square root" of geometry and, just as
understanding the square root of -1 took centuries, the same might
be true of spinors.
<cite>Michael Atiyah</cite></p></blockquote>
<p>Wow, that sounds cool. Maybe I'll be the one to explain spinors to
everyone.</p>
<h2>Some basics</h2>
<p>Before delving into the nitty gritty of what a spinor actually is,
there are a few things that need to be understood. This is a matrix:</p>
<p>$$
M =
\begin{bmatrix}
a & b \
c & d
\end{bmatrix}
$$</p>
<p>This is that same matrix, transposed:</p>
<p>$$
M<sup>T</sup> =
\begin{bmatrix}
a & c \
b & d
\end{bmatrix}
$$</p>
<p>This is the conjugate transpose, usually denoted by an asterisk or
dagger:</p>
<p>$$
M^{\dagger} =
\begin{bmatrix}
a^\ast & c^\ast \
b^\ast & d^\ast
\end{bmatrix}
$$</p>
<p>This matrix $M$ is a unitary matrix, if and only if:</p>
<ul>
<li>$M^{\dagger}M = MM^{\dagger} = I$</li>
<li>Multiplying by $M$ preserves the inner product of two complex
vectors. i.e. $Mx \cdot My = x \cdot y$</li>
<li>$|\det(M)| = 1$</li>
</ul>
<p>In case you're wondering, the inner product of two vectors in
$\mathbb{C}^2$ is defined as:</p>
<p>$$(a,b) \cdot (c,d) = ac^\ast + bd^\ast$$</p>
<p>This is known as the Hermitian inner product.</p>
<p>There are a few other conditions for a matrix to be unitary, but
they'd require a longer explanation. In any case, the most important
parts are there.</p>
<p>Importantly, all of these properties mean that unitary $n \times n$
matrices form a group, $U(n)$ under matrix multiplication.</p>
<p>If we take all of the elements of $U(n)$ with determinant 1, we get
a special unitary group, $SU(n)$.</p>
<h2>Special cases of special groups</h2>
<p>There are loads of things you can do with these groups (mostly to do
with quantum mechanics), but something that really caught my eye was
the fact that there exists an injective mapping from $SU(2)$ to
$\mathbb{H}$. That is interesting. Obviously, the mapping can't be
a bijection, since not all quaternions correspond to unitary
matrices. Maybe that's not obvious yet.</p>
<p>To see how $\mathbb{H}$ relates to $2 \times 2$ matrices, let's look
at the mapping from from $\mathbb{C}$. Complex numbers in the form
$a+bi$ can be represented by a real matrix in the form:</p>
<p>$$
\begin{bmatrix}
a & -b \
b & a
\end{bmatrix}
$$</p>
<p>Using that, it seems natural to say that $\mathbb{H}$ is isomorphic to
$2 \times 2$ complex matrices in the form:</p>
<p>$$
X =
\begin{bmatrix}
\alpha & \beta \
-\beta^\ast & \alpha^\ast
\end{bmatrix}
$$</p>
<p>Where $X = a + b\mathbf{i} + c\mathbf{j} + d\mathbf{k}$, $\alpha =
a+bi$ and $\beta = c + di$.</p>
<p>If we remember the handy quaternion multiplication table, use the
right hand rule for vector multiplication, or use
<a href="https://upload.wikimedia.org/wikipedia/commons/c/c0/Cayley_graph_Q8.svg">this</a>
image (very intuitive), we remember that:</p>
<p>$$
\mathbf{i}\mathbf{j} = \mathbf{k},\;\;
\mathbf{j}\mathbf{k} = \mathbf{i},\;\;
\mathbf{k}\mathbf{i} = \mathbf{j}
$$</p>
<p>The only way you can get a matrix in the form above and stick to these
rules is to define the unit quaternions as:</p>
<p>$$
\mathbf{i} =
\begin{bmatrix}
i & 0 \
0 & -i
\end{bmatrix}!,\;\;
\mathbf{j} =
\begin{bmatrix}
0 & 1 \
-1 & 0
\end{bmatrix}!,\;\;
\mathbf{k} =
\begin{bmatrix}
0 & i \
i & 0
\end{bmatrix}
$$</p>
<p>Obviously, the only quaternions that will end up with determinant 1
will be the ones of norm 1 - the unit quaternions. Thus, the mapping
is obviously injective, since not all quaternions have this.</p>
<p>The norm of a quaternion is simple to work out, since (intuitively)
norm is a generalised idea of the length of a vector. For example, for
the quaternion $X$ I mentioned earlier, its norm would be
$\sqrt{a<sup>2</sup>+b<sup>2</sup>+c<sup>2</sup>+d<sup>2</sup>}$.</p>
<p>Just a note: if you want to multiply quaternions $X$ and $Y$, it's
easier to treat their real parts and imaginary parts separately. If we
say $X = (x,\vec{x})$ and $Y=(y,\vec{y})$:</p>
<p>$$
XY = (xy - \vec{x} \cdot \vec{y}) + (x\vec{y} + y\vec{x} + \vec{x} \times \vec{y})
$$</p>
<p>This is a lot easier because $\vec{x}$ and $\vec{y}$ are just ordinary
vectors in $\mathbb{R}^3$, which are very easy to deal with.</p>
<h2>What use is this?</h2>
<p>That seems really useless, but remember that quaternions are widely
used in computer graphics (among other places) to represent rotations
in three dimensions, which means $SU(2)$ is also isomorphic to
$SO(3)$. Not only that, there are dozens of uses in quantum mechanics
to do with spin, QCD and, you guessed it, spinors.</p>
<p>But what is a spinor? You could find out on the Wikipedia page, but
chances are you won't understand any of it without either spending a
long time trawling through a massive tree of articles or having
someone explain it.</p>
<p>I would explain it now, but I have other things to do.</p>
<h2>That's it?! But I still have no idea what a spinor is!</h2>
<p>Well then, I guess you should come to Maths Society on Thursday the
5th of February!</p>
<p>While I definitely won't have anything finished before the day, you
can check out any work in progress at
<a href="http://kaashif.co.uk/code/spinor.git/">http://kaashif.co.uk/code/spinor.git/</a>.</p>
ShareLaTeX on OpenBSDurn:https-kaashif-co-uk:-2014-12-06-sharelatex-on-openbsd2014-12-06T00:00:00Z2014-12-06T00:00:00ZKaashif Hymabaccus
<p>The other day, I was trying to access <a href="http://sharelatex.com">http://sharelatex.com</a> at
school, and it didn't really work, probably due to a combination of
Internet Explorer and possibly an overzealous filter that could have
been blocking something. That's what I thought, anyway, until I tried
it on Chrome and it still didn't work. Odd. The best solution was
obviously to set up my own ShareLaTeX instance on my server.</p>
<!-- more -->
<h2>Dependencies</h2>
<p>On the
<a href="https://github.com/sharelatex/sharelatex/wiki/Dependencies">ShareLaTeX wiki page</a>
about dependencies, I saw that I needed Node.js, Grunt, Redis,
MongoDB, Aspell, and TeXLive. All of these are packaged in OpenBSD and
they can all be easily installed:</p>
<div class="highlight"><pre><span></span>$ sudo pkg_add node redis mongodb texlive_texmf-full aspell latexmk
</pre></div>
<p>After that, I installed Grunt using npm:</p>
<div class="highlight"><pre><span></span>$ sudo npm install -g grunt-cli
</pre></div>
<p>That's that. I read through the instructions and they said I needed to
configure MongoDB, but that's not actually necessary.</p>
<h2>The hard part</h2>
<p>I chose to install into /var/www, as per the recommendations on the
wiki, so I cloned the repo into /var/www/sharelatex and ran the
commands. It's important to note that Grunt expects the make binary in
the PATH to be GNU make, and I couldn't be bothered to find some way
to change this expectation, so I moved BSD make and symlinked GNU make
in its place:</p>
<div class="highlight"><pre><span></span>$ mv /usr/bin/make /usr/bin/bmake
$ ln -s /usr/local/bin/gmake /usr/bin/make
</pre></div>
<p>Now I was ready to install billions of npm packages and let Grunt set
up trillions of config files:</p>
<div class="highlight"><pre><span></span>$ git clone -b release <span class="se">\</span>
https://github.com/sharelatex/sharelatex.git <span class="se">\</span>
/var/www/sharelatex
$ <span class="nb">cd</span> /var/www/sharelatex
$ npm install
$ grunt install
</pre></div>
<p>That shouldn't show too many errors. Now, I followed the rest of the
instructions on the wiki page I linked earlier. In case you're too
lazy to go there, they're reproduced below (and edited for BSD):</p>
<p>Make a sharelatex user, and chown all files to it:</p>
<div class="highlight"><pre><span></span>$ useradd -b /var/www/sharelatex -G sharelatex sharelatex
$ chown -R sharelatex:sharelatex /var/www/sharelatex
</pre></div>
<p>Move the config files to a better place:</p>
<div class="highlight"><pre><span></span>$ mkdir /etc/sharelatex
$ mv /var/www/sharelatex/config/settings.development.coffee <span class="se">\</span>
/etc/sharelatex/settings.coffee
</pre></div>
<p>Edit that config file and make sure the dir variables read as follows:</p>
<pre><code>DATA_DIR = '/var/lib/sharelatex/data'
TMP_DIR = '/var/lib/sharelatex/tmp'
</code></pre>
<p>Make all of the directories:</p>
<div class="highlight"><pre><span></span>$ mkdir -p /var/lib/sharelatex/data/<span class="o">{</span>user_files,compiles,cache<span class="o">}</span>
$ mkdir -p /var/lib/sharelatex/tmp/<span class="o">{</span>uploads,dumpFolder<span class="o">}</span>
$ chown -R sharelatex:sharelatex /var/lib/sharelatex
</pre></div>
<p>That's it, there are only a few things left to do.</p>
<h2>A problem and a fix</h2>
<p>I tried to run ShareLaTeX, and it wouldn't work. It seems like
there's a well-known bug in Node.js (or something like that) that
causes it to fail unless you do the following:</p>
<div class="highlight"><pre><span></span>$ <span class="nb">cd</span> /var/www/sharelatex
$ rm -rf web/node_modules/bcrypt
$ npm install
</pre></div>
<p>Apparently, it's something to do with dependencies, but it doesn't
matter, I just ran the above command and everything ended up working.</p>
<h2>Init script</h2>
<p>Obviously, the Upstart script supplied is impossible to use on
OpenBSD, but it's not too hard to whip up an rc.d script. Here's the
one I use:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
<span class="c1"># /etc/rc.d/sharelatex</span>
<span class="nv">daemon</span><span class="o">=</span>/usr/bin/tmux
<span class="nv">daemon_user</span><span class="o">=</span>sharelatex
<span class="nv">daemon_flags</span><span class="o">=</span><span class="s2">"new-session -s sharelatex -d 'grunt run'"</span>
. /etc/rc.d/rc.subr
rc_stop<span class="o">(){</span>
<span class="si">${</span><span class="nv">rcexec</span><span class="si">}</span> <span class="s2">"sudo -u sharelatex </span><span class="si">${</span><span class="nv">daemon</span><span class="si">}</span><span class="s2"> kill-server"</span>
<span class="o">}</span>
rc_cmd <span class="nv">$1</span>
</pre></div>
<p>Obviously, it's a bit primitive, since it just runs in tmux with no
logging and is killed by killing tmux, but it does work, and you can
start and stop it.</p>
<p>The last thing I needed to do is to add everything to
/etc/rc.conf.local:</p>
<div class="highlight"><pre><span></span><span class="nv">pkg_scripts</span><span class="o">=</span>redis mongod sharelatex
</pre></div>
<p>And start it up with:</p>
<div class="highlight"><pre><span></span>$ /etc/rc.d/sharelatex startbash
</pre></div>
<p>Now, it should work. If you've been following along, browse to port
3000 on your server to check it out. I advise setting up Apache or
Nginx (or some other web server) as a reverse proxy and using TLS to
access it. Configuring web servers isn't within the scope of this
post, though.</p>
<p>Check out <a href="http://sharelatex.com">http://sharelatex.com</a>, though, it's really cool. Maybe
you'll want to set it up yourself, too (if you haven't already)!</p>
Switching to Mercurialurn:https-kaashif-co-uk:-2014-11-15-switching-to-mercurial2014-11-15T00:00:00Z2014-11-15T00:00:00ZKaashif Hymabaccus
<p>UPDATE: I now use Mercurial on the client side (i.e. everything I do
locally involves <code>hg</code>) and Git on the server-side. It just makes it
easier to mirror to Gitorious, GitLab, etc. You can view all the
repos at <a href="http://kaashif.co.uk/code">http://kaashif.co.uk/code</a>.</p>
<p>Earlier in the year, I was getting curious about version control</p>
<!-- more -->
<p>systems other than Git. Git was my first VCS, and the one I was most
familiar with, but I didn't want to be...locked in (cue
gasps). Obviously, there isn't any potential for actual lock-in, but
there is a Git (and even more worryingly, a GitHub) monoculture
developing that I don't want to be part of.</p>
<p>I could lie and say that it was for practical reasons, but at that
time, I was concerned with the fact I was becoming mainstream in my
choice of VCS. The first alternative I looked to was Darcs.</p>
<h2>Migrating to Darcs</h2>
<p>When I switched to Darcs, the first thing I had to do was convert all
of my repos from Git. This posed a bit of a problem, as 99% of guides
are for converting <em>from</em> Darcs <em>to</em> Git, which is obviously not what
I want. In the end, I just settled for:</p>
<div class="highlight"><pre><span></span>$ <span class="k">for</span> repo <span class="k">in</span> *<span class="p">;</span> <span class="k">do</span>
> <span class="o">(</span><span class="nb">cd</span> <span class="nv">$repo</span><span class="p">;</span> rm -rf .git<span class="p">;</span> darcs initialize<span class="o">)</span><span class="p">;</span>
> <span class="k">done</span>
</pre></div>
<p>That's not ideal, because I'm essentially nuking all of my history for
no reason, but as a lone developer, it doesn't matter. So that was
fine. I guess I can't hold it against them that no-one wants to
migrate to Darcs. Well actually, I can, but I chose not to.</p>
<p>Eventually, after setting up a darcs account and pushing all my repos
to my server, I decided I wanted some sort of cgit-esque web
interface. I was pleasantly surprised to discover that you don't
necessarily need anything special to browse Darcs repos, you <em>could</em>
just use a normal web server's directory listing capabilities. I did
that for a while, but it was pretty bad. In the absence of any bundled
solution, I took to the web and found
<a href="http://blitiri.com.ar/p/darcsweb/">darcsweb</a>, which wasn't too
bad. Sure the last release was in 2008, but that means it's
feature-complete and <em>mature</em>.</p>
<h2>Darcs: the good, the bad and the exponentially complex</h2>
<p>Darcs is good. It's written in Haskell. Everyone loves Haskell. You
know who loves Haskell the most? The guys who wrote the Glasgow
Haskell Compiler. But, uh, they don't use Darcs, they use Git. They
<em>used</em> to use Darcs, but it's apparently too slow for them.</p>
<p>I didn't think that would affect me: GHC is an absolutely massive project.</p>
<p>I was wrong.</p>
<p>After trying to commit the entirety of the OpenBSD source tree (a VCS
should be able to handle this) and doing some merge-fu, it was time to
go back to Git. Evidently, Darcs' merging algorithms and patch algebra
needed some work.</p>
<h2>Why switch?</h2>
<p>After the recent news that the Go team decided to switch to Git, I
decided once again to switch away from Git. This time, I wouldn't come
back. The obvious choice was now Mercurial: it is widely touted as
being easy to use, competetive with Git in terms of speed, and
overall, it's supposed to be actually usable. Also, Mozilla use
Mercurial for their massive projects, so at least <em>someone</em> other than
the creators use Mercurial.</p>
<p>You might be wondering why I told that story about Darcs. Well, it's
mostly just so that you have more than 2 data points to compare. Let's
get comparing, then.</p>
<h2>Mercurial is really, really easy</h2>
<p>Git has a propensity for being really hard to use and cryptic. There's
the classic example of reverting a file. How do you do that?</p>
<p>(Git)</p>
<div class="highlight"><pre><span></span>$ git checkout file
</pre></div>
<p>(Mercurial)</p>
<div class="highlight"><pre><span></span>$ hg revert file
</pre></div>
<p>Well I suppose you'd eventually remember that checkout means
revert... not that bad, right? Well, putting aside the fact that there
exists a <code>git revert</code> command that does something completely
different, sure. It's not that bad.</p>
<p>Let's look at reverting to the last commit.</p>
<p>(Git)</p>
<div class="highlight"><pre><span></span>$ git reset --hard c0mm1th45h
</pre></div>
<p>(Mercurial)</p>
<div class="highlight"><pre><span></span>$ hg rollback
</pre></div>
<p>I know, I know, I've cherry-picked examples that paint Git in a bad
light. Whatever fruit I've been picking, the fact is that Mercurial is
a lot easier to use, and a lot harder to mess up than Git is.</p>
<p>Also, it's shockingly easy to convert a Git repo (or any other repo)
to Mercurial. Let's say there's a repo in "myrepo":</p>
<div class="highlight"><pre><span></span>$ hg convert myrepo
</pre></div>
<p>Then there'll be a converted Mercurial repo in myrepo-hg. That's
fantastic!</p>
<p>So, after some quick Googling and playing around in test repos, I
converted all of my Git repos and pushed everything to BitBucket. But
I needed to set Mercurial up on my server.</p>
<h2>hgweb</h2>
<p>Hgweb comes with Mercurial and is, as the name sort of suggests, the
gitweb of Mercurial. It was a breeze to set up, you just fill in the
path to the config in the script itself, set it up as a CGI script,
and write a config file. Mine looks sort of like this:</p>
<pre><code>[paths]
blog = /home/hg/blog
tau = /home/hg/tau
</code></pre>
<p>And a load of other repos. You can take a look at <a href="http://hg.kaashif.co.uk">http://hg.kaashif.co.uk</a>.</p>
<p>Wow, that was barely a paragraph. Mercurial really is easy!</p>
<h2>Committing to BitBucket from a hook</h2>
<p>I really like it when I can self-host 90% of everything and just use
GitHub, Gitorious and now BitBucket as a glorified backup. I already
have an online repo browser and SSH server, all I need is for my
server to push to BitBucket whenever there are incoming changes.</p>
<p>This was surprisingly easy. All I had to do was edit the ~/.hgrc of
the hg user and add a global hook and path to a script:</p>
<pre><code>[hooks]
incoming = /usr/local/bin/push_to_bitbucket
</code></pre>
<p>Now, when there are <em>incoming</em> changes, they'll be received then
pushed to BitBucket. Well, they will be after I write the script:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
hg push ssh://hg@bitbucket.org/kaashif/<span class="k">$(</span>basename <span class="k">$(</span><span class="nb">pwd</span><span class="k">))</span>
</pre></div>
<p>Within 30 mins, I had everything set up. A lot quicker than I was able
to do it with Git, although I suppose I now have the benefit of
hindsight and experience.</p>
<p>I've even heard that with hg-git, I <em>never</em> have to use Git, even when
upstream is Git-only. Maybe this time, my migration will be permanent.</p>
Pkgsrc on Slackwareurn:https-kaashif-co-uk:-2014-10-11-pkgsrc-on-slackware2014-10-11T00:00:00Z2014-10-11T00:00:00ZKaashif Hymabaccus
<p>Back in the day, I used to use Slackware. It was the best distro
around and all the cool kids used it. Nowadays, it's rather different:
a much lower proportion of people use Slackware. Despite the efforts
of Eric and Patrick (and whoever else), Slackware isn't really all
that popular. It's still a solid distro, though. There is one problem</p>
<!-- more -->
<p>that seems to annoy people.</p>
<p>No dependency management.</p>
<h2>But isn't that a selling point of Slackware?</h2>
<p>Maybe it is, but it causes a lot more problems than it solves. If
you're going for a system where not a single thing was installed that
you didn't want and where you've built all the packages and kernels
from source, then you're a lot better off with Gentoo (which has a
much cooler selling point: USE flags).</p>
<p>Using binary packages without dependency management just causes
problems. If you try to use sbopkg (the SlackBuild package manager),
you quickly find that having to build from source and not being able
to manage dependencies is the worst of every single world: installing
anything takes ages and the builds will all probably fail halfway
through.</p>
<p>Of course, there is a "solution" in place: queues. You're supposed to
look up the whole dependency tree beforehand and queue up all of the
packages, one by one. Pretty tedious.</p>
<p>There are alternatives, like slapt-get, but those require you to
surrender all control to the package manager, at which point you might
as well go back to Ubuntu. So we want dependency management and
control. Where can we turn? Of course, NetBSD's pkgsrc.</p>
<h2>Getting started</h2>
<p>You might think that since it's part of the NetBSD project, pkgsrc can
only run on NetBSD. That's not true, it can run on any vaguely POSIXy
OS. To get started on Slackware, you need to make sure you have a few
of the sets. I'm not sure how many you actually need, but this is what
I got it down to:</p>
<ul>
<li>a</li>
<li>ap</li>
<li>d</li>
<li>l</li>
<li>n</li>
</ul>
<p>The need for "a" is obvious: that's the base. You need "d" and "l" for
compiling, which we'll be doing a lot of. "n" for downloading
pkgsrc. "ap" for, well, being able to do things while you build
everything (which could take a while). Eventually, you could uninstall
"ap" and "n" and replace everything with programs from the pkgsrc
tree. You can't do that yet, though.</p>
<p>Next, get and extract the pkgsrc tree:</p>
<div class="highlight"><pre><span></span><span class="c1"># cd /usr</span>
<span class="c1"># wget ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz</span>
<span class="c1"># tar xzvf pkgsrc.tar.gz</span>
</pre></div>
<p>Now, there is a ports-looking sort of thing under
/usr/pkgsrc. Nifty. But you need NetBSD's make to use it. Luckily,
there's a bootstrap script to get you started.</p>
<div class="highlight"><pre><span></span><span class="c1"># cd /usr/pkgsrc/bootstrap</span>
<span class="c1"># ./bootstrap</span>
</pre></div>
<p>Something should happen and at the end, there should be some binaries
in /usr/pkg/bin and /usr/pkg/sbin. That's where all of the pkgsrc
installed stuff will go, so you might want to add the following to
your shell's profile:</p>
<div class="highlight"><pre><span></span><span class="nb">export</span> <span class="nv">PATH</span><span class="o">=</span><span class="nv">$PATH</span>:/usr/pkg/bin:/usr/pkg/sbin
</pre></div>
<p>Additionally, you might want man pages.</p>
<div class="highlight"><pre><span></span><span class="nb">export</span> <span class="nv">MANPATH</span><span class="o">=</span><span class="nv">$MANPATH</span>:/usr/pkg/man
</pre></div>
<p>Now everything's set up to start building. You might not have an X
server yet and even if you do, you'll want to replace it with
pkgsrc's. Installing a port is easy:</p>
<div class="highlight"><pre><span></span>$ <span class="nb">cd</span> /usr/pkgsrc/x11/modular-xorg-server
$ sudo bmake install
</pre></div>
<p>On my old laptop, this took several hours. If you have a more recent
machine, it shouldn't take too long.</p>
<p>While building, you might think "Hey, why's it installing Perl, I
already have that!". That's a valid question, but pkgsrc doesn't
detect dependencies by looking in /usr/bin/, it searches its own
package database, which is completely empty.</p>
<p>This means that you'll end up building Perl, Python and dozens of
libraries you already have. After pkgsrc is done, you might as well
uninstall all of the Slackware packages you don't need. Go to
/var/log/packages and and <code>removepkg</code> everything you don't want.</p>
<h2>What now?</h2>
<p>You can search the ports tree with /usr/pkgsrc/pkglocate for all your
favourite software and install it. There's more software in pkgsrc
than there is in the default Slackware repos and it's a lot more
convenient than sbopkg. For more info:
<a href="http://www.netbsd.org/docs/pkgsrc/">http://www.netbsd.org/docs/pkgsrc/</a>.</p>
<p>Disclaimer: I do not use pkgsrc on Slackware on a day-to-day basis, so
don't blame me for your problems. Send something in to the pkgsrc
mailing list or some NetBSD IRC channel instead.</p>
NetBSD "Review"urn:https-kaashif-co-uk:-2014-10-07-netbsd-review2014-10-07T00:00:00Z2014-10-07T00:00:00ZKaashif Hymabaccus
<p>Earlier today, I was discussing operating systems and came onto the
subject of ease of installation. Which OS had the easiest installer?
The obvious answers would tend towards OSs with GUI installers, but is
that really <em>easy</em>? Sure, it could be familiar, but there's a lot more
you have to do with GUI installers compared to, say, OpenBSD's</p>
<!-- more -->
<p>installer, where you can just mash enter (other than typing the root
password) and get a usable system. But then, my attention was drawn to
the text-based faux-GUI installers of FreeBSD and NetBSD. Were they
any good? I had no idea.</p>
<p>Essentially, what started out as idle thinking over whether NetBSD was
any good (I already tried out FreeBSD and indeed used it a lot for
servers a while back) turned into this "review" of NetBSD. I'm careful
with that word, because that implies my opinion is to be trusted when
in fact, I don't know all that much about OS design.</p>
<h2>The machine</h2>
<p>I'll be installing NetBSD on (not a ThinkPad this time) an old Dell
Studio 15 laptop with a Core 2 Duo, ATI Radeon graphics and a
reasonably OK 1200x800 screen. OK, that's not great, but I've used a
lot worse - this is at the higher end.</p>
<h2>Getting installation media</h2>
<p>This bit was pretty easy. I just searched "NetBSD", clicked on the
first link that mentioned mirrors, found one close to me, and
downloaded an ISO, which I burnt to a disk. Just for reference, the
location of the ISO I downloaded was
<a href="ftp://ftp2.fr.netbsd.org/pub/NetBSD/NetBSD-6.1.5/amd64/installation/boot.iso">ftp://ftp2.fr.netbsd.org/pub/NetBSD/NetBSD-6.1.5/amd64/installation/boot.iso</a>,
which is a link to the amd64 image from a French mirror.</p>
<p>While some people might think burning to a disk is anachronistic, I
like having a little memento of my first NetBSD installation. Maybe
I'll even buy a set of NetBSD CDs if I end up using NetBSD enough.</p>
<h2>Installing</h2>
<p>Booting from the CD went fine. There were no surprises when it came to
picking a language, keyboard layout and choosing what sort of
installation (installation without X, full install etc). I did run
into a bit of a problem when it came to partitioning the disk,
though.</p>
<p>The installer prompts you whether you want to use the whole disk
(yes), whether you want to overwrite the partitions that already exist
(yes) and whether you want to look at the finished partition table
(yes).</p>
<p>It was all well and good up to this point, where I saw that NTFS and
ext2 partitions were still present on my disk, and that the NetBSD
part of my disk had not used the whole disk: only 300GB out of the
500GB. There wasn't any obvious "delete partition" button, so I
changed the type of the partition to "unused", which seemed to delete
the partitions. The NetBSD part was still a bit undersized, but it
really didn't matter: I was "warned" that I needed at least a few GB
of space to install NetBSD at the start.</p>
<p>I suppose I could live with a bit of unused space - I probably
wouldn't use it anyway.</p>
<h2>Getting the sets</h2>
<p>For the uninitiated, installation sets are the actual stuff that is
extracted and installed onto the hard disk: the stuff on the CD is
just the installer, a way of getting the network set up to download
the sets and install them.</p>
<p>There was a bit of a problem, however: iwn wifi adapters don't tend to
work out of the box with completely free OSs.</p>
<p>Surprisingly, when I selected DHCP to automagically set up my network,
something seemed to be happening: I got an IP from somewhere, a
gateway, a default DNS server... this must have been from someone's
passwordless network down the road. I tried to change the settings to
those of my network, but I couldn't find where to enter the SSID or
WPA key. I could have looked harder, but I had a Ethernet cable so I
just used that.</p>
<p>My troubles didn't end there, however. The installer pinged the router
to make sure it was working. It was. Reassured, I tried to install the
sets from FTP using the ftp2.fr.netbsd.org mirror. After a few minutes
of waiting on "Connecting to ftp2.fr.netbsd.org", it told me it
couldn't connect. Maybe they suddenly went down? It's possible. Next,
I tried ftp.netbsd.org, the master site. There's no way that could be
down, but I was told that I couldn't connect.</p>
<p>The router was fine, the firewall was fine, but no internet on my
NetBSD installer, despite it working on my laptop an inch away. Since
I could still ping things on the LAN from the installer, I assumed it
was some network config mistake and that I'd better just download the
sets to my HTTP server and point the installer there, <em>inside</em> the
LAN.</p>
<p>That seemed to work fine and I was able to get to the "Congratulations
... you have installed NetBSD ... reboot". So I did.</p>
<h2>The first boot</h2>
<p>While rebooting, I decided to read the massive warnings that appeared,
warning me to change some rc.conf variables or not be able to boot to
multiuser. Sure enough, I rebooted and it wouldn't go into
multi-user. My bad, I suppose. I remounted the root read-write and
edited rc.conf, changing "rc_configured=NO" to "YES". Why that is
needed is beyond me, it seems to serve no purpose. Maybe it does, but
that shouldn't be my problem.</p>
<p>Now I had to add a user (why wasn't this done already, by the
installer?) and configure sudo access. Wait, no, sudo isn't installed!
So Lua is in the kernel, but sudo isn't in base. Great
priorities. Maybe there's an actual reason for this, but again, I
don't really care.</p>
<p>After doing all of that, I do end up with a mildly usable system with
vi, su, csh... Who am I kidding, an OS isn't even almost usable unless
you can get onto the internet, so let's do that next.</p>
<h2>Networking</h2>
<p>OpenBSD uses /etc/hostname.if files to manage configs for each
interface. Debian uses /etc/networking/interfaces. Everyone seems to
use something different, and I was right to suspect NetBSD was not
unique in its uniqueness. This was something that couldn't be fumbled
through by trial and error, I had to look it up on the internet, using
the (actually quite good) NetBSD guide
http://www.netbsd.org/docs/guide/en/chap-net-practice.html .</p>
<p>So it turns out they use /etc/ifconfig.xxx files, which is actually a
whole lot more sensible than what OpenBSD calls them. The
configuration syntax is 99% the same as what OpenBSD uses, so I can't
fault it there. This is, of course, because it's essentially a script
that runs ifconfig for each line.</p>
<p>After consulting
<a href="http://wiki.netbsd.org/tutorials/how_to_use_wpa_supplicant/">this page</a>
and finding out that NetBSD uses wpa_supplicant, I worked out how to
set it up from the examples and now I could start the network!</p>
<pre><code># /etc/rc.d/network restart
Stopping network.
...
add net default: gateway 192.168.0.1: Network is unreachable
</code></pre>
<p>Uh... I suppose I have to add an /etc/ifconfig.iwn0 that reads "up",
just to make sure the OS knows I want iwn0 up. No that still doesn't
work. Maybe if I reboot, it'll "just work".</p>
<h2>The second boot</h2>
<p>After the second boot, everything works as I'd expect. Now networking
works, I have a user (kaashif, obviously). All I need to do is to get
package management working and I'll be able to get everything exactly
the same as I have it on my current box. After consulting the FTP
mirror to find the right package path (/pub/pkgsrc/something/version/
or something similar), I looked
<a href="http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/doc/pkgsrc.html#using-pkg">here</a>
only to discover that it works almost the same as it does on OpenBSD,
the environment variable even has the same name: PKG_PATH. So I set
that and...</p>
<pre><code># pkg_add sudo
</code></pre>
<p>Wow, everything seems to work! Let's try X:</p>
<pre><code>$ startx
</code></pre>
<p>That works and starts up twm. From here on out, everything would be
that same as on any Unix-like OS.</p>
<h2>Conclusions</h2>
<p>Quite a bit more hassle than OpenBSD, I think, although it could just
be that I'm more used to OpenBSD. There is no doubt, however, that
OpenBSD's installer lets you get a system up and running
<em>faster</em>. That's not too useful in and of itself, since I don't know
how good NetBSD as a day-to-day OS and I don't plan to use it.</p>
<p>I like OpenBSD's WPA implementation more, there are fewer programs to
worry about and fewer knobs to fiddle with in rc.conf (none,
actually). Also, I prefer OpenSMTPD to Postfix. And a recent version
of pf to whatever NetBSD comes with now (the configs are
ever-so-slightly different).</p>
<p>These are all tiny complaints, though. With more usage, I'm sure I'd
come to see that NetBSD is at least usable.</p>
Learning Racketurn:https-kaashif-co-uk:-2014-09-25-learning-racket2014-09-25T00:00:00Z2014-09-25T00:00:00ZKaashif Hymabaccus
<p>I've heard a lot of things about Racket (well really, things about
many different Lisp dialects), and most of them were good. Recently, I
decided to try to decide between Haskell and a Lisp once and for
all. I wanted to go for a Lisp-1, since they keep functions and values
in the same namespace, which is how it <em>should</em> be. Eventually, after</p>
<!-- more -->
<p>trying out Guile, Chicken Scheme, and a few others, I settled on
Racket.</p>
<h2>Setting Racket up</h2>
<p>This was pretty easy. Unlike with Guile, OpenBSD has a very up-to-date
Racket package, so I just installed that.</p>
<pre><code># pkg_add racket
</code></pre>
<p>Now I had a Racket interpreter living at /usr/local/bin/racket. Sure,
I could have just run that and started to play around, but I really
wanted to install a mode for Emacs, which is what all serious Lisp
users do. A quick Google revealed geiser-mode as the premier Scheme
interaction mode. Opening up Emacs, I installed that with a quick "M-x
package-install RET geiser-mode RET", and that was that.</p>
<p>There were a few bindings I had to get used to, but the real
interesting part of my experience was using Racket, not my editor
(although geiser-mode is pretty good).</p>
<p>After playing around for a bit and installing paredit-mode, I was
ready to get started.</p>
<h2>First impressions</h2>
<p>The first thing you notice when using a Lisp is the large number of
parentheses. In Racket, sometimes square brackets are used to clear
things up, which really helps. I've never seen this in Common Lisp or
other Schemes, so this must be something Racket-specific.</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nv">a</span> <span class="mi">5</span><span class="p">)</span>
<span class="p">(</span><span class="nv">b</span> <span class="mi">10</span><span class="p">))</span>
<span class="p">(</span><span class="nb">+</span> <span class="nv">a</span> <span class="nv">b</span><span class="p">))</span>
</pre></div>
<p>That seems pretty trivial and non-threatening, right? Right? Well, it
doesn't stay very nice for long unless you use square brackets or some
other distinct type of bracket.</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nv">a</span> <span class="p">(</span><span class="nb">map</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nv">x</span><span class="p">)</span> <span class="p">(</span><span class="nv">equal?</span> <span class="nv">x</span> <span class="s">"hello"</span><span class="p">))))</span>
<span class="p">(</span><span class="nv">b</span> <span class="p">(</span><span class="nv">get-field</span> <span class="nv">info</span> <span class="nv">some-struct</span><span class="p">)))</span>
<span class="p">(</span><span class="nb">map</span> <span class="nb">list</span> <span class="nv">a</span> <span class="nv">b</span><span class="p">))</span>
</pre></div>
<p>Maybe you don't think that's too bad, but it looks a lot better with
square brackets. Mainly, it lets you keep track of which parens close
with expressions. Even with rainbow-delimiters mode, it's really
difficult! It's a bit less difficult in Racket, though.</p>
<p>Coming from Haskell, it was a bit jarring to see that there was no
type safety at all in default Racket. You could have something like
this:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nv">equal?</span> <span class="p">(</span><span class="nb">read-line</span><span class="p">)</span> <span class="s">"error out please"</span><span class="p">)</span>
<span class="p">(</span><span class="nb">read-line</span> <span class="mi">4</span> <span class="mi">5</span><span class="p">)</span>
<span class="p">(</span><span class="nb">+</span> <span class="mi">3</span> <span class="mi">4</span><span class="p">))</span>
</pre></div>
<p>Of course, since <code>(read-line)</code> doesn't take any arguments, you'll get
an error at runtime complaining about giving it the wrong number of
arguments. But that's the point, you only get the error at
<em>runtime</em>. In fact, if you never type in "error out please" you'll
<em>never</em> get an error. Oh man, that's scary: you're supposed to catch
all errors at compile-time with a good type system.</p>
<h2>Typed Racket</h2>
<p>In Racket, you start every file with a declaration of what language
you're using. Usually:</p>
<pre><code>#lang racket
</code></pre>
<p>But sometimes you might be writing a document or info file for raco,
so you'll want a different one. Here's one I always use, especially
after that horror story about runtime errors:</p>
<pre><code>#lang typed/racket
</code></pre>
<p>Pretty self-explanatory, I think. Here's how some normal Racket code
might look:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">define</span> <span class="p">(</span><span class="nv">square</span> <span class="nv">n</span><span class="p">)</span>
<span class="p">(</span><span class="nb">*</span> <span class="nv">n</span> <span class="nv">n</span><span class="p">))</span>
<span class="p">(</span><span class="nv">define</span> <span class="nv">my-numbers</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="mi">2</span> <span class="mi">5</span> <span class="mi">3</span> <span class="mi">8</span> <span class="mi">5</span><span class="p">))</span>
</pre></div>
<p>In Typed Racket, you can put in type annotations, too! It's usually
pretty easy to see what's meant. Here are the appropriate type
annotations for the two bindings above:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="err">:</span> <span class="nv">square</span> <span class="p">(</span><span class="nv">-></span> <span class="nv">Number</span> <span class="nv">Number</span><span class="p">))</span>
<span class="p">(</span><span class="err">:</span> <span class="nv">my-numbers</span> <span class="p">(</span><span class="nv">Listof</span> <span class="nv">Integer</span><span class="p">))</span>
</pre></div>
<p>Those actually do look similar to type annotations in Haskell:</p>
<div class="highlight"><pre><span></span><span class="nf">square</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="p">(</span><span class="kt">Num</span><span class="w"> </span><span class="n">a</span><span class="p">)</span><span class="w"> </span><span class="ow">=></span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="ow">-></span><span class="w"> </span><span class="n">a</span><span class="w"></span>
<span class="nf">myNumbers</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="p">[</span><span class="kt">Int</span><span class="p">]</span><span class="w"></span>
</pre></div>
<p>Personally, I like the Haskell syntax more. Of course, since Racket is
a Lisp, the syntax can be changed infinitely, so that's not really a
complaint.</p>
<p>All in all, the Racket REPL is cool (although nothing special
nowadays), the language is good if unsafe, Typed Racket is nicer,
Haskell, ML, OCaml and other "real" functional languages still feel
nicer to me, though.</p>
<p>A few days isn't enough to really get to know a language, though, so
maybe I'm wrong. Until someone tells me so, I won't use any Lisp as
extensively as I do Haskell, mostly because it feels unsafe.</p>
Emacs is greaturn:https-kaashif-co-uk:-2014-09-18-emacs-is-great2014-09-18T00:00:00Z2014-09-18T00:00:00ZKaashif Hymabaccus
<p>I've seen many completely stupid articles where people furiously
circlejerk over how Vim is the best and nothing will ever come close,
but it's rare that I see anyone write much about Emacs, probably
because fewer people use it (or maybe Emacs attracts a different
demographic). It's rare I see an article like</p>
<!-- more -->
<p><a href="https://medium.com/@mkozlows/why-atom-cant-replace-vim-433852f4b4d1">this</a>
one, where the author's stupidity is front and center.</p>
<p>I admit that Atom can't replace Vim, as the title says, but here is
the kicker:</p>
<blockquote><p>If this sounds a bit commonplace, it's because Emacs' big idea has
been widely influential and extensibility is today a standard feature
in any serious editor. Sublime Text uses Python instead of Lisp, and
Atom uses Coffeescript, but the fundamentals of commands and keymaps
are built in to the core. Even Vim has absorbed Emacs' extensibility:
Vim script can define new functions, which can be mapped to command
keystrokes.</p></blockquote>
<p>While Emacs' big idea has caught on, hardly any program, let alone any
editor lets you customise it in the way Emacs does. For example, it is
entirely possible (although stupid) to use Emacs as your web browser,
email client, it's even possible to use Vim in Emacs, with
evil-mode. Hell, I'd argue that with Emacs' nigh-infinite
extensibility, evil-mode is even better than the "real deal" - you get
all of the fantastic Emacs modes you can't get in Vim, due to its
limited extensibility.</p>
<p>Here's another gem:</p>
<blockquote><p>This philosophy of minimalist commands that can be composed together
is the fundamental originating philosophy of Unix, and Vim exemplifies
it like no other editor.</p></blockquote>
<p>Unix philosophy = good. OK. I get it. I hear the same thing from
anti-systemd zealots who spout this tripe without knowing what it
means. The Unix philosophy tells you to write programs that do one
thing and do it well. Programs. One thing. What we have learnt from
the blog post is that you shouldn't find the best editor, you should
find the most minimalistic editor with composable commands. Somehow, I
think the author uses Vim with many plugins instead of piping text
through program after program, each of which only does one thing.</p>
<p>The lesson here is that you should look for the <em>best</em> tool, not the
one that most fits into your twisted, misinterpreted view of a
philosophy almost entirely unrelated to what your tool should do.</p>
Patching OpenBSDurn:https-kaashif-co-uk:-2014-08-24-patching-openbsd2014-08-24T00:00:00Z2014-08-24T00:00:00ZKaashif Hymabaccus
<p>Recently, I've been trying to understand the ins and outs of CVS in
order to be able to contribute to OpenBSD without messing up anything. I
have sent a few patches to ports@, but anything complex was beyond my
abilities until recently.</p>
<!-- more -->
<h2>How does OpenBSD's contributing system work?</h2>
<p>OpenBSD uses a tried and tested method of accepting contributions from
users that has remained unchanged for many years - a mailing list.
Anyone can send in a patch to ports@openbsd.org or tech@openbsd.org and
it will be looked at by the people with commit access and (hopefully)
committed into the tree.</p>
<p>But where's this "tree"? What is it? OpenBSD uses CVS as its version
control system. While it is a bit old, they have their reasons for using
it, so it's best not to complain. Essentially, all history is stored on
the server, not locally, which is very different from how a distributed
VCS would do it. It doesn't affect actual usage of CVS, though,
so don't worry about it.</p>
<h2>Checking out the source</h2>
<p>From here on out, I'll assume you're trying to update a program in the
ports tree you might use. The example program I'll be using is
"editors/joe", which is Joe's Own Editor, a very simple editor inspired
by Emacs and WordStar (yeah, it's not the <em>newest</em> editor either).</p>
<p>The first step is finding a nearby CVS repo. A good place to look is on
<a href="http://www.openbsd.org/anoncvs.html#CVSROOT">this list</a>. Pick one
that's near you and export the CVSROOT variable where it says
"CVSROOT=anoncvs@blahblah". For example, I'm in the UK, so I'll pick a
server in Europe and export CVSROOT:</p>
<pre><code>$ export CVSROOT=anoncvs@ftp5.eu.openbsd.org:/cvs
</code></pre>
<p>The next step is checking out the ports tree from the server:</p>
<pre><code># cd /usr
# cvs checkout ports
</code></pre>
<p>Now you have a fully updated ports tree in /usr/ports! Note that I
didn't specify any flags: on OpenBSD, there is a default ~/.cvsrc that
comes by default in every user's home. Mine looks like this:</p>
<pre><code># $OpenBSD: dot.cvsrc,v 1.1 2013/03/31 21:46:53 espie Exp $
#
diff -uNp
update -P
checkout -P
</code></pre>
<h2>Making a patch</h2>
<p>Now, you might want to find the file you want to change. In this
example, it's in editors/joe, so lets go there in the ports tree:</p>
<pre><code># cd /usr/ports/editors/joe
</code></pre>
<p>I'm not going to explain how to edit Makefiles and update ports in
OpenBSD, a guide for that already exists
<a href="http://www.openbsd.org/faq/ports/guide.html">here</a>. Instead, lets
assume you already know how to make changes (and you will, after reading
that guide).</p>
<pre><code># vi Makefile
**make some changes**
</code></pre>
<p>If you delete any files or add any files, you have to do <code>cvs delete
<file></code> or <code>cvs add <file></code> for the changes to be tracked properly. Do
this before making a patch in the next step.</p>
<p>Now you have to get the changes you just made into an email, which is
easy using <code>cvs diff</code>. Run <code>cvs diff</code> and redirect its output to a file
somewhere.</p>
<pre><code># cvs diff > /tmp/joe.diff
</code></pre>
<p>If you look at /tmp/joe.diff, you'll see that all the changes you made
have been recorded in that file, in the format used by patch(1). You
don't need to know how to apply the patch, that will be done by whoever
wants to try out your changes. For completion's sake, you do that by
running <code>patch -p0 < /tmp/joe.diff</code> in the right directory
(/usr/ports/editors/joe, in this case).</p>
<h2>Mailing that patch to ports@</h2>
<p>In case you don't know how mailing lists work, let me explain. You send
an email to an address (in this case, ports@openbsd.org), and it is
processed by the mail server and sent out to everyone subscribed to the
list, who can then apply the patch.</p>
<p>In your email client, write a new message with the subject "UPDATE: joe
3.7 -> 3.8" (with the right versions, of course), to be sent to
"ports@openbsd.org" and CC'ed to whoever was listed as the port
maintainer in the Makefile. Write something short and descriptive in the
body of the message, like this:</p>
<pre><code>Hello ports@,
This is an update to joe which fixes this bug and that bug and adds a
useful feature.
OK?
</code></pre>
<p>In the rest of the message, paste your diff. It's easier for everyone if
you just put your diff in the body of the message, because they can then
just apply your email as the patch and everything will work out fine.</p>
<p>Now send the mail and wait.</p>
<p>If you get a response, take into consideration any criticism and send a
new diff. If someone important (usually with an @openbsd.org email
address) tells you it's OK, congratulations! They'll apply your patch,
commit it to CVS and you'll have contributed to OpenBSD!</p>
Sambaurn:https-kaashif-co-uk:-2014-08-22-samba2014-08-22T00:00:00Z2014-08-22T00:00:00ZKaashif Hymabaccus
<p>Recently, I've been trying to get away from pre-packaged file sharing
solutions (e.g. FreeNAS) and trying to set up the services they provide
from scratch. While I obviously won't be able to write a web GUI or
create a whole distro, that simply isn't necessary. What is necessary is
setting up a file share and appropriate read/write permissions.</p>
<!-- more -->
<h2>What is Samba?</h2>
<p>Among other things, Samba is a collection of daemons - winbindd, nmbd
and smbd - that let you provide file shares and printers to any client
capable of communicating with Samba. I don't really care about the
printer part, but it's nice to know about.</p>
<p>Essentially, if you have a Windows machine, Samba lets a Unix machine
announce itself and provide file shares when you click on your Unix box
in the file manager. You can do the same with free tools, on Unix, but I
don't think any Unix has a default that's as easy to use as Windows', as
much as it pains me to say.</p>
<h2>Setting it up</h2>
<p>If you have a Windows PC, you might have noticed that your Unix box
doesn't show up on the network, because it's not broadcasting its name.
That's easy to fix, just start <code>nmbd</code>! Assuming OpenBSD:</p>
<pre><code># /etc/rc.d/nmbd -f start
nmbd(ok)
</code></pre>
<p>Now you should see your hostname appear on the network from a Windows
machine or other Samba client.</p>
<p>While that may be cool, it isn't very useful. You might want to share
some files. Let's assume you have a directory you want to share, at
/home/samba. If someone with no valid credentials walks into your house
with their laptop, do you want them to be able to read your share?
Chances are you do, since you might want people to see your pictures
from their phone or whatever. You don't necessarily want everyone to
have <em>write</em> access, though.</p>
<p>Keeping that in mind, let's take a look at /etc/samba/smb.conf, which is
the config for smbd, the "main" Samba daemon that takes care of
authentication, printers and, what we want, file shares.</p>
<pre><code>$ cat /etc/samba/smb.conf
# This is the main Samba configuration file. You should read the
# smb.conf(5) manual page in order to understand the options listed
# here. Samba has a huge number of configurable options (perhaps too
# many!) most of which are not shown in this example
#
# For a step to step guide on installing, configuring and using samba,
# read the Samba-HOWTO-Collection. This may be obtained from:
# http://www.samba.org/samba/docs/Samba-HOWTO-Collection.pdf
#
# Many working examples of smb.conf files can be found in the
</code></pre>
<p>That goes on and on and on... So let's empty the file first.</p>
<pre><code># : > /etc/samba/smb.conf
</code></pre>
<p>Now there's nothing in it, and you're free to put this in there with
whatever editor you want:</p>
<pre><code>[global]
workgroup = WORKGROUP
server string = Kaashif's Server
security = user
map to guest = Bad User
guest account = nobody
log file = /var/log/samba/smbd.%m
max log size = 50
[Public]
comment = Public Files
path = /home/samba
public = yes
writable = no
browseable = yes
guest ok = yes
write list = @staff
</code></pre>
<p>Let's go through that line by line:</p>
<pre><code>[global]
</code></pre>
<p>These are settings global to all shares and printers.</p>
<pre><code>workgroup = WORKGROUP
</code></pre>
<p>This doesn't mean anything significant to a home user, but it should be
set to the same as whatever your other PCs are set to. WORKGROUP is the
default, so leave it like this.</p>
<pre><code>server string = Kaashif's Server
</code></pre>
<p>A short description of your server.</p>
<pre><code>security = user
map to guest = Bad User
guest account = nobody
</code></pre>
<p>If a user trying to access your shares is not recognised, it is mapped
to the guest user, which is an alias for the "nobody" user locally. So
whatever permissions you set for the /home/samba directory, it should be
readable by the "nobody" user.</p>
<pre><code>log file = /var/log/samba/smbd.%m
max log size = 50
</code></pre>
<p>This creates a log file for each machine that accesses your shares. Have
a look in /var/log/samba after you've accessed the share to see what
the logs look like.</p>
<pre><code>[Public]
comment = Public Files
path = /home/samba
public = yes
writable = no
browseable = yes
guest ok = yes
write list = @staff
</code></pre>
<p>"Public" is the name of the share. The rest is self-explanatory, other
than the "write list" line, which means any user in the login class
"staff" will be allowed to write. If your user (for example, "fred") is
not in this class, then add him:</p>
<pre><code># usermod -L staff fred
</code></pre>
<p>This means that if you give Samba the username "fred" and his password,
you can write to the share, assuming the file system permissions allow
fred to write there.</p>
<h2>Starting Samba</h2>
<p>Instead of starting smbd and nmbd separately, you can start them
together using the "/etc/rc.d/samba" script:</p>
<pre><code># /etc/rc.d/samba -f start
smbd(ok)
nmbd(ok)
</code></pre>
<p>If you want it to start at boot, edit "/etc/rc.conf.local" and add the
line:</p>
<pre><code>samba_flags=""
</code></pre>
<p>That should be that, and you should be able to see your server on the
network, click on it, and be able to see and download any files from
"/home/samba" on your server.</p>
Haskell isn't difficulturn:https-kaashif-co-uk:-2014-07-25-haskell-isn-t-difficult2014-07-25T00:00:00Z2014-07-25T00:00:00ZKaashif Hymabaccus
<p>People like to say that learning Haskell is hard, because a pure, lazy,
functional language isn't something most people are used to. The simple
fact of the matter is that most people (nowadays, at least) will be
introduced to programming through Python, Javascript or maybe even Java
or C#. The problem with learning dynamically typed languages that don't tell you</p>
<!-- more -->
<p>about types is that you don't get to see an integral part of how your
program works. It's worse if you program in a weakly typed language like
Perl or Javascript - you won't have any concept of types unless you read
around, which is a poor substitute for actually using types.</p>
<h2>"But I tried to learn Haskell and it was hard!"</h2>
<p>Most people who try to learn Haskell fail to "get it" for a while. After
writing programs in Haskell, going through a few tutorials, seeing some
examples of code, and having the power of purity and lazy evaluation
click, people tend to find Haskell a lot easier. The problem is that
most people don't get past the initial barrier to entry, which is
"unlearning" the paradigms they're used to.</p>
<p>Coming from Java and C#, they'll have to unlearn that style of OOP and
get used to Haskell's way of doing things. Coming from Javascript and
PHP, they'll have to learn about types. In fact, unless the beginner
we're talking about comes from ML, OCaml or similar, they will have to
get used to how much types are used in Haskell in ways they may not be
used to.</p>
<h2>Nullable Types</h2>
<p>In C, let's say, you might want a function which takes a name and does
something with it, returning something else, perhaps a boolean
indicating whether the name is valid and in some database somewhere:</p>
<div class="highlight"><pre><span></span><span class="kt">bool</span><span class="w"> </span><span class="nf">validate</span><span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="w"> </span><span class="n">name</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// some code</span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>That is all well and good, but what if the pointer passed to the
function, instead of pointing to an array like you'd expect, is null?
You'd have to add in a null check.</p>
<div class="highlight"><pre><span></span><span class="kt">bool</span><span class="w"> </span><span class="nf">validate</span><span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="w"> </span><span class="n">name</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nb">NULL</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// some code</span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>That's not terribly shocking, but it's a bit inconvenient. That seems
like a low-level problem you'd never get in a higher-level language like
Java, for example. That is wrong. A similar function would look like
this:</p>
<div class="highlight"><pre><span></span><span class="kt">bool</span><span class="w"> </span><span class="nf">validate</span><span class="p">(</span><span class="n">String</span><span class="w"> </span><span class="n">name</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">null</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">false</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="c1">// code</span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Why are our types nullable? That seems like a terrible design decision
which results in NullPointerExceptions for everyone. Abstracting the
hardware away hasn't solved this problem automatically, it seems.</p>
<p>In Haskell, types are not nullable. You can instead use optional types.
For example, if you have a function that could fail, you either return
"Just" the answer or "Nothing":</p>
<div class="highlight"><pre><span></span><span class="nf">validate</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kr">if</span><span class="w"> </span><span class="n">condition</span><span class="w"> </span><span class="kr">then</span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="n">answer</span><span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="kt">Nothing</span><span class="w"></span>
</pre></div>
<p>While this sort of thing can and is done in other languages, beginners
tend not to be exposed to them. Even if they were, the fact is that most
code out there is riddled with checks for null.</p>
<p>Even worse, most people are unaware of how easy it is to code such a
solution when you have a good algebraic type system:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span><span class="w"> </span><span class="kt">Maybe</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="kt">Nothing</span><span class="w"></span>
</pre></div>
<p>In other languages, this sort of thing would probably done via sentinel
values, which brings with it a whole host of problems. Beginners would
have to unlearn that habit, because it's all to easy to code a
"getIndex" function that returns "255" when there's an error, despite
the fact that 255 is a valid list index for an element.</p>
<h2>Loops? What loops?</h2>
<p>This really threw me off when I started learning Haskell. Back when I
started to learn Haskell, I thought it looked simple enough to write a
simple text adventure in. In Python, such a game could be boiled down
to:</p>
<div class="highlight"><pre><span></span><span class="n">player</span> <span class="o">=</span> <span class="n">Player</span><span class="p">()</span>
<span class="k">while</span> <span class="n">player</span><span class="o">.</span><span class="n">alive</span><span class="p">:</span>
<span class="c1"># get input, update world etc</span>
</pre></div>
<p>The key part is that there is a main game loop. This is how it is in
most games: there is some sort of loop that is run over and over until
the player dies or quits. Simple enough, right?</p>
<p>My first experience with Haskell and IO was with the typical "Hello,
world!" program:</p>
<div class="highlight"><pre><span></span><span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">putStrLn</span><span class="w"> </span><span class="s">"Hello, world!"</span><span class="w"></span>
</pre></div>
<p>That seems simple, too. Surely it won't be too hard to do something like
this:</p>
<div class="highlight"><pre><span></span><span class="nf">player</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">new</span><span class="w"> </span><span class="kt">Player</span><span class="w"></span>
<span class="nf">main</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">while</span><span class="w"> </span><span class="p">(</span><span class="n">alive</span><span class="w"> </span><span class="n">player</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">play</span><span class="w"> </span><span class="n">game</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>Well, that would be very hard for a beginner to do, considering that
"player" is a constant (there are no variables in Haskell) and there are
<em>no loops</em> in Haskell, unless you install "monad-loops" and use whileM,
untilM and the like.</p>
<p>The preferred solution, I now know, is to write a recursive function
that uses the State monad to pass along a Player data type. A beginner
has no idea how to approach a problem in Haskell when their usual tools
are not present.</p>
<h2>Monads?</h2>
<p>In imperative, "traditional" languages, there is no concept of the
monad. In Haskell, it's everywhere, even the "Hello, world!" program was
100% monadic: the main function always has type "IO ()". The "IO" means
we're in the IO monad.</p>
<p>While this would be a great time to have a terrible monad analogy
tutorial, the easiest way of understanding monads, I think, is to come
at them from a mathematical angle. I think I did this in that other post
about functors <a href="http://kaashif.co.uk/2014/02/05/functor/">here</a>. A
monad is just a container for a value with a few functions defined:
"return" takes a value and puts it into a container, and "join" turns a
value in a container in a container into a value in a container.</p>
<p>That's really simple, but also really useful. Most beginners are not
exposed to anything like monads or functors, so they are naturally very
hard to grasp.</p>
<h2>Conclusion</h2>
<p>Don't be discouraged because you find learning Haskell, Lisp, APL,
Forth, or any other esoteric but useful langage hard. The only reason it
seems difficult is that you're starting from scratch: it gets easier
once you know the basics.</p>
How I share a file, simplyurn:https-kaashif-co-uk:-2014-06-08-how-i-share-a-file-simply2014-06-08T00:00:00Z2014-06-08T00:00:00ZKaashif Hymabaccus
<p>Earlier, I saw <a href="http://reinehr.me/how-i-simply-share-a-file">this</a>
article claiming to describe how to share a file "simply" by running
Python's web server module (with either Python 2 or 3). While that may
be <em>easy</em>, it's not simple, and certainly not fast.</p>
<!-- more -->
<p>The largest problem with using Python or Ruby to run a web server
instead of a single-purpose program is initialisation time. Regardless
of performance under load or latency or whatever, the fact is that
running "python3 -m http.server" takes tens of seconds to reach a usable
state on an older machine, and minutes on even older ones. This is
unacceptable if all you want to do is share a text file which would take
less than a second to download - it's ludicrous to spend more time
waiting for a server to start than actually using it.</p>
<h2>The alternative</h2>
<p>I use <a href="http://www.acme.com/software/thttpd/">thttpd</a>.
While the site may not look modern and it seems like they went out of
their way to make it look ugly, the software that runs it is very
performant, portable, and everything you could ever want in a
quick-and-dirty web server solution.</p>
<p>Even better it starts within a second even on really old hardware. If
you want to share a directory, here's what you do:</p>
<pre><code>$ thttpd
</code></pre>
<p>That was very simple, fast and easy! The problem is that you need to be
root to bind to port 80, so just kill that process and start another
one, like this:</p>
<div class="highlight"><pre><span></span>$ thttpd -p <span class="m">8000</span>
</pre></div>
<p>That can be run as any user, since port 8000 is high enough not to need
root to be used. Since it daemonizes straight away, to kill it, you need
to use <code>pkill</code>.</p>
<div class="highlight"><pre><span></span>$ pkill thttpd
</pre></div>
<p>And that's that. A <em>simpler</em> than starting up an interpreter for a
programming language and loading modules within modules of code just to
serve a couple of files.</p>
<h2>Scripting it</h2>
<p>While that's all well and good, just like the author of the article I
linked, I want to be able to run <code>share</code> and share my files, so I wrote
a script. This is going beyond the realm of simple, but it'll certainly
be convenient.</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
<span class="nb">echo</span> -n <span class="s2">"Starting thttpd: "</span>
<span class="k">if</span> thttpd -p <span class="m">8000</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">echo</span> <span class="s2">"done"</span>
<span class="k">else</span>
<span class="nb">echo</span> <span class="s2">"failed"</span>
<span class="k">fi</span>
</pre></div>
<p>That does start the server, but doesn't give us any way to kill it and
it doesn't tell us our IP address (which we might not know due to DHCP
on an unfamiliar network). After <code>echo "done"</code>, add in something to tell
us our IP, then wait for the user to press enter and kill the server:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
<span class="nb">echo</span> -n <span class="s2">"Starting thttpd: "</span>
<span class="k">if</span> thttpd -p <span class="m">8000</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">echo</span> <span class="s2">"done"</span>
ifconfig <span class="p">|</span> grep inet
<span class="nb">read</span> line
pkill thttpd
<span class="k">else</span>
<span class="nb">echo</span> <span class="s2">"failed"</span>
<span class="k">fi</span>
</pre></div>
<p>That is OK, but we can do better to show us our IP - as it is, the
script outputs this:</p>
<pre>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
inet6 fe80::213:e8ff:fe73:485%iwn0 prefixlen 64 scopeid 0x2
inet 192.168.0.9 netmask 0xffffff00 broadcast 192.168.0.255
</pre>
<p></pre></p>
<p>Sure the IP is in there, but it's too messy. I replaced <code>grep inet</code> with
something a little more sophisticated:</p>
<div class="highlight"><pre><span></span>grep -o <span class="s1">'[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}'</span>
</pre></div>
<p>That regex shows us all of the IPs in ifconfig's output, but it also
shows the broadcast address and doesn't show us the port. To fix that,
we can pipe the whole thing into this:</p>
<div class="highlight"><pre><span></span>awk <span class="s1">'!/255$/{ print $0 ":8000"}'</span>
</pre></div>
<p>The finished script looks like this:</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
<span class="nb">echo</span> -n <span class="s2">"Starting thttpd: "</span>
<span class="k">if</span> thttpd -p <span class="m">8000</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">echo</span> <span class="s2">"done"</span>
ifconfig <span class="se">\</span>
<span class="p">|</span> grep -o <span class="s1">'[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}'</span> <span class="se">\</span>
<span class="p">|</span> awk <span class="s1">'!/255$/{ print $0 ":8000"}'</span>
<span class="nb">read</span> line
pkill thttpd
<span class="k">else</span>
<span class="nb">echo</span> <span class="s2">"failed"</span>
<span class="k">fi</span>
</pre></div>
<p>And its output looks like this:</p>
<pre><code>Starting thttpd: done
127.0.0.1:8000
192.168.0.9:8000
</code></pre>
<p>Now you can type <code>share</code> and have a web server up in a minimum of time.</p>
Workflowurn:https-kaashif-co-uk:-2014-05-31-workflow2014-05-31T00:00:00Z2014-05-31T00:00:00ZKaashif Hymabaccus
<p>The inspiration for this came from
<a href="http://blog.z3bra.org/2014/05/grok-that-workflow.html">this</a>
blog post, where the author describes how he uses his computer. While he
does use CRUX, a GNU/Linux distro and I use OpenBSD, our workflows are
actually surprisingly similar, which can, in part, be attributed to the</p>
<!-- more -->
<p>inspiration for the ports system both CRUX and OpenBSD took from
FreeBSD, although OpenBSD is obviously a lot closer to the original.</p>
<h2>Using OpenBSD</h2>
<p><a href="http://openbsd.org">OpenBSD</a> is a very traditional Unix-like OS that
doesn't get in your way or do anything for you. In contrast to the
author of the blog post I linked to, I appreciate a good set of defaults
and a smooth install. Indeed, you can install OpenBSD by mashing enter
and only stopping to enter a root password, and you'll end up with a
very sane set of defaults - an X server, some X programs, tmux, and not
to mention the other software that comes in the base system - an FTP
server, Nginx, and so on.</p>
<p>Even with all of this "bloat", the install footprint is well under a
gigabyte and RAM usage (checked from the text console) stays under 32
MB. So either most GNU/Linux distros are using all that disk space and
RAM to do things I don't want or need, or they do the minimum, but a lot
less efficiently than OpenBSD.</p>
<p>To summarise, if you want to use OpenBSD, here are the steps to follow:</p>
<ol>
<li>Download an install image from
<a href="http://ftp.openbsd.org/pub/OpenBSD/5.5/">here</a></li>
<li>Boot it from USB, CD or wherever</li>
<li>Hit enter until it tells you to stop</li>
<li>Reboot</li>
<li>Install packages</li>
</ol>
<p>After that, it's really identical to using Solaris, AIX or another BSD -
OpenBSD is fully POSIX compliant. That <em>does</em>, however, mean that it's
slightly different from the GNU userland (cp -R instead of cp -r, for
instance).</p>
<p>Note that I didn't mention ports: the idea with OpenBSD is that you use
the packages - the maintainers put a lot of effort into making sure you
don't need to compile packages from ports yourself. Largely, every
flavour of package you want already exists and has been compiled. Maybe
you want vim with Perl, Python and Ruby support - good news! There's a
package with that compiled in. You want rsync to use libiconv? They've
accounted for that use case too! Ports should only be downloaded and
used in very niche cases.</p>
<h2>Window Manager</h2>
<p>I'm not very adventurous with my WM, I just use i3, a very popular WM. I
say that like it's a bad thing, but there's a reason it's popular - it
does what it says on the tin very well. The default keybindings are fine
for me, although I use Mod4 (super) instead of Mod1 (alt). You can find
more about it <a href="http://i3wm.org/docs/userguide.html">here</a>, and to
install it, you should be able to find it in your OS's repos. It's in
OpenBSD under "i3", unsurprisingly.</p>
<p>For the uninitiated, it's a tiling window manager, which means that,
most of the time, my windows take up the whole screen and when there are
multiple, they "tile", meaning they are sized such that there are no
gaps and no windows behind another. Here's a GIF of i3 in action:</p>
<p><a href="/static/work1.gif"><img src="/static/work1.gif" alt="" /></a></p>
<h2>Web Browsing</h2>
<p>This is the activity I spend most of my time doing. Nowadays, I use
<a href="http://mason-larobina.github.io/luakit/">luakit</a>, which is a
Webkit-based browser with Vim-like keybindings, mostly based on
Pentadactyl, a popular Firefox plugin I used to use.</p>
<h2>Checking Email</h2>
<p>The activity I spend the second-most amount of time on while on my
laptop is checking my mail. I'm subscribed to a few mailing lists, but
the one I check the most is misc@openbsd.org. A close second is
tech@openbsd.org. While these aren't overly high-volume lists, it takes
a while for each message to be fetched from the server, and this must be
done one-by-one as I view them if I were to use IMAP.</p>
<p>Those seconds add up. Maybe they don't, but waiting a second between
reading messages is very annoying, so my only option was to store the
messages locally, allowing me to read them with no network delay.
Setting this up was easy, I just used the same Mutt config as I used
when on IMAP and configured it for local mail in a Maildir located at
<code>~/mail/</code>. Even better, this meant I could configure my system MTA to
deliver all system mail there, too (cron job errors, security output,
nightly backups).</p>
<p>Here are the relevant lines from my <code>.muttrc</code>:</p>
<pre><code>set mbox_type=Maildir
set spoolfile = "~/mail/"
set folder = "~/mail/"
</code></pre>
<p>To actually fetch the mail, I used getmail, a small Python script (can
be installed with <code>pkg_add getmail</code>). Setting that up was very simple,
too, just install and place the following in <code>.getmail/getmailrc</code>:</p>
<pre><code>[retriever]
type = SimplePOP3SSLRetriever
server = pop.example.com
username = username
password = password
[destination]
type = Maildir
path = ~/mail/
user = yourusername
[options]
read_all = false
</code></pre>
<p>Getmail can be run by simply running <code>getmail</code>, but this is very
annoying to do every time you want to read mail, and sort of defeats the
point - this was supposed to <em>save</em> time. I set up a cron job for it -
place the following in the file opened via <code>crontab -e</code>:</p>
<pre><code>30 * * * * getmail
</code></pre>
<p>That runs getmail every 30 minutes, meaning your mail stays up to date.
If you're ever waiting for an important message, you can still run
<code>getmail</code> to refresh your inbox directly, although it is mildly annoying
to open your shell every time you want to do that - why not bind it to a
key in Mutt?</p>
<p>To do that, add the following to your <code>.muttrc</code>:</p>
<pre><code>macro index "^" "!getmail<enter>"
</code></pre>
<p>That means that, whenever in the index, you can press "^" to get any new
mail.</p>
<h2>No place like ~</h2>
<p>You already know I have a <code>~/mail/</code> directory, but I have a few more:</p>
<pre><code>~ >> tree -L 2
.
|-- bin -> etc/scripts/bin
|-- etc
| |-- README.md
| |-- [...]
| `-- zsh
|-- mail
| |-- cur
| |-- new
| `-- tmp
|-- notes
|-- sent
|-- src
| |-- advent
| |-- [...]
| `-- www
`-- var
|-- documents
`-- music
47 directories, 4 files
</code></pre>
<p>As you can see, I keep my source code in <code>~/src/</code>, data files in
<code>~/var/</code>, mail in <code>~/mail/</code>, scripts in <code>~/bin/</code> and configs in
<code>~/etc/</code>. My etc directory is filled with configs which are symlinked to
from their expected locations. I'm sure I've written a post about it at
some point.</p>
<p>Just like most people, I'm a fan of text files, but not any ordinary
text files - most of my notes are written using Emacs' org-mode, which I
believe is a note-taking environment second to none. Unless I'm
programming in a REPL language (Ruby, Python, Lisp), I tend to use Vim.
I favour Emacs for those languages because...well, you have to use it to
really understand what I mean - look up some cool things you can do with
SLIME.</p>
<p>You can check out my dotfiles
<a href="http://code.kaashif.co.uk/kaashif/dotfiles">here</a>.</p>
Installing OpenBSD on a T61urn:https-kaashif-co-uk:-2014-05-08-installing-openbsd-on-a-t612014-05-08T00:00:00Z2014-05-08T00:00:00ZKaashif Hymabaccus
<p>I'm sure lots of people (dozens, perhaps) have installed OpenBSD on
ThinkPad T61s of some description, but with the recent release of
OpenBSD 5.5, lots of documentation has become (or already was, and now
is even more so) obsolete, like
<a href="http://www.thinkwiki.org/wiki/Installing_OpenBSD(4.5_-current">this article</a>_on_a_Thinkpad_T61)</p>
<!-- more -->
<p>, which deals with OpenBSD 4.5, which is <em>really</em> old - it's from
slightly over 5 years ago! OK, that may not seem too bad, but much has
changed in OpenBSD since then, so you might want to reevaluate the
situation if you're thinking of upgrading or something.</p>
<h2>Hardware</h2>
<p>The ThinkPad I bought isn't the best T61, but it is a decent, reliable
laptop that served its previous owner for 7 years and will serve me for
a while longer. The important bits of the laptop:</p>
<ul>
<li>1280x800 screen - not quite the 1680x1050 IPS screen of the better
model of T61, but good enough for me, certainly.</li>
<li>Intel GMA 965 graphics</li>
<li>1 GB of RAM</li>
<li>250 GB hard drive</li>
<li>Smart card reader</li>
<li>Fingerprint reader</li>
<li>Intel WiFi Link 4965</li>
</ul>
<p>I made sure that everything I actually use is supported by free drivers.
Funnily enough, the T61 I have also supports coreboot, but if you want
help using flashrom or something, you'd be better served by looking at
their wiki, taking advice from me would probably result in a bricked
laptop.</p>
<h2>OpenBSD Installation</h2>
<p>With the release of OpenBSD 5.5, the project also released USB disk
images you can dd straight to install media instead of having to install
to the USB drive then boot from the ramdisk kernel. Now, it's a lot
easier to install! Assuming you have a 64-bit CPU, you'll want to get
the install media
<a href="http://ftp.openbsd.org/pub/OpenBSD/5.5/amd64/install55.fs">here</a>. Every
T61 comes with a 64-bit Core 2 CPU, so this is the one you want if
you've got a T61. If you have a T60, you might need to get the i386
install media instead, but as far as I'm aware, they're all Core 2 as
well.</p>
<p>After getting the install media, you'll want to dd it to a disk. Make
sure to modify the following command for your disk:</p>
<div class="highlight"><pre><span></span>$ dd <span class="k">if</span><span class="o">=</span>install55.fs <span class="nv">of</span><span class="o">=</span>/dev/sd1c
</pre></div>
<p>After that, the USB drive is bootable, so reboot, wait for the kernel
messages to scroll by (all of that text with a blue background), and
you'll be taken to a screen where you'll be asked the following
question:</p>
<pre><code>Welcome to the OpenBSD/i386 5.5 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell?
</code></pre>
<p>Here, you'll want to type "i" and hit enter, you're installing. Unless
you're upgrading, but then, you probably don't need my help.</p>
<p>The majority of the questions are things any GNU/Linux, or *BSD user
should be able to answer, so I won't go into detail on those. Instead,
I'll focus on the T61-specific parts. In case you want help with a
general OpenBSD installation, look to
<a href="http://www.openbsd.org/faq/faq4.html">the OpenBSD project's guide</a>.</p>
<h2>Getting the network to work</h2>
<p>When you see this:</p>
<pre><code>Available network interfaces are: iwn0 re0 vlan0.
Which one do you wish to configure? (or 'done') [re0]
</code></pre>
<p>If you try to use the wireless card straight away, it will not work, and
the kernel will spit out the error "iwn-4965: could not read firmware".
Due to licensing issues, the OpenBSD project is not allowed to
distribute the firmware for Intel wireless cards on the installation
media. The best they can do is provide firmware packages
<a href="http://firmware.openbsd.org/firmware/5.5/iwn-firmware-5.10p0.tgz">here</a>.</p>
<p>You don't actually need the network to install OpenBSD, so I advise that
you type "done" and leave the networking be (unless you can connect to
the internet using Ethernet) until you boot into OpenBSD for the first
time.</p>
<p>When you reboot, you have a few choices:</p>
<ul>
<li>Find an Ethernet cable to get the firmware</li>
<li>Never use the internet</li>
<li>Put the firmware on a USB drive using another PC</li>
</ul>
<p>I didn't have any spare Ethernet cables on my desk, but I did have
another laptop, do I went for the third option. If you do have an
Ethernet cable and a router, pick "re0" as the network interface to
configure during the installation, and everything will work fine and
you'll be able to download the WiFi firmware through the internet, on
your laptop.</p>
<p>Assuming you went the same route I did, you'll want to create a new
file system on the now-useless installation media, so you can use it to
store firmware.</p>
<div class="highlight"><pre><span></span>$ mkfs_msdos /dev/sd1c
</pre></div>
<p>Then take out the USB drive, download the firmware on another PC onto
the drive, and install the firmware package on your laptop with pkg_add:</p>
<div class="highlight"><pre><span></span>$ sudo pkg_add -Dnosig iwn-firmware-5.10p0.tgz
</pre></div>
<p>You need to tell pkg_add to ignore signatures, because the firmware
packages are signed by a different key to the normal packages, so
pkg_add will reject firmware packages' signatures.</p>
<h2>Graphics acceleration</h2>
<p>Since my T61 has an Intel integrated graphics card, it just works
automatically, without me having to do anything special. You can tell if
OpenBSD detects your Intel card during boot because the font changes to
something that looks a bit different.</p>
<p>If you're still in doubt, try this:</p>
<div class="highlight"><pre><span></span>$ dmesg <span class="p">|</span> grep intel
</pre></div>
<p>My laptop shows this:</p>
<pre><code>inteldrm0 at vga1
drm0 at inteldrm0
inteldrm0: 1280x800
</code></pre>
<p>That means that the Direct Rendering Manager works, and has managed to
work out the resolution of the laptop's screen. This means that X will
work without any xorg.conf hacks. If you have an Nvidia card, my
condolences. The nv driver sucks, despite the stellar work of the
OpenBSD project does keeping it maintained. The fact is that reverse
engineering is no substitute for actual documentation, so users who have
ATI/AMD or Intel cards are far better off than Nvidia users.</p>
<h2>Extra features</h2>
<p>I haven't tried the smart card reader since I don't have a smart card,
and I haven't gotten around to trying the fingerprint reader. It might
work, but I don't really care. The one time I did try to use a
fingerprint reader, it was cool but pointless.</p>
<p>The PCMCIA port works, I tried it with an 3Com EtherLink III card I had
lying around, and it works perfectly.</p>
<p>The trackpad works very well - two finger scrolling works and all three
buttons work fine out of the box. Even that odd scrolling thing that
happens when you press button 3 and use the clit mouse works. No-one
ever uses it, though, but it's nice to know that it works.</p>
<h2>Summary</h2>
<p>All in all, my experience with the T61 just goes to show that OpenBSD
works great as a laptop OS, especially compared to the OS that came with
the laptop (Windows 7 and driver CDs everywhere).</p>
<p>Maybe at some point I'll buy a newer ThinkPad and see how the radeondrm
support is firsthand.</p>
Configuring Mutturn:https-kaashif-co-uk:-2014-04-18-configuring-mutt2014-04-18T00:00:00Z2014-04-18T00:00:00ZKaashif Hymabaccus
<p>On all of my computers, I like being efficient. That means eliminating
everything which uses all that precious CPU time and using applications
which are very customisable and configurable. These sorts of
applications tend to be text-based, which is, in my eyes, a good thing,
since they'll show you the information you need with a minimum of</p>
<!-- more -->
<p>distraction.</p>
<p>Mutt is a text based email client, and I think it's very well suited to
the task - generally, when reading emails, you are dealing with text.
The few times there are images, they're usually annoying HTML emails
from websites or as an attachment, meaning it's not integral to the
viewing of the email body itself.</p>
<p>Now that you're sold on the idea of a text-only email client, we can get
started installing, configuring, and using it.</p>
<h2>Installing Mutt</h2>
<p>Mutt should be in the package manager repository for your OS. On
FreeBSD, you can install it with <code>pkg install mutt</code>. On Debian, it's
<code>apt-get install mutt</code>. The command to run should be similar on most
operating systems. If you're on Windows, I suggest you install Cygwin
and build from source, since that's really the only way you can get
Mutt.</p>
<h2>Configuring Mutt</h2>
<p>If you just run <code>mutt</code>, you'll see that you've opened the local mail box
for your current user. In decades past, local mail was actually really
important - people sent mail to and from each other's servers, not
centralised servers like Outlook or Gmail. In any case, you don't want
this - you likely want to access your Gmail account, which is what I'll
be showing you how to do. The instructions for Yahoo or Outlook should
be similar, but consult their help pages for the server address and
port.</p>
<p>You need to create/edit the file <code>~/.muttrc</code> and input the following:</p>
<pre><code># The bit before @gmail.com in your address
set imap_user = "myname"
# Google's IMAP server, used to view mail
set folder = "imaps://imap.gmail.com:993"
# Google's SMTP server, used to send mail
set smtp_url = "smtp://myname@smtp.gmail.com:587/"
# Displayed in the "from" field of your mail
set from = "myname@gmail.com"
# Change this to your preferred editor
set editor = "vim + -c 'set textwidth=72' -c 'set wrap'"
# Sets your default inbox to the one on the IMAP server
set spoolfile = "+INBOX"
# Makes sure all your folders are polled for new mail
set imap_check_subscribed=yes
# Interval to check for new mail
set mail_check = 120
# Time to wait for server if connection drops
set timeout = 300
set imap_keepalive = 300
# Where Mutt keeps drafts
set postponed = "+[GMail]/Drafts"
# Mutt's cache settings, important for speed
set header_cache=~/.mutt/cache/headers
set message_cachedir=~/.mutt/cache/bodies
# Where mutt stores SSL certificates
set certificate_file=~/.mutt/certificates
# Mutt will never move mail from Gmail to your local mailbox
set move = no
# Include message you're replying to in the reply
set include
# Sorting by threads is very useful for keeping your mail in order
set sort = 'threads'
set sort_aux = 'reverse-last-date-received'
# Sets the height of the window opened to view a message
set pager_index_lines = 10
# I prefer not to show some of the headers, they get in the way
ignore "Authentication-Results:"
ignore "DomainKey-Signature:"
ignore "DKIM-Signature:"
hdr_order Date From To Cc
# A useful key binding to refresh your inbox
bind index "^" imap-fetch-mail
</code></pre>
<p>That might seem like a lot of variables, but they're all well-named and
well-documented in <a href="http://www.mutt.org/doc/manual/">Mutt's online manual</a>.
The manual is very extensive, and documents every variable you see
above.</p>
<h2>Opening Mutt and logging in</h2>
<p>With the settings above, you're all set to open Mutt again. Type <code>mutt</code>
into a terminal and you'll see a prompt telling you whether to accept
Google's SSL certificates. Press "a" for "accept always" a few times
until all the certs are accepted.</p>
<p>Next, you'll see a password prompt, like this:</p>
<pre><code>Password for kaashifhymabaccus@gmail.com@imap.gmail.com:
</code></pre>
<p>Of course, type in your password, and it'll be sent off over an SSL
connection to Google's IMAP server.</p>
<p>If all goes well, the next thing you see should be your inbox. If it
hasn't worked, I advise that you ask on an IRC channel. Although it
isn't strictly related to Arch, #archlinux on irc.freenode.net is very,
very active, so you're likely to get an answer there.</p>
<h2>Using Mutt</h2>
<p>You can use the up and down arrow keys to scroll through the mail, as
well as the standard Page Down and Page Up keys. You can also type in a
number and a prompt should appear like this:</p>
<pre><code>Jump to message: 1
</code></pre>
<p>If you look ot the far left of Mutt, you should see a column of numbers.
Type in any one of those, and Mutt will jump to it.</p>
<p>If you want to open a message, press enter over one, and the bottom 4/5
of the screen should display the message, with the message inde open in
the top 1/5.</p>
<p>When you're at the index (the screen you see just after loggging in),
you should see this at the top:</p>
<pre><code>q:Quit d:Del u:Undel s:Save m:Mail r:Reply g:Group ?:Help
</code></pre>
<p>You'll see bars like this at all screens of Mutt, so press "?" if you
forget what a key does.</p>
<p>To send a message, press "m". You'll have to say who it's for and what
it's subject is, then your editor will open, where you can type out the
body of the message. It's best to keep the message to 72 columns wide,
to maximise viewing pleasure for everyone involved.</p>
<p>After quitting your editor, you'll be taken to a screen with this at the
top:</p>
<pre><code>y:Send q:Abort t:To c:CC s:Subj a:Attach file d:Descrip ?:Help
</code></pre>
<p>It should be obvious how to send the mail or change the headers.</p>
<p>If you use mailing lists a lot (like I do), then you'll want to change
mailboxes. You do that by pressing "c", typing "=mymailbox", and
pressing enter. Of course, replace "mymailbox" with the actual name of
the mailbox.</p>
<p>At this point, you should be able to use your new email client for most
day-to-day tasks. If you need to do something more advanced, you should
look at the manual I linked earlier.</p>
<p>For other interesting and useful text-based programs,
<a href="http://inconsolation.wordpress.com/">this blog</a> isn't a bad place to go.</p>
Ideas for a projecturn:https-kaashif-co-uk:-2014-04-18-ideas-for-a-project2014-04-18T00:00:00Z2014-04-18T00:00:00ZKaashif Hymabaccus
<p>There is always a lot of buzz around the idea of "learning to program".
While I think it's very important that children learn logical thinking
and problem solving, I also realise that the majority of children, and
people in general, would probably not benefit from a very language
specific, rote learning based, generally old-fashioned approach to</p>
<!-- more -->
<p>teaching programming. Those that would enjoy programming largely have
the aptitude to learn it themselves, with a bit of guidance. I think
that the best way to learn is to do - write programs that serve some
purpose or have some goal in mind - like a game or web app.</p>
<p>Obviously, my aim isn't to convert people who hate computers into
hackers, but to offer some ideas and advice to the people who are trying
to learn, but only find books and articles which walk them through
syntax and have some exercises, but no larger projects to complete to
draw together everything that they know.</p>
<p>Since writing simulations and AI programs are probably a bit beyond most
beginners, and boring system utilities are generally too tedious for a
beginner to feel motivated to write, I think the best direction to go in
for a project would be a game.</p>
<h2>But games aren't serious programming!</h2>
<p>I've never heard anyone say the above, but someone might. If they did,
they'd be completely and utterly wrong. If you're trying to learn about
object oriented programming, a game is perfect! OOP is intuitive if you
think of the objects as real things (a player, enemy, level etc). That
might not be entirely in the spirit OOP was conceived in, but it works
for teaching the basics of OOP: encapsulation, polymorphism etc.</p>
<p>The jist of it is that when you tell an object to do something and fetch
an answer, you don't need to know how the answer is obtained, just the
answer - the functionality is <em>encapsulated</em>. In turn, this means that
you can change the internal structure of the object without
repercussions, provided the interface remains the same. Here's an
example, in Python.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Enemy</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">search</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># Very long and complicated search algorithm</span>
<span class="k">return</span> <span class="n">direction_moved</span>
<span class="n">enemy</span> <span class="o">=</span> <span class="n">Enemy</span><span class="p">()</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">direction</span> <span class="o">=</span> <span class="n">enemy</span><span class="o">.</span><span class="n">search</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"Enemy moved "</span> <span class="o">+</span> <span class="n">direction</span> <span class="o">+</span> <span class="s2">" while searching for you"</span>
</pre></div>
<p>In the while loop, the Enemy instance's search method is called and a
result obtained. The person programming the while loop does not have to
know anything about search algorithms or how to write one - the search
method takes care of all of that. Even better, if the programmer who
wrote the search algorithm changes his mind about it, whoever wrote the
while loop doesn't have to change anything.</p>
<p>Games, while they don't have a reputation as serious business among
laymen, can be very useful as vehicles for learning about maths, physics
and, most importantly for me, computer science.</p>
<h2>Where should I start?</h2>
<p>That small example, while sort of neat to someone who's never seen a
working program before, isn't even valid code, so you can't use it as a
starting point.</p>
<p>Before you start a project, I'd advise you learn how to program and
solve some simple exercises. A good website for this, is
<a href="http://codeabbey.com">codeabbey.com</a>. Their exercises are quite good
for getting to grips with the basics of a language, and get harder as
you go.</p>
<p>After you've done that, you can start thinking about what sort of game
or program you want to make. Since it's best not to over-complicate
things, I suggest you write a text-based game. Since that is a bit
broad, let me suggest an adventure game, like "adventure" or
"battlestar", which you may have heard of. You probably haven't, so
<a href="http://i.imgur.com/aHa3cN3.png">here</a> is an example. That's a
screenshot of Zork, a classic text-based adventure game. Essentially,
you get a bit of description of your surroundings, then you can type in
commands like "go north" or "eat leaflet", and the game responds
accordingly. Not very hard to program, but very entertaining.</p>
<p>To make such a game, all you need is to be familiar with standard input
and output. Everything else is optional, but it's very easy for such a
game to be very deep and complex, both for the programmer and player,
believe it or not. Here is a very basic example, in Python.</p>
<div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="s2">"Welcome to Generic Text Adventure!"</span><span class="p">)</span>
<span class="n">running</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">while</span> <span class="n">running</span><span class="p">:</span>
<span class="n">command</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s2">"Enter command: "</span><span class="p">)</span>
<span class="k">if</span> <span class="n">command</span> <span class="o">==</span> <span class="s2">"quit"</span><span class="p">:</span>
<span class="n">running</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">elif</span> <span class="n">command</span> <span class="o">==</span> <span class="s2">"die"</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"You died!"</span><span class="p">)</span>
<span class="n">running</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"Command not recognised!"</span><span class="p">)</span>
</pre></div>
<p>That's not a very fun game, but it's easy to see how it could be
extended using your programming knowledge. Perhaps the command processor
should be extracted into a separate function? Maybe you could put it
into an object of its own and load the list of command from a text file?
The possibilities are endless, and the end result will always be
something workable, because the game is so simple that you'll never get
into problems with graphics drivers or network latency or anything you
didn't create yourself.</p>
<p>If you're looking for a simple project and you haven't already made a
game - make one! It can't hurt you, and it might expose some weaknesses
in your knowledge or just make you feel better about yourself.</p>
How this blog worksurn:https-kaashif-co-uk:-2014-04-13-how-this-blog-works2014-04-13T00:00:00Z2014-04-13T00:00:00ZKaashif Hymabaccus
<p>UPDATE: I now use Hakyll. See <a href="http://kaashif.co.uk/about">http://kaashif.co.uk/about</a> for
more. Also, 100% of the information in this blog post is now wrong or outdated.</p>
<p>When I decided I wanted to write a blog, I had to come up with some way
of writing posts (in a markup format which isn't HTML), and serving them
somehow.</p>
<!-- more -->
<p>First, I turned to Jekyll, a static blog generator written in Ruby and
used by GitHub. That turned out to work quite well, but I wanted more
control over the process of post generation. Next, I went for a CGI
application written in good old tried-and-tested Perl. That, too worked,
but CGI is a bit of a primitive technology, and didn't scale well when I
stress-tested it. Next on my list was a Python WSGI app, which ran in a
container which served pages. That was OK, and was the most stable
incarnation of my website, until I realised how incredibly insecure it
was compared to some of the other options I had available. Also, I was
getting bored of a solution which worked <em>too</em> well to be any fun.</p>
<p>Obviously, I had the right idea in the first place, with generating
static pages, which I could then transfer (using rsync or similar) to a
chrooted Nginx server running in a virtual machine (it isn't any more or
less secure than running it normally, but it helps me sleep at night),
but I couldn't lose face and go back to Jekyll, I'd be humiliated (in my
eyes only, though - most other people wouldn't care)! So I had to write
a Jekyll clone in a language that was safe, not prone to bugs, had a
rich package ecosystem (not unlike the Python package index or Ruby's
gems) and was pleasing to write. Obviously, the only sane option was
Haskell, and it's the language I chose to write my blog generator in. I
called it Muon, because it seems like a cool name - catchy, 4 characters
and easy to
type.</p>
<h2>Getting Muon</h2>
<p>At some point, when I consider Muon to be ready for use by normal
people, or at least, as normal as a Haskell package user can be, I will
upload it to <a href="http://hackage.haskell.org">Hackage</a> and you'll be able to
install it with a simple:</p>
<div class="highlight"><pre><span></span>$ cabal install muon
</pre></div>
<p>As it is, you have to get the latest snapshot from my darcs repository.
You can still use cabal to install it, though, so dependencies are
handled automatically.</p>
<div class="highlight"><pre><span></span>$ darcs get http://repos.kaashif.co.uk/muon
$ <span class="nb">cd</span> muon
$ cabal install
</pre></div>
<p>It might take a while to build the dependencies from source, so if your
OS's package manager already has prebuilt binary packages for them, look
inside muon.cabal to see what you need to install, and install those.</p>
<h2>Using Muon</h2>
<p>If you want a quick rundown of what you can do with Muon, have a look at
the README file in the darcs repo. If you're too lazy for that, just
access Muon's help:</p>
<div class="highlight"><pre><span></span>$ muon <span class="nb">help</span>
</pre></div>
<p>In reality, it doesn't matter what you type as the command - muon
outputs a help message for all inputs that aren't already a command. So
that's everything other than "generate", "init" and "upload".</p>
<p>It's worth noting that, at the moment, my server is hard coded into
Muon's upload command, so you'll have to rsync, scp or FTP the files to
your web server yourself, unless you happen to also have a web server
called "webserver" serving from /var/www/htdocs, which is entirely
possible.</p>
<h2>Lessons learnt</h2>
<p>Before writing Muon, I had the idea in my head that Haskell was useless
for writing anything useful. That is clearly not the case. In fact, I'd
say that my Haskell code is cleaner and easier to read (for me) than my
Haskell code is cleaner and easier to read by necessity - Haskell's
syntax demands it. Also, I find the "name arg = blah $ blah arg" easier to
read than "def name(arg): return blah(blah(arg))", due to an allergy to
parentheses.</p>
<p>There is always this talk of explaining monads using analogies and shit
to make them simpler or easier to understand, but the reality is simple:
monads are boxes you put types into. If you have an IO String, it's just
a String inside a box labeled IO you have to use \<- and fmap to get at.
Once that is realised, you can abandon all of this "monad is a burrito"
crap and actually start using them. I'm sure that doesn't count as an
analogy, if it does, I'm a complete hypocrite.</p>
<p>Anyway, Haskell isn't too hard to learn. Neither is Common Lisp. It gets
really hairy when you move onto Template Haskell and macros in Lisp, but
even then, once you get the hang of it, you'll be defining new syntax
constructs without much effort.</p>
<p>Happy Haskelling!</p>
How not to run a websiteurn:https-kaashif-co-uk:-2014-04-12-how-not-to-run-a-website2014-04-12T00:00:00Z2014-04-12T00:00:00ZKaashif Hymabaccus
<p>After a few months of running a website, there are a few things I have
realised about how I ran my server when it was first delivered, and how
I run it now. The changes have been, for the most part, for the better.
Needless to say, when I first started out, I was clueless, overeager and
far too ambitious with my plans for "the next Facebook" or something</p>
<!-- more -->
<p>equally stupid. Mistakes were made along the way, and I'll make sure I
never make them again.</p>
<h2>Using FTP</h2>
<p>Using FTP <em>at all</em> is a mistake, but my blunder was twofold - I had an
account with write access to my file store which was not backed up.
Worse yet, the account in question had a six character password
consisting of letters and numbers which could have been brute-forced
within seconds by a decent CPU.</p>
<p>Of course, when I first set up my FTP server, I was thinking more about
how cool it was to be able to transfer files using command line FTP
clients and how cool that was. It's not really <em>that</em> cool and one day,
the weight of the security vulnerabilities I had exposed came crashing
down on me. In every directory of
<a href="http://files.kaashif.co.uk">files.kaashif.co.uk</a>, there were an
"index.php" and "index.aspx" files containing links to some online Bible
shop. It could have been much worse - I could have been hosting my
site using FTP to some VPS somewhere and my site could have been turned
into an advert for Bibles. Luckily, my server was only set up to serve
static files, with no index pages and certainly no PHP or other
scripting languages installed.</p>
<p>The fix was simple, I stopped using FTP and went back to using the
chrooted Apache server that comes with OpenBSD 5.4. Of course, now that
it's going to be removed from base, I should probably switch to Nginx,
but that can wait - OpenBSD's Apache, while old and probably insecure,
isn't anywhere nearly as bad as having an FTP server with a weak
password.</p>
<p>I count myself lucky that I chrooted my account into the FTP user's home
directory. If I hadn't, the entire machine I was running the server o
would have been compromised and I'd have had to spend 10 minutes
spinning up a new one. Terrifying to think about.</p>
<h2>Backups</h2>
<p>As I mentioned, I didn't back much of anything up. Mostly, my backups
consisted of spreading files over every PC I owned in the hopes that
none of them would fail. This worked for static, never-changing files
like pictures and videos, but had no place as a strategy when dealing
with ever-changing code and documents. The first crisis I faced involved
a power cut. Now, as you may know, FreeBSD systems use the Unix File
System (UFS2) by default. While it's OK, a far better alternative is
ZFS. I was not using ZFS at the time of the power cut, and just happened
to be doing something disk IO intensive at the time. This was a recipe
for disaster, as this caused the file system on one of my USB disks to
die.</p>
<p>Eventually, the power came back on and I had to pick up the pieces.
After booting into single-user mode and fscking the disks, all but one
disk survived with only a few jumbled inodes here and there. The last
disk could not be recovered for some reason. I ran some data recovery
tools and I did manage to, eventually, save most of the data, but I
learnt three important lessons:</p>
<ul>
<li>Keep 3 copies of everything</li>
<li>Keep those copies on different storage media</li>
<li>Have some off-site backups</li>
</ul>
<p>I heard those somewhere (probably from /usr/games/fortune), and it seems
reasonable.</p>
<p>Nowadays, I tend to back my storage array to:</p>
<ul>
<li>Another storage array (this is more redundancy than backup)</li>
<li><a href="http://tarsnap.com">Tarsnap</a>, an online backup site</li>
<li>For the really static data, I have a pile of DVDs</li>
</ul>
<p>It is impossible for me to erase some of those backups without
physically destroying them, which can't be done by accident. I think
that, if there is another power cut, I'll be fine.</p>
<p>Those are the big two things I've done. It's worth noting that, since I
only deal in public key authentication, not transmitting passwords over
SSL, my site and SSH access to it was not affected by the recent OpenSSL
debacle. Perhaps there's an OpenSSH issue lurking in the shadows,
revealing private keys to hackers. That's unlikely, though, black hats
would have already exploited it if such a thing were possible.</p>
<p>In any case, my pile of DVDs are safe from electronic intrusion, unless
you count lightning.</p>
Creating a GNU/Linux distrourn:https-kaashif-co-uk:-2014-04-03-creating-a-gnu-linux-distro2014-04-03T00:00:00Z2014-04-03T00:00:00ZKaashif Hymabaccus
<p>UPDATE: The project died, it went nowhere.</p>
<p>On <a href="http://nixers.net">nixers.net</a>, the IRC channel of which I spend a bit of
time in, there has been a bit of a stir as the community tried to decide on a
project to commit to. The idea of creating a distro of some OS came up. A few
people wanted a BSD-based distro, but it was decided that the Linux kernel was</p>
<!-- more -->
<p>where the hardware support was, so that's the direction we went. After working
out that we wanted a musl libc based distro with a focus on minimalism, someone
drew our attentions to <a href="http://git.2f30.org/morpheus/">morpheus</a>, a GNU/Linux
distro that seems awfully similar to what we were doing, and had been
established for a very long time compared to our still-hypothetical project
(started in September 2013). After that, it seems that the buzz around making
our own distro has died down. Nevertheless, I'm still up for it, and played
around with building a few live USB images of TinyCore GNU/Linux (basically just
Linux plus a rather fully featured initrd) and FreeBSD (something more
substantial). Although I have nothing concrete yet, I have a few ideas on what I
want in a distro, and what my environment usually ends up like after I'm done
customising the package manager, programs, and so on.</p>
<h2>Package Manager</h2>
<p>This was a subject of much contention, but I think the best path for a new
distro run by a small community would not be to write our own package manager,
but to use one that already exists and works. I suggested <code>pkgsrc</code>, the NetBSD
package manager, which has been ported to GNU/Linux and has been proven to work
well. Indeed, I have personal experience using <code>pkgsrc</code> on Slackware, and I'm
confident it'd be perfect for our needs. It:</p>
<ul>
<li>has over 12000 packages ready for use</li>
<li>comes with a binary package manager, <code>pkgin</code></li>
<li>is a tried and tested system for packages</li>
<li>is bug-tested and securely audited by the NetBSD project</li>
</ul>
<p>Writing our own package manager would require a monumental effort if we ever
wanted to support anything more than a handful of programs, which is a large
failing of morpheus' ports tree, and a reason it can never be anything more than
a toy distro. Installing a package from source would be as simple as:</p>
<div class="highlight"><pre><span></span><span class="c1"># cd /usr/ports/category/program</span>
<span class="c1"># make install clean</span>
</pre></div>
<p>Similarly, installing from binary packages would require a simple:</p>
<div class="highlight"><pre><span></span><span class="c1"># pkgin install program</span>
</pre></div>
<p>This is assuming that we have built all packages, hosted them on a file server,
and pointed <code>pkgin</code> there by default, which is definitely doable.</p>
<p><code>pkgsrc</code> has ports for an X11 server, thousands of applications and scripts,
and, most importantly, a way to keep the tree up to date. All of these features
are things we'd have to poorly reproduce in custom, buggy, hacked-together
scripts if we wanted to roll our own solution. Thankfully, this is not
necessary.</p>
<h2>Init System</h2>
<p>A while back, there was a big kerfuffle about Debian switching to systemd, then
Ubuntu doing the same. This came as a shock to many, but to others, it
represents a necessary shift towards a more modern init system. While I wasn't
shocked, I wouldn't use systemd either. My belief is that /sbin/init should do
as little as possible before handing off init to less important userspace
programs (e.g. /etc/rc) and maybe waiting to run a shutdown script. This is
exactly what <a href="http://git.2f30.org/sinit/">sinit</a> does and it is incredibly
simple. We may end up writing our own init for fun and profit, but it'll be
very, very similar to sinit. The plan, I'd imagine, is to populate /etc/rc.d/
with shell scripts, similar in format to OpenBSD's or FreeBSD's, but with our
own /etc/rc.subr functions.</p>
<p>Alternatively, the init scripts could take the form of sysvinit services, which
are essentially self-contained shell scripts. The latter option is the easiest
to implement, even if the first isn't <em>that</em> hard to implement. In any case, the
distro will end up with:</p>
<ul>
<li>Minimal /sbin/init binary</li>
<li>Simple service files requiring little parsing</li>
<li>/etc/rc.conf based service activation</li>
</ul>
<p>Or at least something similar. Slackware's system of adding execute permissions
to init scripts to activate them is also something worth considering. Also, the
<code>chmod +x /etc/rc.d/service</code> could be aliased to <code>initctl enable service</code>, to
make it look like a lot of work has been done. At this point, we have basically
overtaken sysvinit in features, and are catching up with systemd rapidly.</p>
<h2>File System</h2>
<p>Back when a BSD kernel was still on the table, ZFS was also on the table.
Unfortunately, even in FreeBSD, ZFS root is experimental, and it's even more so
on Linux. Thus, ZFS is not an option, but btrfs may be. It's stable, has many of
the features of ZFS, and is used in many high-profile distros, like Fedora. In
the end, though, considering that ease-of-use needs to be a consideration, it
would probably be easier to use ext4. That said, btrfs is a very attractive file
system, but for now, ext4 is a lot easier.</p>
<h2>Default Shell</h2>
<p>The default shell, while not discussed at length yet, will no doubt be the
subject of many an argument on the IRC channel. After a few discussions, I got
the impression that others had the impression that zsh was bloated and slow, and
that OpenBSD's ksh derivative, pdksh, was lighter. While this may be true in
relative terms, I urged them to take into account than both shells used less
than 1 MB of memory and both were responsive, even on an old 1995 ThinkPad
running other programs in the background.</p>
<p>In fact, here is the line from top(1) for a zsh instance on my laptop right now:</p>
<pre><code>PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1457 kaashif 1 52 0 37172K 1724K ttyin 0 0:00 0.00% zsh
</code></pre>
<p>So you can see, zsh uses 1724K of memory. Not exactly "bloated", is it?</p>
<p>Evidently, givina users a choice of <em>modern</em> shells like bash and zsh is far more
important than saving the miniscule amount of memory switching to a gimped shell
like pdksh would give. Perhaps it'd be worth it on an embedded system, but that
is most certainly not the target audience.</p>
<h2>The Plan</h2>
<p>Taking into account what I have written here, I think the distro is headed in a
good direction, provided development ever gets off the ground. While there are
some disagreements about specifics, the overall sentiment is one of agreement,
considering the sorts of people that congregate on #nixers (nee #unixhub) and
their desires: minimalism, pragmatism and utility.</p>
<p>I hope our hacking will be happy and productive.</p>
Vimurn:https-kaashif-co-uk:-2014-03-01-vim2014-03-01T00:00:00Z2014-03-01T00:00:00ZKaashif Hymabaccus
<p>When I first started programming, I barely had any idea of what constituted a
good text editor, or why I'd want to use some old, texty editor from the 90s
which didn't even have most of the features I took for granted in the IDE I was
using at the time. Maybe this had something to do with one of my first languages
being Java, which is widely considered an IDE language, but I went through the</p>
<!-- more -->
<p>same thing when I started learning C and Python. Why take all this time to learn
Vim when I could just open Gedit or Kate and get to work immediately?</p>
<p>As always, with this type of "Vim changed my life" blog post, I could go on to
describe my experience of slogging through hours of using Vim, getting used to
it, then wondering how I ever coped using terrible editors which weren't even
modal and didn't even have macros. I won't do that, since I'd just be
regurgitating very generic stuff. Instead, I'll distill all of that down into a
few points:</p>
<ul>
<li>Modal editing</li>
<li>Text objects and motions</li>
<li>Plugins</li>
</ul>
<p>Modal editing is actually very simple. At its most basic level, there are three
modes (there are really more, but those aren't as important) - normal mode,
insert mode and command line mode. Normal mode is where one should spend most of
their time - it's where you move around the text, yank and put (copy and paste),
execute macros and so on. Insert mode is where you type text and it is written
to the file you have open. While in normal mode, typing "dw" would delete a
word, doing the same in insert mode actually inserts a literal "dw" into the
text. A mistake most beginners make is trying to spend all of their time in
insert mode, because it's the most similar to what they already know (Notepad,
Gedit, etc). Command mode is where you type commands into a command line prompt,
which is needed for more complex commands which can't be accessed through key
combinations unless you remap keys manually.</p>
<p>Text objects are key combinations that represent structures found in text. For
example, "i(" represents the text inside brackets and "t." represents the text
up to the next full stop. Motions are keys you press to move around text - there
are the usual "hjkl" keys to move left, down, up and right, but also "w" and "gg",
to move to the next word or the start of the file. There are many, many more
objects and motions.</p>
<p>Plugins are scripts which extend Vim's functionality. You are probably already
familiar with plugins or extensions for web browsers, so there isn't really any
need to explain what they are or do, suffice it to say that they add or improve
Vim's features.</p>
<h2>Vim in action</h2>
<p>Reading about text editing is all well and good, but it doesn't allow you to
really get a full grasp of the power of Vim. Neither does looking at a static
series of screenshots - you can't really see a comparison between your current
editor and Vim unless you see it in action. So here are some GIFs of Vim in
action, demonstrating the power I mentioned.</p>
<h3>Underlining text</h3>
<p><img src="/static/vim1.gif" alt="" /></p>
<p>If you're using Markdown or reStructuredText or anything similar, it's a common
task to "underline" a title by putting a few equals signs or dashes under it.
Using Notepad (for instance), you'd probably press enter to go to the next line,
mash "=" until you get the desired look, and call it a day there. In Vim, you
can yank the line, put it on the next line, and replace that line's characters
with "=". This is quick and simple, but does seem like a bit much to learn just
to underline some text, especially when this sequence in Vim translates to a key
combination of "yypv$r=o".</p>
<h3>Repeating commands</h3>
<p><img src="/static/vim2.gif" alt="" /></p>
<p>In Vim, any sequence of actions you do can be recorded to a macro. It's very
easy, you just press "q" followed by the key you want the macro to be assigned
to (I always choose "w" since it's next to "q" on most keyboards), then do
whatever it is you want to do, and press "q" again to stop the recording. Let's
say I recorded the action of underlining a title into a macro. If I want to
repeat the macro 3 times, I just type "3@w". If I want to repeat it 1000 times,
I'd just type "1000@w". This comes in very useful for more complicated macros,
like if you wanted to format a large list into a nicely formatted table.</p>
<h3>Changing text within delimiters</h3>
<p><img src="/static/vim3.gif" alt="" /></p>
<p>This is another very common problem - haven't you ever wanted to change the text
inside quotes, or inside brackets? The answer to that question is always yes,
and Vim has a very easy way of doing it. Changing the text inside double quotes
is simply "ci"", for "change inside " ". Notice how I didn't have to move to the
first quote - Vim moves automatically to the first quote on the same line. You
can do this with any other delimiter, like curly braces, single quotes, but also
for blocks and sentences. When I say "block", I'm referring to a block of a
program, which is a useful text object to have when your blocks aren't delimited
by curly braces.</p>
<h3>Using regular expressions</h3>
<p><img src="/static/vim4.gif" alt="" /></p>
<p>You might be familiar with regular expressions, so you might not be surprised to
hear that Vim can use them extensively - not a surprise for a program that deals
with text. If not, this is a quick demonstration of reversing the lines of a
file. You might find this useful if you're viewing a log file and you want the
lines to be from newest to oldest, but the point is that Vim has a native regex
engine that can be accessed quickly in command mode by using ":g".</p>
<h3>Git integration</h3>
<p><img src="/static/vim5.gif" alt="" /></p>
<p>Lots of programmers use Git, so it's natural that Vim, a programmer's tool, has
a variety of plugins that claim to integrate with Git. Fugitive is one of the
best, in my opinion. You can see how effortless it is to view the authors of
every line in a file (although in this case, all of the authors are "kaashif"),
add the file to the staging area using ":Gwrite" and commit it, using
":Gcommit". I don't see how it could get any easier, especially considering the
tab completion.</p>
<p>This was by no means a Vim tutorial, just a demonstration of what one can do if
you know Vim, and a pretty limited one at that. Learning the basics of Vim is
quite easy, all one needs to do (assuming Vim is installed) is run the "vimtutor"
program in a terminal and it'll walk you through the basics. If you're on
Windows, you can go to <a href="http://www.vim.org/download.php">the Vim website</a> and
download their installer for gVim, which is Vim with a slightly graphical
interface. After that, I think you can run "vimtutor" from cmd.exe, but I'm not
sure.</p>
<p>I'm tempted to sign off by saying "happy hacking!", but that's far too
cheesy. Happy editing!</p>
My Desktopurn:https-kaashif-co-uk:-2014-02-09-my-desktop2014-02-09T00:00:00Z2014-02-09T00:00:00ZKaashif Hymabaccus
<p>Over the years, I've used a few different OSes and desktop environments, and the
one I use currently is portable to many operating systems, mostly due to the
efforts of the writers of i3 over at i3wm.org, but also the
standards-compliance of POSIX, meaning that my shell scripts (which you can find
<a href="https://gitorious.org/kaashif-config-files/dotfiles/">here</a>) work on all</p>
<!-- more -->
<p>operating systems worth using (naturally, this excludes Windows).</p>
<p>EDIT: That link doesn't work. <a href="http://kaashif.co.uk/cgit/dotfiles">This</a>
does.</p>
<p>You might already know how to manage dotfiles effectively with GNU Stow, because
that's exactly how I manage mine - it allows me to install the config files I
need, and no more. This isn't what I want to write about, though, I've already
written something about that - I want to show off the final result of my
dotfiles - my desktop.</p>
<p><a href="/static/desk1.png"><img src="/static/desk1.png" alt="" /></a></p>
<p>This is what my blank desktop looks like. There isn't much to look at, and I
haven't put any fancy conky widgets or anything there because I hardly ever just
look at my wallpaper anyway. That's the Haskell logo, it seems as good a logo as
any to have on a desktop I never see (other choices include Puffy, Tux and
Beastie, all OS logos).</p>
<p>At the top is a bar, created by piping conky output into dzen2. The music player
is mpd (music player daemon), which I can control using my phone, command line
clients, GUI clients, scripts, and so on. Some might say running a service to
play music is overkill, but it's actually more lightweight than most solutions,
and is very Unixy, too.</p>
<p><a href="/static/desk2.png"><img src="/static/desk2.png" alt="" /></a></p>
<p>If there is an application I want to launch that I haven't already bound to some
keyboard chord (for example, Super-C for Firefox and Super-G for gVim), I use
dmenu to get to it. I had to apply a plethora of patches to get dmenu to support
colour changing, Xft fonts, and variable height/width. It turns out that you can
get a pre-patched version of dmenu <a href="https://bitbucket.org/melek/dmenu2">here</a>.
While it is called "dmenu2", it's only dmenu with a few patches, and it's not an
official suckless.org tool anyway. All of the output on he bar comes from all
the normal Unix status programs - df, uptime, date, and so on. I don't remember
where I got the icons, but I saw <a href="https://github.com/windelicato">Bill Indelicato</a>
using icons which looked similar, so maybe I got them there or we both got them
from the same place.</p>
<p><a href="/static/desk3.png"><img src="/static/desk3.png" alt="" /></a></p>
<p>Using the magic of mpc (media player client) and dmenu, I managed to get dmenu
to create a randomly shuffled list of artists in my music collection. When I
select one of them, the current playlist is replaced by the discography of the
chosen artist. It's very convenient, only a Super-Shift-D away - far faster than
opening a music player, going to a list of artists... I still do that, but less
often, since most of the time I just want some randomly chosen music, not
anything specific.</p>
<p><a href="/static/desk4.png"><img src="/static/desk4.png" alt="" /></a></p>
<p>I'd like to say this is what my laptop usually looks like, but the reality is
that I've just opened some random files in /usr/include with gVim. Well, this
might be what a kernel hacker's laptop looks like sometimes - mine is mostly
just the one shell script or Haskell program open, and usually a browser with
Reddit or 4chan on the side. The window manager is i3, by the way, a WM I like
because of its simple and efficient <em>manual</em> tiling - none of that XMonad
dynamic crap, way too inflexible and sometimes annoying.</p>
<p>I hope this has given you some insight into how I use my laptop, but
realistically, you've probably just commented on the similarities between my
desktop and someone on /g/'s. Fair enough, I haven't worked hard enough on my
desktop for it to be very unique, and I don't intend to, because my setup works
for me, and lets me work without using a mouse, so it's a win-win-win for me, my
"hacker cred" (I hear the kids using Ruby and node.js say that nowadays) and my
laziness.</p>
Functors in Haskellurn:https-kaashif-co-uk:-2014-02-05-functors-in-haskell2014-02-05T00:00:00Z2014-02-05T00:00:00ZKaashif Hymabaccus
<p>Whenever you hear something about Haskell, chances are it sounds arcane and
involves lots of complicated and intimidating mathematical language. Well, the
truth is that all this talk of 'endofunctors' and 'monoids' is really
unnecessary, if the concept of functors is explained using a simple analogy.</p>
<!-- more -->
<p>If you have a situation where a type contains other types, you are dealing with
a functor. For example, when you have a list of integers, the list is the
functor. If you have a herd of sheep, the herd is the functor. At its core, this
is what a functor is - something which contains types and can be mapped over.
That last part is crucial and is central to how functors are defined in Haskell,
but are not central to how one should think about functors.</p>
<h2>In practice</h2>
<p>At a GHCi prompt, you can import a few modules here and there with the word
'functor' in them, but at this stage, you do not require any of the more complex
functions, you only need one - fmap. The functor we'll be using as an example is
the 'Maybe' functor, which contains either a single value ('Just' the value) or
Nothing. If this is confusing to you, just think of the 'Just' constructor as
representing a box you put types into, and the 'Nothing' constructor as
representing an empty box. So 'Just 1' can be represented as 'put the value 1
into a box'. This isn't entirely contrived, the 'Maybe' functor does have a use
in functions that can fail, so can return an empty box or a box with a result in
them.</p>
<p>Let's examine the type of 'Just' using ':t'.</p>
<div class="highlight"><pre><span></span><span class="kt">Prelude</span><span class="o">></span><span class="w"> </span><span class="kt">:</span><span class="n">t</span><span class="w"> </span><span class="kt">Just</span><span class="w"></span>
<span class="kt">Just</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="ow">-></span><span class="w"> </span><span class="kt">Maybe</span><span class="w"> </span><span class="n">a</span><span class="w"></span>
</pre></div>
<p>If you are familiar with Haskell syntax, you should know that this means that
'Just' takes any type and turns it into a 'Maybe' functor containing that type.
No surprises here, that is the definition of a functor.</p>
<p>Let's say we have a few functions that return numbers contained in a Maybe
functor, and we want to combine them in some way. When mapping over a single
functor, the function you use is fmap, like so:</p>
<div class="highlight"><pre><span></span><span class="kt">Prelude</span><span class="o">></span><span class="w"> </span><span class="n">fmap</span><span class="w"> </span><span class="p">(</span><span class="o">+</span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">Just</span><span class="w"> </span><span class="mi">3</span><span class="p">)</span><span class="w"></span>
<span class="kt">Just</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
</pre></div>
<p>This is fine for function that only take 1 argument, but remember, we want to
apply this to multiple arguments. If we define a function like so, which takes
three (non-Maybe) numbers and adds them together, it becomes obvious that using
fmap on its own won't work:</p>
<div class="highlight"><pre><span></span><span class="kt">Prelude</span><span class="o">></span><span class="w"> </span><span class="kr">let</span><span class="w"> </span><span class="n">f</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="n">z</span><span class="w"> </span><span class="ow">=</span><span class="w"> </span><span class="n">x</span><span class="o">*</span><span class="n">y</span><span class="o">*</span><span class="n">z</span><span class="w"></span>
</pre></div>
<p>If we just try to fmap our three-operand function over a Maybe functor, and
examine its type, we see the following:</p>
<div class="highlight"><pre><span></span><span class="kt">Prelude</span><span class="o">></span><span class="w"> </span><span class="kt">:</span><span class="n">t</span><span class="w"> </span><span class="p">(</span><span class="n">fmap</span><span class="w"> </span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">Just</span><span class="w"> </span><span class="mi">3</span><span class="p">))</span><span class="w"></span>
<span class="p">(</span><span class="n">fmap</span><span class="w"> </span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">Just</span><span class="w"> </span><span class="mi">3</span><span class="p">))</span><span class="w"> </span><span class="ow">::</span><span class="w"> </span><span class="kt">Num</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="ow">=></span><span class="w"> </span><span class="kt">Maybe</span><span class="w"> </span><span class="p">(</span><span class="n">a</span><span class="w"> </span><span class="ow">-></span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="ow">-></span><span class="w"> </span><span class="n">a</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>This means that fmap has transformed our function which took things of typeclass
Num into a partially applied function, which still takes things of typeclass
Num, but the function itself is within the box of the Maybe functor. This means
that we want to map a Maybe function over a Maybe Num. If the function takes yet
another argument, doing this will return yet another function. If not, it'll
give us the answer, wrapped in a functor. If we want to apply a function with
multiple arguments to functors, we need to use Applicative functors.</p>
<div class="highlight"><pre><span></span><span class="kt">Prelude</span><span class="o">></span><span class="w"> </span><span class="kr">import</span><span class="w"> </span><span class="nn">Control.Applicative</span><span class="w"></span>
</pre></div>
<p>And then, we can use the <code><*></code> operator, which takes a function in a functor and
applies it to a value in a functor. We can use it in more trivial cases like so:</p>
<div class="highlight"><pre><span></span><span class="kt">Prelude</span><span class="o">></span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="p">(</span><span class="o">*</span><span class="mi">4</span><span class="p">)</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="mi">3</span><span class="w"></span>
<span class="kt">Just</span><span class="w"> </span><span class="mi">12</span><span class="w"></span>
</pre></div>
<p>We can see that <code>(*4)</code> takes one value, so we are given the answer we want.
Applying this to our situation gives us this:</p>
<div class="highlight"><pre><span></span><span class="kt">Prelude</span><span class="o">></span><span class="w"> </span><span class="p">(</span><span class="n">fmap</span><span class="w"> </span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">Just</span><span class="w"> </span><span class="mi">3</span><span class="p">))</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
<span class="kt">Just</span><span class="w"> </span><span class="mi">60</span><span class="w"></span>
</pre></div>
<p>We can improve the look of this ugly statement by using fmap as an infix
function.</p>
<div class="highlight"><pre><span></span><span class="kt">Prelude</span><span class="o">></span><span class="w"> </span><span class="n">f</span><span class="w"> </span><span class="p">`</span><span class="n">fmap</span><span class="p">`</span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
</pre></div>
<p>Because this is such a common use case, there is some syntactic sugar provided -
fmap can be replaced with <code><$></code>.</p>
<div class="highlight"><pre><span></span><span class="kt">Prelude</span><span class="o">></span><span class="w"> </span><span class="n">f</span><span class="w"> </span><span class="o"><$></span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="o"><*></span><span class="w"> </span><span class="kt">Just</span><span class="w"> </span><span class="mi">5</span><span class="w"></span>
</pre></div>
<p>At this point, we just have the function and its arguments as we usually would,
just with some extra operators thrown in. The most important thing you should
take away from this is that using Applicative is far better than using liftM and
its numbered equivalents, as Applicative can be applied to an arbitrary number
of arguments. Monads are useful in other cases, and Maybe is actually a monad,
but thinking about it like a monad rarely helps.</p>
Updating DNS recordsurn:https-kaashif-co-uk:-2014-01-22-updating-dns-records2014-01-22T00:00:00Z2014-01-22T00:00:00ZKaashif Hymabaccus
<p>When running a website on a residential connection, a problem one might run into
is the dynamic IP address usually assigned by one's ISP. There are a few dynamic
DNS services which basically let you have a subdomain (e.g.
mydomain.example.com) and let you update it to point to your IP address whenever
it changes. At one time, your IP might be 10.0.0.1, and your domain correctly</p>
<!-- more -->
<p>resolves to 10.0.0.1, but after you reboot your modem or router, your IP may
change to 10.0.1.2 and your domain will still point to your old IP (thus won't
work), until your dynamic DNS client somehow updates the A record of your domain
when your IP changes.</p>
<p>Assuming that getting a static IP is impossible, there is only one solution:
dynamic DNS. While it is possible to transfer your records to someone other than
your domain registrar to manage DNS records, I just keep all things related to
my domain with Namecheap, which is where my domain was registered.</p>
<h2>Choosing a client</h2>
<p>Several websites, DynDNS and NoIP being examples, offer their own clients. You
can install these, put your site-specific username and password into a config
file, and let your DNS be handled automatically. Usually, these programs can do
a lot more than update your DNS, and are needlessly complicated if that's all
you want to do. The same goes for most multi-service dynamic DNS clients, for
example, ddclient.</p>
<p>Most services provide a URL with which you can update your A records. Fetch this
URL, and the A records on your domain will be changed to match the IP you
fetched the URL with. It's a very simple system, and can be automated within
minutes, with a simple script. Because I like to keep my servers bare, and avoid
installing anything bulky (like Python, Ruby, and the like), I'm going to ues
Perl, which is in the base system of every good *BSD, and comes installed by
default on most GNU/Linux distros.</p>
<h2>Writing the client</h2>
<p>There is some Perl boilerplate we have to get out of the way - the shebang and
some "use" statements.</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env perl</span>
<span class="k">use</span> <span class="nn">strict</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">warnings</span><span class="p">;</span>
</pre></div>
<p>It's worth commenting on my use of "env". On FreeBSD (the OS my servers use),
Perl is located at /usr/local/bin/perl, while on most GNU/Linux systems, it's at
/usr/bin/perl. Using "env" avoids the issue of finding the Perl binary, and is
more cross-platform.</p>
<p>Next, we have to find the URL, password, hosts, and domain we will use. Namecheap uses
the word "host" to mean the subdomain, e.g. the "www" part of "www.fsf.com". For
this example, let's just say we want "ftp", "www" and "@", "@" being no
subdomain, e.g. "mydomain.com".</p>
<div class="highlight"><pre><span></span><span class="k">my</span> <span class="nv">$password</span> <span class="o">=</span> <span class="s">"blahblahblah"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$domain</span> <span class="o">=</span> <span class="s">"mydomain.com"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">@hosts</span> <span class="o">=</span> <span class="p">(</span><span class="s">"ftp"</span><span class="p">,</span> <span class="s">"www"</span><span class="p">,</span> <span class="s">"@"</span><span class="p">);</span>
<span class="k">my</span> <span class="nv">$update_url</span> <span class="o">=</span> <span class="s">"http://dynamicdns.com/update?domain=$domain&password=$password&host="</span><span class="p">;</span>
</pre></div>
<p>Notice we didn't attempt to put the hosts into the initial definition of the
update URL. We will do that when we loop over the array and fetch the URL using
cURL, thus updating the domain.</p>
<div class="highlight"><pre><span></span><span class="k">foreach</span><span class="p">(</span><span class="nv">@hosts</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">my</span> <span class="nv">$final_url</span> <span class="o">=</span> <span class="s">"$update_url$_"</span><span class="p">;</span>
<span class="k">my</span> <span class="nv">$output</span> <span class="o">=</span> <span class="sb">`curl -s "$final_url"`</span><span class="p">;</span>
</pre></div>
<p>So far, we have fetched the URL, so the IP of the host's A record should be
updated at this point. We still need to check for errors and such, so let's do
that next.</p>
<div class="highlight"><pre><span></span> <span class="k">if</span> <span class="p">(</span><span class="nv">$output</span> <span class="o">=~</span><span class="sr"> /<ErrCount>0/</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">print</span><span class="p">(</span><span class="s">"Update of $_.$domain succeeded."</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">else</span>
<span class="p">{</span>
<span class="k">print</span><span class="p">(</span><span class="s">"Update of $_.$domain failed!"</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>It's possible to use XML::Simple and actually parse the XML output by the server
handling DNS updates, but that would be overkill and waste everyone's time. So I
just regex the output for errors, it works fine, and the problem is generally
obvious if it fails, which is why I didn't print the output. It's perfectly
possible to add some verbosity and do all of this "correct" stuff, but no-one,
not even you, will ever care (unless you just do it for practice).</p>
<p>So save this script in /usr/local/bin/dnsup, or somewhere else appropriate and
memorable.</p>
<h2>Scheduling updates</h2>
<p>Your IP address can change instantly at any moment, so you must balance the
downtime users experience with the practical considerations of running a script
repeatedly. Personally, I run the script every 10 minutes. My IP only seems to
change once in a blue moon, and when it does (due to me rebooting the firewall
or router), I tend to notice right away, get impatient, and run the script
manually. That's beside the point, though, just add this to your /etc/crontab to
make it run every 10 minutes.</p>
<pre><code>*/10 * * * * nobody /usr/local/bin/dnsup
</code></pre>
<p>I run it as "nobody" to avoid any unnecessary root usage. What if you made a
mistake in the script and overwrote / with garbage? A program having privileges
is bad, don't do it.</p>
<p>At this point, you should be free of any possibly proprietary, bloated DNS
clients and you should have an automatically updating domain.</p>
What's a monoid?urn:https-kaashif-co-uk:-2014-01-14-what-s-a-monoid2014-01-14T00:00:00Z2014-01-14T00:00:00ZKaashif Hymabaccus
<p>If you have spent time on programming or technology boards like /g/ or
/r/programming, chances are you might have heard the word "monad" thrown around
a lot. You may have even heard the oft misquoted phrase "A monad is a monoid in
the class of endofunctors" intended to be a joke, or to scare programmers away
from scary functional languages like Haskell. The truth is that monads aren't</p>
<!-- more -->
<p>even slightly the same sort of thing in Haskell, but they do relate somewhat.
Monoids are, in actual fact, very, very simple.</p>
<h2>Sets and operators</h2>
<p>Let's take an example: suppose you have a set of things <em>S</em> and a single binary
operator. This binary operator takes two things from your set and outputs a
third thing, which is also in your set. There is no pair of operands for which
the result is outside the set you started with - the set is closed. The
set also contains an identity element - there exists an element which does not
change other elements when combined with them, using your binary operator. The
operator must also be associative, i.e. the order in which the operations are
evaluated does not change the results. These laws are expressed more clearly
as follows:
$$S \circ S \to S$$
$$i \circ e = e \text{ for all } e \text{ in } S$$
$$a \circ (b \circ c) = (a \circ b) \circ c$$</p>
<p>We could make this an even more relatable example by using the set of real
numbers and the addition operation, forming a monoid and satisfying the laws as
follows:
$$\mathbb{R} + \mathbb{R} \to \mathbb{R}$$
$$0 + e = e \text{ for all } e \text{ in } \mathbb{R}$$
$$a + (b + c) = (a + b) + c$$</p>
<p>This means that whenever you're adding real numbers (which we all do on a daily
basis), you're utilising monoids. Notice that subtraction and division are not
associative, so cannot form monoids. We aren't limited to sets of numbers
either, you can form monoids from matrices, vectors and functions. Indeed, a set
can contain any sort of structure, meaning monoids can be very general or very
specific, depending on the sort of structures it contains. For example, defining
a monoid of numbers and addition is useful in a small number of cases, while a
monoid of functions and function composition is very useful in many cases.
Monoids can be generalised, and the resulting generalisation will be very
useful, generally.</p>
<h2>Generalising monoids</h2>
<p>While it is all well and good having a single set and a single binary operator,
which maps members of that one set to members of the same set, it would be
useful to have a structure defined as an arbitrary number of sets and an
arbitrary number of functions. We could define such a structure as
consisting of the following:</p>
<ul>
<li>A set of sets</li>
<li>A set of functions mapping sets to other sets</li>
</ul>
<p>In some cases, rather than having sets of sets within our structure, we may want
to use magmas or semigroups, or something unrelated to groups entirely. For this
reason, it is better to say that our new structure contains a set of objects, and
morphisms between these objects. "Object" is a term which refers to any
algebraic structure, and "morphism" refers to any possible mapping between these
objects. So now, our structure consists of:</p>
<ul>
<li>A set of objects</li>
<li>A set of morphisms mapping objects to other objects</li>
</ul>
<p>This structure is not all that different to a monoid, and it becomes very
obvious if we say a monoid merely consists of:</p>
<ul>
<li>A singleton, containing a single set (e.g. all real numbers, all functions)</li>
<li>A single morphism (e.g. addition, function composition)</li>
</ul>
<p>Since there is only one object, the morphism can only map the object to itself,
thus is always an endomorphism. We can also think about the binary operation
(e.g. addition) as a set of unary operations (e.g, add 1, add 2, add 3), and
see the monoid as a set <em>S</em>, and a set of unary functions equal in order to <em>S</em>.
We get the same result whether we think about a single endomorphism or multiple,
so it doesn't really matter.</p>
<p>What we are slowly working towards, as our definitions get progressively more
general, is a category. Categories are incredibly useful and can become so
general that they are used to formalise <em>all</em> of mathematics. They actually have
uses too, such as considering the objects as representing types in a programming
language, and the morphisms as representing functions.</p>
Writing Unix manual pagesurn:https-kaashif-co-uk:-2014-01-08-writing-unix-manual-pages2014-01-08T00:00:00Z2014-01-08T00:00:00ZKaashif Hymabaccus
<p>There are a few very important things that everyone involved in software
(particularly free software) can do to help out. The most important is to file
detailed and helpful bug reports, so the developers working on your favourite
program can get the problem fixed. Since it is not very hard to write a bug
report, and projects generally have their own bug report guidelines, I won't</p>
<!-- more -->
<p>talk about that. The second most important thing you can do is writing guides,
manuals and the like, in order to get more people using the software without
headaches. Readily-available documentation not only saves the users time, but
helps the developers to spread their software, since it might develop a
reputation for being easy to learn to use, or any number of other things.</p>
<p>I'm a bit biased, since I'm a Unix user, but I think that a good place to start
is to write manual pages. There have been a few times when I was trying to do
something with a program, I had trouble, typed <code>man program</code>, saw that there was
no manual page, and installed another program, one that did not force me to
rely on guides written on blogs and such, which may not be accurate or up to</p>
<p>usually write one in Pod, compile it into groff, and submit it to the devs
upstream. It's not hard, in fact, it's very easy.</p>
<h2>What does a manual page look like?</h2>
<p>Here is the manual page for write(1), a pretty vanilla Unix command.</p>
<pre><code>WRITE(1) BSD General Commands Manual WRITE(1)
NAME
write -- send a message to another user
SYNOPSIS
write user [tty]
DESCRIPTION
The write utility allows you to communicate with other users, by copying
lines from your terminal to theirs.
When you run the write command, the user you are writing to gets a mes-
sage of the form:
...
</code></pre>
<p>Most man pages follow the convention of "NAME", "SYNOPSIS", "DESCRIPTION", "SEE
ALSO", and optionally "AUTHOR" and "BUGS".</p>
<h2>What does this look like in Pod?</h2>
<p>Pod is very simple and was designed to be used for Perl. Despite that, it is
very readable and it should be possible to get the gist of what this Pod markup
means just by looking at it:</p>
<div class="highlight"><pre><span></span> <span class="o">=</span><span class="n">head1</span> <span class="n">NAME</span>
<span class="n">program</span> <span class="o">--</span> <span class="n">does</span> <span class="n">something</span>
<span class="o">=</span><span class="n">head1</span> <span class="n">SYNOPSIS</span>
<span class="n">program</span> <span class="p">[</span><span class="o">-</span><span class="n">o</span> <span class="n">I</span><span class="o"><</span><span class="n">optional</span> <span class="n">argument</span><span class="o">></span><span class="p">]</span> <span class="n">positional</span> <span class="n">argument</span>
<span class="o">=</span><span class="n">head1</span> <span class="n">DESCRIPTION</span>
<span class="n">Describes</span> <span class="n">what</span> <span class="n">program</span> <span class="n">does</span> <span class="n">in</span> <span class="n">more</span> <span class="n">detail</span><span class="o">.</span>
<span class="o">-</span><span class="n">o</span><span class="p">,</span> <span class="o">--</span><span class="n">optional</span>
<span class="n">describes</span> <span class="n">what</span> <span class="n">option</span> <span class="n">does</span>
<span class="o">=</span><span class="n">head1</span> <span class="n">SEE</span> <span class="n">ALSO</span>
<span class="n">ls</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="nb">write</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="o">=</span><span class="n">head1</span> <span class="n">AUTHOR</span>
<span class="n">Kaashif</span>
<span class="o">=</span><span class="n">head1</span> <span class="n">BUGS</span>
<span class="n">Report</span> <span class="n">bugs</span> <span class="n">to</span> <span class="n">rms</span><span class="nv">@gnu</span><span class="o">.</span><span class="n">org</span>
</pre></div>
<p>As you can see, "=head1" denotes a top level header. This is all we really need
for writing a simple manual page. To convert it to groff, you can use the
following command:</p>
<div class="highlight"><pre><span></span>$ pod2man my_first_page.pod
</pre></div>
<p>And a file called "my_first_page.man" will be created with the appropriate
groff markup. It is my opinion that Pod is much better than groff if we're just
writing a manual page, and this can be verified if you look at the groff output
- it's not very readable unless you know your groff.</p>
<p>You might not always want to create manual pages, you can use <code>pod2html</code> and
<code>pod2latex</code> to create web pages and LaTeX source which can be put into another,
longer document. In fact, I could have written this post in Pod (but I didn't),
it's very powerful, considering its simplicity.</p>
Reviving an old ThinkPadurn:https-kaashif-co-uk:-2013-12-23-reviving-an-old-thinkpad2013-12-23T00:00:00Z2013-12-23T00:00:00ZKaashif Hymabaccus
<p>While I did have some old hardware lying around, I had never committed to
actually getting that hardware usable. By that, I mean I had never tried to
browse the web, read emails and that sort of day-to-day stuff on anything older
than a few years. To see if it were really possible, I decided to buy an old
ThinkPad (a 760EL from 1995) and see if I could get it working. Before I</p>
<!-- more -->
<p>started, I remembered that people like <a href="http://kmandla.wordpress.com/">K.Mandla</a>
had already done things like this before, but he had such luxuries as 32 MB of
RAM, a CD drive, a <em>USB port</em>. Really, he was pushing the boundaries of what
could be considered low-end (right?). Here is what my ThinkPad 760EL had when I
started off (I have upgraded it a bit since then):</p>
<ul>
<li>Pentium I 133 MHz</li>
<li>16 MB RAM</li>
<li>2.1 GB hard drive</li>
<li>Trident TGUI 9660 graphics</li>
<li>3.5 inch floppy drive</li>
<li>800x600 TFT display</li>
</ul>
<p>The lack of network support and any removable mass storage meant installing
anything was impossible, so I bought a 3Com Etherlink III PCMCIA Ethernet card,
which is well supported by both NetBSD and OpenBSD, the two candidates for the
OS that would eventually be on my laptop.</p>
<h2>Installing OpenBSD</h2>
<p>The minimum memory requirements for OpenBSD 5.4, as listed in INSTALL.i386, are
either 24 or 32 MB of RAM. NetBSD 6.1.2 requires 20 MB. I had only 16. After
some consulting with misc@, I was advised to either get more memory or install
an older release. Since the latter would involve running 5+ year old unsupported
software (a terrible idea in all cases), I decided to open up my ThinkPad and
take a look at the RAM modules. After reseating the only module to make sure I
was familiar with the process, I rebooted to find that...I had 32 MB of RAM?!
Apparently, 16 MB was soldered on the mainboard and another 16 was removable.</p>
<p>That was very fortunate, because that meant I could install OpenBSD 5.4! I could
have also gone the NetBSD route, but it required 5 floppies, while OpenBSD only
required one (floppies are actually quite expensive nowadays). The installation
went well - my NIC was automatically detected and configured and the sets
downloaded and extracted without a hitch.</p>
<h2>Configuring the text console</h2>
<p>OpenBSD doesn't have much framebuffer support, and there isn't much interest in
writing drivers for images (and other complex 2D graphics) in the framebuffer
either. This means we're limited to text unless we set up X, which performs
abysmally on this hardware. The first order of business is getting more
characters on the screen. There are instructions to do that <a href="http://www.openbsd.org/faq/faq7.html#80x50">here</a>. It boils down to doing three things:</p>
<ul>
<li>Loading a half-height font</li>
<li>Deleting all the screens configured to use the full-height font</li>
<li>Creating new screens which use the new font</li>
</ul>
<p>Here is what the relevant part of my /etc/rc.local looks like:</p>
<pre><code>wsconscfg -dF 1
wsconscfg -dF 2
wsconscfg -dF 3
wsconscfg -t 80x50 1
wsconscfg -t 80x50 2
wsconscfg -t 80x50 3
</code></pre>
<p>So I just did that for screens 1, 2, and 3, so I can now see twice the amount of
text.</p>
<h2>Text Applications</h2>
<p>The OpenBSD project provides an extensive collection of packages, including web
browsers and text editors which can function in the text console. Here are a few
applications I make use of:</p>
<ul>
<li>tmux (terminal multiplexer)</li>
<li>vim (the best text editor)</li>
<li>elinks (text web browser)</li>
<li>mutt (email client)</li>
<li>ssh (remote shell access)</li>
</ul>
<p>Using those, I can access the web. check my mail, and write these posts. There
is not much else I need to do on a day-to-day basis.</p>
<h2>Getting X to work</h2>
<p>Unsurprisingly, I could not find much documentation on getting an X server to
work on a Trident video card from 18 years ago. This meant I had to fiddle
around with the files in xorg.conf.d a bit. <code>X -configure</code> segfaults every time
I run it, meaning it's not much help. That doesn't matter, though, because the
configuration file it generates only does the obvious - it changes the driver to
"trident", sets up the screen with the correct resolution, these are all things
I already knew. To get my card to work, I had to add several options to the
"Device" section. Here is what that part of my config looked like:</p>
<pre><code>Section "Device"
Identifier "gfxcard"
Driver "trident"
Option "NoAccel" "True"
Option "ShadowFB" "Enable"
Option "NoPciBurst" "Enable"
Option "FramebufferWC"
EndSection
</code></pre>
<p>To clarify, this did not go into my xorg.conf, I created a new file in
/usr/X11R6/share/X11/xorg.conf.d called 99-trident.conf. The "99" ensured that
this file would be sourced last, and would override any other device settings.
Documentation on what these options do can be found
<a href="http://www.x.org/releases/X11R7.5/doc/man/man4/trident.4.html">here</a>. If the
resolution is incorrect (it wasn't in my case), you may have a timing issue,
which can be fixed by adding one of these options:</p>
<pre><code>Option "UseTiming1024" # For 1024x768
Option "UseTiming800" # For 800x600
</code></pre>
<p>With these settings, one finds that the performance of X is very, very bad on
this hardware. After spending hours getting it to work, I decided to stick to
the text console, where perfomance was far greater. I felt it was worth the
tradeoffs (fonts, resolution, colours).</p>
<h2>The final result</h2>
<p><img src="http://i.imgur.com/94ndnYw.png" alt="Browsing the web" />
<img src="http://i.imgur.com/PXQmHPn.png" alt="Screenfetch" />
It's worth mentioning that the screenfetch script took about 30 seconds to
display anything, and that elinks is completely unusable unless I let it take up
the whole screen. These screenshots are really just for show. I can't show you
what my screen looks like when I'm doing real work, because OpenBSD has no
facilities in place to take framebuffer shots (like fbgrab).</p>
Using GNU Stowurn:https-kaashif-co-uk:-2013-12-01-using-gnu-stow2013-12-01T00:00:00Z2013-12-01T00:00:00ZKaashif Hymabaccus
<p><code>stow</code> is a cool little Perl script which basically just creates and deletes
symlinks. That sounds pointless, but let me explain with an example. Let's say
you want to install a program using the usual <code>make install</code>, which probably
installs into /usr/local, which means it's separated from the rest of your
system, which is managed with a "real" package manager. Unless you're using a</p>
<!-- more -->
<p>system like BSD ports or Gentoo portage, you cannot have packages which are both
managed by the package manager <em>and</em> customised according to your needs. In
fact, if you need to do any in-depth modification of the source, I'd say you
have no other option than using <code>make</code>, meaning you have a whole mess of files
under /usr/local. This wouldn't be a problem with package managers, since this
"mess" of files can be removed by uninstalling the package, which usually has a
list of files it installed, ensuring all stray man pages and example configs are
purged from your filesystem. If you used <code>make install</code>, and there is no <code>make
deinstall</code> or similar, then you're out of luck. The only way to remove the
programs you just installed is to wade through the mess that is /usr/local and
remove the files manually. That's where <code>stow</code> comes in.</p>
<p>The idea of <code>stow</code> is that you compile your programs normally using <code>make</code>, but
install it into a different directory, usually something like
/usr/local/stow/$PKGNAME. This means there is an entire directory tree in that
directory containing all of the files that would have been installed by that
Makefile. You then enter the /usr/local/stow directory and use the command <code>stow
$PKGNAME</code>, which creates appropriate symlinks in the directory /usr/local to
make it seem as if you just installed the package with <code>make install</code> into
/usr/local. At first, this seems completely useless, but it means you have a
guaranteed way of deleting that package - tell <code>stow</code> to delete all the
symlinks. You do this by going into /usr/local/stow and typing <code>stow -D
$PKGNAME</code>, which removes all of the symlinks in /usr/local. Here is a series of
commands someone using <code>stow</code> might execute.</p>
<div class="highlight"><pre><span></span>$ sudo mkdir /usr/local/stow/my_program
$ <span class="nb">cd</span> /tmp/my_program
$ make install <span class="nv">DESTDIR</span><span class="o">=</span>/usr/local/stow/my_program
$ <span class="nb">cd</span> /usr/local/stow
$ stow my_program
</pre></div>
<p>This makes it very simple to upgrade packages or keep multiple concurrent
versions lying around.</p>
<h2>Using Stow for dotfile management</h2>
<p>Admittedly, <code>stow</code> is not really that useful on a bog standard GNU/Linux desktop
or laptop, since most packages would be compiled with the usual combinations of
compile-time features enabled and available in package repos. The behaviour of
<code>stow</code> can be exploited to provide some other uses, however, one of which
involves dotfiles.</p>
<p>Let's say a user named bob wants to check his dotfiles into his favourite
version control system and deploy them easily and quickly on other computers.
Right now he has a home directory which looks like this:</p>
<pre><code>bob
|-- .bash_profile
|-- .bashrc
|-- .config
|-- .mutt
| |-- config_1
| |-- config_2
| `-- config_3
|-- .vim
| |-- bundle
| `-- ftplugin
`-- .vimrc
</code></pre>
<p>Since <code>stow</code> works by recreating a tree of symlinks in the directory above where
it is invoked, bob can make a directory called "~/dotfiles" or similar, put all
of his dotfiles in there, and stow will create symlinks for him in the directory
above "~/dotfiles", his home. His ~/dotfiles should look something like this:</p>
<pre><code>bob
`-- dotfiles
|-- bash
| |-- .bash_profile
| `-- .bashrc
|-- config
| `-- .config
|-- mutt
| `-- .mutt
| |-- config_1
| |-- config_2
| `-- config_3
`-- vim
|-- .vim
| |-- bundle
| `-- ftplugin
`-- .vimrc
</code></pre>
<p>This means that he can deploy his dotfiles by going into the dotfiles directory
and executing <code>stow bash config mutt vim</code>, which creates symlinks in ~/,
duplicating the hierarchy he had in place before. Advantages include: being able
to selectively deploy configs only for programs you need on that PC, easy
integration with Git, Mercurial and the like as the dotfiles directory can be
checked into VCS. Stow has certainly avoided me a few headaches, I recommend it
for anyone who needs to use multiple machines.</p>
Emacsurn:https-kaashif-co-uk:-2013-11-30-emacs2013-11-30T00:00:00Z2013-11-30T00:00:00ZKaashif Hymabaccus
<p>For a few years now, I have been using Vim to edit config files,
program in C, Python, even Lisp (people apparently think that Vim
isn't the best for programming in Lisp). This isn't because I took a
side in the so-called "editor wars", it's just because it came
preinstalled on the first GNU/Linux system I used, Debian. Over time,</p>
<!-- more -->
<p>I abandoned IDEs and moved to a full Vim setup, which works well for
programming in C and Python, which is what I do most of the
time. I like to think my text-editing environment is actually quite
good, and that Vim's modal editing model has aided my programming. But
there was always the elephant in the room - Emacs. I had always heard
about it from the usual "Why I use Emacs" and "Vimscript
considered harmful" blog posts but I had never used it, mostly because
I was too entrenched in Vim's editing model. Then I heard about
evil-mode and viper-mode, the Emacs vi emulation modes, and I wondered
whether Emacs' sheer flexibility was a good reason to use it. It
seemed that this was the reason most people used it (not specifically
for vi emulation, but for the many, equally exciting modes), so maybe
I should try it out.</p>
<h2>Modal editing</h2>
<p>For the uninitiated, here is a quick rundown of Vim. There are 2 modes
you need to know about: Insert and Normal. Normal mode is the mode you
are supposed to spend most of your time in: you can execute motions,
editing commands, macros, searches, everything you need to edit text
apart from actually inserting (typing) any text. Insert mode is
entered by typing "i" and exited by typing ESC. Between these actions,
you can type text and it will be inserted into the file you're
editing. Most newbies attempt to spend most of their time in Insert
mode, but this eschews the real power of Vim, the easy composition of
motions and commands that allow efficient editing. For example, to
delete the next 10 words in Normal mode, you type "10dw". In Insert
mode, you must tediously hold down the arrow key and backspace. It
gets better, too, you can delete everything in quotes with "di'" for
"delete inside '" and so on.</p>
<h2>Emacs' model</h2>
<p>Emacs has a completely different model, when you type anything, it
appears in the buffer, just as if you were in Gedit, Kate, or any
other "normal" editor. Emacs is just as powerful as Vim, however, it
just relies on a system of key chords to execute commands and
motions. For example "C-k" deletes to the end of the line (C- means hold
down Control and press a key, M- means the same thing, but with Alt),
and "C-s /" moves to the next "/". Some might say that these key
chords have a habit of twisting your hands into unnatural positions,
but the advantage of having a built-in Lisp interpreter is that you
can script anything you want, however complex, resulting in the
included viper-mode and easily installable evil-mode, which "fix"
Emacs for Vim users.</p>
<h2>My setup</h2>
<p>I haven't opted to go for any vi emulation mode in my use of Emacs,
I'm using Emacs as Stallman intended. Surprisingly, my hands don't
hurt as much as I thought they would, and I certainly don't have RSI
or any wrist injuries. The first thing I noticed when I opened Emacs
was...well, just look at this:</p>
<p><img src="/static/emacs.png" alt="Emacs defaults" /></p>
<p>It's not very pretty, not in the slightest. The first order of
business was to change the colour theme. It was not immediately
apparent how to do this using the ".emacs" init file, so I resorted to
using the GUI, which was quite a bit easier. While there, I also
changed my font and hid some of the GUI elements, so it almost looked
like a terminal, but with more colours and variable font
sizes. Surprisingly, saving the settings did not save them in some
unreadable binary format or weird markup language, it just appended
some elisp (Emacs Lisp) to my ".emacs" file, which was nice. Here is
the result of that:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="nv">custom-set-variables</span>
<span class="o">'</span><span class="p">(</span><span class="nv">blink-cursor-mode</span> <span class="no">nil</span><span class="p">)</span>
<span class="o">'</span><span class="p">(</span><span class="nv">custom-enabled-themes</span> <span class="p">(</span><span class="k">quote</span> <span class="p">(</span><span class="nv">wombat</span><span class="p">)))</span>
<span class="o">'</span><span class="p">(</span><span class="nv">inhibit-startup-screen</span> <span class="no">t</span><span class="p">)</span>
<span class="o">'</span><span class="p">(</span><span class="nv">tool-bar-mode</span> <span class="no">nil</span><span class="p">)</span>
<span class="o">'</span><span class="p">(</span><span class="nv">tooltip-mode</span> <span class="no">nil</span><span class="p">))</span>
<span class="p">(</span><span class="nv">custom-set-faces</span>
<span class="o">'</span><span class="p">(</span><span class="nv">default</span> <span class="p">((</span><span class="no">t</span> <span class="p">(</span><span class="ss">:family</span> <span class="s">"Terminus"</span> <span class="ss">:foundry</span> <span class="s">"xos4"</span> <span class="ss">:slant</span> <span class="nv">normal</span> <span class="ss">:weight</span> <span class="nv">normal</span> <span class="ss">:height</span> <span class="mi">90</span> <span class="ss">:width</span> <span class="nv">normal</span><span class="p">)))))</span>
</pre></div>
<p>Now I had to get line numbers, which was as simple as appending
<code>(global-linum-mode 1)</code> to my ".emacs". To get a space between the
numbers and the buffer text (which isn't the default, oddly), I had to
append <code>(setq linum-format "%d ")</code>. At this point, Emacs was starting
to look good. Its real test would be the ease of use of its package
system. At the end of all of this customisation, my Emacs looks like
this:</p>
<p><img src="/static/emacs-new.png" alt="New Emacs" /></p>
<h2>package.el</h2>
<p>Emacs comes with a package manager called, imaginatively,
<code>package.el</code>. It doesn't come with a very expansive list of repos, so
I added a few, using the example on the EmacsWiki to guide me. Here is
what I had to add:</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="k">setq</span> <span class="nv">package-archives</span> <span class="o">'</span><span class="p">((</span><span class="s">"gnu"</span> <span class="o">.</span> <span class="s">"http://elpa.gnu.org/packages/"</span><span class="p">)</span>
<span class="p">(</span><span class="s">"marmalade"</span> <span class="o">.</span> <span class="s">"http://marmalade-repo.org/packages/"</span><span class="p">)</span>
<span class="p">(</span><span class="s">"melpa"</span> <span class="o">.</span> <span class="s">"http://melpa.milkbox.net/packages/"</span><span class="p">)))</span>
</pre></div>
<p>Now I had 3 repos to install my favourite packages from. I needed
something to auto pair parenthesis and the like, so I installed
autopair, which was as simple as "M-x package-install autopair RET".
You can also browse a long list of all packages using "M-x
list-packages", and search it with the usual C-s and C-r. This is
better, in some respects, than my method of managing Vim
bundles. Vundle requires you to edit ".vimrc" and add something to a
list of bundles, then run ":BundleInstall" in Vim. This is a longer
process than Emacs' centralised package repo system, which is more
convenient. On the other hand, Vundle lets you install bundles from
any Git repo, meaning it's a lot easier to install random bundles you
find on the internet. I haven't noticed anything regarding
plugin/bundle/package quality, all of the extensions/add-ons/scripts I
use are quite good, probably because they're all free software and
anyone can contribute. That's one thing all people on all sides of the
Editor War can agree on.</p>
<h2>Conclusion</h2>
<p>I can't really say anything about Emacs until I use it more, but I do
like the windows more than Vim's odd buffer system. That's all for
now. Maybe I'll have more of an opinion when I get to writing large
projects.</p>
Introduction to Curn:https-kaashif-co-uk:-2013-11-27-introduction-to-c2013-11-27T00:00:00Z2013-11-27T00:00:00ZKaashif Hymabaccus
<p>This tutorial is designed for those who have programmed before, perhaps in a
higher level language like Python or Ruby. It's not too hard to understand for
those who are completely inexperienced, but some knowledge of functions, data
structures and pointers might help. Most of the low-level stuff will be new to
high level programmers, however.</p>
<!-- more -->
<h2>Setting up the compiler</h2>
<p>On GNU/Linux, most BSDs and quite a few other operating systems, <code>gcc</code>, the C
compiler of the GNU Compiler Collection, is included. If not, it'll be
accessible through your package manager. For example, on FreeBSD, where <code>clang</code>
is the default, you'd install <code>gcc</code> with a simple <code>pkg install gcc</code>. OpenBSD
comes with <code>gcc</code> by default, and so do the vast majority of GNU/Linux distros.
If not, you know what to do. The process of actually compiling source will be
elaborated on later. For now, here is a command you can use to compile programs:</p>
<div class="highlight"><pre><span></span>$ gcc -o hello hello.c
</pre></div>
<p>This compiles the source code in the text file <code>hello.c</code> and compiles it into
the executable <code>hello</code>. The use of libraries will be covered later, or maybe not
at all. Look it up on StackOverflow or something.</p>
<h2>Hello, World!</h2>
<p>The standard "Hello, world!" program is the de facto standard for introducing a
programming language. Here is a basic implementation:</p>
<div class="highlight"><pre><span></span><span class="c1">// This is a comment</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="w"> </span><span class="p">()</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"Hello, world!"</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>The first line includes the standard header containing IO functions. In C, when
you include a header, the C preprocessor just inserts the text contained in that
header wherever you place the <code>#include</code>. There really isn't anything more to it
than that, no complex module importing hoops to jump through, like in Python or
Java.</p>
<p>The main function is self-explanatory, but requires a little exposition. Main
functions in C always return an int. This is the return code of the program,
which is used in scripts and in the shell to determine whether the program
succeeded. Generally, zero means a success while anything else means a failure.
You could also use some macros (basically constants) defined in stdlib.h to
denote failure and success exit codes. If we did that, we could rewrite the
source as follows:</p>
<div class="highlight"><pre><span></span><span class="cm">/* You can also do comments like this */</span><span class="w"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdlib.h></span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="w"> </span><span class="p">()</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"Hello, world!"</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">EXIT_SUCCESS</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>The advantages of using <code>EXIT_SUCCESS</code> and <code>EXIT_FAILURE</code> generally include
things like portability across platforms where 1 could mean failure on one,
while -1 could mean the same thing on another. These constants are still ints,
so you don't have to change anything else to get the main function working. I'd
advise you use these constants, since you'll have included stdlib.h most of the
time anyway. I'll use them for the rest of this tutorial.</p>
<p>The <code>printf</code> function isn't very complicated. Its function is obvious, but it
does have a few quirks you might need to know about.</p>
<h2>Printing</h2>
<p>Let's say you had an int and you wanted to print it out, along with some words.
In Python this would be simple:</p>
<div class="highlight"><pre><span></span><span class="n">print</span><span class="p">(</span><span class="s">"The value of my integer is %d"</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="n">my_int</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>That syntax actually comes from C, <code>printf</code> has a very similar syntax:</p>
<div class="highlight"><pre><span></span><span class="n">printf</span><span class="p">(</span><span class="s">"The value of my integer is %d"</span><span class="p">,</span><span class="w"> </span><span class="n">my_int</span><span class="p">);</span><span class="w"></span>
</pre></div>
<p>The format string and its arguments are all arguments to <code>printf</code>, and you can
have as many of them as you like. For example:</p>
<div class="highlight"><pre><span></span><span class="n">printf</span><span class="p">(</span><span class="s">"My car cost %d pounds and weighs %f kg"</span><span class="p">,</span><span class="w"> </span><span class="n">my_int</span><span class="p">,</span><span class="w"> </span><span class="n">my_float</span><span class="p">);</span><span class="w"></span>
</pre></div>
<p>Simple so far, right? You may already be familiar with the idea of output
streams if you use the command line. Everything <code>printf</code> prints, by default,
goes to standard output or <code>stdout</code>. This is the output that is piped to other
programs and written to text files when you redirect the output using ">" or
"|". Sometimes you want to output debug or error information which is supposed
to only be read by a human. For this you use stderr, or the standard error
output stream. Since <code>printf</code> only prints to <code>stdout</code>, we must use another
function, <code>fprintf</code>. It is used as follows:</p>
<div class="highlight"><pre><span></span><span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span><span class="w"> </span><span class="s">"This is an error"</span><span class="p">);</span><span class="w"></span>
</pre></div>
<p>That is essentially all the printing you'll ever need to know.</p>
<h2>Functions</h2>
<p>C is a statically, strongly typed language. There is no type coercion or
automatic choosing of types for you. This keeps programs simple and easy to
understand, and helps avoid undefined behaviour. This is relevant because
function definitions start with a return type and a list of parameters, all
specifying their type. If we look at the following function, we can see what I
mean.</p>
<div class="highlight"><pre><span></span><span class="kt">void</span><span class="w"> </span><span class="nf">say_hello</span><span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="w"> </span><span class="n">name</span><span class="p">)</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="n">printf</span><span class="p">(</span><span class="s">"Hello, %s!"</span><span class="p">,</span><span class="w"> </span><span class="n">name</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>You should know what <code>void</code> means. If you tried to pass a float or a struct of
some kind to this function, it would not compile. I assume returning other types
doesn't require any explanation or patronising "exercises".</p>
<h2>Arrays, Strings and Pointers</h2>
<p>You might have noticed that the syntax for passing a string to a function is a
bit odd. Well, that's because all strings in C are really just arrays of
characters, represented by the <code>char</code> data type. This will become clear in the
following code snippet:</p>
<div class="highlight"><pre><span></span><span class="kt">char</span><span class="w"> </span><span class="n">hello</span><span class="p">[</span><span class="mi">10</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"hello"</span><span class="p">;</span><span class="w"></span>
</pre></div>
<p>That compiles and behaves as expected. You can pass the variable <code>hello</code> to any
function that expects a C-style string and it will work. But isn't hello an
array, not a <code>char*</code>, whatever that means? Well, you might think so in a
language without manual memory management, but in C, arrays are merely pointers
to the first element in an array.</p>
<p>The thing which confuses most new C programmers is the concept of pointers. It
all becomes vey simple if you just ditch the analogies and realise that a
pointer is a variable which stores the address of the data being pointed to. So
the pointer <em>points</em> to the data in RAM, but is not actually the data itself,
merely an address. If you know that, the act of "dereferencing" a pointer is
also very easy, it's just accessing the data being pointed to. Here is some code
to explain:</p>
<div class="highlight"><pre><span></span><span class="kt">int</span><span class="w"> </span><span class="n">my_int</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">4</span><span class="p">;</span><span class="w"></span>
<span class="c1">// Creates an integer with the value of 4</span>
<span class="kt">int</span><span class="o">*</span><span class="w"> </span><span class="n">to_int</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&</span><span class="n">my_int</span><span class="p">;</span><span class="w"> </span>
<span class="c1">// Creates a pointer to an integer, which is then set to the </span>
<span class="c1">// address of the integer</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"%d"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="n">to_int</span><span class="p">);</span><span class="w"></span>
<span class="c1">// Dereferences the pointer and prints the value</span>
</pre></div>
<p>There are some new operators in there: the <code>&</code> operator, which takes a variable
of any type and returns the memory address at which the variable is stored. The
<code>*</code> operator takes a pointer and returns the data stored there. It essentially
reverses the <code>&</code> operation.</p>
<h2>Memory allocation</h2>
<p>We have established that pointers hold the address of a block of memory, which
usually has some data in it. Letting the compiler allocate the appropriate
amount of memory is fine in some cases, but what if we need to allocate an
amount of memory which we only know at runtime. An obvious use case is when
copying large files - we cannot simply allocate 4 GB at compile time and hope
the file fits, we must get the file size from the filesystem and allocate that
much memory. This is done using dynamic memory management.</p>
<p>The two most important functions for this are <code>malloc</code> and <code>free</code>. <code>malloc</code>
takes an integer denoting the number of bytes to allocate, and returns a pointer
to the allocated block of memory. There is a problem here - you do not always
know the number of bytes taken up by a data type, it could vary from platform
to platform. You can find out using the <code>sizeof</code> function, which takes a data
type and returns an int telling you how many bytes that data type takes up per
instance. Let's say you wanted to allocate memory for an array of 24 ints. You'd
combine <code>sizeof</code> and <code>malloc</code> to produce the following:</p>
<div class="highlight"><pre><span></span><span class="kt">int</span><span class="o">*</span><span class="w"> </span><span class="n">my_array</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">malloc</span><span class="p">(</span><span class="mi">24</span><span class="o">*</span><span class="k">sizeof</span><span class="p">(</span><span class="kt">int</span><span class="p">));</span><span class="w"></span>
</pre></div>
<p>And now the pointer <code>my_array</code> points to an allocated block of memory which can
fit 24 ints. But how do you access the ints? Well, the variable's name should
give you a clue - it's just an array. You can use this like an array, because it
<em>is</em> an array. This is all there is to arrays in C, pointers to memory.</p>
<p>After you're done with the array of ints, you may not want the memory to stay
allocated. If you allocate memory throughout your program but never deallocate
it, this causes the memory usage of your program to grow and is known as a
memory leak. These can be avoided by deallocating (or freeing) memory using the
<code>free</code> function, which takes a pointer and frees the memory it points to.</p>
<div class="highlight"><pre><span></span><span class="n">free</span><span class="p">(</span><span class="n">my_array</span><span class="p">);</span><span class="w"></span>
</pre></div>
<p>In your programs, every call to <code>malloc</code> should be accompanied by a call to
<code>free</code> at some point, to eliminate memory leaks. This is very important when
you're dealing with large files or databases, where leaking memory could cause
gigabytes of damage, causing the OS to swap pages to disk and slow down or even
crash.</p>
Email is the Futureurn:https-kaashif-co-uk:-2013-11-12-email-is-the-future2013-11-12T00:00:00Z2013-11-12T00:00:00ZKaashif Hymabaccus
<p>Often, people look at me oddly when I suggest that they email me something. "Why
can't I just send it to you on Facebook or Skype?" they say. Well, it doesn't
have to be those two media of communication, but it's usually something like
that. When I say often, I also mean that this has only happened on two
occasions, so bear that in mind as I make things up about the types of people</p>
<!-- more -->
<p>who ask me this.</p>
<p>I'd like to think that the people who advocate use of Facebook or Skype are the
sorts of people who move from social media site to site, forgetting about their
old posts from years ago on Livejournal or MySpace, saying they don't matter.
Maybe the content of the messages didn't matter, but if they did, wouldn't you
want them preserved somewhere permanent, somewhere you could always access them,
regardless of how successful the site is this quarter and whether it has to shut
down? That's not the best argument for using email, since it means you'd have to
save and back up your emails yourself - this is all irrelevant, since my point
is not that you should use email because it lets you archive all of your creepy
private messages. No, my point relates to the control the social media websites
have over you.</p>
<h2>Isn't this just another rant about free software?</h2>
<p>No, not exactly. You could have a proprietary email client and it would still be
better than using a proprietary social media SaaSS (Service as a Software
Substitute). Those social media websites, while they may restrict your freedom
for other reasons, primarily rob you of control over your computing. When you
upload a picture to Facebook, then edit it, should you not have done that using
a program on your own PC, instead of a less functional, half-baked "web app"
version of the program? You could argue that it's more convenient, but this is
only true because you are already entrenched in Facebook's lock-in scheme. If
you did all of your editing locally, you would not be able to make the same
complaint.</p>
<p>Doing your editing (whether it be of text, images, or music) on your own PC,
using a program you have <em>some</em> control of, offers you far more choice and
control over your experience than using a webpage to do it. If said website
(this does not only apply to Facebook) changes their UI, you are powerless to
protest, let alone change it back. There are two ways to avoid this situation:
only use websites licensed under free copyleft licenses (such as the Affero GPL)
or do eveything locally, where you have the most control over what software is
used to do the editing. In an ideal world, you'd use a free software program to
edit your content, removing the issue of someone having control over you
entirely.</p>
<p>After you edit your content, uploading it to a SaaSS social network would be
nullifying any advantages to gained by editing locally. Ideally, you'd want to
do as much as possible locally - editing content, addressing it, but not sending
it, as doing that locally is impossible, and the <em>only</em> thing which should be
done over a network. A protocol exists to send messages over the internet, it's
called SMTP.</p>
<h2>Email is limited and inconvenient</h2>
<p>I'd argue that logging into a website or using an app which I cannot examine or
audit is a far greater inconvenience than being "forced" to resort to mailing
lists and email clients. Of course, the majority of people do not think this, so
I'll have to convince them with arguments which appeal to laziness or
convenience. As Larry Wall said, "The three principal virtues of a programmer
are Laziness, Impatience, and Hubris". Hubris doesn't really come into this
issue, but the other two definitely do.</p>
<p>I'll use an example to illustrate. What if you want to put all messages starting
with the word "Important", containing the word "deadline" and with a date in the
format "YYYY-MM-DD". Maybe that's a bit specific, but I'm sure that filtering
for something that specific is impossible without access to scripting tools. Lo
and behold, such tools exist on your machine, but not on the SaaSS website!
Using utilities like procmail, or even just a Perl script, you could easily walk
a Maildir and put all messages matching the above conditions into a separate
folder. Usually, it will just be things like <code>moveto("programming") if
($subject~=/^GitHub/)</code>, but it is important to have tools which work in all
cases, not just the cases the programmers at Proprietary inc. see fit to program
in. Any arguments relating to the odd SaaSS tool which provides greater
functionality than a local program are flawed because of the inherent
limitations and danger of putting a group in charge of communication for another
large group.</p>
<h2>My social network is free, is there still a problem?</h2>
<p>People in charge of projects like Diaspora and GNU Social might tell you there
isn't, but I think there is. Who actually uses Diaspora? Who has even <em>heard</em> of
GNU Social? Maybe I should kickstart them and get all my friends to use them,
but what's the point when I have another method of communication which is almost
as ubiquitous as internet access - email. Everyone has an email address and
everyone will for the forseeable future. The first email standard was published
in 1973 and is still followed (with some revisions and additions) to this day.
That's <em>forty years</em> of use, something Facebook, Google+ and Twitter can never
hope to achieve.</p>
<p>In 2005, it may have been tempting to say that MySpace would still be in use in
10 years. Right now, that might be said about Twitter or LinkedIn. No-one would
ever think of saying the same things if asked whether they would be used in
2053, 40 years into the future. Not only would people say this about email, it
has already happened. A standards compliant email client from 2013 would not
fail to read an email from 1973, or send an email using SMTP (standardised in
1982). Email was the past, it is the present, and it will be the future. What
more could you ask for in a communication protocol?</p>
How to Wipe a Diskurn:https-kaashif-co-uk:-2013-11-06-how-to-wipe-a-disk2013-11-06T00:00:00Z2013-11-06T00:00:00ZKaashif Hymabaccus
<p>This article is not only about disk wiping, it will hopefully teach
you something about using some GNU command line tools . This tutorial
was written on my ThinkPad, which runs Debian, so the output should be
pretty similar to what you'd get on Ubuntu, Mint or any other Debian-
or Ubuntu-based systems. Basically, if you're using something</p>
<!-- more -->
<p>non-standard, you already know that you are, since those types of
operating systems are few and far between.</p>
<h2>Getting into the right environment</h2>
<p>Since you can't reliably wipe a disk with the OS which is on the disk you'll be
wiping, you need some way to run an OS from something other than the hard drive.
Enter live CDs, DVDs, and USB drives. You can download a disk image and write it
to a drive any way you want to, it doesn't matter, as long as the disk boots
into a GNU userland of some sort. I recommend Debian, you can get a selection of
disk images <a href="http://www.debian.org/distrib/">here</a>. Burn a disc, write it to USB
or whatever. After you do that, insert it into your PC and reboot. Make sure
the removable media of your choice is higher in the boot order of the BIOS than
the hard drive, or this won't work. When you reboot, you will hopefully be
confronted with a bootloader menu with several options. Pick the one which
sounds most like "Try before you install" or "Live DVD", and you will be put
into a GNU/Linux environment, hopefully with some sort of shell prompt. You are
now ready to execute some commands!</p>
<h2>Which drive are we wiping?</h2>
<p>The first thing you need to do is find out which disk you need to wipe. Your USB
drive may be one of the drives detected by the OS, so it's important you wipe
the right thing. Even if you only have one drive, it's best to check which drive
you're wiping just to make sure. The command to list block devices is <code>lsblk</code>.
It is used as follows:</p>
<div class="highlight"><pre><span></span>$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda <span class="m">8</span>:00 <span class="m">232</span>.9G <span class="m">0</span> disk
<span class="sb">`</span>-sda1 <span class="m">8</span>:10 <span class="m">232</span>.9G <span class="m">0</span> part
sdb <span class="m">8</span>:16 <span class="m">1</span> <span class="m">14</span>.4G <span class="m">0</span> disk
<span class="sb">`</span>-sdb1 <span class="m">8</span>:17 <span class="m">1</span> 739M <span class="m">0</span> part /
</pre></div>
<p>Here we see two disks: sda and sdb. With the Linux kernel, drives are given
names based on what type of drive they are and the order in which they were
detected. In this case, the 250 GB hard drive is detected first, so it ends in
"a". All disks are given the "sd" prefix, unless they're really old IDE hard
drives. The USB drive was detected 2nd, so its name is "sdb". So we have
established that we want to wipe the drive "sda". Note that we are targeting the
drive itself, not any of its partitions ("sdaX").</p>
<p>But how do we get to that drive? This is an question which really exposes the
convenient and useful nature of the Unix philosophy. Specifically, the part
which says all programs should aim to represent as much as possible using files.
With this in mind, it's logical to say that the drive "sda" is represented
somewhere on the filesystem. It happens to be in the directory "/dev", with all
the other device files. Essentially, we will have to wipe the <em>file</em> "/dev/sda",
which is, for all intents and purposes, the disk.</p>
<h2>Wiping the disk</h2>
<p>To wipe a disk, we have to first consider what we actually want to do to the
disk. Of course, we want to erase the data on that disk permanently. Since
erasing it by replacing the partitions doesn't actually erase the data (it is
still there, it just cannot be read without restoring the partition table), we
have to actually overwrite all of the data with some other data. We could write
lots of random data, but remember that generating random noise takes time and
CPU power. Instead, let's just overwrite it all with zeroes, which is very
simple. Linux users do things like this so often that the devs saw fit to add a
virtual device consisting entirely of zeroes, at "/dev/zero". You can think of
this as a disk of infinite size consisting entirely of zeroes. Our aim has
changed from the initial "I want to wipe a hard drive" to "I want to overwrite
/dev/sda with data from /dev/zero".</p>
<p>There is a command to copy and convert data, and that program is <code>dd</code>. We'd use
it to copy data from /dev/zero to /dev/sda as follows. Be careful, because the
following command will irreversibly overwrite all of data on your primary hard
drive. Make sure you have the right drive and that you really want to do this.</p>
<div class="highlight"><pre><span></span>$ sudo dd <span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>/dev/sda
</pre></div>
<p>You won't see any output unless there's an error, so don't worry about dd's
complete silence. A lack of errors means it's working! Now, you should switch to
a different terminal, and let dd run in its own. You can do this by simply
opening another terminal window, if you're using a GUI, or press CTRL-ALT-F2, to
switch to the 2nd virtual terminal, if you're in text-only mode.</p>
<h2>What to do while you wait</h2>
<p>Now is a good a time as any to tell you about several useful features of Unix
systems which let you find and learn about commands without resorting to the
internet (they'd probably point you to this anyway). I am talking about the Unix
manual pages. They are accessed through the <code>man</code> command, followed by the name
of the program you want to learn more about. For example:</p>
<pre><code>$ man man
MAN(1) Manual pager utils MAN(1)
NAME
man - an interface to the on-line reference manuals
</code></pre>
<p>And a lot more information. You can exit the manual pager by pressing "q". This
is not all you can do while you wait, if you read the man page for "dd", you
might find a way to make it print how much it has copied. The man page is
intended to be a reference for experienced users, so don't worry if you don't
understand it.</p>
<p>On any OS, processes are not only known by the human-readable names of the
programs, they are also known by Process IDs, or PIDs. When a program is run, it
is assigned a PID. This means that the higher the PID, the later in the boot
process or interactive session it was run. After a program terminates, its PID
is recycled and given to the next program to be spawned, or just left unused. To
find out the PID of the dd process you ran earlier, we can use the program
<code>pgrep</code> which takes a program name, and outputs all of the numerical PIDs
associated with programs with that name.</p>
<div class="highlight"><pre><span></span>$ pgrep dd
</pre></div>
<p>It's safe to assume that the most recently started instance of dd is the one we
just started - the one with the highest PID. Now that we know its PID, we can
start sending it signals. The program to send a signal to a running program is
<code>kill</code>, which is a bit of a misnomer, because not all of the signals it's
capable of sending actually kill the process. In this next command, substitute
"$pid" with the PID of your dd process.</p>
<div class="highlight"><pre><span></span>$ sudo <span class="nb">kill</span> -USR1 <span class="nv">$pid</span>
</pre></div>
<p>This command won't output anything in the terminal you run it in. Instead, it
sends a signal to dd to make it print out how much it has copied. This
information would be in the terminal that dd was run from. If you're using a GUI
terminal, switch to the window dd was run from. If you're in text-only mode,
switch back to the 1st virtual terminal by pressing CTRL-ALT-F1.</p>
<h2>Closing remarks</h2>
<p>I hope you've learnt about more than just how to wipe a disk, although that is a
useful skill, too. If you want to learn about Unix using the built-in system
tools, you always have your trusty man pages, but also another new tool:
<code>apropos</code>. It takes a list of keywords, and parses the man pages, searching for
commands which match your description of what you want to do. For example:</p>
<pre><code>$ apropos extract archive
unrar (1) - extract files from rar archives
unar (1) - extract archive file content
</code></pre>
<p>It generally tends to output many results, not all of which are commands.
Remember "man man", the manual page for man? It had a list of section numbers
and what they mean. We can see these section numbers in the search results for
apropos, in the brackets just after the program name. To save you some effort...</p>
<pre><code>$ man man
1 Executable programs or shell commands
2 System calls (functions provided by the kernel)
3 Library calls (functions within program libraries)
4 Special files (usually found in /dev)
5 File formats and conventions eg /etc/passwd
6 Games
7 Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7)
8 System administration commands (usually only for root)
</code></pre>
<p>So out of all the results in that list, only the ones in sections 1 and 8 are
usable as programs from the shell prompt. Using these tools, your knowledge of
Unix can grow without resorting to searching the internet for hacks other people
have made. Even after you know the ins and outs of a program, the man pages are
still useful for when you can't remember the order the arguments go in, or the
switch to make a program change behaviour, or things along those lines.</p>
Why I Use FreeBSDurn:https-kaashif-co-uk:-2013-10-28-why-i-use-freebsd2013-10-28T00:00:00Z2013-10-28T00:00:00ZKaashif Hymabaccus
<h2>Installing packages from source</h2>
<p>Recently, I had SSHed into one of Debian Stable virtual machines I was using as
a file server. The main services I was running were FTP, an HTTP server with a
directory listing and a Samba server, with shares set up for a few users on my</p>
<!-- more -->
<p>LAN. All in all, it wasn't anything very complex, it was extremely
run-of-the-mill, and could have almost been installed by <code>tasksel file-server</code>
or some similar list of packages almost certainly installed on thousands of
servers.</p>
<p>A problem arose with the server after I realised I needed to enable several
options in ProFTPd, the FTP server I was using. There isn't a way to do this
easily on Debian, as far as I'm aware. I know about the <code>apt-get source</code>
command, but, as far as I have read, that command can only build packages with
the options that the person who made the source package specified. Essentially,
it's a system focused not on customisation of packages, but on staying on the
bleeding edge of updates for that package. I don't really see any point in this,
so <code>apt-get source</code> seems useless for my purposes. It would be useful in a
context similar to that of <code>sbopkg</code>, the SlackBuilds package manager, which is
useful because it provides access to a wider variety of packages than official
mirrors. Debian has all of the packages one could desire prepackaged and
mirrored, so the one possible use of <code>apt-get source</code> is null.</p>
<p>At this point, the only option available to me was to download the source from
proftpd.org and use the Makefiles provided with the source to compile and
install ProFTPd. There are several problems with this approach:</p>
<ul>
<li><p><code>dpkg</code>, the Debian package manager, is never involved, meaning dependencies
and upgrades cannot be handled automatically - it is up to <em>me</em> to fetch the
tarball and recompile it.</p></li>
<li><p>Having a multitude of extracted tarballs, CVS repositories and Git
repositories lying around on my system causes clutter and confusion. One might
say that naming and organising my directories should be my responsibility. The
fact of the matter is that it should not be handled manually - that is a
time-wasting and error-prone approach.</p></li>
<li><p>This solution is not scalable. With dozens of packages across many servers
needing to be recompiled whenever there is an important security fix, I could
find myself without any time to devote to real, important server maintenance
tasks. Eventually, manual compilation of packages would become impossible.</p></li>
</ul>
<p>This is a stark contrast to FreeBSD's solution to the problem of package
customisation. On FreeBSD (and any other *BSD), the ports collection provides a
convenient way to compile your own packages. Quite simply, you search for your
packages using "whereis" or "make search", navigate to its location under
/usr/ports, then <code>make</code> it. That is obviously not the only way to make use of
ports - that method suffers from essentially all of the same problems as before!</p>
<p>If we look back to the original example, that of ProFTPd, I can demonstrate the
power of ports. Using <code>sysutils/portmaster</code>, a handy script for building, then
installing packages from source, I can install ProFTPd with my desired
configuration:</p>
<div class="highlight"><pre><span></span>$ sudo portmaster ftp/proftpd
<span class="o">===</span>>>> Port directory: /usr/ports/ftp/proftpd
<span class="o">===</span>>>> Gathering distinfo list <span class="k">for</span> installed <span class="nv">ports</span>
<span class="o">===</span>>>> Launching <span class="s1">'make checksum'</span> <span class="k">for</span> ftp/proftpd <span class="k">in</span> background
<Most of the output removed <span class="k">for</span> brevity>
<span class="o">===</span>>>> The following actions were performed:
Installation of ftp/proftpd <span class="o">(</span>proftpd-1.3.4d<span class="o">)</span>
</pre></div>
<p>Basically, I specified which port to install, which caused portmaster to begin
the process of downloading the source, checking its integrity and most
importantly, launching <code>make config</code>, which opens a menu with the compile
options for ProFTPd. After I select the ones I want, I can then let portmaster
compile the port into a package, which is installed using the FreeBSD package
manager. From this point, I can either update it from a mirror of binary
packages (this would be faster, but the package would not have my compile-time
options enabled) or upgrade it using portmaster. The process of upgrading
hundreds of packages from source is a single command: <code>portmaster -a</code>. The
convenience and power of FreeBSD ports is only one of the reasons I use it and
the other *BSDs on my servers.</p>
<h2>Jails</h2>
<p>On Debian and indeed all other GNU/Linux operating systems, you have a few
choices when it comes to virtualisation. You can go with KVM, Xen or VirtualBox,
all of which are the standard, heavyweight, fully virtualised solutions.
Alternatively, if you think the immense overhead that comes with fully
virtualising a server is too much, you can use Linux containers. There are
problems with using LXC, however, problems which are extremely important for security
conscious sysadmins.</p>
<p>If a malicious hacker gains access to your LXC and manages to use a privilege
elevation exploit to gain root on that box, it is not only the virtual box which
is compromised - root within an LXC translates to root outside the container.
This means that LXCs are essentially useless if you are virtualising for the
sake of security. Similarly, shutting down an LXC using the <code>shutdown</code> binary
results in the host machine shutting down. While this is less serious from a
security perspective than the gaining root (since you need to be root to
shut down in the first place), it can be irritating.</p>
<p>Jails on FreeBSD face none of these issues. FreeBSD jails have existed since
FreeBSD 4.0, which was released 13 years ago. A massive amount of development
has been targeted at adapting the FreeBSD kernel and userspace to playing nicely
within a jail and when hosting jails; this effort has resulted in an excellent
virtualisation solution. Much like a traditional virtual machine, a jail has its
own kernel, its own procfs, its own devices and everything you would find in a
real FreeBSD system. Using <code>make buildworld && make installworld</code> into an
appropriate jail directory is all that is needed to create a jail. After this is
done and the networking is set up, the environment viewed from within a jail is
indistinguishable from a real FreeBSD system.</p>
What is LaTeX?urn:https-kaashif-co-uk:-2013-09-26-what-is-latex2013-09-26T00:00:00Z2013-09-26T00:00:00ZKaashif Hymabaccus
<p>You probably have not heard of LaTeX before now. If you have, then it is likely
that you have no idea what LaTeX is, save for a vague feeling that it relates to
documents in some way. By the end of this short post, you will not only know
what LaTeX is, but be able to understand why people use it, and what its
advantages are. While you probably won't switch to LaTeX immediately, you might</p>
<!-- more -->
<p>be able to see yourself doing it if you ever have to write a non-trivial
document. Non-trivial means, in this context, something longer than a few dozen
pages, or something requiring a considerable amount of formatting which may take
a while in your WYSIWYG editor of choice.</p>
<h2>What is LaTeX?</h2>
<p>LaTeX is a document preparation system, in the words of latex-project.org. That
seems a bit vague, so I'll say that it's the name of a system involving two
things: the writing of source "code" files in the LaTeX "language" and the
compilation of that source into a final, prepared document. So you'd have your
text file (referred to as LaTeX source) which may look something like this:</p>
<div class="highlight"><pre><span></span><span class="k">\documentclass</span><span class="nb">{</span>article<span class="nb">}</span>
<span class="k">\title</span><span class="nb">{</span>My Article<span class="nb">}</span>
<span class="k">\author</span><span class="nb">{</span>Me<span class="nb">}</span>
<span class="k">\date</span><span class="nb">{</span><span class="k">\today</span><span class="nb">}</span>
<span class="k">\begin</span><span class="nb">{</span>document<span class="nb">}</span>
<span class="k">\maketitle</span>
This is my document.
<span class="k">\end</span><span class="nb">{</span>document<span class="nb">}</span>
</pre></div>
<p>Then you'd run a LaTeX compilation command, like <code>pdflatex myarticle</code>, resulting
in something like this:</p>
<p><img src="http://kaashif.co.uk/static/pdflatex1.png" alt="The result" /></p>
<h2>Why use LaTeX?</h2>
<p>It appears very pointless and silly to install a suite of document preparation
programs and learn how to use a very large set of typesetting commands just to
be able to write documents. Despite what you may think, it is not. LaTeX confers
several advantages to the document writer.</p>
<p>The process of writing a document then compiling it divorces formatting and
content. This means that, if you just want to write a 10,000 word essay, you
will only have to type the 10,000 words into your favourite text editor and you
won't have to worry about faffing around with fonts, margins and whatnot. You
might argue that the defaults in a WYSIWYG office program (such as LibreOffice
Writer) are good enough that you won't have to faff around anyway. You'd be
wrong. When you are writing a title, for example, you must first center the
text, make it bold or underlined, maybe increase the font size. This is all
pointless and gets in the way of creating the content, which is what actually
matters. With LaTeX, you write all of the content <em>then</em> worry about formatting.
Even better, you could leave that to someone else.</p>
<p>The clear divide between source files and final documents allows you to write
one source file and compile it into any of a variety of formats. This is not
possible with formats such as Microsoft's OOXML. It is impossible to
100% accurately convert documents to and from this format. That is a
rather extreme example, but it is clear that the plain text standards
(ASCII, Unicode) are completely open and that thousands of plain text
editors exist. This allows you to write LaTeX source in whichever editor
you want, and to compile it into whichever format you require, whether
it be PDF, HTML, PostScript, or any other format.</p>
<p>The vast number of packages available for LaTeX online is staggering.
Visiting the LaTeX website tells you that packages for everything from
mathematical formula rendering to bibliography generation exist and are
available freely. In LibreOffice Writer, for example, to create a
section, you'd have to create a human readable line of text with some
sort of formatting to distinguish it from the surrounding text. In
LaTeX, you'd use a section command, like so:</p>
<div class="highlight"><pre><span></span><span class="k">\section</span><span class="nb">{</span>Why I love LaTeX<span class="nb">}</span>
Reasons.
<span class="k">\section</span><span class="nb">{</span>Why I hate clouds<span class="nb">}</span>
More reasons.
</pre></div>
<p>These commands are machine-readable, thus readable to the LaTeX compiler you are
using. This means it is possible for a list of sections, subsections and
subsubsections to be generated - this uses a simple <code>\tableofcontents</code> command
from one of LaTeX's packages. The flexibility and power of LaTeX becomes obvious
when you start dealing with generating glossaries and indexes of thousand-page
books.</p>
<h2>How do I learn LaTeX?</h2>
<p>I'm not going to write a sub-par LaTeX tutorial just for the sake of it. It'd be
a waste of both my time and yours. Instead, I'll point you to <a href="http://en.wikibooks.org/wiki/LaTeX/Introduction">this
site</a>, which is a far better
intro to LaTeX than I could ever write. There is also substantial documentation
of many LaTeX features on that Wikibook, making it useful as a reference as well
as a tutorial.</p>
Web Servers urn:https-kaashif-co-uk:-2013-09-17-web-servers2013-09-17T00:00:00Z2013-09-17T00:00:00ZKaashif Hymabaccus
<p>Imagine you're a person on some sort of device, using the internet. You see all
of these websites and what do you ask? "How can I set up a web server?", of
course. If you did not ask that question, then this guide is not for you.
Anyway, down to business. You will need:</p>
<!-- more -->
<ul>
<li>A server (an old laptop, desktop or other computing device)</li>
<li>Access to the internet</li>
</ul>
<p>See, it's not <em>that</em> hard, despite what all of these hosting companies would
have you believe. The only money you might have to spend is for a domain, which
I'll get into after you've set up everything.</p>
<h2>Installing an OS</h2>
<p>The first thing you have to do when setting up a server is to choose the OS
which will form a base for all of your services to from. I recommend Debian
GNU/Linux, because it's pretty stable, there are many, many packages for it, and
lots of guides tend to be written for it due to its popularity as a server OS.
Feel free to choose something else, but remember that "apt-get" won't usually
work on CentOS or anything else. I'm not going to walk you through installing
Debian since it's so easy due to the way the installer works. There are a few
things you should note.</p>
<ul>
<li>Please connect to a router or switch via Ethernet. You don't want a wireless
connection between your router and server to become a bottleneck.</li>
<li>Either set up a static IP when the Debian installer prompts you, or set up a
static IP for your server's MAC address in your router's DHCP server settings.</li>
</ul>
<p>Both of these things are quite important. Without a static IP, your router won't
know where to send packets it receives on the HTTP port.</p>
<h2>Installing a web server</h2>
<p>There are quite a few web servers to choose from. The most popular two are Apace
and Nginx (pronounced "engine x"), and for the purposes of this guide, I'll be
using Apache. This isn't really for any reason, I'm just more familiar with it.
Run <code>apt-get install apache2</code> as root to install the apache2 web server. It
should enable itself as a service which runs at boot, but in case it doesn't
<code>update-rc.d apache2 defaults</code> should do it.</p>
<h2>Serving content</h2>
<p>You should be familiar with writing HTML, CSS and JavaScript, so I won't bore
you with the details of how to make your first website. All of the content
served by your server lives in <code>/var/www/</code>. If you list the files in that
directory, you should find only one file named "index.html". If you visit the IP
address of your server on your local area network (for example, 192.168.0.5, or
similar) in your browser, you should see the test web page for Apache. You can
edit index.html, add some more pages and some images, then you'll have a
website. But what good is a website if you can't access it from
the internet?</p>
<h2>IPv4 addresses and NAT</h2>
<p>Before you can get your server online, you have to understand how computers
communicate with each other across the internet. Each packet which is sent
anywhere on the internet contains the address of the recipient. This address is
an IP address, usually an IPv4 address, which you have come across before (think
xxx.xxx.xxx.xxx patterns of numbers). When you ping a website, say google.com,
that domain name is resolved into an IPv4 address using DNS. The details of DNS
are irrelevant, but it basically looks at the DNS records associated with a
domain, finds the A record, and translates the domain name into the IP of the A
record. This is what happens:</p>
<p><img src="http://kaashif.co.uk/static/images/outbound.png" alt="Outbound packets" /></p>
<p>So that's what you see when you try to communicate with someone else over the
internet. You see a single IP address which refers to their LAN. But what if
there are multiple computers on the LAN? The public, or external, IP address
usually points to the router. This router takes packets and routes them to the
computers on the LAN. This means that the computers on the LAN are not directly
accessible from the internet, they must first go through the router. This also
means that the LAN is completely separate from the rest of the internet - there
is no end to end connectivity, all packets go through a router. The translation
of the external IP (the one everyone else sees) and the internal IP (the one you
and your router see) is known as Network Address Translation, or NAT.</p>
<p>The main consequence of this is that you cannot point someone to your server
using the internal IP address (192.168.x.x, usually), you must instead set up a
rule for your router to pass all incoming requests on a port (for HTTP that is
port 80) to the internal IP address of your server.</p>
<p><img src="http://kaashif.co.uk/static/images/inbound.png" alt="Inbound packets" /></p>
<h2>Getting online</h2>
<p>The first order of business is getting a domain name. If you visit <a href="http://freedns.afraid.org/">this
website</a> and create an account, you will be able to
choose from a wide variety of free subdomains (e.g. your-name.mooo.com). You
cannot create a free domain (your-name.com), since those require registration at
a domain registrar (like Namecheap or GoDaddy) in exchange for money. This means
you are stuck with "my-cool-site.mooooo.com" or whatever you pick, until you
stop being cheap. There are instructions on setting up your DNS records to
update automatically, but for now you can visit <a href="http://checkip.dyndns.org/">this
page</a> and put that into the A record for your
domain. If you do try to go to this domain, you won't be able to, because your
ports aren't open! Since instructions vary from router to router, you should
just go to <a href="http://portforward.com/english/routers/port_forwarding/">this helpful website</a>, find your
router and follow the instructions to forward port 80 to the static IP of your
server.</p>
<p>Your website may or may not be fully functional at this point. If not, search
the internet, ask on StackOverflow or whatever your preferred help website is.
The important thing is that you do not link to this post, because I won't take
any responsibility for what you do with your server.</p>
How to use GPGurn:https-kaashif-co-uk:-2013-08-11-how-to-use-gpg2013-08-11T00:00:00Z2013-08-11T00:00:00ZKaashif Hymabaccus
<h2>Why am I writing this?</h2>
<p>I have looked up "how to use gpg" so many times, on so many websites,
and have found every guide to be focused on something I don't use or
worded in such a way that I get confused and revoke all of my keys (that
hasn't actually happened...yet). I thought I'd whip up a quick guide</p>
<!-- more -->
<p>that could serve as a reference for future Kaashif, who may not remember
anything about GPG other than <code>gpg -ear</code> and <code>gpg -d</code>.</p>
<h2>Installing GPG</h2>
<p>This is easy. Most distros come with it, for package signing among
other things. The ones that don't have it easily installable from
their package repos as either "gpg", "gpg2", "gnupg" or "gnupg2". While
GPG and GPG 2 are actually different programs, many distros don't make
the distinction, since hardly anyone uses GPG1 anymore.</p>
<h2>Generating a key</h2>
<p><code>gpg --gen-key</code>
You have to be an idiot to get this wrong. Defaults are fine, unless
someone has broken RSA with quantum magic. Make sure the email is right.</p>
<h2>After generating a key</h2>
<p>Two things:<br/>
1. Create a revocation cerificate <code>gpg --output revokecert --gen-revoke $KEY</code><br/>
2. Back up everything<br/>
I somehow managed to lose two GPG private keys, of which I had only
generated one revocation certificate. I'll never make that mistake again
- I have it backed up on a CD, on a USB drive and on a server. Nothing
off-site, though, so someone could theoretically burn down my house and
I'd lose everything.</p>
<h2>How to use your newfound encryption powers</h2>
<p>To encrypt plain text from stdin, just do <code>gpg -ear $KEY</code> The $KEY
refers to the <em>recipient</em>. It's fine to use your own pubkey when
testing, but you have to use the pubkey of the person who will decrypt
the text! That's the cornerstone of everything to do with keys. Imagine
someone saying "I'll send you this lock only I have the key to", that
would be idiotic when they have the means available to send you a lock
only you have the key to.</p>
<p>If someone sends you a properly encrypted message, invoke <code>gpg -d</code>.
Since you should only have one private key at this point, it'll take
input from stdin which, hopefully, has been encrypted with your pubkey
and can be decrypted with your private key.</p>
<h2>Signatures</h2>
<p>Let's say someone doesn't want to use GPG because they're too lazy (a
very realistic scenario). Maybe you're posting on a mailing list, where
GPG isn't necessary, and just annoys everyone. You still want people to
know that you sent the message and not an imposter with fake headers,
correct? Well you're in luck, you can attach a GPG signature to your
messages. This is basically a copy of the message which can be decrypted
with your public key. Since you are the only person with the private
key, you must have been the person to sign the message. The command to
use is <code>gpg --clearsign</code>. No need to specify a key, because you only
have one private key</p>
<h2>Encrypting files into a binary format</h2>
<p>Remember using <code>gpg -ear</code>? The "a" means ASCII. Take that out and it
magically outputs a binary file, with the input filename and a ".gpg"
extension.</p>
<h2>What is my key?</h2>
<pre><code>-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.14 (GNU/Linux)
mQENBFH38X4BCAC25Ra3yhPjXtqWmYbxHHG3Esn4en9z0yWCE4AukNUm8MX1kPna
1TqBFkw8WhhQKV1v+U0T18zoWwpMm1tUJdVWaUVc2/4iyR49d3SI81K+g7CuQuz1
YjyMG1zOzWeswfcJjZF4+Ti/fZIR7fNc+neAfGg9WMpAvdfMWAjuuV44vrAZQey+
bgpVN3uEJYntyzJRkgkqTNS2JatrL0IOVmfKtrNzuHQy5THJn2uDm9+Eg/tVe0Jp
bfz5AJKqqGUpE9jitCKc55n5xvJrlOQyWaIWuSiaKRRRiupswVAVoEa5y8JOckyb
l5q10F6hCIWq+ohGV/huGSvMNAMoU9+8vdXbABEBAAG0R0thYXNoaWYgSHltYWJh
Y2N1cyAoUHJvZ3JhbW1lciwgc3lzYWRtaW4pIDxrYWFzaGlmaHltYWJhY2N1c0Bn
bWFpbC5jb20+iQE4BBMBAgAiBQJR9/F+AhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIe
AQIXgAAKCRBa0dhzPoELBO6IB/40BRr2DrajYJ6y9yGpUHJIPT+KC90i62r0D3vM
raHz3/shBOOJEeyqhxcmwByWkhBRjEkkt3xgaWlj6XzvuBY457p54f6bIKeKwXpT
WJAVdhM2VSQdTyX/Svo3lnVYv0bozbRIb88M6FvTF8Cv631zSImAAKuPD2X7ZYl1
2p3gLVWB//vkAr2WAJjq1qrcmoVtixbs5HeM65MR7hcE30vCJzswev7m+4mQXFR3
LoMNoC1Zx2iYBNgUNMpoGaGdPTohMD8gCklr86R+OzCORrWyKBl4qc608Dmt+myG
Rs5OH6c9yBYiHIfc9UYaMoPXIdvQBwO/4bOOOZbwqp7Onjy7uQENBFH38X4BCADo
Obc//asuZBCJtf2GSZGrWvWJVYv06a3noIBb9TG+6fAZ40c1zlPJzCSa6wU8aeXn
6UvbQI7W014wWlO+JvvpaZzEsJ+qnxkZQqEne1BqTb32OmIJLInDQhsZsoR/PNSQ
KRUshS9kLBSez2EBA7rV2cJ6X2a2Dtb75PlzysjmHrws1ZOelYRu3DorYfUQ05jL
IhCFQRCHgryK0yD+mZy55F9JHGHWooTGTStBmNW5dDpXdPUcHeZm5ICXkuQC1tzV
apW/vy77PB3ZVTxX/wEuATgbfbPiGpvqVQWr+CTnQOpVGoAMByHdlkHaKtGnjUlI
vp9Xep+cbS4GOjpJ0khZABEBAAGJAR8EGAECAAkFAlH38X4CGwwACgkQWtHYcz6B
CwTqWgf/ROZ5mmloJ/86iCeGzxzIHMWF4m8dwMGa3SZ310umsl83ydM5hixmRM43
cABzEq7sbgirmg31GAkGwU0dQ6z9R4TSSbDFS8nz1EztvuNabMTPfc9AdtE/ig2P
o6Hul8n3A6330RDk94QtqBw3Eppsr6PgQ+hA2rbfy7YRca6p100cC9yOAdc5gvmr
0qTfFtX71/Nxrcok+88uOk3PMwyvW/6HCs9m7jfx2RZe3DmQ5ykZ7qMe5YM4xGKy
SteLT0+yQNETungL5lyC0V/JAgAIQXItQytEXL9TEKeEP2jCrRD4lDa98qMkbNDI
ba2AGJ2aj3+u9787VaY7bSRFQH8Kwg==
=1hN6
-----END PGP PUBLIC KEY BLOCK-----
</code></pre>
<h2>How did I generate that?</h2>
<p><code>gpg --export -a $KEY</code>. Once again, you can take out the "-a" and add an
"--output <file>" to get binary output.</p>
<h2>Where do I find more public keys?</h2>
<p>Go to a keyserver, like pgp.mit.edu. You should also submit your public
key there by invoking <code>gpg --send-keys --keyserver pgp.mit.edu $KEY</code>.
The key will propagate to other servers, so you <em>cannot</em> delete or edit
a key once it's there. Make sure everything is correct and backed up.
Don't search my name, I don't want to be embarrassed. If you must, my
key is the most up to date one, I lost the old one, and revoked the
other one...due to losing it. Do what I say, not what I do.</p>
<h2>Summary</h2>
<p><code>gpg -ear $KEY</code> - Encrypt plaintext from stdin</p>
<p><code>gpg -d</code> - Decrypt plaintext from stdin</p>
<p><code>gpg --export -a $KEY</code> - Export ASCII-armoured key to stdout</p>
<p><code>gpg --import $FILE</code> - Import key from a file</p>
<p><code>gpg --clearsign</code> - Sign a message from stdin, leaving the message human-readable</p>
<p><code>gpg --detach-sig $SIGFILE $FILE</code> - Sign a file and create a detached signature in another file</p>
<p><code>gpg --some-sort-of-command --output $FILE</code> - Do something, then output to a file</p>