Playing with LDoms, OpenBSD and Solaris
2017-12-03
A few weeks ago, I got my hands on a Sun T2000 server. It's got an UltraSparc T1 CPU, 32 threads, 32 GB of memory, a Sun XVR-300 GPU and what sounds like a huge jet engine mounted at the front.
It's a great machine (although maybe not as a workstation...), and there are a few things unique to SPARC that I've really been looking forward to playing around with. Mostly LDoms (logical domains - Sun's virtualization technology), OpenBSD on a beefy sparc64 (compared to my older UltraSparcs anyway) and Solaris (just as a curiosity).
Which OS to run on the bare metal?
I have a few choices here, but it essentially boiled down to OpenBSD or Solaris. There are a few factors to compare, but I settled on OpenBSD, for a few reasons.
Better software support. No software nowadays has Solaris in mind and I have no clue where to get updates for free. Oracle makes it impossible to download anything related to Solaris without having a support contract. OpenBSD has regular updates (for free) and a ports tree filled with up-to-date software.
I don't have any clue how to use Solaris and learning it is a waste of time.
If there's a problem, I won't know where to ask for help. I can't contact the devs and there isn't really a big Solaris community online.
And OpenBSD supports all the cool stuff I want to do fairly well anyway, so picking it is a no-brainer.
Someone not acquainted with the state of affairs when using old Sun hardware might ask why I'm not using some Linux distro. Surely there must be one out there that does what I need, and isn't Linux more popular and thus more well-supported than OpenBSD? Well there is a distro that does LDoms...Oracle Linux.
I think I'll pass on that one. And I can't find anyone running Debian as a primary domain on SPARC, so I guess you can't do it.
Installing the primary OS
This went so painlessly I don't even feel the need to write anything about it. Just put the disk in and everything works as expected. I guess you could netboot, it's not too difficult, but using an install DVD is way easier.
Setting up LDoms
This is actually really easy too. OpenBSD supports LDoms remarkably well, due to the efforts of Mark Kettenis (and others, probably). Take a look at this guide to see how I did it. With the T2000, there are a few differences, but these are confined to the ILOM/ALOM.
Where Ted types start /SYS, I type poweron. He types start
/SP/console, I type console.
There is little real difference between the ALOM of the T2000 and the
ILOM of the T5120 as far as I can tell. ldomctl from OpenBSD can't
see the I/ALOM anyway, since that lives below OpenBoot.
I ended up with the following config:
domain primary {
vcpu 16
memory 16G
}
domain openbsd {
vcpu 8
memory 8G
vdisk "/home/kaashif/vm/install62.iso"
vdisk "/home/kaashif/vm/openbsd.img"
vnet
}
domain solaris {
vcpu 8
memory 8G
vdisk "/home/kaashif/vm/install-solaris-10.iso"
vdisk "/home/kaashif/vm/solaris.img"
vnet
}
Where solaris.img and openbsd.img are files created with
truncate:
$ gtruncate -s 10G openbsd.img
This creates a sparse file rather than actually filling some file with zeroes.
Networking
For this, I just bridged vnet0, vnet1 and em0 (the interface connected to my LAN):
# ifconfig bridge0 up add vnet0 add vnet1 add em0
This lets the LDoms speak to my LAN as if they were real. I haven't got DHCP to work, but that hasn't been an issue.
What to do with the LDoms?
As of now, I haven't actually got the Solaris LDom to boot. I boot from the install media, install, but then it refuses to boot from the disk. Since this was just a toy, I might just abandon it.
In future, I might give the Debian SPARC port a spin, just to see if it's not a steaming mess. Same goes for NetBSD. Expect a post or two in the future about these.
Dmesg!
Here is the dmesg from before I sliced it up with LDoms:
console is /virtual-devices@100/console@1
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2017 OpenBSD. All rights reserved. https://www.OpenBSD.org
OpenBSD 6.2-current (GENERIC.MP) #307: Wed Oct 11 15:17:26 MDT 2017
deraadt@sparc64.openbsd.org:/usr/src/sys/arch/sparc64/compile/GENERIC.MP
real mem = 34225520640 (32640MB)
avail mem = 33610276864 (32053MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root: SPARC Enterprise T2000
cpu0 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu1 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu2 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu3 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu4 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu5 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu6 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu7 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu8 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu9 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu10 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu11 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu12 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu13 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu14 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu15 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu16 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu17 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu18 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu19 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu20 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu21 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu22 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu23 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu24 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu25 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu26 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu27 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu28 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu29 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu30 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
cpu31 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1200 MHz
vbus0 at mainbus0
"flashprom" at vbus0 not configured
cbus0 at vbus0
vldc0 at cbus0
vldcp0 at vldc0 chan 0x0: ivec 0x0, 0x1 channel "hvctl"
"ldom-primary" at vldc0 chan 0x1 not configured
"fmactl" at vldc0 chan 0x3 not configured
vldc1 at cbus0
"ldmfma" at vldc1 chan 0x4 not configured
vldc2 at cbus0
vldcp1 at vldc2 chan 0x14: ivec 0x28, 0x29 channel "spds"
"system-management" at vldc2 chan 0xd not configured
vcons0 at vbus0: ivec 0x111, console
vrtc0 at vbus0
"fma" at vbus0 not configured
"sunvts" at vbus0 not configured
"sunmc" at vbus0 not configured
"explorer" at vbus0 not configured
"led" at vbus0 not configured
"flashupdate" at vbus0 not configured
"ncp" at vbus0 not configured
vpci0 at mainbus0: bus 2 to 7, dvma map 80000000-ffffffff
pci0 at vpci0
ppb0 at pci0 dev 0 function 0 "PLX PEX 8532" rev 0xbc
pci1 at ppb0 bus 3
ppb1 at pci1 dev 1 function 0 "PLX PEX 8532" rev 0xbc
pci2 at ppb1 bus 4
em0 at pci2 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x795, address 00:14:4f:e1:c8:82
em1 at pci2 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x796, address 00:14:4f:e1:c8:83
ppb2 at pci1 dev 2 function 0 "PLX PEX 8532" rev 0xbc
pci3 at ppb2 bus 5
ppb3 at pci1 dev 8 function 0 "PLX PEX 8532" rev 0xbc: msi
pci4 at ppb3 bus 6
ppb4 at pci1 dev 9 function 0 "PLX PEX 8532" rev 0xbc
pci5 at ppb4 bus 7
mpi0 at pci5 dev 0 function 0 "Symbios Logic SAS1064E" rev 0x02: msi
mpi0: UNUSED, firmware 1.9.0.0
scsibus1 at mpi0: 63 targets
sd0 at scsibus1 targ 0 lun 0: <HITACHI, H101414SCSUN146G, SA25> SCSI3 0/direct fixed naa.5000cca00098ddfc
sd0: 140009MB, 512 bytes/sector, 286739329 sectors
vpci1 at mainbus0: bus 2 to 9, dvma map 80000000-ffffffff
pci6 at vpci1
ppb5 at pci6 dev 0 function 0 "PLX PEX 8532" rev 0xbc
pci7 at ppb5 bus 3
ppb6 at pci7 dev 1 function 0 "PLX PEX 8532" rev 0xbc
pci8 at ppb6 bus 4
ppb7 at pci8 dev 0 function 0 "Intel 41210 PCIE-PCIX" rev 0x09
pci9 at ppb7 bus 5
ebus0 at pci9 dev 2 function 0 "Acer Labs M1533 ISA" rev 0x00
com0 at ebus0 addr 3f8-3ff ivec 0x2: ns16550a, 16 byte fifo
ohci0 at pci9 dev 5 function 0 "Acer Labs M5237 USB" rev 0x03: ivec 0x7c1, version 1.0, legacy support
ohci1 at pci9 dev 6 function 0 "Acer Labs M5237 USB" rev 0x03: ivec 0x7c3, version 1.0, legacy support
pciide0 at pci9 dev 8 function 0 "Acer Labs M5229 UDMA IDE" rev 0xc4: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide0: using ivec 0x7c4 for native-PCI interrupt
atapiscsi0 at pciide0 channel 0 drive 0
scsibus2 at atapiscsi0: 2 targets
cd0 at scsibus2 targ 0 lun 0: <TEAC, DW-224SL-R, 1.0B> ATAPI 5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Acer Labs OHCI root hub" rev 1.00/1.00 addr 1
usb1 at ohci1: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Acer Labs OHCI root hub" rev 1.00/1.00 addr 1
ppb8 at pci8 dev 0 function 2 "Intel 41210 PCIE-PCIX" rev 0x09
pci10 at ppb8 bus 6
ppb9 at pci7 dev 2 function 0 "PLX PEX 8532" rev 0xbc
pci11 at ppb9 bus 7
em2 at pci11 dev 0 function 0 "Intel 82571EB" rev 0x06: ivec 0x7d6, address 00:14:4f:e1:c8:84
em3 at pci11 dev 0 function 1 "Intel 82571EB" rev 0x06: ivec 0x7d7, address 00:14:4f:e1:c8:85
ppb10 at pci7 dev 8 function 0 "PLX PEX 8532" rev 0xbc: msi
pci12 at ppb10 bus 8
radeondrm0 at pci12 dev 0 function 0 "ATI FireGL V3100" rev 0x80
drm0 at radeondrm0
radeondrm0: ivec 0x7d4
ppb11 at pci7 dev 9 function 0 "PLX PEX 8532" rev 0xbc: msi
pci13 at ppb11 bus 9
umass0 at uhub0 port 2 configuration 1 interface 0 "USB2.0 Flash Disk" rev 2.00/0.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus3 at umass0: 2 targets, initiator 0
sd1 at scsibus3 targ 1 lun 0: <USB2.0, Flash Disk, 2.60> SCSI2 0/direct removable serial.1221323400000000847F
sd1: 998MB, 512 bytes/sector, 2043904 sectors
uhub2 at uhub1 port 1 configuration 1 interface 0 "Atmel UHB124 hub" rev 1.10/3.00 addr 2
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
bootpath: /pci@780,0/pci@0,0/pci@9,0/scsi@0,0/disk@0,0
root on sd0a (8b6ac481aa65550f.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
WARNING: clock lost 6494 days -- CHECK AND RESET THE DATE!
BIOS signature incorrect 0 0
error: [drm:pid0:r100_cp_init_microcode] *ERROR* radeon_cp: Failed to load firmware "radeon-r300_cp"
error: [drm:pid0:r100_cp_init] *ERROR* Failed to load firmware!
drm:pid0:r300_startup *ERROR* failed initializing CP (-2).
drm:pid0:r300_init *ERROR* Disabling GPU acceleration
drm:pid0:radeon_bo_unpin *WARNING* 0x40027322390 unpin not necessary
radeondrm0: 1024x768, 8bpp
wsdisplay0 at radeondrm0 mux 1
wsdisplay0: screen 0 added (std, sun emulation)