Show Notes: extras.show/74

The post Brunch with Brent: Philip Muller | Jupiter Extras 74 first appeared on Jupiter Broadcasting.

Show Notes/Links: https://www.bsdnow.tv/340

The post Check My Sums | BSD Now 340 first appeared on Jupiter Broadcasting.

Show Notes: linuxunplugged.com/320

The post RHELhide | LINUX Unplugged 320 first appeared on Jupiter Broadcasting.

##Headlines
###DragonflyBSD 5.4 released

DragonFly version 5.4 brings a new system compiler in GCC 8, improved NUMA support, a large of number network and virtual machine driver updates, and updates to video support. This release is 64-bit only, as with previous releases.
The details of all commits between the 5.2 and 5.4 branches are available in the associated commit messages for 5.4.0rc and 5.4.0.

• Big-ticket items
• Much better support for asymmetric NUMA (Non-Uniform Memory Access) configurations. In particular, both the memory subsystem and the scheduler now understand the Threadripper 2990WX’s architecture. The scheduler will prioritize CPU nodes with direct-attached memory and the memory subsystem will normalize memory queues for CPU nodes without direct-attached memory (which improves cache locality on those CPUs).
• Incremental performance work. DragonFly as a whole is very SMP friendly. The type of performance work we are doing now mostly revolves around improving fairness for shared-vs-exclusive lock clashes, reducing cache ping-ponging due to non-contending SMP locks (i.e. massive use of shared locks on shared resources), and so forth.
• Major updates to dports brings us to within a week or two of FreeBSD’s ports as of this writing, in particular major updates to chromium, and making the whole mess work with gcc-8.
• Major rewriting of the tty clist code and the tty locking code, significantly improving concurrency across multiple ttys and ptys.
• GCC 8
• DragonFly now ships with GCC 8.0, and runs as the default compiler. It is also now used for building dports.
• GCC 4.7.4 and GCC 5.4.1 are still installed. 4.7.4 is our backup compiler, and 5.4.1 is still there to ensure a smooth transition, but should generally not be used. buildworld builds all three by default to ensure maximum compatibility.
• Many passes through world sources were made to address various warnings and errors the new GCC brought with it.
• HAMMER2
• HAMMER2 is recommended as the default root filesystem in non-clustered mode.
• Clustered support is not yet available.
• Increased bulkfree cache to reduce the number of iterations required.
• Fixed numerous bugs.
• Improved support on low-memory machines.
• Significant pre-work on the XOP API to help support future networked operations.
• Details
• Checksums
  MD5 (dfly-x86_64-5.4.0_REL.img) = 7277d7cffc92837c7d1c5dd11a11b98f
MD5 (dfly-x86_64-5.4.0_REL.iso) = 6da7abf036fe9267479837b3c3078408
MD5 (dfly-x86_64-5.4.0_REL.img.bz2) = a77a072c864f4b72fd56b4250c983ff1
MD5 (dfly-x86_64-5.4.0_REL.iso.bz2) = 4dbfec6ccfc1d59c5049455db914d499
• Downloads Links

DragonFly BSD is 64-bit only, as announced during the 3.8 release.

• USB: dfly-x86_64-5.4.0_REL.img as bzip2 file
• ISO: dfly-x86_64-5.4.0_REL.iso as bzip2 file
• Uncompressed ISO: dfly-x86_64-5.4.0_REL.iso (For use with VPS providers as an install image.)

###Down the Gopher hole with OpenBSD, Gophernicus, and TLS

In the early 2000s I thought I had seen the worst of the web – Java applets, Macromedia (>Adobe) Flash, animated GIFs, javascript snow that kept you warm in the winter by burning out your CPU, and so on. For a time we learned from these mistakes, and started putting the burden on the server-side – then with improvements in javascript engines we started abusing it again with JSON/AJAX and it all went down hill from there.

Like cloud computing, blockchains, machine learning and a tonne of other a la mode technologies around today – most users and service providers don’t need websites that consume 1GB of memory processing JS and downloading 50MB of compressed data just to read Alice’s one-page travel blog or Bob’s notes on porting NetBSD to his blood-pressure monitor.

Before the HTTP web we relied on Prestel/Minitel style systems, BBS systems, and arguably the most accessible of all – Gopher! Gopher was similar to the locally accessed AmigaGuide format, in that it allowed users to search and retrieve documents interactively, with links and cross-references. Its efficiency and distraction-free nature make it attractive to those who are tired of the invasive, clickbait, ad-filled, javascript-laden web2/3.x. But enough complaining and evangelism – here’s how to get your own Gopher Hole!

Gophernicus is a modern gopher daemon which aims to be secure (although it still uses inetd -_-); it’s even in OpenBSD ports so at least we can rely on it to be reasonably audited.

If you need a starting point with Gopher, SDF-EU’s wiki has a good article here.

• https://sdfeu.org/w/tutorials:gopher

Finally, if you don’t like gopher(1) – there’s always lynx(1) or NCSA Mosaic!

• https://cryogenix.net/NCSA_Mosaic_OpenBSD.html

I’ve added TLS support to Gophernicus so you don’t need to use stunnel anymore. The code is ugly and unpolished though so I wouldn’t recommend for production use.

• https://github.com/0x16h/gophernicus
• https://github.com/0x16h/gophernicus/blob/master/INSTALL.openbsd

##News Roundup
###OpenBSD in Stereo with Linux VFIO

I use a Huawei Matebook X as my primary OpenBSD laptop and one aspect of its hardware support has always been lacking: audio never played out of the right-side speaker. The speaker did actually work, but only in Windows and only after the Realtek Dolby Atmos audio driver from Huawei was installed. Under OpenBSD and Linux, and even Windows with the default Intel sound driver, audio only ever played out of the left speaker.
Now, after some extensive reverse engineering and debugging with the help of VFIO on Linux, I finally have audio playing out of both speakers on OpenBSD.

• VFIO

The Linux kernel has functionality called VFIO which enables direct access to a physical device (like a PCI card) from userspace, usually passing it to an emulator like QEMU.
To my surprise, these days, it seems to be primarily by gamers who boot Linux, then use QEMU to run a game in Windows and use VFIO to pass the computer’s GPU device through to Windows.
By using Linux and VFIO, I was able to boot Windows 10 inside of QEMU and pass my laptop’s PCI audio device through to Windows, allowing the Realtek audio drivers to natively control the audio device. Combined with QEMU’s tracing functionality, I was able to get a log of all PCI I/O between Windows and the PCI audio device.

• Using VFIO

To use VFIO to pass-through a PCI device, it first needs to be stubbed out so the Linux kernel’s default drivers don’t attach to it. GRUB can be configured to instruct the kernel to ignore the PCI audio device (8086:9d71) and explicitly enable the Intel IOMMU driver by adding the following to /etc/default/grub and running update-grub
With the audio device stubbed out, a new VFIO device can be created from it
Then the VFIO device (00:1f.3) can be passed to QEMU
I was using my own build of QEMU for this, due to some custom logging I needed (more on that later), but the default QEMU package should work fine. The events.txt was a file of all VFIO events I wanted logged (which was all of them).
Since I was frequently killing QEMU and restarting it, Windows 10 wanted to go through its unexpected shutdown routine each time (and would sometimes just fail to boot again). To avoid this and to get a consistent set of logs each time, I used qemu-img to take a snapshot of a base image first, then boot QEMU with that snapshot. The snapshot just gets thrown away the next time qemu-img is run and Windows always starts from a consistent state.
QEMU will now log each VFIO event which gets saved to a debug-output file.
With a full log of all PCI I/O activity from Windows, I compared it to the output from OpenBSD and tried to find the magic register writes that enabled the second speaker. After days of combing through the logs and annotating them by looking up hex values in the documentation, diffing runtime register values, and even brute-forcing it by mechanically duplicating all PCI I/O activity in the OpenBSD driver, nothing would activate the right speaker.
One strange thing that I noticed was if I booted Windows 10 in QEMU and it activated the speaker, then booted OpenBSD in QEMU without resetting the PCI device’s power in-between (as a normal system reboot would do), both speakers worked in OpenBSD and the configuration that the HDA controller presented was different, even without any changes in OpenBSD.

A Primer on Intel HDA
Most modern computers with integrated sound chips use an Intel High Definition Audio (HDA) Controller device, with one or more codecs (like the Realtek ALC269) hanging off of it. These codecs do the actual audio processing and communicate with DACs and ADCs to send digital audio to the connected speakers, or read analog audio from a microphone and convert it to a digital input stream. In my Huawei Matebook X, this is done through a Realtek ALC298 codec.
On OpenBSD, these HDA controllers are supported by the azalia(4) driver, with all of the per-codec details in the lengthy azalia_codec.c file. This file has grown quite large with lots of codec- and machine-specific quirks to route things properly, toggle various GPIO pins, and unmute speakers that are for some reason muted by default.
The azalia driver talks to the HDA controller and sets up various buffers and then walks the list of codecs. Each codec supports a number of widget nodes which can be interconnected in various ways. Some of these nodes can be reconfigured on the fly to do things like turning a microphone port into a headphone port.
The newer Huawei Matebook X Pro released a few months ago is also plagued with this speaker problem, although it has four speakers and only two work by default. A fix is being proposed for the Linux kernel which just reconfigures those widget pins in the Intel HDA driver. Unfortunately no pin reconfiguration is enough to fix my Matebook X with its two speakers.
While reading more documentation on the HDA, I realized there was a lot more activity going on than I was able to see through the PCI tracing.
For speed and efficiency, HDA controllers use a DMA engine to transfer audio streams as well as the commands from the OS driver to the codecs. In the output above, the CORBWP=0; size=256 and RIRBRP=0, size=256 indicate the setup of the CORB (Command Output Ring Buffer) and RIRB (Response Input Ring Buffer) each with 256 entries. The HDA driver allocates a DMA address and then writes it to the two CORBLBASE and CORBUBASE registers, and again for the RIRB.
When the driver wants to send a command to a codec, such as CORB_GET_PARAMETER with a parameter of COP_VOLUME_KNOB_CAPABILITIES, it encodes the codec address, the node index, the command verb, and the parameter, and then writes that value to the CORB ring at the address it set up with the controller at initialization time (CORBLBASE/CORBUBASE) plus the offset of the ring index. Once the command is on the ring, it does a PCI write to the CORBWP register, advancing it by one. This lets the controller know a new command is queued, which it then acts on and writes the response value on the RIRB ring at the same position as the command (but at the RIRB’s DMA address). It then generates an interrupt, telling the driver to read the new RIRBWP value and process the new results.
Since the actual command contents and responses are handled through DMA writes and reads, these important values weren’t showing up in the VFIO PCI trace output that I had gathered. Time to hack QEMU.

• Logging DMA Memory Values in QEMU

Since DMA activity wouldn’t show up through QEMU’s VFIO tracing and I obviously couldn’t get Windows to dump these values like I could in OpenBSD, I could make QEMU recognize the PCI write to the CORBWP register as an indication that a command has just been written to the CORB ring.
My custom hack in QEMU adds some HDA awareness to remember the CORB and RIRB DMA addresses as they get programmed in the controller. Then any time a PCI write to the CORBWP register is done, QEMU fetches the new CORB command from DMA memory, decodes it into the codec address, node address, command, and parameter, and prints it out. When a PCI read of the RIRBWP register is requested, QEMU reads the response and prints the corresponding CORB command that it stored earlier.
With this hack in place, I now had a full log of all CORB commands and RIRB responses sent to and read from the codec:
An early version of this patch left me stumped for a few days because, even after submitting all of the same CORB commands in OpenBSD, the second speaker still didn’t work. It wasn’t until re-reading the HDA spec that I realized the Windows driver was submitting more than one command at a time, writing multiple CORB entries and writing a CORBWP value that was advanced by two. This required turning my CORB/RIRB reading into a for loop, reading each new command and response between the new CORBWP/RIRBWP value and the one previously seen.
Sure enough, the magic commands to enable the second speaker were sent in these periods where it submitted more than one command at a time.

• Minimizing the Magic

The full log of VFIO PCI activity from the Windows driver was over 65,000 lines and contained 3,150 CORB commands, which is a lot to sort through. It took me a couple more days to reduce that down to a small subset that was actually required to activate the second speaker, and that could only be done through trial and error:

• Boot OpenBSD with the full list of CORB commands in the azalia driver
• Comment out a group of them
• Compile kernel and install it, halt the QEMU guest
• Suspend and wake the laptop, resetting PCI power to the audio device to reset the speaker/Dolby initialization and ensure the previous run isn’t influencing the current test (I’m guessing there is an easier to way to reset PCI power than suspending the laptop, but oh well)
• Start QEMU, boot OpenBSD with the new kernel
• Play an MP3 with mpg123 which has alternating left- and right-channel audio and listen for both channels to play

This required a dozen or so iterations because sometimes I’d comment out too many commands and the right speaker would stop working. Other times the combination of commands would hang the controller and it wouldn’t process any further commands. At one point the combination of commands actually flipped the channels around so the right channel audio was playing through the left speaker.

• The Result

After about a week of this routine, I ended up with a list of 662 CORB commands that are needed to get the second speaker working. Based on the number of repeated-but-slightly-different values written with the 0x500 and 0x400 commands, I’m guessing this is some kind of training data and that this is doing the full Dolby/Atmos system initialization, not just turning on the second speaker, but I could be completely wrong.
In any case, the stereo sound from OpenBSD is wonderful now and I can finally stop downmixing everything to mono to play from the left speaker. In case you ever need to do this, sndiod can be run with -c 0:0 to reduce the channels to one.
Due to the massive size of the code needed for this quirk, I’m not sure if I’ll be committing it upstream in OpenBSD or just saving it for my own tree. But at least now the hardware support chart for my Matebook is all yeses for the things I care about.
I’ve also updated the Linux bug report that I opened before venturing down this path, hoping one of the maintainers of that HDA code that works at Intel or Realtek knew of a solution I could just port to OpenBSD. I’m curious to see what they’ll do with it.

###Why BSD/OS is the best candidate for being the only tested legally open UNIX

• Introduction

The UNIX® system is an old operating system, possibly older than many of the readers of this post. However, despite its age, it still has not been open sourced completely. In this post, I will try to detail which parts of which UNIX systems have not yet been open sourced. I will focus on the legal situation in Germany in particular, taking it representative of European law in general – albeit that is a stretch, knowing the diversity of European jurisdictions. Please note that familiarity with basic terms of copyright law is assumed.

• Ancient UNIX

The term “Ancient UNIX” refers to the versions of UNIX up to and including Seventh Edition UNIX (1979) including the 32V port to the VAX. Ancient UNIX was created at Bell Laboratories, a subsidiary of AT&T at the time. It was later transferred of the AT&T UNIX Support Group, then AT&T Information Systems and finally the AT&T subsidiary UNIX System Laboratories, Inc. (USL). The legal situation differs between the United States of America and Germany.
In a ruling as part of the UNIX System Laboratories, Inc. v. Berkeley Software Design, Inc. (USL v. BSDi) case, a U.S. court found that USL had no copyright to the Seventh Edition UNIX system and 32V – arguably, by extension, all earlier versions of Ancient UNIX as well – because USL/AT&T had failed to affix copyright notices and could not demonstrate a trade secret. Due to the obsessive tendency of U.S. courts to consider themselves bound to precedents (cf. the infamous Pierson v. Post case), it can be reasonably expected that this ruling would be honored and applied in subsequent cases. Thus under U.S. law, Ancient UNIX can be safely assumed to belong in the public domain.
The situation differs in Germany. Unlike the U.S., copyright never needed registration in order to exist. Computer programs are works in the sense of the German 1965 Act on Copyright and Related Rights (Copyright Act, henceforth CopyA) as per CopyA § 2(1) no. 1. Even prior to the amendment of CopyA § 2(1) to include computer programs, computer programs have been recognized as copyrightable works by the German Supreme Court (BGHZ 112, 264 Betriebssystem, no. 19); CopyA § 137d(1) rightly clarifies that. The copyright holder at 1979 would still have been USL via Bell Labs and AT&T. Copyright of computer programs is transferred to the employer upon creation under CopyA § 69(1).
Note that this does not affect expiry (Daniel Kaboth/Benjamin Spies, commentary on CopyA §§ 69a‒69g, in: Hartwig Ahlberg/Horst-Peter Götting (eds.), Urheberrecht: UrhG, KUG, VerlG, VGG, Kommentar, 4th ed., C. H. Beck, 2018, no. 16 ad CopyA § 69b; cf. Bundestag-Drucksache [BT-Drs.] 12/4022, p. 10). Expiry occurs 70 years after the death of the (co-)author that died most recently as per CopyA § 65(1) and 64; this has been the case since at least the 1960s, meaning there is no way for copyright to have expired already (old version, as per Bundesgesetzblatt Part I No. 51 of September 16, 1965, pp. 1273‒1294).
In Germany, private international law applies the so-called “Territorialitätsprinzip” for intellectual property rights. This means that the effect of an intellectual property right is limited to the territory of a state (Anne Lauber-Rönsberg, KollisionsR, in: Hartwig Ahlberg/Horst-Peter Götting (eds.), ibid., pp. 2241 et seqq., no. 4). Additionally, the “Schutzlandprinzip” applies; this means that protection of intellectual property follows the lex loci protectionis, i.e. the law of the country for which protection is sought (BGH GRUR 2015, 264 HiHotel II, no. 25; BGH GRUR 2003, 328 Sender Felsberg, no. 24), albeit this is criticized in parts of doctrine (Lauber-Rönsberg, ibid., no. 10). The “Schutzlandprinzip” requires that the existence of an intellectual property right be verified as well (BGH ZUM 2016, 522 Wagenfeld-Leuchte II, no. 19).
Thus, in Germany, copyright on Ancient UNIX is still alive and well. Who has it, though? A ruling by the U.S. Court of Appeals, Tenth Circuit, in the case of The SCO Group, Inc. v. Novell, Inc. (SCO v. Novell) in the U.S. made clear that Novell owns the rights to System V – thus presumably UNIX System III as well – and Ancient UNIX, though SCO acquired enough rights to develop UnixWare/OpenServer (Ruling 10-4122 [D.C. No. 2:04-CV-00139-TS], pp. 19 et seq.). Novell itself was purchased by the Attachmate Group, which was in turn acquired by the COBOL vendor Micro Focus. Therefore, the rights to SVRX and – outside the U.S. – are with Micro Focus right now. If all you care about is the U.S., you can stop reading about Ancient UNIX here.
So how does the Caldera license factor into all of this? For some context, the license was issued January 23, 2002 and covers Ancient UNIX (V1 through V7 including 32V), specifically excluding System III and System V. Caldera, Inc. was founded in 1994. The Santa Cruz Operation, Inc. sold its rights to UNIX to Caldera in 2001, renamed itself to Tarantella Inc. and Caldera renamed itself The SCO Group. Nemo plus iuris ad alium transferre potest quam ipse habet; no one can transfer more rights than he has. The question now becomes whether Caldera had the rights to issue the Caldera license.
I’ve noted it above but it needs restating: Foreign decisions are not necessarily accepted in Germany due to the “Territorialitätsprinzip” and “Schutzlandprinzip” – however, I will be citing a U.S. ruling for its assessment of the facts for the sake of simplicity. As per ruling 10-4122, “The district court found the parties intended for SCO to serve as Novell’s agent with respect to the old SVRX licenses and the only portion of the UNIX business transferred outright under the APA [asset purchase agreement] was the ability to exploit and further develop the newer UnixWare system. SCO was able to protect that business because it was able to copyright its own improvements to the system. The only reason to protect the earlier UNIX code would be to protect the existing SVRX licenses, and the court concluded Novell retained ultimate control over that portion of the business under the APA.” The relevant agreements consist of multiple pieces:
the base Asset Purchase Agreement “APA” (Part I)
the base Asset Purchase Agreement “APA” (Part II)
the Operating Agremeent and Amendment 1 to the APA
the Amendment 2 to the APA
The APA dates September 19, 1995, from before the Caldera license. Caldera cannot possibly have acquired rights that The Santa Cruz Operation, Inc. itself never had. Furthermore, I’ve failed to find any mention of Ancient UNIX; all that is transferred is rights to SVRX. Overall, I believe that the U.S. courts’ assesment of the facts represents the situation accurately. Thus for all intents and purposes, UNIX up to and including System V remained with Novell/Attachmate/Micro Focus. Caldera therefore never had any rights to Ancient UNIX, which means it never had the rights to issue the Caldera license. The Caldera license is null and void – in the U.S. because the copyright has been lost due to formalities, everywhere else because Caldera never had the rights to issue it.
The first step to truly freeing UNIX would this be to get Micro Focus to re-issue the Caldera license for Ancient UNIX, ideally it would now also include System III and System V.

• BSD/OS

Another operating system near UNIX is of interest. The USL v. BSDi lawsuit includes two parties: USL, which we have seen above, and Berkeley Software Design, Inc. BSDi sold BSD/386 (later BSD/OS), which was a derivative of 4.4BSD. The software parts of the BSDi company were acquired by Wind River Systems, whereas the hardware parts went to iXsystems. Copyright is not disputed there, though Wind River Systems ceased selling BSD/OS products 15 years ago, in 2003. In addition, Wind River System let their trademark on BSD expire, though this is without consequence for copyright.
BSD/OS is notable in the sense that it powered much of early internet infrastructure. Traces of its legacy can still be found on Richard Stevens’ FAQ.
To truly make UNIX history free, BSD/OS would arguably also need to see a source code release. BSD/OS at least in its earliest releases under BSDi would ship with source code, though under a non-free license, far from BSD or even GPL licensing.

• System V

The fate of System V as a whole is difficult to determine. Various licenses have been granted to a number of vendors (Dell UNIX comes to mind; HP for HP-UX, IBM for AIX, SGI UNIX, etc.). Sun released OpenSolaris – notoriously, Oracle closed the source to Solaris again after its release –, which is a System V Release 4 descendant. However, this means nothing for the copyright or licensing status of System V itself. Presumably, the rights with System V still remain with Novell (now Micro Focus): SCO managed to sublicense rights to develop and sell UnixWare/OpenServer, themselves System V/III descendants, to unXis, Inc. (now known as Xinuos, Inc.), which implies that Xinuos is not the copyright holder of System V.
Obviously, to free UNIX, System V and its entire family of descendants would also need to be open sourced. However, I expect tremendous resistance on part of all the companies mentioned. As noted in the “Ancient UNIX” section, Micro Focus alone would probably be sufficient to release System V, though this would mean nothing for the other commercial System V derivatives.

• Newer Research UNIX

The fate of Bell Labs would be a different one; it would go on to be purchased by Lucent, now part of Nokia. After commercial UNIX got separated out to USL, Research UNIX would continue to exist inside of Bell Labs. Research UNIX V8, V9 and V10 were not quite released by Alcatel-Lucent USA Inc. and Nokia in 2017.
However, this is merely a notice that the companies involved will not assert their copyrights only with respect to any non-commercial usage of the code. It is still not possible, over 30 years later, to freely use the V8 code.

• Conclusion
  In the U.S., Ancient UNIX is freely available. People located everywhere else, however, are unable to legally obtain UNIX code for any of the systems mentioned above. The exception being BSD/OS, assuming a purchase of a legitimate copy of the source code CD. This is deeply unsatisfying and I implore all involved companies to consider open sourcing (preferably under a BSD-style license) their code older than a decade, if nothing else, then at least for the sake of historical purposes. I would like to encourage everybody reading this to consider reaching out to Micro Focus and Wind River Systems about System V and BSD/OS, respectively. Perhaps the masses can change their minds.

A small note about patents: Some technologies used in newer iterations of the UNIX system (in particular the System V derivatives) may be encumbered with software patents. An open source license will not help against patent infringement claims. However, the patents on anything used in the historical operating systems will certainly have expired by now. In addition, European readers can ignore this entirely – software patents just aren’t a thing.

###OpenBGPD – Adding Diversity to the Route Server Landscape

• Introduction

As of last year, there was effectively only a single solution in the Route Server vendor market: the BIRD Internet routing daemon. NIC.CZ (the organisation developing BIRD) has done fantastic work on maintaining their BGP-4 implementation, however, it’s not healthy to have virtually every Internet Exchange Point (IXP) in the RIPE NCC service region depend on a single open source project. The current situation can be compared to the state of the DNS root nameservers back in 2002 – their dependence on the BIND nameserver daemon and the resulting development of NSD as an alternative by NLnet, in cooperation with the RIPE NCC.
OpenBGPD used to be one of the most popular Route Server implementations until the early 2010s. OpenBGPD’s main problem was that its performance couldn’t keep up with the Internet’s growth, so it lost market share. An analysis by Job Snijders suggested that a modernised OpenBGPD distribution would be a most viable option to regain diversity on the Route Server level.

• Missing features in OpenBGPD

The following main missing features were identified in OpenBGPD:

• Performance

In previous versions of OpenBGPD, the filtering performance didn’t allow proper filtering of all EBGP sessions. Current best practice at IXP Route Servers is to carefully evaluate and validate of all routes learned from EBGP peers. The OpenBGPD ruleset required to do correct filtering (in many deployment scenarios) was simply too lengthy – and negatively impacted service performance during configuration reloads. While filtering performance is the biggest bottleneck, general improvements to the Routing Information Base were also made to improve scalability. IXP Route Servers with a few hundred peering sessions are commonplace and adding new sessions shouldn’t impact the Route Servers’ service to other peers. We found that performance was the most pressing issue that needed to be tackled.

• Lack of RPKI Origin Validation

As we’ve seen, Internet operators are moving to adopt RPKI based BGP Origin Validation. While it was theoretically possible to emulate RFC 6811-style Origin Validation in previous versions of OpenBGPD, the required configuration wasn’t optimised for performance and wasn’t user friendly. We believe that BGP Origin Validation should be as easy as possible – this requires BGP-4 vendors to implement native, optimised routines for Origin Validation. Of course, enabling Origin Validation shouldn’t have an impact on performance either when processing BGP updates or when updating the Route Origin Authorisation (ROA) table itself.

• Portability

OpenBGPD is an integral part of OpenBSD, but IXPs may prefer to run their services infrastructure on an operating system of their choice. Making sure that there’s a portable OpenBGPD version which follows the OpenBSD project release cycle will give IXPs this option.

• Development steps

By addressing the issues mentioned above, we could bring back OpenBGPD as a viable Route Server implementation.
Since I was one of the core OpenBGPD developers, I was asked if I wanted to pick up this project again. Thanks to the funding from the RIPE NCC Project Fund, this was possible. Starting in June 2018, I worked full time on this important community project. Over the last few months, many of the problems are already addressed and are now part of the OpenBSD 6.4 release. So far, 154 commits were made to OpenBGPD during the 6.4 development cycle – around 8% of all commits ever to OpenBGPD! This shows that due to funding and dedicated resources, a lot of work could be pushed into the latest release of OpenBGPD.

• OpenBGPD 6.4

The OpenBGPD version, as part of OpenBSD 6.4 release, demonstrates great progress. Even though there have been many changes to the core of OpenBGPD, the released version is as solid and reliable as previous releases and the many bug fixes and improvements make this the best OpenBGPD release so far. The changes in the filter language allow users to write more efficient rulesets while the introduction of RPKI origination validation fixes an important missing feature. For IXPs, OpenBGPD now is an alternative again. There are still open issues, but the gap is closing!

• Feature highlights

The following changes should be highlighted:

• Introduction of background soft-reconfiguration on config reload. Running the soft-reconfiguration task in the background allows for new updates and withdraws to be processed at the same time. This improves convergence time – one of the key metrics for Route Servers.
• BGP Origin Validation when a roa-set is configured Every EBGP route announcement is validated against the locally configured VRP table entries. Depending on the validation process’s outcome, the validation state is set to valid, invalid or not found. The filter language has been extended to allow checking for the origin validation state, and thanks to this, it is possible to deny invalid prefixes or regard valid prefixes different to the ones that aren’t found. The roa-set table is read from the configuration file and updated during configuration reloads. On production systems reloading the roa-set and applying it to all prefixes is done in a couple of seconds.
• Fast prefix-set lookups In OpenBSD 6.3 prefix-sets got introduced in OpenBGPD. A prefix-set combines many prefix lookups into a single filter rule. The original implementation wasn’t optimised but now a fast trie lookup is used. Thanks to this, large IRR DB prefix tables can now be implemented efficiently.
• Introduction of as-sets Similar to prefix-sets, as-sets help group many AS numbers into a single lookup. Thanks to this, large IRR DB origin AS tables can be implemented efficiently.
  Introduction of origin-sets
• Looking at the configurations of Route Servers doing full filtering, it was noticed that a common lookup was binding a prefix to an origin AS – similar to how a roa-set is used for RPKI. These origin-set tables are used to extend the IRR prefix lookup and generated from alternative sources.
• Improving third party tools

Users can only benefit from the changes introduced in OpenBGPD 6.4 when the surrounding 3rd party tools are adjusted accordingly. Two opensource projects such as bgpq3 and arouteserver are frequently used by network operators and IXPs to generate BGP configurations. Thanks to our contributions to those projects, we were able to get them ready for all the new features in OpenBGPD.

• bgpq3 was extended to create as-set and prefix-set tables based on IRR DB entries. This is replacing the old way of doing the same with a large amount of filter rules. Thanks to the quick response from the bgpq3 maintainer, it was possible to ship OpenBSD 6.4 with a bgpq3 package that includes all the new features.
• arouteserver was adjusted to implement RPKI roa-set, as-set, prefix-set, and origin-set to generate a much better-performing configurations for the 6.4 version. With the v0.20.0 release of arouteserver, IXPs are able to generate an OpenBGPD configuration which is a ton faster but also implements the new functionalities. Looking at YYCIX (the resident IXP in Calgary, Canada) the ruleset generated by arouteserver was reduced from 370,000 rules to well under 6,000 rules. This resulted in the initial convergence time dropping from over 1 hour to less than 2 minutes, and subsequent configuration reloads are hitless and no longer noticeable.
• What still needs to be done

A sizeable chunk of work still left on the table is the rework of the RIB data structures in OpenBGPD – these haven’t been changed since the initial design of OpenBGPD in 2003. There’s currently ongoing work (in small steps, to avoid jeopardising the stability of OpenBGPD) to modernise these data-structures. The goal is to provide better decoupling of the filter step from storing RIB database changes, to pave the way to multi-threaded operations at a later point.

• Looking forward
• Job Snijders oversaw this year’s fundraising and project management, he adds:

It’s been incredibly productive to create an environment where a core developer is allowed to work full time on the OpenBGPD code base. However, it’s important to note there still is room for a number of new features to help improve its operational capabilities (such as BMP, RFC 7313, ADD_PATH, etc). It’d be beneficial to the Internet community at large if we can extend Claudio Jeker’s involvement for another year. Open source software doesn’t grow on trees! Strategic investments are the only way to keep OpenBGPD’s roadmap aligned with Internet growth and operator requirements.

##Beastie Bits

• DragonFly – git: annotated tag v5.5.0 created
• Torchlight 2 on NetBSD
• Older, but still good USENIX Login Article on Capsicum
• The Super Capsicumizer 9000
• Dedicated and Virtual Server PXE provisioning tool
• Cirrus CI have announced FreeBSD support
• NetBSD PineBook Gameplay
• BSDCan 2019 CfP is out
• Allan’s first ZFS array, Zulu, turned 7 years old on Nov 29th

##Feedback/Questions

• Malcom – Installing Drivers in Development
• Samir – Introduction to ZFS
• Newnix – Drive Failures

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

The post OpenBSD in Stereo | BSD Now 275 first appeared on Jupiter Broadcasting.

##Headlines
###OpenBSD/NetBSD on FreeBSD using grub2-bhyve

When I was writing a blog post about the process title, I needed a couple of virtual machines with OpenBSD, NetBSD, and Ubuntu. Before that day I mainly used FreeBSD and Windows with bhyve. I spent some time trying to set up an OpenBSD using bhyve and UEFI as described here. I had numerous problems trying to use it, and this was the day I discovered the grub2-bhyve tool, and I love it!
The grub2-bhyve allows you to load a kernel using GRUB bootloader. GRUB supports most of the operating systems with a standard configuration, so exactly the same method can be used to install NetBSD or Ubuntu. First, let’s install grub2-bhyve on our FreeBSD box:

# pkg install grub2-bhyve

To run grub2-bhyve we need to provide at least the name of the VM. In bhyve, if the memsize is not specified the default VM is created with 256MB of the memory.

# grub-bhyve test
GNU GRUB version 2.00
Minimal BASH-like line editing is supported. For the first word, TAB lists possible command
completions. Anywhere else TAB lists possible device or file completions.


grub>

After running grub-bhyve command we will enter the GRUB loader. If we type the ls command, we will see all the available devices. In the case of the grub2-bhyve there is one additional device called “(host)” that is always available and allows the host filesystem to be accessed. We can list files under that device.

grub> ls
(host)
grub> ls (host)/
libexec/ bin/ usr/ bhyve/ compat/ tank/ etc/ boot/ net/ entropy proc/ lib/ root/ sys/ mnt/ rescue/ tmp/ home/ sbin/ media/ jail/ COPYRIGHT var/ dev/
grub>

To exit console simply type ‘reboot’. I would like to install my new operating system under a ZVOL ztank/bhyve/post. On another terminal, we create:

# zfs create -V 10G ztank/bhyve/post

If you don’t use ZFS for some crazy reason you can also create a raw blob using the truncate(1) command.

# truncate -s 10G post.img

I recommend installing an operating system from the disk image (installXX.fs for OpenBSD and NetBSD-X.X-amd64-install.img for NetBSD). Now we need to create a device map for a GRUB.

cat > /tmp/post.map << EOF
(hd0) /directory/to/disk/image
(hd1) /dev/zvol/ztank/bhyve/post
EOF

The mapping files describe the names for files in the GRUB. In our case under hd0 we will have an installation image and in hd1 we will have our ZVOL/blob. You can also try to use an ISO image then instead of using hd0 device name use a cd0. When we will run the grub-bhyve command we will see two additional devices.

# grub-bhyve -m /tmp/post.map post
grub> ls
(hd0) (hd0,msdos4) (hd0,msdos1) (hd0,openbsd9) (hd0,openbsd1) (hd1) (host)

The hd0 (in this example OpenBSD image) contains multiple partitions. We can check what is on it.

grub> ls (hd0,msdos4)/
boot bsd 6.4/ etc/

And this is the partition that contains a kernel. Now we can set a root device, load an OpenBSD kernel and boot:

grub> set root=(hd0,msdos4)
grub> kopenbsd -h com0 -r sd0a /bsd
grub> boot

After that, we can run bhyve virtual machine. In my case it is:

# bhyve -c 1 -w -u -H \
-s 0,amd_hostbridge \
-s 3,ahci-hd,/directory/to/disk/image \
-s 4,ahci-hd,/dev/zvol/ztank/bhyve/post \
-s 31,lpc -l com1,stdio \
post

Unfortunately explaining the whole bhyve(8) command line is beyond this article. After installing the operating system remove hd0 from the mapping file and the image from the bhyve(8) command. If you don’t want to type all those GRUB commands, you can simply redirect them to the standard input.

cat << EOF | grub-bhyve -m /tmp/post.map -M 512 post
set root=(hd0,4)
kopenbsd -h com0 -r sd0a /bsd
boot
EOF

###My FreeBSD Story

My first devices/computers/consoles (not at the same time) that I remember were Atari 2600 and Pegasus console which was hardware clone of the Nintendo NES.
Back then I did not even knew that it was Atari 2600 as I referred to it as Video Computer System … and I did not even knew any english by then. It took me about two decades to get to know (by accident) that this Video Computer System was Atari 2600
Then I got AMIGA 600 computer (or should I say my parents bought it for me) which served both for playing computer games and also other activities for the first time. AMIGA is the computer that had the greatest influence on me, as it was the first time I studied the books about Amiga Workbench operating system and learned commands from Amiga Shell terminal. I loved the idea of Ram Disk icon/directory on the desktop that allowed me to transparently put any things in system memory. I still miss that concept on today’s desktop systems … and I still remember how dismal I was when I watched Amiga Deathbed Vigil movie.
At the end of 1998 I got my first PC that of course came with Windows and that computer served both as gaming machine and as well as typical tool. One time I dig into the internals with Windows Registry (which left me disgusted by its concepts and implementation) and its limited command line interface provided by CMD.EXE executable. I remember that the heart of this box was not the CPU or the motherboard but the graphics accelerator – the legendary 3Dfx Voodoo card. This company (3Dfx) – their attitude and philosophy – also left solid fingerprint on my way. Like AMIGA did.
After ‘migration’ from AMIGA to PC it never again ‘felt right’. The games were cool but the Windows system was horrible. Time has passed and different Windows versions and hardware modifications took place. Windows XP felt really heavy at that time, not to mention Windows 2000 for example with even bigger hardware requirements. I also do not understand all the hate about Windows ME. It crashed with the same frequency as Windows 98 or later Windows 98 Second Edition but maybe my hardware was different ??
I do not have any ‘mine’ screenshots from that period as I lost all my 40 GB (huge then) drive of data when I moved/resized the partition with Partition Magic to get some more space from the less filled C: drive. That day I learned hard that “there are people who do backups and people who will do backups”. I never lost data again as I had multiple copies of my data, but the same as Netheril fall the lost data was was gone forever.
I always followed various alternatives which led me to try Linux in 2003, after reading about various distributions philosophies I decided to run Slackware Linux with KDE 3. My buddy used Aurox Linux by then (one of the few Linux distributions from Poland) and encouraged me to do the same – especially in the context of fixing possible problems as he already knew it and also as he recently dumped Windows system. But Slackware sounded like a better idea so I took that path instead. At first I dual booted between Windows XP and Slackware Linux cause I had everything worked out on the Windows world while I often felt helpless in the Linux world, so I would reboot into Windows to play some games or find a solution for Linux problem if that was required. I remember how strange the concept of dual clipboards (PRIMARY and SECONDARY) was for me by then. I was amazed why ‘so much better’ system as Linux (at least marketed that way) needs a system tray program to literally manage the clipboard. On Windows it was obvious, you do [CTRL]+[C] to copy and [CTRL]+[V] to paste things, but on Linux there (no I know its X11 feature) there were two clipboards that were synchronized by this little system tray program from KDE 3. It was also unthinkable for me that I will ‘lost’ contents of last/recent [CTRL]+[C] operation if I close the application from which the copy was made. I settled down a little on Slackware but not for long. I really did not liked manual dependency management for packages for example. Also KDE 3 was really ugly and despite trying all possible options I was not able to tweak it into something nice looking.
After half a year on Slackware I checked the Linux distributions again and decided to try Gentoo Linux. I definitely agree with the image below which visualizes Gentoo Linux experience, especially when You install it for he first time ??
Of course I went with the most hardcore version with self building Stage 1 (compiler and toolchain) which was horrible idea at that time because compilation on slow single core machine took forever … but after many hours I got Gentoo installed. I now have to decide which desktop environment to use. I have read a lot of good news about Fluxbox at that time so this is what I tried. It was very weird experience (to create everything in GUI from scratch) but very pleasant one. That recalled me the times of AMIGA … but Linux came in the way too much often. The more I dig into Gentoo Linux the more I read that lots of Gentoo features are based on FreeBSD solutions. Gentoo Portage is a clone of FreeBSD Ports. That ‘central’ /etc/rc.conf system configuration file concept was taken from FreeBSD as well. So I started to gather information about FreeBSD. The (then) FreeBSD website or FreeBSD Ports site (still) felt little outdated to say the least but that did not discouraged me.
Somewhere in 2005 I installed FreeBSD 5.4 on my computer. The beginnings were hard, like the earlier step with Gentoo but similarly like Gentoo the FreeBSD project came with a lot of great documentation. While Gentoo documentation is concentrated within various Gentoo Wiki sites the FreeBSD project comes with ‘official’ documentation in the form of Handbook and FAQ. I remember my first questions at the now nonexistent BSDForums.org site – for example one of the first ones – how to scroll the terminal output in the plain console. I now know that I had to push Scroll Lock button but it was something totally new for me.
Why FreeBSD and not OpenBSD or NetBSD? Probably because Gentoo based most their concepts on the FreeBSD solutions, so that led me to FreeBSD instead of the other BSD operating systems. Currently I still use FreeBSD but I keep an steady eye on the OpenBSD, HardenedBSD and DragonFly BSD solutions and improvements.
As the migration path from Linux to FreeBSD is a lot easier – all configuration files from /home can be just copied – the migration was quite fast easy. I again had the Fluxbox configuration which I used on the Gentoo. Now – on FreeBSD – it started to fell even more like AMIGA times. Everything is/has been well thought and had its place and reason. The documentation was good and the FreeBSD Community was second to none.
After 15 years of using various Windows, UNIX (macOS/AIX/HP-UX/Solaris/OpenSolaris/Illumos/FreeBSD/OpenBSD/NetBSD) and UNIX-like (Linux) systems I always come to conclusion that FreeBSD is the system that sucks least. And sucks least with each release and one day I will write why FreeBSD is such great operating system … if I already haven’t

##News Roundup
###OpenBSD on the Desktop: some thoughts

I’ve been using OpenBSD on my ThinkPad X230 for some weeks now, and the experience has been peculiar in some ways.
The OS itself in my opinion is not ready for widespread desktop usage, and the development team is not trying to push it in the throat of anybody who wants a Windows or macOS alternative. You need to understand a little bit of how *NIX systems work, because you’ll use CLI more than UI. That’s not necessarily bad, and I’m sure I learned a trick or two that could translate easily to Linux or macOS. Their development process is purely based on developers that love to contribute and hack around, just because it’s fun. Even the mailing list is a cool place to hang on! Code correctness and security are a must, nothing gets committed if it doesn’t get reviewed thoroughly first – nowadays the first two properties should be enforced in every major operating system.
I like the idea of a platform that continually evolves. pledge(2) and unveil(2) are the proof that with a little effort, you can secure existing software better than ever.
I like the “sensible defaults” approach, having an OS ready to be used – UI included if you selected it during the setup process – is great.
Just install a browser and you’re ready to go.
Manual pages on OpenBSD are real manuals, not an extension of the “–help” command found in most CLI softwares. They help you understand inner workings of the operating system, no internet connection needed. There are some trade-offs, too.
Performance is not first-class, mostly because of all the security mitigations and checks done at runtime.
I write Go code in neovim, and sometimes you can feel a slight slowdown when you’re compiling and editing multiple files at the same time, but usually I can’t notice any meaningful difference. Browsers are a different matter though, you can definitely feel something differs from the experience you can have on mainstream operating systems. But again, trade-offs.
To use OpenBSD on the desktop you must be ready to sacrifice some of the goodies of mainstream OSes, but if you’re searching for a zen place to do your computing stuff, it’s the best you can get right now.

###The history of file type information being available in Unix directories

The two things that Unix directory entries absolutely have to have are the name of the directory entry and its ‘inode’, by which we generically mean some stable kernel identifier for the file that will persist if it gets renamed, linked to other directories, and so on. Unsurprisingly, directory entries have had these since the days when you read the raw bytes of directories with read(), and for a long time that was all they had; if you wanted more than the name and the inode number, you had to stat() the file, not just read the directory. Then, well, I’ll quote myself from an old entry on a find optimization:
[…], Unix filesystem developers realized that it was very common for programs reading directories to need to know a bit more about directory entries than just their names, especially their file types (find is the obvious case, but also consider things like ‘ls -F’). Given that the type of an active inode never changes, it’s possible to embed this information straight in the directory entry and then return this to user level, and that’s what developers did; on some systems, readdir(3) will now return directory entries with an additional d_type field that has the directory entry’s type.
On Twitter, I recently grumbled about Illumos not having this d_type field. The ensuing conversation wound up with me curious about exactly where d_type came from and how far back it went. The answer turns out to be a bit surprising due to there being two sides of d_type.
On the kernel side, d_type appears to have shown up in 4.4 BSD. The 4.4 BSD /usr/src/sys/dirent.h has a struct dirent that has a d_type field, but the field isn’t documented in either the comments in the file or in the getdirentries(2) manpage; both of those admit only to the traditional BSD dirent fields. This 4.4 BSD d_type was carried through to things that inherited from 4.4 BSD (Lite), specifically FreeBSD, but it continued to be undocumented for at least a while.
(In FreeBSD, the most convenient history I can find is here, and the d_type field is present in sys/dirent.h as far back as FreeBSD 2.0, which seems to be as far as the repo goes for releases.)
Documentation for d_type appeared in the getdirentries(2) manpage in FreeBSD 2.2.0, where the manpage itself claims to have been updated on May 3rd 1995 (cf). In FreeBSD, this appears to have been part of merging 4.4 BSD ‘Lite2’, which seems to have been done in 1997. I stumbled over a repo of UCB BSD commit history, and in it the documentation appears in this May 3rd 1995 change, which at least has the same date. It appears that FreeBSD 2.2.0 was released some time in 1997, which is when this would have appeared in an official release.
In Linux, it seems that a dirent structure with a d_type member appeared only just before 2.4.0, which was released at the start of 2001. Linux took this long because the d_type field only appeared in the 64-bit ‘large file support’ version of the dirent structure, and so was only return by the new 64-bit getdents64() system call. This would have been a few years after FreeBSD officially documented d_type, and probably many years after it was actually available if you peeked at the structure definition.
As far as I can tell, d_type is present on Linux, FreeBSD, OpenBSD, NetBSD, Dragonfly BSD, and Darwin (aka MacOS or OS X). It’s not present on Solaris and thus Illumos. As far as other commercial Unixes go, you’re on your own; all the links to manpages for things like AIX from my old entry on the remaining Unixes appear to have rotted away.
Sidebar: The filesystem also matters on modern Unixes
Even if your Unix supports d_type in directory entries, it doesn’t mean that it’s supported by the filesystem of any specific directory. As far as I know, every Unix with d_type support has support for it in their normal local filesystems, but it’s not guaranteed to be in all filesystems, especially non-Unix ones like FAT32. Your code should always be prepared to deal with a file type of DT_UNKNOWN.
It’s also possible to have things the other way around, where you have a filesystem with support for file type information in directories that’s on a Unix that doesn’t support it. There are a number of plausible reasons for this to happen, but they’re either obvious or beyond the scope of this entry.

###Multiboot Pinebook KDE neon

Recently a KDE neon image for the Pinebook was announced. There is a new image, with a handful of fixes, which the KDE Plasma team has been working on over the past week and a half.
Here’s a picture of my Pinebook running KDE neon — watching Panic! At the Disco’s High Hopes — sitting in front of my monitor that’s hooked up to one of my openSUSE systems. There are still some errata, and watching video sucks up battery, but for hacking on documentation from my hammock in the garden, or doing IRC meetings it’s a really nice machine.
But one of the neat things about running KDE neon off of an SD card on the Pinebook is that it’s portable — that SD card can move around. So let’s talk about multiboot in the sense of “booting the same OS storage medium in different hardware units” rather than “booting different OS from a medium in a single hardware unit”. On these little ARM boards, u-boot does all the heavy lifting early in the boot process. So to re-use the KDE neon Pinebook image on another ARM board, the u-boot blocks need to be replaced.
I have the u-boot from a Pine64 image (I forget what) lying around, 1015 blocks of 1024 bytes, which I can dd over the u-boot blocks on the SD card, dd bs=1k conv=notrunc,sync if=uboot.img of=/dev/da0 seek=8, and then the same SD card, with the filesystem and data from the Pinebook, will boot on the Pine64 board. Of course, to move the SD card back again, I need to restore the Pinebook u-boot blocks.
Here’s a picture of my Pineboard (the base is a piece of the garden fence, it’s Douglas pine, with 4mm threaded rods acting as the corner posts for my Pine64 mini-rack), with power and network and a serial console attached, along with the serial console output of the same.
The nice thing here is that the same software stack runs on the Pine64 but then has a wired network — which in turn means that if I switch on the other boards in that mini-rack, I’ve got a distcc-capable cluster for fast development, and vast NFS storage (served from ZFS on my FreeBSD machines) for source. I can develop in a high(er) powered environment, and then swap the card around into the Pinebook for testing-on-the-go.
So to sum up: you can multiboot the KDE neon Pinebook image on other Pine64 hardware (i.e. the Pine64 board). To do so, you need to swap around u-boot blocks. The blocks can be picked out of an image built for each board, and then a particular image (e.g. the latest KDE neon Pinebook) can be run on either board.

##Beastie Bits

• Unexpected benefit with Ryzen – reducing power for build server
• Happy #CIDRDay!
• Absolute FreeBSD 3e ship date
• MWL FreeBSD talk @ October 9th 2018 – MUG Meeting
• MeetBSD Oct 19-20
• October’s London *BSD meetup – 9th Oct 2018
• NRW BUG Meeting at Trivago Oct. 9
• Lars Wittebrood blogs about his visit to EuroBSDCon 2018
• EuroBSDcon 2018 OpenBSD slides available
• EuroBSDCon conference site has most slides as well

##Feedback/Questions

• Brad – Unmounted ZFS sends
• Niclas – Report from a Meetup
• Ghislain – Bhyve not used?
• Shane – zpool history and snapshots

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

The post File Type History | BSD Now 266 first appeared on Jupiter Broadcasting.

##Headlines
###Silent Fanless FreeBSD Desktop/Server

Today I will write about silent fanless FreeBSD desktop or server computer … or NAS … or you name it, it can have multiple purposes. It also very low power solution, which also means that it will not overheat. Silent means no fans at all, even for the PSU. The format of the system should also be brought to minimum, so Mini-ITX seems best solution here.

I have chosen Intel based solutions as they are very low power (6-10W), if you prefer AMD (as I often do) the closest solution in comparable price and power is Biostar A68N-2100 motherboard with AMD E1-2100 CPU and 9W power. Of course AMD has even more low power SoC solutions but finding the Mini-ITX motherboard with decent price is not an easy task. For comparison Intel has lots of such solutions below 6W whose can be nicely filtered on the ark.intel.com page. Pity that AMD does not provide such filtration for their products. I also chosen AES instructions as storage encryption (GELI on FreeBSD) today seems as obvious as HTTPS for the web pages.

• Here is how the system look powered up and working

This motherboard uses Intel J3355 SoC which uses 10W and has AES instructions. It has two cores at your disposal but it also supports VT-x and EPT extensions so you can even run Bhyve on it.

• Components

Now, an example system would look like that one below, here are the components with their prices.

• $49 CPU/Motherboard ASRock J3355B-ITX Mini-ITX
• $14 RAM Crucial 4 GB DDR3L 1.35V (low power)
• $17 PSU 12V 160W Pico (internal)
• $11 PSU 12V 96W FSP (external)
• $5 USB 2.0 Drive 16 GB ADATA
• $4 USB Wireless 802.11n
• $100 TOTAL

The PSU 12V 160W Pico (internal) and PSU 12V 96W FSP can be purchased on aliexpress.com or ebay.com for example, at least I got them there. Here is the 12V 160W Pico (internal) PSU and its optional additional cables to power the optional HDDs. If course its one SATA power and one MOLEX power so additional MOLEX-SATA power adapter for about 1$ would be needed. Here is the 12V 96W FSP (external) PSU without the power cord.

This gives as total silent fanless system price of about $120. Its about ONE TENTH OF THE COST of the cheapest FreeNAS hardware solution available – the FreeNAS Mini (Diskless) costs $1156 also without disks.

You can put plain FreeBSD on top of it or Solaris/Illumos distribution OmniOSce which is server oriented. You can use prebuilt NAS solution based on FreeBSD like FreeNAS, NAS4Free, ZFSguru or even Solaris/Illumos based storage with napp-it appliance.

###An annotated look at a NetBSD Pinebook’s startup

• Pinebook is an affordable 64-bit ARM notebook. Today we’re going to take a look at the kernel output at startup and talk about what hardware support is available on NetBSD.
• Photo
• Pinebook comes with 2GB RAM standard. A small amount of this is reserved by the kernel and framebuffer.
• NetBSD uses flattened device-tree (FDT) to enumerate devices on all Allwinner based SoCs. On a running system, you can inspect the device tree using the ofctl(8) utility:
• Pinebook’s Allwinner A64 processor is based on the ARM Cortex-A53. It is designed to run at frequencies up to 1.2GHz.
• The A64 is a quad core design. NetBSD’s aarch64 pmap does not yet support SMP, so three cores are disabled for now.
• The interrupt controller is a standard ARM GIC-400 design.
• Clock drivers for managing PLLs, module clock dividers, clock gating, software resets, etc. Information about the clock tree is exported in the hw.clk sysctl namespace (root access required to read these values).

# sysctl hw.clk.sun50ia64ccu0.mmc2
hw.clk.sun50ia64ccu0.mmc2.rate = 200000000
hw.clk.sun50ia64ccu0.mmc2.parent = pll_periph0_2x
hw.clk.sun50ia64ccu0.mmc2.parent_domain = sun50ia64ccu0

Digital Ocean
https://do.co/bsdnow

###BSDCan 2018 Trip Report: Mark Johnston

BSDCan is a highlight of my summers: the ability to have face-to-face conversations with fellow developers and contributors is invaluable and always helps refresh my enthusiasm for FreeBSD. While in a perfect world we would all be able to communicate effectively over the Internet, it’s often noted that locking a group of developers together in a room can be a very efficient way to make progress on projects that otherwise get strung out over time, and to me this is one of the principal functions of BSD conferences. In my case I was able to fix some kgdb bugs that had been hindering me for months; get some opinions on the design of a feature I’ve been working on for FreeBSD 12.0; hear about some ongoing usage of code that I’ve worked on; and do some pair-debugging of an issue that has been affecting another developer.
As is tradition, on Tuesday night I dropped off my things at the university residence where I was staying, and headed straight to the Royal Oak. This year it didn’t seem quite as packed with BSD developers, but I did meet several long-time colleagues and get a chance to catch up. In particular, I chatted with Justin Hibbits and got to hear about the bring-up of FreeBSD on POWER9, a new CPU family released by IBM. Justin was able to acquire a workstation based upon this CPU, which is a great motivator for getting FreeBSD into shape on that platform. POWER9 also has some promise in the server market, so it’s important for FreeBSD to be a viable OS choice there.
Wednesday morning saw the beginning of the two-day FreeBSD developer summit, which precedes the conference proper. Gordon Tetlow led the summit and did an excellent job organizing things and keeping to the schedule. The first presentation was by Deb Goodkin of the FreeBSD Foundation, who gave an overview of the Foundation’s role and activities. After Deb’s presentation, present members of the FreeBSD core team discussed the work they had done over the past two years, as well as open tasks that would be handed over to the new core team upon completion of the ongoing election. Finally, Marius Strobl rounded off the day’s presentations by discussing the state and responsibilities of FreeBSD’s release engineering team.
One side discussion of interest to me was around the notion of tightening integration with our Bugzilla instance; at moment we do not have any good means to mark a given bug as blocking a release, making it easy for bugs to slip into releases and thus lowering our overall quality. With FreeBSD 12.0 upon us, I plan to help with the triage and fixes for known regressions before the release process begins.
After a break, the rest of the morning was devoted to plans for features in upcoming FreeBSD releases. This is one of my favorite discussion topics and typically takes the form of have/need/want, where developers collectively list features that they’ve developed and intend to upstream (have), features that they are missing (need), and nice-to-have features (want). This year, instead of the usual format, we listed features that are intended to ship in FreeBSD 12.0. The compiled list ended up being quite ambitious given how close we are to the beginning of the release cycle, but many individual developers (including myself) have signed up to deliver work. I’m hopeful that most, if not all of it, will make it into the release.
After lunch, I attended a discussion led by Matt Ahrens and Alexander Motin on OpenZFS. Of particular interest to me were some observations made regarding the relative quantity and quality of contributions made by different “camps” of OpenZFS users (illumos, FreeBSD and ZoL), and their respective track records of upstreaming enhancements to the OpenZFS project. In part due to the high pace of changes in ZoL, the definition of “upstream” for ZFS has become murky, and of late ZFS changes have been ported directly from ZoL. Alexander discussed some known problems with ZFS on FreeBSD that have been discovered through performance testing. While I’m not familiar with ZFS internals, Alexander noted that ZFS’ write path has poor SMP scalability on FreeBSD owing to some limitations in a certain kernel API called taskqueue(9). I would like to explore this problem further and perhaps integrate a relatively new alternative interface which should perform better.
Friday and Saturday were, of course, taken up by BSDCan talks. Friday’s keynote was by Benno Rice, who provided some history of UNIX boot systems as a precursor to some discussion of systemd and the difficulties presented by a user and developer community that actively resist change. The rest of the morning was consumed by talks and passed by quickly. First was Colin Percival’s detailed examination of where the FreeBSD kernel spends time during boot, together with an overview of some infrastructure he added to track boot times. He also provided a list of improvements that have been made since he started taking measurements, and some areas we can further improve. Colin’s existing work in this area has already brought about substantial reductions in boot time; amusingly, one of the remaining large delays comes from the keyboard driver, which contains a workaround for old PS/2 keyboards. While there seems to be general agreement that the workaround is probably no longer needed on most systems, the lingering uncertainty around this prevents us from removing the workaround. This is, sadly, a fairly typical example of an OS maintenance burden, and underscores the need to carefully document hardware bug workarounds. After this talk, I got to see some rather novel demonstrations of system tracing using dwatch, a new utility by Devin Teske, which aims to provide a user-friendly interface to DTrace. After lunch, I attended talks on netdump, a protocol for transmitting kernel dumps over a network after the system has panicked, and on a VPC implementation for FreeBSD. After the talks ended, I headed yet again to the hacker lounge and had some fruitful discussions on early microcode loading (one of my features for FreeBSD 12.0). These led me to reconsider some aspects of my approach and saved me a lot of time. Finally, I continued my debugging session from Wednesday with help from a couple of other developers.
Saturday’s talks included a very thorough account by Li-Wen Hsu of his work in organizing a BSD conference in Taipei last year. As one of the attendees, I had felt that the conference had gone quite smoothly and was taken aback by the number of details and pitfalls that Li-Wen enumerated during his talk. This was followed by an excellent talk by Baptiste Daroussin on the difficulties one encounters when deploying FreeBSD in new environments. Baptiste offered criticisms of a number of aspects of FreeBSD, some of which hit close to home as they involved portions of the system that I’ve worked on.
At the conclusion of the talks, we all gathered in the main lecture hall, where Dan led a traditional and quite lively auction for charity. I managed to snag a Pine64 board and will be getting FreeBSD installed on it the first chance I get. At the end of the auction, we all headed to ByWard for dinner, concluding yet another BSDCan.

• Thanks to Mark for sharing his experiences at this years BSDCan

##News Roundup
###Transparent network audio with mpd & sndiod

Landry Breuil (landry@ when wearing his developer hat) wrote in…

I've been a huge fan of MPD over the years to centralize my audio collection, and i've been using it with the http output to stream the music as a radio on the computer i'm currently using…

audio_output {
       type            "sndio"
       name            "Local speakers"
       mixer_type      "software"
}
audio_output {
       type            "httpd"
       name            "HTTP stream"
       mixer_type      "software"
       encoder         "vorbis"
       port            "8000"
       format          "44100:16:2"
}
this setup worked for years, allows me to stream my home radio to $work by tunnelling the port 8000 over ssh via LocalForward, but that still has some issues:

a distinct timing gap between the 'local output' (ie the speakers connected to the machine where MPD is running) and the 'http output' caused by the time it takes to reencode the stream, which is ugly when you walk through the house and have a 15s delay
sometimes mplayer as a client doesn't detect the pauses in the stream and needs to be restarted
i need to configure/start a client on each computer and point it at the sound server url (can do via gmpc shoutcast client plugin…)
it's not that elegant to reencode the stream, and it wastes cpu cycles
So the current scheme is:

mpd -> http output -> network -> mplayer -> sndiod on remote machine
|
-> sndio output -> sndiod on soundserver
Fiddling a little bit with mpd outputs and reading the sndio output driver, i remembered sndiod has native network support… and the mpd sndio output allows you to specify a device (it uses SIO_DEVANY by default).

So in the end, it's super easy to:

enable network support in sndio on the remote machine i want the audio to play by adding -L<local ip> to sndiod_flags (i have two audio devices, with an input coming from the webcam):
sndiod_flags="-L10.246.200.10 -f rsnd/0 -f rsnd/1"
open pf on port 11025 from the sound server ip:
pass in proto tcp from 10.246.200.1 to any port 11025
configure a new output in mpd:
audio_output {
       type            "sndio"
       name            "sndio on renton"
       device          "snd@10.246.200.10/0"
       mixer_type      "software"
}
and enable the new output in mpd:
$mpc enable 2
Output 1 (Local speakers) is disabled
Output 2 (sndio on renton) is enabled
Output 3 (HTTP stream) is disabled
Results in a big win: no gap anymore with the local speakers, no reencoding, no need to configure a client to play the stream, and i can still probably reproduce the same scheme over ssh from $work using a RemoteForward.

mpd -> sndio output 2 -> network -> sndiod on remote machine
|
-> sndio output 1 -> sndiod on soundserver
Thanks ratchov@ for sndiod :)

###MirBSD’s Korn Shell on Plan9 Jehanne

Let start by saying that I’m not really a C programmer.
My last public contribution to a POSIX C program was a little improvement to the Snort’s react module back in 2008.
So while I know the C language well enough, I do not know anything about the subtleness of the standard library and I have little experience with POSIX semantics.
This is not a big issue with Plan 9, since the C library and compiler are not standard anyway, but with Jehanne (a Plan 9 derivative of my own) I want to build a simple, loosely coupled, system that can actually run useful free software ported from UNIX.
So I ported RedHat’s newlib to Jehanne on top of a new system library I wrote, LibPOSIX, that provides the necessary emulations. I wrote several test, checking they run the same on Linux and Jehanne, and then I begun looking for a real-world, battle tested, application to port first.
I approached MirBSD’s Korn Shell for several reason:

• it is simple, powerful and well written
• it has been ported to several different operating systems
• it has few dependencies
• it’s the default shell in Android, so it’s really battle tested

I was very confident. I had read the POSIX standard after all! And I had a test suite!
I remember, I thought “Given newlib, how hard can it be?”
The porting begun on September 1, 2017. It was completed by tg on January 5, 2018. 125 nights later.
Turn out, my POSIX emulation was badly broken. Not just because of the usual bugs that any piece of C can have: I didn’t understood most POSIX semantics at all!

iXsystems

###Static site generator with rsync and lowdown on OpenBSD

• ssg is a tiny POSIX-compliant shell script with few dependencies:

• lowdown(1) to parse markdown,

• rsync(1) to copy temporary files, and

• entr(1) to watch file changes.

• It generates Markdown articles to a static website.

• It copies the current directory to a temporary on in /tmp skipping .* and _*, renders all Markdown articles to HTML, generates RSS feed based on links from index.html, extracts the first <h1> tag from every article to generate a sitemap and use it as a page title, then wraps articles with a single HTML template, copies everything from the temporary directory to $DOCS/

Why not Jekyll or “$X”?

• ssg is one hundred times smaller than Jekyll.

ssg and its dependencies are about 800KB combined. Compare that to 78MB of ruby with Jekyll and all the gems. So ssg can be installed in just few seconds on almost any Unix-like operating system.
Obviously, ssg is tailored for my needs, it has all features I need and only those I use.
Keeping ssg helps you to master your Unix-shell skills: awk, grep, sed, sh, cut, tr. As a web developer you work with lots of text: code and data. So you better master these wonderful tools.

• Performance

100 pps. On modern computers ssg generates a hundred pages per second. Half of a time for markdown rendering and another half for wrapping articles into the template. I heard good static site generators work—twice as fast—at 200 pps, so there’s lots of performance that can be gained.

###Why does FreeBSD have virtually no (0%) desktop market share?

• Because someone made a horrible design decision back in 1984.

In absolute fairness to those involved, it was an understandable decision, both from a research perspective, and from an economic perspective, although likely not, from a technology perspective.

• Why and what.

The decision was taken because the X Window System was intended to run on cheap hardware, and, at the time, that meant reduced functionality in the end-point device with the physical display attached to it.
At the same time, another force was acting to also limit X displays to display services only, rather than rolling in both window management and specific widget instances for common operational paradigms.
Mostly, common operational paradigms didn’t really exist for windowing systems because they also simply didn’t exist at the time, and no one really knew how people were going to use the things, and so researchers didn’t want to commit future research to a set of hard constraints.
So a decision was made: separate the display services from the application at the lowest level of graphics primitives currently in use at the time.

• The ramifications of this were pretty staggering.

First, it guaranteed that all higher level graphics would live on the host side of the X protocol, instead of on the display device side of the protocol.
Despite a good understanding of Moore’s law, and the fact that, since no X Terminals existed at the time as hardware, but were instead running as emulations on workstations that had sufficient capability, this put the higher level GUI object libraries — referred to as “widgets” — in host libraries linked into the applications.
Second, it guaranteed that display organization and management paradigms would also live on the host side of the protocol — assumed, in contradiction to the previous decision, to be running on the workstation.
But, presumably, at some point, as lightweight X Terminals became available, to migrate to a particular host computer managing compute resource login/access services.

• Between these early decisions reigned chaos.

Specifically, the consequences of these decisions have been with us ever since:
Look-and-feel are a consequence of the toolkit chosen by the application programmer, rather than a user decision which applies universally to all applications.
You could call this “lack of a theme”, and — although I personally despise the idea of customizing or “theming” desktops — this meant that one paradigm chosen by the user would not apply universally across all applications, no matter who had written them.
Window management style is a preference.
You could call this a more radical version of “theming” — which you will remember, I despise — but a consequence to this is that training is not universal across personnel using such systems, nor is it transferrable.
In other words, I can’t send someone to a class, and have them come back and use the computers in the office as a tool, with the computer itself — and the elements not specific to the application itself — disappearing into the background.
Both of these ultimately render an X-based system unsuitable for desktops.
I can’t pay once for training. Training that I do pay for does not easily and naturally translate between applications. Each new version may radically alter the desktop management paradigm into unrecognizability.

• Is there hope for the future?

Well, the Linux community has been working on something called Wayland, and it is very promising…
…In the same way X was “very promising” in 1984, because, unfortunately, they are making exactly the same mistakes X made in 1984, rather than correcting them, now that we have 20/20 hindsight, and know what a mature widget library should look like.
So Wayland is screwing up again.
But hey, it only took us, what, 25 years to get from X in 1987 to Wayland in in 2012.
Maybe if we try again in 2037, we can get to where Windows was in 1995.

##Beastie Bits

• New washing machine comes with 7 pages of open source licenses!
• BSD Jobs Site
• FreeBSD Foundation Update, May 2018
• FreeBSD Journal looking for book reviewers
• zedenv ZFS Boot Environment Manager

Tarsnap

##Feedback/Questions

• Wouter – Feedback
• Efraim – OS Suggestion
• kevr – Raspberry Pi2/FreeBSD/Router on a Stick
• Vanja – Interview Suggestion

• Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv

The post Silence of the Fans | BSD Now 253 first appeared on Jupiter Broadcasting.