Jupiter Broadcasting

The French CONnection | BSD Now 213

chris

RSS Feeds:

MP3 Feed | iTunes Feed | HD Vid Feed | HD Torrent Feed

Become a supporter on Patreon:

– Show Notes: –

Headlines

Recap of EuroBSDcon 2017 in Paris, France

The story of PR 219251

As I threatened to do in my previous post, I’m going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places
Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out.
It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic.
This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included.

We try to fetch a counter value here, but instead we dereference a bad pointer. There’s two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find.
In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists. The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host.
Great, we know what’s wrong, but now we need to work out how we can get into this state. At this point we’re going to have to learn something about how rules and states get cleaned up in pf. Don’t worry if you had no idea, because before this bug I didn’t either.
The states keep a pointer to the rule they match, so when rules are changed (or removed) we can’t just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire.
When rules are removed pf_unlink_rule() adds then to the V_pf_unlinked_rules list (more on that funny V_ prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that’s done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it). That can be a lot of work if we’ve got a lot of states, so pf_purge_thread() breaks that up into smaller chuncks, iterating only part of the state table on every run.
We iterate over all of our virtual pf instances (VNET_FOREACH()), check if it’s active (for FreeBSD-EN-17.08, where we’ve seen this code before) and then check the expired states with pf_purge_expired_states(). We start at state ‘idx’ and only process a certain number (determined by the PFTM_INTERVAL setting) states. The pf_purge_expired_states() function returns a new idx value to tell us how far we got.
So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that’s changed whenever we change the currently active vnet (say with CURVNET_SET() or CURVNET_RESTORE()). That’s tracked in the ‘curvnet’ variable. In other words: there are as many V_pf_vnet_active variables as there are vnets: number of vnet jails plus one (for the host system).
Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on.
That of course means that we think we’ve run through all of the states in a vnet while we really only checked some of them. So when pf_purge_unlinked_rules() runs it can end up free()ing rules that actually are still in use because pf_purge_thread() skipped over the state(s) that actually used the rule. The problem only happened if we reloaded rules in the host, because the active ruleset is never free()d, even if there are no states pointing to the rule.
That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, V_pf_purge_idx, and then the problem is gone. As is often the case, the solution to a fairly hard problem turns out to be really simple.

vBSDcon 2017: BSD at Work

The third biennial vBSDcon hosted by Verisign took place September 7th through 9th with the FreeBSD Developer Summit taking place the first day. vBSDcon and iXsystems’ MeetBSD event have been alternating between the East and West coasts of the U.S.A. and these two events play vital roles in reaching Washington, DC-area and Bay Area/Silicon Valley audiences. Where MeetBSD serves many BSD Vendors, vBSDcon attracts a unique government and security industry demographic that isn’t found anywhere else. Conference time and travel budgets are always limited and bringing these events to their attendees is a much-appreciated service provided by their hosts.
The vBSDcon FreeBSD DevSummit had a strong focus on OpenZFS, the build system and networking with the FreeBSD 12 wish list of features in mind. How to best incorporate the steady flow of new OpenZFS features into FreeBSD such as dataset-level encryption was of particular interest. This feature from a GNU/Linux-based storage vendor is tribute to the growth of the OpenZFS community which is vital in light of the recent “Death of Solaris and ZFS” at Oracle. There has never been more demand for OpenZFS on FreeBSD and the Oracle news further confirms our collective responsibility to meet that demand.
The official conference opened with my talk on “Isolated BSD Build Environments” in which I explained how the bhyve hypervisor can be used to effortlessly tour FreeBSD 5.0-onward and build specific source releases on demand to trace regressions to their offending commit. I was followed by a FreeNAS user who made the good point that FreeNAS is an exemplary “entry vector” into Unix and Enterprise Storage fundamentals, given that many of the vectors our generation had are gone. Where many of us discovered Unix and the Internet via console terminals at school or work, smart phones are only delivering the Internet without the Unix. With some irony, both iOS and Android are Unix-based yet offer few opportunities for their users to learn and leverage their Unix environments.
The next two talks were The History and Future of Core Dumps in FreeBSD by Sam Gwydir and Using pkgsrc for multi-platform deployments in heterogeneous environments by G. Clifford Williams. I strongly recommend that anyone wanting to speak at AsiaBSDCon read Sam’s accompanying paper on core dumps because I consider it the perfect AsiaBSDCon topic and his execution is excellent. Core dumps are one of those things you rarely think about until they are a DROP EVERYTHING! priority. G. Clifford’s talk was about what I consider a near-perfect BSD project: pkgsrc, the portable BSD package manager. I put it up there with OpenSSH and mandoc as projects that have provided significant value to other Open Source operating systems. G. Clifford’s real-world experiences are perfectly inline with vBSDcon’s goal to be more production-oriented than other BSDCons.
Of the other talks, any and all Dtrace talks are always appreciated and George Neville-Neil’s did not disappoint. He based it on his experiences with the Teach BSD project which is bringing FreeBSD-based computer science education to schools around the world. The security-related talks by John-Mark Gurney, Dean Freeman and Michael Shirk also represented vBSDcon’s consideration of the local community and made a convincing point that the BSDs should make concerted efforts to qualify for Common Criteria, FIPS, and other Government security requirements. While some security experts will scoff at these, they are critical to the adoption of BSD-based products by government agencies.
BSD Now hosts Allan Jude and Benedict Reuschling hosted an OpenZFS BoF and Ansible talk respectively and I hosted a bhyve hypervisor BoF. The Hallway Track and food at vBSDcon were excellent and both culminated with an after-dinner dramatic reading of Michael W. Lucas’ latest book that raised money for the FreeBSD Foundation. A great time was had by all and it was wonderful to see everyone!

News Roundup

FreeBSD 10.4-RC2 Available

OpenBSD changes of note 628

EuroBSDCon in two weeks. Be sure to attend early and often.

There’s a README for the syspatch build system if you want to run your own.

Sigh, here we go again.

Screencasting with OpenBSD

Any USB microphone should appear as a new audio device. Here is the dmesg for my mic by ART:

uaudio0 at uhub0 port 2 configuration 1 interface 0 "M-One USB" rev 1.10/0.01 addr 2
uaudio0: audio rev 1.00, 8 mixer controls
audio1 at uaudio0

audioctl can read off all of the specific characterisitcs of this device

$ audioctl -f /dev/audio1 | grep record
mode=play,record
record.rate=48000
record.channels=1
record.precision=16
record.bps=2
record.msb=1
record.encoding=slinear_le
record.pause=0
record.active=0
record.block_size=1960
record.bytes=0
record.errors=0

Now test the recording from the second audio device using aucat(1)

aucat -f rsnd/1 -o file.wav

If the device also has a headset audio can be played through the same device.

aucat -f rsnd/1 -i file.wav

The rate at which a framebuffer for your video card is a feature of the hardware and software your using, and it’s often very slow. x11vnc will print an estimate of the banwidth for the system your running.

x11vnc
...
09/05/2012 22:23:45 fb read rate: 7 MB/sec

This is about 4fps. We can do much better by using a virtual framebuffer. Here I’m setting up a new screen, setting the background color, starting cwm and an instance of xterm

Xvfb :1 -screen 0 720x540x16 &
DISPLAY=:1 xsetroot -solid steelblue &
DISPLAY=:1 cwm &
DISPLAY=:1 xterm +sb -fa Hermit -fs 14 &

Much better! Now we’re up around 20fps.

x11vnc -display :1 &
...
11/05/2012 18:04:07 fb read rate: 168 MB/sec

Make a connection to this virtual screen using raw encoding to eliminate time wasted on compression.

vncviewer localhost -encodings raw

A test recording with sound then looks like this

ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 ~/out.avi

Note: always stop the recording and playback using q, not Ctrl-C so that audio inputs are shut down properly.

Xephyr is perhaps the easiest way to run X with a shadow framebuffer. This solution also avoids reading from the video card’s RAM, so it’s reasonably fast.

Xephyr -ac -br -noreset -screen 800x600 :1 &
DISPLAY=:1 xsetroot -solid steelblue &
DISPLAY=:1 cwm &
DISPLAY=:1 xrdb -load ~/.Xdefaults &
DISPLAY=:1 xterm +sb -fa "Hermit" -fs 14 &

Capture works in exactally the same way. This command tries to maintain 12fps.

ffmpeg -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi
To capture keyboard and mouse input press Ctrl then Shift. This is very handy for using navigating a window manager in the nested X session.

I have sometimes found it helpful to launch applications and arrange them in a specific way. This will open up a web browser listing the current directory and position windows using xdotool

DISPLAY=:1 midori "file:///pwd" &
sleep 2
DISPLAY=:1 xdotool search --name "xterm" windowmove 0 0
DISPLAY=:1 xdotool search --class "midori" windowmove 400 0
DISPLAY=:1 xdotool search --class "midori" windowsize 400 576

This will position the window precisely so that it appears to be in a tmux window on the right.

If you find that the audio is way out of sync with the video, you can ajust the start using the -ss before the audio input to specify the number of seconds to delay. My final recording command line, that delays the audio by 0.5 seconds, writing 12fps

ffmpeg -ss 0.5 -f sndio -i snd/1 -y -f x11grab -r 12 -s 800x600 -i :1.0 -vcodec ffv1 -acodec copy ~/out.avi

If you’re trying to record a terminal session, tmux is able to share a session. In this way a recording of an X framebuffer can be taken without even using the screen. Start by creating the session.

tmux -2 -S /tmp/tmux0

Then on the remote side connect on the same socket

tmux -2 -S /tmp/tmux0 attach

Grabbing a screenshots on Xvfb server is easily accomplished with ImageMagick’s import command

DISPLAY=:1 import -window root screenshot.png

The first step is to ensure that the clip begins and ends where you’d like it to. The following will make a copy of the recording starting at time 00:00 and ending at 09:45

ffmpeg -i interactive-sql.avi \
-vcodec copy -acodec copy
-ss 00:00:00 -t 00:09:45
interactive-sql-trimmed.avi
mv interactive-sql-trimmed.avi interactive-sql.avi

Setting the gain correctly is very important with an analog mixer, but if you’re using a USB mic there may not be a gain option; simply record using it’s built-in settings and then adjust the levels afterwards using a utility such as normalize. First extact the audio as a raw PCM file and then run normalize

ffmpeg -i interactive-sql.avi -c:a copy -vn audio.wav
normalize audio.wav

Next merge the audio back in again

ffmpeg -i interactive-sql.avi -i audio.wav \
-map 0:0 -map 1:0 -c copy interactive-sql-normalized.avi

The final step is to compress the screencast for distribution. Encoding to VP8/Vorbis is easy:

ffmpeg -i interactive-sql-normalized.avi -c:v libvpx -b:v 1M
-c:a libvorbis -q:a 6 interactive-sql.webm

H.264/AAC is tricky. For most video players the color space needs to be set to yuv420p. The -movflags puts the index data at the beginning of the file to enable streaming/partial content requests over HTTP:

ffmpeg -y -i interactive-sql-normalized.avi -c:v libx264 \
-preset slow -crf 14 -pix_fmt yuv420p -movflags +faststart \
-c:a aac -q:a 6 interactive-sql.mp4

TrueOS @ Ohio Linuxfest ’17!

Beastie Bits

Feedback/Questions