codeblog code is freedom — patching my itch

July 11, 2011

aliens hat-tip

Filed under: Blogging,Debian,Multimedia,Ubuntu — kees @ 6:04 pm

Picked up a Doctor Who comic today and saw a nice hat-tip to (or composite ship design plagiarism of) Aliens.

The Colonial Marines ship “Sulaco”, from Aliens, 1986:
aliens ship

The Scavengers ship, from the Doctor Who Spam Filtered story, 2011:
drwho art

Such a great ship. Not even remotely made to look aerodynamic.

And to make this almost related to Ubuntu and Debian, here was my command line to remove exif data from the image I took with my phone:

mogrify -strip spam-filtered.jpg

© 2011 – 2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

September 14, 2010

my part in the ecosystem

I was asked to write about what I do at Canonical and what I do in the Free Software community at large. There is obviously a great deal of overlap, but I’ll start with the things I’m involved with when I’m wearing my “Ubuntu” hat.

My primary job at Canonical is keeping Ubuntu secure. This means that I, along with the rest of the Ubuntu Security Team, coordinate with other Free Software distributions and upstream projects to publish fixes together so that everyone in the community has the smallest possible window of vulnerability, no matter if they’re running Ubuntu, Debian, RedHat/Fedora, SUSE/openSUSE, Gentoo, etc. Between vendor-sec, oss-security, and the steady stream of new CVEs, there is plenty going on.

In addition to updates, the Security Team works on pro-active security protections. I work on userspace security hardening via patches to gcc and the kernel, and via build-wrapper script packages. Much of this work has been related trying to coordinate these changes with Debian, and to clean up unfinished pieces that were left unsolved by RedHat, who had originally developed many of the hardening features. Things like proper /proc/$pid/maps permissions, real AT_RANDOM implementation, upstreaming executable stack fixing patches, upstreaming kernel NX-emu, etc. Most of the kernel work I’ve done has gotten upstream, but lately some of the more aggressive protections have been hitting frustrating upstream roadblocks.

Besides the hardening work, I also improve and support the AppArmor Mandatory Access Control system, as well as write and improve confinement profiles for processes on Ubuntu. This work ends up improving everyone’s experience with AppArmor, especially now that it has gotten accepted upstream in the Linux kernel.

I audit code from time to time, both “on the clock” with Canonical and in my free time. I’m no Tavis Ormandy, but I try. ;) I’ve found various security issues in Xorg, Koffice, smb4k, libgd2, Inkscape, curl+GnuTLS, hplip, wpa_supplicant, Flickr Drupal module, poppler/xpdf, LimeSurvey, tunapie, and the Linux kernel.

With my Canonical hat off, I do all kinds of random things around the Free Software ecosystem. I’m a sysadmin for kernel.org. In Debian, I maintain a few packages, continue to try to push for security hardening, and contribute to the CVE triage efforts of the Debian Security Team.

I’ve written or maintain several weird projects, including MythTVFS for browsing MythTV recordings, GOPchop for doing non-encoding editing of MPEG2-PS streams, Perl’s Device::SerialPort module, and the TAP paging server Sendpage.

For a selection of things I’ve contributed to other project, I’ve implemented TPM RNG access in rng-tools, made contributions to Inkscape‘s build and print systems, implemented CryptProtect for Wine, wrote a PayPal IPN agent in PHP that actually checks SSL certificates unlike every other implementation I could find, added additional protocol-specific STARTTLS negotiations to OpenSSL, implemented the initial DVD navigation support in MPlayer, updated serial port logic in Scantool for communicating with vehicle CAN interfaces, tried to add support for new types of timeouts in Snort and Ettercap, fixed bugs in mutt, and added HPUX audio support to the Apple ][ emulator XGS.

As you can see, I like making weird/ancient protocols, unfriendly file formats, and security features more accessible to people using Free Software. I’ve done this through patches, convincing people to take those patches, auditing code, testing fixes and features, and doing packaging work.

When I go to conferences, I attend UDS, DefCon, OSCon, and LinuxCon. I’ve presented in the past at OSCon on various topics including security, testing, and video formats, and presented at the Linux Security Summit (miniconf before LinuxCon this year) on the need to upstream various out-of-tree security features available to the Linux kernel.

I love our ecosystem, and I love being part of it. :)

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

May 27, 2010

too lazy to recompile

Filed under: Blogging,Debian,Multimedia,Ubuntu — kees @ 9:13 pm

I have a camera that takes .MP4 files. MythTV’s Gallery plugin has a static list of extensions it recognizes. MP4 is not one of them. I can rename all my files (which is wrong — they’re MP4s not something else), or I can edit the source and recompile.

Or, since I want to watch them RIGHT NOW, I can binary-edit the shared library and replace something I’ll not have any of (WMV) with MP4.

# strings -a /usr/lib/mythtv/plugins/libmythgallery.so | grep WMV 
*.WMV
# sed -i -e 's/\.WMV/.MP4/' /usr/lib/mythtv/plugins/libmythgallery.so

Ta-da. Just need to make sure I don’t change the size of the binary. :)

(Oddly, I’m not too lazy to open a bug and propose a patch.)

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

December 16, 2009

headache empathy

Filed under: Blogging,Debian,Multimedia,Ubuntu — kees @ 9:20 pm

Run “gstreamer-properties“, click the “Video” tab, change Default Input Plugin to “Custom”, and add this Pipeline:

v4l2src ! ffmpegcolorspace ! vertigotv ! ffmpegcolorspace

Now when Empathy video-chatting with a friend, you can give them a headache!

Or give yourself a headache by trying it directly from the command line:

gst-launch v4l2src ! ffmpegcolorspace ! vertigotv ! ffmpegcolorspace ! xvimagesink

Feel free to replace “vertigotv” with any other or more of the video effects listed in “gst-inspect effectv“.

Here’s me with edgetv ! vertigotv:

Edge Vertigo

© 2009 – 2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

April 19, 2009

recording from PulseAudio

Filed under: Blogging,Debian,General,Multimedia,Ubuntu — kees @ 11:42 am

Every PulseAudio “Sink” has a “Source” named “monitor”. This lets you attach to a given Sink and chain more stuff to it, for example, recording the audio that is playing through PulseAudio at any given moment. This is very handy for creating, for example, PubQuiz-style clips of songs, movies, etc.

Here is a script to find the monitor for the most recently added Sink, record from it, and shove it through “sox” to get a WAV instead of raw sound data (requires recent sox, Pulse, etc):

#!/bin/bash
WAV="$1"
if [ -z "$WAV" ]; then
    echo "Usage: $0 OUTPUT.WAV" >&2
    exit 1
fi
rm -f "$WAV"

# Get sink monitor:
MONITOR=$(pactl list | egrep -A2 '^(\*\*\* )?Source #' | \
    grep 'Name: .*\.monitor$' | awk '{print $NF}' | tail -n1)
echo "set-source-mute ${MONITOR} false" | pacmd >/dev/null

# Record it raw, and convert to a wav
echo "Recording to $WAV ..."
echo "Close this window to stop"
parec -d "$MONITOR" | sox -t raw -r 44k -sLb 16 -c 2 - "$WAV"

© 2009 – 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

October 22, 2008

using hdhomerun IR with lirc

Filed under: Debian,Multimedia,Ubuntu — kees @ 5:18 pm

Followed the instructions and got lirc configured with my HDHomeRun’s IR:


# ID=$(hdhomerun_config discover | cut -d" " -f3)
# hdhomerun_config $ID set /ir/target "10.0.0.100:5000 store"
# echo 'REMOTE_LIRCD_ARGS="-H udp -d 5000"' >> /etc/lirc/hardware.conf
# /etc/init.d/lirc restart
# /etc/init.d/gdm restart

© 2008, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

July 13, 2008

zooming in Xine

Filed under: Blogging,Multimedia,Ubuntu — kees @ 11:34 pm

I use Xine to watch DVDs. In the past I’ve encountered “full screen” (4:3) DVDs that carried a wide-screen (16:9) image. This means there were black bars on the top and bottom of the video frame. When watching this sort of video on a 16:9 monitor, you end up with a full border of black surrounding the image. I have encountered this much more frequently when recording standard definition TV that contains wide-screen video. For example, many music videos on MTV have a wide aspect, but are displayed with top/bottom bars in the 4:3 standard definition frame:

16:9 displayed in 4:3 with black top/bottom bars

Displayed on a 16:9 monitor, in Xine:

16:9 within 4:3 on a 16:9 display resulting in black border

In MythTV, there is a “zoom” function that zooms the video, matching the width of the frame to the width of the display. This ends up cropping the top and bottom black bars, allowing the zoom to fit to the width of the frame:

zoomed to 16:9, cropping unneeded 4:3 bars

I have been unable to find such a feature in either Xine or MPlayer. A weekend ago I ran into another DVD doing the wide-screen-in-4:3 trick, and wrote a patch to Xine to create a zoom post-processing filter. Now I can start Xine like this:

xine --disable-post --post zoom path/to/video

And Ctrl-Alt-Shift-P will let me enable-disable post processing. In my case, I’ll be mapping the VPProcessEnable to the same lirc button I use for zooming in MythTV.

© 2008 – 2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

August 2, 2006

mythtv cutlist to mplayer EDL file

Filed under: Multimedia — kees @ 8:47 pm

I was too lazy to walk over to my TV, so I decided to watch my MythTV recordings on my desktop without having installed a MythTV frontend. Via the magic of MythTVfs, I started watching a recent Stargate episode.

Before the opening credits had finished, I knew I was already going to miss MythTV’s commercial flagging. So started the investigation into where in the world MythTV stores that information. I was imagining adding “.edl” files to MythTVfs automatically, etc.

In MythTV 0.19, using “mythcommflag”, you can get the “Cut” list, but not the “Commercial Skip” list. (Think of the latter as a “Cut Hint” list.) The command for this is:

mythcommflag –getcutlist -c 1059 -s 20060728210000

In MythTV 0.20, you’ll be able to use “–getskiplist”. Since I’m still using 0.19, I had to go directly to the “mythconverg” database to get the details. The marks are stored in the “recordedmarkup” table. The mark types I care about are: 4: Commercial Start, 5: Commercial End. This SQL query gets me somewhere:

SELECT mark, type FROM recordedmarkup WHERE chanid = “1059” AND starttime = “20060728210000” AND (type = 4 OR type = 5) ORDER BY mark;

However, mplayer’s EDL file format expects time, not frame number. MythTV stores frame number. Luck for us, it’s all NTSC MPEG2, so we’re at 29.97 frame per second, and I can modify the SQL:

SELECT mark/29.97, type FROM recordedmarkup WHERE chanid = “1059” AND starttime = “20060728210000” AND (type = 4 OR type = 5) ORDER BY mark;

Now I just have to get the pairs on a single line with a trailing “0” for mplayer to know to skip that time frame:

echo ‘SELECT mark/29.97 FROM recordedmarkup WHERE chanid = “1059” AND starttime = “20060728210000” AND (type = 4 OR type = 5) ORDER BY mark;’ | mysql -B –skip-column-names | xargs -l2 | awk ‘{print $0 ” 0″ }’

Combined with some logic to extract the channel and starttime for a given recording, I’ve now got a really crazy wrapper script that’ll let me mplayer a recording after generating an EDL cutlist.

(With thanks to Ken’s excellent collection of “merge pairs of lines into single lines” short cuts.)

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

July 27, 2006

I love Open Source Software

Filed under: Multimedia — kees @ 12:45 am

Today, Randy let me borrow his awsome presentation remote. It’s basically an RF remote, whose other end is a USB fob, that acts as a keyboard. The two “next” and “previous” buttons on the remote map to “PageUp” and “PageDown” keys, which worked great for the Crucible and Xen presentation Bryce and I gave this afternoon.

Tomorrow, for my MythTV presenation, I’m using OpenOffice.org, which makes a distinction between “Space” and “PageDown”. “PageDown” goes literally to the next slide, where as “Space” triggers the next animation within a slide. Since my presentation slides have a ton of “reveals”, I need to be pressing “Space”, not “PageDown”. A search of the Logitech page yielded no info on changing the remote’s config, so I opted to using “xmodmap” to get the job done for me.

Using “xev”, I found that the keysym for “PageDown” on my keyboard is 0xff56, and the keysym for “Space” is 0x20. So, I blew away my PageDown key, and replaced it with Space:

echo keysym 0xff56 = 0x20 | xmodmap –

Now the remote works just how I need it. :)

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

June 4, 2006

frontline assembly samples

Filed under: Blogging,Multimedia — kees @ 8:08 am

Every once in a while, I hear something in a movie and yell, “Hey! That’s in a Frontline song!” This time it was while catching a portion of the The Abyss, where a big chunk of gear misses falling on their habitat, and they kind of laugh nervously. Well, that nervous laugh was looped for about for about 30 seconds at the start of “Victim of a Criminal” from Millennium. I found it at about the same time I thought to put “Frontline Assembly samples” into Google, which gave me this page, which is a list of all the samples that they used. Robocop, Aliens, and the one that haunted me for a while before this Abyss incident: Stargate and “send in the probe”.

This is what my brain is filled with: sound effects. Great. :)

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

May 22, 2006

TiVoConnect dissector for Ethereal

Filed under: Multimedia,Networking — kees @ 9:03 pm

Over the weekend, I coded up a protocol dissector in Ethereal for the TiVoConnect Discovery Protocol. The protocol is very simple, but I still wanted the satisfaction of seeing it listed by name when scanning through my home network captures while debugging Galleon/TiVo traffic.

Ethereal has great developer documentation. It was easy to find and got me coding right away with a skeleton dissector. I just love the projects with these kind of to-the-point examples. The only thing I felt was missing from their README.developer was something showing that the dissector routine could return gboolean, letting a dissector reject being attached to a given packet.

There were other clearly written dissectors that I used for reference: DNS, Yahoo, and Syslog. They seemed to answer most of the more subtle questions I had about rewriting column text, scanning the packet, and dealing with other special cases.

Hopefully the patch will get accepted. I even did the randomized testing the wiki recommended. :)

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

May 4, 2006

mythtv listing-update time-rotation

Filed under: Multimedia — kees @ 7:33 pm

A few months ago, zap2it‘s renewal survey included a set of questions that were not questions, but rather pleas that people move their database updates to a non-standard time. The default install of MythTV has a hard-coded time in the middle of the night to contact the zap2it servers for TV listing updates. Unfortunately, this means zap2it’s servers were (are?) getting crushed on the hour, across 4 time-zones, in the middle of the night.

To solve this little problem in MythTV, I turned off the built-in “mythfilldatabase” execution, and moved to doing it via crontab. zap2it said they had virtually no load during the day-time, so I moved the update to a little after noon, sticking the following in /etc/cron.d/mythtv-backend:

09 12 * * * mythtv mythfilldatabase –quiet

It dawned on me today while messing around with the “at” scheduler that I could actually randomize when during the day the mythfilldatabase runs. If you wanted to run the job anywhere in a 12 hour (720 minute) period after 9am, you could do it this way:

0 9 * * * mythtv echo “mythfilldatabase –quiet” | at + $(( RANDOM % 720 )) minutes

(My bash manpage says the low-order bits of $RANDOM are as random as the high-order bits, so this is “safe”. If you don’t trust your version of bash, you could use $(( 720 * RANDOM / 32767 )) instead. IANAMG*, YMMV, OMGPONIES.)

*I Am Not A Math Geek

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

careful with mysql downgrades

Filed under: Multimedia — kees @ 1:59 am

I ran into a nasty bug while attempting to downgrade a MySQL database. I had been running my MythTV machine on Debian Unstable, but recently reinstalled to Ubuntu. This has the unfortunate consequence of going from MySQL 4.1 to MySQL 4.0. The “mysqldump” option “–compat=mysql40” kind of forgets to include the “auto_increment” flag for tables creation. This caused my subsequent MythTV 0.18 to 0.19 upgrade attempt to instantly bomb, since all the INSERTs expecting the PRIMARY KEY to increment as new stuff was inserted … didn’t.

Once I split the dumps into tables (-t) and data (-d) with different “–compat” levels and hand-edited the tables, everything was “fine” again. I actually got the whole system up and on its feet again, with no loss of Stargate SG-1 episodes. ;)

So, now all I have to fight with is Xv on an old S3 card. Looks like new versions of Xorg don’t aim Xv to the right place. And, mysteriously, the S3 card’s Xv implementation lacks the XV_SWITCHCRT attribute, so I can’t just use “xvattr” to fix it, like I do I my laptop. Aaagh.

I wonder if something like the xorg.conf’s Option MonitorLayout “TV,CRT” might help it? I’ll try that tomorrow.

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

April 11, 2006

construction movies

Filed under: Multimedia — kees @ 7:55 pm

From the time-lapse construction webcam I set up with Brian, I built a pair of AVIs. These were made with ffmpeg so they use the FMP4 codec. Windows folks can find ffdshow in a number of places.

Left view movie (31M), set to Delerium’s “Silence” (with Sarah Mclachlan).
Right view movie (24M), set to a remix of Everything But the Girl’s “Like the Deserts Miss the Rain”.

I tried to build these movies showing only day-light hours, on work days. A few holidays sneak in, though. (There’s a longer section of a few days where no one shows up for work across Thanksgiving, for example.)

The AVI frame rate is 25fps, with each frame jumping 10 minutes. The effective speed is 14400:1. They span the time from July 26 2005 through January 21 2006. (The right-side camera was added on August 29.)

My least favorite easter egg is where the room-light-shield I taped up to keep room glare off the camera peels off the window and hangs in front of one of the cameras during Christmas and New Year’s vacation (when no one was around to fix it).

My most favorite easter egg is near the latter half when a pile drilling rig is parked in the foreground. Over the course of the day, the hydraulics holding the drill up at an angle bleed out, and the drill slowly pitches forward. I just love it; it’s exactly the kind of event no one notices at the time because it’s happening so slowly. Seen in time-lapse, though, it’s very obvious — it’s the only thing moving at all! :)

Enjoy!

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

April 1, 2006

NetFlix movie downloading

Filed under: Multimedia,Vulnerabilities — kees @ 12:22 pm

Netflix accidentally lets you download movies for free. I reported this on March 18th, but they still haven’t replied. It’s been 2 weeks, so I’m posting the details now.

While digging through Netflix’s javascript I found a function named “startDownload“. I was originally just curious about the AJAX responsible for the movie info popup boxes, but this proved much more interesting.

I’m guessing they must be beta-testing this for some accounts because nothing visible through my account ever calls “startDownload“, but I could still use it.

Turns out the function handles a bitrate selection, and then just rewrites the URL a little. You can get the same affect by just adding “&download=avi&br=4” to the end of a movie info URL. For example, this is the URL to get info about Ice Age, and this is the URL to download Ice Age. This even seems to work without being logged in.

I haven’t had time to check if everything in their library is downloadable, but of the 6 or so I tried, they all worked. If anyone finds a cut-off date, let me know.

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

March 27, 2006

presenting at OSCon 2006

Filed under: Blogging,Multimedia — kees @ 6:43 pm

Woohoo! I got accepted to present at OSCon again! I’m really excited about this one, too — I get to present about something non-work-related. The title of my presentation is “DVR Happiness: Gluing MythTV and TiVo together with Galleon“. Here is my proposed outline:

  1. Intro to DVRs
    • TiVo: have you been under a rock?
    • MythTV: learn all about video standards.
  2. TiVo Gets You A Lot
    • Hacked TiVos can do great things
    • Is your TiVo a tool or a toy?
    • Stock TiVos can do cool stuff too
    • ToGo: move video from TiVo to PC
    • GoBack: move video from PC to TiVo
    • MP3s: streaming from anywhere
    • Image Galleries: beyond just snapshots
    • Galleon Gets You More
    • Implements the server-side of TiVo features
    • On-the-fly format conversion
  3. MythTV Gets You The Most
    • Making Tivo recordings available to MythTV
    • Format conversion
    • Making MythTV recordings available to TiVo
    • Mounting a MythTV filesystem with FUSE
    • Making your MythTV remote make noises
    • Short-cuts with the Linux IR daemon

EDIT: WordPress pisses me off so very much when it comes to lists, indenting, and code snippets. Some day, I will switch to something that just lets me type in HTML and doesn’t try to “fix” it for me afterwards. *fume*

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

January 7, 2006

cheap auto-rotoscoping

Filed under: Multimedia — kees @ 2:35 pm

In the back of my head, I’ve been wondering about various ways I could use my $20 camcorder. Making cheap movie shorts, like SNL’s “Lazy Sunday” come to mind. There are no fancy zooms or special effects. It’s all editing and audio. Bryce kicked some ideas my way, most of which include using the dogs and/or cats as the primary actors (I can pay them in kibble). He also suggested renewing my father’s Rottweiler Camcorder with a much smaller device. I think the images would be mostly obscured by my dog’s chin. We’ll see. I also worry it may suffer from being summarily ingested.

While playing Name that Tune 80s DVD Edition, the a-ha video for “Take On Me” came on. The rotoscoping used to create the animated parts made me think of Inkscape‘s autotracing function, and I lept up to go investigate the possibilities.

So far, I’ve played with two styles. One leaves the autotraced fill areas (which makes a video look like a really freaky cartoon), and one that reduces the fill opacity, and adds line density so it looks more like a regular outlined cartoon. That one tends to be distracting, though, since the edges keep jumping all over the place.

The autorotoscope script requires mplayer, ImageMagick, autotrace, and ffmpeg.

Here are the results:

These AVIs use xvid for their video codec. If you don’t already have it, you can get them from here.

© 2006 – 2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

December 23, 2005

mythtvfs

Filed under: Multimedia — kees @ 2:58 pm

My first working crack at getting a filesystem overlay working between MythTV, Galleon, and my TiVo is finished. If you’re brave, check it out.

© 2005 – 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

December 22, 2005

galleon and FUSE

Filed under: Multimedia — kees @ 9:46 pm

Every few years, I end up relearning Java for some project or another. Today, that project is Galleon, the fantastic TiVo Home Media Server. I’ve been sort of mildly involved in another person’s attempts to get MythTV and his TiVo working nicely together.

The basic situation is this: I want to be able to watch stuff I’ve recorded on my MythTV box via my TiVo. Galleon already does this.

Using the “GoBack” TiVo feature, Galleon acts like another TiVo on the network, and sends a show to the TiVo. Normally this is used to send a .TiVo file from your computer back to the TiVo. In the case of TiVo-to-TiVo communication, they include the metadata about the show (title, air date, duration, description, etc) before starting the data transfer. Galleon stores the metadata to a local database when it downloads shows via the ToGo feature.

It is possible to send any valid MPEG stream to the TiVo, but unless the Galleon database has metadata for the show, there will be nothing but the filename on the TiVo end when it transfers. In the case of MythTV shows, the metadata is contained in the MythTV database. I’m hoping to create a MythTV “application” for Galleon that will connect to the MythTV database, and populate the TiVo with the needed metadata.

Since I’m so green on Java, I’m doing something else as a “proof-of-concept”. It was suggested that some of the metadata could be encoded into the filename. This requires two halves: a parser in Galleon to extract the data, and the files to be named with their metadata encoded.

On the Galleon side, I’m digging around with the StringTokenizer, and generally getting my feet wet with the Galleon source and banging my head on the java compiler.

On the filesystem side, I’m going to use FUSE to create an overlay filesystem that queries the MythTV database, and builds a list of files based on an NFS mount’s contents. (Which I’ll NFS mount from my MythTV box.)

Mostly I just wanted to write a FUSE application. :)

© 2005 – 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

December 12, 2005

xmltv article

Filed under: Multimedia — kees @ 11:59 am

My friend Brian just had his article-writing debut on O’Reilly‘s ONLamp site. He wrote about how to use XMLTV to help manage your TV viewing schedules if you don’t have a DVR doing it for you (or not doing it well enough).

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

October 23, 2005

mythtv original air date visibility

Filed under: Multimedia — kees @ 3:12 pm

I have been really unhappy with MythTV’s visibility of program “Original Air Date” information, which as far as I could tell is only visible through the Program Finder. I wanted to be able to see original air date while I browsed my recordings. Digging through the MythTV code has proven very difficult. The documentation has been minimal, and I haven’t found any tutorials on theme creation, which seems to be where all the visible components of the mythfrontend get their details from.

While looking for the bleeding-edge code, I did find http://cvs.mythtv.org/ which is actually a Subversion repository, bug tracker, and wiki. There a nice start to information there, including doxygen output. Also, the #mythtv-users channel on freenode has a nice MythTV FAQ.

The bulk of the display stuff I was looking from takes place in programs/mythfrontend/playbackbox.cpp (thank Bryce). The “cursorDown” function led me through to the “update*” functions, and eventually ToMap/SetText calls, which load program information into a hash, and then pass that hash to the theme engine.

libs/libmythtv/programinfo.cpp has ToMap defined, and all the various hash keys are visible, including the original air date variable I was looking for:

progMap[“title”] = title;
progMap[“subtitle”] = subtitle;
progMap[“description”] = description;

progMap[“originalairdate”]= originalAirDate.toString(dateFormat);

SetText is in libs/libmyth/uitypes.cpp. Hash items are uppercased to match %-enclosed words from the themes. The first “|” seen is to identify “what appears in front”, and the second is “what’s after”.

I modified the ui.xml from my theme (G.A.N.T. currently) from:

  <value>%SUBTITLE|”|”
%%STARS%%DESCRIPTION%</value>

to:

  <value>%SUBTITLE|”|” %%ORIGINALAIRDATE|(|)
%%STARS%%DESCRIPTION%</value>

So now, when I scroll down to Smallville, I see in the description box:

“Aqua” (2005-10-20)
During a beach party Lois hits her head when she jumps into the lake, and …

Ta-da! Original Air Date in parens. Now, being able to see the year is important, so I had to change my date format to one that included the year, but it’s ugly. To fix this, I need to actually change code. In “MythDateFormat” from programs/mythfrontend/globalsettings.cpp, I added:

gc->addSelection(sampdate.toString(“ddd MMM d, yyyy”), “ddd MMM d, yyyy”);

Now I just have to get it compiled. :)

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

October 16, 2005

mythtv button sounds

Filed under: Multimedia — kees @ 11:51 am

Well, I’ve discovered that my “slow to respond” UI was entirely due to the lirc “repeat” settings. I’ve eliminated (“repeat=0”) the repeat settings for the Esc, Up, Down, Left, Right, Space, and Return buttons. I discovered this only after seeing that ircat was just as slow to respond. I’ve hacked together a “make noise” script (named “irnoise”) that runs along with mythfrontend:

#!/bin/bash
export SOUNDS=~mythtv/sounds

ircat mythtv | while read NAME; do
  case “$NAME” in
   Return|Space)
    SOUND=select.wav
    ;;
   *)
    SOUND=default.wav
    ;;
  esac
  #echo “$NAME: $SOUND”
  aplay -q “$SOUNDS”/”$SOUND”
done

This gives me my “boop” and “click” noises for all remote buttons. Yay! Since I couldn’t find the official TiVo noises, and I don’t feel like taking my TiVo apart right now, I just grabbed some noises I found online. From the “MiscWAVs.zip”, I used “THUD.WAV” and “BTN_DWN.WAV”:

sox src/THUD.WAV -t wav -c 1 -s -w -r 48000 default.wav resample
sox src/BTN_DWN.WAV -t wav -c 1 -s -w -r 48000 -v 2 select.wav resample
normalize-audio -m default.wav select.wav

Not a lot of complaints left with my MythTV installation. :)

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

mythtv dpi

Filed under: Multimedia — kees @ 9:43 am

I accidentally fixed my “fonts are too small” problem. While reading the MythTV development notes, there was mention of everything being designed for a 100dpi screen resolution. As it turns out, my Xorg config was defaulting to 75dpi, so I forced it with the DisplaySize option. Since my video card uses 640×480 for it’s NTSC output, I had to modify the recommended settings to use:

DisplaySize 162 121 # 100 DPI @ 640×480

I was surprised to discover that this solved my font size issues. I had so totally given up on the font size problem I didn’t even list it was a problem in the prior MythTV blog entry. :) I was actually expecting fonts to get even smaller, but I guess this changes how font selection is done, and as a result, everything appears sane now. Neato!

Also yesterday, in the hopes of reducing the effects of the “crappy audio” problem, and allowing multiple programs to have the soundcard open, I figured out how to get ALSA working natively within MythTV. In the MythTV configurations, the sound device should be “ALSA:default” instead of “/dev/dsp”, and the mixer should be “default” instead of “/dev/mixer”. The start of this was gleaned from the link I mentioned earlier. Mixer settings were found through trial and error. Strangely, MythMusic had a separate playback configuration, so I had to change that to “ALSA:default” as well.

Now that ALSA is being used, the audio choppiness has not returned. I can still get desync’d A/V, but I think that’s entirely due to disk latency issues, or something like that. Usually when it happens, if I pause or restart playback, it goes away. Also, since multiple programs can open the ALSA device and play sounds at the same time (thank you ALSA dmix), I can start looking at how to add a tool to play “button press notification” sounds. I’ll initial probably use something like “ircat” piped to a reader just to get a proof-of-concept. Then I’ll find some hooks in mythfrontend to attach it to instead. Eventually, I was a themeable visual notification. I should probably join the dev list to make sure other folks aren’t already working on this.

I’ve also found mention of “show type priority bumping”, where “New Episode” can trigger a bump in the priority of a recording. This may be a good step towards recording new Stargates at high priority, but reruns at very low priority. In the priority adjusting tool, I can see the third row for this kind of priority bump, but I can’t find the UI elements to adjust it.

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

October 15, 2005

mythtv OMG

Filed under: Multimedia — kees @ 9:25 am

I was preparing myself to deal with the “pain” of moving to a multimedia system that didn’t have all the feature tweaks I need. MythTV has surprised me in that after only a week, and I’ve solved almost all my issues through just finding the right configuration options.

As of about Tuesday (2 days after the initial “commitment”), I was ready to call it “better than my existing system”. It had a few glitches that bugged me, but overall, it had many many more features than I was expecting. After watching Smallville Thursday night, I’m a total freak for MythTV. Smallville was basically the first “Production” show MythTV recorded for me. I had recorded “Medium” earlier in the week, and that served as a good way to feel out the interface. Smallville is the real test because it’s at position 1 in my TiVo (and MythTV) recording priorities. I had no irritations while watching it. Nicely done, MythTV.

Rewind to Saturday. Bryce and I spent about 12 hours straight digging through KnoppMyth both on his new system and my machine that I brought over to his place. By the end of it, I had entirely reinstalled my system with Debian Sid, and installed the most current ivtv drivers, with the apt-able myth binaries. We had figured out how to get KnoppMyth running with the newer tuner chip, but Bryce’s HD audio card wasn’t supported in that version of the kernel. Let me just say, everyone should just start with the latest ivtv driver. It detected everything correctly right off the bat. On Sunday, Bryce installed Gentoo, and got the latest ivtv, etc, everything was happy.

Early this week I toyed with the remote control settings, and discovered a whole mess of MPlayer commands I didn’t know about that let you control playback speed (including fast audio!), OSD text (so I can have a visible indication that I’m paused), etc. After restoring my other MPlayer defaults (16M cache, readable font, etc), MPlayer stuff was in great shape again, including DVD playback. I also programmed my spare TiVo remote to control my stereo power and volume. I’m down to 1 remote finally! (I was surprised that the TiVo remote programming codes aren’t online anywhere. The only guide seems to be in the TiVo itself.)

Yesterday I discovered MythWeb. I must have been blind to miss it before. That’ll teach me not to read the entire EFF MythTV guide first. Full scheduling, guide data, recorded show lists, and most importantly, the ability to adjust the keybindings for the various MythTV modules. In the MythMusic module, I was infuriated that “PgDn” would skip to the next song, instead of (wait for it) paging down in the list of songs. I just can’t understand why such a massively counter-intuitive setting is the default.

The commercial detection system is greate so far. It’s already painful for me to go back to using my TiVo where I have to press the “skip 30 seconds” button ten times to get past all the commercials.

Current issues:

  • Interface is slow. Everything (especially the video browser) is slow to respond. I miss not having an audible notification that a remote button was received, but there should at least be SOME kind of visible change if a button is pressed.
  • Intermitant crappy audio playback. Something goofy happens on playback sometimes where the audio is just totally trashed. I just have to quit the playback and try again. I wonder if switching everything to using ALSA would make things better.
  • No way to record the same show with two priorities. I want to have “New Episodes Only” for “Stargate: SG-1” at a high priority, but “Any time, any channel” for it at a very low priority. I haven’t figured out how to do this yet. I think I will have to write special recording rules for it in SQL somewhere secret.

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

September 16, 2005

song tagging

Filed under: Multimedia — kees @ 6:03 pm

Jimmac you are my hero! I was just complaining about needing to tag a lot of my early music rips so they would be sensible in MythTV (and on my iPod), and poof, there’s the answer in my RSS feed reader. tagtool is exactly what I was looking for. It’s going to take me about 10 minutes to tag a little over 5000 mp3s. And for 9 minutes and 30 seconds of that, I’m going to be looking up publication years, which is the only thing not in the pathname that’s going to get sucked into the ID3 tag. Great, great tool!

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

September 14, 2005

tivo meets mythtv

Filed under: Multimedia — kees @ 11:50 am

As Bryce has already detailed, a few of us have started examining MythTV. While TiVo has served me pretty well, it sounds like now that the recording restriction flag has been tested it won’t be too long now before the content providers start actually using it. Supposedly the latest alarm is unfounded, in that it was a mistake. Whether it was a test, a mistake, it does show that my TiVo is now prepared to cripple itself.

MythTV has come a long way now. It seems that it’s a viable alternative for a DVR. Within our MythTV Club, we’re each going to have different problems. Doug will be fighting DVD playback via his PVR-350, Bryce will be playing with double recording sources, and I will be fighting with the video library playback capabilities. The EFF has a great write-up on installing MythTV for broadcast HDTV support. I think HD broadcast recording is going to come pretty late in our designs.

My existing video navigator (not the TiVo) is a very simple filesystem browser that just launches “mplayer” or “xmms” on the files it finds. (Holy crap, I set that up in 2002?!) MythTV’s “mythvideo” plugin is a much smarter browser for both video and audio, but its interface is wildly different from my current system. Especially for audio, which is very ID3-tag heavy. Very few of my early CD rips have any ID3 tags at all. The built-in MythTV player is also very very broken for seeking, and for unusual formats. I have a lot of movie trailers in Quicktime that MythTV refuses to play. I suspect that the video library issues will be the easiest to solve, though, since I can select a different viewer, etc. All that I have set up already on my existing navigator.

The nice thing about MythTV’s video/audio browser, though, is all the metadata support. I can pull down DVD and CD covers, etc. It’s way prettier than my existing system too. My goal is to get MythTV set up on my existing TV computer this week. Once it has replaced the filesystem browser, I’ll have the same functionality as the old system, and gain the ability to move on to real DVR activity in the future. I want to match my TiVo’s recording schedule, and then probably run in parallel until TiVo really gets DRM going.

I’m curious to see what the Netflix/TiVo joint venture is going to look like. I suspect the recent DRM testing is for the Netflix offering. If that’s true, I’ll wait until I see it, and then cancel my service. I’d really like to support TiVo, but not if they’re trying to cripple my time-shifting/long-term storage capabilities. I wonder how long it’ll take me to kernel-monte the TiVo into a MythTV frontend. All I really need is a cross-compiler. Sure would be nice to reuse my TiVo as a MythTV frontend if it comes to that.

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

August 28, 2005

cvs camcorder

Filed under: Multimedia,Reverse Engineering — kees @ 12:43 am

I was able to get my hands on a CVS camcorder this past week. The unit is very cool. It doesn’t have any features of a “real” video camera, but I view it like a super-version of my Digital camera which can take 15 seconds worth of video at a time. This one is only limited by available storage (about 20 minutes). The resolution can be turned up to 640×480, too.

Initially I couldn’t get a cable built for it. I had a spare USB cable and a spare Palm Pilot cable to merge, but after soldering it all together, the device wouldn’t take an address (but my laptop noticed it was getting plugged in). After a hint from Linus (“plug-ins are detected via power-draw, everything else is over the data lines”), I realized that perhaps the USB data wires were reversed (the cable has got to flip them on at least one end of the cable… I just got unlucky).

The software for downloading the videos had been ported to Linux, and I worked on some more code to have a one-shot ability to download all the videos from the camera. Now I’ve got autoconf/automake working for it, and have gotten myself added to the Saturn Tools project where we can all work on the code through SourceForge’s CVS. Much easier than endlessly trading patches. :)

So far, I’ve managed to wreck the splash-screen image during camera bootup. Something is busted with the usb_bulk_write calls, and the device stops taking writes after 4k worth of data. Hopefully I can get that repaired so I can be greeted by the Laughing Man. I’ve also got a JPG all ready to replace the logo screen, “powered by pure digital technology”, with my oh-so-clever “powered by pwnd digital technology”.

© 2005 – 2016, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

May 2, 2005

I feel dirty

Filed under: Multimedia — kees @ 11:13 pm

I’ve just published srcfilter. It runs under Wine, which is the only reason I kept working on it. Coding in the Windows style has this sickening allure, like an overly sweet smell. I know I don’t want it, but it seems so tasty. (Oh, DWORD, how I hate thee. All your instance names start with “dw”.)

Anyway, srcfilter uses the DirectShow API to initialize a SourceFilter from a DLL you supply. It then pulls a specified source file through the DLL, writing the output to a file. Similiar things can be done with the GraphEdit tool that comes with the DirectX SDK. If you wanted to set up a Graph that uses a SourceFilter to read a file and a Dump sink to write output, you can use srcfilter to do the same thing. And you wouldn’t need Windows to do it.

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

April 28, 2005

video editing with gopchop

Filed under: Multimedia — kees @ 10:16 am

Well, in the last few weeks, after finally getting MPEG2-PS streams out of my series 2 TiVo, I’ve been actively using gopchop again. It had stagnated, but it seems that people with hardware MPEG2 encoders continue to use it. I should have realized sooner that they are gopchop’s audience. It’s traditionally been a rather fragile bit of code for software-generated MPEG2-PS streams, but pretty stable for the more regular hardware-encoder streams. Between my recent increase in its use, and two large patch bundles I was sent, I’m trying to pick up development again.

One thing that has been particularly frustrating when dealing with MPEG2 video has been dealing with the rendering of the pictures. MPEG2 stores its pictures “out of order”. And by “order” I mean display order. They’re in order for decoding, but not for display. libmpeg2 handles all this for me, except for the part where I need to stop on a specific picture. For an accurate GOP splitting UI, I need to display the last picture of a GOP. Depending on the stream, libmpeg2 may not have rendered the last picture, since it’s waiting for more B or P frames before it knows the decoding is done. (For example, B frames depend on frames in the future, so you can’t render them until you see further down the stream, etc.)

My plan to deal with this is to force-feed libmpeg2 with End-Of-Stream packets, and not render stuff until I’m on the picture I want. For example, to display the initial B or P frames from a GOP, I may need to process the entire prior GOP first. For some types of streams, this may end up being very CPU expensive, since they may have hundreds of pictures in each GOP. But, this is why MPEG2 editing is hard. :)

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

April 16, 2005

referer headers

Filed under: Multimedia,Security — kees @ 2:33 pm

I’m surprised that anyone still uses referer headers as a “security” measure. I’ve come across this several times recently. I’ll select a URL out of firefox, and paste it onto a curl -O command line, only to end up with a 0-sized file. And usually if I just add -e [site URL] to the command line, poof there’s my file. Most recently, I found this when trying to download the freely available Nine Inch Nails samples.

Seriously, what’s the point of doing this test? I don’t understand at all. If you want people to download a file in their web browser, do you think they can’t figure this out?

© 2005, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

Older Posts »

Powered by WordPress