LCA 2009: Day One, Mini-conferences

A brief summary of the talks I attended today ...

My LCA 2009 Flickr photostream

NDB / MySQL Cluster by Monty Taylor (Sun Microsystems) mordred@sun.com
- URLs ...
  - https://launchpad.net/ndb-bindings
  - ndb-connectors@lists.mysql.com
  - http://lists.mysql.com
  - IRC #mysql-ndb at freenode.net
- High availability, 5 9's (99.999%)
- Sub-second fail-over
- Configurable redundancy, e.g. number of replicas
  - Best to use just 2 replicas, using more isn't necessarily a good thing
- High performance through parallelism
- For simple primary key look-ups as fast as memcached
  - But, harder to configure
  - Doesn't implement foreign keys
*** Fully asynchronous, includes an Event API ***

- Data and indices kept in main memory and check-pointed to disk
- Share nothing, commodity PCs, commodity interconnects, no expensive shared disk
  - No single point of failure
- Originally a project at Ericsson, then MySQL and now Sun Microsystems
- Hash (MD5) on the primary key and split across systems accordingly
- Storage Engine API (per table configuration)
  - Like a Virtual File System
  - Different implementation, e.g. InnoDB, MyISAM, NDB, Falcon
  - NDB doesn't exist in Drizzle yet
  - NDB only really exists in the MySQL cluster fork
- mysqld talks to cluster of NDB nodes, using NDB wire protocol
- Using NDB directly is roughly 5 times faster than SQL
- With 16 nodes has achieved ...
  - Using MySQL: 120,000 transactions per second
  - Using NDB 5.1 directly: 500,000+ transactions per second
- Complete NDB API ...
  - Primary Key operations: readTuple, writeTuple, deleteTuple
  - Index Scan - readTuples
  - Table Scan - readTuples
  - Unique Index Scan - readTuples
  - All other operations can be broken down into combinations of these
- NDB API / Bindings for Java (most complete), Python, Ruby, Perl, C#, Lua
- Require ...
  - MySQL installed with libs and headers
  - MySQL 5.1 CGE 6.2 required (and 6.3.14 or 6.3.16 will also work)
- C++ API (libndbclient) wrapped with SWIG
  - Java bindings (NDBJ) or JNI via SWIG
  - Java NDB object is not thread-safe, need to have one NDB instance per thread
- REST API: http://code.google.com/p/mod-ndb/
  - Returns: JSON, XML, YAML

Transcoding in MythTV by Paul Wayper

- Capture analogue signal can result in significant storage requirements
  - Good idea to transcode from RTJPEG into MPEG-4
- Remove "cut regions", e.g. advertisements, etc.
  - Editing using "e" key whilst watching
  - "z" to import commercial flagged regions
- Transcoding ...
 - Removes previously set "cut regions", but not "flagged regions"
 - Downsize physical area
 - Recompress using MPEG-4 (fixed bit rate with quality settings) and downsample audio
 - Exporting to other media
   - Using nuvexport to reformat and export to external storage
     - But, easier to just add storage and kept video stored in MythTV
   - User jobs are harder to configure, separate script, can be performed automatically
     after transcoding, environment variables for parameters
     - An example: export to mobile phone 3gpp files
- Ended up talking a lot about his custom notebook cover made from laser-cut wood veneer

Panel on Geek Parenting by LinuxChix

- Q: How did you start out as a "geek parent: ?

  A: Reading books on parenting
  A: Lessons learned from parent's parenting
  A: Listen to all the details and throw it all away apart from the "nuggets of wisdom"
  A: Not something you can prepare for intellectually, because it's not an intellectual process
  A: Freaking out from total information overload. Avoid taking on too much information from outside
  A: Funniest book: "Perfect parents" (history of parenting)

- Q: What type of teaching, before your kids started formal education ?

  A: Reading to them and music
  A: The simple things that we all take for granted, e.g. language and mathematics
  A: Lots of group activities, getting them out in the open
  A: Parent-child relationship goes both ways, e.g. parent also learns from child
  A: Allow child's simple explorations become opportunities for "finding out about the world"
  A: Read "thousands" of books to kids, counting games (with numbers)
  A: Looking for toys with some sort of "creative construction" element, e.g. Lego
  A: Less interest in games with fixed or limited outcomes, rather than more open-ended
  A: Capacity to learn driven by motivation
  A: Engage them in activities: physical, social, intellectual, artistic
  A: Know when to "stop" ... when some activity isn't working
  A: Know when not to "stop" ... don't let them learn to give up when things get hard

- Q: Double challenge for parents of girls, encouraging interest in maths and science
  A: Social stigma associated with girls and geeky, technical subjects ... still true ?
  A: Are toys becoming more segregated into boys and girls ?
  A: Still opportunities for balance between girl and boy interests.

- Q: Juggling parenting and your career ?
  A: Divorced, by self, entrepreneur ... organised with a regular routine, balance, prioritise
  A: Important to spent more time at home with your children, if at all possible
  A: Having a full-time, start at home Mum can be good, if at all possible
     But, need to ensure that you are very engaged as a parent when you are at home

  Gever Tulley: TED Talk: 5 Dangerous things your kids should do !
  http://www.ted.com/index.php/talks/gever_tulley_on_5_dangerous_things_for_kids.html

System Administration Lightning Talks

VESPER (Virtual Embraced Space ProbER) by Sungho Kim
- http://sysadmin.miniconf.org/lightning09.html#01
- Uses KProbes (Kernel Dynamic Probes = automated breakpoint, small C functions)
  - http://sourceware.org/systemtap/kprobes
- Gathers information from the guest operating system under test

Mondorescue via PXE by Simon Lyall
- http://sysadmin.miniconf.org/lightning09.html#03
- http://www.mondorescue.org and http://trac.mondorescue.org
- Documentation: /usr/share/docs/HOWTO.pxe (?)
- Backup and Recovery tool
- Creates bootable images of running servers
- Intended for system and appications
- Not for backing up all your data, e.g. not database, not video
- # mount ...; mondoarchive ...; umount ...
- Full or incremental back-ups
- Use with PXE for booting and restoring system
- Also good for cloning machines
- Read the Trac FAQ (more up-to-date) after the main FAQ

Think by Frank Sainsbury
- http://sysadmin.miniconf.org/lightning09.html#04
- We are paid to "think"
- Frank has been at the "rough end" of some pretty poor thinking
- Works for someone who wants everything and wants it yesterday
- Project: Places SMS messages into an Oracle database
- Based on Windows 2003 server and Oracle database
- Tactical thinking: How to fix the problem
- Strategic thinking: Longer term ... but no-one had thought about that, e.g. keep it simple
- Should perform strategic thinking, even if it is a (so called) simple task
- Security has become a bit part of what we do these days
- Running 2 versions of Apache, versions of Java, JavaScript, Mono and many, many other things !
- Need shared access to good Oracle database models for a specific domain, improve implementation time
- Caught out by RedHat ES deployment issue
- Need all Linux hackers / system administrators to *think*, collaborate and do it better !

System administration - a programming perspective by Devdas Bhagat
- http://sysadmin.miniconf.org/lightning09.html#02
- Need to think in terms of multiple systems, not single systems
- Need to have centralised configuration system management
- System administrator also needs to be a programmer to solve the problems
- Opposing view: Centralised configuration can be overkill
- Centralised deployment management ?
- Use "puppet" ... http://reductivelabs.com/trac/puppet/wiki/PuppetIntroduction
- Test Driven Systems Administration ?
- System Administrators need to work much more closely with Developers, particularly for production environment
- Developers need to test against an environment that is as close as possible to the production environment

Large CPU masks by Rusty Russell

- Kernel primitives ...
  - Runtime NR_CPUS and cpus_weight_nr() and for_each_cpu_mask_nr(...)
  - set_cpus_allowed_ptr()
  - Magic declaration and allocation macros
  - Returns pointer to cpumask for specified node
- These primitives are applied on an "as needed basis" ,,,
- And trial-and-error, only fix when broken
- What if distributions set CONFIG_NR_CPUS=4096, even if you only have 2 CPUs ?
- 2.6.29 aim is to remove (almost) all cpmasks from x86 stack
- CONFIG_CPUMASK_OFFSTACK: currently only set by x86/64 MAXSMP or via DEBUG_CPUMASKS
- We do have a typedef "cpumask_var_t" and also "alloc_cpumask_var(&var, fp)"
- Need to provide new kernel API, switch all code across, then deprecate the old kernel API
  - Usually takes at least one cycle of kernel development
- 2.6.30 aim is to remove all cpumasks and old operators from everywhere on x86
- Fairly good set-up for "static" per-cpu variables ...
  - DEFINE_PER_CPU(type, name)
  - __get_cpu_var(name)
  - get_cpu_var(name), put_cpu_var_name(), per_cpu(var, cpu)
- Also have "temporary" system (now 5 years old) for "dynamic" per-cpu variables ...
  - Mirror "static" per-cpu variables, but uses alloc / free
- Do have a real allocator inside the module code, but never exposed because per-cpu space is severely limited
Longer term: Growing per-cpu areas ?
  - Virtual contiguous layout scheme ?
  - Then networking code can move back into using per-cpu areas

Presence Awareness for MythTV by Jonathan Oxer
- SuperHouse: Channel 31 in Melbourne, later in the year
- Blog: jon.oxer.com.au
- Flames: jon@oxer.com.au
- Web: http://superhouse.tv
- Using Arduino + PHP code to play/pause MythTV
- Multi-room MythTV and follow-me
- Presence loss: query location, parse and log -> Last event for a particular person
- Presence gain: read log, change channel and play from given point in movie
  - Problem MythTV back-end doesn't provide play video from given point in movie
- Use "zoneminder" and security cameras
- Facial recognition: Intel video library "opencv"

MythTV Lightning Talks

MythTV usability: Customisation and tweaking versus the Wife Acceptance Factor !
- HTPC - Home Theatre PC
  - Two vague terms crammed together ?!?
  - HTPC - Customisation ?
  - MythTV - Convergence ?
  - PVR - Convenience ?
- Not having to mess with a collection of video tapes or DVDs
- Customizations: Best image quality, scaling
- Windows: FFDShow http://www.avsforum.com ... difficult to configure, too many hours required
- Good versus Bad customisation
  - Good: Learning remote, Useful jobs (copy and transcode, burn to DVD), MythTV menus
  - Useful: Remote admin: ssh -L 9999:localhost:80 host

Linux kernel Panel

- Worst bug introduced into the kernel ?
  - Memory leak in the ICMP stack, took a year to discover
  - Removed semicolon (?) which broke part of the build for 6 weeks, before it was discovered
  - Broke the build in Linus' tree
  - Broke code turning off barriers, which wasn't possible to test, but broke for someone in the field
  - Forgot some parenthesis so that a divide-by-two became multiple-by-two, lead to allocation
    growing too fast.  Took 2 years to discover.

- How did you become a kernel hacker ?
  - After fixing lots of kernel bugs (in Debian)
  - Was Mac OS hacker, started playing with Linux, noticed it didn't run so well, knew hardware
    pretty well, and slowly started hacking kernel more and more
  - Started hacking on hand-rolled BSD kernels in embedded systems.  Moved across to working on XFS.
  - Started working at Sequent on MPP systems.  Hired by IBM, started working on broken Linux kernel
    things
  - Started kernel hacking in 1996, was sick of working on minority O.S.
    Linux looked like the next bug thing.  First patch was to ISO9960 file system

- What development environments and work flow process do you use ?
  - VI, GIT, make and shell scripts
  - Emacs (minimal features)
  - Anything that runs in an xterm, e.g. VIM, user mode Linux, sometimes Linux on ESX
  - KVM or VMWare to test ISO of distribution
  - Make and test on real hardware, due to using drives for device specific hardware
  - Would be useful to have tool to many KVM images
  - Use "quilt" for work-flow
  - Should use "guilt", which is "quilt" on GIT :)
- When will you be finished [as a kernel hacker] ?

  - When I've drunk away the less of my brain cells
  - When it stops being fun
  - When I can buy my very own goat farm [laughter]

- Worst part of being a kernel hacker ?
  - There is no holiday, it's 24x7 !  Always someone asking a question
  - Having the control-D key wearing out on all my keyboards
  - Too much stuff to keep track off
  - If you aren't deleting your email intray messages, you're not a kernel hacker
  - Can't keep up with all of Linux Kernel email list messages
  - No-one person has a full grasp on the whole kernel ... but, it's getting worse
  - Other people breaking your stuff
  - Coming back from vacation ... with pages and pages of messages
  - Other people touching stuff [laughter]
  - Other people fixing a problem, but the solution is almost worst than the problem !
    - Especially, when it's Linus' [laughter]  Linus is perfect [laughter]

- Have you been flamed by Linus ?

  - YES.  YES.  YES.  YES.  YES
  - Being flamed for coding style
  - Flamed at 2007 kernel site regarding scalability for device probing ...
    - "Stupid SGI people with their stupid problems, should stop bothering other people with their problems"
  - Broke something for a Mac Mini ... and Linus has a Mac Mini !
  - Linus flamed someone (and everyone involved in Linux security) as "wanking around",
      which means he really disagrees with you
    - This one was picked up on by all the technical press, e.g. article in The Inquirer
    - Linus is really quoted properly in context

- Question regarding security
  - Lots more code being committed these days, but not more code review
  - Some third-party organisations now have financial incentive to find security problems
    - Leads to more reports
  - Kernel hackers don't have the same types of incentives to find these sort of problems
  - Again: Just lots more code these days
  - Can't stop bugs getting into the kernel without slowing down the rate of development ... so ...
    Need some tool that will allow you to patch them out in the field as they are discovered [laughter]