Life in Code

SQL Koan

2011-10-29T00:13:00.000-07:00

It's not that profound, but I sure do like it: a simple, elegant example of a self-join that gives a truth table for the NOT DISTINCT FROM operator.

  WITH x(v) AS (VALUES (1),(2),(NULL))
  SELECT l.v, r.v, l.v = r.v AS equality,
    l.v IS NOT DISTINCT FROM r.v AS isnotdistinctfrom
  FROM x l, x r;

Thanks to Sam Mason via the PostgreSQL - general mailing list.

Why can't ads be more fun?

2011-10-28T21:42:00.000-07:00

The worst part about advertisements these days, in my not-so-humble opinion, is the degree to which they belittle the user. Well-endowed lady wants to be my Facebook friend? "Click here to confirm request." Riiight. Lonely singles near me? Yeah, sure. OH GOOD LORD, A THREAT HAS BEEN IDENTIFIED! Oh, wait, y'all really are evil! On the infrequent occasions that I do watch TV, the ads are less flagrantly insidious, but they're nonetheless relentlessly patronizing.

The second-worst part of ads, IMNSHO, is the brute-force repetition. The same inane 30 seconds once? I can do that, but five times in an hour? You've got to be kidding me. Nope, no kidding here. Just 30 seconds of the same inanity, again and again.

Consequently, I find unique and intelligent ads rather inspiring. I know, it shouldn't be this way. One might thing that unique and inspiring would be more common amongst something so prevalent. Anyway...

Some of the GEICO ads that have played on Hulu lately have managed to avoid the "worst part". They're short and playful, vaguely reminiscent of [adult swim] commercials. "Is the sword mightier than the pen" is my current favorite; I still sometimes chuckle when I see it. They still fail occasionally in the repetition department. If you're a large international corporation, how hard is it to make more than a handful of moderately-entertaining 30-second spots?

Which brings me back to banner ads. I just saw this at the top of ars technica today for the first time, and it really caught my eye. I don't read or write Javascript, but I found myself puzzling through it and then following the link.

At the top of the page, I find this banner, which visually reminds me where I came from, that I've come to the right place, and gives me another little puzzle. Neat. Thank you, Heinz College of Carnegie Mellon.

None of this is revolutionary, but I do hope it is evolutionary. Ads don't have to be boring, and I'm guessing that interesting and intelligent ads are more effective even as they're less painful and insulting. My guess is that a generational change-over in marketing departments and their managers is underway, and will be slow. Still, I look forwards to a new generation of more modern advertising approaches that gives me something to think while my eyeballs are held hostage.

FFT / Power Spectrum Box-and-Whisker Plot with Gggplot2

2011-10-06T20:28:00.000-07:00

I have a bunch of time series whose power spectra (FFT via R's spectrum() function) I've been trying to visualize in an intuitive, aesthetically appealing way. At first, I just used lattice's bwplot, but the spacing of the X-axis here really matters. The spectra's frequencies aren't regularly-spaced categories, which is the default of bwplot. If all the series are of the same length, then the coefficients of the are all estimated at the same frequencies, and one can use plyr's split-apply-combine magic to compute the distribution of coefficients at each frequency.

I spent some time trying to faithfully reproduce the plotting layout of the figures produced by, e.g. spectrum(rnorm(1e3)) with respect to the axes. This was way more annoying than i expected...

To include a generic example, I started looking around for easily generated, interesting time series. The logistic map is so damned easy to compute and generally pretty interesting, capable of producing a range of frequencies. Still, I was rather surprised by the final output. Playing with these parameters produces a range of serious weirdness -- apparent ghosting, interesting asymmetries, etc..

Note that I've just used the logistic map for an interesting, self-contained example. You can use the exact same code to generate your own spectrum() box-and-whisker plot by substituting your matrix of time series for mytimeseries below. Do note here that series are by row, which is not exactly standard. For series by column, just change the margin from 1 to 2 in the call to adply below.

## box-and-whisker plot of FFT power spectrum by frequency/period
## vary these parameters -- r as per logistic map
## see http://en.wikipedia.org/wiki/Logistic_map for details
logmap = function(n, r, x0=0.5) { 
    ret = rep(x0,n)
    for (ii in 2:n) {
        ret[ii] = r*ret[ii-1]*(1-ret[ii-1])
    }
    return(ret)
}

## modify these for interesting behavior differences
rfrom = 3.4
rto = 3.7
rsteps=200
seqlen=1e4

require(plyr)
require(ggplot2)

mytimeseries = aaply(seq(from=rfrom, to=rto, length.out=rsteps), 1,
function(x) {
       logmap(seqlen, x)
})

## you can plug in any array here for mytimeseries
## each row is a timeseries
## for series by column, change the margin from 1 to 2, below
logspec = adply( mytimeseries, 1, function(x) {
       ## change spec.pgram parameters as per goals
       ret = spectrum(x, taper=0, demean=T, pad=0, fast=T, plot=F)
       return( data.frame(freq=ret$freq, spec=ret$spec))
})

## boxplot stats at each unique frequency
logspec.bw = ddply(logspec, 'freq', function(x) {
           ret = boxplot.stats(log10(x$spec));
           ## sometimes $out is empty, bind NA to keep dimensions correct
           return(data.frame(t(10^ret$stats), out=c(NA,10^ret$out)))
})

## plot -- outliers are dots
## be careful with spacing of hard returns -- ggplot has odd rules
## as close to default "spectrum" plot as possible
ggplot(logspec.bw, aes(x=1/(freq)))  +  geom_ribbon(aes(ymin=X1,
ymax=X5), alpha=0.35, fill='green')  +
       geom_ribbon(aes(ymin=X2, ymax=X4), alpha=0.5, fill='blue') +
geom_line(aes(y=X3))  +
       geom_point(aes(y=out), color='darkred') +
       scale_x_continuous( trans=c('inverse'), name='Period')  +
       scale_y_continuous(name='Power', trans='log10')

Following my nose further on the logistic map example, I also used the animate package to make a movie that walks through the power spectrum of the map for a range of r values. Maybe one day I'll set this to music and post it to Youtube! Though I think some strategic speeding up and slowing down in important parameter regions is warranted for a truly epic tour of the logistic map's power spectrum.

require(animation)
saveVideo({
       ani.options(interval = 1/5, ani.type='png', ani.width=1600, ani.height=1200)
        for (ii in c(seq(from=3.2, to=3.5, length.out=200), seq(from=3.5, to=4, length.out=1200))) {
            spectrum(logmap(1e4, ii), taper=0, demean=T, pad=0, fast=T, sub=round(ii, 3))
        }
},  video.name = "logmap2.mp4", other.opts = "-b 600k")

Bat Country

2011-10-06T16:49:00.001-07:00

I've spent a lot of time thinking about and using R's spectrum() function and the Fast Fourier Transform (FFT) in the last 5+ years. Lately, they've begun to remind me a little of a Theremin: simple to use, difficult to master.

While prepping a figure for the R Graph Gallery (which is awesome, by the way), I ran across this curious example -- its striking visual appearance was definitely not what I was expecting.

I decided to use the logistic map to generate an ensemble of time series with a range of frequencies. The logistic map goes through characteristic period doubling, and then exhibits broad-spectrum "noise" at the onset of chaos. And it's really easy to compute. So, I can tune my time series to have a range of frequency distributions.

In this example, I'm in the periodic regime (r=3.5, period 4) . So why am I getting this Batman motif?

Actually, it's weirdly simple. This is an artifact of the (default) tapering that R's spectrum() function applies to the series before the FFT is computed. In theory, the Fourier Transform assumes an infinite-length sequence, while in practice the FFT assumes the series is circular -- in essence, gluing the beginning and end of the series together to make a loop. If the series is not perfectly periodic, though, this gluing introduces a sharp discontinuity. Tapering is typically applied to reduce or control this discontinuity, so that both ends gently decline to zero. When the series is genuinely periodic, though, this does some weird effects. In essence, the taper transfers power from the fundamental frequencies in a curious way. Note that the fundamental period is 4, as seen by the peak at frequency 1/4, with a strong harmonic at period 2, frequency 1/2.

Zero-padding has a similar effect. If your series is zero (or relatively small) at both ends, then you could glue it together without introducing discontinuities, but peaks near the beginning and the end will still be seen as near by, resulting in spurious peaks. In any case, R pads your series by default (fast=T) unless the length of the series is highly composite (i.e. it can be factored into many small divisors). Below, it turns out that a series of length 1e5 is composite enough that R doesn't pad it. 1e5? Totally different results.

If I understand this correctly, both padding and tapering are attempts to reduce or control spectral leakage. We've constructed an example where no spectral leakage is expected from the original time series, which closely fits the assumptions of the FFT without modification.

The moral? Know thy functions (and their default parameter values, and their effects)!

## using R -- 
## logistic map function:
logmap = function(n, r, x0=0.5) { 
    ret = rep(x0,n)
    for (ii in 2:n) { 
        ret[ii] = r*ret[ii-1]*(1-ret[ii-1])
    }
    return(ret)
}

## compare:
spectrum(logmap(1e5, 3.5), main='R tapers to 0.1 by default')
spectrum(logmap(1e5, 3.5), taper=0, main='No taper')
spectrum(logmap(1e5+1, 3.5), taper=0, main='Non-composite length, fast=T pads by default')
spectrum(logmap(1e5, 3.5), taper=0.5, sub='Lots of padding approximately recovers "correct" results')

Efficient loops in R -- the complexity versus speed trade-off

2011-06-18T20:49:00.013-07:00

I've written before about the up- and downsides of the plyr package -- I love it's simplicity, but it can't be mindlessly applied, no pun intended. This week, I started building a agent-based model for a large population, and I figured I'd use something like a binomial per-timestep birth-death process for between-agent connections.

My ballpark estimate was 1e6 agents using a weekly timestep for about 50 years. This is a stochastic model, so I'd like to replicate it at least 100 times. So, I'll need at least 50*50*100 = 250,000 steps. I figured I'd be happy if I could get my step runtimes down to ~1 second -- dividing the 100 runs over 4 cores, this would give a total runtime of ~17.5 hours. Not short, but not a problem.

At first, I was disheartened to see the runtime of my prototype step function stretching into the minutes. What's going on? Well, I'd used plyr in a very inappropriate way -- for a very large loop. I began to investigate, and discovered that writing an aesthetically unappealing while gave me a 30+x speed-up.

All of which got me thinking -- how expensive are loops and function calls in R? Next week I'm leading a tutorial in R and C++ using the wonderful Rcpp and inline packages here at Santa Fe Institute's 2011 Complex Systems Summer School. Might this make a nice example?

It does, and in spades. Below are the test functions, and you can see that the code complexity increases somewhat from one to the other, but outcomes are identical, again with 30+x speedup for each subsequent case. Here, I'm using the native R API. I also tested using Rcpp to import rbinom(), but that ended up taking twice as long as the naive while loop.

So, the moral of the story seems to be that if you can write a long loop in pure C++, it's a really easy win.

Note -- The as< double >(y); in src below doesn't seem to copy-and-paste correctly for some reason. If testfun2 doesn't compile, check to make sure this bit pasted correctly.

The pure R function definitions

## use aaply -- the simplest code
require(plyr)
testfun0 <- function(x, y)  aaply(x, 1, function(x) rbinom(1, x, y)

## rewrite the above as an explicit loop
testfun1 = function(nrep, x, y) {
    ## loop index
    ii<-0;
    ## result vector
    ret<-rep(0, nrep);
    while(ii < nrep) {
        ii<-ii+1;
        ## how many successes for each element of bb?
        ret[ii] <- rbinom(1, x[ii], y)
    }
    return(ret)
}

Rcpp function definitions (with lots of comments)

## define source code as string, pass to inline
src <-  ' 
    // Cast input parameters to correct types
    // clone prevents direct modification of the input object
    IntegerVector tmp(clone(x));
    double rate = as< double >(y); 
    // IntegerVector inherits from STL vector, so we get standard STL methods
    int tmpsize = tmp.size(); 
    // initialize RNG from R set.seed() call
    RNGScope scope; 
    // loop
    for (int ii =0; ii < tmpsize; ii++) {
        // Rf_rbinom is in the R api
        // For more details, See Rcpp-quickref.pdf, "Random Functions"
        // also, Rcpp provides "safe" accessors via, e.g., tmp(ii)
        tmp(ii) = Rf_rbinom(tmp(ii), rate);
    };
    // tmp points to a native R object, so we can return it as-is
    // if we wanted to return, e.g., the double rate, use:
    // return wrap(rate)
    return tmp;
'

require(inline)
## compile the function, inspect the process with verbose=T
testfun2 = cxxfunction(signature(x='integer', y='numeric'), src, plugin='Rcpp', verbose=T)

## timing function
ps <- function(x) print(system.time(x))

The tests

## Input vector
bb <- rbinom(1e5, 20, 0.5)
## test each case for 2 different loop lengths 
for( nrep in c(1e4, 1e5)){
    ## initialize RNG each time with the same seed
    ## plyr
    set.seed(20); ps(cc0<- testfun0(nrep[1:nrep], bb, 0.1))
    ## explicit while loop
    set.seed(20); ps(cc1<- testfun1(nrep, bb, 0.1))
    ## Rcpp
    set.seed(20); ps(cc2<- testfun2(bb[1:nrep], 0.1))
    print(all.equal(cc1, cc2))
}


## output
   user  system elapsed 
  3.392   0.020   3.417 
   user  system elapsed 
  0.116   0.000   0.119 
   user  system elapsed 
  0.000   0.000   0.002 
[1] TRUE
   user  system elapsed 
 37.534   0.064  37.601 
   user  system elapsed 
  1.228   0.000   1.230 
   user  system elapsed 
  0.020   0.000   0.021 
[1] TRUE

Postlude

After posting this to the very friendly Rcpp-devel mailing list, I got an in-depth reply from Douglas Bates pointing out that, in this case, the performance of vanilla apply() beats the while loop by a narrow margin. He also gives an interesting example of how to use an STL template and an iterator to achieve the same result. I admit that templates are still near-magic to me, and for now I prefer the clarity of the above. Still, if you're curious, this should whet your appetite for some of the wonders of Rcpp.

## Using Rcpp, inline and STL algorithms and containers
 
 ## use the std::binary_function templated struct
inc <-  '
  struct Rb : std::binary_function< double, double, double > {
     double operator() (double size, double prob) const { return ::Rf_rbinom(size, prob); }
  };
'

## define source code as a character vector, pass to inline
 src <-  ' 
     NumericVector sz(x);
     RNGScope scope;
 
     return NumericVector::import_transform(sz.begin(), sz.end(), std::bind2nd(Rb(), as< double >(y)));
'
 
## compile the function, inspect the process with verbose=TRUE
f5 <- cxxfunction(signature(x='numeric', y='numeric'), src, 'Rcpp', inc, verbose=TRUE)

The importance of being unoriginal (and befriending google)

2011-06-11T19:29:00.000-07:00

In search of bin counts

I look at histograms and density functions of my data in R on a regular basis. I have some idea of the algorithms behind these, but I've never had any reason to go under the hood until now. Lately, I've been looking using the bin counts for things like Shannon entropy ( in the very nice entropy package. I figured that binning and counting data would either be supported via a native, dedicated R package, or quite simple to code. Not finding the former ( base graphics hist() uses .Call("bincounts"), which appears undocumented and has a boatload of arguments ), I naively failed to search for a package and coded up the following.

myhist = function(x, dig=3)  {
    x=trunc(x, digits=dig);
    ## x=round(x, digits=dig);
    aa = bb = seq(0,1,1/10^dig);
    for (ii in 1:length(aa)) {
        aa[ii] = sum(x==aa[ii])
    };
    return(cbind(bin=bb, dens=aa/length(x)))
}


## random variates
test = sort(runif(1e4))
get1 = myhist(test)

Trouble in paradise

Truncate the data to a specified precision, and count how many are in each bin. Well, first I tried round(x) instead of trunc(x), which sorta makes sense but gives results that I still don't understand. On the other hand, trunc(x) doesn't take a digits argument? WTF? Of course, I could use sprintf(x) to make a character of known precision and convert back to numeric, but string-handling is waaaaaay too much computational overhead. Like towing a kid's red wagon with a landrover...

Dear Google...

An hour of irritation and confusion later, I ask google and, small wonder, the second search result links to the ash package that contains said tool. And it runs somewhere between 100 and 1,000 times faster. It doesn't return the bin boundaries by default, but it's good enough for a quick-and-dirty empirical probability mass distribution.

To be fair, there's something to be said for cooking up a simple solution to a simple problem, and then realizing that, for one reason or another, the problem isn't quite as simple as one first thought. On the other hand, sometimes we just want answers. When that's the case, asking google is a pretty good bet.

## their method
require(ash)
get2 = bin1(test, c(0,1), 1e3+1)$nc

Problems with plyr -- the memory/complexity trade-off

2011-05-10T03:42:00.000-07:00

Two types of R users

My overwhelming impression from UseR 2010 is that, generally speaking, there are 2 types of regular R users -- those who have heard and are made uncomfortable by the idea of the *apply() functions, and those who really get it. In UNM R programming group that I've been leading for about a year now, I've really tried to get people over the hump and into the second group. Once there, folks seem to really appreciate the amazing power of vectorization in R, and begin to enjoy writing code. The conceptual clarity of:

  mymatrix = matrix(1:10, nrow=10)
  apply(mymatrix, 1, sum)
  apply(mymatrix, 2, sum)

over the clunky:

  rowSums(mymatrix)
  colSums(mymatrix)

may not be immediately apparent. Eventually, though, folks go searching for things like rowMedian() and become frustrated that R doesn't "have all the functions" that they need. Well, R does, once you grok apply().

Hadley's Magic Brainchild

In the last year, I've had some serious ahah! moments with plyr. Lately, I've added the reshape package to the mix to achieve some serious R 1-liner Zen. Multidimensional arrays to long-form dataframes?

  myarray = array(0, dim=c(3,4,5,6,10), dimnames=list(a=1:3, b=1:4, c=1:5, d=letters[1:6], e=LETTERS[1:10]))
  melt( adply(myarray, 1:4, I))

Sure. No problem. It's really that easy?!

An unexpeceted bonus? This way of thinking lends itself nicely to thinking of explicit parallelism. If you can describe the problem as a single recipe that's done for each "piece" of a whole, then you're one step away from using all the cores on your machine to solve a problem with foreach. Need to apply some sort of long-running analysis to each element of a list, and want to return the results as a list? Why write a for loop when you can do:

## prep
## warning -- this  may not be fast
## use smaller matrices for smaller machines! 
  nn = 2^20
  rr = 2^10
  mylist = list(a=matrix(rnorm(nn), nrow=rr), b=matrix(rnorm(nn), nrow=rr), c=matrix(rnorm(nn), nrow=rr))

## analysis
  myeigs = foreach( ii = iter(mylist), .combine=c) %do% { print('ping'); return(list(eigen(ii)))}

and then, once it works, change %do% to %dopar%, add the following, and you're off to the races!

  require(multicore)
  require(doMC)
  ## use the appropriate # of cores, of course
  registerDoMC(cores=4)
  myeigs1 = foreach( ii = iter(mylist), .combine=c) %dopar% { print('ping'); return(list(eigen(ii)))}

Compared to something like llply, your dimnames don't automatically propagate to the results, but I think this is still pretty amazing. Complete control and simple parallelism.

Debugging with %dopar% is tricky, of course, because there are separate stacks for each item (i think), and messages (such as calls to print()) don't return to the console as you might expect them to. So, when in doubt, back off to %do%.

What could possibly go wrong?

The only problem with all of this is, when tasks are embarassingly parallel, that data also becomes embarassingly parallel to point where it no longer fits into memory. Thus, I returned today to a long-running bootstrap computation to find R consuming ~7GB RAM, 5+ GB swap, and this message:

Error: cannot allocate vector of size 818.6 Mb
Enter a frame number, or 0 to exit   
...
4: mk.bootrep(zz, 75)
5: mk.boot.R#165: aaply(tmpreps, .margins = 1, function(x) {
6: laply(.data = pieces, .fun = .fun, ..., .progress = .progress, .drop = .dro
7: list_to_array(res, attr(.data, "split_labels"), .drop)
8: unname(unlist(res))

What's happening is that plyr is trying to do everything at once. As anyone who's used grep can tell you, doing one row at a time, or streaming data is often a much better idea. I got the intended results from above by pre-allocating an array and writing each item of my results list into the array in a loop in seconds, and barely broke 3 GB of RAM usage.

Now, nothing here is really news. The dangers of "Growing Objects" is covered in Circle 2 of Burns Statistics' wonderful R Inferno. Still, plyr strikes me as an interesting case where reducing conceptual complexity can lead to a rather steep increase in computational complexity. And the most interesting thing of all is that it happens quite suddenly above a certain threshold.

Parting thoughts

I wonder if there's any major barrier to a stream=TRUE argument to the plyr functions -- I haven't thought about it too much, but imagine that you'd also need a finalizer function to prepare the return object to be written/streamed into. At what point is it easier to just do by hand with a for loop?

Honestly, I don't know the answer. I don't do too many things that break plyr, but I've learned how important it is to understand when I'm likely exceed its limits. .variables in ddply is another one that I've learned to be careful with. If, after subdividing your input data.frame, you end up with 1e5 or 1e6 pieces, things start to break down pretty fast.

Honestly, I love writing solutions with the *apply() family and the ddply functions. I think it makes code cleaner and more logical. Watching the light of this dawn in other R users' eyes is truly exciting. Yet it pays to remember that, like all things, it has its limits. In this case, one seems to reach the limits suddenly and harshly.

Why PKI matters.

2011-03-29T04:16:00.000-07:00

This article landed in my inbox this week in a newsletter from the EFF. I usually don't read them, but the term "meltdown" caught my eye, what with all the nuke new this month. They also managed to work in "too big to fail", and neither reference was hyperbolic. The internet depends on a level of trust, and (surprise) there are people working to co-opt that trust.

A number of my friends think I'm a little weird for using real passwords, for not sharing them, etc. But I read Cryptonomicon and friends; I have at least one box exposed to the real, honest-to-god, jungle-out-there internet; I have a healthy fear of all the things that could go wrong. As part of my lab group's data entry project, I recently registered my first SSL certificate from http://www.startssl.com/ (free!), and learned quite a bit about PKI in the process.

Still, reading an article like this really drives home both the complexities and importance of the global PKI system. Trust is difficult when it's strung across the globe on fiber optics cables, and enforced by our inability to quickly factor very large numbers. But old-school techniques of impersonation and breaking and entering will always be with us. I may trust google.com, but how do I know that it's actually them? PKI.

How cool is that?
Very.

Looking at the "Curse of Dimensionality" with R, foreach, and lattice

2011-03-20T22:52:00.000-07:00

Here are the results of a "Curse of Dimensionality" homework assignment for Terran Lane's Introduction to Machine Learning class. Pretty pictures, interesting results, and a good exercise in explicit parallelism with R.

It's neat to see distance scaling linearly with standard deviation, and linearly with the Lth-root for an L-norm metric (e.g. Euclidean = L2 norm). It took me a while to "get" how the maximum metric worked -- it's simpler than I was trying to make it, though I'm still not sure I can explain it properly.

In general, I'm a big fan of R's lattice package, though I'm never quite happy with the legend/key settings.

The code is below. One thing that I've learned from this exercise is that I prefer not to have the foreach call in the outermost loop. For one, error and/or progress reporting from inside a foreach isn't immediately emitted to the console, and it's annoying/confusing to have a long run with no feedback. Second and less aesthetically, it seems that foreach excels when it has jobs of medium complexity iterated over a medium number of elements. Too few elements + very complex job means one or more cores will finish early and sit idle. Too many elements + very simple job means too many jobs will be spawned, and (I'm guessing here) a lot of time will be wasted with job setup and teardown.

dimcurse = function(dims=1:1e3, .sd=c(1e-2, 0.1, .2, .5, 1), reps=1e3, cores=4,
                    rfun=rnorm, .methods=c('manhattan', 'euclidean', 'maximum'), .verbose=F){
    ## examine distance between 2 points in dims dimensions for different metrics
    ## chosen at random according to rfun (rnorm by default)
    ##
    require(foreach)
    require(doMC)
    registerDoMC
    registerDoMC(cores=cores)
    ## use multicore/foreach on the outside (each sd)
    finret = foreach(ss=iter(.sd), .combine=rbind, .verbose=.verbose) %dopar% {
        ## pre-allocate, don't append/rbind!
        ret = data.frame(.sd=1:(reps*length(dims))*length(methods), .dim=1, .rep=1, .dist=1, method=factor(NA, levels=.methods))
        ii = 1
        for (.dim in dims) {
          for (method in .methods) {
            ## reset .rep counter
            .rep = 1
            while (.rep <= reps) {
                ## 2 points, one per row, randomly chosen
                tmp = matrix(rfun(2*.dim, sd=ss), nrow=2)
                ## calc dist
                mydist = as.numeric(dist(tmp, method=method))
                ## fill return dataframe
                ret[ii, ] = list(ss, .dim, .rep, mydist, method)
                ## update rep counter
                .rep = .rep +1
                ## update ret row counter
                ii = ii +1
            }
          }
        }
        return(ret)
    } ## rbind happens here
    return(finret)
}

For plotting, I've used:

xyplot(.dist ~ (.dim)|method, groups=.sd, data =aa, pch='.',
    par.settings=list(background=list(col='darkgrey')), layout=c(3,1),
    scales=list(y=list(relation='free'), alternating=1),
    auto.key=list(space='bottom', columns=5, points=F, lines=T),
    xlab='Dimension', ylab='Distance', main='Distance between 2 N-dimensional points \n drawn from a normal distribution with varying sd (legend) using different metrics (columns)',
)

xyplot(.dist ~ (.dim)|as.character(.sd), groups=method, data =aa, pch='.',
    par.settings=list(background=list(col='darkgrey')), layout=c(5,1),
    scales=list(y=list(relation='free'), alternating=1),
    auto.key=list(space='bottom', columns=3, points=F, lines=T),
    xlab='Dimension', ylab='Distance', main='Distance between 2 N-dimensional points \n drawn from a normal distribution with varying SD (columns) using different metrics (legend)',
)

snow and ssh -- secure inter-machine parallelism with R

2011-02-24T21:41:00.007-07:00

I just threw a post up on Revolutions, which got a lot longer than I planned. And got me thinking. And reading (see refs in previous post). And trying. Turns out that it was way easier than I thought!

The problem:

From the blog post:
"
OpenSSH is now available on all platforms. A sensible solution is to have *one* cluster type -- ssh. Let ssh handle inter-computer connection and proper forwarding of ports, and for each machine to connect to a local port that's been forwarded.

... It seems like most of the heavy lifting is done by foreach, and that engineering a simple makeSSHcluster and doSSH to forward ports, start slaves, and open socket connections should be tractable.

... the SSHcluster method would require minimal invasion on those machines -- just ability to execute ssh and Rscript on the remote machines -- not even login privileges are required!
"

Requirements:

R must be installed on both machines, as well as the snow package. On the machine running the primary R session (henceforth referred to as thishost, the foreach and doSNOW packages must be installed, as well as an ssh client. On remotehost, Rscript must be installed, an ssh server must be running, and the user must have permissions to run Rscript and read the R libraries. Note that Rscript is far from a secure program to allow untrusted users to run. For the paranoid, place it in a chroot jail...

Step-by-step:

The solution here piggy-backs on existing snow functionality. All commands below are to be run from thishost, either from commandline (shown by $) or the R session in which work is to be done (shown by \>). You will need to replace remotehost with the hostname or ip address of remotehost, but localhost should be entered verbatim.

Note that the ssh connections are much easier using key-based authentication (which you were using already, right?), and a key agent so you don't have to keep typing your key password again and again. Even better, set up an .ssh/config file for host-specific aliases and configuration.

Now, onto the code!

$ ssh -f -R 10187:localhost:10187 remotehost sleep 600  ## 10 minutes to set up connection

> require(snow); require(foreach); require(doSNOW);
> cl = makeCluster(rep('localhost',3), type='SOCK', manual=T)  ## wait for 3 slaves

## Use ssh to start the slaves on remotehost and connect to the (local) forwarded port

$ for i in {1..3}; 
$  do ssh -f remotehost "/usr/lib64/R/bin/Rscript ~/R/x86_64-pc-linux-gnu-library/2.12/snow/RSOCKnode.R 
$       MASTER=localhost PORT=10187 OUT=/dev/null SNOWLIB=~/R/x86_64-pc-linux-gnu-library/2.12"; 
$ done

## R should have picked up the slaves and returned.  If not, something is wrong.
## Back up and test ssh connectivity, etc.

> registerDoSNOW(cl)

>  a <- matrix(1:4e6, nrow=4)
>  b <- t(a)
>  bb <-   foreach(b=iter(b, by='col'), .combine=cbind) %dopar%
>    (a %*% b)

Conclusions?

This is a funny example of %dopar%; it's minimal and computationally useless, which is nice for testing. Still, it allows you to directly test the communication costs of your setup by varying the size of a. In explicit parallelism, gain is proportional to (computational cost)/(communication cost), so the above example is exactly what you don't want to do.

Next up:

Write an R function that uses calls to system to set the tunnel up and spawn the slave processes, with string substitution for remotehost, etc.
Set up a second port and socketConnection to read each slave's stdout to pick up errors. Perhaps call warning() on the output? Trouble-shooting is so much easier when you can see what's happening!
Clean up the path encodings for SNOWLIB and RSOCKnode.R. R knows where it's packages live, right?
As written, a little tricky to roll multiple hosts into the cluster (including thishost). It seems possible to chose port numbers sequentially within a range, and connect each new remotehost through a new port, with slaves on thishost connecting to a separate, non-forwarded port.

Those are low-hanging fruit. It would be nice to cut out the dependency on snow and doSNOW. The doSNOW package contains a single 1-line function. On the other hand, the snow package hasn't been updated since July 2008, and a google search show up many confused users and few success stories (with the exception of this very helpful thread, which led me down this path in the first place).

+1 if you'd like to see a doSSH package.
+10 if you'd like to help!

ssh port forwarding

2011-02-24T17:52:00.004-07:00

Is it just me, or is the openssh port forwarding syntax *really* confusing? Every time I have to work through examples afresh. It's ridiculously powerful -- who can argue with key-based authentication on a user-land vpn, and it's available for all major OSes. Still, I wish I *really* understood the syntax.

I suppose practice makes perfect. I'm working on a small project now. With any luck, examples to come.

References:
Openssh FAQ
Red Hat Magazine

Postgreql Hacks -- firing a trigger "for each transaction"

2011-02-03T02:55:00.001-07:00

The problem

I spent today learning about triggers in postgresql - there are triggers "for each statement" and "for each row". I went searching for triggers "for each transaction", and found a few other folks with the same question but no compelling answer.

My use case is this - I have 2 tables, a parent (entries) and child (reports), with a one-to-many relationship. I need to add multiple reports for one entry in a single transaction, and then update the associated entry if the transaction succeeds. Here are the table definitions:

create schema transtest;
set searchpath to transtest,public;
    -- it's nice to have a sandbox
    -- remember to reset searchpath when done

create table myents ( id serial primary key, sum integer);
    -- the entries table
create table myrecs ( id serial primary key, entid integer references myents (id), cases integer);
    -- the reports table, many reports per entry
create table myids ( id integer primary key references myents (id));
    -- table used to store trigger state

The solution

It appears that deferred constraint triggers are *almost* "per transaction". But constraint triggers are *always* for each row. Yet Postgresql 9.0 allows conditions on triggers - so all I need is to store some state of whether to fire the trigger or not for each row. First, the function that I'll be using in the condition:

-- function for when-clause (condition) in trigger
-- add id to the "to-do" list if it's not there already
-- return True only if it's been added to the least
create or replace function f_condition(myid integer) returns boolean as $_$
    DECLARE
        myret boolean := True;
        newid integer;
    BEGIN
        RAISE NOTICE 'Return Value here is %', myret;
        select count(*) into newid from myids where id = myid;
        if newid > 0 then -- id is there, return False 
           myret := False; 
        else    -- add id to queue, return True
            insert into myids (id) values (myid);
        end if;
        return myret;
    END;
$_$ language plpgsql;

Next, the trigger function, which will clean up after the above function when it's done:

-- trigger function to rebuild myent.sum for this myrecs.id
create or replace function f_refresh() returns trigger as $_$
    DECLARE
        mysum integer;
        myid integer;
    BEGIN
        IF (TG_OP = 'DELETE') THEN
            myid := OLD.entid;
        ELSE 
            myid := NEW.entid;
        END IF;
        RAISE NOTICE 'Return Value here is %', myid;
        select sum(cases) into mysum from myrecs where entid = myid;
            -- compute entry sum from records and update it.
            -- PL/pgSQL rules on queries are quirky.  Read the docs.
        update myents set sum = mysum where id = myid;
        delete from myids where id = myid;
            -- clean up
    RETURN NULL;
    END;
$_$ language plpgsql;

Finally, the trigger definitions using the above.

-- only fire trigger once per myent.id using when clause (condition)
-- eval is deferred until transaction concludes
create constraint trigger t_cache after insert or update 
    on myrecs initially deferred for each row 
    when ( f_condition(NEW.entid) )  -- comment out for pre pg 9.0
    execute procedure f_refresh();

-- need 2, one for each for NEW (insert and update) and OLD (delete)
create constraint trigger t_cache_del after delete
    on myrecs initially deferred for each row 
    when ( f_condition(OLD.entid) )
    execute procedure f_refresh();

The test

All software needs to be tester, right?
I don't have pg9.0 yet, and it won't hit Ubuntu mainline until Ubuntu Natty, but it looks easy enough to install via a backports repo. So, this isn't fully tested. Comment out the "when" lines in the trigger defines and it will work, albeit running once per row *after* the commit.

In any event, here goes! The sum should be zero until after the commit. If the when clause works correctly, then the trigger should only fire (and emit a notice) once. Likewise, the myids table should be empty.

insert into myents (sum) VALUES ( 0);
begin;
    insert into myrecs (entid, cases) VALUES ( 1, 0), (1, 1), (1,2);
    select * from myents;  -- 0
    select * from myids;   -- entid from above
commit;
select * from myents;   -- new sum
select * from myids;   -- empty

References -- Postgresql Docs:

2011-02-02T03:15:00.002-07:00

I've been obsessively following Egypt for the past few days. Both the BBC and Al Jazeera have excellent coverage, as well as Google Realtime. After days of seeing BBC's "Have your say" links on the BBC ( and reading tech and business stories to boot ), I sent one in. Here goes: ### Begin Comments The latest comments from both Mubarak and Obama indicate that change is on the horizon. How close the horizon actually is seems to be the biggest question on people's minds. Several days ago, in the space of minutes, some 80 million people vanished from the internet (see http://www.renesys.com/blog/). This likely happened at direct orders from the current regime - who else wields such power? This is one of the most significant "black marks" against the current regime - a totalitarian attempt to curtail information flow. A resumption of basic information services is an absolutely prerequisite to any transition. If the current regime is serious about stability, they will "turn the lights back on" ASAP. The fact that they haven't done so yet casts serious doubt on their intentions.

User Interface Zen

2010-06-06T20:32:00.003-07:00

Before Apple made pretty screens on cellphonesand album art that you could flip through, there was Synaptics. Have you ever seen someone drawing on a pad bound to a laptop in a coffeeshop? Synaptics. Have them you ever done a 2-fingered scroll? Synaptics. I was a perennial hater of laptop touchpads. Imprecise, too much work. A mouse gives wide sweeps, high precision. Then i slowly worked myself up to almost-full-time keyboard usage, a long barrage of shortcuts that i could fire off without ever lifting my fingers, frame after frame of text editor, browser, google scholar, google maps, and back. I've always really *liked* keyboards, and now i absolutely depend upon them. Re-enter the 2.2 pound netbook. That's right - the 1kg laptop. Alright, it has an annoyingly small battery compared to a very slightly larger 2.8 pound laptop, and the weight gain is more or less obviated by the more frequent need to carry a 0.3 pound charger. Whatever. It still lasts for twice as long as my "real" laptop (a thinkpad t61 that i would *never* put in my lap, whose screen and keyboard i nonetheless adore ). Point is... The touchpad has come a long way. So far, in fact, that touchpads are now officially cool, as far as i can tell. With 2 fingers, i can now give a *lot* of information, while moving my fingers less than a mouse. The secret, from my point of view, is synaptics, and more specifically, synclient. Under ubuntu, the following sets thresholds of two finger motions, adds rules for "two finger tap as middle click" (Xwindows paste), and disables end scrolling (otherwise known as the devil). I'm using an Asus 1005HAB (N270, $250 @ Best Buy with WinXP, after some deliberation, and some pretty aweful "the customer is a retard" corporate customer service. As an extended sidenote, do yourself a favor and remove the "rescue partition" before it wipes your computer clean with a random, unconfirmed hard drive repartition when you accidentally boot into it with grub). It's the model with the little bumps on a touchpad that is flush with the case. I'm surprised at how not-annoying the bumps are. After turning up the sensitivity, it's, um, magic? I think "scroll down" and it suddenly happens. The wonders of multitouch. The happiness of the not-a-huge-tome laptop! Here's my multitouch script: #!/bin/bash ## synclient EmulateTwoFingerMinZ=5 synclient EmulateTwoFingerMinW=10 synclient VertTwoFingerScroll=1 synclient HorizTwoFingerScroll=1 synclient VertScrollDelta=75 synclient HorizScrollDelta=100 synclient JumpyCursorThreshold=100 synclient VertEdgeScroll=0 synclient HorizEdgeScroll=0 synclient TapButton2=2 synclient TapButton3=3

Little R == r

2010-04-20T23:51:00.003-07:00

There's big R, the R that I use to do most my work, the environment that makes pretty graphics, et. al. It's like matlab, only cooler. Or more cool. Or less uncool. You can see my prejudices here. Today i discovered little R. It's like big R, only little. Holy shit. Dirk gives a thorough rundown here http://dirk.eddelbuettel.com/code/littler.html Suffice to say, for someone who's been using pipes and #!/usr/bin constructs for years (though not quite yet decades), this is cooler than cool. One might say, super-cool. It's also a nice intro to R for some of the systems geeks out there. Need a million random numbers uniformly distributed between 0 and 1, specified to 7 decimal points? Need it in a file? Need it fast? r can help: time r -q -e 'for (i in rnorm(1e6)) cat(sprintf("%1.7f\n", i))' >> randomnums Or perhaps you have a million numbers in some file that you would like to plot as a histogram, fast, every day, in an automated fashion, from the command line... cat randomnums | r -e 'myrandoms <- as.numeric(readLines()); png(filename="myplot.png"); plot(histogram(myrandoms)); dev.off()' >/dev/null No, it won't mungle strings with the ease of perl, but it can chew a spreadsheet and spit it out *fast*. And since it's a stream, you can always pipe it to/from perl. If you ask me, pretty fucking cool.

The new face of CouchSurfing

2010-04-07T20:02:00.002-07:00

Really? I've gotten a few borderline disrespectful CS requests lately (my profile is here). Sure, young and inexperienced, have mercy on them, etc. etc. Nonetheless, I think it's time for a bit of gentle schooling... The Request >USERNAME: *** >GENDER: Male >AGE: 23 >LOCATION: United States - New York - *** > >ARRIVAL DATE: 4/9/10 >DEPARTURE DATE: 4/13/10 >NUMBER OF PEOPLE: 1 >ARRIVING VIA: Plane > >Hi, > >My name is *** ***. I'm going to Albuquerque this weekend to attend a seminar at the **** Institute. I know it's short notice, but I was wondering if you would be able to host. It's a lot of last minute planning, I just created this account because my girlfriend suggested I use couchsurfing to try to find a place to stay, so sorry that my profile isn't very extensive. > >Thanks! > >*** My Reply First rule of a successful couchsearch - read the profile of the person that you're addressing. Second rule of a successful couchsearch - tell the person what specifically about them you think would make you a good match. what do you bring to the table? What about them interests you? Third rule of a successful couchsearch - add relevant information and a picture or 2 to your profile. A couchsearch is asking someone to take time out of a busy life to provide you with a place to stay. It makes sense to reciprocate the effort by spending some time to write out a real, decent profile that tells your prospective host something about who you are. Good luck! christian

Postgres + PL/R = magic information swiss army knife

2010-01-13T03:44:00.006-07:00

I just wrote up an extensive example using PL/R to build a logistic map for a range of values of r. The end result is a pretty picture generated within the database and handed back to the webserver. Very cool. For complete details, see http://www.joeconway.com/web/guest/pl/r/-/wiki/Main/Bytea+Graphing+Example

thesis take 7 part 2

2009-10-27T02:37:00.001-07:00

human agriculture, which is central to human ecological dominance, in fact relies upon ecosystem disturbance.

as humans stabilize niches against disturbance, characteristic changes in species abundance and guild specialization may be expected. using two model guild strategies for exploiting disturbance (fire and flood), i aim to characterize the effect of disturbance supression. negotiation of abiotic and intra-guild constraints are the primary mechanisms of ecological prosperity that are considered.

thesis topic take 7

2009-10-27T02:28:00.001-07:00

environmental disturbance is a key ecological determinant of species composition. long lives and lack of motility in woody plants can lead to characteristic relationships to disturbance, either through close life-cycle coupling to dominant abiotic disturbance (i.e. flooding), or through biotic forcing of disturbance (i.e. fire).
these relationships to disturbance are mediated by nutrient and organism cycling, yielding causal, fitness-driven mechanisms of species abundance in woody plants. woody plants provide a fundamental, irreplacable ecosystem service of energy production. driven by intra-guild competition and abiotic limitations such as water, nitrogen, and phosporus, many woody plants have adapted to exploit a priori disturbance, or to drive new disturbance.

contrary to contemporary memes, many terrestrial ecosystems have been subject forceful stabilization as a result of human activity. humans supression of fire and flooding has ecological consequences, especially for woody plan

biking and pixing and blogging

2009-10-24T17:34:00.001-07:00

dont try this at home, kids. weird downtown industrial scene, not much traffic.

bonus pic - tincture making

2009-10-16T01:47:00.001-07:00

this is percolation, michael moore (the herbalist) style, easily the most irritating method imaginable, but the man is right. it works. well. this is a ridiculous amount of coltsfoot. if you got lung issues, lemme know...

the herb is soaked for some time, and then packed into the cut bottle with a coffee filter below and above, not too tight, not too loose, then alcohol is poured on top, and it seeps through and carries the goodness away with it. the cap on the bottom is screwed just so, such that one or two drops a second leave one bottle asnd enter the other.

this particular batch is pretty bitter. the color tells me that it more-or-less worked as advertised. one outstanding question is whether "waste" of alcohol or plant matter by using excess of each is preferable. tincture-making is a fine line between the two. depends on the plant (and the booze), i suppose...

baseline

2009-10-16T01:38:00.001-07:00

the sg of water should be 1.0000000. it's close enough. sanitizer water in the pot in the background. the decidedly unsexy part of brewing - keeping everything clean. ive got some strange strains living down here...

it floats!

2009-10-16T01:33:00.001-07:00

here's the sweet-and-sour water getting tested with the hydrometer in the test tube. the higher the hydrom floats, the denser the liquid, the more sugar is dissolved in it... specific gravity, s.g. = 1.15... that's a *lot* of sugar. my fingres are sticky nasty now.

my new favorite toy

2009-10-16T01:27:00.001-07:00

it's a laser thermometer - some real star trek shit! im cooling the honey water in a sink of cold water. it's not perfect, but pointing the magic yellow thing at the surface and pressing the button gives me the temp within a few degrees. 90 degF is definitely too high for the yeasties.

scales how i love thee

2009-10-16T01:20:00.001-07:00

measuring out acid blend - 2.5 g tartaric, 4 g malic acid, conservative for 2+ gallons, but more can always be added later. the acid is tasty in it's own right, but it gives the yeasties a leg-up during the ferment.

more sugar == more alcohol

2009-10-16T00:50:00.000-07:00

some extra honey that was crystallized (~500g) and some priming sugar (corn, 140g) into a quart of water). need ~12% alcohol for it to be delf preserving. honey makes tasty brew, in any case. this is the last of my *easy* fermentables. time to buy a new bag of sugar...

This message has been sent using the picture and Video service from Verizon Wireless!

To learn how you can snap pictures and capture videos with your wireless phone visit www.verizonwireless.com/picture.

Note: To play video messages sent to email, Quicktime@ 6.5 or higher is required.

prickly pear wine from the student ghetto

2009-10-16T00:46:00.001-07:00

1 gallon fresh prickly pear juice, 1 gallon water, 1kg sugar (good stuff, not the ghetto smiths bleached shit), a pinch of vitamin c (1.5g), a few dashes of diammonium phosphate (8g), some yeast energizer (1g), all cooled to 25 degC. bubbled in oxygen for 2 minutes. specific gravity is 1.065... potential alcohol @ 9%, way too low. must add more sugar.

This message has been sent using the picture and Video service from Verizon Wireless!

To learn how you can snap pictures and capture videos with your wireless phone visit www.verizonwireless.com/picture.

Note: To play video messages sent to email, Quicktime@ 6.5 or higher is required.

of a few things large and many things small

2009-08-30T01:26:00.001-07:00

reading Robinson's Red Mars reminds me of a basic and oft-under (or un-) appreciated principle of ecology: change in the ecospere tends to flow from small to large, and not vise versa.

sure, planets have colided. asteroids caused massive turnover in earth species composition. nonetheless, where did life come from? the smallest of units, testing a galaxy of possibilities, some succeeding, some aggregating.

the little things. we dream grand dreams of galaxies, of control. seen from a plane, humans' works are not vast. none but our roads and our lights warrant note. points and lines.

ozone hole; global warming; oxygen atmosphere: consequences of small units, people and plants and refridgerators, units repeated to vast proportions over time, an inflation of individually insignificant individuals that, together over time, effects nontrivial change.

this is the law of history as well as ecology. the pyramids, slaves. democracy with it's masses, intel's chips.

small isn't trivial.

classroom economics experiments

2009-08-07T17:11:00.001-07:00

i've been thinking lately about inflation and the meaning of money lately. i think a classroom experiment with monopoply money and a real lemonade stand would be great. folks draw some sort of income (perhaps making the lemonade and selling it to the vendor), vendors get paid a fixed turn-based wage. if the resale price of lemonade rises 10% every turn, the "real value" of money immediately becomes apparent.

as an added capitalist spin, savings might be used to buy a stand or "manufacturing plant", and employees are hired from the participants. finally, adding or removing money from the system, a bank with a savings rate...

Albany NY

2009-07-23T08:29:00.001-07:00

Mo says, "an owl and a bunny went to Albany".

A view of the Hudson

2009-07-23T06:25:00.001-07:00

It's a *very* pretty ride, and we both got seats on the good (west) side of the train. It's amazing how quickly a city of millions gives way to a sparsely-populated expanse of water and trees. Sailboats are scattered picturesquely about.

Safely on the train

2009-07-23T06:18:00.001-07:00

after a half-mad rush through the subway. Morning rush hour was, thankfully, merciful. Epic confusion in Penn Station of the "where in god's name anre the amtrack trains" variety in the *dead midst* of morning rush hour. Nonetheless, success with minutes to spare! No assigned seating, and Mo and i get split, which is probably for the best at this point. We're both borderline hungover, dangerously caffeinated, and tragically sleep-deprived. But we are safe, and the train is spacious and relaxed, and the aforementioned pbr begins to sing its sweet siren song from the bag, otherwise known as "late lunch"...

A last look @ Williamsburg

2009-07-23T06:08:00.001-07:00

We eat a hurried few bites in front of our heap of bags. Very pretty morning, the air is misty and thick. We're told that a tropical depression is in-bound, that it will starting raining around noon and continue for days. We'll be safely on the train by then. Good timing.

The swank Williamsburg Apt

2009-07-23T05:55:00.001-07:00

Where we stayed. Very cozy, replete with romantic fire escape to sit on in the cool evening air.

Some of our luggage

2009-07-23T05:52:00.001-07:00

that we will haul on our backs through the tunnels to Penn Station. Amomng other indispensables? 9 cans of pbr. Not light.

Good morning sunshine

2009-07-23T04:23:00.001-07:00

After a few hours of sleep, we crawl towards breakdast, Penn station, and Canada.

Happy Hour

2009-07-22T16:42:00.001-07:00

We savor the peace of having not killed each other during the long, hot day. Cold Brooklyn Brewery lager and smoking on the non-smoking, zoo-cage patio is the perfect pairing with this moment of foot-blissed circumspection.

I didn't take this picture

2009-07-22T16:36:00.001-07:00

It's actually a still frame from the Matrix, a scene that ended up on the editing room floor. Mo was too fast for the cameras. From what I've heard, technology is catching up. Until then we'll have to settle for Keanu and special effects...

Rush hour

2009-07-22T16:33:00.001-07:00

to Brooklyn on the L. I got some annoyed looks for pointing a camera at folks. See previous post r.e. subway confrontation avoidance...

Rachael's Hypothesis

2009-07-22T16:29:00.001-07:00

of New Yorkers' fetish with shoes is that everyone looks down while riding the subway, and this makes shoes a dominant mode of slef-expression. I lose in this game, but my feet don't ache too bad after a long day's stroll...

Subwyas don't kill people

2009-07-22T16:26:00.001-07:00

People kill people (or themselves), as the saying goes.

Ok, i'm a big fat train dork. Sense of power, hint of danger as the beast rushes up with a gust of air, smell of electric motors and oil, the subterranean labyrinth. I'd definitely rather live here than in the Mall of America in some post-apocalyptic future? Seen "Dark Days"? If not, it will in all likelyhood forever change the way you see subway tunnels. Oh, and there's a book from the late 80s or early 90s, "Slake's something"...

This message has been sent using the picture and Video service from Verizon Wireless!

To learn how you can snap pictures and capture videos with your wireless phone visit www.verizonwireless.com/picture.

Note: To play video messages sent to email, Quicktime@ 6.5 or higher is required.

Trash

2009-07-22T16:16:00.001-07:00

I can respect a city that leaves large piles of trash lying conspicuously around before removal.

Years ago (2004?) i was wandering through Central Park, up towards Harlem i think, and i discovered the Park's composting facilities. It was a 5 year old's dream, a leaf pile the size of a castle (albeit shredded and rapidly decomposing), french drain-style aeration pipes threaded throughout. It was spring, cold still, with the last crocuses and the first daffodils blooming, and there was a great quantity of microbially-warmed air pouring forth from this mountain of leaves. It totally changed the way i saw Central Park *and* composting.

A counting clock

2009-07-22T16:10:00.001-07:00

Looking south past union station. The clock is counting something awefully fast... microseconds since Bush left office?

Not Square

2009-07-22T16:06:00.001-07:00

I have this thing for oblique-cornered buildings - fond memories from Prince Ave in Athens? ABQ is pretty square; we don't have these...

Hiding from the sun

2009-07-22T16:01:00.001-07:00

As it beat down upon us. Found an unairconditioned goodwill to try on polyester blazers in - downright swampy! Scored some good clothes for cheap, though. This is our idea of souvenirs - things we can wear, "yeah, i went shopping in downtown Manhattan last summer"... not the pinnacle of fashion, goodwill, but it will suffice for us!

Looking Uptown

2009-07-22T15:56:00.001-07:00

Big roads, big buildings. Saw a ghostbike near here. Mo was wondering how often people get killed on bikes in Manhattan. Now we know - 14 in 2008. Per capita, I'm guessing that's less than ABQ...

Credit Unions Weather Financial Storm

2009-07-22T11:45:00.001-07:00

Whether Ukrainia weathers Putin remains to be seen...

The Daylilies of Stuyvesant Square @ 15th st & Perlman

2009-07-22T11:00:00.001-07:00

We made it to Manhattan, all 3 minutes away by the L train. A break in the shade, cool breeze moving in. I haven't seen so many nice trees all in the same place in a long time...

Williamsburg Coffeehouse Bathroom

2009-07-22T10:56:00.001-07:00

Public restrooms rare - this one came with coffee, a nice window-seat view of the street, and sweet ipod hipster music that was accidentally stopped when i asked who a song was by... Looking @ music schedule in the Viilage Voice makes me want time/$$ to see shows. Cool stuff, $10. Pocket flask? $6 pints are rough!

8am train out of Penn Station tomorrow, so no late night tonight. Still jet/sleep-schedule-lagged...

Picked up a 6 pack of Dogfish Head 60min IPA last night for $13, warm 12pk of pbr for $9 on Bedford St. Cold pbr was a dollar more. Strangely enough, i saw 12pks of warm natural light for $6.50 today in RiteAid. What? Beer that fell off the back of the truck? This is Brooklyn, after all. $0.50 a beer even for the cheapest of packaged corporate swill seems a little odd.

this one's for mike d

2009-07-22T07:27:00.001-07:00

Brooklyn bug, replete with authentic art.

Alive if damp at Laguardia

2009-07-21T19:31:00.001-07:00

We're in NYC! Yay! Awesome seatmate advised us extensively. Life is good, if blearyily sleep-deprived.

This message has been sent using the picture and Video service from Verizon Wireless!

To learn how you can snap pictures and capture videos with your wireless phone visit www.verizonwireless.com/picture.

Note: To play video messages sent to email, Quicktime@ 6.5 or higher is required.

Houston Airport Smokers Lounge

2009-07-21T13:04:00.001-07:00

yes, that would be outside.

It amazes me that the city of choice for NASA mission control and a major hub airport is so perpetually overcast. Air like peanut butter, with some hot, dirty mopwater mixed in for good measure.

Continental staff is a rather moribund crew. Our 1.5 hr layover has magically extended itself an hour - its shaping up to be a long day, what with impromptu going away party and all. Still, if we get there i'll be happy.

Traveling with green machine, trying to download MTA subway maps. I forgot my credit card sized MTA map at home. Alas. I *did* remember my passport, which is still valid ('til 2016, which makes me wonder what the world will look like then...).

Cool-looking Brooklyn couchsurfers and a big bag of homemade snacks for the airport wait 'til then. Not Texas, here we come!

This message has been sent using the picture and Video service from Verizon Wireless!

To learn how you can snap pictures and capture videos with your wireless phone visit www.verizonwireless.com/picture.

Note: To play video messages sent to email, Quicktime@ 6.5 or higher is required.

thus we begin!

2009-07-21T09:57:00.001-07:00

houston to nyc to montreal and back, lord willing!

What my violin really sound (and look) like

2009-06-25T23:02:00.004-07:00

I've been trying to spend some time with the violin on a semi-regular basis. When it's more regular, it sounds less bad. I have a vague idea when i'm playing whether i'm horrible or tolerable, but listening to one's own self is a whole other thing. Today it was raining, so i had to play indoors. I hooked up rosegarden and set a cheap but remarkably effective gooseneck headset mic in the middle of the room and started playing. You don't really want to hear it, trust me. I'm trying to play more longingly minor keys because i like the sounds of them. They sound like i feel a lot of the time. Still, my fingers don't really know where they are. While i'm playing, the "music in my head" cancels out many of these mistakes before they reach my ear. In the recording i can "see" where i'm trying to go, but i hear all the mistakes quite clearly, too! I was checking on the web to see if Rosegarden has a spectogram viewer inside it when i found this wonderful tool for audio visualizion - sonic visualizer. I haven't been this impressed with audio software since, oh, rosegarden? It's currently devouring 100% of one of the cores of my CPU rendering the full 10 minute spectrograph as i write this, but it updates to the current view very quickly. It's user controls are *smooooooth*. Here's a spectrograph view of 15-odd seconds (don't know why the time axis isn't labeled here...), with frequency and amplitude scales shown on the left. I never really thought about it, but the harmonics are more widely spaced for high notes, accounting for the perceived "pureness" of the tone? You can see the vibrato, too. Pretty cool. Click on the image to see it in decent-resolution.

Lattice magic - top-to-bottom plotting order in xyplot

2009-06-19T17:29:00.010-07:00

I've been wondering about this for a while - an alternative to as.table that fills from top-to-bottom first. Working with paired timeseries, it's much easier to compare vertically, with time-axis already aligned. Deepayan's method is fast and general, and that's awesome...

From R-help: Hi, Using library(lattice), is there any way to tell xyplot to plot panels top to bottom, then left to right (i.e. panels are appended vertically, then horizontally). as.table changes the plot direction from left-to-right then top-to-bottom, to right-to-left then bottom- to-top, but that's not quite what I want to do. Thanks Yan

Deepayan says:

> tmp.tr3 <- xyplot(y ~ x | a, data=tmp, as.table=TRUE)

Another high level option is to change the rule determining how packets are chosen for a given panel in the layout.

print(tmp.tr3,  packet.panel = function(layout, row, column, ...) { 
    layout <- layout[c(2, 1, 3)]
    packet.panel.default(layout = layout, row = column, column = row, ...) 
})

This effectively transposes the layout, which (along with as.table=TRUE) is what you want.
-Deepayan

Obfuscated SQL

2009-06-15T02:13:00.006-07:00

I've finally learned to create views early and often in postgres... They're like variables (with lazy evaluation). Only different. A lot different, in ways that make as much sense to me as conjugating weird tenses in french. Here i've used multiple views stacked atop each other. It's not the fastest way, but for hundreds of thousands of records, it's fast enough. I have multiple loggers in different wells. Every well has its own datum. Every file belongs to a logger, which in turn belongs to a well, and requires the correct datum. I've had to join and union several different subqueries and views. It's a god-aweful mess, but i think i finally got it.

--here's the magic: v_sourcedat
--matches source files to datums
create or replace view v_sourcedat as
        select  unit_id||well_num as well, datum_m, gw.sn, source
                from v_loggerdat wd,
                ( SELECT tmp.sn, min(tmp."time")::date AS start,
                        tmp.source FROM tmp_gwdata tmp
                        GROUP BY tmp.source, tmp.sn)
                gw
                where gw.sn = wd.sn::int
                and gw.source not in (select source from v_sourcetowell)
        union select unit_id||well_num as well, datum_m, gwo.sn, source
                from v_loggerdat wd,
                (select * from v_sourcetowell)
                gwo
                where gwo.well = wd.unit_id||well_num
        order by well;

-- using v_sourcedat, as above
-- the final data
create or replace view v_gwdata_4hr as
        SELECT datum.well, datum.sn, gw."time",
                gw.dtw::numeric(10,4) - datum.datum_m::numeric(10,4) AS dtw,
                gw.dtwsd, gw.temp, gw.tempsd
        FROM v_sourcedat datum
        JOIN gwdata_4hr gw using (source)
        WHERE gw.temp <> 3::numeric
        AND gw.dtw > (-0.5) AND gw.dtw < 4.5
        AND gw."time" < '2010-01-01 00:00:00'::timestamp without time zone
        ORDER BY datum.well, datum.sn, gw."time";

flying standby

2009-06-12T07:27:00.001-07:00

After a long and futile day at the airport yesterday watching summer afternoon thunderstorms make a hash of my travel plans, i booked for the 6am out of ABQ.

A brief list of miraculous events proceed:
1. Megan graciously and unexpectedly offers to drop me off at the god-aweful hour of 5am.

2. My alarm fails to wake me at the even more god-aweful hour of 4:30, but a gate-change triggers a call from Southwest at 5:20am. I'm obviously late.

3. There's leftovers in BAM's fridge, and megan spots me a V8.

4. At security, i'm shuttled into the pilot/express line. I fly through security despite an irrationally long 5:40am "normal" line.

5. I don't feel like a total zombie...

6. Pressure-induced condensation vortex from turbelence in the streamline off the wing during descent braking.

7. belgian waffle with real butter

8. getting on the plane from denver to baltimore.

it's shaping up to be areally nice day. i'm sure i'll keel over eventually, in a safe place...

Postgresql Poetry? Aggregate median with PL/R

2009-06-06T18:16:00.009-07:00

No, no poetry yet, but i think i'm getting closer to thinking like a query planner. Today i learned aggregates. Have i mentioned how much i love the postgresql documentation? Unknownst to myself before today, postgres's aggregates are smart - they process as a stream when at all possible - so average keeps track of the running sum and number, and calculates the average at the end. Which makes median tricky. It depends on the length of input. I have a median implimentation in PL/Perl that queries the table directly, asking it for length, and then extracts the needed records. While this is efficient, it's much closer to magic than i'm comfortable with. Armed with PL/R, I create a median function with the proper type, and then I create the aggregate using that function as the final calc, and a simple accumulator as the transistion function. This isn't very efficient when the "group by" clause give large groups to be passed into the aggregate function, since the whole group has to be stored in memory. But for lots of small groups (say, aggregating 4 samples per hour or 24 samples per day over a year or two), it gives ~20% performance gains. Throw a "limit 5" clause in, and the performance gain increases to ~100%. Here's a link to some truly amazing stuff folks are doing with PL/R. -- sql code follows -- first create the function, then the aggregate

CREATE OR REPLACE FUNCTION median(vals numeric[]) RETURNS float AS '
median(vals)
' LANGUAGE 'plr' STRICT;

CREATE AGGREGATE median (numeric)
(
sfunc = array_append,
stype = numeric[],
initcond = '{}',
finalfunc = median
);

--And usage
select sn, median(dtwm), count(dtwm), date_trunc('day', timestamp) as day from
gwdata where timestamp < '2003-01-01' group by sn, day limit 5;   
-- sn  | median | count |          day  -----+--------+-------+------------------------  
-- 5526 |   1.59 |    24 | 2002-12-26 00:00:00-07  
-- 5539 | 1.7605 |    24 | 2002-02-27 00:00:00-07  
-- 5522 |  0.737 |    24 | 2001-10-03 00:00:00-07  
-- 5517 |   0.96 |    24 | 2001-11-05 00:00:00-07  
-- 5513 | 1.3855 |    24 | 2001-09-07 00:00:00-07  
-- Time: 247.126 ms     

-- Here's the PL/Perl median() function.  
-- Much less straightforwards, included for historical interest BEGIN { strict->import(); }
my ($tname,$cname) = @_;
my $SQL = "SELECT count($cname) AS t FROM $tname";
my $rc = spi_exec_query($SQL);
my $total = $rc->{rows}[0]{'t'};
$total < offset =" ($total-1)/2;" sql = "SELECT $cname AS median FROM $tname     ORDER BY $cname OFFSET $offset LIMIT 1" sql = "SELECT avg($cname) AS median FROM   (SELECT $cname FROM $tname    ORDER BY $cname      OFFSET $offset LIMIT 2   ) AS foo" rc =" spi_exec_query($SQL);">{rows}[0]{median};

Dear google - "a driver is not registered for the url sdbc:postgresql"

2009-06-01T22:31:00.004-07:00

I have an openoffice database project built around a postgresql database using the postgresql sdbc driver, Everything worked really well on my old machine. Why doesn't it work on the new one? Does the driver not work for openoffice 3? Is it a jaunty issue? I followed the instructions and read the f@#$*'ing manual! The error message is the title of this post - google doesn't give a fix, per se, and shows just a few hits, but reveals that it's a 32/64 bit issue. Well then. There appears to be a jaunty package, openoffice.org-sdbc-postgresql. Does that work in and of itself (after i uninstall the bad 32 bit one)? Nope. But executing ### dpkg -L openoffice.org-sdbc-postgresql | grep zip shows me where the file i need is, and executing ### unopkg add `dpkg -L openoffice.org-sdbc-postgresql | grep zip` installed it for me. Everything works - hoorah!

If only it were that simple... Enforced upgrade from feisty to jaunty

2009-05-25T19:32:00.005-07:00

Pl/R abjectly refuses to build under Intrepid (8.10). No package for it, requires postgresql source, and even then it fails. But wait! There's a Jaunty package. A good excuse to upgrade (and postpone real work for another few hours)? Can it be done over a remote ssh without knocking the box over? On the upside, Oh My Lord, the campus network is Mad-Fast. The new ubuntu install tools (do-release-upgrade) helpfully inform me: "You have to download a total of 1735M. This download will take about 3 hours 40 minutes with a 1Mbit DSL connection and about 2 days 19 hours with a 56k modem. " Um, yeah. 5 minutes later... Done http://ftp.osuosl.org jaunty/main postgresql-contrib-8.3 8.3.7-1 [81%] 9469kB/s 35s Holy mother-of-wow. I love my campus, and i highly recommend ftp.osuosl.org for West Coasters looking for a screaming mirror. Of course, unpacking takes a while. First problems arise therein: "dpkg: libjack0: dependency problems, but removing anyway as you request: stk depends on libjack0 (>= 0.109.2); however: Package libjack0 is to be removed. mplayer depends on libjack0 (>= 0.109.2); however: Package libjack0 is to be removed." NOOOOoooo! Give me back my libjack! This is a work computer, and libjack is definitely not for work, though the psychic relief of multitrack recording with rosegarden definitely helps. What gives? Total upgrade takes ~50 minutes. At last: System upgrade is complete. Restart required To finish the upgrade, a restart is required. If you select 'y' the system will be restarted. Continue [yN] y Broadcast message from xian@ocimum (/dev/pts/2) at 20:19 ... The system is going down for reboot NOW! It goes down, and doesn't come back up. Looks like i'll be driving into the lab today...

Exploits in Brute Repetition - RODBC and Postgresql and Postgis and UnixODBC and Ubuntu

2009-05-22T21:16:00.007-07:00

Database admin will make me old before my time... It's a wicked-cool toolchain, but byzantine to say the least. This is the third time i've gone through this as a fresh installation, and it sucks every time. Here i'm installing the database engine, setting up the database, adding db user, and restoring a dump. This time, I've documented it... May you never have to do this from scratch through the google-tron! #!/bin/bash ## steps to create frs database from scatch on ubuntu ## with system user as cluser superuser ## postgis setup included for package postgresql-8.3 sudo apt-get install postgresql-8.3 postgresql-8.3-postgis sudo -u postgres createuser --superuser $USER sudo -u postgres psql -c "alter role $USER encrypted password '*****'" createdb $DB createlang plpgsql $DB psql -d $DB -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql psql -d $DB -f /usr/share/postgresql-8.3-postgis/spatial_ref_sys.sql ## http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=441797 ## postgis version issues... # cd /usr/lib/postgresql/ # ln -s 8.3 8.2 # sudo ln -s liblwgeom.so.1.3 liblwgeom.so.1.2 # time psql -d frs -f db.full-2008-06-24 # psql -d frs -f /usr/share/postgresql-8.3-postgis/lwpostgis_upgrade.sql ## odbc sudo apt-get install r-cran-rodbc sudo apt-get install unixodbc psqlodbc sudo odbcinst -i -d -f /usr/share/psqlodbc/odbcinst.ini.template sudo su cat /usr/share/doc/odbc-postgresql/examples/odbc.ini.template >>~/.odbc.ini ### edit the latter - name used to call by, db, passwd, etc. ### unixODBC doesn't do ident authentication??? exit chmod 600 ~/.odbc.ini ### tune the server ### suggestions here: http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server ### and here: http://www.postgresql.org/docs/current/static/runtime-config-resource.html ### this machine has 4gb ram, services small number of concurrent connections sudo vi /etc/postgresql/8.3/main/postgresql.conf sudo sysctl -w kernel.shmmax=268435456 sudo sysctl -w kernel.shmall=4194304 sudo vi /etc/sysctl.conf ## set shmmax and shmall across reboots # pulling data from postgres into R using a custom function - not too shabby for a half a million records! # system.time((tmp=query('v_dtw', clause="limit 500000" ))) # user system elapsed # 25.502 18.277 46.285

Adventures in R Recursion

2009-05-19T19:23:00.002-07:00

Spent the day on the coffee shop patio fielding OLPC questions - from "Wow, that's cool, what is it?" to "Where can i get one?" I demonstratively poured some tea on the keyboard at one point for waterproof emphasis to "coooool"s. Good times. Installed R on the green machine (which is now runing happy-fun-times modified ubuntu. Spent time working out of Venables and Ripley's "S Programming" book - trying to keep some finals week brain momentum going... Plunking example code into the green machine. I kept the screen backlight off, just using sunlight and low battery while the sun was up. When i took a break from coding, i could hit the power button to put it to sleep. So, i worked through some recursion examples, in the meantime discovering my own knowledge-gaps in indexing ( this[-index] is the compliment of this[index] for a numeric index vector, whereas this[!index] is the compliment for a logical index vector) and list packing. Also, found a test example of "=" not the same as "<-" a=1:10 > index1 = a>2 ; any(index2<-(a>2)) ; identical(index1, index2) [1] TRUE [1] TRUE > index1 = a>5 ; any(index2=(a>5)) ; identical(index1, index2) [1] TRUE [1] FALSE ### index2 assignment isn't made... why?! I find recursion confusing; debugging recursive functions is especially so! The fruits of my labor are here, with examples of all of the above... mk.recur = function(a,lim=5, fac=2){ ## takes a numeric vector, returns a list of numeric vectors ## with elements greater than lim, multiplied by fac recursively if(any(index<-a>lim)) {ret=list(a[index])} if(length(a[!index])==0) {return(ret)} ret=append(ret, Recall(fac*a[!index], lim, fac)) return(ret) }

the trigonometry of seeing and being seen (and watching it all happen)

2009-05-12T23:41:00.001-07:00

in day-to-day exchanges, we often indicate our attention to each other, whether or not we're actually listening. a nod of a head, "uhuh", raised eyebrows, smirks & frowns & winks. all these are directional; their target is clear.

at a distance, it's harder to know whether he's checking you out or reading your shirt. when 2 people are in close conversation, it's trivial to tell that they're "gazing into each other's eyes".

basic trigonometry shows that the farther the distance, the less the angle. further, to watch a watcher watching is even trickier - i'm uncertain of the angle between watcher, watched, and me, let alone where their eyes point.

all of this makes it devilishly hard to tell if someone's checking someone other than you out (and which part) if they're sufficiently far away from either. yet two people from afar notice quickly if they're checking each other out - i know where i am in space.

try it. i bet the best you can tell is whether someone's looking at you or not...

grading tests is good for something

2009-04-24T01:07:00.001-07:00

amazing revelation - they always seem so at the time.

bloodflow takes a while to catch up with thought. i do the same thing over & over for 5 minutes, and im in the groove.

fMRI tells me so.

same thing for an hour and im toast. i reckon it's some combination of glucose supply and metabolic waste. ah, the likitations of the flesh.

fieldwork

2009-04-06T17:03:00.001-07:00

Woke up at 6 am - the earliest in 6 months maybe? Spent the day in the field picking up all those aging, half-broken groundwater loggers and bringing them home for good. No more Bosque del Apache, which is kinda sad. No more Lemitar, which is really nice. No more Southern NM mosquitos! Workday in the sun makes boy happy.

A delicate operation

2009-03-26T23:09:00.009-07:00

I formalized my Current Research Statement today for a fellowship that I'm not, at this moment, a smashing candidate for. I beat my head against the conclusion, and i'm still not exactly happy with it, but i attached it and pressed send nonetheless. It's amazing how many errors go unchecked until i print it out and see it on paper... As always, seeing it in latex makes me happy (though i'm not sure how i feel about this ultra-low-res blogger pic - click for something that doesn't make your eyes bug out!):

Getting out of town, getting blown around

2009-03-23T21:17:00.003-07:00

We managed to get out of town for 2 nights. The first night we found an awesome flat-bottomed spruce hollow, sheltered and moist with plenty of firewood, a few miles away from the VLA. It was magical to look down on the VLA on the drive out of the mountains. The second day was terribly windy. Gusts to 45 miles an hour? Sounds about right. The car wobbling around on the highway. Somehow we managed to find a strangely sheltered camp spot in the middle of the woods - some quirk of local topography sent the wind whistling over our heads - to cook our mac-n-cheese and snuggle into bed with the sake. We booked it back home today to get M. to class. I still have a head cold, so today hasn't been too productive otherwise. Here's where we went: http://maps.google.com/maps?f=d&source=s_d&saddr=87106&daddr=Forest+Rd+549%2FNM-52+to:34.114292,-109.493866+to:US-191+to:87106&hl=en&geocode=%3B%3BFfSKCAIdlkF5-Q%3BFaQOGQId8rZ7-Q%3B&mra=ls&sll=35.160337,-108.132935&sspn=1.609939,3.523865&ie=UTF8&ll=34.800272,-107.882996&spn=1.617033,3.523865&t=h&z=9

the most exciting part of my spring break

2009-03-18T21:36:00.001-07:00

is not grading genetics 202 exams. something like 250 tests at about 30 tests an hour (counting setup overhead... hopefully data entry won't blow that average too much) gives me 8 hours of grading. yep, sounds about right. 8 hours of numbing repetition, some students bright, some cleuless, most somewhere in the middle.

in an interview, the famous evolutionary biologist remarked that patience is necessary for science (manipulative paraphrase, that). paint me red and call me a scientist in training. patience and repitition are by no means my strong points!

waiting for a sandwich

2009-03-17T19:02:00.001-07:00

i'm presented with this jem of modern commerce. yes, the white can on the right actually says "free cocaine"... what will these folks think of next? fat-free heroin?

its the little breaks in life, waiting in line, for laundry, short enough to pause but not long enough to do anything useful, that sends these strange little missives out into the ether...

and yes, the lack of cut-and-paste on this phone is truly a sorrow.

darwin continued - see previous post first!

2009-02-12T00:01:00.001-07:00

It gets weirder. Evidence that females prefer manly men when fertile, and caring men when not... Females, like the nest-robber bird, can see big gains by capturing the choicest "extra-marital" sperm for babies, and getting the hubby to feed and cloth them.

"choice" men, for their part, can gain from this arrangement if they can detect fertile women.... or they can lose, unless they guard the "old lady" from the "wolves". my quotes.

the cost-benefit matrix is pretty simple - all women benefit from one caregiver and one or more hunks, same person or different. some men benefit from caring for one female, some from sleeping with many and caring for one or none, and some take care of others' kids!

Darwin's are examples of theories so clearly articulated as to be self-evident to the receptive listener (though he provided heaps of evidence in support thereof). Here too i see sense. The bar is the microcosm, & experiment bears it out. My fave - fertile fems prefer smell of symmetric males.

Darwin Day Dr. Thornberg and the evolution of human female sexuality

2009-02-11T23:44:00.000-07:00

By luck or by fate, his talk was moved and i managed to catch it today, the day before Darwin's 200th birthday. It's still blowing my mind.

Jumping ahead, humans are the only known species in which females don't advertise fertility, In fact, it seems they hide it.

What better place to observe female fertility concealment and mate guarding by males than at a bar.

Xiph.org -- Ogg's ugly step-child lives on

2009-02-11T20:19:00.005-07:00

For years i compressed music into ogg vorbis, before ultimately admitted loosing the device-war. Ipods don't play ogg, which sucks. But i have no control over Apple. (I can tell old age is coming quickly, since i don't really care too much about things i can't control anymore. It's the things i can control that i can regret at my own leisure). Drawing similarities between ogg's parent organization, Xiph.org, and OLPC, is tempting. They both promised to save the world in the face of static, property-ized, inferior technology (mp3 for the former, something more amorphous for the XO). They both were non-profits that, to an arguable extent, over-sold and under-delivered. The OLPC is (mostly) dead. Long live Pixel Qi. Ogg Vorbis is (mostly) dead. Long live Flac. Flac is my new favorite format. The compressor is blazing fast compared to lossy compressors like mp3 and ogg vorbis - limited, seemingly, by disk I/O. As a format, it seems to do everything better than WAV, while consuming a fraction of the space. I can't seek through them, they can't contain the most basic of meta-data, let alone real honest-to-god ID3 tags. With blank DVDs the price they are, and lossy audio codec wars heating back up with AAC, AAC+, etc., i'm stashing "the goods" away in lossless flac for future transcoding to whatever the heck i want, as needed. ----- edit: The above is a bit too harsh on Xiph.org, i suppose. After peering around, the have a plethora of codex alive and kicking - Speex, OggPCM, a low latency format, and HARK! Theora just hit 1.0 with a $100,000 grant from Mozilla. Holy crap! So where does that put us? The ipod video *definitely not* supporting Theora? I'm not talking about the Ipod because i have one or love one. I'm talking about it because people buy them and use them in shockingly large numbers. Firefox is capturing marketshare, and is a good vehicle to consume Theora on. Still, what format plays on the device that fits in someone's pocket has a huge impact on the usefulness of the format -- and i don't expect lossless video, really ever. What would be the point? What i really want is programmable hardware. I don't want use my laptop as a phone or an alarm clock! But I can't really program my alarm clock, let alone my phone, and that really bugs me. So, as a final and decidedly asymptotic aside, i want to mention that intel just announced an upgrade of it's Rio Rancho, NM chip fab to 35nm process (yes, there's sometimes real news in the Albuquerque Journal after all...). I mention this to highlight how few nanometers there are between 35 and 0! The race towards high transistor density is grinding towards completion. I hope (though i'm not holding my breath) that we're approaching a new era of *smarter* hardware, not just blazing fast oh-my-god-so-much-in-a-tiny-package hardware -- and my first criteria of smart is *programmable*, making the format wars obsolete. How'd'ya think big brother would feel about that?

sleep cont

2009-02-05T14:01:00.000-07:00

1 dropperful, twice a day. It's deep red, a tastes a bit like sunshine, in an earthy sort of way.

Superifially, i feel more relaxed. I'm still not "thirsty" for sleep, 6 or 7 hours seems to do me, which is a bit surprising. This morning, though, the dreams. I was in a hurricane in coastal alaska. Don't ask me how... Not scary, more like thrilling, watching the waves build. Then the eye came over, and i ran out into the sudden clear, giddy. I've always wanted to see the eye of a hurricane.

I feel refreshed. The sun's out. I'm glad that, so it seems, the worst is over.

On Seasons and Sleep

2009-02-05T13:57:00.001-07:00

"Sleeping is my hobby"

That quote from one of my bosses. I tend to concur. In the runOup to the winter solstice, i found myself with an insatiable hunger for sleep. 9 to 10 solid hours, day after day. It was magical & comforting & confusing all at once. I could weather a single night of 5 hours of sleep once a week perhaps, and then i had to hibernate. Any less sleep & i soon became a zombie, walking brain-dead.

The sun started coming back, the weather became mild, and the semester started. That last one, i'm sure, affected me most in the short-term. The sudden rush of people, ideas, and deadlines was pure overstimulation. For days at a time, i was awake at night, restless. I lay in bed, closed my eyes, and opened them a few hours later. It's not that i was "trying" to go to sleep, There was simply little sleep to be had, and little rest in the sleep that came.

I started to fall apart a bit.

I started on St. john's wort tincture last sunday. Frsh stuff, made in town.

What i use, and how

2009-02-03T22:11:00.007-07:00

I've been trying to set up my first dual-screen set up, and make it ultra-sweet. I'm using ubuntu and openbox, and i find myself searching for and rediscovering sites and techniques, along with new ones. So, here's a quick document of what i'm using on my desktop! First off, my two favorite desktop art sites are here - sooooo nice http://www.pixelgirlpresents.com/ http://www.deviantart.com/ There's a nice nvidia card in this machine, so I can use the "nvidia-settings" program to set up twinview super-quick to have 2 separate X displays. To run a program on a particular one, i just precede the command with DISPLAY=:0.0 One remarkable discovery was switch2, which handles gtk themes. I'm a black-background fanatic. i can't believe i didn't find this earlier. i always assumed that the window manager handled coloring inside windows, but *obvious in retrospect* that's the application's job, with help from gtk. Yeeesh, learn something profound with each new install! I picked up some themes from here: http://www.gnome-look.org/, though it took me a while to figure out how to use them.... sigh. I settled on a theme called "kore". I really like it's Qt firefox buttons, and rounded everything. I'm using .xsession to start and position everything - using gdm, I select the "Run Xclients script". This is ***far*** from counter-intuitive. So, here's my xsession, that points out a lot of the programs i settled on... DISPLAY=:0.1 openbox & #### i have to start one on each display #DISPLAY=:0.1 ggl-gtk & ### google applets, don't work for some reason xmodmap -e 'keycode 66 = Control_L' & # Make capslock a control key xmodmap -e 'clear Lock' & xmodmap -e 'add Control = Control_L' & #DISPLAY=:0.1 gkrellm -c c2& DISPLAY=:0.0 feh --bg-center ~/.config/openbox/Underwater-City_1600.jpg ###from pixelgirlpresents.com DISPLAY=:0.0 fbpanel & ### supports transparency! DISPLAY=:0.0 gkrellm & ### all-in-one monitoring! DISPLAY=:0.0 firefox & xscreensaver --nosplash ### i use this to lock screen DISPLAY=:0.1 xfce4-terminal --geometry 85x31+0+0 --hide-borders --hide-menubar & ### look ma! a terminal!!! #DISPLAY=:0.1 xfce4-terminal & DISPLAY=:0.0 amarok --display :0.0 & DISPLAY=:0.0 exec openbox Here's the interesting screen and here's where i do all the work - no borders, no scroll bar, just the tab indicators at the top. the mouse wheel and Ctrl-Shift-Up still scrolls, and the X paste buffer does it's duty. Hmmm, looks like gkrellmd is doing WAY more work than it needs to... For my part, real work was obviously lacking so far today! Yeesh, let me count the ways i'm not doing work... 1,2,3,4, a, b, .... -

Tech Withdrawal and the Ebay Wall of Shame

2009-01-25T15:03:00.009-07:00

I bought a used LG vx9800 phone on ebay from a flesh-and-blood human (not a mega-seller) a few months ago. I love it, and I've become melded to it, borg-style, since then. I lost my phone charger about 2 weeks ago. My phone died a few days later. A day or two after that, I started retracing my steps to find it, to no avail. Now i'm, for all practical purposes, a cripple. I'm reading scientific papers and hitting upon words i don't know, and I have no google sms dictionary to rescue me. I can't send myself deadline reminders, or post half-useless ruminations from the coffee-shop while waiting for an electrical outlet to free up. As it stands now, I'm running an electrical cable to my laptop across a busy coffeeshop isle, and i have to re-plug it in after it's kicked out every 5 or 10 minutes... In a desperate attempt to return my life to normalicy, I turn to ebay to get a decent-priced charger. Yet I made the not-uncommon mistake of going too cheap. 3 chargers for $10, shipping included. I figure if one's a lemon, then I still have 2, which allows me to lose one and still have a phone. Nope. They all suck. Sure, I know they're going to be ultra-cheap, but they should work, right? Wrong. They're effectively designed *not* to charge my phone. Within 5 minutes of charging, the phone says "charge complete", and it's obviously not. ( I should point out that a bad charger can be worse than no charger at all (in a sense). A mis-behaved charger can overload and overheat the battery, essentially destroying it in a surprisingly short period of time.) In short, I should have known better. On a related note, ebay's profits from ebay.com are down steeply for the year ( see here). Paypal and the more recently acquired skype, along with advertising, are holding the dogs at bay for now ( see here ). It's shit like like this that is eating ebay alive from the inside. Here's where I bought my charger. From the looks of it, they're rapidly shoveling ultra-cheap Asian electronics to market, getting a 99.2% approval rating on enormous volume, which means lots of unhappy folks in absolute numbers. Here's one good post that addresses their current crisis ( see here ), but it looks like a *lot* of folks, both buyers and sellers, are pretty pissed off ( see here ) Looking back in time, things get even weirder. Ebay purchased a 25% stake in Craigslist and then sued them??? ( see here ). I've drawn heavily from the NYT here, but the WSJ isn't free... My best ebay purchases, over time, have been second-hand electronics. I regularly include the word "used" in my search terms. There are some great deals out there on second-hand 70's electronics (provided it doesn't cost $50 to ship - an well-made 1970s reciever/amp can easily weigh 40+ pounds). It's like going to a great big yardsale that's open at times other than Saturday from 6am-2pm. What's the lesson here? I re-ordered 2 OEM (original equipment manufacturer) chargers - i.e. they're made by LG. We'll see if this works any better. Still, it's another week before I get the rest of my life back. How many more days before my outgoing voicemail message that pleads, "the fastest way to contact me is to send a text message to this number" once again becomes true? How many days before I'm carrying a 5.2 oz dictionary in my backpack?

why i'm nocturnal

2009-01-12T20:04:00.000-07:00

from time to time, especially after a long interlude of relative peace & self-determination (like the one after the holidays), i forget just how much i dislike crowds & traffic.

then i wake up one day relatively early & leave my cave relatively early in order to accomplish some grand & useful task in pursuit of the higher good, something that involves a business establishment that closes @ 5pm, or requires sunlight.

the onslaught, the horror! where did all these people come from, and where are they going?!

today i checked the groundwater wells down by the river. i can't do it after sunset, not at all. trust me. i tried. it's much warmer in the day, too, &very peaceful and pretty.

getting there is murder, tho.

i go to winnings for coffee and breakfast. its packed. people are talking to me. i'm talking back. why?! i just want to eat and read the paper.

next, bumper-to-bumper traffic on the one ways. where are so my folks going @ 3pm? interstate construction.

ick.

The best/worst sunset in town

2009-01-09T17:36:00.000-07:00

Few things hightlight the mind's eyes ability to filter and highlight like cellphone pics. Here's an especially dramatic one taken of the East Mountains at sunset, with a luminously full moon hanging overhead.

This intersection of Lomas and 12th gives one of the best views of the mountains in downtown. Too bad its framed by powerlines and 6 lanes of rush-hour traffic and fumes.

Somehow my mind and my eye are very good at picking out and focusing on the wonder of the sunset-peach-colored mountains and bits of cloud in the distance. I'd linger, but the fumes of ABQ's numerous beaters, and the frantic pace of the intersection, are harder to ignore. Instead, i snap a picture and trot away down quiet, house-lined 12th which, pleasant as it is, lacks the view.

This pic looks nothing like what i saw.

Human eyes are mechanical and informational wonders (see new years post!), functioning in a wider range of lighting than any camera, and responding faster. Cell cameras are the opposite...

google text

2009-01-09T17:25:00.001-07:00

i've been having a lot of fun with google's sms search engine (see http://sms.google.com). it sounds so simple as to be trivial, but its really groovy.

example: text "w zipcode" to 46645, and get this ( for 9 jan 2009):

Weather: Albuquerque, NM 87106 42F, Cloudy Wind: N 19 mph Hum: 42% Fri: 25F-54F, Mostly Sunny Sat: 22F-47F, Mostly Sunny Sun: 25F-47F, Clear

"w", of course, stands for weather. "d" is define, etc. exactly what "etc." is, though, can be ambiguous.

i send two msgs, "help" and "info" and get useful summaries on how to use the system in response (the wonders of self-describing software!), and i learn that a search for "current movie city" should give me local showtimes.

unless, of course, the current movie is a particular oliver stone flick named "w."

it's an great question - how much is the software expected to infer from context, and how much contextual info is the user expected to provide? personally, with a specialized interface like txtmsg where brevity is a virtue, i really like the "keyword query" model, where the keyword can be abbrev to a single letter.

google, if you're listening use m for movie! 20-some letters await good context assignments...

Real Good Food - Que Huong Vietnaemese

2009-01-07T22:59:00.000-07:00

We're cold. It's windy. We need hot salty spicy noodle soup, so we go looking for Cafe Da Lat. It's disappeared off the north side of central and, presumably, the face of Albuquerque.

Just across the street from Talin, hark! Vietnamese with tons of neon beer signs and lots of cars parked in a small lot. Bingo!

Big, exciting menu. Cheap as all get-out, fast service, *strong tea*, and really good hot noodle soup.

No take out menu. Just a business vard that says "the best vietnamese food in town". So far, i agree...
262-0575

Vintage Cookbooks from Internet Archive

2009-01-05T20:35:00.001-07:00

With my foray back into getting the XO to work, I've been obsessing about ebooks lately. Enter the internet archive. For an illuminating glimpse, check this out: http://internetarchive.wordpress.com/2008/03/03/vintage-cookbooks/, and more generally, http://www.archive.org/details/texts

You can learn a lot from the tabloids...

2009-01-05T18:24:00.001-07:00

Myself, I thought she was too fucked up to care...

2009-01-05T18:20:00.000-07:00

the wall of "do we get to leave today?"

everything's looking clear so far...

WHERE AM I?

2009-01-05T18:19:00.003-07:00

HIA --- HOUSTON GEORGE BUSH INTERNATIONAL

2009-01-05T18:17:00.002-07:00

the first in a series of xmas cellphone airport photos

An airport terminal in Houston aka HAI - oh-so-post-modern and, i imagine, rather difficult to vacuum.

An Lutheran-American Xmas Eve

2009-01-05T17:46:00.004-07:00

The Xmas Eve Candlelight and Carols service: a right & proper time & place to greet Pride in our hearts' anteroom as it hangs its hat upon our imagined dignity.

a family, holding a hand-lettered sign, on a sidewalk by a stoplight amidst a TN big-box Xmas wonderland. i walk past, across a fast food parking lot, the only other pedestrian on Xmas eve, as a warm, light mist falls. traffic in the fading light. all parties feign nonchalance.

Xmas: putative season of humility. Our gifts are surely never given to impress, to inflate our own imagined goodness! So plump with self-satisfaction that we easily and unironically ignore our last nagging inclinations towards largess and contrition.

And the Xmas carols! Written for leaner, darker times? These songs, out of childhood, now echos and simulcrums of hope. The Culture of Now, superbowl and starbucks and more than 100 million cars in 50 states, already has its own answers, priced to move for the holiday season.

My pride helps me safely scoff.

In Cynic am i cloaked;
Untouchable;
Judging but unJudgeable:

i'm a culture-hero. Like a super-hero, only hip-er.

Rising above it all... (and whatever is it?) Is this why we drink and smoke and fuck? Our culture, historically lacking in uncommercialized transpersonal alternatives. Like fish into a cave. we've slowly blinded to the strength of our self-affirmations. Tag,we're it.

In pain, we suddenly question our inalienable rightness, our moral superiority. Recessions breed innovation; depressions are lulls in which listeners hear troubadours above the Now Culture Roar. Even then it lies, hungry, in wait.

Wine & Communion Wafers: to me the taste, give or take, of roman/spanish/dutch/german/english pillage and war, & the ovens of imperial entitlement. It sounds ridiculous until i try to "eat and drink - this is the body and blood of the lord god, given for you" with a stright face. No one else is laughing. A touch of evil, not unlike bitters, is useful in treating ethical constipation and moral flatulence:

a hint of naked envy;
a dram of guiltless gluttony;
an ounce of unapologetic lust;
a pinch of piracy?

Stomachics for our putatively immortal souls, purging us of a great excess of ourselves.

My favorite part of church is that we don't clap for each other.

OLPC: Why Sugar Sucks - A Really Long List

2009-01-03T19:31:00.002-07:00

Why OLPC Software Sucks

olpc's "sugar" operating system takes the cake for the most unresponsive OS ever, and that's saying a lot! this is a pic of me trying to open an activity bundle, an ".xo" file. As i type this, nothing has changed. ctlr-alt-f2-ing over to a real, honest-to-god terminal shows me that unzip has eaten my XO's rather limited AMD GEODE processor for now. WTF?

Steps to cause users great software discomfort:

1. Use redhat's fedora core, and alter it almost beyond recognition - the result is "sugar", an awkward operaring system that steals valuable devel time from the OSS world.

2. insist that every previously-written piece of software must be "sugarized" to work with XO. Generate buzz and intial devel activity that result in lots of orphaned software projs.

3. write a new browser. nevermind many decent ones already exist.

4. Refuse to play nice with anyone, or admit mistakes *ever*

I've been trying to use sugar for about a year now. It's still terrible. It's more terrible than i can summarize in the requisite 1000 characters of an MMS, so i now continue my diatribe from the green machine itself.

Other things i hate:

* No man pages - the most flagrant sign that "it just plays linux on TV"

* No swap, since it has a flash "hard drive" that would be degraded by swap's frequent writing. Too bad linux on 256MB ram *really* needs swap. Eric's performance hacks, especially the incredible compcache greatly alleviate this evil (keep an eye on compcache - changing the nature of linux computing!). Still, it's non-trivial to permanently enable compcache. Another example of "it just plays linux on TV"

* No mp3 capability by default; installation must be done via commandline. I'm no fan of mp3. I used ogg for years before i finally admitted defeat. We're stuck in an mp3 world. As such. the XO is a victim of idealogical zealotry.

* Battery life - claims of 10-12 hours were a major selling point for me. A year later. sugar's best power saving features are still *experimental*, for god's sake! 3.5 hours today from full charge, 1/3 of that with backlight off, reading by sunllight.

* No hibernate, nor the possibility of it. On my thinkpad, i can press a button and suspend my machine to disk, leave it my backpack for weeks, and get my in-use desktop back with the press of a button. With OLPC's emphasis on power, i thought the XO could at least match this feat.

Nope. At best, it goes sleep. Leave it asleep on the shelf for a month or two, and the battery gets so dead that it WON'T CHARGE. A special EC program (batman.fth) must be used to flash the battery. All this requires an easily-obtained developer's key. What's secure about requiring this key???

* Ebook reading still not functional. Flipping through pdfs is painfully slow. Hit the wrong button and *surprise* you're at the end of the book, you've lost your place, and it will take 5 or 10 minutes to get back.

Overall, the XO's software sucks. Bad.

The hardware is amazing. I've put the keyboard under running water. It's 3 pounds, had a nice handle, and a small powerwart. I can read the screen in DAYLIGHT!

So what's a geek to do? Buy a 16gb SDHC sandisk extreme II (15mb/s) card ($40 at writing) and install ubuntu. That's what i'm doing.

2009-01-03T13:58:00.001-07:00

blunt objects;
eye trauma;
orbital fracture:

file under things not to do on new year's eve.

after 2 days of seeing double (vertical displacement) a sore left eye, and numbness on that side of my nose, i started to wonder if a doctor might have some wisdom. it was fri the 2nd of this new year, not a holiday per se, so i should be able to schedule some non-emergency visit, right?

wrong on all accounts. another case of "go ask google". after talking to a very nice soul @ NewMexNurseHotline (24/7/365&free!), calling the still-closed UNM student health center, & pres. hospital, the emer room was still my only official option.

30 min of google later, i have found my symptoms and causes described exactly. the eyeball rests in an orb of fragile bones that tend to crack before the eyeball pops under sudden pressure (lucky for us... im not sure popped eyeballs heal so gracefully). unless severe, it heals itself.

so i scheduled an eye exam for next week. i need new glasses anyway. oh, and here's my self-diagnosis.

Why blog?

2008-12-30T21:28:00.002-07:00

This is the picture i have on my desktop. I wish i could remember where i found it, because i love it, and someone else made it, and i wish i could give them credit for it.

Are blogs inherently narcissistic?

As a *nix technologist, i use google to fix 75% of my problems *very* quickly. Google just finds the already-published answers, many of which come from personal blogs, short write-ups of problems solved.

It's like stone soup

As an aside, I read a lot of newspapers. There's a huge, ongoing conversation about how newspapers *create* news, while bloggers move it around, pick out the nice parts, etc. Are bloggers destroying newspapers? I reckon it's a question worth asking, but with respect to technology, it's a null one.

Personal observation in general, and micro-problem-solving in particular, is where blogs excel. I'm a little embarrassed that i started so late, but i've gotten to the point in my life where, more and more, i use technology to solve a clear-and-present problem. I need it to fit me.

Enter cellphone blogging.

I need it to be simple, to work as near the speed-of-thought as possible. Keybindings are a great example. I learn new firefox and vim keybindings because they allow me to work *much* more rapidly, with much less thought, once my fingers finally learn them.

Sometimes computers are a pain to find, or there's no internet access, or my battery's down, or i only have time to check my email and run. On the other hand, my cellphone can stay on for days without a charge, (finally!) has a thumb keyboard and camera attached, and can almost always send pix messages. Alternately, I can use my laptop with gmail; then i get a real keyboard and firefox's spellchecker, and i can compose a well thought-out email quickly and easily.

I don't want a blog to take time out of my life. Rather, i want to put a little bit of effort into it here and there to make my life better. For me, these are the things of which speed-of-thought blogging is made. Only now do i enjoy tossing things into this vast and mercurial stone soup cauldron.

2008-12-30T19:24:00.001-07:00

which way is south?

snow is still on the ground on the north side of this massive 4-wheeled thing that never moves, after 3 days of relatively warm and sunny weather.

this still gives me a moment of pause - unexpectedly logical, a trick, like navigating by moss on the north side of trees.

*edit* my cellphone camera doesn't have the best auto-exposure... i had to pull the pic into digikam, which does have some really nice auto-adjust tools, and then re-post it. you can see the car, the shadow of the tree, the sidewalk, and the faint crust of crunchy snow on the grass, right?

2008-12-30T16:26:00.001-07:00

first post from phone - 1000 chars is a big upgrade from twitter for some stuff.

next: www.x14n.org... automagic link interpretation!

finally, the knoxville *linens'n'things*, post-pillage by holiday shoppers. the fixtures are for sale!

*edit* - i'm so stoked that actually works. verizon "pix text" to go@blogger.com can get pics from phone to blog just fine, with auto-link-detection. life is sweet. I did have to afteredit the image to rotate it...

first post

2008-12-30T15:47:00.000-07:00

crosspost: http://www.twitter.com/prosopis annoying - links not autorecognized... what's up with that? I *really* hate having to click a toolbar button to change text. Yes, I'm a text-input primadonna.