R (15) Admin (12) programming (11) Rant (6) personal (6) parallelism (4) HPC (3) git (3) linux (3) rstudio (3) spectrum (3) C++ (2) Modeling (2) Rcpp (2) SQL (2) amazon (2) cloud (2) frequency (2) math (2) performance (2) plotting (2) postgresql (2) DNS (1) Egypt (1) Future (1) Knoxville (1) LVM (1) Music (1) Politics (1) Python (1) RAID (1) Reproducible Research (1) animation (1) audio (1) aws (1) data (1) economics (1) graphing (1) hardware (1)

15 February 2016

Shiny on Webfaction: VPS installion without root

I've been using Webfaction (plug) as an inexpensive managed VPN. Part of me wants VPS root access, but I'm mostly happy to leave the administrative details to others. Webfaction seems to be a good example of a common VPS plan: user-only access in a rich development environment. Compilers, zsh, and even tmux are available from the shell, making this a very comfortable dev environment overall.

Most times root doesn't matter, but sometimes it complicates new software installs. I've been looking forwards to testing R's webapp package Shiny, but all of the docs assume root access (and some even state that it's required). I set off without knowing if this would work, attempting to see how far I could get. What follows is a (hopefully) reproducible account of a user-land install of R & Shiny via ssh on a Webfaction slice. To the best of my knowledge, this requires only standard development tools, and so should(??) work.

In the following I use [tab] to indicate hitting tab key for auto-completion. The VPS login username is [user]. [edit] means call your editor of choice (vim, emacs, or, god forbid, nano). This assumes you are using bash (which seems to be the default shell on most VPNs).

Prepare the build environment

## ssh to webhost
## make directories, set paths, etc
## source build dir
mkdir ~/src
## software install dir
mkdir ~/local
## personal content dir
## some hosts have /tmp set noexec?
mkdir src/tmp
## Install software here

## set paths:  
echo 'export PATH=$PATH:~/local/bin:~/local/shiny-server/bin' >> ~/.bashrc
echo 'export TMPDIR=$HOME/src/tmp' >>~/.bashrc

## check that all is well
[edit] ~/.bashrc
## update env
. .bashrc
[Ref: temp dir and R packages]

Install R from source: fast and (mostly) easy

cd ~/src
tar xzf R-3.2.3.tar.gz
cd R-[tab]
./configure --prefix=$INSTPREFIX
## missing library, search and add directory
CPPFLAGS=/usr/lib/jvm/java/include/ make
make install
cd ~

Prep R environment

## The following commands are in R
install.packages(c('shiny', 'rmarkdown'))
## From the shell:
## on a headless / no-X11 box, need cairo for png
echo "options(bitmapType='cairo')" >> ~/.Rprofile
## check that all is well
[edit] ~/.Rprofile
[Ref: R png without X11]

Install cmake (if needed)

## first install cmake - skip if's already available 
`which cmake`
## nothing?  continue
## NOTE - I'm using the source tarball here, not binaries
tar xzf cmake-[tab]
cd cmake-[tab]
./configure --prefix=$INSTPREFIX
make install

Install Shiny Server

## From shell
cd ~/src
git clone
cd shiny-server
## "make install" Complains about no build dir
## I'm not sure what happens here, but this seems to work
PYTHON=`which python`
mkdir build
./bin/npm --python="$PYTHON" rebuild 
./bin/node ./ext/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js --python="$PYTHON" rebuild 
make install
[Ref: shiny build docs]

Configure Shiny Server

All of the Shiny Server docs assume the config file is located in /etc/, which I don't have access to. There's _zero_ documentation on running shiny, nor does running shiny-server -h or shiny-server --help provide any indication. Trial and error and reading source code on github finally leads to shiny-server path-to-config-file. So, let's make a shiny site!
## Nest content in ~/var
mkdir $CONTENTDIR/shiny
cp -rp ~/src/shiny-server/samples $CONTENTDIR/shiny/apps
mkdir $CONTENTDIR/shiny/logs
## copy the packaged settings template to the content dir
cp ~/src/shiny-server/config/default.config $CONTENTDIR/shiny/server.conf
[edit] $CONTENTDIR/shiny/server.conf
## server.conf content follows:
run_as [user];
## leave location as-is
## substitute var with $CONTENTDIR if needed
    site_dir /home/[user]/var/shiny/apps;
    log_dir /home/[user]/var/shiny/logs;    
## save file
## back at shell, run shiny, put in background
shiny-server ~/var/shiny/server.conf &
[Ref: Shiny-server docs]


Shiny should give messages about Starting listener on First up, let's use ssh to connect remote port 3838 to a local port. This allows local testing without deployment. As an aside, if you're not using ~/.ssh/config on a local machine to manage keys and hostname shortcuts, you should!
## on local machine:
ssh -nNT -L 9000: [user]@webhost
Now, if all went well, you should be able to navigate to the welcome page via browser on local machine:

Once shiny is working, don't forget to take a look at your logs:
ls -alh $CONTENTDIR/shiny/logs

I had trouble with the packaged rmd example app (which renders a .Rmd file). Reading logs showed install issues with pandoc, and I had to manually fiddle with the links:

ln -s $INSTPREFIX/shiny-server/ext/pandoc/static/pandoc $INSTPREFIX/shiny-server/ext/pandoc/
[Ref: port forwarding]


For a full production environment, you would want a process monitor to keep shiny-server running, as well a public-facing server. See your webhost's documentation for process monitors. More details on shiny-server and apache are here (I haven't tried these proxy methods).

Finally, a more conventional approach using root access on a VPS (such as DigitalOcean) is available here.

Update - 17 Feb 2016: Deployment Logistics

After a day of kicking the tires, I'm happy to report Shiny-server is working well on Webfaction in production mode. Two points:

Making a webapp. In the Webfaction control panel, I added a custom application. In the following, substitute [appname] for the value entered in the Name field. For App category I selected "Websockets", and then clicked "Save". Copy the port number. Edit the server.conf file from above, replacing the number in listen 3838; with the port number copied from Webfaction. Finally, create a website, add a name (can be the same as [appname] from above), and a domain. It typically takes a few minutes for DNS changes to propagate.

The above steps creates a directory named $HOME/webapps/[appname]. I placed the server.conf file here, created app and log directories, and then updated server.conf to reflect the new locations:

## Create the following directories
## add these paths to server.conf, 
## and don't forget the trailing ; 
mkdir $HOME/[appname]/logs
## shiny app files go here:
mkdir $HOME/[appname]/app

[Ref: Webfaction custom application]
[Ref: Webfaction Applications and Websites]

Running the server. Shiny-server will use a PID file, which makes job-spawning a simple shell script + cron job. If shiny-server is already running, it will recognize the PID file and not start another process. I made the following script:

## executable shell script names $HOME/bin/
## make sure to run: chmod +x $HOME/bin/
## using full path                                                        
$HOME/local/shiny-server/bin/shiny-server $APPROOT/server.conf --pidfile=$PIDFN>> $APPROOT/logs/server.log 2>&1 &
Now run crontab -e and add an entry for the script (above):
## try once an hour, on the 10th minute of the hour
10 * * * * /home/[user]/bin/
Finally, take a look at memory usage. If you exceed memory limits, Webfaction automatically kills everything. And R's memory use grows with more connections (which themselves persist, because websockets). Webfaction distributes a nice python script that shows per-process and total memory usage.

[Ref: shiny-server systemd script (shows commandline usage)]
[Ref: Webfaction cron]

I should point out that I like Webfaction (plug) well enough to pay them money. Their intro plan is $10/month for 1GB RAM + 100GB full SSD, with a 1-month free trial. I like that the webfaction user-base is big enough that lots of my questions are already answered, but small enough that staff actually answer new questions.

I've done my best to document exactly what I did, but I'm sure there are typos. Let me know if you encounter any issues!


  1. Thanks for the detailed instructions, very helpful for those of us who know R but not Linux. I got all the way up to cmake -DCMAKE_INSTALL_PREFIX=$INSTPREFIX and have an error that CMake 2.8.10 or higher is required but I'm running version 2.8.4. I likely installed an older version years ago. Deleting the directory isn't making it go away. Any suggestions on how I can make the system use my newly installed cmake 3.4?

    1. What do you get from `which cmake`
      Perhaps modify your path so that the custom cmake takes precedence?


  2. which cmake returns /local/bin/cmake not the custom one. Will try to modify path, thanks

  3. Thanks - Nice bit of work. Without it I would have walked away from Webfaction.

    I did run into a problem with it being unable to find javas' jni.h but worked around that.

  4. Hi,

    Could you tell me a bit more about using webfaction as a vpn? I will be spending time in China, and would like to not pay for a separate vpn since I already have a webfaction account.

  5. What do you want to know? I quite like it - it comes with a development environment pre-installed. To be honest, ssh should work the same the world over,
    and ssh should handle most single-user tasks. Network bandwidth and path will likely be the most important.
    Here's results from ping to a Chinese ntp server (I have no idea where).

    --- ping statistics ---
    17 packets transmitted, 17 received, 0% packet loss, time 16752ms
    rtt min/avg/max/mdev = 193.344/193.529/193.825/0.430 ms

    I got similar results for the other 2 Chinese ntp servers - low hundreds time, almost zero variance. I take that as a good sign - that roundtrip time is limited by geography rather than network conditions, but again it's Sunday there now...

    Any other questions?

  6. Awesome Manual!
    Would be lost without it, but i got only to the point to run r. Trying to install the shiny-server reuslts after the "make" command in the following:

    -- extracting... done
    [ 50%] No patch step for 'node'
    [ 60%] No update step for 'node'
    [ 70%] Performing configure step for 'node'
    File "/home/dmh2office/src/shiny-server/external/node/node-v0.10.40/configure", line 452
    fpu = 'vfpv3' if armv7 else 'vfpv2'

    i found out that should be a problem of python but if I put "python -V" I get:
    Python 2.7.11

    Do you have an idea what can be this issue about! Thanks a lot!

  7. got it somehow installed with
    npm install -g shiny-server/ --python=python2.7