tag:blogger.com,1999:blog-4695461192314112143.post2303763110083314164..comments2023-02-28T06:08:03.248-07:00Comments on Life in Code: snow and ssh -- secure inter-machine parallelism with Rxianhttp://www.blogger.com/profile/01991290472959345882noreply@blogger.comBlogger8125tag:blogger.com,1999:blog-4695461192314112143.post-43363480493362663832012-12-16T19:50:48.946-07:002012-12-16T19:50:48.946-07:00Thank you so much for your post. I don't have ...Thank you so much for your post. I don't have any real experience with Unix systems but after some trying I've managed to repeat your code on AWS Ubuntu instances and RStudio server. That was cool :-)<br />Could you please share an example of your R-code to manage multiple remotehosts? I feel myself more comfortable with R than with SSH terminal.<br />Thanks a lot. Regards, Tim.Timhttps://www.blogger.com/profile/02978898195506998856noreply@blogger.comtag:blogger.com,1999:blog-4695461192314112143.post-27297851929987593862011-03-23T01:08:03.528-07:002011-03-23T01:08:03.528-07:00In the example I gave, I was spawning multiple sla...In the example I gave, I was spawning multiple slaves on one machine named "remotehost". You could wrap another bash do loop around this with multiple hostnames, but I prefer to handle that part in R -- then each machine name in the R list of remotehosts can have a distinct number of slaves associated with it, and an R function calls the bash loop once per host.xianhttps://www.blogger.com/profile/01991290472959345882noreply@blogger.comtag:blogger.com,1999:blog-4695461192314112143.post-11542189494664584962011-03-16T09:01:47.216-07:002011-03-16T09:01:47.216-07:00In "do ssh -f remotehost ...", shouldn&#...In "do ssh -f remotehost ...", shouldn't "remotehost" depend on i?<br />I mean something like<br /><br />do ssh -f "remotehost$i" ...matteo.grigolettohttps://www.blogger.com/profile/02482246700205201660noreply@blogger.comtag:blogger.com,1999:blog-4695461192314112143.post-47289443500080212492011-03-16T09:00:36.463-07:002011-03-16T09:00:36.463-07:00In "do ssh -f remotehost ...", shouldn&#...In "do ssh -f remotehost ...", shouldn't "remotehost" depend on i?<br />I mean something like<br /><br />do ssh -f "remotehost$i" ...matteo.grigolettohttps://www.blogger.com/profile/02482246700205201660noreply@blogger.comtag:blogger.com,1999:blog-4695461192314112143.post-62009247339213586212011-03-10T04:04:24.798-07:002011-03-10T04:04:24.798-07:00Thanks!
Dirk's post really helped me step thr...Thanks!<br /><br />Dirk's post really helped me step through the trouble-shooting by deconstructing the process into orthogonal steps. The ssh connections seem to be the most fragile part, and in some senses easiest to test.<br /><br />I'm curious of your setup. Do these steps work within your network and only fail when you try to cross sub-networks?<br /><br />If you're interested in this, I'd appreciate your thoughts and testing. The easiest way to get my attention is twitter -- @prosopis :)xianhttps://www.blogger.com/profile/01991290472959345882noreply@blogger.comtag:blogger.com,1999:blog-4695461192314112143.post-23853837068401980092011-03-09T08:01:43.404-07:002011-03-09T08:01:43.404-07:00Really nice post.
I have been trying to do this f...Really nice post.<br /><br />I have been trying to do this for quite some time now, but I always come back to emacs and tramp, because I am not able to get this working.<br /><br />When I follow your steps closely I can establish the connection and even register the backend, but when I try to use it with %dopar% I always get<br />Error in serialize(data, node$con) : ignoring SIGPIPE signal<br /><br />I suspect this to be something connected with the firewall or the connection beeing between seperate sub-networks, as I have been using snow within one network successfully for a while.<br />Stange thing is, that I use reverse tunnelling regularly for other purposes than R without problems.<br /><br />Any ideas by any chance?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-4695461192314112143.post-36502164713349350002011-03-02T00:42:13.194-07:002011-03-02T00:42:13.194-07:00From the redis package manual:
"Redis suppo...From the redis package manual:<br /> "Redis supports a trivially simple and insecure authentication method. This function implements it. [...] You should not use this function. If you need a secure key/value database, it’s best not to use Redis for now."<br /><br />So no, using redis for IPC isn't what I'm looking for :)<br /><br />Just to clarify, "what I want" is the *thinnest* possible stack to securely deploy %dopar% on multiple machines, with as few privileges as possible. I'm curious to hear folks' thoughts on MPI -- at first blush, it sounded like "just another piece of software to learn and install", but I'm curious now...xianhttps://www.blogger.com/profile/01991290472959345882noreply@blogger.comtag:blogger.com,1999:blog-4695461192314112143.post-90765430969528809302011-03-02T00:04:47.805-07:002011-03-02T00:04:47.805-07:00have you tried the doRedis package?
it is very muc...have you tried the doRedis package?<br />it is very much what you want.Anonymousnoreply@blogger.com