Current Volume | Browse | Search | RSSHome | Instructions for Authors | JSS Style Guide | Editorial Board

Authors: Michael Kane, John W. Emerson, Stephen Weston
Title: [download]
(3147)
Scalable Strategies for Computing with Massive Data
Reference: Vol. 55, Issue 14, Nov 2013
Submitted 2012-09-05, Accepted 2013-04-18
Type: Article
Abstract:

This paper presents two complementary statistical computing frameworks that address challenges in parallel processing and the analysis of massive data. First, the foreach package allows users of the R programming environment to define parallel loops that may be run sequentially on a single machine, in parallel on a symmetric multiprocessing (SMP) machine, or in cluster environments without platform-specific code. Second, the bigmemory package implements memory- and file-mapped data structures that provide (a) access to arbitrarily large data while retaining a look and feel that is familiar to R users and (b) data structures that are shared across processor cores in order to support efficient parallel computing techniques. Although these packages may be used independently, this paper shows how they can be used in combination to address challenges that have effectively been beyond the reach of researchers who lack specialized software development skills or expensive hardware.

Paper: [download]
(3147)
Scalable Strategies for Computing with Massive Data
(application/pdf, 329.5 KB)
Supplements: [download]
(236)
bigmemory.4.4.5-1.tar.gz: R source package
(application/x-gzip, 183.3 KB)
[download]
(231)
foreach.1.4.1-1.tar.gz: R source package
(application/x-gzip, 351.2 KB)
[download]
(224)
v55i14-replication.zip: Replication R code for examples from the paper
(application/x-zip-compressed, 2.3 KB)
[download]
(406)
Downloading_data.txt Instructions for downloading data set.
(text/plain, 195 Bytes)
Resources: BibTeX | OAI
Creative Commons License
This work is licensed under the licenses
Paper: Creative Commons Attribution 3.0 Unported License
Code: GNU General Public License (at least one of version 2 or version 3)
Current Volume | Browse | Search | RSSHome | Instructions for Authors | JSS Style Guide | Editorial Board