The byteline tool is a stand-alone utility for distributing large files across large computer clusters by sending a single copy of the data through the network, each computer making a local copy in the process. This method also seems to be known as a "pipelined copy."
This package also includes an "object-style" C pthreads wrapper to implement threads, semaphores, and message queues. The library is used by the byteline tool.
Available through the project web page.
Links:
Individual developers (just J. Tseng for now):
This code was developed for use in the Level 3 Trigger at the Collider Detector at Fermilab (CDF). It was sponsored by the United States Government Department of Energy under cooperative agreement with the Massachusetts Institute of Technology Laboratory for Nuclear Science.
The Level 3 Trigger is a cluster of (at last count) 266 Linux PC's which performs on-line filtering of interesting proton-antiproton collisions. The filter program analyzes patterns among the million or so readout channels of CDF, and the byteline utility regularly distributes new filter executables and calibration databases, which amount to hundreds of megabytes, to the cluster nodes while starting up the data acquisition systems.
 MIT |
 LNS |
 FNAL |
 CDF |