|
Harnessing Distributed Computing Power
using Open Source Tools
转自:http://www.chessbrain.net/hdcp.html
主题:利用开放源代码工具收集分布式计算力量
概述:使用开放源代码工具创建分布式计算项目的白皮书
出自:《ChessBrain.NET》
发表时间:2004年2月1日
内容:
Harnessing Distributed Computing Power
using Open Source Tools
Carlos Justiniano
ChessBrain.NET
NordU 2004 USENIX Conference
Copenhagen, Denmark
February 1st 2004
ABSTRACT
Distributed Computing projects are successfully utilizing the idle processing power of millions of computers. This paper examines how Open Source tools can be used to harness the vast computing potential of Internet connected machines.
Because distributed computing projects depend on individuals, who generously volunteer the use of their machines, the human and machine dynamics that makes distributed computing projects possible will be closely examined.
The second half of this paper explores technical issues and examines how open source tools and technologies are helping to pave the way for new distributed computing projects.
1. PART ONE - DISTRIBUTED COMPUTING, human and machine dynamics
1.0 INTRODUCTION TO DISTRIBUTED COMPUTING
Distributed Computing (DC) can be defined as the use of computers and their resources from geographically dispersed locations. By definition, the word distributed implies: dispersed, partitioned, and perhaps loosely connected. The term distributed computing has become synonymous with Internet connected machines that work together to accomplish one or more goals. The ability to use the resources of remote machines amounts to the technological equivalent of sharing. In fact, the earliest systems were referred to as timesharing systems.
The basic principles behind distributed computing are by no means new. From the moment there were two computers present side by side, there must have been a desire to connect them. Indeed, the desire to connect research centers and computing resources eventually led to the creation of the Internet.
1.0.1 Cluster Computing
As machines became more powerful, researchers began exploring ways to connect collections (clusters) of smaller machines to build less expensive substitutes for the larger, more costly systems.
In 1994, NASA researchers, Thomas Sterling and Don Becker connected 16 computers to create a single cluster. They named their new cluster, Beowulf, and it quickly became a success. Sterling and Becker demonstrated using commodity-off-the-shelf (COTS) computers as a means of aggregating computing resources and effectively solving problems, which typically required larger and more dedicated systems.
Beowulf clusters are now commonplace, with large clusters being built using thousands of machines. Assembling clusters is no longer the purview of computer scientist, using the freely available ClusterKnoppix, which is a modified distribution of Knoppix Linux and OpenMosix, students can quickly connect a group of machines to form a cluster.
While speaking at a Mac OS X conference, technology publisher, Tim O'Reilly, told his audience that they were are all Linux users. He went on to explain that the search engine, Google, is powered by the world's largest Linux cluster3. In fact, the cluster consists of more than ten thousand servers according to a Google press release4.
1.0.2 Peer-to-Peer Computing
Although, distributed computing predates the Internet, it continues to touch nearly every aspect of our computing lives. Beyond web browsing and email, the use of instant messaging and P2P file sharing increasingly rank high in the lives of hundreds of millions of people.
Consider the acronym, P2P, it means Peer to Peer. In a P2P system peers connect to one another rather then to a central location. The concept seems new to some, because the Internet they experience most, is a system called the World Wide Web. In contrast, P2P systems rightfully bypass websites and in turn communicate directly with one another. The point is noteworthy because it underscores a core philosophy, which has given birth, and continues to guide the development of many of the Internet systems we now take for granted. The Internet was not conceived of as a system involving centralized control, rather, early pioneers went out of their way to ensure the Internet would exist and flourish as a decentralized system.
Peer-to-Peer systems are allowing machines to connect with one another to share resources. The resources might involve file sharing and remote storage, or the sharing of processing power. Increasingly researchers are exploring the potential for the sharing of processing resources. Studies show that most computers are vastly underutilized. This squandering of resources doesn’t just involve home computers; very few businesses operate on 24-hour shifts, leaving vast numbers of machines sitting idle for large periods of time. If one also considers the millions of smart cash registers at retail outlets and markets, it seems that the underutilization of computing power appears everywhere.
1.1. HARNESSING THE VAST POTENTIAL OF DISTRIBUTED MACHINES
It’s important to point out that the primary lure of harnessing remote computing power is largely one of economics. After all, if limited financial resources were not of concern, what researcher wouldn’t want a massive cluster of dedicated machines? Reality being what it is… dedicated resources tend to be in high demand and researchers find themselves writing proposals pleading for precious time on large compute clusters. It’s easy to envision a researcher (who potentially holds the code that might unlock a medical cure) being denied access to centralized and dedicated computing resources.
Because the lack of computing resources may come from political, economic, and social barriers, researchers are turning to the Internet as a viable way to continue their computationally intensive work. What is equally interesting is that ordinary citizens (many of whom pay the taxes which fund major institutions) are providing the machines which support distributed computing based projects!
1.1.1 Distributed Computing Applied to Computation
Distributed computing offers researchers the potential of solving complex problems using many dispersed machines. The result is faster computation at potentially lower costs when compared to the use of dedicated resources. The term Distributed Computation has been used to describe the use of distributed computing for the sake of raw computation rather than say, remote file sharing, storage or information retrieval.
This paper will tend to focus mainly on distributed computation, however, many of the concepts and tools apply to other distributed computing projects.
1.1.2 Distributed computation: a Community Approach to Problem Solving
Distributed Computation offers researchers an opportunity to distribute the task of solving complex problems onto hundreds and in many cases thousands of Internet connected machines. Although, the network is itself distributed, the research and end user participants form a loosely bound partnership. The resulting partnership is not unlike a team or community. People band together to create and connect the resources required for the achievement of a common goal. A fascinating aspect of this continues to be humanity’s willingness to transcend cultural barriers.
1.1.3 Successful Projects
The earliest known DC projects used email to distribute tasks and results in the late 1980’s. At the time, most people considered massively distributed projects to be largely idealistic. The notion that ordinary people would pay for and allow their computers to participate in research projects was foolhardy at best.
Participants in a distributed computing project bare the cost of equipment, electricity, and any necessary upgrades. The fact that millions of people now participate in distributed computing projects is a testament to an evolving technology landscape.
Distributed Computing entered the public spotlight, On October 24th 1997, when the New York Times published an article entitled "Cracked Code Reveals Security Limits" to report on the distributed.net project’s achievement in deciphering the RSA 56-bit encryption challenge. Distributed.net proved the viability of using massively distributed machines over a public network when they found the encryption key that unlocked RSA’s encoded message: "The unknown message is: It's time to move to a longer key length".
The following year, a group of researchers at the University of California in Berkeley launched the SETI@home project. While SETI@home hasn’t found any signs of alien life, it has demonstrated a staggering use of computational power. SETI@home researcher, David Anderson compared the computing power available to the project against the ASCI White super computer which was built by IBM for the U.S. Department of Energy. The machine cost US $110 million, weighed 106 tons, and offered a peak performance of 12.3 TFLOPS. Anderson went on to state that SETI@home was faster than the ASCI White at less than 1% of the cost. The reduction in expenditure was due to the fact that not only were the computations distributed… but so were the costs!1
The AspenLeaf DC website (http://www.aspenleaf.com/distributed/) currently lists over 50 active projects, ranging from Analytical Spectroscopy to the Zeta grid working in areas such as Art, Cryptography, Life Sciences, Mathematics and Game research. Not surprisingly, the category of mathematics features the largest number of DC projects.
[ Last edited by 碧城仙 on 2004-11-17 at 12:33 PM ] |
|