Architecture
Hadoop consists of the Hadoop Common which provides access to the filesystems supported by Hadoop. The Hadoop Common package contains the necessary JAR files and scripts needed to start Hadoop. The package also provides source code, documentation, and a contribution section which includes projects from the Hadoop Community.
For effective scheduling of work, every Hadoop-compatible filesystem should provide location awareness: the name of the rack (more precisely, of the network switch) where a worker node is. Hadoop applications can use this information to run work on the node where the data is, and, failing that, on the same rack/switch, reducing backbone traffic. The Hadoop Distributed File System (HDFS) uses this when replicating data, to try to keep different copies of the data on different racks. The goal is to reduce the impact of a rack power outage or switch failure so that even if these events occur, the data may still be readable.
A small Hadoop cluster will include a single master and multiple worker nodes. The master node consists of a JobTracker, TaskTracker, NameNode, and DataNode. A slave or worker node acts as both a DataNode and TaskTracker, though it is possible to have data-only worker nodes, and compute-only worker nodes; these are normally only used in non-standard applications. Hadoop requires JRE 1.6 or higher. The standard startup and shutdown scripts require ssh to be set up between nodes in the cluster.
In a larger cluster, the HDFS is managed through a dedicated NameNode server to host the filesystem index, and a secondary NameNode that can generate snapshots of the namenode's memory structures, thus preventing filesystem corruption and reducing loss of data. Similarly, a standalone JobTracker server can manage job scheduling. In clusters where the Hadoop MapReduce engine is deployed against an alternate filesystem, the NameNode, secondary NameNode and DataNode architecture of HDFS is replaced by the filesystem-specific equivalent.
Read more about this topic: Apache Hadoop
Famous quotes containing the word architecture:
“The two elements the traveler first captures in the big city are extrahuman architecture and furious rhythm. Geometry and anguish. At first glance, the rhythm may be confused with gaiety, but when you look more closely at the mechanism of social life and the painful slavery of both men and machines, you see that it is nothing but a kind of typical, empty anguish that makes even crime and gangs forgivable means of escape.”
—Federico García Lorca (1898–1936)
“Polarized light showed the secret architecture of bodies; and when the second-sight of the mind is opened, now one color or form or gesture, and now another, has a pungency, as if a more interior ray had been emitted, disclosing its deep holdings in the frame of things.”
—Ralph Waldo Emerson (1803–1882)
“For it is not metres, but a metre-making argument, that makes a poem,—a thought so passionate and alive, that, like the spirit of a plant or an animal, it has an architecture of its own, and adorns nature with a new thing.”
—Ralph Waldo Emerson (1803–1882)