Apache’s Hadoop: 10 Things You Need to Know


While it may have a funny name, Hadoop provides all of the key elements which make this open-source software framework a powerhouse for those needing speed, strength and dependability. Here’s a breakdown of its key features.

Origins of Apache and Hadoop

The Apache Software Foundation is a nonprofit institution set up to provide support to open-source software projects for the common good. This foundation has over 350 open source projects and initiatives that provide different technological solutions. Hadoop is one of these projects. Hadoop is  a Java-based programming environment that is geared to support hte storage of extremely huge data blocks in a distributed computing outfit.

Hadoop was created by Doug Cutting and Mike Cafarella in 2005. Their aim was to return web search results faster by distributing data calculation across different computers so multiple tasks could be accomplished without stress. Another search engine project called Google was also in progress during this period, and both were based on the same concept- storing and processing data in distributed way so that relevant web search result could be returned faster.

In 2006, Cutting joined Yahoo and took with him the Nutch project as well as the idea based on Google’s early work with automating distributed data storage and processing. The Nutch project was divided into two parts; the web crawler portion remained as Nutch while the distributed computing and processing portion became Hadoop, named after a toy elephants of Cutting’s son. In 2008, Yahoo released Hadoop as an open-source project. Today, Hadoop’s framework and ecosystem of technologies are maintained by the non-profit Apache Software Foundation (ASF), a global community of software developers and contributors.

Six FeaturesUnlike the traditional enterprise, Hadoop is known for its power of distributed processing, Hadoop can handle vast volumes of structured and unstructured data more efficiently than the traditional enterprise data warehouse. This means that the initial cost saving is dramatic with Hadoop, while it can continue to grow as the organizational data grows.

  1. EconomicalThe economical nature of Hadoop cannot be over emphasized since it provides huge cost savings. This enables customers to get just the specific elements needed and add nodes as needs increase. Hadoop has the ability of using commodity direct attached storage and share the cost of the network and computers it run on. Its distribution helps in facilitating data transfer rates more rapidly. Hadoop quickly emerged as a foundation for vast data processing thanks to its economical nature.
  2. Solid Ability to Support Map Reduce Workload. In terms of solid ability and supporting Map Reduce Workload, Hadoop is succinct and flexible. Hadoop has the ability to deliver data into the computing substructure at a huge data rate, which is also the requirement of big data workloads. Hadoop without stress can surpass two gigabits per second per computer into the map-reduce layer on a very low cost shared network. This is a feature that other frameworks have problems with, which is a key reason many developers prefer Hadoop.
  3. Fault Tolerance. Web development and coding are intricate areas that need precision. Some platforms through their programming have made it difficult for revision in cases of error, but this is not the case with Hadoop. This outfit has the ability to protect data and application processing from hardware failure. In the event of a node going down, jobs are automatically redirected to other nodes to make sure the distributed computing does not fail and multiples of every single data continue to be stored automatically.In Hadoop, the level of replication is configured and this makes Hadoop a very reliable data system. This is a distinct advantage over other open-source platforms like ASP.NET, whose nodes are lost in the event of accidental rebuts or power off. With Hadoop, data is stored in HDFS where it is replicated in two other locations, and even if one or two of the systems collapses, the file will be available on the third system at least which provides a superior level of fault tolerance.
  4. Flexibility in Data Processing. Flexibility is one feature that Hadoop prides itself on. One of the greatest problems that have plagued platforms even to this date, is the handling unstructured data. Many organizations can only boast of roughly 20% well-structured data, while the rest is unstructured whose values have largely been ignored due to lack of technology to analyze it.With Hadoop, the processing and management of data before storage is a guaranteed. Structured or unstructured, encoded or formatted, using this platform ensures data processing. Hadoop can be trusted in bringing values to the table where unstructured data can be useful in decision making processes.  With Hadoop you can store as much data as you want and decide later on how to use them. Texts, images and videos are included to give users the best orientation. Hadoop is extremely good at high quality volume batch processing because of its ability to do parallel processing. This can be performed through batch processes ten times faster than on a single thread server or on the mainframe.
  5. Hadoop Ecosystem is Strong. Strength is another benefit Hadoop has over other platforms like ASP.NET and some of the others. Hadoop’s Ecosystem is extremely strong and well suited to meet the critical needs of developers of both small to large organizations. Hadoop’s Ecosystem comes with a suite of tools and technologies that enables it to deliver a variety of data processing. This ecosystem comes with an array of projects like Map Reduce, Hive, HBase, Zookeeper, Hcatalog and Apache Pig among others, while more tools continue to be developed and additional technologies added to the ecosystem as the market grows.
  6. Scalability. Scalability is another feature programmers seek in their choice platform since it predetermines the capability of the system and its network. Hadoop is an open source platform and runs on industrial standard hardware. This feature makes Hadoop an extremely scalable platform where new nodes can be easily added. Equally, as data volume of processing needs grow, the user can increase nodes without altering anything in the existing systems or program. With Hadoop, the possibility of growing your system to handle more data is a guaranteed; simply by adding nodes with little administration needed.

Four Essential Components

    1. Hadoop Common:  This is a set of common elements that utilizes and support other Hadoop modules. It contains Java libraries and utilities for other Hadoop modules.
    2. Hadoop Distributed File System (HDFS): HDFS protects data. It is thanks to this feature that the fear of losing important data has become a thing of the past. HDFS also provides reliable data storage and access across all the nodes in a Hadoop Cluster. It is linked to the file systems on many local nodes to create a single file system. Data in Hadoop clusters is broken down into smaller pieces called “blocks” and is distributed out to various nodes in the cluster. In this way, the map and reduce function can be executed on smaller subsets of the larger data set and with this, the provision of scalability that is needed for big data processing are provided. This feature is made possible through the HDFS of Hadoop.

Important Points:

  1. `HDFS stores the files by dividing them into blocks of 64 MB, thus minimizing the search cost.
  2. HDFS uses a MASTER / SLAVE architecture:

  • Name Node: (MASTER), responsible for managing the files and the metadata (information of files, blocks and location of the same in data nodes).
  • Data Node: have the responsibility of storing and retrieving blocks. They form a cluster where the blocks are replicated (by default 3 times) on DataNodes to guarantee fault tolerance.

3. Map Reduce: It is a programming framework of Hadoop suitable for writing applications that process                        large amounts of structured and unstructured data in parallel across a cluster of thousands of machine and it is also called the heart of Hadoop.  Important Points:

  • Job Tracker:
    • Ability to handle job metadata.
    • There is exactly one Job Tracker per cluster.
    • It is the point of interaction between users and the MapReduce framework.
    • Users send MapReduce jobs to the Job Tracker, which puts them in a queue of pending jobs and executes them on the order of arrival.
    • The Job Tracker manages the assignment of tasks and delegates the tasks to the Task Trackers.
  • Task Trackers:
    • Run job requests from Job Trackers
    • Get the code to be executed
    • Applies the specific job configuration.
    • Have communication with the Job Tracker: Sending of the output, completing tasks, updating tasks, etc.
  • Map:
    • Take as input a pair (key, value) and return a list of pairs (key2, value2). This operation is performed in parallel for each input data pair.
    • Next, MapReduce groups all the pairs generated with the same key of all the lists, creating a list by each one of the keys generated.
  • Reduce:
    • This task is performed in parallel taking as input each list of those obtained in the Map and producing a collection of values. The mode of operation is as follows:
      • The data obtained from the Map phase are ordered so that the key-value pairs are contiguous (sort phase).
      • If Reduce runs in distributed mode, the data must first be copied to the local file system in the copy phase. After this process, an addition phase is attached, the file is merged in an orderly fashion. The output will consist of an output file per run task executed.

4. YARN: Yet Another Resource Navigator is the architectural center of Hadoop that allow multiple data processing engines such as interactive SQL, real time streaming, data science and batch processing to handle data storage in a single platform. YARN is a cluster management technology and it is also known as the next generation map reduce which assigns CPU, memory and storage to application running on a Hadoop cluster. It enable application framework other than Map Reduce to run on Hadoop and opening up a wealth of possibilities.

Important Points:

  • Resource Manager (RM): This element is global and takes care of all resource management.
  • Application Master (AM): This element is per application and is responsible for scheduling and monitoring tasks.
  • The Resource Manager and the Node Manager (NM) slave of each node form the working environment. The Resource Manager distributes and manages resources among all applications in the system. The Application Master handles resource negotiation with the Resource Manager and Node Manager to be able to execute and control the tasks, that is, it requests resources to work

ConclusionThe major aspects that intrigue and interest web developers in a platform include flexibility, scalability and the ability to process huge amounts of data on multiple applications. These are all features that Hadoop possesses which has made the platform a great asset to coders. The fact that data can be retrieved and revised easily in the event of error also endears this platform to many. This is unlike other open source platforms which have difficulty rendering processes along with long complicated set up procedures. With Hadoop, the idea is to make programming accessible and enjoyable even to novices. The fact that this platform option is also affordable is just icing on the cake.

HadoopJohn Mathieu

John Mathieu

Hadoop[layerslider id=”11″][layerslider id=”12″]