Spark: Big Data Cluster Computing in Production

TIPS, TRICKS, AND SOLUTIONS FOR USING SPARK IN PRODUCTION

Spark′s popularity means the field is expanding in terms of both use and capability. Faster than Hadoop and MapReduce, but compatible with Java®, Scala, Python®, and R, this open source clustering framework is becoming a must–have skill. Spark: Big Data Cluster Computing in Production goes beyond the basics to show you how to bring Spark to real–world production environments. With expert instruction, real–life use cases, and frank discussion, this guide helps you move past the challenges and bring proof–of–concept Spark applications live.

Fine–tune your Spark app to run on production data Manage resources, organize storage, and master monitoring Learn about potential problems from real–world use cases, and see where Spark fits best Estimate cluster size and nail down hardware requirements Tune up performance with memory management, partitioning, shuffling, and more Ensure data security with Kerberos Head off Spark streaming problems in production Integrate Spark with Yarn, Mesos, Tachyon, and more

Ilya Ganelin is a data engineer working at Capital One Data Innovation Lab. Ilya is an active contributor to the core components of Apache Spark and a committer to Apache Apex.

Ema Orhian is a Big Data Engineer interested in scaling algorithms. She is the main committer on jaws–spark–sql–rest, a data warehouse explorer on top of Spark SQL.

Kai Sasaki is a software engineer working in distributed computing and machine learning. He is a Spark contributor who develops mainly MLlib, ML libraries.

Brennon York has been a core contributor to Apache Spark since 2014 including development on GraphX and the core build environment.