Those curious about what's coming in Apache Spark 1.6 can now get hands-on experience with the latest changes to the big data processing framework.
Databricks, a major commercial contributor to Spark and provider of a cloud-hosted platform for running Spark applications, is offering Spark 1.6 in preview on its services. Those who want to try out Spark on their own can obtain Spark 1.6 pre-release code directly from Apache.
[ Learn how to unlock the power of the Internet of things analytics with big data tools in InfoWorld's downloadable Deep Dive. | Explore the current trends and solutions in BI with InfoWorld's Extreme Analytics blog. ]
More headroom
This new version packs major changes to the way Spark handles memory. Earlier editions of Spark used to subdivide available memory into two partitions, one for data and one for execution, and required users to figure out how much memory to split between the two. This might have been one of the reasons for InfoWorld contributor Ian Pointer's complaints about Spark's memory management.
To read this article in full or to leave a comment, please click here
Read: Spark 1.6 feeds big data's hunger for memory
Previous Topic
