APACHE SPARK OPTIMISATIONS

Context Ambitious goal and the scope of the problem We were chasing a critical, futuristic business requirement for our big data pipelines achieving “1-day planning,” defined as a total execution window of 24 hours or less. The entire process is extensive, processing a 7TB data volume each cycle across 19 distinct plans (16 Spark pipelines and 3 Data Science plans). The sheer size and complexity…

This is an excerpt. The full article continues at Flipkart.