Spark YARN [static] Resource Allocation

Here’s the basic spark-submit command you might be using in production

--packages org.apache.spark:spark-avro_2.11:2.4.0
--class class-name
--master yarn
--deploy-mode cluster
--driver-memory 4g
--num-executors 2
--executor-memory 2g
--executor-cores 2
--files sample_conf.json

tl;dr: skip the reading and calculate the resources

Formula to calculate spark resources

Lifecycle of A Spark Application [Simple Overview]

Before we dive deep, in-a-nutshell, when a lifecycle of a spark application is:

Spark Execution flow
  1. user submits the job
  2. resource manager allocates resource to start the application master (AM)
  3. AM is created simultaneously along with the driver in an YARN container
  4. spark driver interacts with RM to get the available resources on the worker nodes
  5. RM allocates resources
  6. spark driver starts the executors on the worker nodes
  7. executors report back to the driver
  8. driver sends the result back to the client

Memory Management Overview

Using the above resource calculation strategy, you might have the spark.executor.memory value. You need to know how it is being used.

Spark YARN Resource Management

user memory

Memory used for the RDD transformations defined by user. For instance, mapPartitions transformations.

reserved memory 300MB

The memory is reserved for system and is used to store Spark’s internal objects.

spark memory { 60% of (Java Heap — 300MB) }

Further divided into spark.memory.fraction and spark.memory.storageFraction

1. spark.memory.fraction

  • The execution happens here. Memory used for shuffle, sort, aggregate etc..
  • Spilling on disk when exceeds the limit, but the blocks from this pool cannot be forcefully evicted by other threads.

2. spark.memory.storageFraction

  • Responsible for cached data, accumulator, broadcast variables
  • Expresses the size of R as a fraction of M (default 0.5). R is the storage space within M where cached blocks immune to being evicted by execution.

You can evict the block from Storage Memory, but cannot do so from Execution Memory

Food for thought:

● It is more important that you know how to set parallelism. Will be in a separate post

● It is a good point that each JVM-based worker can have multiple “cores” that run tasks in a multi-threaded environment. There are benefits to running multiple executors on a single node (single JVM) to take advantage of the multi-core processing power, and to reduce the total JVM overhead per executor. Obviously, the JVM has to startup and initialize certain data structures before it can begin running tasks.

● spark.cores.max = the maximum amount of CPU cores to request for the application from across the cluster (not from each machine)


A Senior Engineer/Bigdata enthusiast.