Skip to content

Spark Network

Abstract

In some scenario you would like to configure your spark punchline network setting. For instance, limitting the range of ports that spark can use...

Follow this documentation for your purpose.

Limiting the number of Ports availables for Spark applications

By default, spark is configured to bind it's spawned process on random ports, be it the driver, blockmanager, driver ui, and so on...

Because of security concerns, this behavior is unacceptable. In most cases, we want to restrict ports avaibles for an application to a finite range.

This can be achieved per Spark Punchline by playing with the below parameters:

  • spark.driver.port
  • spark.driver.blockManager.port
  • spark.blockManager.port
  • spark.port.maxRetries

spark.port.maxRetries takes an integer as value and defines how many available ports will be allowed for spark.driver.port, spark.driver.blockManager.port and spark.blockManager.port where all three of them takes a starting port integer as value.

Behind the scene, spark will try to bind on a defined starting port and increment it by one if the port tried port is already in use until reaching initial defined port + spark.port.maxRetries.