SPARK_LOCAL_DIRS parameter in the
/tmp/jaytest. There is one of these in each worker and one in the master.
home/ubuntu/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh, and modified the file to contain this line:
Permissions for each of the
spark-env.sh files are
-rwxr-xr-x, and for the jaytest folders are
As far as I can tell this is in line with all the advice I've read online. However, when I load some data into the cluster it still ends up in
/tmp, rather than
I have also tried setting the
spark.local.dir parameter to the same directory, but also no luck.
Can someone please advise on what I might be missing here?
Edit: I'm running this as a standalone cluster (as the answer below indicates that the correct parameter to set depends on the cluster type).
"this will be overridden by SPARK_LOCAL_DIRS (Standalone, Mesos) or LOCAL_DIRS (YARN) environment variables set by the cluster manager." source - https://spark.apache.org/docs/2.3.1/configuration.html