history
A web server that renders SparkUIs of completed applications.
The recommended way of starting and stopping a HistoryServer is through the scripts start-history-server.sh and stop-history-server.sh.
The recommended way of starting and stopping a HistoryServer is through the scripts start-history-server.sh and stop-history-server.sh. The path to a base log directory, as well as any other relevant history server configuration, should be specified via the $SPARK_HISTORY_OPTS environment variable. For example:
export SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=/tmp/spark-events" ./sbin/start-history-server.sh
This launches the HistoryServer as a Spark daemon.
A web server that renders SparkUIs of completed applications.
For the standalone mode, MasterWebUI already achieves this functionality. Thus, the main use case of the HistoryServer is in other deploy modes (e.g. Yarn or Mesos).
The logging directory structure is as follows: Within the given base directory, each application's event logs are maintained in the application's own sub-directory. This is the same structure as maintained in the event log write code path in EventLoggingListener.