A web server that renders SparkUIs of completed applications.
The recommended way of starting and stopping a HistoryServer is through the scripts start-history-server.sh and stop-history-server.sh.
The recommended way of starting and stopping a HistoryServer is through the scripts start-history-server.sh and stop-history-server.sh. The path to a base log directory is must be specified, while the requested UI port is optional. For example:
./sbin/spark-history-server.sh /tmp/spark-events ./sbin/spark-history-server.sh hdfs://1.2.3.4:9000/spark-events
This launches the HistoryServer as a Spark daemon.
A web server that renders SparkUIs of completed applications.
For the standalone mode, MasterWebUI already achieves this functionality. Thus, the main use case of the HistoryServer is in other deploy modes (e.g. Yarn or Mesos).
The logging directory structure is as follows: Within the given base directory, each application's event logs are maintained in the application's own sub-directory. This is the same structure as maintained in the event log write code path in EventLoggingListener.