Package org.apache.camel.component.spark

Class SparkEndpoint

  • All Implemented Interfaces:
    AutoCloseable, org.apache.camel.CamelContextAware, org.apache.camel.Endpoint, org.apache.camel.IsSingleton, org.apache.camel.Service, org.apache.camel.ShutdownableService, org.apache.camel.spi.HasId, org.apache.camel.StatefulService, org.apache.camel.SuspendableService
    @UriEndpoint(firstVersion="2.17.0",
             scheme="spark",
             title="Spark",
             syntax="spark:endpointType",
             producerOnly=true,
             category={BIGDATA,IOT},
             headersClass=SparkConstants.class)
public class SparkEndpoint
extends org.apache.camel.support.DefaultEndpoint
    Send RDD or DataFrame jobs to Apache Spark clusters.

    • Field Summary

      • Fields inherited from class org.apache.camel.support.service.BaseService

        BUILT, FAILED, INITIALIZED, INITIALIZING, lock, NEW, SHUTDOWN, SHUTTING_DOWN, STARTED, STARTING, status, STOPPED, STOPPING, SUSPENDED, SUSPENDING

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.camel.Consumer createConsumer​(org.apache.camel.Processor processor)  
      org.apache.camel.Producer createProducer()  
      protected void doInit()  
      SparkComponent getComponent()  
      org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> getDataFrame()  
      DataFrameCallback getDataFrameCallback()  
      EndpointType getEndpointType()  
      org.apache.spark.api.java.JavaRDDLike getRdd()  
      RddCallback getRddCallback()  
      boolean isCollect()  
      void setCollect​(boolean collect)
      Indicates if results should be collected or counted.
      void setDataFrame​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)
      DataFrame to compute against.
      void setDataFrameCallback​(DataFrameCallback dataFrameCallback)
      Function performing action against an DataFrame.
      void setEndpointType​(EndpointType endpointType)
      Type of the endpoint (rdd, dataframe, hive).
      void setRdd​(org.apache.spark.api.java.JavaRDDLike rdd)
      RDD to compute against.
      void setRddCallback​(RddCallback rddCallback)
      Function performing action against an RDD.

      • Methods inherited from class org.apache.camel.support.DefaultEndpoint

        configureConsumer, configureExchange, configurePollingConsumer, configureProperties, createAsyncProducer, createEndpointUri, createExchange, createExchange, createPollingConsumer, doStart, doStop, equals, getCamelContext, getEndpointKey, getEndpointUri, getExceptionHandler, getExchangePattern, getId, getPollingConsumerBlockTimeout, getPollingConsumerQueueSize, hashCode, isAutowiredEnabled, isBridgeErrorHandler, isLazyStartProducer, isLenientProperties, isPollingConsumerBlockWhenFull, isPollingConsumerCopy, isSingleton, setAutowiredEnabled, setBridgeErrorHandler, setCamelContext, setComponent, setEndpointUri, setEndpointUriIfNotSpecified, setExceptionHandler, setExchangePattern, setLazyStartProducer, setPollingConsumerBlockTimeout, setPollingConsumerBlockWhenFull, setPollingConsumerCopy, setPollingConsumerQueueSize, setProperties, toString

      • Methods inherited from class org.apache.camel.support.service.BaseService

        build, doBuild, doFail, doLifecycleChange, doResume, doShutdown, doSuspend, fail, getStatus, init, isBuild, isInit, isNew, isRunAllowed, isShutdown, isStarted, isStarting, isStartingOrStarted, isStopped, isStopping, isStoppingOrStopped, isSuspended, isSuspending, isSuspendingOrSuspended, resume, shutdown, start, stop, suspend

      • Methods inherited from interface org.apache.camel.Endpoint

        getEndpointBaseUri, isSingletonProducer

      • Methods inherited from interface org.apache.camel.Service

        build, close, init, start, stop

      • Methods inherited from interface org.apache.camel.ShutdownableService

        shutdown

      • Methods inherited from interface org.apache.camel.StatefulService

        getStatus, isRunAllowed, isStarted, isStarting, isStopped, isStopping, isSuspending

      • Methods inherited from interface org.apache.camel.SuspendableService

        isSuspended, resume, suspend

    • Method Detail

      • doInit

        protected void doInit()
               throws Exception
        Overrides:
        doInit in class org.apache.camel.support.DefaultEndpoint
        Throws:
        Exception

      • createProducer

        public org.apache.camel.Producer createProducer()
                                         throws Exception
        Throws:
        Exception

      • createConsumer

        public org.apache.camel.Consumer createConsumer​(org.apache.camel.Processor processor)
                                         throws Exception
        Throws:
        Exception

      • getComponent

        public SparkComponent getComponent()
        Overrides:
        getComponent in class org.apache.camel.support.DefaultEndpoint

      • setEndpointType

        public void setEndpointType​(EndpointType endpointType)
        Type of the endpoint (rdd, dataframe, hive).

      • getRdd

        public org.apache.spark.api.java.JavaRDDLike getRdd()

      • setRdd

        public void setRdd​(org.apache.spark.api.java.JavaRDDLike rdd)
        RDD to compute against.

      • setRddCallback

        public void setRddCallback​(RddCallback rddCallback)
        Function performing action against an RDD.

      • getDataFrame

        public org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> getDataFrame()

      • setDataFrame

        public void setDataFrame​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)
        DataFrame to compute against.

      • setDataFrameCallback

        public void setDataFrameCallback​(DataFrameCallback dataFrameCallback)
        Function performing action against an DataFrame.

      • isCollect

        public boolean isCollect()

      • setCollect

        public void setCollect​(boolean collect)
        Indicates if results should be collected or counted.