public class DatahubClient extends Object
Modifier and Type | Class and Description |
---|---|
static class |
DatahubClient.ShardState
shard的状态
|
Constructor and Description |
---|
DatahubClient(Odps odps,
String projectName,
String tableName,
String datahubEndpoint)
Datahub服务入口类
|
Modifier and Type | Method and Description |
---|---|
String |
getProjectName() |
List<Long> |
getShardList() |
HashMap<Long,DatahubClient.ShardState> |
getShardStatus()
查询DatahubClinet对应的table拥有的shard在服务端的状态
|
TableSchema |
getStreamSchema() |
TableSchema |
getStreamSchemaFromServer() |
String |
getTableName() |
Date |
getTableReplicatedTimeStamp()
返回一个写入datahub的pack的时间戳,datahub确保小于等于这个时间戳的pack都已经进入离线ODPS
|
Date |
getTableTimestamp()
返回一个时间戳,datahub确保小于等于这个时间戳的pack都已经进入离线ODPS
|
void |
loadShard(long shardNumber)
在ODPS hub服务上启用shard
|
DatahubReader |
openDatahubReader(long shardId)
创建DatahubReader读取指定shard
|
DatahubReader |
openDatahubReader(long shardId,
String packId)
创建DatahubReader读取指定shard
|
DatahubWriter |
openDatahubWriter()
创建DatahubWriter
|
DatahubWriter |
openDatahubWriter(long shardId)
创建DatahubWriter写入指定shard
|
PackReader |
openPackReader(long shardId) |
PackReader |
openPackReader(long shardId,
String packId) |
ReplicatorStatus |
QueryReplicatorStatus(long shardId)
在ODPS hub查询非分区表拷贝到离线集群的状态
|
ReplicatorStatus |
QueryReplicatorStatus(long shardId,
PartitionSpec partitionSpec)
Deprecated.
返回的packid不一定是对应partition的, 只能通过比较大小来判断, 不要使用是否相等来判断
|
void |
setEndpoint(String endpoint)
设置DatahubServer地址
|
void |
waitForShardLoad()
同步等待 load shard 完成
默认超时时间为 120000ms
|
void |
waitForShardLoad(long timeout)
同步等待 load shard 完成
最大超时时间为 120000ms
|
public DatahubClient(Odps odps, String projectName, String tableName, String datahubEndpoint) throws OdpsException
odps
- odps对象projectName
- 对应project名称tableName
- 对应table名称datahubEndpoint
- datahub服务地址,公网用户使用 http://dh.odps.aliyun.com,ecs或内网用户请使用 http://dh-ext.odps.aliyun-inc.comOdpsException
public String getProjectName()
public String getTableName()
public void loadShard(long shardNumber) throws OdpsException
shardNumber
- 需要启用的shard数量OdpsException
public void waitForShardLoad() throws OdpsException
OdpsException
public void waitForShardLoad(long timeout) throws OdpsException
timeout
- 超时时间,单位是毫秒
若该值超过 120000ms,将等待 120000msOdpsException
public HashMap<Long,DatahubClient.ShardState> getShardStatus() throws OdpsException, IOException
OdpsException,
- IOExceptionOdpsException
IOException
@Deprecated public ReplicatorStatus QueryReplicatorStatus(long shardId, PartitionSpec partitionSpec) throws OdpsException
shardId
- 需要查询的shardIdpartitionSpec
- 查询的分区,分区表必选, 非分区表可以为nullOdpsException
public void setEndpoint(String endpoint) throws OdpsException
没有设置DatahubServer地址的情况下, 自动选择
endpoint
- OdpsException
public ReplicatorStatus QueryReplicatorStatus(long shardId) throws OdpsException
shardId
- 需要查询的shardIdOdpsException
public Date getTableReplicatedTimeStamp() throws IOException, OdpsException
IOException,
- DatahubExceptionIOException
OdpsException
public Date getTableTimestamp() throws IOException, OdpsException
*
- @return timestampIOException,
- DatahubExceptionIOException
OdpsException
public TableSchema getStreamSchema()
public TableSchema getStreamSchemaFromServer() throws OdpsException
OdpsException
public DatahubWriter openDatahubWriter(long shardId) throws OdpsException, IOException
shardId
- 需要写入数据的shardIdOdpsException,
- IOExceptionOdpsException
IOException
public DatahubWriter openDatahubWriter() throws OdpsException, IOException
OdpsException,
- IOExceptionOdpsException
IOException
public DatahubReader openDatahubReader(long shardId) throws OdpsException, IOException
shardId
- 需要读取数据的shardIdOdpsException,
- IOExceptionOdpsException
IOException
public DatahubReader openDatahubReader(long shardId, String packId) throws OdpsException, IOException
shardId
- 需要读取数据的shardIdpackId
- 指定读取的packIdOdpsException,
- IOExceptionOdpsException
IOException
public PackReader openPackReader(long shardId) throws OdpsException, IOException
OdpsException
IOException
public PackReader openPackReader(long shardId, String packId) throws OdpsException, IOException
OdpsException
IOException
Copyright © 2020 Alibaba Cloud Computing. All rights reserved.