链接：https://pan.baidu.com/s/1n5nhrcvdsNScTmjL2g5zmg 

提取码：5dlf 

 

The Spark Streaming integration for Azure Event Hubs provides simple parallelism, 1:1 correspondence between Event Hubs partitions and Spark partitions, and access to sequence numbers and metadata. Table of Contents Linking User Configurations Connection String EventHubsConf Creating a Direct Stream Creating an RDD Obtaining Offests Storing Offsets Checkpoints Your own data store Recovering from Failures with Checkpointing Managing Throughput Deploying Linking For Scala/Java applications using SBT/Maven project definitions, link your application with the following artifact: 1 groupId = com.microsoft.azure 2 artifactId = azure‐eventhubs‐spark_2.11 3 version = 2.3.16 45 or

67 groupId = com.microsoft.azure 8 artifactId = azure‐eventhubs‐spark_2.12 9 version = 2.3.16 For Python applications, you need to add this above library and its dependencies when deploying your application. See the Deploying subsection below. User Configuration Connection String An Event Hubs connection string is required to connect to the Event Hubs service. You can get the connection string for your Event Hubs instance from the Azure Portal or by using the ConnectionStringBuilder in our library. Azure Portal When you get the connection string from the Azure Portal, it may or may not have the EntityPath key. Consider: 1 // Without an entity path 2 val without = "Endpoint=ENDPOINT;SharedAccessKeyName=KEY_NAME;SharedAccessKey=KEY" 34 // With an entity path 5 val with = "Endpoint=sb://SAMPLE;SharedAccessKeyName=KEY_NAME;SharedAccessKey=KEY;Enti tyPath=EVENTHUB_NAME" To connect to your EventHubs, an EntityPath must be present. If your connection string doesn't have one, don't worry! This will take care of it: 1 import org.apache.spark.eventhubs.ConnectionStringBuilder 2

3 val connectionString = ConnectionStringBuilder(without) // defined in the previous code block 4 .setEventHubName("EVENTHUB_NAME") 5 .build ConnectionStringBuilder Alternatively, you can use the ConnectionStringBuilder to make your connection string. 1 import org.apache.spark.eventhubs.ConnectionStringBuilder 23 val connectionString = ConnectionStringBuilder() 4 .setNamespaceName("NAMESPACE_NAME") 5 .setEventHubName("EVENTHUB_NAME") 6 .setSasKeyName("KEY_NAME") 7 .setSasKey("KEY") 8 .build EventHubsConf All configuration relating to Event Hubs happens in your EventHubsConf . To create an EventHubsConf , you must pass a connection string: 1 val connectionString = "YOUR.CONNECTION.STRING" 2 val ehConf = EventHubsConf(connectionString) Please read the Connection String subsection for more information on obtaining a valid connection string. Additionally, the following configurations are optional: