PySpark – PySpark Streaming
Table Of Contents:
- What is Spark Streaming?
- What is Structured Streaming?
- Key Concepts
- Example Code
- Spark Streaming vs. Structured Streaming
- Use Cases
(1) What Is Spark Streaming ?
(2) What Is Structured Streaming ?
(3) Key Concepts:
(4) Example Code
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("StructuredStreamingExample").getOrCreate()
# Read stream from a socket source
df = spark.readStream.format("socket").option("host", "localhost").option("port", 9999).load()
# Word count logic
words = df.selectExpr("explode(split(value, ' ')) as word")
word_counts = words.groupBy("word").count()
# Write the results to the console
query = word_counts.writeStream.outputMode("complete").format("console").start()
query.awaitTermination()
(5) Spark Streaming vs. Structured Streaming
(6) Use Cases
