-
PySpark – Spark Application Lifecycle Overview
PySpark – Spark Application Lifecycle Overview Table Of Contents: Spark Application Starts Driver Program Is Launched Cluster Manager Allocates Resources Job is Created on Action DAG Scheduler Breaks Job into Stages Tasks are Sent to Executors Results Returned to Driver SparkContext Stops / Application Ends (1) Spark Application Start from pyspark.sql import SparkSession spark = SparkSession.builder.appName("MyApp").getOrCreate() We need to first initialize a Spark application to enable distributed data processing with Apache Spark. you are initializing a Spark application. This is the entry point for using Spark. (2) Driver Program Is Launched from pyspark.sql import SparkSession # This runs on the
-
PySpark – Apache PySpark Ecosystem Overview.
PySpark – Apache PySpark Ecosystem Overview. Table Of Contents: SparkContext RDD (Resilient Distributed Dataset) DataFrame Spark SQL SparkSession MLlib Spark Streaming / Structured Streaming GraphX / GraphFrames Data Sources & Integration Deployment & Cluster Management PySpark Libraries (1) Spark Context from pyspark import SparkContext sc = SparkContext("local", "MyApp") (2) RDD (Resilient Distributed Dataset) rdd = sc.parallelize([1, 2, 3, 4]) rdd2 = rdd.map(lambda x: x * 2) (3) DataFrame from pyspark.sql import SparkSession spark = SparkSession.builder.appName("App").getOrCreate() df = spark.read.csv("data.csv", header=True) (4) Spark SQL df.createOrReplaceTempView("people") spark.sql("SELECT * FROM people WHERE age > 30").show() (4) SparkSession spark = SparkSession.builder.appName("App").getOrCreate() (5) MLlib from pyspark.ml.classification
-
PySpark – PySpark Vs Pandas Vs Dask .
PySpark – PySpark Vs Pandas Vs Dask Table Of Contents: PySpark Vs Pandas Vs Dask . Use Case-Based Comparison . Summary . (1) PySpark Vs Pandas Vs Dask (2) Use Case-Based Comparison . (3) Summary
-
PySpark – Why Use PySpark Over Python ?
PySpark – Why Use PySpark Over Python ? Table Of Contents: Why Use PySpark Over Python ? Distributed Computing. Big Data Support. Lazy Evaluation. In Built Fault Tolerance. Support For SQL, ML, Streaming and Graphs. Cluster Deployment. Optimized Engine. (1) Why Use PySpark Over Python ?
-
PySpark – What Is PySpark ?
PySpark – What Is Pyspark ? Table Of Contents: What Is PySpark ? What Is Distributed Computing ? What Happens If I Have A Single Computer With Me How The Task Will Get Distributed ? How Spark Works On Single Core Device ? (1) What Is PySpark ? (2) What Is Distributed Computing ? (3) What Happens If I Have A Single Computer With Me How The Task Will Get Distributed ? (4) How Spark Works On Single Core Device ?
-
PySpark – Syllabus
PySpark – Syllabus Table Of Contents:
-
NLP – BERT Architecture
NLP – BERT Architecture Table Of Contents: Introduction to BERT BERT Architecture Input Representation Pretraining Objectives Fine-Tuning BERT Variants of BERT BERT Evaluation and Benchmarks Advanced Concepts Implementation with Libraries Limitations and Challenges Applications of BERT (1) Introduction To BERT. (2) BERT – Questions What is BERT and the transformer, and why do I need to understand it? Models like BERT are already massively impacting academia and business, so we’ll outline some of the ways these models are used, and clarify some of the terminology around them. What did we do before these models? To understand these models, it’s important to look
-
NLP – Syllabus
NLP – Syllabus Table Of Contents: Introduction To NLP. NLP Tools & Libraries. NLP Data Formats. NLP Pipeline. Text Preprocessing Steps. Regular Expression In NLP. Embedding Techniques. Sequence Modeling. Transformers and Pre-trained Models. Evaluation Metrics Advanced NLP Tasks. Real-World NLP Projects Ethical Considerations (1) Introduction To NLP. What is NLP ? Real-World Applications of NLP . Challenges in NLP . Differences Between NLP, NLU, and NLG . Rule-Based vs Statistical vs Neural NLP (2) NLP Tools & Library NLTK (Natural Language Toolkit) spaCy (Fast, production-ready NLP tasks) TextBlob (Simpler NLP tasks) Gensim (Topic modeling & word embeddings) Flair (Zalando) (Sequence
-
Transformer – Interview Question & Answer
Transformer – Interview Questions Table Of Contents: Basic Understanding Attention Mechanism Model Architecture Training Transformer Model Prediction Transformer Model Interpretability & Optimization Practical/Advanced Concepts Beyond Vanilla Transformers (1) Basic Understanding What is the Transformer model and why was it introduced? What are the key components of a Transformer? What does the term “attention” mean in NLP? How is a Transformer different from an RNN/LSTM? What are the main advantages of Transformers over RNNs? What is positional encoding, and why is it needed in Transformers? Can you explain how a Transformer model works? What are Query, Key, and Value in the
-
Transformers – Syllabus
