Big Data Analytics Revision Guide

Pandas Dataframes & manipulation.
NumPy Numerical computing & arrays.
Matplotlib/Seaborn Statistical visualizations.
PySpark Interface for Spark using Python.
Scikit-learn Machine learning (regression, classification).
TensorFlow/PyTorch Deep learning frameworks.

The modern standard: Combines the flexibility/low cost of a Lake with the performance/ACID transactions of a Warehouse.
• Tech: Databricks, Delta Lake, Apache Iceberg.

Stores data in flexible, JSON-like documents (BSON).

• Dynamic Schema: No predefined structure required.
• Sharding: Distributes data across servers for horizontal scaling.
• GridFS: Used for storing binary files > 16MB (images/videos).

Best For IoT data, content management, social media logs.

1. Input Split: Data is divided into chunks (HDFS blocks).
2. Map Phase: Processes records into intermediate (Key, Value) pairs.
3. Shuffle & Sort: Transfers and organizes data across the network so all values for the same key reach the same reducer.
4. Reduce Phase: Aggregates intermediate values into the final result.

• Combiners: "Mini-reducers" that aggregate data locally on the mapper node to reduce network traffic.
• Speculative Execution: Identifies slow tasks ("stragglers") and launches redundant copies on other nodes to finish faster.
• Partitioners: Custom logic to determine which reducer a key is sent to (prevents data skew).
• Distributed Cache: Efficiently shares small lookup tables or side-data across all mapper nodes.
• Input Formats: Using SequenceFile or Parquet is more efficient than plain TextInputFormat.

Apache Spark is up to 100x faster than Hadoop MapReduce because it processes data directly in RAM across cluster nodes instead of writing intermediate results to disk after every operation.

Allocates physical resources (CPU/RAM) across the cluster. Supports YARN, Mesos, Kubernetes, and Standalone mode.

• Initialization: Always use SparkSession.builder.
• Optimization: Use .filter() early to reduce data size before a shuffle.
• Storage: Use Parquet format for output; it is columnar and significantly faster than CSV.
• Cleaning: Use dropna() or fillna() to handle missing records.

df.filter(col("status") == "Active").groupBy("id").agg(sum("amount"))

High-dimensional data refers to datasets where the number of features (p) is much larger than the number of observations (n). This leads to the "Curse of Dimensionality":

• Overfitting: Models capture noise rather than patterns.
• Sparsity: Data points become isolated, making distance metrics (like Euclidean) less meaningful.
• Computation: Exponentially higher processing requirements.

Spark enables clustering of millions of points through:

• Mini-Batch K-Means: Processes subsets of data at each iteration to save memory.
• K-Means|| (Parallel): An optimized initialization method (similar to K-Means++) that reduces the number of passes over data.
• Feature Scaling: Using StandardScaler is critical because K-Means is distance-based and sensitive to varying scales.

Typical workflow for a scalable ML model:
from pyspark.ml.clustering import KMeans
from pyspark.ml.feature import VectorAssembler, StandardScaler

1. Assemble: Combine columns into a single features vector.
2. Scale: Standardize features to have mean=0 and std=1.
3. Fit: Run KMeans.fit() on the clustered environment.

Contains loops to retain sequential dependencies (hidden state).
• Challenge: Vanishing Gradient problem.
• Solution: LSTM (Long Short-Term Memory) units.
• Best For: NLP, time-series forecasting, speech-to-text.

VAEs learn probabilistic latent representations of data.

• Encoder: Maps input data to a probability distribution (latent space).
• Latent Space: A compressed, continuous bottleneck layer.
• Decoder: Reconstructs data from samples taken from the latent space.

Applications: Image generation, anomaly detection, drug discovery.