BigDL Documentation

---

BigDL makes it easy for data scientists and data engineers to build end-to-end, distributed AI applications. The BigDL 2.0 release combines the original BigDL and Analytics Zoo projects, providing the following features:

• DLlib: distributed deep learning library for Apache Spark

• Orca: seamlessly scale out TensorFlow and PyTorch pipelines for distributed Big Data

• RayOnSpark: run Ray programs directly on Big Data clusters

• Chronos: scalable time series analysis using AutoML

• PPML: privacy preserving big data analysis and machine learning (experimental)

---

Quick Start

• TensorFlow 1.15 Quickstart
• Keras 2.3 Quickstart
• TensorFlow 2 Quickstart
• PyTorch Quickstart
• RayOnSpark Quickstart

User Guide

• Python User Guide
• Scala User Guide
• Colab User Guide
• Docker User Guide
• Hadoop/YARN User Guide
• K8s User Guide
• Databricks User Guide
• Developer Guide

DLlib Overview

• DLlib User Guide
• Keras-Like API
• Spark ML Pipeline Support

Orca Overview

• The Orca Library
• Orca Context
• Distributed Data Processing
• Distributed Training and Inference
• Distributed Hyper-Parameter Tuning
• RayOnSpark

Chronos Overview

• Chronos User Guide
• AutoTSEstimator
• TSDataset and Forecaster
• Anomaly Detector

PPML Overview

• PPML User Guide

Common Use Case

• Use torch.distributed in Orca
• Use Spark Dataframe for Deep Learning
• Use Distributed Pandas for Deep Learning
• Enable AutoML for PyTorch
• Use AutoXGBoost to auto-tune XGBoost parameters

Python API

• Orca API
• AutoML API
• Friesian Feature API
• Chronos API

Real-World Application

• Presentations
• Powered By