Databricks is a web based platform for working with Apache Spark. It provided automated cluster management and iPython-styled notebooks. Azure data bricks is the jointly developed data and AI cloud service from Microsoft and Databricks for data analytics, data science, data engineering and machine learning.
Azure Data Bricks Architecture
Azure Databricks is a cloud service that lets you set up and use a cluster of Azure Instances with Apache Spark Installed with a Master-Worker nodal dynamic(similar to a local Hadoop/Spark cluster).
Benefits of using Azure Databricks:
Optimized Spark Engine – Data processing with auto-scaling and Spark optimized for up to 50x performance gains.
Machine Learning – Pre-configured environments with frameworks such as PyTorch, TensorFlow and sci-kit learn installed.
Mlflow – Track and share experiments, reproduce runs and manage models collaboratively from a central repository.
Benefits of using Databricks Vs SSIS
Exposure of Distribute computing memory.
Cost effective with respect to cluster management.
Scheduling can happen in Databricks UI itself.
Data can be exposed to any cloud platform like AWS/Azure/GCP.
SSIS is licensed with free and paid versions but Databricks is “Pay as you go” plan.
SSIS can handle only structured data but Databricks can handle both structured and unstructured.
Features of Azure Databricks:
Collaborative Notebooks – Quickly access and explore data, find and share new insights and build models collaboratively with the languages and tools of your choice.
Delta Lake – Bring data reliably and scalability to your existing data lake with an open source transactional storage layer designed for the full data lifecycle
Integration with Azure Services – Complete your end-to-end analytics and machine learning solution with deep integration with Azure services such as Azure Data Factory, Azure Data Lake Storage, Azure Machine Learning and Power BI.
Interactive Work spaces – Easy and seamless coordination between Data Analysts, Data Scientist, Data Engineers and Business Analyst to ensure smooth collaboration.
Enterprise Grade Security – Security provided by Microsoft Azure ensures protection of data with storage services and private work spaces.
Production Ready – Easily run, implement and monitor your heavy data-oriented jobs and job-related statistics.