Authentic Databricks Certification Exam Databricks-Machine-Learning-Professional Questions Answers

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. It allows users to express streaming computations the same way as batch computations on static data, using DataFrame and Dataset APIs. Structured Streaming can handle both unbounded and bounded data sources, and can process data in micro-batches or continuously. Structured Streaming can be used to provide continuous processing of data for machine learning applications, such as data ingestion, preprocessing, feature engineering, and model inference. StructuredStreaming can also integrate with MLflow and Delta Lake to enable end-to-end machine learning pipelines with tracking, reproducibility, and governance123 References:

• Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) - Spark 3.2.0 Documentation
• Machine Learning with Structured Streaming - Databricks
• [Continuous Machine Learning with Structured Streaming and MLflow - Databricks]

Concept drift in machine learning and data mining refers to the change in the relationships between input and output data in the underlying problem over time. In other domains, this change maybe called “covariate shift,” “dataset shift,” or “nonstationarity.” Concept drift can affect the performance and accuracy of predictive models that assume a static relationship between input and output variables. Concept drift can be caused by various factors, such as changes in user behavior, environmental conditions, market trends, etc. Concept drift can be detected and handled by various methods, such as periodic retraining, online learning, ensemble methods, etc References:

• Concept drift - Wikipedia
• A Gentle Introduction to Concept Drift in Machine Learning
• Model Drift & Machine Learning: Concept Drift, Feature Drift, Etc.
• Data Drift vs. Concept Drift: What Is the Difference?

The mlflow.sklearn module provides an API for logging and loading scikit-learn models. To load a model as an sklearn object for batch deployment, the mlflow.sklearn.load_model function can be used. This function takes a model_uri argument that specifies the location of the model to load. The model_uri can be a local file path, a remote file path, or a Databricks or MLflow Model Registry URI. The function returns an sklearn model object that can be used for inference or further training. The function also preserves the model signature and the input example, if they were logged with the model. References:

• mlflow.sklearn Documentation, p. 1-2
• MLflow Models Documentation, Model Signature and Input Example, p. 6-7