- Large Scale Machine Learning with Spark
- Large Scale Machine Learning with Spark
- Credits
- About the Authors
- About the Reviewer
- www.Packtpub.com
- Preface
- 1. Introduction to Data Analytics with Spark
- 2. Machine Learning Best Practices
- What is machine learning?
- Machine learning tasks
- Practical machine learning problems
- Most widely used machine learning problems
- Large scale machine learning APIs in Spark
- Practical machine learning best practices
- Best practice before developing an ML application
- Good machine learning and data science worth huge
- Best practice – feature engineering and algorithmic performance
- Beware of overfitting and underfitting
- Stay tuned and combining Spark MLlib with Spark ML
- Making ML applications modular and simplifying pipeline synthesis
- Thinking of an innovative ML system
- Thinking and becoming smarter about Big Data complexities
- Applying machine learning to dynamic data
- Best practice after developing an ML application
- Best practice before developing an ML application
- Choosing the right algorithm for your application
- Summary
- 3. Understanding the Problem by Understanding the Data
- 4. Extracting Knowledge through Feature Engineering
- The state of the art of feature engineering
- Best practices in feature engineering
- Feature engineering with Spark
- Advanced feature engineering
- Summary
- 5. Supervised and Unsupervised Learning by Examples
- 6. Building Scalable Machine Learning Pipelines
- Spark machine learning pipeline APIs
- Cancer-diagnosis pipeline with Spark
- Cancer-prognosis pipeline with Spark
- Market basket analysis with Spark Core
- OCR pipeline with Spark
- Topic modeling using Spark MLlib and ML
- Credit risk analysis pipeline with Spark
- Scaling the ML pipelines
- Tips and performance considerations
- Summary
- 7. Tuning Machine Learning Models
- Details about machine learning model tuning
- Typical challenges in model tuning
- Evaluating machine learning models
- Validation and evaluation techniques
- Parameter tuning for machine learning models
- Hypothesis testing
- Machine learning model selection
- Summary
- 8. Adapting Your Machine Learning Models
- 9. Advanced Machine Learning with Streaming and Graph Data
- 10. Configuring and Working with External Libraries
In this chapter, we have shown several machine learning applications and tried to differentiate between Spark MLlib and Spark ML. We also showed that it's really difficult to develop a complete machine learning application using only Spark ML or Spark MLlib.
However, we would like to argue that a combined approach, or interoperability, between these two APIs would be best for these purposes. In addition, we learned how to build an ML pipeline by using both Spark ML libraries and how to scale-up the basic model by considering some performance considerations.
Tuning an algorithm or machine learning application can simply be thought of as a process by which one goes through to optimize the parameters that impact the model in order to enable the algorithm to perform at its best (in terms of runtime and memory usage).
In Chapter 7, Tuning Machine Learning Models, we will discuss more about tuning machine learning models. We will try to reuse some of the applications from this chapter and Chapter 5, Supervised and Unsupervised Learning by Examples, to tune up the performance by tuning several parameters.
-
No Comment