- Large Scale Machine Learning with Spark
- Large Scale Machine Learning with Spark
- Credits
- About the Authors
- About the Reviewer
- www.Packtpub.com
- Preface
- 1. Introduction to Data Analytics with Spark
- 2. Machine Learning Best Practices
- What is machine learning?
- Machine learning tasks
- Practical machine learning problems
- Most widely used machine learning problems
- Large scale machine learning APIs in Spark
- Practical machine learning best practices
- Best practice before developing an ML application
- Good machine learning and data science worth huge
- Best practice – feature engineering and algorithmic performance
- Beware of overfitting and underfitting
- Stay tuned and combining Spark MLlib with Spark ML
- Making ML applications modular and simplifying pipeline synthesis
- Thinking of an innovative ML system
- Thinking and becoming smarter about Big Data complexities
- Applying machine learning to dynamic data
- Best practice after developing an ML application
- Best practice before developing an ML application
- Choosing the right algorithm for your application
- Summary
- 3. Understanding the Problem by Understanding the Data
- 4. Extracting Knowledge through Feature Engineering
- The state of the art of feature engineering
- Best practices in feature engineering
- Feature engineering with Spark
- Advanced feature engineering
- Summary
- 5. Supervised and Unsupervised Learning by Examples
- 6. Building Scalable Machine Learning Pipelines
- Spark machine learning pipeline APIs
- Cancer-diagnosis pipeline with Spark
- Cancer-prognosis pipeline with Spark
- Market basket analysis with Spark Core
- OCR pipeline with Spark
- Topic modeling using Spark MLlib and ML
- Credit risk analysis pipeline with Spark
- Scaling the ML pipelines
- Tips and performance considerations
- Summary
- 7. Tuning Machine Learning Models
- Details about machine learning model tuning
- Typical challenges in model tuning
- Evaluating machine learning models
- Validation and evaluation techniques
- Parameter tuning for machine learning models
- Hypothesis testing
- Machine learning model selection
- Summary
- 8. Adapting Your Machine Learning Models
- 9. Advanced Machine Learning with Streaming and Graph Data
- 10. Configuring and Working with External Libraries
In this Chapter, we applied the basic data manipulations with RDDs, Dataset and DataFrame APIs. We also learn how to do some complex data manipulation through these APIs. We tried to focus on data manipulations, to understand a practical machine learning problem Spam-filtering. In addition to these, we showed how to read the data from different sources. Analyzing and preparing your data to understand the spam filtering as an example.
However, we did not develop any complete machine learning application, since our target was just to show you the basic data manipulation on the experimental Datasets. We intended to develop complete ML application in Chapter 6, Building Scalable Machine Learning Pipelines.
Which features should be used to create a predictive model is not only a vital question but also a difficult question that may require deep knowledge of the problem domain to be answered. It is possible to automatically select those features in data that are most useful or most relevant for the problem someone is working on. Considering these questions, the next chapter covers the feature engineering in detail, explaining the reasons why to apply it along with some best practices in feature engineering. Some topics which are still unclear will be clearer in the next chapter.
-
No Comment