- Large Scale Machine Learning with Spark
- Large Scale Machine Learning with Spark
- Credits
- About the Authors
- About the Reviewer
- www.Packtpub.com
- Preface
- 1. Introduction to Data Analytics with Spark
- 2. Machine Learning Best Practices
- What is machine learning?
- Machine learning tasks
- Practical machine learning problems
- Most widely used machine learning problems
- Large scale machine learning APIs in Spark
- Practical machine learning best practices
- Best practice before developing an ML application
- Good machine learning and data science worth huge
- Best practice – feature engineering and algorithmic performance
- Beware of overfitting and underfitting
- Stay tuned and combining Spark MLlib with Spark ML
- Making ML applications modular and simplifying pipeline synthesis
- Thinking of an innovative ML system
- Thinking and becoming smarter about Big Data complexities
- Applying machine learning to dynamic data
- Best practice after developing an ML application
- Best practice before developing an ML application
- Choosing the right algorithm for your application
- Summary
- 3. Understanding the Problem by Understanding the Data
- 4. Extracting Knowledge through Feature Engineering
- The state of the art of feature engineering
- Best practices in feature engineering
- Feature engineering with Spark
- Advanced feature engineering
- Summary
- 5. Supervised and Unsupervised Learning by Examples
- 6. Building Scalable Machine Learning Pipelines
- Spark machine learning pipeline APIs
- Cancer-diagnosis pipeline with Spark
- Cancer-prognosis pipeline with Spark
- Market basket analysis with Spark Core
- OCR pipeline with Spark
- Topic modeling using Spark MLlib and ML
- Credit risk analysis pipeline with Spark
- Scaling the ML pipelines
- Tips and performance considerations
- Summary
- 7. Tuning Machine Learning Models
- Details about machine learning model tuning
- Typical challenges in model tuning
- Evaluating machine learning models
- Validation and evaluation techniques
- Parameter tuning for machine learning models
- Hypothesis testing
- Machine learning model selection
- Summary
- 8. Adapting Your Machine Learning Models
- 9. Advanced Machine Learning with Streaming and Graph Data
- 10. Configuring and Working with External Libraries
In this chapter, we have shown how to develop large-scale machine learning applications from real-time Twitter stream data and graph data. We have discussed the social network and time-series data analysis. In addition, we also developed an emerging recommendation application by using the content-based collaborative filtering algorithms of Spark MLlib to make movie recommendations for users. These applications, however, can be extended and deployed for other use cases.
It is worth noting that the current implementation of Spark contains a few implemented algorithms for the streaming or network data analysis. However, we can hope that, for example, GraphX will be improved in the future and extended for not only Scala, but for Java, R, and Python too. In the next chapter, we will focus on how to interact with external data sources to make the Spark working environment more diverse.
-
No Comment