Containerization is emerging as the way forward for the software industry as it brings forth a newer and richer way of building and bundling any kind of software, shipping and running them everywhere. That is the fast-evolving aspect of containerization promises and provides software portability, which has been a constant nuisance for IT developers and administrators for long decades now. The Docker idea is flourishing here because of a number of enabling factors and facets. This section is specially prepared for telling the key use cases of the Docker idea.
Workflows are a widely accepted and used abstraction for unambiguously representing the right details of any complicated and large-scale business and scientific applications and executing them on distributed compute systems such as clusters, clouds, and grids. However, workflow management systems have been largely evasive on conveying the relevant information of the underlying environment on which the tasks inscribed in the workflow are to run. That is, the workflow tasks can run perfectly on the environment for which they were designed. The real challenge is to run the tasks across multiple IT environments without tweaking and twisting the source codes of the ordained tasks. Increasingly the IT environments are heterogeneous with the leverage of disparate operating systems (OSes), middleware, programming languages and frameworks, databases, etc. Typically workflow systems focus on data interchange between tasks and environment-specific. The same workflow, which is working fine in one environment, starts to crumble when it is being migrated and deployed on different IT environments. All kinds of known and unknown dependencies and incompatibilities spring up to denigrate the workflows delaying the whole job of IT setup, application installation and configuration, deployment, and delivery. Containers are the best bet for resolving this imbroglio once for all.
Chao Zheng and Douglas Thain (Integrating Containers into Workflows: A Case Study Using Makeflow, Work Queue, and Docker)has done a good job of analyzing several methods in order to experimentally prove the unique contributions of containers in empowering workflow / process management systems. They have explored the performance of a large bioinformatics workload on a Docker-enabled cluster and observed the best configuration to be locally managed containers that are shared between multiple tasks.
(Douglas M. Jacobsen and Richard Shane Canon) – Currently containers are being overwhelmingly used for the web, enterprise, mobile and cloud applications. However, there are questions being asked and doubts being raised on whether containers can be a viable runtime for hosting technical and scientific computing applications. Especially there are many high-performance computing applications yearning for perfect a deployment and execution environment. The authors of this research paper have realized that Docker containers can be a perfect answer for HPC workloads.
In many cases, users desire to have the ability to easily execute their scientific applications and workflows in the same environment used for development or adopted by their community. Some researchers have tried out the cloud option but the challenges there are many. The users need to solve how they handle workload management, file systems, and basic provisioning. Containers promise to offer the flexibility of cloud-type systems coupled with the performance of bare-metal systems. Furthermore, containers have the potential to be more easily integrated into traditional HPC environments which mean that users can obtain the benefits of flexibility without the added burden of managing other layers of the system (i.e. batch systems, file systems, etc.).
Minh Thanh Chung and the team have analyzed the performance of virtual machines and containers for high-performance applications and benchmarked the results that clearly show containers are the next-generation runtime for HPC applications. In short, Docker offers many attractive benefits in an HPC environment. To test these, IBM Platform LSF and Docker have been integrated outside the core of Platform LSF and the integration leverages the rich Platform LSF plugin framework.
We all know that the aspect of compartmentalization is for resource partitioning and provisioning. That is, physical machines are subdivided into multiple logical machines (virtual machines and containers). Now on the reverse side, such kinds of logical systems carved out of multiple physical machines can be linked together to buildavirtual supercomputer to solve certain complicated problems. Hsi-En Yu and Weicheng Huang have described how they built a virtual HPC cluster in the research paper "Building a Virtual HPC Cluster with Auto Scaling by the Docker". They have integrated the auto-scaling feature of service discovery with the lightweight virtualization paradigm (Docker) and embarked on the realization of a virtual cluster on top of physical cluster hardware.
Csaba Rotter and the team has explored and published a survey article on the title "Using Linux Containers in Telecom Applications". Telecom applications exhibit strong performance and high availability requirements, therefore running them in containers requires additional investigations. A telecom application is a single or multiple node application responsible for a well-defined task. Telecom applications use standardized interfaces to connect to other network elements and implements standardized functions. On top of the standardized functions, a telecom application can have vendor-specific functionality. There is a set of quality of service (QoS) and quality of experience (QoE) attributes such as high availability, capacity, performance / throughput, etc. The paper has clearly laid out the reasons for the unique contributions of containers in having next-generation telecom applications.
Efficient Prototyping of Fault Tolerant Map-Reduce Applications with Docker-Hadoop (Javier Rey and the team) – Distributed computing is the way forward for compute and data-intensive workloads. There are two major trends. Data becomes big and there are realizations that big data leads to big insights through the leverage of pioneering algorithms, script and parallel languages such as Scala, integrated platforms, new-generation databases, and dynamic IT infrastructures. MapReduce is a parallel programming paradigm currently used to perform computations on massive amounts of data. Docker-Hadoop1, a virtualization testbed conceived to allow the rapid deployment of a Hadoop cluster. With Docker-Hadoop, it is possible to control the nodes characteristics and run scalability and performance tests that otherwise would require a large computing environment. Docker-Hadoop facilitates simulation and reproduction of different failure scenarios for the validation of an application.
Interactive Social Media Applications - AlinCalinciuc and the team has come out with a research publication titled as OpenStack and Docker: building a high-performance IaaS platform for interactive social media applications. It is a well-known truth that interactive social media applications face the challenge of efficiently provisioning new resources in order to meet the demands of the growing number of application users. The authors have given the necessary description on how Docker can run as a hypervisor, and how the authors could manage to enable for the fast provisioning of computing resources inside of an OpenStack IaaS using the nova-docker plug-in that they had developed.