Computational Fluid Dynamics (CFD) is a technique used to analyze how fluid (air, water, and other fluids) flow over or inside objects of interest. CFD is a mature field that originated several decades ago and is used in fields of study related to manufacturing, healthcare, the environment, and aerospace and automotive industries that involve fluid flow, chemical reactions, or thermodynamic reactions and simulations. Given the field’s progress and long history, it is beyond the scope of this book to discuss many aspects of this field. However, these video links may be a great way for readers to get up to speed on what CFD is and some CFD tools, and best practices on AWS:
In this chapter, we are going to review the field of CFD and provide insights into how Machine Learning (ML) is being used today with CFD. Additionally, we will examine some of the ways you can run CFD tools on AWS.
We will cover the following topics in this chapter:
You should have the following prerequisites before getting started with this chapter:
In the following section, we will introduce CFD through an example application problem – designing a race car!
CFD is the prediction of fluid flow using numerical analysis. Let’s break that down:
Now, imagine you are the team principal for a new Formula 1 (F1) team who is responsible for managing the design of a new car for the upcoming race season. The design of this car has to satisfy many new F1 regulations that define constraints of how the car can be designed. Fortunately, you have a large engineering team that can manage the design and manufacturing of a new car that is proposed. The largest teams spend millions of dollars on just the conceptual design of the car before even manufacturing a single part. It is typical for teams to start with a baseline design and improve this design iteratively. This iterative improvement of design is not unique to racing car development; think of the latest version of the iPhone in your pocket or purse, or how generations of commercial passenger aircraft designs look similar but are very different. You task your engineers with designing modifications to the existing car using Computer-Aided Design (CAD) tools, and after a month of working on potential design changes, they show you the design of your team’s latest car (see Figure 11.1). This looks great!
Figure 11.1 – F1 car design
However, how do you know whether this car will perform better on track? Two key metrics that you can track are as follows:
Figure 11.2 shows the direction of these two forces applied to the F1 car:
Figure 11.2 – Drag and downforce directions on the F1 car
Now, one way to measure drag and downforce is to manufacture the entire car, drive around a track with force sensors, and report this back to the team – but what if you had a different design in mind? Or a variation in one of the components of your car? You would have rebuilt these components, or the entire car, and then perform the same tests, or run a scale model in a wind tunnel – these options can be very time-consuming and very expensive. This is where numerical analysis codes such as CFD tools become useful. With CFD tools, you can simulate different flow conditions over the car and calculate the drag and downforce.
It is typical in CFD to create a flow domain with the object of interest inside it. This can look similar to Figure 11.3 for external flows (for example, the flow around the vehicle). On the other hand, you could have internal flows where the domain is defined in the object itself (such as the flow inside a bent pipe). In Figure 11.3, the green and blue surfaces represent the inlet and outlet in this domain. Air flows from the inlet, over and around the car, and out through the outlet.
Figure 11.3 – CFD domain defined around the F1 car
The car and the domain so far are conceptual ideas and need to be represented as objects or files that CFD code can ingest and use. A typical file format used to represent objects is the Stereolithography (STL) file format. Each object is represented as a set of triangles, and each triangle is represented by a set of 3D points. The same car in an STL format is shown in Figure 11.4 – the car is now a collection of tens of thousands of triangles.
Figure 11.4 – F1 car in an STL format
We can now use this car object and mesh the CFD domain. Creating a mesh, or meshing, is the process of creating grid points in the CFD domain where numerical equations related to fluid flow are to be solved. Meshing is a very important process, as this can directly influence results, and also sometimes cause the numerical simulation to diverge or not solve.
Note
Meshing techniques and details about the algorithms used are beyond the scope of this book. Each solver tool implements different meshing techniques with various configurations. Teams spend a significant amount of time getting high-quality meshes while balancing the complexity of the mesh to ensure faster solving times.
Once the mesh is built out, it may look similar to Figure 11.5. We see that there is a concentration of grid cells closer to the body. Note that this is a slice of the mesh, and the actual mesh is a 3D volume with the bounds that were defined in Figure 11.3.
Figure 11.5 – CFD mesh built for the F1 car case
Once the mesh is built out, we can use CFD solvers to calculate the flow around this F1 car, and then post-process these results to provide us with predictions for drag and downforce. Figure 11.6 and Figure 11.7 show typical post-processed images involving streamlines (the white lines in the images representing how fluid flows around the body), a velocity slice (the magnitude of the velocity on a plane or cross-section of interest), pressure on the car body (redder regions are higher pressure), and the original car geometry for context.
Figure 11.6 – Post-processed results for the F1 car case showing streamlines and velocity slice
Figure 11.7 shows a different output visualization of pressure on the surface of the car, along with the streamlines in a perspective view.
Figure 11.7 – Post-processed results for the F1 car case showing pressure on the car body with streamlines
In summary, running a CFD case involves the following:
In the next section, we will discuss a few ways of running CFD analyses on AWS according to our documented best practices.
CFD, being very compute-intensive, needs to be scaled massively to be practical for companies that depend on analysis results to make decisions about their product designs. AWS allows customers to run CFD simulations at a massive scale (thousands of cores), on-demand, with multiple commercial and open source tools, and without any capacity planning or up-front capital investment. You can find many useful links related to CFD on AWS here: https://aws.amazon.com/hpc/cfd/.
As highlighted at the outset of this chapter, there are several commercial and open source tools available to solve your CFD problems that run at scale on AWS. Some of these tools are as follows:
In this chapter, we will be providing you with examples of how to set up and use OpenFOAM. For other tools, please take a look at this workshop provided by AWS: https://cfd-on-pcluster.workshop.aws/.
Note
Note that the AWS Well-Architected pillar defines best practices for running any kind of workload on AWS. It includes the best practices for designing architectures on AWS with the following pillars: Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, and Sustainability.
If you are unfamiliar with the Well-Architected Framework, you can read about it in detail here: https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html/.
We will now discuss two different ways of running CFD simulations on AWS: using ParallelCluster and using CFD Direct.
On AWS, these Well-Architected best practices are encapsulated in a solution called AWS ParallelCluster that you can launch in your AWS account. ParallelCluster lets you configure and launch an entire HPC cluster with a simple Command-Line Interface (CLI). The CLI also allows you to dynamically scale resources needed for your CFD (and other HPC) applications as needed, in a secure manner. Popular schedulers such as AWS Batch or Slurm can be used to submit and monitor jobs on ParallelCluster. Here are some steps to follow for installing ParallelCluster (note that a complete set of steps can be found on the official AWS documentation page for ParallelCluster here: https://docs.aws.amazon.com/parallelcluster/latest/ug/install-v3-pip.html).
This helps us have access to a full IDE on the cloud on a specified instance type, with temporary, managed credentials that can be used to launch ParallelCluster. Follow the instructions here to launch an AWS Cloud9 IDE: https://docs.aws.amazon.com/cloud9/latest/user-guide/setup-express.html.
Once you have created your Cloud9 IDE, navigate to the terminal as shown in the instructions here: https://docs.aws.amazon.com/cloud9/latest/user-guide/tour-ide.html#tour-ide-terminal.
Once you are inside the terminal, do the following:
...
python3 -m pip install "aws-parallelcluster" --upgrade –-user
...
...
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash
chmod ug+x ~/.nvm/nvm.sh
source ~/.nvm/nvm.sh
nvm install --lts
node –version
...
pcluster version
{
"version": "3.1.3"
}
Let’s move on to step 3.
Before you launch ParallelCluster, you need to define parameters using the configure command:
... pcluster configure ...
The command-line tool will ask you the following questions for creating a configuration (or config, for short) file:
This creates a config file that can be found in ~/.parallelcluster and modified before the creation of the cluster. Here is an example of a ParallelCluster config file:
... Region: us-east-2 Image: Os: alinux2 HeadNode: InstanceType: c5.4xlarge Networking: SubnetId: subnet-0e6e79abb7ed2452c Ssh: KeyName: pcluster Scheduling: Scheduler: slurm SlurmQueues: - Name: queue1 ComputeResources: - Name: c54xlarge InstanceType: c5.4xlarge MinCount: 0 MaxCount: 4 - Name: m516xlarge InstanceType: m5.16xlarge MinCount: 0 MaxCount: 2 Networking: SubnetIds: - subnet-09299f6d9ecfb8122 ...
A deeper dive into the intricacies of the ParallelCluster config file can be found here: https://aws.amazon.com/blogs/hpc/deep-dive-into-the-aws-parallelcluster-3-configuration-file/.
Once you have verified the config file, use the following command to create and launch ParallelCluster:
... pcluster create-cluster --cluster-name mycluster --cluster-configuration config { "cluster": { "clusterName": "mycluster", "cloudformationStackStatus": "CREATE_IN_PROGRESS", "cloudformationStackArn": "arn:aws:cloudformation:us-east-2:989279443319:stack/mycluster/c6fdb600-d49e-11ec-9c26-069b96033f9a", "region": "us-east-2", "version": "3.1.3", "clusterStatus": "CREATE_IN_PROGRESS" } }...
Here, our cluster has been named mycluster. This will launch a CloudFormation template with the required resources to work with ParallelCluster, based on the config file you previously defined. The following services are used by AWS ParallelCluster:
For more details on the services used, please refer to the links provided in the References section of this chapter. A simplified architecture diagram of AWS ParallelCluster is shown in Figure 11.8 – more details can be found on the following blog: https://aws.amazon.com/blogs/compute/running-simcenter-star-ccm-on-aws/. Otherwise, see the documentation page for ParallelCluster (https://docs.aws.amazon.com/parallelcluster/latest/ug/what-is-aws-parallelcluster.html).
Figure 11.8 – AWS ParallelCluster architecture
The launch will typically take about 10 minutes and can be tracked both on the console as well as on the CloudFormation page on the AWS Management Console. On the console, the following message will confirm that your launch is in progress:
pcluster list-clusters --query 'clusters[?clusterName==`mycluster`]' [ { "clusterName": "mycluster", "cloudformationStackStatus": "CREATE_IN_PROGRESS", "cloudformationStackArn": "arn:aws:cloudformation:us-east-2:<account_number>:stack/mycluster/c6fdb600-d49e-11ec-9c26-069b96033f9a", "region": "us-east-2", "version": "3.1.3", "clusterStatus": "CREATE_IN_PROGRESS" } ]
Wait for the status to say "clusterStatus": "CREATE_COMPLETE"
To install OpenFOAM on your cluster, see the following:
pcluster ssh -n mycluster -i pcluster.pem
The authenticity of host '3.135.195.149 (3.135.195.149)' can't be established.
ECDSA key fingerprint is SHA256:DZPeIcVRpZDg3VMYhA+2zAvEoLnD3gI6mLVkMPkyg90.
ECDSA key fingerprint is MD5:87:16:df:e7:26:f5:a0:da:a8:3a:7c:c4:c8:92:60:34.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '3.135.195.149' (ECDSA) to the list of known hosts.
Last login: Sun May 15 22:36:41 2022
__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|
...
wget https://sourceforge.net/projects/openfoam/files/v2012/OpenFOAM-v2012.tgz
wget https://sourceforge.net/projects/openfoam/files/v2012/ThirdParty-v2012.tgz
...
...
tar -xf OpenFOAM-v2012.tgz
tar -xf ThirdParty-v2012.tgz
...
...
cd OpenFOAM-v2012
export WM_NCOMPPROCS=36
./Allwmake
...
To install OpenFOAM on all nodes, you can use the sbatch command, and submit the preceding commands as a file named compile.sh: for example, sbatch compile.sh.
Once installation completes, you can run a sample CFD application as shown in step 6.
Here, we will run a sample CFD application using ParallelCluster. First, we access the head node of the cluster we just created using SSH:
... pcluster ssh --cluster-name mycluster -i /path/to/keyfile.pem ...
Make sure you use the same .pem file that you created in step 3!
In this case, we will be running an example from OpenFOAM – incompressible flow over a motorbike. The case files for this case can be found here: https://static.us-east-1.prod.workshops.aws/public/a536ee90-eecd-4851-9b43-e7977e3a5929/static/motorBikeDemo.tgz.
The geometry corresponding to this case is shown in Figure 11.9.
Figure 11.9 – Geometry for the motorbike case in OpenFOAM
To run the case in just the head node, you can run the following commands:
cp $FOAM_TUTORIALS/resources/geometry/motorBike.obj.gz constant/triSurface/ surfaceFeatureExtract blockMesh snappyHexMesh checkMesh potentialFoam simpleFoam
We will go into detail about what these commands do in later sections. For now, our aim is just to run the example motorbike case.
To run the same case in parallel with all your compute nodes, you can use sbatch to submit the following shell script (similar to submitting the installation shell script). We can define some input arguments to the script, followed by loading OpenMPI and OpenFOAM:
... #!/bin/bash #SBATCH --job-name=foam #SBATCH --ntasks=108 #SBATCH --output=%x_%j.out #SBATCH --partition=compute #SBATCH --constraint=c5.4xlarge module load openmpi source OpenFOAM-v2012/etc/bashrc cp $FOAM_TUTORIALS/resources/geometry/motorBike.obj.gz constant/triSurface/
First, we mesh the geometry using the blockMesh and snappyHexMesh tools (see the following code):
surfaceFeatureExtract > ./log/surfaceFeatureExtract.log 2>&1 blockMesh > ./log/blockMesh.log 2>&1 decomposePar -decomposeParDict system/decomposeParDict.hierarchical > ./log/decomposePar.log 2>&1 mpirun -np $SLURM_NTASKS snappyHexMesh -parallel -overwrite -decomposeParDict system/decomposeParDict.hierarchical > ./log/snappyHexMesh.log 2>&1
We then check the quality of the mesh using checkMesh, and renumber and print out a summary of the mesh (see code):
mpirun -np $SLURM_NTASKS checkMesh -parallel -allGeometry -constant -allTopology -decomposeParDict system/decomposeParDict.hierarchical > ./log/checkMesh.log 2>&1 mpirun -np $SLURM_NTASKS redistributePar -parallel -overwrite -decomposeParDict system/decomposeParDict.ptscotch > ./log/decomposePar2.log 2>&1 mpirun -np $SLURM_NTASKS renumberMesh -parallel -overwrite -constant -decomposeParDict system/decomposeParDict.ptscotch > ./log/renumberMesh.log 2>&1 mpirun -np $SLURM_NTASKS patchSummary -parallel -decomposeParDict system/decomposeParDict.ptscotch > ./log/patchSummary.log 2>&1 ls -d processor* | xargs -i rm -rf ./{}/0 ls -d processor* | xargs -i cp -r 0.orig ./{}/0
Finally, we run OpenFOAM through the potentialFoam and simpleFoam binaries as shown here:
mpirun -np $SLURM_NTASKS potentialFoam -parallel -noFunctionObjects -initialiseUBCs -decomposeParDict system/decomposeParDict.ptscotch > ./log/potentialFoam.log 2>&1s mpirun -np $SLURM_NTASKS simpleFoam -parallel -decomposeParDict system/decomposeParDict.ptscotch > ./log/simpleFoam.log 2>&1 ...
You can follow instructions in the following AWS workshop to visualize results from the CFD case: https://catalog.us-east-1.prod.workshops.aws/workshops/21c996a7-8ec9-42a5-9fd6-00949d151bc2/en-US/openfoam/openfoam-visualization.
Let’s discuss CFD Direct next.
In the previous section, we saw how you can run a CFD simulation using ParallelCluster on AWS. Now, we will look at how to run CFD using the CFD Direct offering on AWS Marketplace: https://aws.amazon.com/marketplace/pp/prodview-ojxm4wfrodtj4. CFD Direct provides an Amazon EC2 image built on top of Ubuntu with all the typical tools you need to run CFD with OpenFOAM.
Perform the following steps to get started:
Figure 11.10 – CFD Direct AWS Marketplace offering (screenshot taken as of August 5, 2022)
Follow the instructions and get more help on using CFD Direct’s image here: https://cfd.direct/cloud/aws/.
To connect to the instance for the first time, use the instructions given here: https://cfd.direct/cloud/aws/connect/.
In the following tutorial, we will use the NICE DCV client as a remote desktop to interact with the EC2 instance.
To install NICE DCV, perform the following steps:
wget https://d1uj6qtbmh3dt5.cloudfront.net/nice-dcv-ubuntu2004-x86_64.tgz
tar -xvzf nice-dcv-2022.0-12123-ubuntu2004-x86_64.tgz && cd nice-dcv-2022.0-12123-ubuntu2004-x86_64
sudo apt install ./nice-dcv-server_2022.0.12123-1_amd64.ubuntu2004.deb
sudo systemctl start dcvserver
dcv create-session cfd
Figure 11.11 – Connecting to the EC2 instance using a public IP
Figure 11.12 – Entering a username and password for Ubuntu
Figure 11.13 – Ubuntu desktop provided by CFD Direct
echo $FOAM_TUTORIALS
/opt/openfoam9/tutorials
/opt/openfoam9/tutorials/incompressible/simpleFoam/airFoil2D/
The directory is set up like a typical OpenFOAM case and has the following contents (explore using the tree command on Ubuntu):
/opt/openfoam9/tutorials/incompressible/simpleFoam/airFoil2D/ |-- 0 | |-- U | |-- nuTilda | |-- nut | `-- p |-- Allclean |-- Allrun |-- constant | |-- momentumTransport | |-- polyMesh | | |-- boundary | | |-- cells | | |-- faces | | |-- neighbour | | |-- owner | | `-- points | `-- transportProperties `-- system |-- controlDict |-- fvSchemes `-- fvSolution
Let us explore some of these files, as this will give you an understanding of any OpenFOAM case. The folder called 0 represents the initial conditions (as in, time step 0) for these key quantities we will be solving for:
What do these files look like? Let’s take a look at the U (velocity) file:
FoamFile { format ascii; class volVectorField; object U; } dimensions [0 1 -1 0 0 0 0]; internalField uniform (25.75 3.62 0); boundaryField { inlet { type freestreamVelocity; freestreamValue $internalField; } outlet { type freestreamVelocity; freestreamValue $internalField; } walls { type noSlip; } frontAndBack { type empty; } }
As we can see here, the file defines the dimensions of the CFD domain and the free stream velocity, along with the inlet, outlet, and wall boundary conditions.
The Airfoil2D folder also contains a folder called constant; this folder contains files specific to the CFD mesh that we will be creating. The momentumTransport file defines the kind of models to be used to solve this problem:
simulationType RAS; RAS { model SpalartAllmaras; turbulence on; printCoeffs on; }
Here, we use the SpalartAllmaras turbulence model under the Reynolds-Averaged Flow (RAF) type. For more information about this, please visit https://www.openfoam.com/documentation/guides/latest/doc/guide-turbulence-ras-spalart-allmaras.html.
The boundary file inside the polyMesh folder contains definitions of the walls themselves; this is to let the simulation know what a surface inlet or wall represents. There are several other files in the polyMesh folder that we will not explore in this section.
Inside the System folder, the controlDict file defines what applications to run for this case. OpenFOAM contains over 200 compiled applications; many of these are solvers and preprocessing and post-processing for the code.
Finally, we get to one of the most important files you will find in any OpenFOAM case: the Allrun executable. The Allrun file is a shell script that runs the steps we defined earlier for every typical CFD application in order – to import the geometry, create a mesh, solve the CFD problem, and post-process results.
Depending on the output intervals defined in your ControlDict file, several output folders will be output in the same directory that correspond to different time stamps in the simulation. The CFD solvers will solve the problem until they converge, or until a maximum number of time steps is reached. The output folders will look similar to the timestep 0 folder that we created earlier. To visualize these results, we use a tool called ParaView:
Figure 11.14 – Mesh for the Airfoil 2D case in OpenFOAM
Figure 11.15 – Pressure distribution for the Airfoil 2D case in OpenFOAM
Figure 11.16 – Velocity distribution for the Airfoil 2D case in OpenFOAM
Note that these plots are post-processed by initializing ParaView with the paraFoam executable, which automatically understands output formatted by OpenFOAM cases.
Let us now look at a slightly more complicated case – the flow around a car:
Figure 11.17 – Car geometry (perspective view)
Figure 11.18 – Car geometry (side view)
Figure 11.19 – Mesh created for the car case
Figure 11.20 – Streamlines (black) and pressure distribution (perspective view) created for the car case
Figure 11.21 – Streamlines (black) and pressure distribution (side view) created for the car case
The files needed for the following cases can be found in the ZIP files provided in the GitHub repository here: https://github.com/PacktPublishing/Applied-Machine-Learning-and-High-Performance-Computing-on-AWS/tree/main/Chapter11/runs.
In the next section, we will discuss some advancements in the CFD field related to using ML and deep learning with CFD tools.
CFD, being a field that has been around for decades, has matured to be very useful to companies in various domains and has also been implemented at scale using cloud providers. Recent advances in ML have been applied to CFD, and in this section, we will provide readers with pointers to articles written about this domain.
Overall, we see deep learning techniques being applied in two primary ways:
Figure 11.22 – Pressure distribution generated by a trained GAN (left) and CFD (right)
Similarly, Autodesk trained a network with over 800 examples of cars and can instantaneously predict the flow and drag of a new car body: https://dl.acm.org/doi/10.1145/3197517.3201325 (see figure 11.23).
Figure 11.23 – Flow field and drag coefficient being predicted instantaneously for various car shapes
Figure 11.24 – Velocity (top row) and temperature (bottom row) comparisons of OpenFOAM versus SIMNET from NVIDIA
Let’s summarize what you have learned in this chapter next.
In this chapter, we provided a high-level introduction to the world of CFD, and then explored multiple ways to use AWS to solve CFD problems (using ParallelCluster and CFD Direct on EC2). Finally, we discussed some recent advancements connecting the field of CFD to ML. While it is out of the scope of this book to go into much more detail regarding CFD, we hope that the readers are inspired to dive deeper into the topics explored here.
In the next chapter, we will focus on genomics applications using HPC. Specifically, we will talk about drug discovery and do a detailed walk-through of a protein structure prediction problem.