As this ebook edition doesn't have fixed pagination, the page numbers below are hyperlinked for reference only, based on the printed edition of this book.
Symbols
0-switch coverage 172
404 content injectio 266
A
A/B testing
advantages 245
disadvantages 246
accidental load test 348
ad hoc testing 4
Agile approaches 7
alarms 280
alerts
API commands
checking 110
performing 108
asynchronous applications
checking 174
duplicate messages, checking 174
missing response messages, checking 176
missing update messages, checking 175
out-of-order messages, checking 175
attack area 252
discovering 251
authentication
alternative login methods 260
API authentication 259
tests for login sessions 259, 260
automated testing 25
off precision trading, versus brittleness 140-142
B
Barry Boehm’s spiral model 14
beta testing 31
billion laughs attack 264
black box 47
black-box testing
API testing, performing 108
boundary value analysis, using 119, 120
bugs and features, comparing 143
cause-effect graphing, using 124-127
CRUD testing, performing 110, 111
decision tables, using 122-124
dependent variables, mapping 120-122
equivalence partitioning 117, 118
error guessing, optimizing 134-136
hidden defects, uncovering 133, 134
independent variables, mapping 120-122
negative testing, performing 114, 115
new features, enabling 105-108
numeric processing, testing 132, 133
variable types, testing 127
what to check, determining 138-140
worse-case scenarios, identifying 115-117
blue/green systems 294
boundary value analysis
examples 119
buffer overflow attack 263
bug bounty program
running 272
bugs and feature requests
comparing 143
Business Requirement Document (BRD) 32, 33
C
Cassandra 309
cause-effect graphing
Chaos Monkey 320
Chromatic 218
clickjacking 267
client-server applications
load runner architecture 333, 334
client-side errors
raising 285
code analysis
code coverage
calculating 153
condition coverage 156
condition/decision coverage 157
function coverage 153
modified condition/decision coverage 157, 158
multiple condition coverage 159, 160
parameter value coverage 160
test coverage, selecting 163, 164
code inefficiencies 177
communication failure 318, 319
test cases 156
Confidentiality, Integrity, and data Availability (CIA) 54
Continuous Integration/Continuous Delivery (CI/CD) 93
critical alerts 283
Cross-Origin Resource Sharing (CORS) 268
cross-site scripting attacks 263, 264
CRUD testing
creation operations, testing 113
deletion operations, testing 111, 112
read operations, testing 114
update operations, testing 112, 113
CSRF attacks 268
customer
losing 288
customer outcomes
versus system resources 286
D
test cases 156
decision tables
defect hiding 134
Denial of Service (DoS) 358
dependent variable
descriptive tests 90
advantages and disadvantages 89
destructive testing 307
backup and restore testing 321-324
disaster recovery testing 309
DRY (Don’t Repeat Yourself) 149
dual redundant hardware
logging on 295
Dual-Tone Multi-Frequency (DTMF) 271
dynamic load
dynamic testing 70
E
email text input fields
end-to-end testing 79
equivalence partitioning 117, 118
error
predicting 176
error alerts 283
error case testing
expected versus unexpected problems, classifying 183
network degradation 197
prioritizing 192
user feedback, giving on errors 203, 204
error feedback testing
errors, policing 243
errors, preventing 240
internal failures 243
spurious errors 243
spurious errors, avoiding 242
error guessing 25
error handling
philosophies 184
error handling, philosophies
fail as early as possible 185
fail as little as possible 189, 190
error handling philosophies, unexpected failures
errors, catching in backend 186, 187
errors, catching in frontend 185
errors written, catching into storage 187
failing, early in release cycle 185
stored errors, catching 188
events 280
exploratory testing 4
identifying, if feature is ready 16, 17
naivety while testing, using 20, 21
results, checking 24
spiral model, of test improvement 14-16
F
failover 309
failover server
failing 310
failover testing 309
classes of redundancy 309, 310
error reporting 311
system recovery 313
feature specifications
alternatives, examples 31
arguments, countering against 58-60
completing 51
error cases 53
functional test requirements 52, 53
handover, improving from product owner 36, 37
maintainability 54
non-functional tests 55
requirement content, improving 49-51
requirements format 37
requirement’s priority 39
requirement statements, improving 40
round-up, of requirement statement improvements 48, 49
security test requirements 54
turning, into test plan 57, 58
user experience specification 53
feature specifications, alternatives
Business Requirement Document 32, 33
diagrams 35
implicit specifications 34
round-up 36
technical specification 33
test plans 35
user interface mockups 34
feature specifications, requirements format
numbering 38
sections 39
feature specifications, requirement statements
agreed 44
complete 44
consistent 46
precise statement, correcting 41-43
realistic 44
specific 40
feedback
file inputs
validating 264
file storage
testing 265
file uploads
First-In-First-Out (FIFO) 369
flakes 141
full boundary analysis 119
function coverage 153
G
General Data Protection Regulation (GDPR) 270
generic text input fields
Goldilocks zone
graceful shutdown 310
H
hard shutdown 310
hidden defects
host header attacks 267
HTML injection 262
I
idempotent 174
independent variable
injection attacks
code injection 263
cross-site scripting attacks 263, 264
HTML injection 262
SQL injection 262
testing 261
integration tests
International Software Testing Qualifications Board (ISTQB) 25
iPhone
hidden tools 233
iptables 319
issues
debugging 282
L
laxer specification 46
Linux command prompt 230
load testing 139
design 333
errors, filtering 350
inefficiencies, loading 347
issues, debugging 352
messages, loading between modules 347, 348
performance testing 348
startup and shutdown 346
system limits, increasing 335, 336
load testing, considerations
system resource leaks 339, 340
load testing, design
runner architecture, for client-server applications 333, 334
runner architectures 334
runner functions 335
runner interfaces 334
load testing, errors
result sensitivity 351
load testing, operations
combinations, loading 332
dynamic load, versus static load 330, 331
soak testing, versus spikes of load 331
load testing, performance
bottlenecks, identifying 349
in release cycle 349
log 279
searching 295
logging 295
overview 302
using 295
authentication 256
logging, on dual redundant hardware 295
loop coverage 161
M
maintainability
features 276
goals 278
use cases 276
maintainability, goals
issues, debugging 282
system degradation, identifying 280, 281
system, improving 281
tools, for observability 278-280
maintainability testing
maintenance
version 287
maintenance operations
centralized commands 289
client upgrades 292
overview 294
recovery and rollback 293
testing 287
transitions, testing 294
upgrade control, testing 290
upgrade processes, testing 290, 291
upgrade types, testing 289
worst-case 288
Mean Time Between Failure (MTBF) 343
messages
displaying 298
metrics 280
mock 79
modified condition/decision coverage 157, 158
test cases 158
modules
monitoring 282
active checks, performing 285
automatic problem reporting 287
designing 282
hierarchies of system failures 286
instrumentation 283
interacting, ways 282
overview 287
system resources versus customer outcomes 286
multi-factor authentication (MFA) 256
multiple condition coverage 159
test cases 159
Murphy’s law 127
N
naivety while testing
negative testing
network degradation 197
types 197
Network Link Conditioner 200
Network Time Protocol (NTP) 299
non-redundant VPNs 314
numeric processing
numeric text input fields
testing 129
O
observability
observability, data types
events 280
logs 279
metrics 280
traces 279
P
packet loss 284
PacketStorm 200
parameter value coverage 160
password complexity heuristics 130
password text input fields
testing 129
path coverage 163
performance testing 22
personally identifiable information (PII) 265
poetry 255
positive feedback loops
advantages and disadvantages 88
product owner
handover, improving from 36
R
race condition 345
regression testing 23
release cycle
reliability testing 22
Requests For Quotations (RFQs) 46
round-robin 370
S
scripting attacks 22
security
avoiding, through obscurity 273
considering, beyond beyond the application 273
security scans
security testing
security threats 251
Session Initiation Protocol (SIP) 139
shift-left testing 7
Simple Network Management Protocol (SNMP) 95
soak testing 364
versus spikes of load 331
Socratic method 73
Software Requirements Specification (SRS) 29
specification review meetings
black box, opening 72
checklist, reviewing 74
incorrect requirements, fixing 71, 72
informal reviews 66
inspections 66
need for 64
requirement testing, prioritizing 74
scheduling 70
technical reviews 66
walkthroughs 66
spiral model, of test improvement 14-16
stages 15
Spirent 200
SSH File Transfer Protocol (SFTP) 95
staging areas
advantages and disadvantages 92
using 92
state coverage 162
state transition
static load
static testing 70
code format 149
code functionality 151
considerations 148
defensive coding 150
using 148
stress test architecture 357
advantages 356
disadvantages 356
graceful degradation 362
policed limits 360
positive feedback loops, identifying 360, 361
spikes, in load 366
transaction rates, breaching 357-360
unpoliced limits 360
stub 79
subsystem 300
system
improving 281
systematic testing
system degradation
system failures
hierarchies 286
system recovery 313
system resources
versus customer outcomes 286
error reporting 315
system testing 79
system tests
T
temporary test environments
advantages and disadvantages 91
subsets 96
test design 13
Test Driven Development (TDD) 8, 148
test environment
correct version, setting 93, 94
evaluating 91
testing
testing, levels
examples 80
ordering 81
test plan
curiosity and feedback, using 97, 98
time input fields
testing 130
traces 279
Traffic Shaper XP 200
transaction rates
Two-Factor Authentication (2FA) 37
U
unit test coverage
advantages 164
disadvantages 164
unit tests
universally unique identifiers (UUIDs) 271
upgrades 292
usability testing 21
user experience (UX) testing 207
accessibility, ensuring 223-225
bugs, versus feature requests 247
defining 208
documentation, testing 238, 239
error feedback, testing 240
feature specification 211
information display, in tables 226-228
information display overview 229
information display, testing 225, 226
loading 244
notifications, testing 229
scroll bars, using 218
setup, versus ongoing usage 212-214
touch screens, using 219
usability studies, running 244
user interaction, testing 230
user interface, testing 215
visual elements 228
user-facing textboxes
testing 131
user feedback methods
A/B testing 245
instrumentation 246
questionnaires 245
recorded sessions 245
user interaction testing 230
data structure design 237
features, accessing in multiple ways 236
irreversible operations 234
overview 238
restarts 232
user steps combinations, selecting 231
V
variable types
email text input fields, testing 128, 129
file uploads, testing 131, 132
generic text input fields, testing 127, 128
numeric text input fields, testing 129
password text input fields, testing 129, 130
testing 127
time input fields, testing 130
user-facing textboxes, testing 131
VMWare interface 233
W
warning alerts 283
waterfall model 7
web application security, testing 265
404 content injection 266
clickjacking 267
CSRF attacks 268
host header attacks 267
information leakage 266
long password attac 267
webhooks 175
WET (Write Everything Twice) 149
white-box checks 177
code inefficiencies, checking 177
incorrect values, checking 178
white-box functional testing 145
advantages 146
disadvantages 146
white-box testing 21
WinDriver 200
worse-case scenarios
X
XML formatting
fixing 23