|
|
Overview
An on-going challenge of today’s test world is
ensuring that test systems are designed correctly and that they
continue to function properly. While developing a test system,
engineers create test procedures and set measurement limits to
correctly detect defective products. Once the test methods have
been determined, the test system is developed to confirm that
the software and hardware performs the tasks correctly. When the
system is completed and implemented, changes might arise in the
company, product, or test environment that must be incorporated
into the test system. Verification and Validation (V&V)
processes formally ensure that the test system is developed
correctly and accomplishes its intended purpose.
V&V primarily affects businesses governed by
ISO or FDA procedures or good practices that manufacture
products such as pharmaceuticals or medical devices, or products
for automotive and aeronautical use. Since such products are
highly critical to health and safety, these industries are
subject to formal oversight, including well-defined V&V
processes. Some companies voluntarily invest in formal V&V
processes to reduce costs, or for competitive reasons. If a
company’s competitiveness is based on quality or reliability,
investing in a rigorous V&V process might pay for itself.
NI TestStand software helps engineers develop
test systems by providing tools to create test sequences that
call code modules written in many different programming
languages. TestStand also helps test engineers manage test
limits and configuration. TestStand components are critical to
any V&V effort.
This document discusses V&V as it applies to
test systems developed with TestStand. First, this document
introduces the concepts of verification, validation, and impact
analysis as applied to TestStand. Second, this document examines
the components of TestStand and describes some best practices to
help streamline V&V for each component. Finally, this document
discusses some general purpose best practices for V&V.
Validation and
Verification
 The
governing principles of V&V are well-defined for many
industries, and are outlined by disciplines like Good
Manufacturing Practices (GMP) or by regulation such as ISO-9000,
FDA's 21 CFR, or IEEE Standards. Each V&V system is
similar but uses slightly different terminology to explain the
generic requirements of the two processes. Specific
requirements are usually not defined. This document
explores V&V processes for automated test systems.
Learning the difference between verification
and validation is an important step in preparing to develop and
certify a test system.
Verification
is one or many formal audits to determine if a test system is
built according to specifications provided in a design, drawing,
statement of work, or other similar guideline.
Verification allows an auditor to test at the end of a
development phase if the item being produced meets the
conditions imposed on it. Verification tests can be
performed at several milestones of a product development, and
can be performed on assemblies and subassemblies.
Validation
is the process to evaluate if the system, having already passed
verification to one or more specifications throughout several
phases, is complete and now accomplishes its purpose or intent.
Validation might include a formal pass/fail test procedure, or
it might be a subjective form of a usability study performed
with customers, users, or patients. Validation often
involves some subjective requirements, such as rejects defective
products or has an easy to use interface.
To illustrate the difference between
verification and validation, suppose a test department builds a
simple test fixture for measuring the electrical current
consumption of a unit under test (UUT). The test system
must compare the current of the UUT current against test limits
that require the UUT to consume less than 500mA. If the
system performs the measurement correctly with acceptable
repeatability, the system passes verification.
If a known failure mode exists in
manufacturing, such as a diode installed in reverse, that causes
some parts of the circuit not to activate, the current
consumption might be excessively low, perhaps 150mA.
Unfortunately, in this example the UUTs pass testing on the
verified fixture, and might be shipped. Though the test
system was built correctly according to the specifications, the
system does not serve the purpose it was commissioned to fulfill
and therefore fails validation. The specification and the
test system must be modified to incorporate an upper and a lower
measurement limit, such as 400-500mA.
Performing system verification can be
relatively easy based on a well-written specification, drawing,
or statement of work, and test methods can be very
straightforward so that defects are easy to find, but validation
can be more challenging, as shown by the previous example.
In comparison, validation requirements that
relate specifically to medical devices in the FDA’s 21 CFR are
vague, including phrases that specify that a medical device must
be validated to conform to user needs and intended uses, so the
quality system manager must define the needs and oversee
validation testing. Selecting the validation methods is
not necessarily very easy. For test systems, one method of
defining a validation test might be as simple as keeping track
of known failure modes, and having good and bad product samples
available to help ensure the system detects known defects.
Another method might use a trusted manual test procedure or
include another automated system to validate the results of the
new test system.
Some instances of validation can be
extraordinarily thorough, such as those that surround the
aeronautics, pharmaceuticals, and medical device industries,
because the validation processes involve extremes cases of
safety, quality, or cost. A thorough system validation
might take weeks or months to define and perform. For
example, if a test system uses a switch matrix in a 16x32
configuration, the test engineer might be required to test every
possible combination of connections using a continuity tester
and ensure that no restricted connections are ever made.
Another example might consist of validating a communication
system in which every possible command and sequence of commands
must be tested and validated. Although such validation
processes might seem extreme, is imperative that no damage,
injury, or incorrect results arise under any circumstance.
No written procedures exist to explain what
must be verified or validated, or to define how testing must be
accomplished. The same is true for reverification or
revalidation if changes are made to a test system. The
organization must appoint someone to make recommendations about
test procedures and review and approve them. Although each
company must decide and define how to implement design controls
and change management in their products and test systems, this
document provides some ideas and best practices to help with
defining such policies.
Impact Analysis
Impact analysis concerns the understanding of
consequences of any change that alters the way in which a test
system performs, such as replacing a failing instrument or
modifying an algorithm or setting. Changes might be due to
maintenance or repair, or trying to improve or correct the
performance of the system. A change might cause the system
to operate improperly in a noticeable or even unnoticeable
manner, and might result in product recalls, production
stoppages, or other interference with business. A change
might also cause the system operate properly but affect the
outcome or test results, and cause incorrect decisions about
tested products.
The cost of a missed change or an incorrect
validation can be extremely large, such as shipping bad products
to customers. In some industries, shipments of faulty
products can result in a product recall. The FDA reserves
the right to take regulatory action. One company reported
that they take extra measures beyond typical V&V processes
because even a tiny reduction in stock price might cost them
billions of dollars.
Assume that every change you make will require
revalidation. No simple rule exists for deciding what you can
change, how you can make changes, or what effects will arise
from any change. Understanding the components of TestStand and
the impact of changes made to the components can help make
changes easier to plan for, easier to recognize, and can
minimize the scope and difficulty of testing the changes.
Components of
TestStand
TestStand is not a traditional programming
language, and therefore possesses distinct advantages that
facilitate V&V processes. TestStand includes a very
modular architecture made up of basic building blocks that allow
your test system to operate however you want. The
following sections describe the TestStand components as methods
to facilitate design decisions to better manage V&V processes.
Sequences & Steps
The most fundamental part of a TestStand
system is a sequence. In a sequence, you define the overall
behavior of the system and the settings and limits applied to
each test. Even in the simplest sequence, much of the work
involves creating and configuring steps in the sequence.
You can add sequencing actions, such as looping properties,
preconditions, and other configurations that directly influence
which steps execute and in what order. For each step, you
can select a number of settings, such as how to record results,
what expressions to evaluate, or what actions to take if the
step fails. Both built-in and custom steps have options
that determine if a step passes or fails, and if the UUT passes
or fails. Some steps determine failure from measurement
limits that you set. For example, for the numeric limit step you
can require that a measurement be within a range.
Changes to the steps in a TestStand sequence
fundamentally modify how a test system operates. Changes
include renaming steps, modifying the preconditions that allow a
step to run, or even deleting an obsolete step. Such
changes might be critical and require revalidation of the
system. Other changes might be somewhat minimal and require
minimal validation. Each step has some amount of
interaction with the steps before and after it.
Determining the effect of a change requires a specific study of
how those steps work together and where each on gets information
it needs to run.
The following are some best practices to help
you make changes to sequence files easier to recognize,
understand, and test:
- Use Case steps instead of
preconditions. Case steps are more visible in the TestStand
environment, are expandable, and can be modified to run
different options and include more than one step per
condition. Preconditions are partially hidden and can only
determine whether to run a single step.
- Use Switch Steps instead of using the
switching options for a step. Switch steps are more visible
in a sequence and make the sequence somewhat more
self-documenting. The switching options for a step are
convenient but are partially hidden and can only be set to
start and stop in relation to a single step. Create written
procedures and TestStand sequences that relate to each other
directly. For example, if a written procedure requires that
you first power on a device and wait for the device to boot,
create two instruction steps in the procedure to correlate
to two steps in TestStand.
- Name every step carefully as a form of
documentation within the sequence. The step name should
describe what the step does and why it performs an action,
if possible. For example, instead of naming a step Wait,
name the step Wait 2 second for system to boot. If the name
requires more information, use the Description property of
the step to specify additional details.
- Organize the sequence so that the Main
sequence is composed almost entirely of Sequence Call
steps. Each Sequence Call step can relate to a single test
that can be given a name, such as Power and Current Test
sequence call followed by the Audio Test sequence call, and
so on for each test in the test procedure. This helps
logically organize the sequence similar to how someone might
describe the sequence verbally, saying “The procedure starts
by doing a power and current test, then it does an audio
test.”
- Create each subsequence to stand alone
and prepare itself to run, including the setup of all
instruments and devices for the subsequence. For example,
if you make changes to the Audio Test, you might want to
run the subsequence over and over before validation to
ensure it runs properly and to avoid having to run the
entire sequence. The test subsequence should power on the
device, set up all necessary switches or instruments, and
execute with noticeably good or bad feedback from the Audio
Test results. If each test runs independently, testing and
retesting can become much easier.
- Avoid using the PreviousStep property
and use variables instead. The PreviousStep propertyrefers
dynamically at runtime to the step that last ran and that
retrieves information from the step. The source of the
information can change if you rearrange or delete steps,
thus resulting in erroneous or missing data. Set a variable
using a the PostExpression property of the step and retrieve
data from the variable as needed.
Code Modules
TestStand does not control instrumentation and
automation hardware directly, but instead calls code modules to
perform these tasks. TestStand relies heavily on the
design and performance of code modules to operate properly.
Code modules are small pieces of code written in a number of
languages, including LabVIEW, C++, C#, and Microsoft Visual
Basic. A test system developer or coworker might create
many of the code modules, while a third party might create other
code modules. Additionally, many steps included with
TestStand are actually code modules written in C and compiled
into DLLs. Changing a code module used in a test sequence
fundamentally affects how TestStand performs actions, but this
can be advantageous because TestStand is a fixed and compiled
application that cannot be changed, and only the code modules
TestStand executes can be modified. Because a code module
must only receive data from TestStand, execute a specific
action, and return data to TestStand, you can limit testing
caused by a code module change.
The following are some best practices for
using code modules:
- Plan ahead when you select parameter
inputs and outputs for the code dodule. Do not rearrange or
rename inputs unless no alternative exists. For example,
when you select a connector pane for a LabVIEW code module,
select a connector pane with extra inputs and avoid changing
the connector pane. Any of these actions might cause the
step to not execute in TestStand or might cause the step to
execute but return results improperly or out of order.
- Design or implement code module testers
that can test all possible inputs and record the output
values for each set of inputs. This can help prove that a
single code module executes correctly as it did before the
change, but you must still prove that new software works
with TestStand, and that it executes and returns correct
results. If the code module passes module testing and can
still be called from TestStand, no change or effect to the
TestStand sequence exists, and you can limit testing.
- Avoid using the Sequence Context to
create variables or to set the value of variables
dynamically. Instead, pass values from code modules to
TestStand using the Module Adapter inputs and outputs to
make setting variables more obvious to see and easy to
track.This also allows you to use the Test Expression button
to verify the datatype or expression and ensure that the
data type or expression evaluates correctlyto help avoid
setting the wrong variable with the wrong data.
Data Types and Process Models
Data types and process models other
characteristic components of TestStand that you can change to
allow advanced programmers to fine tune performance and
behavior. Many users employ these components in the
background without realizing they have done so, and without
modifying the components intentionally. For example, aata
types, such as strings, numbers, and arrays, define the
characteristics of the variables and constants throughout the
system. A process model is a sequence within a sequence
that is generally unseen and that governs how your sequence
starts, executes, stops, and reports data, and manages batch or
parallel testing. You can modify data types and process
models, but leaving them in their default state guarantees
smooth V&V processes. You might think of these components as
being similar to the registry within Microsoft Windows, in which
developers must thoroughly understand changes to avoid
unintended changes to the system. Although some changes to
the process model might seem relatively simple, the complexity
of the changes can affect multiple components in the system,
thus making V&V processes difficult to plan.
The following are some best practices for
using data types and process models:
- Treat changes to the process model as
seriously as you do when you make a registry change or large
operation system change. Consider creating a backup of the
sequence file and process model files, and perhaps a backup
image of the entire hard drive if you might undo the
change. .
- When you create a new data type or
process model, copy an existing version and modify it in
minimal increments, changing some features and testing it at
each phase to ensure that the change occurs as you expected
without many unexpected consequences.
TestStand
Settings
TestStand includes several options and
preferences that you can access from the menus in the TestStand
Sequence Editor or in a TestStand User Interface. Some
settings can be easily changed without any noticeable effects in
operation, and cause no noticeable change in managed or verified
source code files or sequence files, but changes to other
settings might significantly affect the behavior of the system.
One such setting is the database schema that handles how data is
stored in a database, and how TestStand modifies measurements in
preparation for storage. Another setting, which is as
simple enabling a checkbox, can disable all reporting and result
in serious implications to any test system. You must be
aware of what the TestStand settings accomplish, how you intend
to use the settings, and how the settings affect your test
system.
The following are some best practices for
using the options and preferences in TestStand:
- Review all available settings early and
decide how to set them before the system is verified. If
possible, include the settings you want to use in an early
draft of the specification, so that the settings are less
likely to change later. For example, decide what kind of
reports you require and select the appropriate settings.
Try to imagine any changes that you might have to make to
those settings over time. If not conceived early, a change
to a setting would require revalidation. Pay attention to
database and report settings, especially as they relate to
hard drive space or database size. A system that tests 1000
units per day and generates a report for each unit can
become difficult to browse or might fill up the hard drive
on the computer. If a customer must retain all reports but
does not use them on a daily basis, keep the size of the
reports as small as possible. If the customer uses reports
only in case of failure on a test run, but never uses the
reports again, you can instruct TestStand to reuse the
filename and overwrite old reports.
- You can keep the size of reports as
small as possible by turning off reporting for as many steps
as possible. If reporting is activated for every step, test
with just five key measurements might include over one
thousand entries in the database and reports. Many steps,
such as DoWhile, End, For, and Break, do not provide much
useful information in databases or reports. Steps inside
loops write multiple records for each loop. In fact, many
consumers of the reports and database, such as operators and
quality engineers, will likely complain that the reports are
very difficult if they only want to find what failed on a
single test run.
- Use search directories to eliminate the
possibility of finding code in an unexpected place.
- Use relative search paths to locate code
if you decide to move or rearrange folders and files.
Factors External to TestStand
TestStand developers must realize that
numerous factors can affect a test system, including a company's
network configuration, the computer name, or the correct date
and time. Some of the most obvious but often overlooked
settings are those for instruments and the computer itself.
Instrument settings might include NI-DAQmx settings made in NI
Measurement & Automation Explorer (MAX), or GPIB and COM
settings for a device. In Windows, a setting might include
a network drive mapping, or a screen saver setting.
Changes to such settings can interrupt or modify the operation
of a test system and might be partially undetectable, such as an
instrument failing to take valid data, or a motor failing to
move. Such settings can be numerous and validation tests
difficult to design.
The following are some best practices for
handling external factors:
- If you can set any options dynamically,
consider using the Setup step group in a sequence to ensure
the are set correctly.
- If possible, query each setting to
ensure that the setting was accepted by the instrument or
program. If the setting cannot be queried, find where the
setting is stored and read it from a text, INI, or XML
file. The system can verify and even record the state of
items outside of TestStand and keep them under control.
- Find files that contain the settings and
manage them using source code control (SCC).
The preceding sections provided a brief
introduction to the TestStand components that you can change,
resulting in reverification or revalidation of a test system.
Use discipline and best practices to plan ahead and save
unnecessary delay and cost.
General
Best Practices for Validation and Verification
General Best Practices for Validation and
Verification
This section describes more general best
practices that do not apply to any specific TestStand component.
Although not completely exhaustive, the ideas below can help
educate and remind you of items you might want to manage while
developing any test system subject to V&V processes.
Documenting and Gathering Requirements
V&V processes center on well-defined
specifications. Validation can also be subject to some
objective issues, such as loosely defined needs of the
marketplace or the end user. The most important first step
for any test system is to research and document a good working
specification and V&V requirements. Testing can be
difficult if specifications are not specific or leave room for
interpretation or ambiguities. Testing can be halted if
the auditor or observer finds a setting undocumented or
implemented incorrectly. A well-written specification
implemented with care and attention can help ensure a painless
V&V process.
Verification requires one or more design
documents or drawings to govern what the system must accomplish.
The documents and drawings might cover a component, assembly,
or entire system. The specification and test methodology
for verification must be a thoroughly detailed document with as
much information as necessary to create a correct test system.
Thoroughly record any changes to
specifications, whether devised by a customer, by engineers, or
as the result of learning and discovery. Employ a change
order process to record the change and the reason, and to make
the change official. Verification only passes if you match
instructions, settings, and test limits to the correct document.
You might also need to record less obvious changes, such as
Quality Manager agrees to reduce measurement resolution to
reduce test time, which will increase throughput for
Manufacturing Manager. You might want to debate, test,
agree, and release such a change under a change process to
ensure all the changes are correctly implemented.
Designing a validation test might seem like
more of an art than a science, and although wisdom and
experience might seem like the only tools for validation design,
remember that gathering requirements can be revealing and
useful. Techniques might include reviewing past
performance of other test fixtures or products, interviewing
operators and their supervisors, and studying past measurement
data. One company that commonly outsources to test system
integrators performs a detailed review of each project to find
ways to make the next project better, and places those ideas in
a checklist for the next project. .
For example, if you manufacture a product as
simple as a cooling fan for electronic enclosures, compliance
requirements might include electrical emissions testing and
describe the way the emissions must be tested. For the
same product, an audible fan noise might not exist in the
verification spec or test procedure. Assume that a study
of product warranty data reveals that annoyed customers
historically return noisy fans that are too loud in home or
office devices, and the noisy fans came in batches.
Perhaps the test system could perform random sampling of audio
information to save test time while still inspecting the noise
level of the batch. If you treat requirements gathering
with meticulous notes, fact-checking, and curiosity, you can
help design a system that can more easily pass validation.
Take
Advantage of TestStand Modularity
When a system is modular, the system consists
of several modules or components. In comparison to
manufacturing, many parts or subcomponents have their own V&V
processes so that the product can pass through validation with
reduced effort and can incorporate changes and improvements.
For example, in automobiles if an airbag module changes, the
module can undergo some electrical tests to ensure that the next
version of the airbag is functionally equivalent to the
previously validated airbag version. Some non-electrical
requirements, such as speed of deployment, size, shape, and
other factors, can all be verified outside the automobile.
Because the airbag has no effect, for example, on fuel modules
and exhaust systems, no reason exists to reverify or revalidate
the subsystems to approve the vehicle design as a whole. A
revalidation plan might have the fuel system, braking systems,
and others marked as no change but the airbag, the steering
system, and bumper sensors might be marked for a simplified V&V
process. You can apply such modularity to test systems.
In the case of TestStand, you can plan code
modules to be modular in order to streamline V&V processes.
For example, if you have successfully validated version 2.5 of a
sequence and version 1.8 of code moduels, consider the effort
involved to change a measurement limit. If the limit is
hard coded in LabVIEW, you must modify and recompile the code,
which requires you to revalidate the new code, version 1.9.
If you can make the changes outside of the source code, such as
loading limits from a text file, then the validations of
versions 1.8 and 2.5 of your code and sequence file are still
current. You might still need to check that the new text
file is correct, and that it is compatible with the code and
sequence.
Limit
the Interaction between Two Subsystems
One strategy to limit the amount of
revalidation is to include distinct lines of independence in
your system. For example, if you have a medical device
with firmware loaded to it, followed by a number of functional
tests, and a customer-specific file is loaded into memory and
tested before shipping, you can create three separate test
systems that perform these tasks independently of each other.
Changes to one test station do not affect the others. In
the same way you can create a single test system that performs
the work of all three, each of which can be written into its own
sequence file and the sequence files can be called one after the
other from a master sequence file. IN this way you can
easily show which files are modified so you need to only
re-verify the files that change.
Some dependence still exists in the
independent systems, whether they are separate stations or not.
The functional test is dependent on the previous firmware test
working properly. Although they are interdependent, you
want to limit the effects of a change. In this example, the
next step can continue as long as the previous step worked
properly. You can use a step in the functional test to
verify that the loaded firmware is compatible. Because
only one output value affects the next step, validation can be
minimized to ensure that the requirement is met after any
changes.
For example, if test 3 is an Audio Test that
includes a volume test at Low, Medium, and High volume and test
4 performs audio quality testing that must be done at Medium
volume, you might want to power off and reset the audio volume
between tests and design test 4 to set itself to medium volume.
If test 4 is now completely independent on the design and
settings of test 3, a change to test 3 can result in a
revalidation limited to only that single test.
Manage Files Using Source Code Control Tools
Computer-based test systems consist of
numerous files. Source code, sequences, configuration
files, and others change as a test system is developed or
modified.
Code Modules are stored in a format specific
to their platform. For example, LabVIEW generates Virtual
Instrument files called VIs (.vi and control and type Definition
files (.ctl), and organizes those files in LabVIEW project files
(.lvproj). An application written using applications such
as C or Visual Basic includes a number of source files and
compiled DLLs.
TestStand sequence files (.seq) are created
within TestStand and you must manage them in the same way you
manage software code modules. You can also store sequence
files in binary, text, or XML formats. You can organize sequence
files along with your code modules.
Generally, the system designer creates, saves,
and manages the files mentioned in this. One best practice
is to consolidate the files in a single location, or use a
management tool, such as a LabVIEW project or TestStand
workspace, to reference them in various locations. You
must track and archive all files in support of a given
specification.
Files other than those deliberately modified
and collected might include .ini files, XML files, or binary
files that TestStand or another third-party application stores.
For example, TestStand and LabVIEW both use .ini files to store
some configurations and settings made within options and menus.
MAX uses a file designed to be hidden from tampering but that
contains information about your instruments configuration.
You can export most settings from MAX to a backup file (.mce)
for users to manage. Most applications, drivers, or other
tools generate files, and changes to any of these files can
affect V&V processes.
The industry-accepted method for monitoring,
controlling, and storing such files is Source Code Control (SCC)
programs such Subversion (SVN), Perforce, and Microsoft Visual
Source Safe. Many of these programs are designed to
conform to the Microsoft SCC interface and you can use them from
within TestStand or LabVIEW. In some cases, you cannot
modify a file without taking temporary ownership of it and
documenting your changes in order to save them. These
programs can often tell you which files changed as well as
analyze the old and new files to highlight the changes to help
simplify verification. For example, Figures 1 and 2 shows
an example in Subversion, in which only one file,
MySettings.xml, has changed, and in that file only the Sheath
Flow value was modified. Using these programs helps
simplify validation because the programs can demonstrate that
all the other settings have not changed.
Figure 1.
Subversion Showing Only One Modified File in the Working
Directory
Figure 2.
Comparison in Subversion Showing Only One Modified Value in the
File
Verify Files at Run Time
After you complete and verify system
configuration , you must archive the system in such a way that
you can recall and restore, verify, and distribute the system as
the proper specification. For example, if the
MySettings.xml file from the previous example passes V&V along
with a certain TestStand sequence file, you can commit both
files to source code control and submit the files to an
independent document control entity. You might want to
self-audit to verify that the modules are running on the system.
Allow TestStand to verify everything possible during testing,
and even record confirmation of dependant files alongside the
test results in a database or report. For example, you
might accomplish this using a step that can read the checksum of
a file and ensure that it never changes, as shown in Figure 3.
Using file utilities and storing the results along with other
test data creates a complete record that verified files are
being used correctly.
Figure 3.
Third-Party Add-on for TestStand to Verify and Record the
Checksum at Run Time
Hardware Configuration Management
The requirements for hardware configuration
management are no different than for software. You must
properly select, install, program, and configure the instruments
not only for the system but also for each individual test.
For example, a digital multimeter (DMM) or oscilloscope has
several options to configure for proper communication and signal
acquisition that must be verified and validated at the
completion of a test system and for changes to hardware in the
future.
Use software to help validate hardware at run
time. By reading and storing settings or other factors at
run time you can have confidence that items that must be
validated along with the software are set and operating as
intended. For example, your TestStand steps might query
the calibration dates of an instrument to ensure the calibration
is current, and can verify the model number and serial number of
the instrument attached to a COM port to ensure that the
instrument has not been replaced. Designing your test
sequence and even purchasing instruments with these
considerations in mind can help simplify V&V processes.
If you anticipate that your hardware must
change, you must consider the change in a V&V process. If
an instrument fails and another instrument of the same make and
model is inserted, think about what you must accomplish to
verify if it operates correctly, and design a test to ensure
that the change was successful. If the instrument
settings are critical from one instrument to the next, create a
sequence of steps that automatically sets up the instrument when
the sequence detects a new instrument. If another make or
model is required, but the application is hard-coded to use that
instrument, then you cannot avoid revalidation. Using
Interchangeable Virtual Instrument (IVI) drivers and interfaces
for instrument setup can help simplify the transition between
two instruments of the same make or model, or between two
instruments of dissimilar make or model.
Understand
and Manage Software Upgrades
A common question concerns whether you can
upgrade LabVIEW, TestStand, or any other program to take
advantage of new features as they become available. Making
such a software upgrade is always a trigger for a revalidation
and reverification. Treat a potential upgrade as a return
on investment (ROI) exercise. For example, to gain a
streamlined development interface, you might want to upgrade
during development but not after the system is deployed.
However, as is the case with recent TestStand upgrades, improved
execution speed can result in shorter test time, greater
throughput, and greater revenue. In both cases, the cost
of revalidation is the deciding factor, but the cost can also
provide a positive ROI and is therefore worth the expense and
effort.
Another easily overlooked upgrade is Microsoft
Update, which can help to protect a system from attacks and help
repair bugs discovered in Windows. By default, Microsoft
Update occurs automatically on most computers. Other
companies, such as Sun, Apple, and Adobe, also use web-based
automatic updates. Unequivocally, you must disable any
automatic changes and upgrades on any system that is subject to
V&V processes. The changes that automatic updates make are
not predictable and can have unknown effects on operation and
settings. Some updates automatically reboot your computer
after installing.
Your Internet Technology (IT, MIS, or other)
department might have a general policy that they control
computers within the company, including using virus scanning
software, setting security policies like screen savers, and
installing patches and upgrades as needed. A manufacturing
department must work with the IT Department to help manage
TestStand systems by leaving them absolutely untouched.
These computers are not typically used to read email or browse
the Internet and are therefore less susceptible to worms or
viruses. You must decide what items specifically affect
your computers, but your needs might contrast with IT policy,
such as removing virus scanners, turning off screen savers, and
exempting from company-wide upgrades or patches.
Conclusion
Significant challenges cane exist with V&V
processes for any test system. The best practices
described in this document are just a sample of the methodology
that simplifies V&V processes. You must tailor any of
these practices to your company, because some might with your
particular environment. Enforcing some or all of these
disciplines can improve your product quality, yield, rework
efforts, and many other tangible and intangible considerations
in order to reduce costs, turn greater profit, and please
customers.
About
the Author
Joe Spinozzi is the co-founder and Director of
Operations at Cyth Systems. Equipped with 12 years of experience
helping dozens of companies develop products and test systems,
he now leads a team of engineers to plan, architect, and
complete projects across many industries. Operating from
San Diego, California, Cyth Systems is responsible for test
systems large and small currently operating in the United
States, Europe, and Mexico.
back to top
|
|
