Very often, because of multiple reasons, in R&D and in manufacturing we have to conduct experiments. These experiments may be for pure research to see if something at all works, or for Testing or V&V where we test if the system complies with its requirements or for troubleshooting in our assembly line.
In many cases the experiments are very complex and require a lot of effort and resources. In this post I will detail with what are the key factors to consider in order to eventually get the correct experiment planned and executed. Obviously, there are no two experiments that are identical and that being even if sometimes we use the exact same experimental setup for two random experiments. Each experiment has its rounds and angles we must be familiar with in order to achieve a well-executed experiment whose outcome we may rely on.
Experiment Phases – Question, Plan, Execute and Present
There are 4 major stages in every experiment:
Phase 1: Question
This is a very crucial stage which sometimes we skip either due to lack of time or because we think that the objective of the experiment is so obvious and it is not necessary to dwell on. First, if the objective of the experiment is obvious, it shouldn’t be a problem to write it down, should it? Invest the time even in the smallest experiment in the laboratory to jot down a few sentences as to what you are testing and why. Second, having the question posed properly may well save you planning and executing wrongly conceived setups by virtue of the fact that the question was not perfectly clear from the outset and as a result what has to be checked, very much like initial requirements. Obviously, we would like to have more than one objective for the experiment, to kill two birds with one stone so to speak. This is quite alright as long as these objectives can technically live together – a subject that would come up in the planning phase. For Testing or V&V the requirements are the experiment questions. If we have correctly written our System Requirements or Product Requirements we should get the experiment questions for free.
Examples for good experiment questions would be: how much light power in [W] reaches the object if we use a 450-500nm bandpass filter?; if we shut down the chiller do we still have a noise of >10% of the signal?; what is the polarization ratio of the laser in [%]?
Examples for bad experiment questions would be: if we shut down the chiller do we still have the problem? (the question must include the definition what the problem is in this statement); Can we use driver A instead of driver B in our system? (no success criteria here); do we have enough light in our system? (again, enough light has to be defined).
Phase 2: Planning
The planning phase is when we tackle how we answer the question we raised in the first phase. We should state the exact flow of the experiment, the resources needed, the expected time it is going to take and our confidence that the experiment question will be answered. The planning must include success criteria, i.e. for each and every outcome of the experiment we know if it is OK, not OK or needs to fork to another experiment.
See below checklist
Variable & Parameter Isolation & Differentiation
This is the most important bit of the planning phase. If we test a certain parameter variability and its correlation to another parameter, we have to make sure that (a). we plan to check the parameter we intended to and (b). our plan in fact isolates that parameter and does not test multiple variables at once. For example, I tested the impact of linear polarization effect in a certain system that had allegedly been performed before me only to discover that my predecessor had tested the same effect but with circular polarization which had a totally different physical impact.
Gear
What gear is needed for the experiment. Do we have it already? Is it exotic? Expensive? Does it require a special capability that we do not possess in house? Maybe it has a long lead time that requires special attention (yes, we had an experiment with a special lens that originally the quote was for 10 months lead time – we stretched the whole development process differently because of it. Thankfully it went down only to 6 months 
).
Flow
List the sequence or draw a flow chart of it to show exactly what is executed first in the experiment. If you are measuring a lens MTF at best focus, probably this list will have exactly two items however, if the experiment has multiple stages and operations this chart will save you a lot of time and effort.
SW
In small to medium scale experiments we tend to write our own scripts in LabVIEW or Python (or any other scripting language). If, as experiment owners, our core technology is not SW engineering or coding, the SW usually ends up something very limited. There are cases where one or two days of a SW engineer or an Embedded engineer may save us weeks, mainly on repetitive tasks, of effort during the execution of the experiments. Map these operations in your planning, by doing so you will end up much more efficient that way.
Algo & Result Analysis
We sometimes get a very large amount of data as the output of an experiment. Make sure you know how to analyze your results. If there are statistics that are to be made, make sure to plan which statistical analysis tools you are about to use. When planning we should also take into account that we may use some algorithms in situ while the experiment is running. This, of course, makes sense, in order to shorten the duration of the experiment on the one hand and on the other hand having less data for us to analyze at the end of the experiment. If the experiment has a technical specialty related to the result analysis, make sure to consult with them when planning your experiment. For example, testing laser beam diameter would probably require knowing the beam shape of the laser, or other physical parameters that only the Physicist seem to know – have a word with them before going forward with its implementation.
Success Criteria
An experiment is not a product. At the end of each experiment process we should get an answer – each parameter we test should be either numeric (for example if we check a certain laser for its power in mW), OK (threshold), not OK (another threshold), or needs further investigation (try to have these as little as possible). The success criteria must be decided upon before starting.
Human Resources
Who within or outside our team may assist in all the beauty of the experiment and what is their availability. Availability is not a small thing. I have been summoned to assist in many experiments of projects that were not related to me or as an external advisor only because I was available at the time to push the experiment forward.
When we look closely at the checklist above, we may notice that it is a short list of the technical stakeholders of a project, and this is as it should be. See below to the direct relation between a big scale experiment and a full-blown system.
Phase 3: Implementation and Execution
Here there is only one piece of advice – check your experiment setup all the time: check your hardware; check your SW; make sure that none of the assemblies you have is faulty; when you start your day, have a sanity test over the measurement system to make sure it does not need any re-calibration; log in environmental parameters (really hot day, air-con is off etc.) regardless whether you think it is related or not; always timestamp your files and folders so you could have an experiment timeline that would fit the sanity tests that were done. Finally, when possible, have clear experiment notes or an experiment notebook – this last one is a problem, I know, because we sometimes go into the laboratory with our gut feeling and do not write anything. Well, eventually, write something even to yourself, only to be able to retrace your steps afterwards.
And last note about execution, once you have performed everything correctly to your highest standards, trust your results. On that note, I attended a lecture by the Nobel prize winner, Dan Shechtman, where he explained that his discovery of quasicrystals was also measured and observed before and noted in other scientists’ notebooks that did not trust their results and observations.
Phase 4: Presentation
Although this is not officially a formal stage of an experiment, the way we present our experiment results to others and to ourselves may well make the difference between making the right decision and the wrong one. If we were a bit more rigid about experiment execution, we could say that the presentation of our results should be part of the analysis however, I have chosen to have a separate phase for it since in many cases it is difficult to plan how to present our results a priori and we choose how to present the results only after we have seen them.
Take a look at the following charts to see how the same results are presented differently but only one way of presentation really conveys the message. The experiment question is which laser model has the best performance for the range of 40-70mW. The first chart shows the averages of each experiment set while the second one shows all the data points of the all the experiments sets.
Experiment or a Prototype? Or maybe System?
Depends on the experiment question, planning can get to something simple like “Resources: a magnifying glass, an ant and a sunny day. Flow: focus the light on the ant. Success criteria: ant stopped moving for 1 minute = OK, ant keeps running after 1 minute or less = Not OK”.
However, in undismissably high complexity cases the experiments deserve their own System Requirements and System Design. Be mindful of the complexity of the question before you decide to call it “an experiment” or a “prototype” \ “system”. Do not try to keep it small scale when you clearly see that the sub-modules count starts growing, or in other words, if you find yourself having multi-participant meetings twice a week and trying to coordinate half the R&D team for your experiment change the title to a System and act accordingly.
Reviewing Other’s Experiments
There are times when we are sitting on the reviewing side of other’s experiments planning and results. We have to remember to help our teammates to achieve the correct experiment on the one hand, while on the other hand not overruling everything that was done. This is a delicate point where unless the experiment’s execution or planning that has already been done are total errors which is rarely so, then we have to figure out together what conclusions may be extracted from the data at hand and continue forward toward our original goal and answering the questions correctly. BTW, if when you get to the point of experiments that are total errors you have to consider that maybe this particular team member is not a good fit for the team or for that position. Sadly, I have faced such a case and it was a very unpleasant episode.
That’s it for now. Experiments, amongst a lot of other things, are our way of moving technology forwards. We should make sure to set up our experiments right, in a way that we could trust the results and for getting the answers we need for our questions. Alas it is a pity that many experiments have to be repeated because we eventually discover that we’ve missed a parameter in one crucial session that changed the final conclusion to the wrong one, or even worse, finding out at the end of the development process that we got this experiment wrong and now the product is missing a module. I hope this post’s tips will help you plan and execute your next experiment successfully.