Is trial and error good attitude towards problem solving?
A couple of months into the fast paced startup environment right after college, I had plenty of questions on solving problems - about what was the right way to do things. In a startup environment, you are often faced with big problems which are to be solved within a limited time. Which means, you often face a situation where you have to come up with a solution in a time period lesser than what it might take you to actually find and understand an existing solution. "But, how do you know you are right? What if there is a better way, a more strategic way?" - where the questions ringing on my head.
For example, in the startup I previously worked for, we were often faced with problems where you had to classify data among several categories. More or less a spam problem. A naive approach to solving this problem was by simply setting thresholds for values of specific vectors. We adopted this approach because it got things working quickly. We often experimented with these values to ensure there wouldn't be any false positives. We had no idea of what was right or wrong. Very often there were false positives and we would simply tweak the specific value further to get things working. I found this very disturbing.
Sure there were more formal methods to solve this problem like linear regression or Support Vector Machines which could provide better accuracy, but even trying out these models needed you to make heavy assumptions about the data. You had to choose the feature vectors carefully. How do you know which was correct set of vectors? Trial and error (or prior experience of trial and error of course). And this would require time - which you will never have in some startups. But that wasn't the main reason I didn't appreciate this approach - I didn't like trial and error. I wanted to study thoroughly the background and identify the correct model to use for the problem. Which, as it sounds, is very expensive.
There were many similar problems too - Do daily stand-ups help? Are design meetings worth the time? Do we need to document our infrastructure? What were right timeouts to be set on the HTTP clients? What is the advantage of using exponential backoff?
And advice poured in from many directions for problems. If you are startup kid, you a get a lot of advice from the tech community outside (like parents advise their kids). Not all of them were useless advice. But, the one thing that was obvious from all the inputs, is that "it depends". There were too many factors involved. We couldn't directly download a solution like it were a library and simply set the context so that it would work.
If you tried to generalize on a solution for such problems, it looked like trial and error was the only solution that worked. This again, was disturbing. "That is so not the right way", I thought.
So here is what, (I think) I failed to realize back then:
The key idea here is that - with every trial, you find some errors in the assumptions you had made previously. In other words, you understand the problem better. It helps you define the problem more clearly.
It was probably from college, that I picked up the idea of trial and error being ugly. Problems in real life are different from problems in the academic space. In the academic space, problems are well defined. In real life, they aren't. You often have to figure them out or define them yourself. You don't have a standard in which problems are well defined. In such a scenario, I think trial and error is a very good methodology to approach problem solving.
I am not saying trial and error is better than more strategic ways. I am simply saying - trial and error is a good option for defining problems clearly. Post which, more strategic and established solutions would do a great deal in solving the problem more quickly than trial and error. So it wouldn't be fair to rule out trial and error from the "solution tool set". Because, (as some of my mentors have often mentioned) the first step to solving a problem is to ensure that it is well defined.
Similar thoughts:
Tim Harford: Trial, Error and the God complex is almost on similar lines.
A couple of months into the fast paced startup environment right after college, I had plenty of questions on solving problems - about what was the right way to do things. In a startup environment, you are often faced with big problems which are to be solved within a limited time. Which means, you often face a situation where you have to come up with a solution in a time period lesser than what it might take you to actually find and understand an existing solution. "But, how do you know you are right? What if there is a better way, a more strategic way?" - where the questions ringing on my head.
For example, in the startup I previously worked for, we were often faced with problems where you had to classify data among several categories. More or less a spam problem. A naive approach to solving this problem was by simply setting thresholds for values of specific vectors. We adopted this approach because it got things working quickly. We often experimented with these values to ensure there wouldn't be any false positives. We had no idea of what was right or wrong. Very often there were false positives and we would simply tweak the specific value further to get things working. I found this very disturbing.
Sure there were more formal methods to solve this problem like linear regression or Support Vector Machines which could provide better accuracy, but even trying out these models needed you to make heavy assumptions about the data. You had to choose the feature vectors carefully. How do you know which was correct set of vectors? Trial and error (or prior experience of trial and error of course). And this would require time - which you will never have in some startups. But that wasn't the main reason I didn't appreciate this approach - I didn't like trial and error. I wanted to study thoroughly the background and identify the correct model to use for the problem. Which, as it sounds, is very expensive.
There were many similar problems too - Do daily stand-ups help? Are design meetings worth the time? Do we need to document our infrastructure? What were right timeouts to be set on the HTTP clients? What is the advantage of using exponential backoff?
And advice poured in from many directions for problems. If you are startup kid, you a get a lot of advice from the tech community outside (like parents advise their kids). Not all of them were useless advice. But, the one thing that was obvious from all the inputs, is that "it depends". There were too many factors involved. We couldn't directly download a solution like it were a library and simply set the context so that it would work.
If you tried to generalize on a solution for such problems, it looked like trial and error was the only solution that worked. This again, was disturbing. "That is so not the right way", I thought.
So here is what, (I think) I failed to realize back then:
The key idea here is that - with every trial, you find some errors in the assumptions you had made previously. In other words, you understand the problem better. It helps you define the problem more clearly.
It was probably from college, that I picked up the idea of trial and error being ugly. Problems in real life are different from problems in the academic space. In the academic space, problems are well defined. In real life, they aren't. You often have to figure them out or define them yourself. You don't have a standard in which problems are well defined. In such a scenario, I think trial and error is a very good methodology to approach problem solving.
I am not saying trial and error is better than more strategic ways. I am simply saying - trial and error is a good option for defining problems clearly. Post which, more strategic and established solutions would do a great deal in solving the problem more quickly than trial and error. So it wouldn't be fair to rule out trial and error from the "solution tool set". Because, (as some of my mentors have often mentioned) the first step to solving a problem is to ensure that it is well defined.
Similar thoughts:
Tim Harford: Trial, Error and the God complex is almost on similar lines.
