Published 2017-05-30.
Time to read: 4 minutes.
I had occasion to speak with a technical manager at a Very Large Corporation at length recently. He wanted advice on where to find Spark programmers. After my usual 20 questions, I realized that his project did not have any requirement for the usual Spark capabilities:
- No need of machine learning or pattern recognition.
- He emphatically did not want to process streaming data.
- He had a rather small volume of data.
- He had no need of interactive analysis.
- There was no need for ETL.
- “Data enrichment” in his case was no more difficult than joining two SQL tables.
- There was no requirement for trigger event detection.
- ... and there was no requirement for session analysis.
This man just wanted to put “I managed the development of an Apache Spark application” on his resume.
Shine Up That Turd
Buffing up a resume with a nonsensical misapplication of the latest software fashion is nothing new. Few people involved with such a project have a positive experience, however. When complex technologies are misapplied they often fail to deliver value, or the project just fails, which contributes to the trough of disillusionment in the Gartner Hype Cycle.
Consider the cynical and selfish world view that such a person must have to engage in this sort of activity. Software professionals are rewarded for being fashionable, and part of the mystique is to say and do things that others don’t fully comprehend. If one keeps moving fast enough, and learns some key phrases to use as put downs that sound impressive but have no meaning when taken out of context (“your approach would likely cause a data race condition”), others are kept off balance long enough for the perpetrator to achieve advantage.
As W.C. Fields said 100 years ago: “If you can’t dazzle them with brilliance, baffle them with bullshit.” He also made a movie entitled Never Give a Sucker an Even Break... but I digress.
Fashion is the Enemy of Delivering Valuable Results on Time
I often remark that the software business is just like the fashion business. The cycle goes like this:
- The latest thing is preannounced. No documentation is available. Big claims are made.
- Early code is provided, but it is slow, very buggy, there is no documentation, and the wild claims continue.
- Seminars pump up the hype. Arm-waving abounds. ‘Experts’ associated with vendors are interviewed.
- A whole new vocabulary is invented to describe the features. However, since those terms are never defined, few people realize that the message is actually “same old stuff with a new coat of paint.” The cool kids learn the words and can parrot them back they way they heard them, but they don't actually know what they are saying. Even if you knew what the phrase “a monad is a monoid in the category of endofunctors” means, it would not help you add value to a business process... and for most programmers who work in the business world, that is what actually pays their bills.
- Gartner, ThoughtWorks and O’Reilly Radar say nice things about the new tech.
- Marketing and sales activity shifts into high gear. Everyone’s laptop has stickers on it that show the owner is hip.
- Some incomplete and mostly useless docs trickle out. Release candidates are eventually made available but it is not obvious how they provide value without significant magic being applied.
- Version 1 finally arrives, completely useless to everyone except those few customers who paid huge dollars to the vendor, so they could get the bragging rights for what is essentially a custom build.
- No useful benchmarks are available. Bugs crawl out from every crevasse. Documentation reads like it was badly translated from Chinese into Bantu, then into Navajo and then English.
- Version 1.1 comes out, to address stop-ship problems that somehow did not prevent version 1 from shipping.
- Version 2 is announced, completely incompatible with Version 1.x, and it is touted as revolutionary.
- The digital lemmings decide that this is the cliff they hold dearest in their hearts, and proceed to jump from it en masse.
- Gartner, ThoughtWorks and O’Reilly Radar downgrade the new tech to “meh”
- ... and the churn continues.
Documentation is not written to be understood, they are written to encourage payments to the digital elite – who have no interest in providing value; they just want to make money from foolish lemmings. This encourages others to aspire to be elitists. The hype cycle only works because our society accepts bullshit, and software has become a cultural phenomenon. We have the software quality and utility that we deserve as a society. If we want better software, we as a society will have to modify our purchasing behavior accordingly.
The software business has often demonstrated an insatiable need to be cool, at the expense of providing little or no value because evidencing value is either too hard for the many posers in the software business, or too boring for those who are capable but jaded.