Introduction

The dictionary definitions for the word ‘control’ have not been updated since before the advent of computing machinery, approximately 80 years ago. While the previous definitions still apply and are valid, they do not provide a nuanced definition suitable for discussing the control of current technology.

These changes took several human generations to evolve; they did not all happen at once. While writing this document, I came to realize that legal precedent from 150 years ago is becoming increasingly relevant. The advent of new technology has brought us full circle. Read on to learn more.

I wrote this article over a period of 4 weeks and have revised it extensively since then as my thinking developed. As with all the articles on this website, I will continue to update this article as new information becomes available. Furthermore, any such definition would be well served by vigorous discussion.

Control, Defined

Let's start the discussion with a few dictionary definitions for ‘control’, as a noun and a verb. This is not meant to be exhaustive.

Lawinsider.com has some definitions, based on analyzing a corpus of documents, however their corpus appears to only have been drawn from corporate law, and the definitions offered are narrow in scope as a result. More than a dozen definitions are offered. The one that seems most applicable to software is:

Control v. Ownership

In a nutshell, the traditional definition of control is influence or authority over, while ownership is defined as the state of having legal title, which means legal control of the status of something. Ownership might require a legal responsibility to provide some measure of control. Responsibility for outcomes might derive from exercising or failing to exercise proper control. Further examination of specific examples might be instructive and might require a restatement of this short definition.

Control is possible without ownership; multiple scenarios are possible, some of which are mentioned in this article.

Pwned

No discussion these days of the word ‘control’ in a software context would be complete without mentioning the word ‘pwned’. Pwned is a portmanteau of the words ‘power’ and ‘owned’ or ‘perfectly owned’. For video gamers and increasingly in the trade press, the word means an opponent has been defeated, or ‘owned’. Listen to this short video to hear how the word is pronounced.

Note the conflation of the concepts of ownership, control, and dominance.

Assessing Control

Feedback

A person generally considers themselves to control a thing, an animal, or a person when they receive feedback appropriate for the action that they took, including their inaction. For example, flicking a light switch should cause lights that are connected to shine. If a light does not shine, the person might consider the switch or the light bulb to be broken, or perhaps the circuit might be suspect.

If for some reason feedback is suppressed or imperceptible, then the person would also likely believe that they were not in control, unless they believed that plausible explanations existed for the lack of feedback. For example, a person with complete color blindness would likely be aware of their inability to perceive color. Similarly, a person with a hearing defect that prevented them from hearing sounds that might indicate a device is responding to their actions would not expect to hear those sounds.

Control systems theory is usually taught to electrical engineers starting in the third year. I'm not going to get into that material here. In case you would like to know more, below is a technical diagram that suggests some basic concepts, including feedback, for closed-loop control systems. The course linked to has more detail.

Learning New Skills

Learning a skill can enhance one’s ability to control something. For example, learning to ride a bicycle or learning to fly a kite. Regarding flying kites, complete control is impossible because the wind cannot be controlled, but learning how to respond to wind changes is an essential part of learning to fly a kite.

Thus, absolute control over some types of things might not be possible, even if some measure of control can be attained or enhanced through practice.

Inert Objects

Scissors are useful for cutting paper. As inert objects, scissors do nothing on their own. The person holding the scissors decides what to cut, when to cut it, and how and where to perform the action. Sharp scissors can be dangerous when handled improperly, when handled by children, or if a properly trained adult attempts to use them when inebriated.

It is easy to determine who controls inert objects like scissors: whoever is holding the scissors has control of them, provided the inert object is not attached to anything else or is subject to physical forces such as magnetism that might affect the object to a greater degree than the person can reliably overcome.

Someone holding the scissors might be conforming to directions provided by another person, or perhaps the scissors were left out in easy reach of children or the public. Any discussion of liability is out of scope for my expertise.

Losing Control

Someone at the top of a tall ski hill covered in several inches of slightly wet snow could make a large snowball. They completely control the large snowball at this point. However, if they push hard and start rolling it downhill, the snowball could grow rapidly as it accelerates, causing injury, death, and destruction. The snowball would be out of control for most of its downward journey.

This example shows that someone could have control over something and then lose control.

Devices With Computational Capability

One major consideration for devices that contain computational elements stems from the engineering practice of layering successively higher functionality, from hardware to firmware, and software; even the software itself is built in layers. Another term, similar to ‘layers’ that is equally applicable for defining control, is ‘hierarchy’.

Devices that contain general-purpose computational elements have a hierarchy of control categories.

Context

The type of device, and the question of whether it contains a general-purpose computational element, provide a contextually dependent meaning for the word ‘control’. This is true whether the word is used as a noun or verb.

Control Categories

It is useful to distinguish between the following categories of control regarding software. This is not an exhaustive list. The preceding dictionary definitions encompass the categories listed below; the following information enhances the definitions shown earlier without contradicting them.

1. Manipulating user interface controls (users manipulating the user interface or providing data, as designed, such that the software provides a benefit to the user). Perception varies between devices, depending on the larger context. This sub-category distinguishes between normal usage of software and the ability to modify the operational parameters of the software.
1. Car drivers and airplane pilots are tasked with controlling their vehicles, even though they just manipulate the user interface controls. Many or most modern vehicles use software to interpret user actions; physical connections between user interface elements and control surfaces are increasingly rare, especially in large vehicles and electric vehicles. Pilots and drivers are required to exercise judgment and maintain situational awareness while operating the vehicle, such that it remains under their control.
2. Data entry clerks also manipulate the user interface controls in order to type in data, but few would argue that they are in control. These people operate in a very narrow context, where situational awareness is not a factor, and no judgment beyond how to interpret the written data is required. There is nothing that a data entry clerk could normally do during their work that might affect the state of the entire system that they interact with.
3. More examples would likely be instructive.
2. Administrative control: Higher privileged users changing the status of regular users and the data contained in the system.
3. Operational control: Installing and maintaining the software in a physical or virtual system, including physically (re)locating the system.
4. Malicious control: Bad actors altering the access privilege of authorized users, granting access to unauthorized users, altering the data in the system, suppressing or changing the inputs to the system. From the point of view of those responsible for the proper operation of the system, they would perceive malicious control of the system as the system being out of control.
5. Social Control: Affecting the perception of a system, such that the behavior of the users of that system is influenced while interacting with it, or the timing of their interaction is influenced, or their desire to interact with the system is suppressed. Perception is reality, in some sense; in fact, controlling people’s perception of a device is as significant as controlling access to the physical device. For example, if a person believes that an angry software god will strike them dead if they touch a sacred keyboard, the priest advocating such nonsense effectively controls the device.

Denying Control

If someone can prevent another from controlling something, is this not itself a form of control? This could be accomplished by the following:

• Breaking or damaging the device, or otherwise rendering it inoperative
• Blocking access to the device
• Masking the device’s responses
• Suppressing the effects of the user’s actions, or redirecting those actions elsewhere
• Turning off the device
• Associating attempts to control the device with negative consequences, for example, electrically shocking anyone who touches the control surface
• … and oh, so many more reasons!

Artificial Intelligence and Machine Learning

Software capable of learning has become commonplace in business software. Machine Learning (ML) learns and predicts based on passive observations by applying sophisticated statistical methods, whereas Artificial Intelligence (AI) implies an agent interacting with the environment to learn and take actions that maximize its chance of successfully achieving its goals.

There is controversy in the software community on whether ML is a subset of AI or a separate field. For the purposes of this introductory discussion, this is a distinction without a difference; however, one should be aware of the inconsistency of terminology when reading literature. The distinctions might take on more significance in a more advanced discussion, however, depending on the topic. By late 2021, most AI installations were, in fact, ML installations.

In 2020, “50% of respondents reported that their companies have adopted AI in at least one business function.” Note that McKinsey’s report uses the definition that ML is a subset of AI, and in fact, their cited usages of AI are almost exclusively examples of ML.

 –From McKinsey & Company: The State of AI in 2020

Magical Results

ML systems must be trained before they can be used. The training data fed into an ML system defines its future responses.

Although many research papers from 2021 discuss ways to show how results by ML systems could be explained, the reality is that most of these systems currently have no way to explain their results; they operate as a black box, and they are vulnerable to learning unstated bias introduced during initial training.

“Companies increasingly manage risks related to AI explainability… AI high performers remain more likely than others to recognize and mitigate most risks. For example, respondents at high performers are 2.6 times more likely than others to say their organizations are managing equity and fairness risks such as unwanted bias in AI-driven decisions.”

 –From McKinsey & Company: The State of AI in 2020

Magicians seem to perform magic because they do not explain their amazing results. For the ML systems that cannot explain the rationale behind an output or state change, could anyone be said to have complete control over them?

Again, more could be said on this topic, but I will save that discussion for another time.

Unacknowledged Bias

Is AI Impossible To Control?

Completely autonomous AI is upon us, and many well-informed technologists are gravely concerned that such a thing cannot be controlled.

It is instructive to consider a parent’s often futile desire to control their teenage offspring. When teenagers are alone with their friends, the best parents can hope for is that their children were provided good examples while growing up, and were shown how to deal with peer pressure effectively.

ML systems are much the same; the results they produce are due to their training. ML systems cannot be micromanaged to produce correct results.

On March 10, 2022 the US eliminated the human controls requirement for fully automated vehicles. Whom or what controls the vehicle? Can a thing be held accountable for its actions?

Examples

Close analogs exist in the fields of enterprise software, particularly e-commerce, enterprise resource planning (ERP), and software-as-a-service (SaaS).

Examples include recommender systems, image recognition and generation, speech recognition and generation, traffic prediction, weather prediction, email filtering, security, fraud detection, dynamic pricing, and much more.

Following are two examples, drawn from the physical world, with the intention that most readers would find them relatable.

Vacuum Cleaners

The concept of controlling a classically constructed device, such as a traditional vacuum cleaner, differs from controlling a device with a general-purpose computational element, such as the iRobot Scooba® floor washing robot, first available in 2005.

The older vacuum cleaner would only have an on/off switch, perhaps a power selector, and perhaps the ability to adjust various optional attachments. The operator would be able to control where they place the vacuum head by lifting and placing the head in the location where they want to clean.

In contrast, consider a robotic cleaner, such as the 2005 iRobot Scooba® wet vacuum cleaner pictured above. Robotic cleaners find their path to clean autonomously. This type of device usually has sensors that cause them to employ various cleaning strategies when it encounters something that triggers a specific cleaning algorithm.

Clearly, the finer points of the concept of ‘controlling’ a vacuum cleaner vary depending on the nature of the device.

Transportation

The Ford Model T was named the most influential car of the 20th century by the 1999 Car of the Century competition. No one alive today had yet been born when the Model T was first offered for sale in 1908, so here are a few reminders of what technology was like at that time.

The Model T had no battery. The only use of electricity by early versions of this car was for the 4 spark plugs, and they received power produced by a flywheel magneto system. Acetylene gas lamps were used for headlights, and kerosene lamps were used for side and tail lights.

Controlling a Ford Model T was somewhat different from modern cars: once the Model T was started, drivers could set the throttle with their right hand, press on the brake, clutch and reverse pedals, turn the steering wheel, and operate the hand brake. This diagram is from the Model T Ford Club of America:

Cadillac began offering cars with push-button starters in 1912, but few people could afford such a luxury car; instead, the Model T was started with a hand crank. Power steering would not be invented for several years, so the steering wheel was mechanically connected to the steering mechanism for the front wheels. There were no power windows; in fact, early Model Ts did not even have windows in the doors; this could be uncomfortable in bad weather.

Transistors were invented decades later, in 1956, and vacuum tubes were still exotic and expensive in 1918. Consequently, car radios did not appear for another decade, and they would use vacuum tubes until the mid-1960s. Cars would not be mass-produced with automatic transmissions until General Motors introduced the Hydramatic three-speed hydraulic automatic in 1939.

In contrast, the average vehicle in 2022 contains about 50 interconnected microprocessors, and most of those vehicles also have at least one camera. Vehicles from various manufacturers use multiple cameras to park themselves autonomously, for both parallel and regular perpendicular parking.

Some vehicles in 2023 also feature assisted reversing, which assumes steering control to mirror the path the vehicle most recently took going forward. This system makes backing out of a confined parking place easy. All the driver has to do is operate the accelerator and brakes and monitor the surrounding area, while the steering follows the exact path the car took to enter the space.

When using either of these modern features, what controls the car when parking or reversing? The car continually uses its many sensors and internal guidance system to make course corrections. All the driver does is indicate their intent, and the only action they can take is to control the speed, pause or abort the procedure. This seems rather similar to how a rider controls a horse, yet differences do exist.

Clearly, the finer points of the concept of controlling a car differ depending on the nature of the car, and the last word has yet to be spoken on this matter.

Six Levels of Vehicle Autonomy

The Society of Automotive Engineers (SAE) defines six levels of driver assistance technology. These levels have been adopted by the U.S. Department of Transportation (US DOT). The following is taken from the US National Highway Traffic Safety Administration (NHTSA), which is a division of the US DOT:

Autonomous Cars

Self-driving cars have analogs in machine learning systems, which are increasingly incorporated into enterprise software.

Tesla cars were advertised as being designed to provide self-driving features; however, when this was written, many of those features were not enabled yet for all customers, and it seemed uncertain if all of them would ever be enabled. Those advanced features include autopilot, autosteer, smart summon, full self-driving, taking direction from a calendar instead of a human, and self-parking.

Tesla’s Autopilot feature is classified as Level 2 vehicle autonomy, which means the vehicle can control steering and acceleration, but a human in the driver’s seat can must be able to take control at any time.

Regardless of whether the claims made by Tesla are supportable, the concept of controlling such a vehicle is nuanced. When I read articles like Tesla Recalls 362,758 Vehicles Over Full Self-Driving Software Safety Concerns, I wonder if litigation will arise.

Tesla Autopilot Full Self-Driving Hardware from Tesla.

More could be said regarding autonomy and context, but this serves to introduce the topic.

Update 2023-11-01 Tesla Wins Another Lawsuit

Tesla wins first US Autopilot trial involving fatal crash. I am sure we have not heard the last of this issue.

Update 2023-12-06 GM’s Cruise Dismisses 900 Employees, Cutting 24% of Workforce

After several high-profile accidents, California regulators suspended Cruise’s license to operate as authorities accused the company of hiding details. BNN Bloomberg has more details.

Update 2023-12-14 Massive Tesla Recall

Tesla forced to recall just about every car it has ever built – Regulators are fuming over the Autopilot driver assist system

Conclusion

This article is meant to stimulate discussion of a more modern and contextually aware definition for the word ‘control’. Devices that employ computational capability may require a more nuanced definition of control, while devices that go beyond general computational capability and employ machine learning and/or artificial intelligence may require an even more specialized definition.

I have not offered any such definitions; an entire book could be dedicated to deriving them. However, for a specific circumstance, a nuanced and contextually relevant definition could be derived.

Implications

We have discussed the similarity between the concept of controlling a sentient being such as a horse and controlling an autonomous device such as a self-driving vehicle or robotic vacuum cleaner. Court cases that cite horse-and-buggy precedents from 100 years or more ago may soon arise.

Doctrine Of Precedent

On a related note, in 2012, Kyle Graham, Assistant Professor of Law, Santa Clara University, discussed how new technology is gradually included into stare decisis (the doctrine of precedent) in Of Frightened Horses and Autonomous Vehicles: Tort Law and its Assimilation of Innovations.

Addendums

Gone: US Blueprint for an AI Bill of Rights

The White House Office of Science and Technology Policy, while under the Biden administration, released the Blueprint for an AI Bill of Rights as a public policy document. Subsequently, IBM published What is the AI Bill of Rights? The second Trump administration (POTUS 47) removed the document.

The document included five protections, the following 5 core ethical principles, and a call to action to protect the American public’s rights in an automated world.

• Safe and Effective Systems: protection from unsafe or ineffective systems.
• Algorithmic Discrimination Protections: people not face discrimination by algorithms, and systems should be used and designed in an equitable way.
• Data Privacy: people should be protected from abusive data practices via built-in protections, and you should have agency over how their data is used. Big tech companies will fight this every way they can.
• Notice and Explanation: People should know that an automated system is being used and understand how and why it contributes to outcomes that impact them. Good luck with that; today's technology is not designed with that capability in mind.
• Alternative Options: People should be able to opt out, where appropriate, and have access to a person who can quickly consider and remedy problems.

European AI Act and Liability Directive

Europe is leading the way towards a legal framework for AI implementations with the European Artificial Intelligence Act.

“The AI Liability Directive is just a proposal for now, and has to be debated, edited, and passed by the European Parliament and Council of the European Union before it can become law,” as reported by The Register September 29, 2022 in Europe just might make it easier for people to sue for damage caused by AI tech.

TAIHA: Proposed Hazard Analysis

Missy Cummings, Professor and Director of Mason Autonomy and Robotics Center at George Mason University, has written A Taxonomy for AI Hazard Analysis. This paper will be published in the Journal of Cognitive Engineering and Decision Making.

References

1. ‘Control’ from Merriam-Webster Dictionary
2. ‘Control’ from Oxford Dictionary
3. ‘Control’ from Law.com Dictionary
4. ‘Control’ from LawInsider.com
5. Have I Been Pwned goes open source, bags help from FBI
6. haveibeenpwned.com
7. What steps should you take when your email has been pwned?
8. ‘Context’ from Lexico.com
9. McKinsey & Company: The State of AI in 2020
10. ‘Black box’ from Merriam-Webster Dictionary
11. Artificial Intelligence: Explainability, ethical issues and bias
12. Computing Machinery and Intelligence, Mind
13. U.S. eliminates human controls requirement for fully automated vehicles
14. 1999 Car of the Century
15. Acetylene gas lamps
16. Model T Ford Club of America
17. BMW driver assistance
18. Society of Automotive Engineers
19. NHTSA: Automated Vehicles for Safety
20. US National Highway Traffic Safety Administration
21. Telsa autopilot
22. Tesla Recalls 362,758 Vehicles Over Full Self-Driving Software Safety Concerns
23. Tesla Autopilot Full Self-Driving Hardware
24. Tesla videos on Vimeo
25. A Tesla on autopilot killed two people in Gardena. Is the driver guilty of manslaughter?
26. Mercedes-Benz Says Self-Driving Option Ready to Roll
27. Living With Artificial Intelligence, Lecture 1
28. Tesla wins first US Autopilot trial involving fatal crash
29. BNN Bloomberg has more details.
30. Tesla forced to recall just about every car it has ever built – Regulators are fuming over the Autopilot driver assist system
31. Of Frightened Horses and Autonomous Vehicles: Tort Law and its Assimilation of Innovations
32. White House Office of Science and Technology Policy
33. Blueprint for an AI Bill of Rights
34. What is the AI Bill of Rights?
35. European Artificial Intelligence Act
36. Europe just might make it easier for people to sue for damage caused by AI tech
37. Unofficially published
38. Missy Cummings
39. A Taxonomy for AI Hazard Analysis

What do you wish you knew the first time you were an expert witness?

I wish I knew about how client management as an expert witness differs from client management as a computer consultant. When I started as an expert witness, my experience as a computer consultant was:

• Designing and implementing software products
• Automating business processes,
• Evaluating the work of other technologists
• Coaching startup founders
• Doing technical due diligence for potential investors

I had worked with technical people and their colleagues all over the US and Canada. Since then I have also worked in Hong Kong, southern India and Europe, and I have had clients in Oceana.

It is interesting that as an expert, the specialties of the different types of lawyers I have worked with has only somewhat overlapped the range of the non-legal work that I have done. It seems that experience in one field can map non-linearly to other fields, and that always intrigues me; different thought processes from familiar beginnings.

What do you mean by different types of lawyers?

The Yale Law School’s publication from April 2017 entitled Law Firm Practice Area Summary describes 47 types of lawyers. This 16-page document briefly describes each type of lawyer, and includes comments about common behavioral traits of each type. As an expert, I have worked with 4 types of lawyers described in that document to date:

1. General Corporate
2. Litigation
3. Prosecution
4. General Litigation

I have also been approached by white-collar crime lawyers for two cases, and although these high-profile cases were quite intriguing, I was not selected as an expert. One lawyer described the end client as being ‘radioactive’. That client is currently in the news quite a bit.

As a software entrepreneur, I have also worked with 3 additional types of lawyers:

1. Mergers & Acquisitions
2. Cyber Security & Data Protection
3. Licensing and Technology Transactions

As a semi-professional musician and founder of a musical collective, I am interested in working with lawyers who focus on entertainment law, but have not yet had an opportunity to do so.

Like most expert witnesses, I do not have a formal education in law, so it is probably not surprising that I found portions of the Yale publication misleading. It is not my intention to suggest the publication is incorrect, instead I believe that my understanding of the terminology may have involved false cognates. For example, the description for Litigation lawyers says:

I agree with most of the description, for example the adversarial nature of the work. While I do not mind that aspect, most technical people strongly dislike that type of work. I think that the ‘documents’ referred to in the description actually refers to the legal documents filed throughout the case. While the number of filed documents per case for this type of lawyer may be less than for some other types of lawyers, the size of these documents can be daunting.

For example, the principal challenge for one of the patent cases I have worked on as an expert was plowing through 4 long and vaguely worded patents. Together, these patents totaled about 1000 pages. They were deliberately worded to allude to almost anything and everything, while actually saying very little. I knew I was going to be asked in deposition if I had read and understood all of these documents. This was really painful to do: the only way I could get through these documents and retain anything was to spend several days standing, while reading them to myself aloud, and making a brief note of each paragraph after reading it. It was torture! I was indeed asked if I had read them early on during deposition, to which I proudly said, “Every word!” That was expensive for the client, but it was important to be able to say that with confidence, and my reports and testimony were used to successfully invalidate those patents as part of two inter-partes reviews.

As a side note, Malcolm Gladwell gave a nice keynote presentation on a closely related topic with Adam Grant at the 2017 Wharton People Analytics Conference.

Tell me about your first case

My first expert case came to me unexpectedly. The client was a venture capital investment firm, and their BigLaw attorney described my task in a well-written and succinct engagement letter. The client was concerned that a large contractor had done a poor job developing software for one of their startups. The VCs were considering litigation, and wanted to know my opinion of the quality of the software.

Before this time, 20 years ago, I did not have a good understanding of what the term “expert witness” meant. I had previously acted as a computer consultant for over 25 years, and I was familiar with the technology for the case, but my weakness was my lack of experience managing this type of assignment. I did not know about the various types of lawyers, and so did not know how to interface most effectively with the lawyer who engaged me. He was actually outside counsel, and an investment management lawyer, not a litigator. Had I known that at the time, and had I known what that meant, I like to think that I would have tried to approach the case a bit differently. A mentor would have been very helpful to me in this regard. I have since been both protégé and mentor, and I believe there is no better way to get started as an expert witness, than as a protégé of a thoughtful mentor.

This was the first time I had a three-way engagement; in this case the 3 parties were the VC firm (the end client), the law firm, and me. The startup was actually a 4^th party, but as far as I could see, in retrospect, they had no say and could only provide me the information I requested and watch the process unfold. In addition, the software contractor was potentially a 5^th party, and this might have become significant if litigation was pursued, but the complexities of that potential relationship never materialized.

At the time, I lacked the experience to know that my assignment would likely have a better outcome if I had simply asked to meet with the VCs at the beginning of the assignment. The VCs were the ones who needed to make the decision of what to do about their investment. I mostly spoke with the lawyer who had written the engagement letter, and interacted on technical matters directly with the startup. Naturally, lawyers want to shield their client from problems and distractions, but sometimes it is important that the root of an issue be described first-hand to those who can address it.

I focused on the task of evaluating the software and writing my report, and never spoke with anyone from the VC firm. Furthermore, my simplistic understanding was that the task was merely about things (software), as if no people were involved, and analyzing process was out of scope, which meant I did not consider the nature of the communication between the startup and the contractor. Additionally, I might have provided significantly more value if I had spent even a few hours getting to know the VCs.

I approached the task as many consulting engineers would; I examined the software, wrote a report, and worked through several versions with the lawyer. Then, without any discussion with me, the VCs acted: they decided to abandon their investment. I was shocked to learn, after the fact, that they had decided to simply shut down the startup that they had invested $3M of today’s money in. They did not litigate or negotiate. Had I known how to handle the matter better, the result might have been different. I was happy to learn, years later, that the startup somehow survived this trauma, and continues on today.

This outcome did not provide the long-term value to the client that expanding the scope to include an examination of processes might have, so the question of “why did this happen, and how can we improve?” was never addressed. As a subject-matter expert, I could have provided this information quickly and easily. However, the necessary level of client management skill was beyond me at that point in my career.

By the way, Dave Zwieback wrote a good book on this topic, entitled Beyond Blame: Learning From Failure and Success.

What could you have done differently as an expert?

Presenting possibilities as questions at appropriate times can yield much value. Directly answering questions is appropriate when responding in discovery or when testifying in court.

However, when dealing with a client in private, passively providing literal answers to questions posed often does not provide the most value to the client. A more experienced expert might have attempted to engage the end client in a comprehensive dialog to build up a sense of the larger context, and might have revisited the parameters of the case at opportune times with the end client. Note that some countries explicitly disallow this type of interaction; however, this was a US-based client, and I am speaking to accepted US practice.

One way to get the attorney who engaged me to suggest a meeting with the VC firm would have been to ask for clarification on certain issues face-to-face. There are subtleties involved in this type of 3-way engagement, and a good attorney acts as a strategist, rather like a quarterback in American football. The rules of engagement necessitate an arms-length interaction between the expert and the end client; so carefully documenting each interaction can be important for discovery. Having a transcription of a recording of every meeting can be helpful.

My report graded the software written for the startup by the contractor as a failure. I did not spend any time considering how to mitigate the issues that I found because I expected the client would tell me to do that if they wanted to consider mitigation. I also expected that the client would discuss the matter with me before acting; at this point in my career, I was unaware that sometimes a client might just opt to flush problems away. I suspect that internal politics within the VC firm may have used me as a pawn in some sort of negative-sum game.

Nowadays, I am more likely to proactively make a judgment call to spend an extra hour when I feel the effort is appropriate, and then I communicate that I did this with a quick call to the lawyer, along with my initial findings. I have never received any pushback for spending a little extra time in this way, and the lawyers have always welcomed these unexpected results. I don’t overdo this, in fact, I take great care whenever I exercise initiative, but results matter, and occasionally taking a limited risk has allowed me to deliver significantly more value with negligible extra time and cost. The occasional unexpected victory, prudently executed, builds trust. If ever I find that I wasted time doing something that I was not asked to do, I don’t mention it and I don’t bill for it.

One of the principal points that my report highlighted was that the software was written as cut-and-paste, with lots of duplicated code. Software like that can often be optimized by refactoring the repeated parts, and this is easily automated because it is repetitive in nature. If I had engaged with my client better, and got their agreement to spend a few extra hours on the report, I could have provided an indication of the feasibility of rescuing the project. Later in my career, I lead SWAT teams to rescue high-profile software projects that were in trouble, but at this point in my career I just did what I was told, and only asked enough questions to carry out my mission according to a literal interpretation.

How has your understanding of the role of an expert changed?

Litigation could be described as a formal process of assigning blame and consequences. Frequently, litigation leads to both sides losing: losing time, losing energy, losing focus, and not adapting outdated or dysfunctional processes to ever-changing circumstances. This is of course not a universal truth because malicious behavior does exist, but my experience has been that executive ego and badly managed relationships drives a significant percentage of corporate litigation. Relationships are mutual affairs, so all parties involved share responsibility and have an opportunity to make them work better. In a very real sense, initiating litigation could be construed as a failure to negotiate solutions to issues.

In retrospect, I suspect that a principal within the end client, the VC, wanted to assign blame to an external third party, the software contractor. An impartial expert, myopically focused on tasks and things instead of also considering people and processes, was the perfect vehicle to accomplish this.

I suspect that I was used as a tool by an unknown client faction. It is true that the software was substandard, however the question of how this was allowed to happen was never raised and I, as the expert, did not proactively lay any groundwork for mitigation, not the least because I was unclear how I might do so appropriately.

I am not just an expert witness with expertise in technology, I am a practicing technology expert with experience as an expert witness. The fact that I was retained by outside counsel, defined the nature of my engagement, such that the value I provided was suboptimal. When I presented my findings, I could have suggested that if litigation was decided against, that my role be formally changed, from expert witness to management consultant, and that I be retained directly by the VC firm. Because I did not offer this possibility, I did not provide anywhere the near the long-term value that I might have.

This line of thinking leads to the question: what is the role of an expert? Several possible answers might be entertained, depending on circumstances. As always, the expert should be willing to change their opinion based on evidence as it arises. Thus, the goals of the expert might be considered to be a matter for ongoing discussion, at least until litigation might be initiated.

One possible answer might be that an expert should disregard politics. This would make that expert a pawn in the politics of the client. Moreover, a demonstrably politically naive expert lacks credibility, rather like a wind-up doll.

In contrast, if an expert consistently attempts to understand all the forces in play, including politics, but the client denies them access to that information, fine; the expert is demonstrably not naive and therefore could be said to be credible. However, merely accepting client’s assertions at face value and not asking for evidence to support those assertions again shows the expert to be a pawn and not is therefore not credible.

I have since found it very effective to inform the lawyers I am working with that my report will contain a passage detailing the documents I read, and stating the information that I was not allowed to view, and people that I was not permitted to interview. Clarity follows soon after – does the client have something to hide, or are they dealing with me in good faith?

As a first-time expert, I never considered the politics that must have been ongoing within the VC firm, so I was surprised when they discarded their investment. Clearly, there were ‘winners’ and losers in the VC firm as a result of this action. The startup was a big loser, the contractor suffered significant loss, and so did the VC firm as a whole. I doubt any positive lessons were learned by any of the parties involved.

In summary, the principal lesson for me was how I might transition from expert witness to management consultant. If a management consultant is experienced and perceptive enough, and if support can be identified within the client organization, a management consultant could provide insight and metrics to drive a client's change management process.

To illustrate how impactful that might have been, here are a few quotes from an article entitled Consulting Is More Than Giving Advice, published in September 1982 by The Harvard Business Review:

The consultant also has a professional responsibility to ask whether the problem as posed is what most needs solving. Very often, the client needs help most in defining the real issue... Thus, the consultant’s first job is to explore the context of the problem…

Consultants facilitate learning by including members of the organization in the assignment’s processes. For example, demonstrating an appropriate technique or recommending a relevant book often accomplishes more than quietly performing a needed analysis.

With strong client involvement in the entire process, there will be many opportunities to help members identify learning needs. Often a consultant can suggest or help design opportunities for learning about work-planning methods, task force assignments, goal-setting processes, and so on. Though the effective professional is concerned with executive learning throughout the engagement, it may be wise not to cite this as an explicit goal…

Too much talk about client learning comes across as presumptuous – and it is.

Learning during projects is a two-way street. In every engagement, consultants should learn how to be more effective in designing and conducting projects. Moreover, the professional’s willingness to learn can be contagious. In the best relationships, each party explores the experience with the other in order to learn more from it…

Increasingly, the best management consultants define their objective as not just recommending solutions, but also helping institutionalize more effective management processes.

What instruments do you play?

I play guitar, bass, various percussion instruments, saxophone, and I sing. Currently, I am learning to play harp. I blog about the technology I use for recording and streaming on my website, mslinn.com, where I also blog about my experience as an expert witness, and I publish music videos on Facebook at facebook.com/mslinnmusic.

References

1. This audio podcast
2. Yale Law School’s Law Firm Practice Area Summary
3. Malcolm Gladwell and Adam Grant at the 2017 Wharton People Analytics Conference
4. Beyond Blame: Learning From Failure and Success
5. Harvard Business Review: Consulting Is More Than Giving Advice

Nothing Succeeds Like Excess

This article explains how the above quotation is relevant to an expert witness.

Honing My Opinions

My experience as an expert has been that for many of my cases, most of my time is spent preparing for interactions with the other side. My reports are written as a series of drafts that I share with the attorneys that I work with. We discuss the material, and that allows me opportunities to clarify exactly what I mean. The best attorneys that I have worked with have typically challenged the material as it evolves. My responses often get incorporated into the next draft.

As I work through each draft of a report, if there is material that is never the subject of discussion, I must ask myself if that material is fundamental and obvious, or whether it provides a basis for my opinions. Since an expert report concludes with opinions, material that is not fundamental, or necessary background, and is not something that was relied on to form an opinion, might not be germane enough to include in the report.

As a motivating principle, if one views each statement in a report as a potential attack vector that might be exploited by the opposing side, then it follows that removing unnecessary statements makes for a stronger report. This is one of the reasons why a vigorous dialog between an expert and the attorneys that they work with will, in general, yield a better result.

Preparation Scenarios

To date, my most common types of interactions with the opposition have been:

1. Accessing restricted material under supervision.
2. Examination and cross-examination when being deposed.
3. Testifying under oath.

This article explains the type of preparation I do for these circumstances.

Preparing To Access Restricted Material Under Supervision

Computer source code has been the most common type of restricted material that I have reviewed to date. Project-related information is often not as closely guarded. I wrote about how I do source code comparisons previously. This article will not repeat that information.

A software expert who needs to examine source code would ideally like complete access to the material, in their office. They would like to use special software and/or hardware to analyze the material, and they might want to write scripts (small programs) to help them with their task. Sometimes this is possible. Sometimes this is not.

The restrictions on how and where I, as an expert, can examine source code can be profound. I find that imposing these restrictions seriously impacts what can be accomplished, and how the source code review can be approached. They also raise the cost of litigation for plaintiff and defendant significantly; it would be much cheaper for both parties if they would just allow the software expert access to any tools they want. A good expert will eventually find out the facts one way or another anyway, so why waste money? The cost penalty paid by both sides for imposing tight restrictions on experts can become significant. The extra cost comes from extra time spent preparing, and, if permitted by the court, extra time examining evidence with reduced efficiency.

Restricted materials are sometimes provided to an expert in a glass-walled evidence room (sometimes this is called a ‘clean room’), and if they are placed in such a room, then the expert is usually observed the entire time they review the material. Normally, at least one representative of the opposing counsel’s firm observes the expert the entire time that the expert has access to the restricted material. The evidence room usually has no communication devices, and the expert many not bring in a mobile device, camera, etc. A limited amount of Bates-numbered paper is usually provided for the expert to take notes.

Hostile Opposing Counsel

It is not unusual for opposing counsel to be unhelpful. Sometimes, however, opposing counsel crosses the line into unprofessional hostility. While attorneys are expected to be partisan, they are prohibited from tampering with evidence. Sometimes, when their desire not to lose a weak case overcomes their better judgment, they cross the line because they think they can get away with it.

A case that I worked on required me to compare two bodies of code. One body of code came from my client, and I could inspect it at leisure on my laptop over many weeks. The other body of code came from the opposition.

Before providing access to the source code, opposing counsel misrepresented the computer language it in which it was written and the software frameworks that were used. Fortunately, I did not put my faith in the information the opposition provided, and instead I spent quite a lot of time writing scripts to identify unique aspects of my client’s code. I prepared 250 pages of instructions for myself to follow while in the glass-walled evidence room, for various possible scenarios. Armed with these instructions, I quickly discovered exactly which portions of source code were similar between the opposition’s evidence and our side’s evidence.

Every day that I spent in the glass evidence room examining the opposition’s code I found that the evidence had been changed significantly. Directories had reorganized, new directories had been added, and others had been removed. That did not slow me down much due to the nearly fanatical amount of preparation I had done. I knew what to look for, and found it. All the opposition’s shenanigans did not help their case – in fact the judge was not impressed because I chronicled the events in my report. It was satisfying for me to be able to voice strong opinions in my report, backed up by incontrovertible evidence and straightforward reasoning.

The extra cost of the unusual amount of preparation I did for this case was significant. The only reason I felt it necessary to dedicate such an extreme amount of time to preparing was because opposing counsel had previously demonstrated a willingness to exceed proper behavior. Each transgression motivated further transgressions. The risk that they assumed was that, if they lost, and the judge found they violated legal standards, they might be subject to consequences.

Testimony as Performance Art

It is not enough for an expert to know the facts, perform appropriate analysis and provide defensible opinions. Answering questions simply, directly and without embellishment can be surprisingly difficult. Getting it right under pressure, while tired from travel and long days of work, requires extensive practice.

Rehearsals

Preparation for testimony begins with the first draft of my report and includes rehearsals with the attorneys I work with.

Following is a typical sequence of events before my report is submitted. This explanation is rather general, and approximates what might be done in US Federal court and ITC arbitration. Patent-related matters do not necessarily follow this sequence exactly as described, however those sequences are similar enough that this general description should be instructive for those who have not gone through these types of legal processes before.

My report is written with the intention of effectively defending it:

• Summarizing my draft report into the minimum number of key points.
• Summarizing the other side’s expert reports (if available) into the minimum number of key points.
• Role playing testimony against an aggressive attorney (from our side), who tries to find different weaknesses in my responses. Running through every scenario many times, often with a twist.

After my report is submitted to the court, and the other side has responded, the sequence generally continues as follows:

• I follow a similar sequence for preparing drafts of my response to the other side's expert report.
• Role plays are performed before deposition. These usually take place in person with the attorneys I work with.
• Doing a postmortem of each run-through with the participants, using recorded video, is vital.
• Summarizing my responses onto flashcards for each anticipated question has proven to be very helpful.
• Working through the flashcards over several days until neither I nor the attorneys I am working with can make a mistake, no matter how tired we are, means I can be confident that I will do well when testifying.

I want to make a distinction between rote memorization and being able to recall all the salient points of my expert report easily, and to discuss their subtleties in detail. If an expert is viewed as a puppet because they stick verbatim to a script, then their credibility is lost. On the other hand, being able to readily articulate a nuanced understanding of the details of a case is impressive. “Threading the needle” by not falling into technobabble and plainly stating facts, methodology and opinions without being boring can be quite a challenge.

Winning Confidently

Being visibly relaxed and confident while easily handling whatever the other side brings forward goes a long way to winning as an expert. (Please remember that my definition of ‘winning’ may not be what you might expect.)

Confidence comes from knowing that one simply cannot make a mistake. The right words, demeanor and sense of timing have been habituated into the unconscious. Excellence has become natural.

Most questions are of a purely technical nature, but sometimes they pertain to technical marketing. Issues often relate to trade secrets and intellectual property. My responses follow from facts, or stated premises where facts are unavailable.

How Do Experts Win?

As a software expert, I strive to win, but my definition of winning might surprise you: for a non-partisan expert witness, “winning” can only mean that their opinions were accepted.

Independent experts must not have an investment in the legal outcome of a case; conversely, they should not fear any personal impact from performing their duties in a professional and conscientious manner, regardless of the outcome of the case. Experts can “win” even in cases where their client has an unfavorable judgment.

How Do Experts Lose?

Experts can lose 3 ways:

1. By being disqualified as an expert.
2. By impeaching themselves.
3. By having their opinions partially or completely dismissed.

Financial pressure is a common way for clients to attempt to influence an expert’s opinion. If payment is not made in advance, then an expert might be tempted to please their client so full payment is made promptly. This type of financial arrangement sets up a conflict of interest for the expert.

During discovery, I am often asked by the opposing counsel what my financial arrangements with my client are. If an expert agrees to payment terms that puts them into a conflict of interest, opposing counsel may be motivated to attempt to disqualify the expert.

Lawyers are Partisan Advocates

In most of the world, lawyers play by different rules than experts. The French words for male / female ‘attorney’ are ‘avocat’ and ‘avocate’. While lawyers must advocate for their clients, in most of the world experts must not.

US Expert Witnesses Are Allowed To Be Partisan

The USA differs from most other countries in that expert witnesses can be partisan unless they are hired by the court.

I have rebutted reports written by such experts, and my experience has been that the best way to counter a partisan expert is to be demonstrably non-partisan. When an expert says in effect “I formed my opinions soley on the basis of my experience”, and remind everyone of their illustrious credentials without applying analysis to evidence, then their argument is vulnerable to a well-articulated explanation of facts and findings from which opinions follow, given by the other side’s expert.

I normally consult with clients about how facts might be presented and construed, and the strengths and weaknesses of an argument from a technical point of view. However, my opinions are based on fact, I detail my reasoning, and my findings can be reproduced from the evidence. My opinions are defensible. Shills cannot compete against this approach.

While it is regrettable that the USA is out of step with most of the rest of the world in allowing partisan experts, the opinions of partisan experts tend to be disregarded in favor of opinions from trustworthy experts.

Concurrent Evidence

In other parts of the world, experts from both sides can testify together at trial, in a procedure known as concurrent evidence. They discuss the case in front of the judge, and respond to inquiries from the magistrate(s) and attorneys from both sides of the case,

I participated in such an interchange when I was the lead ERP software expert for an ITC arbitration based in Paris. A tribunal of 5 arbitrators asked me and the opposing expert to stand before them, and they questioned us together. Discussion was encouraged. My experience was quite positive.

The other expert was shown to be partisan. While we underwent the procedure of giving concurrent evidence, I reminded the court that the other expert had not provided any evidence to support the assertions upon which his opinions were based, beyond “Because I said so, and I am an expert!” His opinions were soon dismissed, while mine were accepted.

Judge Alsup

William Alsup has been a United States District Judge of the United States District Court for the Northern District of California since 1999. A Law 360 article published January 7, 2021 directly addressed the issue of partisan experts. The first sentence in the article reads:

District Judge William Alsup appeared unlikely to let Finjan Inc. skip out on Juniper Networks’ $8.65 million legal bill following its defeat at trial, calling out Finjan on Thursday for changing its patent infringement theory to inflate damages and lamenting the “standard patent BS by bought-and-paid-for experts.”

Elsewhere in the article, Judge Alsup is quoted as saying:

The bought-and-paid-for experts will say anything. They will say anything, on both sides. I don’t trust a word of that.

It may be legal to be a partisan expert in the United States, but that does not mean it is a good idea for an expert to conduct themselves that way. Experts that prostitute themselves are no match for against a competant and demonstrably impartial expert.

Common Business Practices Can Cause Experts to Lose

Most senior managers in large firms are used to tightly controlling their subordinates. Contractors and employees are saddled with onerous terms because they are often unable to make significant changes to those terms. This is considered to be ‘business as usual’ in many parts of the world.

When engaged under these terms, an expert invariably must adopt a partisan stance. Engaging experts this way makes them vulnerable to a serious attempt to disqualify them.

I am non-partisan expert. My standard Terms and Conditions document details how my relationships with clients are governed. All clients must agree to these terms, without exception. The document has been carefully crafted to ensure that my opinions are free of influence. This allows me to take on controversial cases, and work for either side.

Experts can only be effective if their independent opinions can be expressed and believed.

Experts Must Stay In Their Lane

About a year ago, I was contacted by an attorney who represented a high-profile individual, whom he described as ‘radioactive’. This person had already been found guilty of various matters, while others were ongoing. Would I be interested in being engaged by him as a software expert?

I told the attorney that I, as an expert, am not one to pass judgment. My job is to respond to questions put to me by independently examining evidence, performing analysis and reporting on my findings. It would be inappropriate for me to attempt to do the work that a judge or jury would perform.

I also said that my opinions would be based on my own first-hand examination of the evidence and subsequent analysis. My opinions are often qualified by the probability of certain evidence occurring by chance.

If his client was concerned about what form my opinions might take, I suggested that they quietly engage me before declaring me as an expert. I would examine the evidence and provide a preliminary opinion in confidence. They could then decide if they wanted to declare me as their expert.

Critical Review and Dress Rehearsal

For an expert to be effective, it is not enough to be competent, trustworthy, examine the available evidence, perform the appropriate analysis, and develop defensible opinions. To be effective, an expert must also articulate the issues succinctly and clearly for their audience.

As a software expert, I normally discuss my notes verbally with my client, and then draft a report. Attorneys I work with normally ask me questions throughout the engagement. For example, they might ask me “Mike, you say such-and-such, but someone might understand your statement to mean this-and-that. Is that what you mean?” My response is often something like “No, not at all, you see there are other factors to consider…” By discussing the issues in this way, the attorneys learn a lot about the technical issues, and I greatly improve the way I express my opinions.

This process gets more intense when preparing for deposition or trial. Often we roleplay: one attorney takes on the other side’s possible point of view, and another attorney argues as themselves. Then they cross-examine me, as the expert. We then critique each other on what was said, and how effective the dialog was.

It takes a lot of preparation to be able to readily provide clear, succinct answers to questions. I greatly appreciate the time I have spent preparing for testimony with the attorneys that I have worked with. As a result of our time together, my opinions were expressed effectively and all have been accepted.

Summary

The articles in this series explore topics of interest to attorneys who are contending with cases involving intellectual property issues of software or hardware. This article discussed what ‘winning’ and ‘losing’ looks like for a non-partisan expert, how that differs from how attorneys win and lose, why non-partisan experts can succeed against partisan experts, common business practices that affect an expert’s success, and preparing for testimony.

Acknowledgement

I would like to thank Eur Ing Professor Geoffrey M. Beresford Hartwell CEng for his thoughful review and advice.

ERP & CRM Are Big Business

Enterprise-level customer relationship management (CRM) and enterprise resource planning (ERP) installations are big business. System integration contracts for the largest clients are often priced at hundreds of millions of dollars. ERP and CRM are close cousins, and have grown towards each other over the years, yet one aspect has not changed: when a large contract goes over budget by 10%, this is 10% of a huge number, and lawsuits get filed. The same response is common when projects budgeted at hundreds of millions of dollars get scrapped.

The level of CRM customization generally tends to be very high because marketing and sales processes tend to be very customer-specific. Similarly, the level of ERP customization can also be very high unless the customer’s back office procedures were designed to match the vendor’s idealized procedures, and this is seldom the case. Customization is expensive and is often difficult to manage perfectly because of unique requirements that are often not explicitly or fully stated.

The 5 largest ERP software vendors in 2020 are, in decreasing order of annual sales: SAP, Oracle, Microsoft, Infor, and IFS. The 5 largest CRM vendors, in order, are Salesforce, Adobe, Oracle, SAP, and Microsoft.

Whose Side Am I On?

After reading through a draft of this article I became concerned that it seemed to place most of the blame for failed ERP and CRM projects on customers. That would be inaccurate because customers rely on software vendors, system integrators and consultants to design, implement, install and often maintain ERP / CRM systems. For convenience, let’s collectively call those 3 types of entities suppliers.

Just as customers in general have issues and are rarely completely fault-free, suppliers in general also have issues and are rarely completely fault-free. One source of problems is that for systems of this size, sales people are not personally accountable for the overall success. Instead, sales people are motivated by short-term considerations (making the sale), and often have moved on to other opportunities before long-term considerations such as implementation success and realizing customer value.

Lots of articles are about how to avoid ERP/CRM failure. I am not going to belabor them here. Instead, I just wish to say that each situation is unique, people are inherently selfish and political, and whenever projects requiring clarity of purpose and constancy of vision are initiated that require significant resource and time there are potentially many types of suboptimal outcomes.

ERP Installation Failures

From a recent article in CIO Magazine entitled 16 famous ERP disasters, dustups and disappointments:

Nothing ever goes 100% as planned, that is just how life is. Large, long-term contracts with lots of custom integration will have varied outcomes. New litigation just needs means, motive and opportunity.

Standard Components vs. Custom Software

In life, the path well-traveled is generally safe because most people traverse it without incident. Blazing a new trail is generally fraught with peril, often from unexpected sources, yet this is where innovation and fresh opportunity arise. Programming interfaces (APIs) exhibit the same phenomenon: using standard, well-used interfaces tends to be cheaper than designing and building custom interfaces. That said, sometimes bold measures are easily justified, and it makes sense to attempt to develop new technology to address an urgent need / opportunity.

Litigation is often contemplated when costs are exceeded because:

1. Diagnosing software is generally a difficult task,
       ... so problems lurking within custom interfaces may not be immediately evident.
2. Debugging software is often even harder than diagnosing problems,
       ... so fixes to custom software may be slow to be made available.
3. Building software is labor-intensive and custom software is amortized over only a single company’s needs,
       ... so custom software is expensive and takes a long time to build.

When Suppliers are Blamed

I don’t mean to suggest that customers always blame their suppliers, but this does happen and is often easy to spot if one knows where to look. Conversely, all organizations are imperfect and so are the people who staff them. Problems arise. This is normal. I merely offer technical opinions in response to specific questions. Yes, I strive to achieve a wholistic understanding of any circumstance that I encounter as an expert witness, but that is only to provide meaning to my personal journey. This information that I am providing right now does not in any way describe the totality of all possible circumstances.

Earlier I quoted an article from CIO Magazine. An unhappy SAP customer (Leaseplan) said in their Q2 2019 financial results:

As a software expert, the above paragraph just leaves me shaking my head in disbelief. That paragraph is very wrong on many levels. I won’t go into the details here.

Leaseplan also stated they “planned to build a modular system using best-of-breed third-party components alongside its existing predictive maintenance, insurance claim and contract management systems. It expected this to be more scalable and allow incremental product deployments and updates.” These statements by Leaseplan sound like complete garbage to me, and other industry professionals such as Shaun Snapp of Brightwork Research & Analysis agree. However, I do not concur with many of Mr. Snapp’s other statements. I expect that Leaseplan’s current system integrator will be sued next.

Summary

Plaintiffs and defendants sometimes use the courts as a venue for high-stakes negotiations. Timing is generally coordinated with the expected remaining term of the plaintiff’s senior executives. I don’t mean to suggest that suppliers are always blameless. When they make mistakes, they should be held accountable.

The point of this article is to state that sometimes the simple essence of a case can be heavily obfuscated by the unstated motives of one or more of the parties. As an expert, I must have no vested interest in the outcome. Instead, an expert can only provide value when addressing the technical fundamentals of a case, and those fundamentals by definition cannot be influenced by politics. Ignorance of political and economic forces can make a naïve expert an unwitting pawn. I strive to become aware of all factors pertaining to a case, even non-technical factors, so I can provide unbiased and well-considered opinions.

Acknowledgement

I would like to thank Eur Ing Professor Geoffrey M. Beresford Hartwell CEng for his thoughful review and advice.

References

1. International Chamber of Commerce
2. Publicis/SAP arbitration settlement
3. Alphabet Soup, Oh My! What You Really Need to Know About ERP vs CRM
4. 5 largest ERP software vendors
5. 5 largest CRM vendors
6. Avoid ERP/CRM failure
7. 16 famous ERP disasters, dustups and disappointments
8. Leaseplan Q2 2019 financial results
9. Shaun Snapp of Brightwork Research & Analysis
10. Eur Ing Professor Geoffrey M. Beresford Hartwell CEng

This is my mantra as an expert witness. Read on to learn what I mean.

Requirements, Investigation, Analysis, And Opinion

Before rendering an opinion, a technology expert should first gather the information necessary to perform their analysis. The choice of methodology used for an investigation and analysis is dependent on many factors. When investigating engineering projects, it is often crucial to understand the context provided by the requirements imposed on the engineering team and the technology available during the timeframe of the project.

For the US federal legal system, Daubert is a flexible standard for the admissibility of expert witness testimony that permits novel approaches to analysis, as long as they are well-grounded in fact and present defensible reasoning. It follows that the methodology for distilling evidence from raw data must be defensible, and the results should be reproducible. As well, the methodology for analysis of the distilled evidence must be appropriate. When analyzing engineering projects, this means that the analysis used should be consistent with the project requirements, with due consideration for the technology available during the timeframe of the project, and the range of accepted practices at that time.

The US Federal Rule of Evidence 702 is similarly flexible.

The Expert Mantra

In light of Daubert and Rule 702, and from common sense, I suggest that technology expert opinions should be stated as follows, so they are defensible:

• What I know
• How I know it
• What it means to me

Let’s break this down.

What I Know

What I know is one or more observations gathered from a first-hand inspection of the evidence. For a technology expert faced with voluminous evidence, this often means running programs that extract and summarize information from the evidence. Useful evidence for a software expert might be source code, documentation, transcripts of conversations, or database contents.

I write programs as required, such as custom Microsoft Office macros, custom SQL commands, ad hoc command-line incantations, and/or custom shell scripts, etc.. I sometimes use commercial software tools such as decompilers and integrated development environments (IDEs).

Hearsay is not evidence, and an expert should take all verbal and non-authoritative written information under advisement.

How I Know It

Observations that cannot be replicated are suspect. This is why experts should always, in their reports, strive to document the exact sequence of commands that they performed to obtain the observations that they based their opinions on. This allows others to replicate the expert’s observations of the evidence, which means that their observations can be trusted.

How I know it provides credibility because anyone with access to the same evidence could type along with the expert report and verify that they get the same results.

Microsoft Office Skills

The legal profession relies heavily on Microsoft Office, in particular Word and Excel. While most attorneys are familiar with redlining the difference between two versions of a Word document, many are less familiar with style sheets, and few attorneys indeed are aware of Microsoft Office macros. To be effective when faced with a lot of evidence, experts should be able to develop custom Word and Excel macros. Some custom macros might be simple time savers for repetitive tasks, while other custom macros could be complex programs.

In a recent case, I wrote custom Excel macros to correlate two sources of information into a single timeline with approximately 20,000 entries, which was essential to understanding the history of a project. I could then opine that while the two document sources were known to be incomplete, the information found in each corroborated the other. I also developed a sense of how the individuals interacted with each other, and their technical and managerial strengths and weaknesses. It is amazing what a competent expert can discover when they know what to look for!

What It Means to Me

An expert’s opinion should be based on an analysis of their observations, or in other words what it means to me. They should strive to show how the observations were analyzed without getting lost in technical details. Even better, a confidence level in the results of their analysis should be expressed statistically. The degree of confidence is important because it informs their opinion.

Summary

It is essential for experts to be believable and easily understood. What I know, how I know it, and what it means to me is my mantra for providing value as an expert.

Since 2007, I have created over 250 instructional videos for my online training site ScalaCourses.com. I built the web application that runs the site, wrote most of the content, and did all the videography.

I also create explanatory videos for court cases. This is helpful when complex and/or technical information needs to be presented.

Sample Explanatory Video

Following is an explanatory video that I created for a case, modified so that identifiable information is redacted, and with extra context added so that the video makes more sense to viewers who were not involved in the case. The information presented in this video is one aspect of the digital fingerprint of the ‘noise’ in the software that I discussed in the previous article.

Transcript

Hello, my name is Michael Slinn. This video has been prepared to explain one of the central concepts behind the opinions expressed in my report. At issue is whether the similarities in two bodies of code could be explained as a curious coincidence, or whether the likelihood of this being a random coincidence is extremely improbable. That would allow me to opine wether or not the author(s) of the second program must have accessed the first program so they could literally translate code written in the first computer language into a second computer language.

Question Addressed

The precise question that this video addresses is complex, and can be summarized as:

What is the improbability of 6 items appearing the in same order in two identical lists containing the same 26 items, after shuffling?

Concept: Factorial

This video shows the question visually, and also shows the answer in a compelling manner. In this video I demonstrate the likelihood of 26 routes found in both bodies of code, such that 6 of those routes appear in the same order in each program. This ‘co-incidence’ is approximately 2.5 times more unlikely than the chances that any two random humans have the same fingerprints. First, we need to understand a mathematical concept This concept is interesting and not difficult, I promise!

A factorial is mathematical shorthand. It is just a quick way of writing a tedious multiplication. Factorials are written as an exclamation mark (!) after a number. Let’s see some examples.

2! = 2 x 1 = 2

3! = 3 x 2 x 1 = 6

4! = 4 x 3 x 2 x 1 = 24

5! = 5 x 4 x 3 x 2 x 1 = 120

6! = 6 x 5 x 4 x 3 x 2 x 1 = 720

10! = 10 x 9 x 8 x 7 x 6 x 5 x 4 x 3 x 2 x 1 = 3,628,800

15! = 15 x 14 x 13 x 12 x 11 x 10 x 9 x 8 x 7 x 6 x 5 x 4 x 3 x 2 x 1
    = 1,307,674,368,000
      (1.3 trillion)

20! = 20 x 19 x 18 x 17 x 16 x
      15 x 14 x 13 x 12 x 11 x 10 x 9 x 8 x 7 x 6 x 5 x 4 x 3 x 2 x 1
    = 2,432,902,008,176,640,000
      (about 2 quintillion, or 2 followed by 18 zeroes)

26! = 26 x 25 x 24 x 23 x 22 x 21 x 20 x 19 x 18 x 17 x 16 x
      15 x 14 x 13 x 12 x 11 x 10 x 9 x 8 x 7 x 6 x 5 x 4 x 3 x 2 x 1
    = 403,291,461,126,605,635,584,000,000
      (about 4 octillion, or 4 followed by 26 zeroes)

Analogy

1. Start with a deck of cards.
2. Remove half of them (take away 2 suits).
3. 26 cards remain.
4. Do the same with another deck of cards. Now you have 2 half decks, each of which has the same cards as the other.
5. Shuffle both decks.

Let’s examine the odds of finding the same sequence of cards in both half-decks.

1. What are the odds of finding 2 cards in the same sequence in both half-decks?
2. Notice that the two sequences are in different locations in the deck. The topmost half-deck of cards has the sequence occurring about 2/3 of the way through that half-deck, whereas the bottom-most half-deck of cards has the sequence occurring hear the beginning of that half-deck.
3. The equation for that probability is 26! / 24! = 1 : 650
4. OK, what are the odds of finding 3 cards in the same sequence in both half-decks?
5. The equation for that probability is 26! / 23! = 1 : 15,600
6. Now let’s consider the odds of finding 4 cards in the same sequence in both half-decks.
7. The equation for that probability is 26! / 22! = 1 : 358,800
8. Next let’s consider the odds of finding 5 cards in the same sequence in both half-decks.
9. The equation for that probability is 26! / 21! = 1 : 7,893,600
10. Next let’s consider the odds of finding 6 cards in the same sequence in both half-decks.
11. The equation for that probability is 26! / 20! = 1 : 165,765,600

Notice that the card values, for example jack, queen, king, ace, and the numbers 2 through 9, have no effect on the probability. The uniqueness of low-value cards appearing in a given sequence is the same as the uniqueness of high-value cards appearing in a given sequence.

Summary

This graph shows the dramatic increase in improbability as the number of matching cards increases.

As a point of reference, the odds of 2 people having the same fingerprints is 2.5 times less, or approximately 1:64,000,000! I obtained the information on fingerprint uniqueness from the book entitled Finger Prints (Great Minds) by Francis Galton.

Conclusion

Getting back to comparing software for this court case, I found 26 items that were literal translations of each other in both the plaintiff’s code and the defendant’s code. 6 of these 26 items appeared in the same order in both programs. The probability that this is mere coincidence is so unlikely that it is reasonable to opine that the author of the second program worked from the source code for the first program, which was a trade secret. This constituted a misappropriation of intellectual property.

Thank you.

Since this article discusses software in various forms, we need to define a few terms:

Source code

Computer commands written by a programmer in one or more of many computer languages. Computer languages are classified as either compiled or interpreted.

Interpreted language

A programming language that executes source code directly, without compiling the source code.

Compiled language

A programming language that translates source code into executable object code.

Object code

The executable (ready-to-run) program resulting from compiling source code. Only compiled languages save the generated object code; interpreted languages generally do not produce object code.

Decompile

Reconstitute source code from object code. Decompilation is the opposite of compilation.

Reverse engineer

To take something apart to see how it works, to duplicate or enhance it. When a technology expert reverse-engineers a product for legal purposes the results cannot be used to create a competing product or be publicly shared.

As a software expert, I am able to work with source code and object code, and decompile or reverse-engineer software when necessary. I have also opined on whether a party’s software was copied verbatim, translated or otherwise derived from another party’s software.

Copying is Binary

One defendant’s expert, whose opinions I rebutted, had suggested that the defendant’s copying was insignificant and should be disregarded since only a few thousand lines of code had been translated, amounting to a small percentage of the total amount of the defendant’s code. Just as one cannot be just a little bit pregnant, copying is a binary state; either copying occurred, or it did not.

Once I believe that copying occurred, my job is to quantify the degree of certainty of my opinion. Computing the confidence factor, and communicating it effectively, is an important part of my task as an expert.

Inspecting Compiled Programs

When a copyright notice is embedded in an executable program or is present in source code, I reach my expert opinion easily. Sometimes all it takes is a hexadecimal editor to display this information. The image below shows that a company called Foundstone Inc. copyrighted a program called WebDog File Watch.exe in 2000. Unfortunately for the defendant, this was proof that they had misappropriated the executable program from Foundstone.

A defendant misrepresented a Microsoft Windows program (an ActiveX control) as their own. By examining the executable image, I determined that it had been misappropriated from Microsoft.

Microsoft Word – the Smoking Gun

A defendant included a Features and Benefits document in their CD. This Microsoft Word document was a modified version of the plaintiff’s Microsoft Word document that discussed a commercial product with exactly the same features as the infringing program. The defendant had left Microsoft Word’s revision marking on, so that an audit trail was present in the shipping product. My expert report stated conclusively that the defendant had misappropriated the software.

Sometimes My Task Is Not So Easy

The analysis necessary to form an opinion can be rather involved. The challenge that an expert faces when writing their report is often how to present their findings in a clear and concise manner to a non-technical judge and jury. Later in this article, I will describe a novel approach that can lead to a simple, very convincing report.

Equivalent Programs

A global edit is a text replacement made to all the files in a project. Making global edits to a copy of a project does not alter the fact that the project was copied. Merely changing the names of variables, classes and methods does not alter the meaning of a program. This means that performing global edits on source code results in an equivalent program. The same is true of literally translating a program from one computer language to another.

Le Petit Prince / The Little Prince by Antoine de Saint Exupéry is the world’s most translated book, excluding religious works. Originally published in English and French, this book has been translated into 361 languages. We will use this book as an example of how both types of copying could be accomplished. The same principles apply to software as intellectual property; note that we are not discussing copyright at this time.

Global Edits

In The Little Prince, the main character supposedly came from an asteroid known as B-612. The name of the asteroid could be changed throughout the book without affecting the book’s meaning. Therefore, a global edit that changed the name of the asteroid from B-612 to XYZ123 would mean that the resulting book would still be an equivalent copy of the first because the meaning of the story would not have changed.

Translations

The French version has more words because French is a grammatically longer language than English, but every passage in the English version has the same meaning as the equivalent passage in the French version. Both versions of this document are literal translations of each other because they have the same meaning. The actual words used are insignificant, so long as the meaning does not change. Thus, the English and French versions of the book are copies of each other, even though they are written in different languages.

The AFC Test

Since 1992 the gold standard for source code comparisons in copyright law has been the Abstraction-Filtration-Comparison (AFC) test. To apply this type of analysis, a certain degree of co-operation by both parties with the software expert is necessary. The AFC test requires that both parties allow their source code to analyzed by various software tools. Depending on the size of the software programs, this might require days, weeks or months. The computer that is used to perform the analysis must be specially prepared, and the results must be saved onto some form of storage media for further analysis before the expert can write their report.

The AFC test invariably results in a long and complex report filled with technical information. The abstract concepts in the report can be difficult to communicate to a non-technical judge and jury.

A Novel Approach for Detecting Copying

Comparing digital fingerprints can give compelling results and, if approached in the right way, can be a lot less work than the AFC test.

Let’s say that a music producer is accused of copying a song that was originally published as a vinyl record. Imagine that the complaint stated that the infringement was done in a way that the musical structure, lyrics and arrangement were substantially similar. An expert might attempt to perform an analysis of the structures, lyrics, and arrangements of the two recordings, and opine on the similarities. The problem with this approach is that the report would be rather technical, and there is a risk that the judge and jury might not understand the analysis. Additionally, there is always the risk that a jury might not come to an agreement.

To avoid these issues, the expert might show that the alleged copy was made from a specific vinyl record, not by analyzing the music, but by analyzing the noise (for example, cat scratches). The expert would have to exclude noise added in the production of the original recording (because all originals would have this or similar noise), and focus solely on the noise added to a particular vinyl record (for example, long, jagged cat claw scratches unique to one specific original record.)

If, after the musical content is removed, the remaining noise closely matches the noise on the recording, then the degree of correspondence could be statistically computed. Noise could be compared based on the amplitude of each snap, crackle and pop. The energy spectral density of the noise could be displayed as a timeline, and the degree of correspondence accurately computed. This approach (comparing noise) would actually give a much better result than theoretical analysis of the music, and would be more convincing to a judge and jury.

To summarize, the approach described above compared the digital fingerprints of the noise in two recordings, instead of comparing the fingerprints of the actual recordings.

A Difficult Case Had an Unexpected Benefit

Recently, I was retained on a case that was particularly challenging because both parties refused to allow me to use any tools. Furthermore, the opposing party’s software to be examined was provided on a computer without internet access, and I could only print 100 pages of notes in total during a relatively short visit at the offices of their attorneys. However, necessity is the mother of invention, and I developed two novel approaches that allowed me to opine that there was clear evidence of inappropriate IP infringement.

I took a similar approach as described in the novel approach described above for this case. I would normally analyze input and output patterns, but the defendants chose to make more complete analysis impossible. In the limited time available to me and with only the most rudimentary software tools permitted I was unable to compare processing, object models, SQL tables and other standard aspects of software programs.

I focused instead on the ‘noise’, which in this case were unusual artifacts found in the software. Unfortunately for the defendants, the results of analyzing the software ‘noise’ were incontrovertible. The mathematical probability that the backbone of the defendant’s software would coincidentally have an identical noise fingerprint as the backbone of the plaintiff’s software was vanishingly small. The parties in this case settled at the 11^th hour in the evening before trial.

Summary

Many techniques are available for a technology expert to employ when examining evidence. Out-of-the-box thinking can provide excellent results that are technically sound and present clearly to non-technical audiences. Experts who are skilled at writing Microsoft Office macros can provide tremendous value.

I spoke with a corporate attorney that I have worked with about this, and he said “Rule 401, in my understanding, is not referring to ‘probabality’ as a mathematical matter, but merely tries to reflect that there must be some logical connection between evidence being offered and the claims or defenses.”

Notwithstanding my learned friend’s explanation, there is general agreement that the relevance of expert evidence that arises from the hard sciences requires particularly rigorous reasoning. That reasoning should be provided as a quantitative result, after establishing that the evidence is authentic. Computing probabilities can be most helpful in this regard. Analysis showing strong mathematical probabality can form the basis for well-founded opinions.

For coherence, empirical truth requires a proper suite of propositions that mutually support each other. Experts must adhere to this practice when reviewing technology. As a result, a competent expert should be able to opine on the abilities and work habits of the people who worked together to develop the technology in question, even though they never met, and on whether the technology was created organically or misappropriated.

Certainty

The purpose of probabality is to quantify uncertainty. For example, what is the probabality of rolling a six-sided die and obtaining a 1?

For empirical truth, certainty is generally assumed once a threshold of probabality is achieved. This is the basis, for example, for matching DNA profiles, fingerprint matching and matching retina scans. With due consideration for prosecutor’s fallacy, a higher probabality leads one to believe that the possibility is more likely.

There are 7.5 x 10¹⁸ grains of sand in the world, and there are estimated to be 10²¹ stars in the observable universe. In a recent case, I, in my capacity as a software expert, found and presented evidence that was more unique than all the grains of sand in the entire world, and the number of stars in the observable universe. As a result, I felt confident that the opinions I derived from my analysis of the evidence were well-founded.

Subsequently, in that same case, I was invited to provide technical support during the deposition of one of the other side’s experts. I computed the likelihood or probabality of many statements that the other expert made mere seconds after he uttered them. These real-time probabality computations provided guidance for further questions posed by the attorney that I was supporting.

Summary

This article summarizes the importance of empirical study in the analysis of evidence, and the role that the mathematical determination of probabality plays in determining the value of evidence.

Analyzing evidence for unlikely contents can yield important information that may be the basis for an expert opinion. It is vital that an expert be able to quantify the probabality that anomalies are merely due to random coincidence, and that the expert’s report reflect this in a defensible manner. Since correlation does not imply causation, probabilities must be combined with other factors to have meaning.

Intellectual Propery & Trade Secrets

Intellectual property becomes a trade secret when it is valuable, or is expected to be valuable, provided that a reasonable attempt is made to keep the details secret. Trade secrets differ can just be valuable, and need not be novel.

People, Process and Technology

My qualifications and experience allow me as a technology expert to:

• Respond to questions about a technology
• Respond to questions about a product or family of products
• Compare and contrast technologies and their applications
• Compare and contrast products or families of products
• Respond to questions about the development of a technical product or service
• Respond to questions about potential or actual applications of a technology
• Examine source code and report on similarities to other code
• Reverse engineer products

Often the expert writes a report that addresses specific questions posed by the client, after which the expert discusses the report in confidence with their client. If the report is sent to the opposing side of the case, the expert will most likely need to defend their report under oath.

What is Software?

Software is:

Many people are surprised to learn that software is not just program code. Software is a term that applies to all digital artifacts pertaining to the development of a program, including the program code, documentation, associated data, and more. Non-programmers regularly create new software artifacts; for example, technical writers write documentation, which is a form of software.

Considering that transcripts of conversations between technicians are a primary source of documentation, those transcripts might also be included in the definition of software. This might include Git commits, Jira / Pivotal Tracker / Trello logs, and transcripts of chats in online forums where developers talk to each other about the software project that they are collaborating on. I am skilled at weaving multiple sources of information together into a chronology, and then inferring habitual behavior from the transcript. This insight can be used to determine where to look next, and what to look for. It is amazing what one can find when one knows what to look for!

Software Archeology

Git is a free and open-source distributed version control system (VCS). It became the most popular VCS in 2008.

Git is particularly useful for determining what participants did during a software development project. Because Git normally retains the complete history of all the contributions made by the technical team as they work together to create software, a software expert should examine the Git history early in a case. Sometimes the Git history is doctored in an attempt to remove evidence. In a recent case, I easily identified that the evidence had been doctored, and the person who erased the Git history had to admit during deposition that he had erased some history.

When I start a new case involving software, I ask for copies of all relevant Git repositories. That provides me with a snapshot of the software in question at any moment in time. I can rewind any project’s history, and replay it — commit by commit, then summarize the data and display it.

Elsewhere, on this website, I discuss the technical details of working with Git in depth.

Structuring the Client / Expert Relationship

Sometimes my findings are not what the client wants to hear. The most effective way to ensure that I can be frank with clients is to have them pay in advance. I have found that an evergreen draw-down retainer is mutually convenient.

I was once retained by a company to render a preliminary opinion on whether litigation that they were contemplating might have merit. To develop my opinion, I had to download and reverse engineer the prospective defendant’s software without their knowledge. Having done that, my opinion was that there was likely no intellectual property infringement, and I suggested to the client that they make a business deal with the other party instead.

I suggested making a deal instead of litigation because during my research, I came to believe that the other party had developed better software using a different approach. This suggestion was unpalatable to my client’s ego, but as I explained, until they reworked their product to match the fresh and modern technology used in the other party’s product, my client’s only other recourse would be to compete on a non-technical basis. This would have required them to expend a significant amount of additional resources for sales and marketing, which are generally much more expensive than engineering costs.

My client followed some of my advice, and never initiated litigation; the other party never knew they were in the crosshairs. However, my client did not attempt to develop a business deal.

Games Some Attorneys Play

There are millions of competent technologists, but few are comfortable in the role of high-profile participant in an adversarial justice system. Attorneys should be aware that when a legal case requires the opinion of a technology expert with specific skills, or a specific background, it is not uncommon for certain qualified and experienced experts to be contacted by both sides.

Some attorneys attempt to engage all available experts at no cost to themselves or their clients. It is easy to retain an expert – all that is required is for the expert to agree to the terms expressed in an engagement letter. The letter might mention an attractive hourly rate with favorable terms, but these particular attorneys never assign any work.

Experts that have been engaged in this manner cannot ever work for the other side; they have been taken out of action for the remainder of the case.

The solution that I use is simple: my standard terms and conditions include a non-refundable minimum engagement fee. I have found that a fee equivalent to 2 day’s billings is sufficient to discourage this practice. I stipulate that this minimum fee must be received in full before an engagement is deemed to begin, within a short time period. If the fee is not received in time, then I am available to be engaged by the other side without conflict. I inform prospective clients that they should not tell me any non-public details about the case until their minimum engagement payment has cleared.

Summary

The articles in this series explore topics of interest to attorneys who are contending with cases involving intellectual property issues of software or hardware. This article discussed:

• Services I provide as a software expert.
• Definitions for trade secret, intellectual property, and software.
• Effective and defensible communication between experts, lawyers and the court.
• How I reconstruct a detailed chronology of the dialog that occured during a development project.
• Avoiding insincere engagements that are intended to take experts out of commission.