Abstraction - is it still relevant?

Abstraction - Is It Still Relevant?

Abstraction in software engineering has been challenged with the rise of machine learning (ML). While ML introduces new paradigms and tools, the concept of abstraction, which is taught at the foundational level of software engineering, continues to play a vital role in developing, managing, and scaling systems. Here’s why:

Aim

This article aims to examine the need for human intervention in performing code abstraction in light of the proliferation of ML in software development.

Really? Before we go into this, let's set the stage by revisiting some fundamental points.

What is Abstraction?

Abstraction is the process of simplifying complex systems by focusing on essential features and hiding unnecessary details. It allows developers to break down systems into manageable parts and work at higher levels of understanding without being overwhelmed by complexity.

Without Abstraction?

Without abstraction in software engineering, we would certainly face issues in managing the complexity of software, scalability, reusability, coding collaboration, and maintenance. These issues multiply with the expansion of business processes and, at worst, when fixing vulnerabilities.

How Does the Human Mind Perform Abstraction?

Humans perform abstraction by recognizing patterns, structuring problems into layers, and creating reusable and simplified modular solutions. Microservices are a proven example. I have been doing these. Read more in this article…

How Does ML Perform Abstraction?

To understand this, we need to have a clear understanding of what ML is. It is broadly defined as the capability of a machine to imitate intelligent human behavior. That said, it wouldn’t be far-fetched to question the need for performing code abstraction when we can employ ML to replace the task.

ML learns and relearns according to changes in the code, surpassing the need for human intervention without any fatigue!

Abstraction: A Force Multiplier in Machine Learning

To generalize it, I have termed the process into three stages:

1. Simplified Model Development

In the initial stage, ML libraries like TensorFlow, PyTorch, and Scikit-learn provide abstractions that simplify model creation, training, and deployment. These tools allow practitioners to focus on the problem domain rather than low-level computations, streamlining the development process.

2. Seamless System Integration

The next stage involves embedding ML models into larger systems. Abstractions such as APIs and deployment tools like MLflow and SageMaker treat models as black-box services, simplifying the transition from experimentation to production and ensuring seamless integration into existing systems.

3. Efficient Data Management

In the final stage, data preprocessing abstractions, including pipelines and feature stores, enable consistent and efficient data transformations. These abstractions are critical for ML workflows and help bridge the gap between software engineers and data scientists, fostering effective collaboration without requiring deep domain knowledge.

Replacement or Complement?

We can all agree that ML excels at automating tasks like pattern recognition, optimization, and decision-making within specific contexts. However, there are three points to consider:

• Abstraction requires deep contextual understanding and domain-specific knowledge to effectively identify what details to hide and what to expose.

• Humans must define the boundaries and goals of abstraction, which involves strategic thinking and insight into the broader context, rather than just raw computation.

• Effective abstraction in ML often results from a collaborative effort between humans and machines. While ML can handle complex computations and pattern recognition, human intuition and strategic oversight are crucial for guiding these processes and ensuring they align with broader objectives.

As such, incorporating abstraction into ML tools complements the human approach to large-scale systems.

Conclusion

Abstraction is fundamentally about understanding the purpose of a system:

• Where are the pain points?
• What problems are we solving?
• What should be exposed or hidden to the user or other developers?
• What trade-offs are acceptable?

Therefore, the need for human intervention in performing code abstraction is here to stay, with the assistance of ML tools.