Designing a Secure, Async-First Modular Monolith to Accelerate Developer–Data Scientist Collaboration at Equinor

The application facilitates faster and more reliable insight generation by improving data quality and optimizing the workflow efficiency of traders and analysts at Equinor

The Starting Point

The existing solution, built on Next.js, was already mature and integrated with external data services via GraphQL. However, as queries and algorithms grew more complex, new challenges emerged:
 

• Some of the Business logic tightly coupled to the frontend
• JavaScript limitations for heavy data processing
• Data scientists and backend developers spoke different languages
• Slow iteration cycles, impacting delivery speed

The Problem

Developers and data scientists often speak very different technical languages. This leads to siloed workflows, miscommunication, and slow iteration cycles.

We saw this as an opportunity not just to fix a bottleneck, but to future-proof the entire stack.

Our challenge wasn’t just technical - it was strategic: how do we enable data scientists and developers to collaborate on a unified platform without rewriting the existing solution or slowing down delivery?

What began as a performance bottleneck evolved into an opportunity to architect a modular, async-first platform, one that designed not just to scale, but to accelerate innovation. More than a backend solution, this is a reusable, product-grade foundation for high-performance development, experimentation, and collaboration.

The Critical Question We Faced

How can we scale our platform and decouple business logic without rewriting the current solution- while enabling data scientists to create and test APIs independently, without needing backend expertise?

The goal was ambitious:
 

• Keep the Next.js application responsible for both UI rendering and API handling, with no changes to its current behavior
• Allow it to consume existing and new endpoints in parallel

• Enable data scientists to define, test, and deploy business logic directly via Python

We didn’t want to just patch the architecture; we wanted to future-proof it.

Async-First by Design​​​​

• High concurrency performance: async I/O is ideal for I/O-bound tasks like database access or external service calls
• Efficient resource usage: fewer threads, lower latency
• Faster response times: no blocking during data processing or API chaining.
• Avoids cold starts and connection overhead
• Keeps latency low under high load
• Maintains persistent, efficient access to our MSSQL database

The Opportunity:  Build a Platform, Not Just a Backend
 

If you're only building a backend, you're solving today’s problems. If you're building a platform, you're investing in tomorrow’s success.

Rather than treat the backend as a hidden layer, we turned it into an open, extensible platform. One that:
 

• Supports cross-functional collaboration
• Exposes standardized, documented APIs
• Enables safe experimentation API versioning without backend rewrites

Strategic Platform Goals

1. Async Performance by Default

2. Python-First Logic for Data Scientists

Business logic is now written in Python—enabling data teams to:

• Define endpoints using FastAPI
• Validate inputs/outputs with Pydantic
• Deploy and test with no backend expertise

3. Secure, Scalable Foundation

Built-in auth handling, token validation, container orchestration, and API standardization - all aligned for scale.
 

Horizontal Scaling with Radix

While async enables efficient handling of many I/O-bound tasks within a single process, using uvicorn --workers allows the application to scale across multiple CPU cores, significantly improving performance under high load.

Radix provides the underlying infrastructure to horizontally scale these worker processes across multiple containers or nodes, enabling the application to handle even greater concurrency and throughput.

Radix’s orchestration delivers scalable, low-latency performance for both I/O-bound and CPU-bound scenarios.

 

Tech Stack and Tools

Backend
Python, FastAPI, Uvicorn, Pydantic, Redis, Pandas

Frontend
React, Next.js, TypeScript

Data Layer
SQLAlchemy, SQLModel

Infrastructure & Cloud
Radix, Azure
 

Middleware & Modular Design

The middleware is designed following a service-based architecture and adheres to modular design principles. Each component—such as token-based authorization, request routing - is implemented as an independent service with a clearly defined responsibility. This ensures separation of concerns and single responsibility across the system.

The modularity allows different data services and business logic layers to be developed, maintained, and scaled independently. Business logic is completely decoupled from the middleware, making the system easier to test, extend, and integrate with new services without impacting the core middleware functionalities.

The central middleware handles:

• Token-based authorization
• Request routing
• Data serialization
• Schema validation
   

Real-World Benefits

New services can be deployed without disrupting the existing app—no coupling, no regressions
 

• No additional tooling, backend rewrites, or API scaffolding required.
• Empowered teams. Accelerated iterations. Fewer bottlenecks.
• Every developer works in a consistent, production-like environment—with confidence.
• Encapsulate domain knowledge cleanly.
• Build a middleware that communicates directly with the business logic.
• Expose clean APIs that mirror internal services-without code duplication.
• Fast development cycles.
• Fast iteration - Add new endpoints in <10 mins.
• Shared language - Everyone understood the data contracts.
• Easy onboarding - New devs were productive in days.
• Data scientists didn’t need to learn backend development.

Developer Experience (DX)

Every engineering decision impacts delivery speed, quality, and long-term maintainability. A core part of our platform philosophy was making developer experience (DX) a first-class citizen.

That’s why we built this platform with developer experience at its core - so that teams can focus on delivering value, not wrestling with infrastructure.
 

Orchestration & Local Development Made Simple
 

• All services supporting middleware can be orchestrated using Docker Compose.
• Developers can spin up the full stack locally in minutes, without the need for complex setups.
• Onboarding new team members is frictionless—no manual setup, no guesswork.

Hot Reload & Middleware Debugging

The middleware is fully hot-reloadable, allowing for real-time debugging and iteration without restarting containers.

Developers can inspect and test routing logic in isolation, drastically reducing turnaround time.

TypeScript Support

The Pydantic models not only enforce data integrity on the backend—they automatically generate TypeScript types for the frontend.

• Frontend engineers benefit from full type safety and autocomplete when using APIs.
• Backend and frontend share a single source of truth, reducing bugs and misalignments.
• Fewer blockers, faster iteration, and more confident code pushes across teams.
• The architecture is modular, scalable, and ready to evolve.

We delivered more than a solution. we built a secure, async-first platform that bridges developers and data scientists, and enables rapid, confident growth.

For our client, this means:

• No costly rewrites

• Faster delivery cycles

• A scalable foundation for future products