dharma.h Software Technologies designed a bespoke solution for Lulu and Sky, a leading online fashion portal in India. Their retail online property had been built on Magento, deployed on a monolithic architecture on AWS. They hit a roadblock with scaling up. An increase in customer base spelled problems with processing and delivery of orders. dharma.h stepped in to tackle the twin challenges of sluggish performance and scaling up. The ask was to deliver a robust deployment system that provides a business solution and a technology solution wrapped in one.
Part of the business challenge was to anticipate future growth. For the technology part of the solution design, we addressed multiple needs within a limited timeframe and made optimum use of resources. We stepped in as enablers to remove actual and anticipated hurdles.
The challenges were to:
· provide a robust solution that is highly available and scalable
· equip it to manage large volumes of concurrent order processing and connections
· optimize AWS costs and infrastructure spend
· increase page load speed and response speed manifolds
· build and develop analytics and reporting solution for their business needs
· ensure disaster recovery and fail-safe architecture
· deliver a near to 100% uptime with failovers and backups
· provide CI/CD for seamless deployment and release management
· ensure continuous monitoring, logging and performance optimization
· harden security
· allow continuous threat and vulnerability assessment while keeping the data and the infra safe and secure.
Project Description
Our first step was to assess the architecture. We looked at enhancements and bottlenecks at various levels. We needed to put Magento on steroids to build a solution that can manage traffic and concurrent transaction volumes.
Request Flow:
Our intervention succeeded in improving overall request management at DNS Level and improving endpoint network delivery. We achieved high availability of the DNS by optimizing the use of AWS’s Route53. For enhanced caching, latency and endpoint delivery improvements, we contracted, configured and deployed Akamai CDN.
For security at request origin, we contracted and deployed a cloud-based Layer 7 from Oracle Dyn. The solution deployed is an intelligent threat detection tool designed with proprietary human interaction algorithms, continuous vulnerability assessment, device fingerprinting, and other advanced Layer 7 features. Layer7 provides network and request level security from all traffic originating from the world wide web.
We redesigned the request flow too. Requests from the Layer 7 are now first scanned and then, handed over to the AWS Infrastructure either at the AWS API Gateway or AWS Application Load Balancer. Rulesets configured on the Application Load Balancer direct the request to the requisite resource group. All requests at the API end are logged and analysed with ML-based anomaly detection algorithms. This helps to continuously improvise and build a better and more secure network.
Below is a schematic flow with an overview of what’s deployed at the infrastructure level:
Application Layer:
At the Application Layer, we decoupled various business functions and code blocks. The Application Load Balancer now routes requests based on specific conditions to different Auto Scaling Groups. The decoupling has two advantages. The first is an optimal use of resources and the second is the ability to provide dedicated resources for the specific business functions and code blocks.
For requests terminating at the API Gateway, we used dynamic token-and-key-based authentication for enhanced security on certain API sets. The API Gateway now terminates to either a serverless scalable Lambda code block or an Auto Scale Group designed to serve API requests. Caching at the API level on Akamai CDN has improved scale and concurrency.
The Auto Scale Groups comprises a fleet of compute instances of various sizes. We configured and deployed proper health checks to check application functionality that each compute instance is meant to deliver. Error and other logs on the compute instances were pushed to CloudWatch to preserve instance-level logging. Whenever a new instance is deployed now, it pulls the latest code base from the production repo. All the instances on the Auto Scaling Group are on a Private Subnet, which provides for further security. Application and infrastructure level monitoring are done to help identify and resolve issues in code blocks.
The golden AMI used to deploy on the AWS Auto Scale Group is versioned internally and scripted for System-level enhancements and rollouts.
Using the above, we have been able to make the system more robust, and fail-safe. The Instance Groups are deployed over multiple zones and kept private from the world wide web. No service becomes unhealthy. Logs help us monitor and deploy enhancements continuously.
Media Storage Layer:
The Media Files and Static Files are generated, uploaded or saved to an AWS EFS and/or AWS S3 Buckets. This provides media association to code blocks on instances during scaling.
The Database and Cache Layer:
The Database is deployed on a private network on AWS RDS (A-Z) — with IOPS optimized storage. The tables have been optimized for recovery to any instance in time using log rebuilding. Scheduled backups are configured. AWS DB Insights are used to analyse and optimize queries and bottlenecks at real time. For security, a NAT Gateway is configured for any access to the Database.
Besides the Akamai Caching at delivery end, we have configured and deployed an AWS ElastiCache — Redis Cluster for application level data caching and object caching. The Redis Cluster shares the load of data layer and makes results available quicker.
The database and the data layer form a high-performance powerhouse thanks to the marriage of ElastiCache Redis Cluster to a power database engine with disaster recovery. The database or the data layer is now ready to handle concurrent and large volumes of transactions.
Services & Security Layer:
We enhanced the code base to scale with services such as AWS Simple Queue Management (SQS), Simple Notification Service (SNS), ElastiSearch and Simple Email Service (SES). Athena, CloudWatch and Alarms are used as support services for logging, querying logs and alerts.
The access to the infra is via a VPN which can be accessed via a NAT Gateway. The Application layer is locked away from the external world. All access to the Development and other non-production environment are via Access Keys and Multi Factor Authentication (MFA). Individual IAM Users and Roles with limited scope have been deployed for enhanced security of the infrastructure.
The Deployment Layer (CICD):
All application development, done on Dev environment, are pushed to AWS Code Commit. The code from AWS Code Commit repo is then deployed using AWS Code Pipeline. The Code Pipeline is configured to deploy releases even at peak traffic hours without disrupting user experience. Various branches for Development/UAT and Testing are used before a code block is pushed to production. This allows for better version control and code lifecycle management. The repo can be accessed using IAM Keys making the code base secure.
BI Layer:
We use AWS QuickSights for BI Reporting.
Media Delivery Layer:
The property Lulu and Sky is laden with rich media content. The visuals are an essential, intrinsic and voluminous part of the eCommerce platform. We have integrated ImageDeliver, an Intelligent Image Compression, Optimization and Delivery Solution. ImageDeliver customizes the images for every single user — both on the app and the web property — based on the user’s device, network speed and other attributes. It optimizes the size and clarity of every image delivered to the end-user. The end-user media experience is rich, crisp, detailed and the image is at its smallest size when delivered to the user’s device. This helps increase the overall performance by optimizing and reducing data transfer for the property, both at the app level and on the web.
Mobile App Level Cache Management:
The Mobile App (Android and the iOS) uses Akamai Custom Control libraries to manage caching at the application level. This ensures a better end-user experience. It also helps to customize and deliver a performance filled user experience based on user’s network properties and attributes. Other third-party caching controls are also used at the App level. Object lifecycle policies are deployed to keep the app light and yet heavy on performance.
On the next page is a schematic view of the above layers put together:
Value Delivered
Putting it all together, dharma.h Software Technologies has:
· integrated services to support scale and concurrency,
· configured continuous code and infrastructure level monitoring to identify bottlenecks,
· added multi-layer caching — at object and data level,
· increased performance at endpoint delivery,
· customized end-user experience to their device and network capability,
· built a secure, closed network and application layer to safeguard against external vulnerabilities,
· optimized resources and compute power,
· created logs and used third-party services to monitor application and infra at run time, and
· used ML and learning capabilities for the team to assess, optimize and deliver.
We’ve used Akamai, AWS, Oracle Dyn and ImageDeliver to meet requirements of scale and performance. The core requirements were met with best practices in technology. Magento on steroids, married to the best platform and services, has successfully delivered a fail-safe, secure, robust and scalable deployment.
