Actions

Copy link

Task #144

open

AS AS

Deploy to AWS with SAM container images and harden the Lambdas for the real runtime

Task #144: Deploy to AWS with SAM container images and harden the Lambdas for the real runtime

Added by Akshath Shettigara 7 days ago.

Status:

New

Priority:

Normal

Assignee:

Akshath Shettigara

Target version:

Backlogs

Start date:

10/15/2026

Due date:

% Done:

Estimated time:

Description

Deploy to AWS with SAM container images and harden the Lambdas for the real runtime¶

Get the NSDL pipeline built, deployed to a real AWS TEST stack and working on the actual Lambda runtime, which behaves differently from a local Docker container.

1. Summary¶

The pipeline code was written without a build or deploy environment. This story covers building the two container images, creating the TEST stack, and fixing everything that only failed on the real platform: local disk and Docker problems, a template dependency bug, data-type bugs at the DynamoDB and Lambda boundaries, and four Lambda sandbox restrictions that sam local invoke did not reproduce. Result: the deployed Phase A and Phase B ran end to end against real NSDL and a TEST Frappe site. The stack is nsdl-sez-boe-automation-test in ap-south-1.

2. Requirement¶

#	Requirement	Source	Status
R1	Build both function images and deploy a TEST stack	user	Done
R2	Review the CloudFormation changeset before it is executed	Build Spec / safety check	Done
R3	Test against real deployed resources, not a mock	user	Done
R4	Phase A must run exactly as the daily EventBridge schedule would	user (2026-09-15)	Done
R5	Query window must end the day before the run date	user (2026-09-15)	Done

Requirement that changed: the user first asked to skip local testing, then asked to test locally ("lets test in local first"); the agreed route became deploy real infrastructure first, then invoke. See Decisions.

3. Decisions¶

Decision	Options considered	Why this one	Date
Test route	Local only; deploy then invoke	Local runs still call real DynamoDB, S3, Secrets Manager and NSDL, so real infrastructure is needed anyway	2026-09-13
Move Docker's disk	Keep on C:, move to D:	C: filled up twice during image builds; D: had about 166 GB free	2026-09-13
Clear Docker's image cache	Re-pull only, clear cache	A corrupted cached layer was reused by digest; clearing was the only way to force a fresh pull	2026-09-13
Recreate the failed stack	Update in place, delete and recreate	A stack in ROLLBACK_COMPLETE cannot be updated; nothing had finished creating	2026-09-14
Writable paths	One library fix at a time, `HOME=/tmp` globally	Several libraries assume a writable home; only `/tmp` is writable on Lambda	2026-09-15
Date window end	`run_date`, `run_date - 1`	NSDL's export for the current day is incomplete (user's correction)	2026-09-15

4. Logic & Workflow¶

Deploy flow: sam build -> sam deploy (changeset only) -> review -> execute -> populate secret -> invoke -> read CloudWatch logs -> fix -> rebuild -> redeploy

Build both images; push to ECR (the first push is large, see Limitations).
Create a changeset and review the resources before executing it.
After CREATE_COMPLETE, a human enters the real credentials in the secret.
Invoke the deployed functions and read the full CloudWatch log, not the truncated response.

Defects found and fixed (each only showed up after the previous one was fixed):

#	Symptom	Cause	Fix
1	Stack create rolled back	Usage plan referenced the stage by name, no dependency on the generated stage resource	Reference the stage resource
2	DynamoDB write error on numpy booleans	`_dynamo_safe` only converted floats	Convert numpy booleans and integers
3	Lambda response could not be serialised	Decimal version and numpy QC values in the return value	Sanitise the summary; version as int
4	Phase B push failed with Decimal	Numbers read back from DynamoDB are Decimal	Convert at the read boundary
5	Browser page closed immediately	`/dev/shm` too small on Lambda	`--disable-dev-shm-usage`
6	`--user-data-dir` rejected	Not accepted as a launch argument	`launch_persistent_context` with a profile under `/tmp`
7	Chromium zygote crash	Lambda does not allow the capability drop	`--no-zygote --disable-gpu`
8	Read-only file system under the sandbox user's home	Libraries cache under `$HOME`	EasyOCR model directory in `/tmp`
9	Same error persisted	Another library also wrote under `$HOME`	`HOME=/tmp` as a Lambda environment variable
10	Empty current-day data	NSDL export lags one day	Window ends at `run_date - 1` (8 days inclusive)

5. Files & Components Changed¶

File / resource	Type	Change	Purpose
`template.yaml`	SAM template	Usage-plan stage dependency; `HOME=/tmp` on both functions	Stack creation and runtime fixes
`app/phase_a_fetch/dynamo.py`	Code	Convert numpy types	DynamoDB writes
`app/phase_a_fetch/lambda_handler.py`	Code	Sanitise return value; window end `run_date - 1`; persistent-context launch	Response, dates, browser
`app/phase_a_fetch/nsdl_download.py`	Code	Chromium launch arguments	Run on the Lambda sandbox
`app/phase_a_fetch/ocr_engine.py` and Phase B copy	Code	EasyOCR model directory in `/tmp`	Writable storage
`app/phase_b_deliver/dynamo.py`, `shared/json_utils.py`	Code	Decimal to int/float	Phase B push
`app/phase_b_deliver/lambda_handler.py`	Code	Same browser launch fixes	Phase B browser

6. Configuration & Deployment¶

Region and stack: ap-south-1, nsdl-sez-boe-automation-test; deploy parameters Environment, S3RetentionDays, DailyScheduleCronExpression.
Local tools: Docker Desktop with its disk on a drive with free space, AWS CLI with credentials, SAM CLI.
Deploy: sam build, then create the changeset, review, execute. Later deploys go through GitHub Actions (see nsdl-github-actions-deployment).
After deploy: fill the secret; subscribe an endpoint to the alerts topic.

7. Testing¶

Test	Data / Site	Expected	Actual	Status
`sam build` of both images	Local Docker	Images built	Succeeded after disk and Docker repairs	Pass
Changeset review	New stack	Only additions	25 new resources, nothing modified	Pass
Stack create, first attempt	AWS	CREATE_COMPLETE	Rolled back (usage-plan dependency)	Fail, fixed
Stack create, second attempt	AWS	CREATE_COMPLETE	CREATE_COMPLETE (2026-09-14)	Pass
Phase A via local invoke	Real NSDL	Staged records	Passed locally; the same code failed on the real Lambda	Pass, then Fail on Lambda
Phase A as EventBridge would call it	Deployed Lambda	HTTP 200	HTTP 200 after fixes 5 to 9 (2026-09-15)	Pass

8. Known Limitations & Open Items¶

Each image is about 4.1 GB compressed because easyocr pulls the full PyTorch build; a later story (nsdl-github-actions-deployment) addresses the size.
The first image push from the developer machine took about 45 minutes.
sam deploy --guided is interactive and cannot be driven by an agent; parameters were supplied through samconfig.toml.
Not confirmed: whether the earlier "10 defects" count in other write-ups matches this table exactly; this table lists the ones evidenced in the chat.

9. Status¶

Item	Value
Branch	`main`
Commits	`2e9ba74` (code state captured in the initial commit, 2026-09-15)
PR	None
Version	Not applicable at this point
Deployed to	TEST stack `nsdl-sez-boe-automation-test` (`ap-south-1`)

10. References¶

Related stories: nsdl-lambda-core-pipeline (the code deployed here), nsdl-github-actions-deployment, nsdl-pdf-attachment-be-dta. Attachment: the SAM template (see front matter).

Pushed to Redmine on 15 Oct 2026, 16:35 from nsdl-aws-deploy-lambda-hardening_20261015-1616_20261015-163531.md (README created 15 Oct 2026, 16:35).

Files

Download all files

README.md (7.96 KB) README.md	README: Deploy to AWS with SAM container images and harden the Lambdas for the real runtime	Akshath Shettigara, 10/15/2026 04:35 PM
nsdl-aws-deploy-lambda-hardening_20261015-1616__template.yaml (27.3 KB) nsdl-aws-deploy-lambda-hardening_20261015-1616__template.yaml	SAM template of the deployed stack (current version)	Akshath Shettigara, 10/15/2026 04:35 PM
template.yaml (27.3 KB) template.yaml	YAML file: AWSTemplateFormatVersion: "2010-09-09"	Akshath Shettigara, 10/15/2026 04:35 PM

No data to display