Task #144 » README.md
type: redmine-readme
project: Canofy Process Standardization
subject: Deploy to AWS with SAM container images and harden the Lambdas for the real runtime
story_key: nsdl-aws-deploy-lambda-hardening
generated_at: 2026-10-15T16:16:51+05:30
story_type: task
attachments:
- file: nsdl-aws-deploy-lambda-hardening_20261015-1616__template.yaml
description: SAM template of the deployed stack (current version)
Deploy to AWS with SAM container images and harden the Lambdas for the real runtime
Get the NSDL pipeline built, deployed to a real AWS TEST stack and working on the actual Lambda runtime, which behaves differently from a local Docker container.
1. Summary
The pipeline code was written without a build or deploy environment. This story covers building the two container images, creating the TEST stack, and fixing everything that only failed on the real platform: local disk and Docker problems, a template dependency bug, data-type bugs at the DynamoDB and Lambda boundaries, and four Lambda sandbox restrictions that sam local invoke did not reproduce. Result: the deployed Phase A and Phase B ran end to end against real NSDL and a TEST Frappe site. The stack is nsdl-sez-boe-automation-test in ap-south-1.
2. Requirement
| # | Requirement | Source | Status |
|---|---|---|---|
| R1 | Build both function images and deploy a TEST stack | user | Done |
| R2 | Review the CloudFormation changeset before it is executed | Build Spec / safety check | Done |
| R3 | Test against real deployed resources, not a mock | user | Done |
| R4 | Phase A must run exactly as the daily EventBridge schedule would | user (2026-09-15) | Done |
| R5 | Query window must end the day before the run date | user (2026-09-15) | Done |
Requirement that changed: the user first asked to skip local testing, then asked to test locally ("lets test in local first"); the agreed route became deploy real infrastructure first, then invoke. See Decisions.
3. Decisions
| Decision | Options considered | Why this one | Date |
|---|---|---|---|
| Test route | Local only; deploy then invoke | Local runs still call real DynamoDB, S3, Secrets Manager and NSDL, so real infrastructure is needed anyway | 2026-09-13 |
| Move Docker's disk | Keep on C:, move to D: | C: filled up twice during image builds; D: had about 166 GB free | 2026-09-13 |
| Clear Docker's image cache | Re-pull only, clear cache | A corrupted cached layer was reused by digest; clearing was the only way to force a fresh pull | 2026-09-13 |
| Recreate the failed stack | Update in place, delete and recreate | A stack in ROLLBACK_COMPLETE cannot be updated; nothing had finished creating | 2026-09-14 |
| Writable paths | One library fix at a time, HOME=/tmp globally |
Several libraries assume a writable home; only /tmp is writable on Lambda |
2026-09-15 |
| Date window end |
run_date, run_date - 1
|
NSDL's export for the current day is incomplete (user's correction) | 2026-09-15 |
4. Logic & Workflow
Deploy flow: sam build -> sam deploy (changeset only) -> review -> execute -> populate secret -> invoke -> read CloudWatch logs -> fix -> rebuild -> redeploy
- Build both images; push to ECR (the first push is large, see Limitations).
- Create a changeset and review the resources before executing it.
- After CREATE_COMPLETE, a human enters the real credentials in the secret.
- Invoke the deployed functions and read the full CloudWatch log, not the truncated response.
Defects found and fixed (each only showed up after the previous one was fixed):
| # | Symptom | Cause | Fix |
|---|---|---|---|
| 1 | Stack create rolled back | Usage plan referenced the stage by name, no dependency on the generated stage resource | Reference the stage resource |
| 2 | DynamoDB write error on numpy booleans |
_dynamo_safe only converted floats |
Convert numpy booleans and integers |
| 3 | Lambda response could not be serialised | Decimal version and numpy QC values in the return value | Sanitise the summary; version as int |
| 4 | Phase B push failed with Decimal | Numbers read back from DynamoDB are Decimal | Convert at the read boundary |
| 5 | Browser page closed immediately |
/dev/shm too small on Lambda |
--disable-dev-shm-usage |
| 6 |
--user-data-dir rejected |
Not accepted as a launch argument |
launch_persistent_context with a profile under /tmp
|
| 7 | Chromium zygote crash | Lambda does not allow the capability drop | --no-zygote --disable-gpu |
| 8 | Read-only file system under the sandbox user's home | Libraries cache under $HOME
|
EasyOCR model directory in /tmp
|
| 9 | Same error persisted | Another library also wrote under $HOME
|
HOME=/tmp as a Lambda environment variable |
| 10 | Empty current-day data | NSDL export lags one day | Window ends at run_date - 1 (8 days inclusive) |
5. Files & Components Changed
| File / resource | Type | Change | Purpose |
|---|---|---|---|
template.yaml |
SAM template | Usage-plan stage dependency; HOME=/tmp on both functions |
Stack creation and runtime fixes |
app/phase_a_fetch/dynamo.py |
Code | Convert numpy types | DynamoDB writes |
app/phase_a_fetch/lambda_handler.py |
Code | Sanitise return value; window end run_date - 1; persistent-context launch |
Response, dates, browser |
app/phase_a_fetch/nsdl_download.py |
Code | Chromium launch arguments | Run on the Lambda sandbox |
app/phase_a_fetch/ocr_engine.py and Phase B copy |
Code | EasyOCR model directory in /tmp
|
Writable storage |
app/phase_b_deliver/dynamo.py, shared/json_utils.py
|
Code | Decimal to int/float | Phase B push |
app/phase_b_deliver/lambda_handler.py |
Code | Same browser launch fixes | Phase B browser |
6. Configuration & Deployment
-
Region and stack:
ap-south-1,nsdl-sez-boe-automation-test; deploy parametersEnvironment,S3RetentionDays,DailyScheduleCronExpression. - Local tools: Docker Desktop with its disk on a drive with free space, AWS CLI with credentials, SAM CLI.
-
Deploy:
sam build, then create the changeset, review, execute. Later deploys go through GitHub Actions (seensdl-github-actions-deployment). - After deploy: fill the secret; subscribe an endpoint to the alerts topic.
7. Testing
| Test | Data / Site | Expected | Actual | Status |
|---|---|---|---|---|
sam build of both images |
Local Docker | Images built | Succeeded after disk and Docker repairs | Pass |
| Changeset review | New stack | Only additions | 25 new resources, nothing modified | Pass |
| Stack create, first attempt | AWS | CREATE_COMPLETE | Rolled back (usage-plan dependency) | Fail, fixed |
| Stack create, second attempt | AWS | CREATE_COMPLETE | CREATE_COMPLETE (2026-09-14) | Pass |
| Phase A via local invoke | Real NSDL | Staged records | Passed locally; the same code failed on the real Lambda | Pass, then Fail on Lambda |
| Phase A as EventBridge would call it | Deployed Lambda | HTTP 200 | HTTP 200 after fixes 5 to 9 (2026-09-15) | Pass |
8. Known Limitations & Open Items
- Each image is about 4.1 GB compressed because
easyocrpulls the full PyTorch build; a later story (nsdl-github-actions-deployment) addresses the size. - The first image push from the developer machine took about 45 minutes.
-
sam deploy --guidedis interactive and cannot be driven by an agent; parameters were supplied throughsamconfig.toml. - Not confirmed: whether the earlier "10 defects" count in other write-ups matches this table exactly; this table lists the ones evidenced in the chat.
9. Status
| Item | Value |
|---|---|
| Branch | main |
| Commits |
2e9ba74 (code state captured in the initial commit, 2026-09-15) |
| PR | None |
| Version | Not applicable at this point |
| Deployed to | TEST stack nsdl-sez-boe-automation-test (ap-south-1) |
10. References
Related stories: nsdl-lambda-core-pipeline (the code deployed here), nsdl-github-actions-deployment, nsdl-pdf-attachment-be-dta. Attachment: the SAM template (see front matter).