---
type: redmine-readme
project: Canofy Process Standardization
subject: Deploy to AWS with SAM container images and harden the Lambdas for the real runtime
story_key: nsdl-aws-deploy-lambda-hardening
generated_at: 2026-10-15T16:16:51+05:30
story_type: task
attachments:
  - file: nsdl-aws-deploy-lambda-hardening_20261015-1616__template.yaml
    description: SAM template of the deployed stack (current version)
---

# Deploy to AWS with SAM container images and harden the Lambdas for the real runtime

> Get the NSDL pipeline built, deployed to a real AWS TEST stack and working on the actual Lambda runtime, which behaves differently from a local Docker container.

## 1. Summary
The pipeline code was written without a build or deploy environment. This story covers building the two container images, creating the TEST stack, and fixing everything that only failed on the real platform: local disk and Docker problems, a template dependency bug, data-type bugs at the DynamoDB and Lambda boundaries, and four Lambda sandbox restrictions that `sam local invoke` did not reproduce. Result: the deployed Phase A and Phase B ran end to end against real NSDL and a TEST Frappe site. The stack is `nsdl-sez-boe-automation-test` in `ap-south-1`.

## 2. Requirement
| # | Requirement | Source | Status |
|---|---|---|---|
| R1 | Build both function images and deploy a TEST stack | user | Done |
| R2 | Review the CloudFormation changeset before it is executed | Build Spec / safety check | Done |
| R3 | Test against real deployed resources, not a mock | user | Done |
| R4 | Phase A must run exactly as the daily EventBridge schedule would | user (2026-09-15) | Done |
| R5 | Query window must end the day before the run date | user (2026-09-15) | Done |

Requirement that changed: the user first asked to skip local testing, then asked to test locally ("lets test in local first"); the agreed route became deploy real infrastructure first, then invoke. See Decisions.

## 3. Decisions
| Decision | Options considered | Why this one | Date |
|---|---|---|---|
| Test route | Local only; deploy then invoke | Local runs still call real DynamoDB, S3, Secrets Manager and NSDL, so real infrastructure is needed anyway | 2026-09-13 |
| Move Docker's disk | Keep on C:, move to D: | C: filled up twice during image builds; D: had about 166 GB free | 2026-09-13 |
| Clear Docker's image cache | Re-pull only, clear cache | A corrupted cached layer was reused by digest; clearing was the only way to force a fresh pull | 2026-09-13 |
| Recreate the failed stack | Update in place, delete and recreate | A stack in ROLLBACK_COMPLETE cannot be updated; nothing had finished creating | 2026-09-14 |
| Writable paths | One library fix at a time, `HOME=/tmp` globally | Several libraries assume a writable home; only `/tmp` is writable on Lambda | 2026-09-15 |
| Date window end | `run_date`, `run_date - 1` | NSDL's export for the current day is incomplete (user's correction) | 2026-09-15 |

## 4. Logic & Workflow
Deploy flow: `sam build -> sam deploy (changeset only) -> review -> execute -> populate secret -> invoke -> read CloudWatch logs -> fix -> rebuild -> redeploy`

1. Build both images; push to ECR (the first push is large, see Limitations).
2. Create a changeset and review the resources before executing it.
3. After CREATE_COMPLETE, a human enters the real credentials in the secret.
4. Invoke the deployed functions and read the full CloudWatch log, not the truncated response.

Defects found and fixed (each only showed up after the previous one was fixed):

| # | Symptom | Cause | Fix |
|---|---|---|---|
| 1 | Stack create rolled back | Usage plan referenced the stage by name, no dependency on the generated stage resource | Reference the stage resource |
| 2 | DynamoDB write error on numpy booleans | `_dynamo_safe` only converted floats | Convert numpy booleans and integers |
| 3 | Lambda response could not be serialised | Decimal version and numpy QC values in the return value | Sanitise the summary; version as int |
| 4 | Phase B push failed with Decimal | Numbers read back from DynamoDB are Decimal | Convert at the read boundary |
| 5 | Browser page closed immediately | `/dev/shm` too small on Lambda | `--disable-dev-shm-usage` |
| 6 | `--user-data-dir` rejected | Not accepted as a launch argument | `launch_persistent_context` with a profile under `/tmp` |
| 7 | Chromium zygote crash | Lambda does not allow the capability drop | `--no-zygote --disable-gpu` |
| 8 | Read-only file system under the sandbox user's home | Libraries cache under `$HOME` | EasyOCR model directory in `/tmp` |
| 9 | Same error persisted | Another library also wrote under `$HOME` | `HOME=/tmp` as a Lambda environment variable |
| 10 | Empty current-day data | NSDL export lags one day | Window ends at `run_date - 1` (8 days inclusive) |

## 5. Files & Components Changed
| File / resource | Type | Change | Purpose |
|---|---|---|---|
| `template.yaml` | SAM template | Usage-plan stage dependency; `HOME=/tmp` on both functions | Stack creation and runtime fixes |
| `app/phase_a_fetch/dynamo.py` | Code | Convert numpy types | DynamoDB writes |
| `app/phase_a_fetch/lambda_handler.py` | Code | Sanitise return value; window end `run_date - 1`; persistent-context launch | Response, dates, browser |
| `app/phase_a_fetch/nsdl_download.py` | Code | Chromium launch arguments | Run on the Lambda sandbox |
| `app/phase_a_fetch/ocr_engine.py` and Phase B copy | Code | EasyOCR model directory in `/tmp` | Writable storage |
| `app/phase_b_deliver/dynamo.py`, `shared/json_utils.py` | Code | Decimal to int/float | Phase B push |
| `app/phase_b_deliver/lambda_handler.py` | Code | Same browser launch fixes | Phase B browser |

## 6. Configuration & Deployment
- **Region and stack:** `ap-south-1`, `nsdl-sez-boe-automation-test`; deploy parameters `Environment`, `S3RetentionDays`, `DailyScheduleCronExpression`.
- **Local tools:** Docker Desktop with its disk on a drive with free space, AWS CLI with credentials, SAM CLI.
- **Deploy:** `sam build`, then create the changeset, review, execute. Later deploys go through GitHub Actions (see `nsdl-github-actions-deployment`).
- **After deploy:** fill the secret; subscribe an endpoint to the alerts topic.

## 7. Testing
| Test | Data / Site | Expected | Actual | Status |
|---|---|---|---|---|
| `sam build` of both images | Local Docker | Images built | Succeeded after disk and Docker repairs | Pass |
| Changeset review | New stack | Only additions | 25 new resources, nothing modified | Pass |
| Stack create, first attempt | AWS | CREATE_COMPLETE | Rolled back (usage-plan dependency) | Fail, fixed |
| Stack create, second attempt | AWS | CREATE_COMPLETE | CREATE_COMPLETE (2026-09-14) | Pass |
| Phase A via local invoke | Real NSDL | Staged records | Passed locally; the same code failed on the real Lambda | Pass, then Fail on Lambda |
| Phase A as EventBridge would call it | Deployed Lambda | HTTP 200 | HTTP 200 after fixes 5 to 9 (2026-09-15) | Pass |

## 8. Known Limitations & Open Items
- Each image is about 4.1 GB compressed because `easyocr` pulls the full PyTorch build; a later story (`nsdl-github-actions-deployment`) addresses the size.
- The first image push from the developer machine took about 45 minutes.
- `sam deploy --guided` is interactive and cannot be driven by an agent; parameters were supplied through `samconfig.toml`.
- Not confirmed: whether the earlier "10 defects" count in other write-ups matches this table exactly; this table lists the ones evidenced in the chat.

## 9. Status
| Item | Value |
|---|---|
| Branch | `main` |
| Commits | `2e9ba74` (code state captured in the initial commit, 2026-09-15) |
| PR | None |
| Version | Not applicable at this point |
| Deployed to | TEST stack `nsdl-sez-boe-automation-test` (`ap-south-1`) |

## 10. References
Related stories: `nsdl-lambda-core-pipeline` (the code deployed here), `nsdl-github-actions-deployment`, `nsdl-pdf-attachment-be-dta`. Attachment: the SAM template (see front matter).