Actions
Task #150
open
AS
AS
GitHub Actions deployment pipeline for the NSDL Lambda stack, following the versioning SOP
Task #150:
GitHub Actions deployment pipeline for the NSDL Lambda stack, following the versioning SOP
Start date:
10/15/2026
Due date:
% Done:
0%
Estimated time:
Description
GitHub Actions deployment pipeline for the NSDL Lambda stack, following the versioning SOP¶
Merging a pull request into
testnow builds the two container images and deploys the TEST stack with a least-privilege AWS role. This story also covers the failures met while getting that pipeline to pass.
1. Summary¶
Until now the stack was deployed by hand from a developer machine. The user asked to deploy from GitHub and to follow the company SOP "Code Versioning and Deployment Standards". This story adds the branch and version rules, a Ruff quality gate, a deploy workflow per environment, and an IAM role that GitHub assumes through OIDC (no stored AWS keys). Getting the TEST pipeline green took seven fix rounds; each is listed below. The production workflow exists but has never run. Repo: canofytech/canofy-NSDL-data-extraction.
2. Requirement¶
| # | Requirement | Source | Status |
|---|---|---|---|
| R1 | Follow the SOP: main and test branches, feature/AI-[id]- branches, SemVer with -beta on test |
user + SOP | Done (the Task ID is still TBD) |
| R2 | Merge into test deploys to TEST automatically |
SOP | Done |
| R3 | Release on main deploys to PROD |
SOP | Workflow written, not run |
| R4 | No long-lived AWS keys in GitHub | design | Done (OIDC) |
| R5 | Role permissions limited to this stack's resources | design | Done, with the exceptions in section 8 |
| R6 | Ruff must pass before any deploy | SOP | Done |
| R7 | The Lambda repo holds no Frappe code | user | Done |
3. Decisions¶
| Decision | Options considered | Why this one | Date |
|---|---|---|---|
| One role template per environment | One shared policy for both roles; one template deployed per environment | A shared policy would let the TEST role touch PROD-named resources | 2026-09-15 |
| Trust condition | Branch name; GitHub Environment | Environment test or production is harder to spoof |
2026-09-15 |
| Ruff baseline | Leave findings; fix and format once | The gate would otherwise fail from the first run | 2026-09-15 |
| Role secret location | Environment secret; repository secret | The user did not see the Environments page, and jobs with an environment can read repository secrets | 2026-10-11 |
| PyTorch in the images | Default (CUDA) build; CPU-only build | Lambda has no GPU and the CUDA build overflows the runner disk | 2026-10-11 |
| Artifact bucket and image repos |
--resolve-s3, --resolve-image-repos; explicit values |
The first option needs access to SAM's helper stacks that the role is not allowed to touch | 2026-10-11 |
| Fix permissions | Console edit; stack update | Console edits made the live policy work at once; the same change was committed to the template so they stay in sync | 2026-10-11 |
4. Logic & Workflow¶
PR merged into test -> checkout -> Ruff check and format check -> assume AWS role (OIDC) -> sam validate --lint -> sam build -> sam deploy to stack nsdl-sez-boe-automation-test
- The workflow runs only on push to
test; a merge intomainalone does not deploy. -
deploy-prod.ymlis meant to run on a published release onmain, and reads the retention days and cron expression from environment variables that must be set first, so it fails loudly instead of guessing. - A failed run cannot be fixed by re-running: a re-run uses the original commit. A fix needs a new merge into
test. - Changes to the role template do not change AWS by themselves; the role stack must be updated by someone with IAM rights.
Failures met and fixed, in order:
| # | Symptom | Cause | Fix | Commit |
|---|---|---|---|---|
| 1 | "Could not load credentials" | Role ARN secret not visible to the job | Add AWS_DEPLOY_ROLE_ARN as a repository secret |
None (setting) |
| 2 |
no space left on device in sam build
|
CUDA PyTorch pulled in by easyocr
|
CPU-only torch/torchvision installed first in both Dockerfiles |
79e2a98 |
| 3 | AccessDenied on sam deploy --resolve-s3
|
Role cannot manage SAM's helper stacks | Pass existing bucket and image repositories | 8358abb |
| 4 | Cron parameter cut at the first space | Shell word splitting | Quote the value | e081376 |
| 5 |
CreateChangeSet denied |
Needed on the SAM transform resource | New statement; also lambda:UpdateFunctionEventInvokeConfig
|
0e53e89 |
| 6 | Three function updates failed and the rollback failed | Role lacked the watchdog function, the alerts topic and the dead-letter queue | Add them; run continue-update-rollback from the console |
0bdf75b |
| 7 | EventBridge actions denied | SAM truncates rule names, so the role's name pattern did not match | Pattern for the truncated names, one each for test and prod | 0bdf75b |
5. Files & Components Changed¶
| File / resource | Type | Change | Purpose |
|---|---|---|---|
.github/workflows/deploy-test.yml |
Workflow | New; later explicit bucket and repos, quoted cron | TEST deploy |
.github/workflows/deploy-prod.yml |
Workflow | New | PROD deploy on release |
infra/github-oidc-roles.yaml |
CloudFormation | New; later transform, watchdog/SNS/SQS and Events fixes | Role and policy |
.github/pull_request_template.md |
Template | New | PR rules from the SOP |
VERSION |
File | New (1.0.0-beta; 1.1.0-beta with the OTP login) |
SemVer |
app/phase_a_fetch/Dockerfile, app/phase_b_deliver/Dockerfile
|
Docker | CPU-only PyTorch | Fit the runner disk |
| Python sources | Code | Ruff fixes and formatting (no behaviour change intended) | Quality gate |
6. Configuration & Deployment¶
| Item | Value | Where |
|---|---|---|
| Role stack |
nsdl-sez-boe-github-oidc-test, deployed from infra/github-oidc-roles.yaml with parameters Environment, StackNamePrefix, GitHubOrg, GitHubRepo
|
AWS CloudFormation |
| Secret | AWS_DEPLOY_ROLE_ARN |
GitHub repository secret |
| PROD inputs |
AWS_DEPLOY_ROLE_ARN_PROD, PROD_S3_RETENTION_DAYS, PROD_CRON_EXPRESSION
|
GitHub environment production, not yet set |
| Region | ap-south-1 |
Workflow |
To roll a permission change out: update the role stack from the template, then merge a new commit into test.
7. Testing¶
| Test | Data / Site | Expected | Actual | Status |
|---|---|---|---|---|
| Ruff check and format | Whole repo | Zero findings | Passed after the baseline fix | Pass |
sam validate of the role template |
Local | Valid | Valid | Pass |
| Role stack deploy | AWS account | Role created | Created | Pass |
| Deploy to Test, rounds 1 to 7 | GitHub Actions | Stack updated | Failed six times with the causes above | Fail, fixed |
| Deploy to Test, final | GitHub Actions | Stack updated | Succeeded (user confirmation, 2026-10-11) | Pass |
8. Known Limitations & Open Items¶
- ECR, API Gateway and CloudWatch Logs statements stay broad because those services cannot be scoped by name at creation time.
-
deploy-prod.ymlstill needs the same quoting fix for the cron value, and the PROD role and stack do not exist. Not confirmed beyond that. - The Task ID in branch names is still
TBD; renaming means deleting and re-pushing the remote branch. - Pull requests are created by hand because the GitHub CLI is not installed on the machine.
- The first scaffolding commit went directly to
main, a one-time deviation from the SOP. - The user could not see the repository's Environments page, which suggests limited admin rights on the repo; not confirmed.
9. Status¶
| Item | Value |
|---|---|
| Branch |
main for the scaffolding; feature/AI-TBD-nsdl-otp-login for the fixes |
| Commits |
cb8cffc, 21078a6, 79e2a98, 8358abb, e081376, 0e53e89, 0bdf75b
|
| PR | Several PRs into test created by the user; numbers #1 to #3 appear in the chat, others not confirmed |
| Version | 1.1.0-beta |
| Deployed to | TEST stack, via the pipeline |
10. References¶
Related stories: nsdl-aws-deploy-lambda-hardening (manual deploys before this), nsdl-email-otp-login (the change that triggered the first runs), nsdl-eventbridge-scheduling.
Pushed to Redmine on 15 Oct 2026, 16:35 from nsdl-github-actions-deployment_20261015-1616_20261015-163535.md (README created 15 Oct 2026, 16:35).
Files
No data to display
Actions