Project

General

Profile

Download (8.64 KB)

Task #150 » README.md

README: GitHub Actions deployment pipeline for the NSDL Lambda stack, following the versioning SOP - Akshath Shettigara, 10/15/2026 04:35 PM

 

type: redmine-readme
project: Canofy Process Standardization
subject: GitHub Actions deployment pipeline for the NSDL Lambda stack, following the versioning SOP
story_key: nsdl-github-actions-deployment
generated_at: 2026-10-15T16:16:51+05:30
story_type: task
attachments:

  • file: nsdl-github-actions-deployment_20261015-1616__deploy-test.yml
    description: Deploy to Test workflow (AWS account ID masked)
  • file: nsdl-github-actions-deployment_20261015-1616__deploy-prod.yml
    description: Deploy to Production workflow (not yet exercised)
  • file: nsdl-github-actions-deployment_20261015-1616__github-oidc-roles.yaml
    description: CloudFormation template for the OIDC deploy role and its least-privilege policy

GitHub Actions deployment pipeline for the NSDL Lambda stack, following the versioning SOP

Merging a pull request into test now builds the two container images and deploys the TEST stack with a least-privilege AWS role. This story also covers the failures met while getting that pipeline to pass.

1. Summary

Until now the stack was deployed by hand from a developer machine. The user asked to deploy from GitHub and to follow the company SOP "Code Versioning and Deployment Standards". This story adds the branch and version rules, a Ruff quality gate, a deploy workflow per environment, and an IAM role that GitHub assumes through OIDC (no stored AWS keys). Getting the TEST pipeline green took seven fix rounds; each is listed below. The production workflow exists but has never run. Repo: canofytech/canofy-NSDL-data-extraction.

2. Requirement

# Requirement Source Status
R1 Follow the SOP: main and test branches, feature/AI-[id]- branches, SemVer with -beta on test user + SOP Done (the Task ID is still TBD)
R2 Merge into test deploys to TEST automatically SOP Done
R3 Release on main deploys to PROD SOP Workflow written, not run
R4 No long-lived AWS keys in GitHub design Done (OIDC)
R5 Role permissions limited to this stack's resources design Done, with the exceptions in section 8
R6 Ruff must pass before any deploy SOP Done
R7 The Lambda repo holds no Frappe code user Done

3. Decisions

Decision Options considered Why this one Date
One role template per environment One shared policy for both roles; one template deployed per environment A shared policy would let the TEST role touch PROD-named resources 2026-09-15
Trust condition Branch name; GitHub Environment Environment test or production is harder to spoof 2026-09-15
Ruff baseline Leave findings; fix and format once The gate would otherwise fail from the first run 2026-09-15
Role secret location Environment secret; repository secret The user did not see the Environments page, and jobs with an environment can read repository secrets 2026-10-11
PyTorch in the images Default (CUDA) build; CPU-only build Lambda has no GPU and the CUDA build overflows the runner disk 2026-10-11
Artifact bucket and image repos --resolve-s3, --resolve-image-repos; explicit values The first option needs access to SAM's helper stacks that the role is not allowed to touch 2026-10-11
Fix permissions Console edit; stack update Console edits made the live policy work at once; the same change was committed to the template so they stay in sync 2026-10-11

4. Logic & Workflow

PR merged into test -> checkout -> Ruff check and format check -> assume AWS role (OIDC) -> sam validate --lint -> sam build -> sam deploy to stack nsdl-sez-boe-automation-test

  1. The workflow runs only on push to test; a merge into main alone does not deploy.
  2. deploy-prod.yml is meant to run on a published release on main, and reads the retention days and cron expression from environment variables that must be set first, so it fails loudly instead of guessing.
  3. A failed run cannot be fixed by re-running: a re-run uses the original commit. A fix needs a new merge into test.
  4. Changes to the role template do not change AWS by themselves; the role stack must be updated by someone with IAM rights.

Failures met and fixed, in order:

# Symptom Cause Fix Commit
1 "Could not load credentials" Role ARN secret not visible to the job Add AWS_DEPLOY_ROLE_ARN as a repository secret None (setting)
2 no space left on device in sam build CUDA PyTorch pulled in by easyocr CPU-only torch/torchvision installed first in both Dockerfiles 79e2a98
3 AccessDenied on sam deploy --resolve-s3 Role cannot manage SAM's helper stacks Pass existing bucket and image repositories 8358abb
4 Cron parameter cut at the first space Shell word splitting Quote the value e081376
5 CreateChangeSet denied Needed on the SAM transform resource New statement; also lambda:UpdateFunctionEventInvokeConfig 0e53e89
6 Three function updates failed and the rollback failed Role lacked the watchdog function, the alerts topic and the dead-letter queue Add them; run continue-update-rollback from the console 0bdf75b
7 EventBridge actions denied SAM truncates rule names, so the role's name pattern did not match Pattern for the truncated names, one each for test and prod 0bdf75b

5. Files & Components Changed

File / resource Type Change Purpose
.github/workflows/deploy-test.yml Workflow New; later explicit bucket and repos, quoted cron TEST deploy
.github/workflows/deploy-prod.yml Workflow New PROD deploy on release
infra/github-oidc-roles.yaml CloudFormation New; later transform, watchdog/SNS/SQS and Events fixes Role and policy
.github/pull_request_template.md Template New PR rules from the SOP
VERSION File New (1.0.0-beta; 1.1.0-beta with the OTP login) SemVer
app/phase_a_fetch/Dockerfile, app/phase_b_deliver/Dockerfile Docker CPU-only PyTorch Fit the runner disk
Python sources Code Ruff fixes and formatting (no behaviour change intended) Quality gate

6. Configuration & Deployment

Item Value Where
Role stack nsdl-sez-boe-github-oidc-test, deployed from infra/github-oidc-roles.yaml with parameters Environment, StackNamePrefix, GitHubOrg, GitHubRepo AWS CloudFormation
Secret AWS_DEPLOY_ROLE_ARN GitHub repository secret
PROD inputs AWS_DEPLOY_ROLE_ARN_PROD, PROD_S3_RETENTION_DAYS, PROD_CRON_EXPRESSION GitHub environment production, not yet set
Region ap-south-1 Workflow

To roll a permission change out: update the role stack from the template, then merge a new commit into test.

7. Testing

Test Data / Site Expected Actual Status
Ruff check and format Whole repo Zero findings Passed after the baseline fix Pass
sam validate of the role template Local Valid Valid Pass
Role stack deploy AWS account Role created Created Pass
Deploy to Test, rounds 1 to 7 GitHub Actions Stack updated Failed six times with the causes above Fail, fixed
Deploy to Test, final GitHub Actions Stack updated Succeeded (user confirmation, 2026-10-11) Pass

8. Known Limitations & Open Items

  • ECR, API Gateway and CloudWatch Logs statements stay broad because those services cannot be scoped by name at creation time.
  • deploy-prod.yml still needs the same quoting fix for the cron value, and the PROD role and stack do not exist. Not confirmed beyond that.
  • The Task ID in branch names is still TBD; renaming means deleting and re-pushing the remote branch.
  • Pull requests are created by hand because the GitHub CLI is not installed on the machine.
  • The first scaffolding commit went directly to main, a one-time deviation from the SOP.
  • The user could not see the repository's Environments page, which suggests limited admin rights on the repo; not confirmed.

9. Status

Item Value
Branch main for the scaffolding; feature/AI-TBD-nsdl-otp-login for the fixes
Commits cb8cffc, 21078a6, 79e2a98, 8358abb, e081376, 0e53e89, 0bdf75b
PR Several PRs into test created by the user; numbers #1 to #3 appear in the chat, others not confirmed
Version 1.1.0-beta
Deployed to TEST stack, via the pipeline

10. References

Related stories: nsdl-aws-deploy-lambda-hardening (manual deploys before this), nsdl-email-otp-login (the change that triggered the first runs), nsdl-eventbridge-scheduling.

(1-1/7)