# Build Spec — NSDL SEZ BOE/DTA Automation (for Claude Code)

Execute the phases below **in order**. Each phase lists exact commands/files. Steps marked
**⚠️ MANUAL — HUMAN REQUIRED** cannot be done by an agent (they need real credentials, an AWS
console action, or a business decision) — stop, ask the human to complete them, confirm they're
done, then continue.

Reference source: the existing local pipeline at `fw_verify/` (already proven end-to-end against
real NSDL/Frappe data) is the logic source of truth for Phase A/B business logic — port it, don't
reinvent it.

---

## Phase 0 — Prerequisites

⚠️ **MANUAL — HUMAN REQUIRED**, confirm all before proceeding:
- [ ] AWS CLI configured locally with credentials that have rights to create Lambda, API Gateway,
      DynamoDB, S3, EventBridge, IAM roles, Secrets Manager, SNS, SQS in the target account/region.
- [ ] AWS SAM CLI installed (`sam --version`).
- [ ] Docker installed and running (SAM builds container images for Lambda).
- [ ] NSDL portal credentials for the `siddhartha_ftwz` account available.
- [ ] For each client being onboarded at launch: Frappe site URL, and a **dedicated API user**
      already created on that site scoped to `Create`-only on `FTWZ BOE Import`, `FTWZ DTA Sale`
      (+ child tables), and file upload — nothing broader. (See Phase 6 for exact role setup if
      not yet created.)
- [ ] Confirmation of the S3 retention horizon decision (Blueprint §13, item 1) — needed before
      Phase 3's lifecycle policy is written with a real number instead of a placeholder.

---

## Phase 1 — Repo scaffolding

```bash
mkdir nsdl-sez-boe-automation && cd nsdl-sez-boe-automation
git init
mkdir -p app/phase_a_fetch app/phase_b_deliver shared frappe_app
touch README.md .gitignore
```

`.gitignore` must include:
```
.aws-sam/
__pycache__/
*.pyc
.env
config/config.yaml
samconfig.toml
```

Copy the proven pipeline logic in as a reference (read-only, not shipped):
```bash
mkdir -p reference/fw_verify
# copy fw_verify's pipeline/, state/, utils/, config/ modules here for reference during porting
```

---

## Phase 2 — Port Phase A logic into `app/phase_a_fetch/lambda_handler.py`

Source modules to port from `fw_verify`, in this order, adapting each as described:

1. **`pipeline/step1_download.py`** → NSDL login + Playwright/OCR download.
   - Change: `from_date`/`to_date` must no longer be read from a static config block. Compute them
     at invocation time: `to_date = event["run_date"]`, `from_date = to_date - 7 days`.
   - Change: before logging into NSDL, check the `run_date` control record in DynamoDB
     (schema in Phase 4). If `download_status == "PASSED"` for this exact date and no
     `force_refresh` flag is in the event, skip NSDL login entirely and reuse the existing S3 key.
   - Output: raw `.xlsx` files uploaded to `s3://<bucket>/{run_date}/be_export_v{n}.xlsx` and
     `dta_export_v{n}.xlsx` (version bump only on forced re-download, never a silent overwrite).
2. **`pipeline/nsdl_parser.py`** → parsing logic, ported as-is.
3. **`pipeline/step2_transform.py`** → transform + QC, with these required changes:
   - Build the `IEC → client_name` reverse map once per invocation from the `CLIENTS` block in
     Secrets Manager (see Phase 5 for the secret schema).
   - Normalize IECs identically (strip, zero-pad to a fixed width) on both the secret's
     `client_iec_codes` and the parsed dataframe column, in one shared function.
   - Hard-abort the entire invocation (not a per-record skip) if the reverse map build detects an
     IEC mapped to more than one `client_name` — log this loudly, this is the one error class that
     must never resolve silently.
   - Add QC check #6: within one `request_id` group, assert all rows agree on normalized IEC;
     reject the group (not the batch) if not.
   - Partition rows by resolved `client_name` **before** running the existing 5 QC checks — run
     QC and the success-rate gate independently per client partition, not globally across the run.
   - Only new (non-already-terminal-in-DynamoDB) rows are subject to QC/gating each run — the
     7-day lookback re-covers old rows for safety, it does not mean re-validating them.
   - Write each validated client partition to DynamoDB with `status = "STAGED"` (schema in
     Phase 4) instead of pushing directly to Frappe — Phase A never talks to Frappe.
4. Wire these into one `lambda_handler(event, context)` entry point taking
   `{"run_date": "...", "force_refresh": false}`.

---

## Phase 3 — `template.yaml`: S3 + DynamoDB + EventBridge + Phase A Lambda

```yaml
Resources:
  RawFilesBucket:
    Type: AWS::S3::Bucket
    Properties:
      LifecycleConfiguration:
        Rules:
          - Id: TransitionAndExpire
            Status: Enabled
            Transitions:
              - TransitionInDays: 30
                StorageClass: STANDARD_IA
            ExpirationInDays: <CONFIRMED_RETENTION_DAYS>   # from Phase 0 checklist

  StagedRecordsTable:
    Type: AWS::DynamoDB::Table
    Properties:
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - {AttributeName: request_id, AttributeType: S}
        - {AttributeName: client_name, AttributeType: S}
      KeySchema:
        - {AttributeName: request_id, KeyType: HASH}
      GlobalSecondaryIndexes:
        - IndexName: client_name-status-index
          KeySchema:
            - {AttributeName: client_name, KeyType: HASH}
          Projection: {ProjectionType: ALL}
      TimeToLiveSpecification:
        AttributeName: ttl
        Enabled: true

  RunGateTable:
    Type: AWS::DynamoDB::Table
    Properties:
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - {AttributeName: run_date, AttributeType: S}
        - {AttributeName: client_name, AttributeType: S}
      KeySchema:
        - {AttributeName: run_date, KeyType: HASH}
        - {AttributeName: client_name, KeyType: RANGE}

  RunControlTable:
    Type: AWS::DynamoDB::Table
    Properties:
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - {AttributeName: run_date, AttributeType: S}
      KeySchema:
        - {AttributeName: run_date, KeyType: HASH}

  PhaseAFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      Timeout: 900
      MemorySize: 3008
      Events:
        DailySchedule:
          Type: Schedule
          Properties:
            Schedule: "cron(0 9 * * ? *)"   # confirm desired UTC time with human before deploy
            Input: '{"run_date.$": "$$.State.EnteredTime"}'  # adjust to pass a plain run_date string
      Policies:
        - S3CrudPolicy: {BucketName: !Ref RawFilesBucket}
        - DynamoDBCrudPolicy: {TableName: !Ref StagedRecordsTable}
        - DynamoDBCrudPolicy: {TableName: !Ref RunGateTable}
        - DynamoDBCrudPolicy: {TableName: !Ref RunControlTable}
        - AWSSecretsManagerGetSecretValuePolicy: {SecretArn: !Ref NsdlBoeSecrets}
    Metadata:
      DockerContext: app/phase_a_fetch
      Dockerfile: Dockerfile
```

Note: EventBridge `Schedule` event input needs a real `run_date` string (e.g. via an
`EventBridge Scheduler` resource with a fixed input template) rather than the Step-Functions-style
placeholder above — finalize the exact input wiring against whichever of SAM's schedule resource
types is used, and confirm the cron time with the human before first deploy.

`app/phase_a_fetch/Dockerfile` — base on Playwright's official Python image (already known-good
from `fw_verify`'s local requirements), install `requirements.txt`, `playwright install chromium`,
copy `lambda_handler.py` + ported pipeline modules, set `CMD` to the Lambda handler.

---

## Phase 4 — DynamoDB item shapes (reference while coding, not new resources)

Staged record (`StagedRecordsTable`):
```json
{
  "request_id": "172601351484",
  "client_name": "eap",
  "pipeline": "BE",
  "status": "STAGED",
  "payload": { "...": "..." },
  "run_date": "2026-09-12",
  "staged_at": "2026-09-12T15:11:25Z",
  "error_message": null,
  "error_type": null,
  "retry_count": 0,
  "ttl": null
}
```
Set `ttl` only when `status` becomes `ATTACHED` (epoch seconds, now + 30-90 days). Never set `ttl`
on `STAGED`/`PUSHED` items.

Run gate (`RunGateTable`): `{run_date, client_name, status, record_count, qc_summary, first_error}`

Run control (`RunControlTable`):
```json
{"run_date": "2026-09-12", "be_s3_key": "...", "dta_s3_key": "...", "download_status": "PASSED", "version": 2, "downloaded_at": "..."}
```

---

## Phase 5 — Secrets Manager

⚠️ **MANUAL — HUMAN REQUIRED**: create the secret with real values (never let an agent generate or
guess real NSDL/Frappe credentials). Claude Code should create the **secret resource and its IAM
policy reference** in `template.yaml`, and document the expected shape — the human fills in values
via `aws secretsmanager put-secret-value` or the console.

```yaml
  NsdlBoeSecrets:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: nsdl-boe-automation-secrets
```

Expected JSON shape (document this in README, do not hardcode real values anywhere in the repo):
```json
{
  "NSDL_ACCOUNTS": {"siddhartha_ftwz": {"NSDL_USERNAME": "", "NSDL_PASSWORD": "", "entity_id": ""}},
  "CLIENTS": {
    "eap": {"client_iec_codes": [""], "FRAPPE_URL": "", "api_key": "", "api_secret": ""}
  }
}
```

---

## Phase 6 — Port Phase B logic into `app/phase_b_deliver/lambda_handler.py`

Source: `pipeline/step3_pdf_attach.py` + the direct-push portion of `step2_transform.py`, adapted:

1. Entry point receives `client_name` (from the `X-Frappe-Client` header via API Gateway's
   mapping template — see Phase 7).
2. Query `StagedRecordsTable` via the `client_name-status-index` GSI for
   `client_name = <this> AND status = "STAGED"`. If empty, return
   `{"delivered": 0, "message": "No staged records for client '<x>'."}` immediately.
3. Look up that `client_name`'s Frappe credentials from the Phase-5 secret.
4. For each staged record: `FrappeClient.insert()` directly (ported from `fw_verify`'s existing,
   already-proven push code) using the scoped API user. Update DynamoDB status to `PUSHED` on
   success, or set `error_message`/`error_type` and increment `retry_count` on failure — **do not
   abort the whole batch on one record's failure.**
5. For each newly created `FTWZ BOE Import`: fetch the BOE PDF from NSDL and attach via the
   existing direct scoped file-upload call (ported from `step3_pdf_attach.py`, including the
   existing throttle config `pdf_pause_every`/`pdf_pause_seconds`). Update status to `ATTACHED`
   and set `ttl`.
6. Update the per-client `RunGateTable` row for today's `run_date`.
7. Write the `NSDL Sync Run` audit doc back to that client's Frappe (fire-and-log-only — a failure
   here must not roll back or fail the delivery that already succeeded).
8. Return a summary: `{"delivered": N, "attached_pdfs": M, "request_ids": [...]}`.

⚠️ **MANUAL — HUMAN REQUIRED per client onboarded:** create the dedicated Frappe API user with
`Create`-only role scoped to exactly `FTWZ BOE Import` / `FTWZ DTA Sale` (+ child tables) and file
upload — confirm this role exists and is correctly scoped before that client's `api_key`/`api_secret`
go into the Phase 5 secret.

---

## Phase 7 — API Gateway + Frappe-side scheduled job

`template.yaml` addition:
```yaml
  PhaseBFunction:
    Type: AWS::Serverless::Function
    Properties:
      PackageType: Image
      Timeout: 300
      Events:
        DeliverPending:
          Type: Api
          Properties:
            Path: /sez/deliver-pending
            Method: post
            Auth:
              ApiKeyRequired: true
      Policies:
        - DynamoDBCrudPolicy: {TableName: !Ref StagedRecordsTable}
        - DynamoDBCrudPolicy: {TableName: !Ref RunGateTable}
        - AWSSecretsManagerGetSecretValuePolicy: {SecretArn: !Ref NsdlBoeSecrets}
```
Add the API Gateway request mapping template that extracts `X-Frappe-Client` into the Lambda
event, mirroring the pattern already used for the courier/shipping services in
`canofy-document-extraction-lambda-main`.

`frappe_app/` deliverables (for the human to apply to the shared Canofy app codebase, not this repo's
CI/CD):
- `nsdl_sync_run.json` — new doctype definition per Blueprint §10's field table.
- `nsdl_sync_scheduled_job.py` — calls the **existing shared outbound-call helper** (confirmed to
  already exist — get its exact function signature from the human/codebase before writing this;
  do not guess field names) to hit `/sez/deliver-pending` with `X-Frappe-Client` set to that site's
  own client code, creating an `NSDL Sync Run` doc as the parent record.
- Register the job in that app's `hooks.py` under `scheduler_events`.

⚠️ **MANUAL — HUMAN REQUIRED**: apply `frappe_app/` changes to each client's Canofy site and confirm
the scheduled job is enabled — this repo does not deploy Frappe-side code.

---

## Phase 8 — Monitoring & alerting resources

```yaml
  AlertsTopic:
    Type: AWS::SNS::Topic

  PhaseBDeadLetterQueue:
    Type: AWS::SQS::Queue
  # Attach to PhaseBFunction's EventInvokeConfig as OnFailure destination

  PhaseAErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      Namespace: AWS/Lambda
      MetricName: Errors
      Dimensions: [{Name: FunctionName, Value: !Ref PhaseAFunction}]
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      AlarmActions: [!Ref AlertsTopic]

  WatchdogFunction:
    Type: AWS::Serverless::Function
    Properties:
      Events:
        HourlySchedule:
          Type: Schedule
          Properties: {Schedule: "rate(1 hour)"}
      Policies:
        - DynamoDBReadPolicy: {TableName: !Ref StagedRecordsTable}
        - SNSPublishMessagePolicy: {TopicName: !GetAtt AlertsTopic.TopicName}
```
Watchdog logic: scan `StagedRecordsTable` for items with `status = "STAGED"` and `staged_at` older
than 24h → publish to `AlertsTopic` with the affected `client_name`(s).

⚠️ **MANUAL — HUMAN REQUIRED**: subscribe real email/Slack endpoints to `AlertsTopic` after deploy.

---

## Phase 9 — Local testing before any AWS deployment

```bash
sam build
sam local invoke PhaseAFunction --event events/phase_a_test_event.json
sam local invoke PhaseBFunction --event events/phase_b_test_event.json
```
Create `events/phase_a_test_event.json` (`{"run_date": "<a real recent date>", "force_refresh": false}`)
and `events/phase_b_test_event.json` (with a test `X-Frappe-Client` header) before running.

⚠️ **MANUAL — HUMAN REQUIRED**: provide a safe TEST-environment NSDL/Frappe credential set for this
step — never point local testing at production client Frappe sites.

---

## Phase 10 — Deployment

```bash
sam build
sam deploy --guided --stack-name nsdl-sez-boe-automation-test --parameter-overrides Environment=TEST
```
On success, repeat for production only after Phase 9's test-environment run has been verified by a
human:
```bash
sam deploy --guided --stack-name nsdl-sez-boe-automation-prod --parameter-overrides Environment=PROD
```

⚠️ **MANUAL — HUMAN REQUIRED**: review the SAM changeset before confirming each deploy; do not
auto-approve production deployments.

---

## Phase 11 — Post-deploy verification checklist

- [ ] Manually trigger Phase A once via `aws lambda invoke` with a known `run_date`; confirm S3
      files appear and `RunControlTable` gets a `PASSED` row.
- [ ] Confirm `StagedRecordsTable` has rows for at least one onboarded client with `status = STAGED`.
- [ ] From one client's Frappe site, manually run the new scheduled job once; confirm
      `NSDL Sync Run` doc appears with `status = Delivered` and the corresponding `FTWZ BOE Import`
      record exists with its PDF attached.
- [ ] Confirm the CloudWatch alarm, DLQ, and watchdog all show up correctly in the console and that
      SNS subscriptions are receiving test notifications.

---

## Phase 12 — Client onboarding runbook (repeat per new client)

1. ⚠️ **MANUAL**: create the client's dedicated, narrowly-scoped Frappe API user (Phase 6).
2. ⚠️ **MANUAL**: verify that client's Canofy instance has the matching `Attachment Type` master
   and `FTWZ BOE Import`/`FTWZ DTA Sale` doctype schema (Blueprint §9 onboarding checklist).
3. Add the client's block to the Phase 5 secret (`client_iec_codes`, `FRAPPE_URL`, `api_key`,
   `api_secret`) — ⚠️ **MANUAL**, real credentials.
4. ⚠️ **MANUAL**: apply `frappe_app/` doctype + scheduled job to that client's Canofy site, set its
   own `client_name` in the job config, enable the scheduler.
5. Run Phase 11's verification checklist scoped to just this client.