---
type: redmine-readme
project: Canofy Process Standardization
subject: Attach NSDL print PDFs to DTA Sale and BOE records, with a per-invocation batch cap
story_key: nsdl-pdf-attachment-be-dta
generated_at: 2026-10-15T16:16:51+05:30
story_type: feature
attachments:
  - file: nsdl-pdf-attachment-be-dta_20261015-1616__nsdl_browser.py
    description: Phase B browser helpers including the keep-alive pause
---

# Attach NSDL print PDFs to DTA Sale and BOE records, with a per-invocation batch cap

> Phase B now fetches the NSDL print PDF and attaches it to every delivered record, for DTA Sale as well as BOE, and limits how many PDFs one Lambda run handles so a large backlog cannot hit the 900 s limit.

## 1. Summary
The first Phase B version attached a PDF only to BE (`FTWZ BOE Import`) records and marked DTA (`FTWZ DTA Sale`) records done as soon as they were created, copying a note in the reference pipeline that DTA has no print PDF. The user questioned this and supplied an updated reference pipeline showing that NSDL's Search Request -> Print flow is the same for DTA. DTA records now go through the same PDF step. A keep-alive pause protects the NSDL session during cooldowns, and a batch cap defers the surplus to the next invocation.

## 2. Requirement
| # | Requirement | Source | Status |
|---|---|---|---|
| R1 | DTA Sale documents get the NSDL print PDF attached | user (2026-09-15) | Done |
| R2 | Pausing between downloads must not let the NSDL session expire | updated reference pipeline | Done |
| R3 | One run must not exceed the 900 s Lambda limit on a large backlog | user: "Yes, tune the pause/batch size to avoid the 900s timeout" | Done |
| R4 | A retried or duplicated run must not create duplicate documents | idempotency test | Done (verified) |

## 3. Decisions
| Decision | Options considered | Why this one | Date |
|---|---|---|---|
| Add DTA PDF step | Keep DTA without PDF; ask for NSDL selectors; port updated reference | The updated reference shows the same print button on both request types | 2026-09-15 |
| Handle the cooldown | Shorter pause; keep 120 s every 15 downloads with scrolling | A shorter pause risks a silent NSDL session expiry mid-batch | 2026-09-15 |
| Avoid the 900 s limit | Shrink pauses; cap items per invocation | The design is resumable, so leftover items stay `PUSHED` for the next run | 2026-09-15 |
| Cap value | Not specified by the user | 30 per invocation, as an env var default; adjustable | 2026-09-15 |

The cap applies per invocation, and each invocation serves one client, so clients do not share a budget.

## 4. Logic & Workflow
`push record -> status PUSHED -> (batch of up to 30) -> NSDL login -> for each: search request ID -> Print -> PDF -> attach to the record's DocType -> ATTACHED`

1. A pushed BE or DTA record stays `PUSHED` until its PDF is attached.
2. Each pending entry carries its DocType, so the attach call targets the right document type.
3. After every 15 downloads the browser idles for 120 s while scrolling, so NSDL does not log the session out.
4. At most `MAX_PDF_ATTACHES_PER_INVOCATION` records are processed per run; the rest are processed by the next run.

## 5. Files & Components Changed
| File / resource | Type | Change | Purpose |
|---|---|---|---|
| `app/phase_b_deliver/lambda_handler.py` | Code | DTA goes through the PDF step; DocType in pending entries; batch cap | Core change |
| `app/phase_b_deliver/nsdl_browser.py` | Code | Added `idle_keepalive_pause` | Keep session alive |
| `app/phase_b_deliver/pdf_fetch.py`, `dynamo.py` | Code | Docstrings corrected | Remove the wrong DTA claim |
| `template.yaml` | SAM template | `MAX_PDF_ATTACHES_PER_INVOCATION: "30"` on PhaseBFunction | Configurable cap |

## 6. Configuration & Deployment
| Setting | Value | Where |
|---|---|---|
| `PDF_PAUSE_EVERY` | 15 | Lambda env var |
| `PDF_PAUSE_SECONDS` | 120 | Lambda env var |
| `MAX_PDF_ATTACHES_PER_INVOCATION` | 30 | Lambda env var |

Deployed to the TEST stack as a PhaseBFunction-only change.

## 7. Testing
| Test | Data / Site | Expected | Actual | Status |
|---|---|---|---|---|
| Build check | Local | Builds cleanly | Built | Pass |
| DTA PDF flow | 3 DTA records reset to `PUSHED`, TEST Frappe site | 3 PDFs attached | `attached_pdfs: 3`, no errors | Pass |
| Full re-run with timeout | 64 records, Frappe emptied | One document per request | 45 PDFs before the 900 s limit; the retry attached the other 19 and created nothing new | Pass |
| Duplicate check on Frappe | Same site | 1 document per request ID | 1 each | Pass |

## 8. Known Limitations & Open Items
- 30 DTA documents delivered before this change have no PDF; the user was asked whether to back-fill them. Answer not confirmed.
- The AWS CLI retried the long synchronous invoke; later runs used asynchronous invokes.
- The cap value of 30 is untuned against real backlog sizes. Not confirmed.

## 9. Status
| Item | Value |
|---|---|
| Branch | `main` |
| Commits | `2e9ba74` (initial commit contains this state) |
| PR | None |
| Deployed to | TEST stack, `ap-south-1` |

## 10. References
Related stories: `nsdl-lambda-core-pipeline`, `nsdl-aws-deploy-lambda-hardening`, `nsdl-boe-pdf-print-layout` (PDF page layout), `nsdl-data-parity-dta-be`.