# NSDL SEZ Automation — Login, OTP, CAPTCHA & BOE/DTA Fixes

**Fix report and setup guide**

This document explains:

1. the problems that were reported,
2. the root cause of each problem,
3. what was changed to fix it,
4. how to set up and run the fixed framework,
5. how the fixes were tested.

Only the login (CAPTCHA + OTP) and the BOE/DTA export handling were changed.
Everything after login (parsing, QC, Frappe document creation, PDF attach) is
unchanged.

---

## 1. Problems reported

| # | Problem |
|---|---|
| 1 | The framework picked the **wrong / older OTP mail**, both when pushing mail from Gmail to the Frappe *Communication* doctype and when reading the OTP from *Communication*. |
| 2 | The log almost always said **"no OTP within 90 sec"**, and then a Resend loop started. |
| 3 | **Resend OTP** was clicked without checking that the right OTP had been used, and the reason for a failure was not captured. |
| 4 | The **CAPTCHA** was often read wrongly, and the whole login was **slow**. |
| 5 | When **no BOE file** existed for the IEC and dates, the **whole flow stopped with an error**. DTA was never tried. |

---

## 2. Root-cause analysis

The analysis was done by reading the full framework and checking it against
Frappe v15's own email code (`frappe/email/doctype/email_account/email_account.py`
and `frappe/email/receive.py`). After that, the problems were **reproduced** by
running the *original* code against a mock NSDL portal and a mock Frappe.

### 2.1 Wrong / older OTP picked (problems 1 and 2)

| Root cause | Explanation |
|---|---|
| **90-second wait was too short** | The mail travels NSDL → manager's mailbox → forward → Gmail → Frappe pull. That routinely takes 60–120 s. In the log of 10-Oct, a *successful* OTP arrived after **82 s**, which is just under the limit. |
| **Resend made it worse** | After 90 s the framework clicked **Resend**, so NSDL issued a *new* OTP and the first one became invalid. The first mail then arrived a few seconds later. It was not in the snapshot, so it was treated as "new" and the **invalid OTP was submitted**. NSDL rejected it, and the cycle repeated until "NSDL rejected 3 OTPs". |
| **"New" was decided by `creation` time** | `creation` is the moment Frappe **imported** the mail, not when the mail was sent. Frappe's Email Account uses `UNSEEN` sync and imports *every unread* mail in Gmail. Every old unread OTP mail was therefore imported with a brand-new `creation` and looked fresh. |
| **`pull_emails` can silently do nothing** | When Frappe's scheduler is already pulling the account, `pull_emails` returns HTTP 200 with *"Emails are already being pulled from this account"* and imports nothing. The scheduler job then imported mails *after* the snapshot was taken, so they looked new. |
| **No date checks at all** | The mail's real date (`communication_date`, from the mail's `Date` header) was never checked. Neither was the original "Sent:" time inside the forwarded mail, or the 10-minute validity. A late forward of an old OTP mail was accepted. |

### 2.2 Resend and missing error reasons (problem 3)

| Root cause | Explanation |
|---|---|
| Resend on a timer | Resend was clicked because the mail was slow, not because an OTP had failed. |
| JavaScript alerts lost | NSDL shows errors such as *Invalid OTP* or *Invalid Captcha* as JS `alert()` popups. **Playwright auto-dismisses alerts** when no handler is registered, so the reason was lost. Only a 1,000-character page dump was logged. |
| Resend result not checked | Nothing verified that the Resend click worked. A "limit reached" message, for example, went unnoticed. |
| OTP page reload misread | After a wrong OTP, NSDL reloads the OTP page. The old code saw "still on OTP page" before the error message had loaded. |

### 2.3 CAPTCHA slow and inaccurate (problem 4)

| Root cause | Explanation |
|---|---|
| Slow OCR call | EasyOCR `readtext()` runs the heavy text-**detection** network on 4 image variants each time: about **1.15–1.4 s per read** on CPU. |
| Model loaded late | The OCR model (several seconds to load) was loaded only when the first CAPTCHA appeared. |
| Weak filtering | Filters worked on greyscale only, so the grey noise curve and the border went into the OCR. The NSDL letters are *dark coloured*; the noise is *mid-grey*. |
| Detector dropped letters | The text detector often missed thin letters such as `i`, which gave 4-letter readings. |
| Bad guesses were submitted | No CAPTCHA refresh was used, so a doubtful reading was still submitted. Each wrong guess cost a full page reload. |
| CAPTCHA error not detected | The "Invalid Captcha" alert was lost (see 2.2). A **wrong password** would also have been retried 8 times, which risks locking the account. |
| Fixed sleeps | `wait_for_timeout(1500)` after every action slowed the whole login. |

### 2.4 Missing BOE file stopped everything (problem 5)

| Root cause | Explanation |
|---|---|
| Download timeout raised an error | When NSDL has no BOE data, no file is produced. `expect_download` waited 30 s and raised an exception. |
| One error stopped both modules | That exception broke out of the export loop, so **DTA was never exported**. |
| Both files were mandatory | Step 1 validation and Step 2 both *required* the BE and DTA files to exist. |

---

## 3. What was fixed

### 3.1 OTP — always the newest, valid OTP

**Files:** `utils/otp_mailbox.py`, `utils/nsdl_login.py`

#### The OTP request time is now recorded

When **Login** or **Resend OTP** is clicked, the exact time is stored. This is
the *OTP request time*. It is measured on the **Frappe server clock**, using
the HTTP `Date` header, so a wrong clock on the PC does not matter.

#### Before every OTP request (Login / Resend)

```text
1. Pull the mailbox and WAIT until Frappe confirms a real, completed pull
   (if Frappe says "already being pulled", wait and pull again)
   → every old unread mail in Gmail is imported NOW, before the OTP is requested
2. Snapshot the existing OTP Communications (baseline)
3. Record the request time
4. Click Login / Resend
```

#### Validations every mail must pass

A mail is used **only if all** of these are true:

| # | Validation | Stops |
|---|---|---|
| 1 | Not in the baseline snapshot | Mails that existed before the request |
| 2 | Communication name, mail Message-ID and OTP value never used before | Re-using an OTP |
| 3 | `communication_date` (the mail's own date) ≥ request time | Old unread mails imported late |
| 4 | Original **"Sent:" / "Date:"** time inside the forwarded mail ≥ request time | A *new forward* of an *old* NSDL mail |
| 5 | OTP is still inside the **10-minute validity** | Expired OTPs |
| 6 | Subject contains `SEZOnline Authentication` | Unrelated mails |
| 7 | Exactly one OTP sentence (`<OTP> is your One Time Password`) | Forwards that quote two OTP mails |
| 8 | User ID in the mail = `NSDL_USERNAME` | OTPs of other portal users |

If several mails pass, the **newest** is used.

The clock tolerance (default 20 s) is automatically limited to **half the gap
between two requests**. So after a Resend, a late mail of the cancelled OTP
can never pass for the new one.

Every rejected mail is written to the log once, with the reason. Example:

```text
[otp] Ignoring mail 'FW: SEZOnline Authentication' (j3b0bmbk27): forwarded copy of an
older NSDL mail (original sent 2026-10-12 15:32:27, OTP request #1 was at 2026-10-12 15:36:27).
```

### 3.2 New OTP and Resend flow

```text
OTP page reached
   │
   ▼
wait for a valid OTP mail — up to 9 minutes (never past the 10-min validity)
   │
   ├─ no valid mail at all → STOP with a clear reason
   │                          (lists every mail seen and why it was rejected)
   │
   ▼
check before submit: 6 digits, still valid, still on OTP page
   │
   ▼
fill OTP → verify the value really landed in the box → Submit
   │
   ├─ accepted → home page ✔
   │
   └─ rejected → capture NSDL's reason (alert OR page label) + screenshot
                 │
                 ├─ lockout message?                       → STOP immediately
                 ├─ 3 rejected OTPs?                       → STOP (NSDL locks at 5)
                 ├─ another valid OTP already in mailbox?  → submit that one (no resend)
                 └─ otherwise → wait the 30 s cooldown → click Resend
                                → check the Resend result (limit / block / page lost)
                                → wait for the new OTP again
```

Key rules:

- **Resend is used only after NSDL rejects an OTP.** It is never used just
  because the mail is slow. (Optional: `resend_on_timeout: true` allows one
  resend when *no mail at all* arrived within the validity.)
- Every portal message, **including JavaScript alerts**, is captured and
  classified as one of: `captcha`, `credentials`, `otp_invalid`,
  `otp_expired`, `resend_limit`, `lockout`, `otp_sent`.
- **Wrong User ID / password stops at once.** It is not retried 8 times, to
  avoid an account lock.
- The whole login is written to **`data/logs/<run_date>/login_audit.jsonl`**:
  request times, which mail was submitted, each portal message and its
  classification.

Example `login_audit.jsonl`:

```json
{"time": "...", "event": "otp_requested", "sequence": 1, "at": "2026-10-12T10:07:30+00:00"}
{"time": "...", "event": "otp_submitted", "communication": "vh07zwkg1l"}
{"time": "...", "event": "otp_rejected", "kind": "otp_invalid", "portal_message": "Invalid OTP. Please enter the correct OTP"}
{"time": "...", "event": "otp_resend", "sequence": 2, "portal_message": "OTP has been resent successfully", "kind": "otp_sent"}
{"time": "...", "event": "otp_accepted", "sequence": 2, "communication": "x1k9..."}
```

### 3.3 CAPTCHA — accurate and fast

**File:** `utils/ocr_engine.py`

```text
CAPTCHA image (151 x 63)
   │
   ▼  1. FILTER
   crop grey border
   "darkness" map from the brightest colour channel
      → dark navy / brown / purple letters become black
      → mid-grey noise curves + speckled background become white
   Otsu threshold (with a floor the grey noise cannot reach)
   remove tiny speckles
   │
   ▼  2. RE-SPACE
   cut out every letter and place it on a clean canvas with even gaps
   (5 gap sizes x grey/binary)  +  4 variants of the original layout
   │
   ▼  3. RECOGNISE
   EasyOCR recogniser only (the slow detector is skipped; it also dropped letters)
   OCR model is loaded in the background while the login page opens
   │
   ▼  4. VOTE
   majority vote across all variants → per-character confidence vote
   upper/lower case of c o s u v w x z decided by letter height
   a letter with a dot = i / j
   │
   ▼  5. VALIDATE
   alphanumeric, expected length (5), enough variants agree?
      no  → click the CAPTCHA refresh and read a new image (no wasted login)
      yes → submit
   │
   ▼
   rejected by NSDL? → image saved in captcha_rejected/ (for tuning) → new attempt
```

**Measured result** (151 NSDL-style CAPTCHAs: the real sample plus generated
images in the same style):

| | Before | After |
|---|---|---|
| Exact reads | 50 % | **70 %** |
| Reads ignoring upper/lower case | 70 % | **86 %** |
| Time per read | 1.15 s | **0.32 s** |

Other speed-ups:

- The fixed 1.5 s sleeps were replaced by smart waits that react as soon as
  NSDL answers.
- The mailbox pull and snapshot happen *before* the CAPTCHA is read, so the
  CAPTCHA is submitted while it is fresh.

### 3.4 Missing BOE or DTA — the flow no longer stops

**Files:** `pipeline/step1_download.py`, `pipeline/step2_transform.py`, `pipeline/base.py`

- BOE and DTA are now exported **independently**. A problem in one never
  stops the other.
- A module counts as **"no data"** when any of these happens:
  - NSDL shows a "No Records Found" popup or page message;
  - no file arrives within 30 s and NSDL shows no error;
  - the file has only the header row.
- After download, rows are counted **per configured client IEC**:

| BOE rows for IEC | DTA rows for IEC | Result |
|---|---|---|
| yes | yes | Both are processed |
| none | yes | Logs `No BOE records for IEC ...` → **DTA is processed** |
| yes | none | Logs `No DTA records for IEC ...` → **BOE is processed** |
| none | none | Logs `There are no BOE and no DTA files/records for IEC ... Nothing to process — stopping here.` → run **stops cleanly** (exit code 0) |

- Step 2 now accepts a missing BOE or DTA file and simply skips that pipeline.
- A *real* export failure (navigation error, unexpected portal error) still
  fails Step 1, with a screenshot, after the other module has been tried.

### 3.5 Files changed

| File | Change |
|---|---|
| `utils/otp_mailbox.py` | Rewritten: request-time anchoring, all mail validations, completed-pull check, server clock, time zones |
| `utils/nsdl_login.py` | Rewritten: alert capture, response detection, OTP / Resend flow, error classification, audit log |
| `utils/ocr_engine.py` | Rewritten: new filters, re-spacing, fast recognition, voting, validation, background model load |
| `pipeline/step1_download.py` | Independent BOE / DTA export, no-data detection, per-IEC validation |
| `pipeline/step2_transform.py` | Accepts a missing BOE or DTA file |
| `pipeline/base.py` | Clean stop when there is nothing to process |
| `config/config.py`, `config/*.yaml` | New settings and their validation |
| `main.py` | `--otp-check` now also shows mail date, original sent time and age |
| `requirements.txt` | Added `python-dateutil`, `tzdata` (needed for time zones on Windows) |
| `README.md` | Updated login / OTP / CAPTCHA / missing-report sections |

**Not changed:** `nsdl_parser.py`, `frappe_client.py`, `step3_pdf_attach.py`,
`state/store.py`, `nsdl_browser.py`, `json_utils.py`, `logger.py`.

---

## 4. Setup guide

### 4.1 Install

```powershell
cd NSDL_SEZ_AUTOMATION
python -m venv venv
.\venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m playwright install chromium
```

> `tzdata` (in `requirements.txt`) is required on Windows, because Python on
> Windows has no time-zone database of its own.

### 4.2 OTP mailbox (`nsdlotp@gmail.com`)

1. Turn on **2-Step Verification** for the Gmail account and create an **App Password**.
2. Make sure **IMAP is enabled** in Gmail settings.
3. **Nobody should open this mailbox** in Gmail web, the phone app or Outlook.
   Frappe imports only *unread* mail (`UNSEEN`), so a mail read there is never imported.

### 4.3 Forwarding rule (manager's mailbox)

- **Condition:** from `donotreply-ndmlsupport@ndml.in` **or** subject contains `SEZOnline Authentication`
- **Action:** forward to `nsdlotp@gmail.com`
- Use a **server-side** rule (Outlook "Rules" on Exchange / Microsoft 365), so
  mails are forwarded immediately, even when Outlook is closed.
- The NSDL user's email ID must be **verified** on the NSDL portal.

### 4.4 Frappe Email Account

Open `/app/email-account/new`:

| Field | Value |
|---|---|
| Email Address | `nsdlotp@gmail.com` |
| Email Account Name | e.g. `NSDL SEZ OTP` → this is `OTP_EMAIL_ACCOUNT` |
| Password | Gmail **App Password** |
| Enable Incoming | ✔ |
| Use IMAP | ✔ |
| Incoming Server | `imap.gmail.com`, Use SSL ✔ |
| IMAP Folder table | one row: `INBOX` |
| Append To | empty |
| Email Sync Option | `UNSEEN` |

Save, click **Pull Emails** once, and check `/app/communication`.

**API user:** needs **read** permission on **Communication** and **Email Account**
(System Manager has both). Generate the API key and secret from the User record.

### 4.5 `config/config.yaml`

Per environment:

```yaml
environments:
  TEST:
    NSDL_USERNAME: "canofy01"
    NSDL_PASSWORD: "********"
    FRAPPE_URL: "http://127.0.0.1:8000"
    FRAPPE_API_KEY: "xxxxxxxx"
    FRAPPE_API_SECRET: "xxxxxxxx"
    OTP_EMAIL_ACCOUNT: "NSDL SEZ OTP"   # Email Account NAME, not the email address
```

> The old log showed `Could not read Email Account 'nsdlotp@gmail.com': not found`.
> `OTP_EMAIL_ACCOUNT` must be the **Email Account Name** (e.g. `NSDL SEZ OTP`),
> not the email address.

New and changed settings under `common` (the defaults are safe):

```yaml
  max_login_attempts: 8
  nsdl_response_timeout_seconds: 30   # max wait for NSDL to answer Login / OTP submit
  nsdl_export_timeout_seconds: 30     # max wait for an export file before "no data"
  treat_export_timeout_as_no_data: true

  nsdl_otp:
    otp_validity_seconds: 600         # NSDL OTP validity (10 min)
    wait_seconds: 540                 # was 90 — wait for the mail up to 9 min
    submit_margin_seconds: 30         # don't submit an OTP in its last 30 s
    resend_on_timeout: false          # resend ONLY after NSDL rejects an OTP
    drain_timeout_seconds: 45         # wait for a running Frappe pull to finish
    clock_skew_tolerance_seconds: 20
    verify_forwarded_sent_time: true  # check the original "Sent:" time in forwards
    frappe_time_zone: ""              # "" = read from Frappe, e.g. "Asia/Kolkata"
    mail_time_zone: ""                # zone of the forwarded "Sent:" line ("" = Frappe's)
    resend_cooldown_seconds: 32
    max_resends: 3
    max_submit_attempts: 3

  captcha:
    expected_length: 5                # letters in the NSDL CAPTCHA (null = 4-8)
    min_agreement: 0.34
    max_reads_per_attempt: 4
```

Unsafe values (for example `wait_seconds` above the validity, a cooldown under
30 s, or 5+ submit attempts) are **rejected when the config loads**.

### 4.6 Check the OTP connection (no NSDL login)

Forward one real NSDL OTP mail to `nsdlotp@gmail.com`, then run:

```powershell
python main.py --otp-check --environment TEST
```

Expected output:

```text
Frappe Email Account 'NSDL SEZ OTP' is reachable and receiving mail.
Latest OTP mail : FW: SEZOnline Authentication
Communication   : 0a1b2c3d4e (imported 2026-10-12 15:47:19)
Mail date       : 2026-10-12T10:15:18+00:00 (UTC)
Original sent   : 2026-10-12T10:15:18+00:00 (UTC)
Age             : 121s
Extracted OTP   : 658204
```

### 4.7 Run

| Task | Command |
|---|---|
| Offline test (no NSDL, no Frappe writes) | `python main.py --run-date 2026-09-12 --environment TEST --dry-run` |
| Step 1 only | `python main.py --run-date 2026-09-12 --environment TEST --steps 1` |
| Step 2 only | `python main.py --run-date 2026-09-12 --environment TEST --steps 2` |
| Step 3 only | `python main.py --run-date 2026-09-12 --environment TEST --steps 3` |
| All steps | `python main.py --run-date 2026-09-12 --environment TEST` |
| Custom date range | add `--from-date 01/09/2026 --to-date 05/09/2026` |
| Status | `python main.py --run-date 2026-09-12 --environment TEST --status` |

Exit codes: `0` = success (including a clean "no BOE/DTA data" stop), `1` = a
step failed validation, `2` = config/setup error or gate refused.

### 4.8 Where to look when something fails

| File | Contains |
|---|---|
| `data/logs/<run_date>/run.log` | Full log, including every ignored mail and the reason |
| `data/logs/<run_date>/login_audit.jsonl` | Login timeline: OTP request times, submitted mail, portal messages |
| `data/logs/<run_date>/*.png` | Screenshot at every stop / rejection |
| `data/logs/<run_date>/captcha_rejected/` | CAPTCHA images NSDL rejected (file name = what was read) |

| Message | Meaning / action |
|---|---|
| `No valid OTP mail arrived for request #N ... Mails seen but rejected — ...` | Read the listed reasons. "No new NSDL OTP mail reached Frappe" means: check the forwarding rule, that nobody opens the Gmail inbox, and Email Account → Pull Emails / Error Log. |
| `NSDL rejected N OTPs (last reason: ...)` | NSDL's own reason is shown; compare `login_audit.jsonl` with the real mail (`--otp-check`). |
| `NSDL reports a login block` | Cooling period — wait 30 minutes. |
| `NSDL refused the OTP resend (limit reached)` | NSDL resend limit — wait and re-run. |
| `NSDL rejected the User ID / password` | Fix `NSDL_PASSWORD`. The login is deliberately not retried. |
| `Forwarded 'Sent:' time ... is after the mail date` | Set `nsdl_otp.mail_time_zone` to the manager mailbox's zone. |
| `There are no BOE and no DTA files/records for IEC ...` | Not an error: nothing to process for those dates. |

---

## 5. Testing performed

### 5.1 Test setup

A **mock NSDL portal** and a **mock Frappe** were built to behave like the
real ones:

- **Mock NSDL portal:** login, CAPTCHA images in NSDL style, OTP page,
  Resend, JS alerts, red notices, trade-data export.
- **Mock Frappe:** Email Account, `pull_emails` (including the "already being
  pulled" case), Communication API, and late background imports.

The **real** framework code ran against these mocks with **Playwright + Chromium**.

### 5.2 Login / OTP scenarios — 15 of 15 passed

| # | Scenario | Expected and verified |
|---|---|---|
| 1 | Normal login | 1 OTP submitted, accepted |
| 2 | Old unread OTP mails waiting in Gmail | Old mails ignored, correct OTP used |
| 3 | Old mail imported late by Frappe (after the request) | Rejected by mail date |
| 4 | New forward of an old NSDL mail | Rejected by original "Sent:" time |
| 5 | Frappe "already being pulled" + late import | Waits for a real pull; old mail rejected |
| 6 | Very slow mail (longer than the old 90 s rule) | Waits, **no resend**, accepted |
| 7 | Wrong OTP rejected, another valid OTP already waiting | Uses it **without resend** |
| 8 | Wrong OTP rejected (alert style) | Reason captured → Resend → new OTP accepted |
| 9 | Wrong OTP rejected (page-label style) | Same as 8 |
| 10 | No mail ever arrives | Clean stop with explanation, no resend |
| 11 | Lockout message | Stops immediately |
| 12 | Resend limit message | Captured, stops |
| 13 | Wrong password | Stops after **1** try (no retry) |
| 14 | CAPTCHA rejected twice | Image kept, retried, logged in |
| 15 | `resend_on_timeout: true` | One resend, new OTP accepted |

**Before/after proof:** scenarios 3, 4 and the slow-mail case were also run on
the **original code**. It reproduced exactly the reported problem: wrong OTPs
submitted, Resend loop, and finally *"NSDL rejected 3 OTPs"*.

### 5.3 BOE / DTA scenarios (Step 1 live against the mock + Step 2) — 6 of 6 passed

| Scenario | Result |
|---|---|
| BOE "No Records" popup, DTA file | DTA processed (55 records) |
| BOE file, DTA no answer (timeout) | BOE processed (32 records) |
| Both "no records" | Clean stop with the "no BOE and no DTA" message |
| Both files | Both processed (87 records) |
| BOE file with header only, DTA file | DTA processed |
| Files present but no rows for the IEC | Clean stop with the IEC message |

### 5.4 Unit checks — 21 of 21 passed

These cover:

- OTP extraction, including forwarding headers and an ambiguous double OTP;
- forwarded-time parsing (Outlook and Gmail formats, explicit time zones);
- rejection of stale and expired OTPs;
- the minute-precision boundary;
- config validation of unsafe values.

### 5.5 Regression

- **Dry run:** a full `--dry-run` of the original and the fixed code gave
  **identical** results: same 87 request records, same Frappe POST payloads,
  and Step 3 at 87/87.
- **Untouched files:** the post-login files are byte-for-byte unchanged.
- **Lint:** clean (`pyflakes`).
- **Clean zip:** the packaged zip was extracted fresh and ran successfully.

---

## 6. Known limits / things to confirm on the real portal

- **Real-world testing:** the fixes were tested against mocks, not the live
  NSDL portal and Gmail, because a live login sends a real OTP to the manager.
  The first live run should be watched, and `login_audit.jsonl` checked.
- **CAPTCHA upper/lower case:** if NSDL's CAPTCHA is **not case-sensitive**,
  the effective accuracy is the 86 % figure.
- **CAPTCHA refresh icon:** if the NSDL page has none, a doubtful reading is
  submitted anyway. A rejection then reloads the page, which is slower but
  still works. A refresh icon can be set with `captcha.refresh_selector`.
- **Empty BOE/DTA export:** the exact way NSDL reports this is handled in all
  common forms (popup, page message, no file within 30 s, header-only file).
- **Frappe without `pull_emails`** (older than v15): mails arrive only on the
  Frappe scheduler interval, so the OTP may take several minutes.