Feature #132 » NSDL_SEZ_LOGIN_FIXES_AND_SETUP.md
NSDL SEZ Automation — Login, OTP, CAPTCHA & BOE/DTA Fixes
Fix report and setup guide
This document explains:
- the problems that were reported,
- the root cause of each problem,
- what was changed to fix it,
- how to set up and run the fixed framework,
- how the fixes were tested.
Only the login (CAPTCHA + OTP) and the BOE/DTA export handling were changed.
Everything after login (parsing, QC, Frappe document creation, PDF attach) is
unchanged.
1. Problems reported
| # | Problem |
|---|---|
| 1 | The framework picked the wrong / older OTP mail, both when pushing mail from Gmail to the Frappe Communication doctype and when reading the OTP from Communication. |
| 2 | The log almost always said "no OTP within 90 sec", and then a Resend loop started. |
| 3 | Resend OTP was clicked without checking that the right OTP had been used, and the reason for a failure was not captured. |
| 4 | The CAPTCHA was often read wrongly, and the whole login was slow. |
| 5 | When no BOE file existed for the IEC and dates, the whole flow stopped with an error. DTA was never tried. |
2. Root-cause analysis
The analysis was done by reading the full framework and checking it against
Frappe v15's own email code (frappe/email/doctype/email_account/email_account.py
and frappe/email/receive.py). After that, the problems were reproduced by
running the original code against a mock NSDL portal and a mock Frappe.
2.1 Wrong / older OTP picked (problems 1 and 2)
| Root cause | Explanation |
|---|---|
| 90-second wait was too short | The mail travels NSDL → manager's mailbox → forward → Gmail → Frappe pull. That routinely takes 60–120 s. In the log of 10-Oct, a successful OTP arrived after 82 s, which is just under the limit. |
| Resend made it worse | After 90 s the framework clicked Resend, so NSDL issued a new OTP and the first one became invalid. The first mail then arrived a few seconds later. It was not in the snapshot, so it was treated as "new" and the invalid OTP was submitted. NSDL rejected it, and the cycle repeated until "NSDL rejected 3 OTPs". |
"New" was decided by creation time |
creation is the moment Frappe imported the mail, not when the mail was sent. Frappe's Email Account uses UNSEEN sync and imports every unread mail in Gmail. Every old unread OTP mail was therefore imported with a brand-new creation and looked fresh. |
pull_emails can silently do nothing |
When Frappe's scheduler is already pulling the account, pull_emails returns HTTP 200 with "Emails are already being pulled from this account" and imports nothing. The scheduler job then imported mails after the snapshot was taken, so they looked new. |
| No date checks at all | The mail's real date (communication_date, from the mail's Date header) was never checked. Neither was the original "Sent:" time inside the forwarded mail, or the 10-minute validity. A late forward of an old OTP mail was accepted. |
2.2 Resend and missing error reasons (problem 3)
| Root cause | Explanation |
|---|---|
| Resend on a timer | Resend was clicked because the mail was slow, not because an OTP had failed. |
| JavaScript alerts lost | NSDL shows errors such as Invalid OTP or Invalid Captcha as JS alert() popups. Playwright auto-dismisses alerts when no handler is registered, so the reason was lost. Only a 1,000-character page dump was logged. |
| Resend result not checked | Nothing verified that the Resend click worked. A "limit reached" message, for example, went unnoticed. |
| OTP page reload misread | After a wrong OTP, NSDL reloads the OTP page. The old code saw "still on OTP page" before the error message had loaded. |
2.3 CAPTCHA slow and inaccurate (problem 4)
| Root cause | Explanation |
|---|---|
| Slow OCR call | EasyOCR readtext() runs the heavy text-detection network on 4 image variants each time: about 1.15–1.4 s per read on CPU. |
| Model loaded late | The OCR model (several seconds to load) was loaded only when the first CAPTCHA appeared. |
| Weak filtering | Filters worked on greyscale only, so the grey noise curve and the border went into the OCR. The NSDL letters are dark coloured; the noise is mid-grey. |
| Detector dropped letters | The text detector often missed thin letters such as i, which gave 4-letter readings. |
| Bad guesses were submitted | No CAPTCHA refresh was used, so a doubtful reading was still submitted. Each wrong guess cost a full page reload. |
| CAPTCHA error not detected | The "Invalid Captcha" alert was lost (see 2.2). A wrong password would also have been retried 8 times, which risks locking the account. |
| Fixed sleeps |
wait_for_timeout(1500) after every action slowed the whole login. |
2.4 Missing BOE file stopped everything (problem 5)
| Root cause | Explanation |
|---|---|
| Download timeout raised an error | When NSDL has no BOE data, no file is produced. expect_download waited 30 s and raised an exception. |
| One error stopped both modules | That exception broke out of the export loop, so DTA was never exported. |
| Both files were mandatory | Step 1 validation and Step 2 both required the BE and DTA files to exist. |
3. What was fixed
3.1 OTP — always the newest, valid OTP
Files: utils/otp_mailbox.py, utils/nsdl_login.py
The OTP request time is now recorded
When Login or Resend OTP is clicked, the exact time is stored. This is
the OTP request time. It is measured on the Frappe server clock, using
the HTTP Date header, so a wrong clock on the PC does not matter.
Before every OTP request (Login / Resend)
1. Pull the mailbox and WAIT until Frappe confirms a real, completed pull
(if Frappe says "already being pulled", wait and pull again)
→ every old unread mail in Gmail is imported NOW, before the OTP is requested
2. Snapshot the existing OTP Communications (baseline)
3. Record the request time
4. Click Login / Resend
Validations every mail must pass
A mail is used only if all of these are true:
| # | Validation | Stops |
|---|---|---|
| 1 | Not in the baseline snapshot | Mails that existed before the request |
| 2 | Communication name, mail Message-ID and OTP value never used before | Re-using an OTP |
| 3 |
communication_date (the mail's own date) ≥ request time |
Old unread mails imported late |
| 4 | Original "Sent:" / "Date:" time inside the forwarded mail ≥ request time | A new forward of an old NSDL mail |
| 5 | OTP is still inside the 10-minute validity | Expired OTPs |
| 6 | Subject contains SEZOnline Authentication
|
Unrelated mails |
| 7 | Exactly one OTP sentence (<OTP> is your One Time Password) |
Forwards that quote two OTP mails |
| 8 | User ID in the mail = NSDL_USERNAME
|
OTPs of other portal users |
If several mails pass, the newest is used.
The clock tolerance (default 20 s) is automatically limited to half the gap
between two requests. So after a Resend, a late mail of the cancelled OTP
can never pass for the new one.
Every rejected mail is written to the log once, with the reason. Example:
[otp] Ignoring mail 'FW: SEZOnline Authentication' (j3b0bmbk27): forwarded copy of an
older NSDL mail (original sent 2026-10-12 15:32:27, OTP request #1 was at 2026-10-12 15:36:27).
3.2 New OTP and Resend flow
OTP page reached
│
▼
wait for a valid OTP mail — up to 9 minutes (never past the 10-min validity)
│
├─ no valid mail at all → STOP with a clear reason
│ (lists every mail seen and why it was rejected)
│
▼
check before submit: 6 digits, still valid, still on OTP page
│
▼
fill OTP → verify the value really landed in the box → Submit
│
├─ accepted → home page ✔
│
└─ rejected → capture NSDL's reason (alert OR page label) + screenshot
│
├─ lockout message? → STOP immediately
├─ 3 rejected OTPs? → STOP (NSDL locks at 5)
├─ another valid OTP already in mailbox? → submit that one (no resend)
└─ otherwise → wait the 30 s cooldown → click Resend
→ check the Resend result (limit / block / page lost)
→ wait for the new OTP again
Key rules:
-
Resend is used only after NSDL rejects an OTP. It is never used just
because the mail is slow. (Optional:resend_on_timeout: trueallows one
resend when no mail at all arrived within the validity.) - Every portal message, including JavaScript alerts, is captured and
classified as one of:captcha,credentials,otp_invalid,
otp_expired,resend_limit,lockout,otp_sent. -
Wrong User ID / password stops at once. It is not retried 8 times, to
avoid an account lock. - The whole login is written to
data/logs/<run_date>/login_audit.jsonl:
request times, which mail was submitted, each portal message and its
classification.
Example login_audit.jsonl:
{"time": "...", "event": "otp_requested", "sequence": 1, "at": "2026-10-12T10:07:30+00:00"}
{"time": "...", "event": "otp_submitted", "communication": "vh07zwkg1l"}
{"time": "...", "event": "otp_rejected", "kind": "otp_invalid", "portal_message": "Invalid OTP. Please enter the correct OTP"}
{"time": "...", "event": "otp_resend", "sequence": 2, "portal_message": "OTP has been resent successfully", "kind": "otp_sent"}
{"time": "...", "event": "otp_accepted", "sequence": 2, "communication": "x1k9..."}
3.3 CAPTCHA — accurate and fast
File: utils/ocr_engine.py
CAPTCHA image (151 x 63)
│
▼ 1. FILTER
crop grey border
"darkness" map from the brightest colour channel
→ dark navy / brown / purple letters become black
→ mid-grey noise curves + speckled background become white
Otsu threshold (with a floor the grey noise cannot reach)
remove tiny speckles
│
▼ 2. RE-SPACE
cut out every letter and place it on a clean canvas with even gaps
(5 gap sizes x grey/binary) + 4 variants of the original layout
│
▼ 3. RECOGNISE
EasyOCR recogniser only (the slow detector is skipped; it also dropped letters)
OCR model is loaded in the background while the login page opens
│
▼ 4. VOTE
majority vote across all variants → per-character confidence vote
upper/lower case of c o s u v w x z decided by letter height
a letter with a dot = i / j
│
▼ 5. VALIDATE
alphanumeric, expected length (5), enough variants agree?
no → click the CAPTCHA refresh and read a new image (no wasted login)
yes → submit
│
▼
rejected by NSDL? → image saved in captcha_rejected/ (for tuning) → new attempt
Measured result (151 NSDL-style CAPTCHAs: the real sample plus generated
images in the same style):
| Before | After | |
|---|---|---|
| Exact reads | 50 % | 70 % |
| Reads ignoring upper/lower case | 70 % | 86 % |
| Time per read | 1.15 s | 0.32 s |
Other speed-ups:
- The fixed 1.5 s sleeps were replaced by smart waits that react as soon as
NSDL answers. - The mailbox pull and snapshot happen before the CAPTCHA is read, so the
CAPTCHA is submitted while it is fresh.
3.4 Missing BOE or DTA — the flow no longer stops
Files: pipeline/step1_download.py, pipeline/step2_transform.py, pipeline/base.py
- BOE and DTA are now exported independently. A problem in one never
stops the other. - A module counts as "no data" when any of these happens:
- NSDL shows a "No Records Found" popup or page message;
- no file arrives within 30 s and NSDL shows no error;
- the file has only the header row.
- After download, rows are counted per configured client IEC:
| BOE rows for IEC | DTA rows for IEC | Result |
|---|---|---|
| yes | yes | Both are processed |
| none | yes | Logs No BOE records for IEC ... → DTA is processed
|
| yes | none | Logs No DTA records for IEC ... → BOE is processed
|
| none | none | Logs There are no BOE and no DTA files/records for IEC ... Nothing to process — stopping here. → run stops cleanly (exit code 0) |
- Step 2 now accepts a missing BOE or DTA file and simply skips that pipeline.
- A real export failure (navigation error, unexpected portal error) still
fails Step 1, with a screenshot, after the other module has been tried.
3.5 Files changed
| File | Change |
|---|---|
utils/otp_mailbox.py |
Rewritten: request-time anchoring, all mail validations, completed-pull check, server clock, time zones |
utils/nsdl_login.py |
Rewritten: alert capture, response detection, OTP / Resend flow, error classification, audit log |
utils/ocr_engine.py |
Rewritten: new filters, re-spacing, fast recognition, voting, validation, background model load |
pipeline/step1_download.py |
Independent BOE / DTA export, no-data detection, per-IEC validation |
pipeline/step2_transform.py |
Accepts a missing BOE or DTA file |
pipeline/base.py |
Clean stop when there is nothing to process |
config/config.py, config/*.yaml
|
New settings and their validation |
main.py |
--otp-check now also shows mail date, original sent time and age |
requirements.txt |
Added python-dateutil, tzdata (needed for time zones on Windows) |
README.md |
Updated login / OTP / CAPTCHA / missing-report sections |
Not changed: nsdl_parser.py, frappe_client.py, step3_pdf_attach.py,
state/store.py, nsdl_browser.py, json_utils.py, logger.py.
4. Setup guide
4.1 Install
cd NSDL_SEZ_AUTOMATION
python -m venv venv
.\venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m playwright install chromium
tzdata(inrequirements.txt) is required on Windows, because Python on
Windows has no time-zone database of its own.
4.2 OTP mailbox (nsdlotp@gmail.com)
- Turn on 2-Step Verification for the Gmail account and create an App Password.
- Make sure IMAP is enabled in Gmail settings.
-
Nobody should open this mailbox in Gmail web, the phone app or Outlook.
Frappe imports only unread mail (UNSEEN), so a mail read there is never imported.
4.3 Forwarding rule (manager's mailbox)
-
Condition: from
donotreply-ndmlsupport@ndml.inor subject containsSEZOnline Authentication -
Action: forward to
nsdlotp@gmail.com - Use a server-side rule (Outlook "Rules" on Exchange / Microsoft 365), so
mails are forwarded immediately, even when Outlook is closed. - The NSDL user's email ID must be verified on the NSDL portal.
4.4 Frappe Email Account
Open /app/email-account/new:
| Field | Value |
|---|---|
| Email Address | nsdlotp@gmail.com |
| Email Account Name | e.g. NSDL SEZ OTP → this is OTP_EMAIL_ACCOUNT
|
| Password | Gmail App Password |
| Enable Incoming | ✔ |
| Use IMAP | ✔ |
| Incoming Server |
imap.gmail.com, Use SSL ✔ |
| IMAP Folder table | one row: INBOX
|
| Append To | empty |
| Email Sync Option | UNSEEN |
Save, click Pull Emails once, and check /app/communication.
API user: needs read permission on Communication and Email Account
(System Manager has both). Generate the API key and secret from the User record.
4.5 config/config.yaml
Per environment:
environments:
TEST:
NSDL_USERNAME: "canofy01"
NSDL_PASSWORD: "********"
FRAPPE_URL: "http://127.0.0.1:8000"
FRAPPE_API_KEY: "xxxxxxxx"
FRAPPE_API_SECRET: "xxxxxxxx"
OTP_EMAIL_ACCOUNT: "NSDL SEZ OTP" # Email Account NAME, not the email address
The old log showed
Could not read Email Account 'nsdlotp@gmail.com': not found.
OTP_EMAIL_ACCOUNTmust be the Email Account Name (e.g.NSDL SEZ OTP),
not the email address.
New and changed settings under common (the defaults are safe):
max_login_attempts: 8
nsdl_response_timeout_seconds: 30 # max wait for NSDL to answer Login / OTP submit
nsdl_export_timeout_seconds: 30 # max wait for an export file before "no data"
treat_export_timeout_as_no_data: true
nsdl_otp:
otp_validity_seconds: 600 # NSDL OTP validity (10 min)
wait_seconds: 540 # was 90 — wait for the mail up to 9 min
submit_margin_seconds: 30 # don't submit an OTP in its last 30 s
resend_on_timeout: false # resend ONLY after NSDL rejects an OTP
drain_timeout_seconds: 45 # wait for a running Frappe pull to finish
clock_skew_tolerance_seconds: 20
verify_forwarded_sent_time: true # check the original "Sent:" time in forwards
frappe_time_zone: "" # "" = read from Frappe, e.g. "Asia/Kolkata"
mail_time_zone: "" # zone of the forwarded "Sent:" line ("" = Frappe's)
resend_cooldown_seconds: 32
max_resends: 3
max_submit_attempts: 3
captcha:
expected_length: 5 # letters in the NSDL CAPTCHA (null = 4-8)
min_agreement: 0.34
max_reads_per_attempt: 4
Unsafe values (for example wait_seconds above the validity, a cooldown under
30 s, or 5+ submit attempts) are rejected when the config loads.
4.6 Check the OTP connection (no NSDL login)
Forward one real NSDL OTP mail to nsdlotp@gmail.com, then run:
python main.py --otp-check --environment TEST
Expected output:
Frappe Email Account 'NSDL SEZ OTP' is reachable and receiving mail.
Latest OTP mail : FW: SEZOnline Authentication
Communication : 0a1b2c3d4e (imported 2026-10-12 15:47:19)
Mail date : 2026-10-12T10:15:18+00:00 (UTC)
Original sent : 2026-10-12T10:15:18+00:00 (UTC)
Age : 121s
Extracted OTP : 658204
4.7 Run
| Task | Command |
|---|---|
| Offline test (no NSDL, no Frappe writes) | python main.py --run-date 2026-09-12 --environment TEST --dry-run |
| Step 1 only | python main.py --run-date 2026-09-12 --environment TEST --steps 1 |
| Step 2 only | python main.py --run-date 2026-09-12 --environment TEST --steps 2 |
| Step 3 only | python main.py --run-date 2026-09-12 --environment TEST --steps 3 |
| All steps | python main.py --run-date 2026-09-12 --environment TEST |
| Custom date range | add --from-date 01/09/2026 --to-date 05/09/2026
|
| Status | python main.py --run-date 2026-09-12 --environment TEST --status |
Exit codes: 0 = success (including a clean "no BOE/DTA data" stop), 1 = a
step failed validation, 2 = config/setup error or gate refused.
4.8 Where to look when something fails
| File | Contains |
|---|---|
data/logs/<run_date>/run.log |
Full log, including every ignored mail and the reason |
data/logs/<run_date>/login_audit.jsonl |
Login timeline: OTP request times, submitted mail, portal messages |
data/logs/<run_date>/*.png |
Screenshot at every stop / rejection |
data/logs/<run_date>/captcha_rejected/ |
CAPTCHA images NSDL rejected (file name = what was read) |
| Message | Meaning / action |
|---|---|
No valid OTP mail arrived for request #N ... Mails seen but rejected — ... |
Read the listed reasons. "No new NSDL OTP mail reached Frappe" means: check the forwarding rule, that nobody opens the Gmail inbox, and Email Account → Pull Emails / Error Log. |
NSDL rejected N OTPs (last reason: ...) |
NSDL's own reason is shown; compare login_audit.jsonl with the real mail (--otp-check). |
NSDL reports a login block |
Cooling period — wait 30 minutes. |
NSDL refused the OTP resend (limit reached) |
NSDL resend limit — wait and re-run. |
NSDL rejected the User ID / password |
Fix NSDL_PASSWORD. The login is deliberately not retried. |
Forwarded 'Sent:' time ... is after the mail date |
Set nsdl_otp.mail_time_zone to the manager mailbox's zone. |
There are no BOE and no DTA files/records for IEC ... |
Not an error: nothing to process for those dates. |
5. Testing performed
5.1 Test setup
A mock NSDL portal and a mock Frappe were built to behave like the
real ones:
-
Mock NSDL portal: login, CAPTCHA images in NSDL style, OTP page,
Resend, JS alerts, red notices, trade-data export. -
Mock Frappe: Email Account,
pull_emails(including the "already being
pulled" case), Communication API, and late background imports.
The real framework code ran against these mocks with Playwright + Chromium.
5.2 Login / OTP scenarios — 15 of 15 passed
| # | Scenario | Expected and verified |
|---|---|---|
| 1 | Normal login | 1 OTP submitted, accepted |
| 2 | Old unread OTP mails waiting in Gmail | Old mails ignored, correct OTP used |
| 3 | Old mail imported late by Frappe (after the request) | Rejected by mail date |
| 4 | New forward of an old NSDL mail | Rejected by original "Sent:" time |
| 5 | Frappe "already being pulled" + late import | Waits for a real pull; old mail rejected |
| 6 | Very slow mail (longer than the old 90 s rule) | Waits, no resend, accepted |
| 7 | Wrong OTP rejected, another valid OTP already waiting | Uses it without resend |
| 8 | Wrong OTP rejected (alert style) | Reason captured → Resend → new OTP accepted |
| 9 | Wrong OTP rejected (page-label style) | Same as 8 |
| 10 | No mail ever arrives | Clean stop with explanation, no resend |
| 11 | Lockout message | Stops immediately |
| 12 | Resend limit message | Captured, stops |
| 13 | Wrong password | Stops after 1 try (no retry) |
| 14 | CAPTCHA rejected twice | Image kept, retried, logged in |
| 15 | resend_on_timeout: true |
One resend, new OTP accepted |
Before/after proof: scenarios 3, 4 and the slow-mail case were also run on
the original code. It reproduced exactly the reported problem: wrong OTPs
submitted, Resend loop, and finally "NSDL rejected 3 OTPs".
5.3 BOE / DTA scenarios (Step 1 live against the mock + Step 2) — 6 of 6 passed
| Scenario | Result |
|---|---|
| BOE "No Records" popup, DTA file | DTA processed (55 records) |
| BOE file, DTA no answer (timeout) | BOE processed (32 records) |
| Both "no records" | Clean stop with the "no BOE and no DTA" message |
| Both files | Both processed (87 records) |
| BOE file with header only, DTA file | DTA processed |
| Files present but no rows for the IEC | Clean stop with the IEC message |
5.4 Unit checks — 21 of 21 passed
These cover:
- OTP extraction, including forwarding headers and an ambiguous double OTP;
- forwarded-time parsing (Outlook and Gmail formats, explicit time zones);
- rejection of stale and expired OTPs;
- the minute-precision boundary;
- config validation of unsafe values.
5.5 Regression
-
Dry run: a full
--dry-runof the original and the fixed code gave
identical results: same 87 request records, same Frappe POST payloads,
and Step 3 at 87/87. - Untouched files: the post-login files are byte-for-byte unchanged.
-
Lint: clean (
pyflakes). - Clean zip: the packaged zip was extracted fresh and ran successfully.
6. Known limits / things to confirm on the real portal
-
Real-world testing: the fixes were tested against mocks, not the live
NSDL portal and Gmail, because a live login sends a real OTP to the manager.
The first live run should be watched, andlogin_audit.jsonlchecked. -
CAPTCHA upper/lower case: if NSDL's CAPTCHA is not case-sensitive,
the effective accuracy is the 86 % figure. -
CAPTCHA refresh icon: if the NSDL page has none, a doubtful reading is
submitted anyway. A rejection then reloads the page, which is slower but
still works. A refresh icon can be set withcaptcha.refresh_selector. -
Empty BOE/DTA export: the exact way NSDL reports this is handled in all
common forms (popup, page message, no file within 30 s, header-only file). -
Frappe without
pull_emails(older than v15): mails arrive only on the
Frappe scheduler interval, so the OTP may take several minutes.