OCR configuration

Example customer: Belimo (BEL)

1. [1. How OCR authentication works](#1-how-ocr-authentication-works)
2. [2. Prerequisites in AWS](#2-prerequisites-in-aws)
3. [3. OCR server: find the live files](#3-ocr-server-find-the-live-files)
4. [4. Update property.json](#4-update-propertyjson)
5. [5. Update the Frappe site config](#5-update-the-frappe-site-config)
6. [6. Test end to end](#6-test-end-to-end)
7. [7. Change, rotate or remove a customer](#7-change-rotate-or-remove-a-customer)
8. [8. Troubleshooting](#8-troubleshooting)
9. [9. Security rules](#9-security-rules)
10. [Appendix](#appendix)
11. [A. ocr_add_customer.py](#a-ocr_add_customerpy)
12. [B. Screenshot register](#b-screenshot-register)

# OCR configuration for a Canofy customer

How to connect a customer's Canofy (Frappe) site to the shared OCR service: the **OCR server's `property.json`**, the **Frappe `site_config.json`**, and the network and S3 access around them. Belimo (BEL) is used as the worked example throughout.

## 1. How OCR authentication works

1. The customer's Frappe site sends `POST /generate_ocr` to the OCR server with a **username**, a **password** and the **S3 path** of the PDF.
2. The OCR server looks up the **caller's private IP** in `property.json`. It decrypts that entry's username and password with the `ENCRYPTION_KEY` from `.env` (Fernet) and compares them with what was sent.
3. If they match, it reads the PDF from the entry's **bucket** and returns the extracted data.

Each step is tagged with where it runs:

OCR server, as ubuntu Server via SSM, as frappe Server via SSM, as ssm-user AWS console AWS CloudShell Canofy in the browser

| Where | What must be set | Belimo value |
| --- | --- | --- |
| OCR server `property.json` | `"<ERP private IP>": {"BUCKET NAME", "username" (encrypted), "password" (encrypted)}` | `"172.31.10.27"`, bucket `bel.canofy.tech` |
| OCR server `.env` | `ENCRYPTION_KEY` (shared by every customer, **never change it**) | unchanged |
| Frappe `site_config.json` | `ocr_username`, `ocr_password` (both **base64** of the plain values), `ocr_url` | `ocr_url = http://172.31.35.206:5000/generate_ocr` |
| AWS IAM `OCR-production-S3` | customer bucket in **both** statements | `bel.canofy.tech`, `bel.canofy.tech/*` |
| AWS security group of the OCR server | TCP 5000 from the ERP private IP | `172.31.10.27/32` |

> **Warning:** the username and password must be the **same plain values** on both sides. They're encrypted (Fernet) in `property.json`, and base64-encoded in `site_config.json`.

### Values for this customer

| Variable | Belimo |
| --- | --- |
| Site | `bel.canofy.tech` |
| ERP private IP | `172.31.10.27` (EC2 → instance → Private IPv4) |
| Customer bucket | `bel.canofy.tech` |
| S3 folder (S3 File Settings) | `BEL` |
| OCR server | `172.31.35.206`, port `5000`, service `ocr_app.service` |
| OCR app folder | `/home/ubuntu/canofy_OCR` (owner `ubuntu`) |

## 2. Prerequisites in AWS

### 2.1 OCR access to the customer bucket

AWS console IAM → Policies → **OCR-production-S3** → Edit → JSON. Add these lines to **both** the Allow statement and the Deny/NotResource statement, right after a line that already ends with a comma:

```text
"arn:aws:s3:::bel.canofy.tech",
"arn:aws:s3:::bel.canofy.tech/*",
```

Then **Next → Save changes**. Confirm in CloudShell that the bucket appears 4 times:

AWS CloudShell

```bash
ARN=$(aws iam list-policies --scope Local --query "Policies[?PolicyName=='OCR-production-S3'].Arn" --output text); VER=$(aws iam get-policy --policy-arn "$ARN" --query Policy.DefaultVersionId --output text); aws iam get-policy-version --policy-arn "$ARN" --version-id "$VER" --query PolicyVersion.Document --output json | grep -c "bel.canofy.tech"
```

Figure 1 OCR-production-S3 saved with the BEL lines in the Allow statement (20–21) and the Deny/NotResource statement (40–41).

### 2.2 Network: port 5000

AWS console EC2 → OCR server → Security → security group → **Edit inbound rules** → Add rule: Custom TCP, port **5000**, source **`172.31.10.27/32`**, description `BEL bel.canofy.tech`.

Server via SSM, as ssm-user Test from the customer server. Any HTTP code (400/401/500) means it's reachable; `000` means it's blocked.

```bash
curl -s -m 5 -o /dev/null -w "%{http_code}\n" -X POST http://172.31.35.206:5000/generate_ocr -H "Content-Type: application/json" -d "{}"
```

## 3. OCR server: find the live files

OCR server, as ubuntu Connect to the OCR server through SSM, then confirm the service and its working directory:

```bash
systemctl status ocr_app.service --no-pager | head -5
```

```bash
systemctl cat ocr_app.service | grep -E "WorkingDirectory|ExecStart"
```

Figure 2 WorkingDirectory is /home/ubuntu/canofy_OCR, and gunicorn listens on 172.31.35.206:5000.

Several copies of `property.json` exist. Only the one in the WorkingDirectory is live:

```bash
sudo find / \( -iname "property.json" -o -iname "properties.json" \) -not -path "/proc/*" -not -path "*/node_modules/*" 2>/dev/null
```

Figure 3 Copies found: the live file /home/ubuntu/canofy_OCR/property.json, plus backups and a stray /home/property.json. Don't edit those.

To confirm which path the code opens:

```bash
sudo grep -rn "property.json" /home/ubuntu/canofy_OCR/*.py
```

A relative `"property.json"` means the file in the WorkingDirectory.

Switch to the file owner. As `ssm-user`, you can't read `/home/ubuntu`:

```bash
sudo -iu ubuntu
```

Figure 4 Permission denied as ssm-user; every following OCR-server step runs as ubuntu.

Back up before any change:

```bash
cp ~/canofy_OCR/property.json ~/canofy_OCR/property.json.bak.$(date +%Y%m%d-%H%M) && ls -la ~/canofy_OCR/property.json*
```

```bash
grep -c "^ENCRYPTION_KEY=" ~/canofy_OCR/.env
```

The last command must print `1`.

## 4. Update property.json

Choose **one** of the two methods below. Both use the **existing** key, generate a random per-customer username and password, and print the three Frappe commands needed in section 5.

### 4.1 Method A: helper script (recommended)

AWS console Upload `ocr_add_customer.py` (appendix A) to `s3://bel.canofy.tech/_setup/`. The OCR role can read that bucket.

OCR server, as ubuntu As `ubuntu`:

```bash
aws s3 cp s3://bel.canofy.tech/_setup/ocr_add_customer.py ~/ocr_add_customer.py --region ap-south-1
```

Dry run (it lists the existing IP → bucket entries and checks they all decrypt):

```bash
/usr/bin/python3 ~/ocr_add_customer.py --app-dir /home/ubuntu/canofy_OCR --code BEL --ip 172.31.10.27 --bucket bel.canofy.tech --site bel.canofy.tech --ocr-url http://172.31.35.206:5000/generate_ocr --dry-run
```

Real run (it backs up, writes and validates the file, then prints the Frappe commands):

```bash
/usr/bin/python3 ~/ocr_add_customer.py --app-dir /home/ubuntu/canofy_OCR --code BEL --ip 172.31.10.27 --bucket bel.canofy.tech --site bel.canofy.tech --ocr-url http://172.31.35.206:5000/generate_ocr
```

It refuses to overwrite an IP that already exists. To **rotate** an existing customer's credentials, add `--replace`.

### 4.2 Method B: manual

**Step 1: generate the values.** Keep this terminal open, and don't share its output:

```bash
cd ~/canofy_OCR && /usr/bin/python3 -c "import secrets,base64;from cryptography.fernet import Fernet;k=[l.split('=',1)[1].strip().strip('\"') for l in open('.env') if l.startswith('ENCRYPTION_KEY=')][0];f=Fernet(k.encode());u='bel_ocr_'+secrets.token_hex(3);p=secrets.token_urlsafe(24);print('ENC_USERNAME =',f.encrypt(u.encode()).decode());print('ENC_PASSWORD =',f.encrypt(p.encode()).decode());print(' bench --site bel.canofy.tech set-config ocr_username',base64.b64encode(u.encode()).decode());print(' bench --site bel.canofy.tech set-config ocr_password',base64.b64encode(p.encode()).decode());print(' bench --site bel.canofy.tech set-config ocr_url http://172.31.35.206:5000/generate_ocr')"
```

For a new customer, change `bel_ocr_` and `bel.canofy.tech` in this command.

**Step 2: check the format of the existing entries.** Look at whether values are plain `"gAAAA…"` or `"b'gAAAA…'"`, and use the same style:

```bash
head -c 400 ~/canofy_OCR/property.json; echo
```

**Step 3: edit the file.**

```bash
nano ~/canofy_OCR/property.json
```

Add a comma after the last entry's closing `}`, then this block. Paste with right-click, save with Ctrl+O and Enter, exit with Ctrl+X.

```json
"172.31.10.27": {
    "BUCKET NAME": "bel.canofy.tech",
    "username": "PASTE_ENC_USERNAME",
    "password": "PASTE_ENC_PASSWORD"
}
```

If the SSM terminal merges pasted lines into one, that's fine: JSON ignores line breaks. Only the commas, quotes and braces matter.

### 4.3 Validate (both methods)

This prints only IPs, buckets and True/False:

```bash
cd ~/canofy_OCR && /usr/bin/python3 -c "import json;from cryptography.fernet import Fernet;k=[l.split('=',1)[1].strip().strip('\"') for l in open('.env') if l.startswith('ENCRYPTION_KEY=')][0];f=Fernet(k.encode());d=json.load(open('property.json'));[print(ip,'->',v.get('BUCKET NAME'),'| decrypts:',all(bool(f.decrypt(str(v[x]).strip('b').strip(\"'\").encode())) for x in ('username','password'))) for ip,v in d.items()]"
```

Expected: every customer listed, including `172.31.10.27 -> bel.canofy.tech | decrypts: True`.

Show the plain username of the new entry, to compare with Frappe in section 5.3:

```bash
cd ~/canofy_OCR && /usr/bin/python3 -c "import json;from cryptography.fernet import Fernet;k=[l.split('=',1)[1].strip().strip('\"') for l in open('.env') if l.startswith('ENCRYPTION_KEY=')][0];f=Fernet(k.encode());v=json.load(open('property.json'))['172.31.10.27'];print(f.decrypt(v['username'].strip('b').strip(\"'\").encode()).decode())"
```

**If anything fails, restore the latest backup:**

```bash
cp "$(ls -t ~/canofy_OCR/property.json.bak.* | head -1)" ~/canofy_OCR/property.json
```

### 4.4 Restart the OCR service

The service is shared, so restart it at a quiet moment:

```bash
sudo systemctl restart ocr_app.service && sudo systemctl status ocr_app.service --no-pager | head -5
```

Expected: `active (running)`.

## 5. Update the Frappe site config

Server via SSM, as frappe Connect to the customer server through SSM, then:

```bash
sudo -iu frappe
```

```bash
cd ~/frappe-bench
```

### 5.1 Set the three keys

Paste the three lines printed in section 4. They begin with a **space**, so they stay out of shell history:

```bash
 bench --site bel.canofy.tech set-config ocr_username 'BASE64_USERNAME'
```

```bash
 bench --site bel.canofy.tech set-config ocr_password 'BASE64_PASSWORD'
```

```bash
bench --site bel.canofy.tech set-config ocr_url 'http://172.31.35.206:5000/generate_ocr'
```

If you only have the plain values, encode them yourself. Use `printf '%s'`, not `echo`, because `echo` adds a newline that breaks the match:

```bash
 printf '%s' 'PLAIN_OCR_USERNAME' | base64
```

```bash
 printf '%s' 'PLAIN_OCR_PASSWORD' | base64
```

### 5.2 Check (values masked)

```bash
python3 -c "import json;d=json.load(open('sites/bel.canofy.tech/site_config.json'));[print(k,'=',('*****' if k!='ocr_url' else d[k])) for k in ('ocr_url','ocr_username','ocr_password') if k in d]"
```

Expected:

```text
ocr_url = http://172.31.35.206:5000/generate_ocr
ocr_username = *****
ocr_password = *****
```

Or use Frappe's own view:

```bash
bench --site bel.canofy.tech show-config | grep -E "ocr_url|host_name"
```

### 5.3 Check that the username matches property.json

```bash
python3 -c "import json,base64;d=json.load(open('sites/bel.canofy.tech/site_config.json'));print(base64.b64decode(d['ocr_username']).decode())"
```

This must print **exactly** the same username as the check in section 4.3.

### 5.4 Apply

```bash
bench restart
```

### 5.5 Other config commands

| Task | Command |
| --- | --- |
| View the whole file (contains secrets, so don't share it) | `cat sites/bel.canofy.tech/site_config.json` |
| Read one value as the app sees it | `bench --site bel.canofy.tech execute frappe.conf.get --args "['ocr_url']"` |
| Change the OCR URL | `bench --site bel.canofy.tech set-config ocr_url 'http://<ocr-ip>:5000/generate_ocr'` |
| Remove a key | `bench --site bel.canofy.tech remove-config ocr_url` |
| Check the JSON is valid | `python3 -m json.tool sites/bel.canofy.tech/site_config.json > /dev/null && echo VALID` |

## 6. Test end to end

### 6.1 Find a real PDF in the customer bucket

Server via SSM, as frappe

```bash
aws s3 ls s3://bel.canofy.tech/BEL/ --recursive --region ap-south-1 | grep -i "\.pdf" | tail -5
```

Note the key (the path after the date, size and space).

### 6.2 Call OCR directly with the site's own config

This sends the same credentials the site uses, so it separates configuration problems from application problems. Replace `BEL/your/file.pdf` with the key from 6.1:

```bash
cd ~/frappe-bench && ./env/bin/python -c "import json,base64,requests;c=json.load(open('sites/bel.canofy.tech/site_config.json'));r=requests.post(c['ocr_url'],json={'username':base64.b64decode(c['ocr_username']).decode(),'password':base64.b64decode(c['ocr_password']).decode(),'filename':'BEL/your/file.pdf'},timeout=300);print(r.status_code);print(r.text[:600])"
```

The payload format (`username`, `password`, `filename`) follows the OCR App Setup Guide. If the Canofy app sends it differently, compare with `canofy_tcs/import_integrations/doctype/boe_bridge/boe_bridge.py`.

### 6.3 Test from the application, and read the OCR log

Canofy in the browser Run OCR on a document the way users do, for example from a Bill of Entry or Icegate Intimation. Then:

OCR server, as ubuntu

```bash
sudo journalctl -u ocr_app.service --since "15 minutes ago" --no-pager | tail -80
```

## 7. Change, rotate or remove a customer

| Task | OCR server (as `ubuntu`) | Frappe server (as `frappe`) |
| --- | --- | --- |
| Rotate credentials | Method A with `--replace`, or replace the two values in the entry (method B) | Set `ocr_username` and `ocr_password` again; `bench restart` |
| Customer server got a new private IP | Rename the entry's key to the new IP (`nano`), validate, restart | Nothing |
| OCR server moved | Nothing | `set-config ocr_url` to the new address; `bench restart` |
| Remove a customer | Delete the entry with `nano`, validate, restart | `remove-config` for the three keys |

Every OCR-server change follows the same pattern: **back up → edit → validate (4.3) → restart (4.4)**.

## 8. Troubleshooting

| Symptom | Likely cause | Check / fix |
| --- | --- | --- |
| `curl` prints `000` | Security group or VPC | Add the TCP 5000 rule (2.2); both servers must be in the same VPC |
| `HTTP 500` from OCR (Belimo, open) | See the rows below; the traceback is in the journal | `journalctl -u ocr_app.service` (6.3) |
| `KeyError: '172.31.x.x'` in the journal | Entry missing, wrong IP, or service not restarted | `systemctl show ocr_app.service -p ActiveEnterTimestamp` vs `stat property.json`; restart |
| `InvalidToken` | Entry encrypted with a different key, or a damaged paste | Regenerate the entry (4.1 or 4.2) |
| Authentication or "invalid credentials" message | Plain username/password differ between the two files | Compare 4.3 with 5.3; re-encode with `printf '%s'` |
| `NoSuchKey` / 404 from S3 | Wrong `filename` path or folder case | Compare with `aws s3 ls`; MIL uses lower-case `mil`, BEL uses `BEL` |
| `AccessDenied` from S3 | OCR policy missing the bucket | Section 2.1; the count must be 4 |
| SSL / certificate error for the bucket | Dotted bucket name in the OCR client | Path-style addressing in the OCR client (SLC's `slc.canofy.tech` works today) |
| `Permission denied` on `/home/ubuntu` | Working as `ssm-user` | `sudo -iu ubuntu` |
| `JSONDecodeError` after editing | Missing comma or brace | Restore the backup (4.3) and redo the edit |

## 9. Security rules

- **Never change `ENCRYPTION_KEY`.** It protects every customer's entry.
- The OCR App Setup Guide currently contains the real key in plain text. Replace it with a placeholder in that document, and plan a key rotation (re-encrypt all entries in one change).
- Never paste the plain or encrypted credentials into chat, tickets or email; keep them in the password manager.
- Keep the `property.json.bak.*` files on the server; don't copy them elsewhere.
- Delete `ocr_add_customer.py` from `_setup/` in the customer bucket after use, and keep the master copy in the DevOps repository.

## Appendix

## A. ocr_add_customer.py

Helper for section 4.1. Run on the OCR server as ubuntu with /usr/bin/python3. It reads the key from .env and never prints it.

```python
#!/usr/bin/env python3
"""
Canofy OCR - add a customer (ERP server) to property.json

Run ON THE OCR SERVER, with the same Python the OCR service uses:

  /usr/bin/python3 ocr_add_customer.py --code BEL --ip 172.31.10.27 --bucket bel.canofy.tech --site bel.canofy.tech --dry-run
  /usr/bin/python3 ocr_add_customer.py --code BEL --ip 172.31.10.27 --bucket bel.canofy.tech --site bel.canofy.tech

What it does
  * reads ENCRYPTION_KEY from ~/canofy_OCR/.env (never typed, never printed)
  * generates a random OCR username + password for this customer
  * Fernet-encrypts them exactly like encrypt.py, and checks they decrypt
  * refuses to overwrite an existing IP entry (use --replace to rotate)
  * backs up property.json, writes the new entry, validates the JSON
  * prints the 3 'bench set-config' commands to run on the ERP server
    (ocr_username / ocr_password base64-encoded, ocr_url)
"""
import argparse
import base64
import json
import os
import secrets
import shutil
import socket
import sys
from datetime import datetime

from cryptography.fernet import Fernet


def read_env_key(env_path):
    with open(env_path) as f:
        for line in f:
            line = line.strip()
            if line.startswith("ENCRYPTION_KEY="):
                return line.split("=", 1)[1].strip().strip('"').strip("'")
    sys.exit(f"ERROR: ENCRYPTION_KEY not found in {env_path}")


def own_private_ip():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        s.connect(("10.255.255.255", 1))
        return s.getsockname()[0]
    finally:
        s.close()


def main():
    home = os.path.expanduser("~")
    p = argparse.ArgumentParser()
    p.add_argument("--code", required=True, help="customer code, e.g. BEL")
    p.add_argument("--ip", required=True, help="ERP server PRIVATE IP, e.g. 172.31.10.27")
    p.add_argument("--bucket", required=True, help="customer S3 bucket, e.g. bel.canofy.tech")
    p.add_argument("--site", required=True, help="Frappe site name, e.g. bel.canofy.tech")
    p.add_argument("--app-dir", default=os.path.join(home, "canofy_OCR"))
    p.add_argument("--ocr-url", help="default: http://<this server private IP>:5000/generate_ocr")
    p.add_argument("--replace", action="store_true", help="rotate credentials of an existing IP entry")
    p.add_argument("--dry-run", action="store_true")
    a = p.parse_args()

    prop_path = os.path.join(a.app_dir, "property.json")
    env_path = os.path.join(a.app_dir, ".env")
    for path in (prop_path, env_path):
        if not os.path.isfile(path):
            sys.exit(f"ERROR: not found: {path}  (use --app-dir if the OCR app lives elsewhere)")

    fernet = Fernet(read_env_key(env_path).encode())
    with open(prop_path) as f:
        props = json.load(f)
    if not isinstance(props, dict):
        sys.exit("ERROR: property.json is not a JSON object")

    # Report existing entries (IP -> bucket) and sanity-check their format
    print(f"property.json has {len(props)} entries:")
    for ip, entry in props.items():
        print(f"  {ip:16} -> {entry.get('BUCKET NAME')}")
        u = str(entry.get("username", ""))
        if u.startswith("b'") or u.startswith('b"'):
            print("    NOTE: this entry stores values as b'...' text; new entry uses plain token text")
        try:
            fernet.decrypt(u.strip("b'\"").encode())
        except Exception:
            print("    WARNING: this entry does not decrypt with the current key")

    if a.ip in props and not a.replace:
        sys.exit(f"ERROR: {a.ip} already exists (bucket {props[a.ip].get('BUCKET NAME')}). Use --replace to rotate.")
    for ip, entry in props.items():
        if entry.get("BUCKET NAME") == a.bucket and ip != a.ip:
            print(f"WARNING: bucket {a.bucket} is already used by {ip}")

    username = f"{a.code.lower()}_ocr_{secrets.token_hex(3)}"
    password = secrets.token_urlsafe(24)
    enc_user = fernet.encrypt(username.encode()).decode()
    enc_pass = fernet.encrypt(password.encode()).decode()
    assert fernet.decrypt(enc_user.encode()).decode() == username
    assert fernet.decrypt(enc_pass.encode()).decode() == password

    new_entry = {"BUCKET NAME": a.bucket, "username": enc_user, "password": enc_pass}
    ocr_url = a.ocr_url or f"http://{own_private_ip()}:5000/generate_ocr"

    print(f"\nNew entry for {a.ip}: BUCKET NAME={a.bucket}, username/password = Fernet tokens (verified)")
    if a.dry_run:
        print("\nDRY RUN - property.json not changed.")
        return

    backup = f"{prop_path}.bak.{datetime.now():%Y%m%d-%H%M%S}"
    shutil.copy2(prop_path, backup)
    props[a.ip] = new_entry
    tmp = prop_path + ".tmp"
    with open(tmp, "w") as f:
        json.dump(props, f, indent=4)
    with open(tmp) as f:
        json.load(f)  # validate before replacing
    os.replace(tmp, prop_path)
    print(f"Backup : {backup}")
    print(f"Written: {prop_path}  ({len(props)} entries)")

    b64 = lambda s: base64.b64encode(s.encode()).decode()
    print("\n=========== RUN THESE ON THE ERP SERVER (as frappe, in ~/frappe-bench) ===========")
    print(f"bench --site {a.site} set-config ocr_username {b64(username)}")
    print(f"bench --site {a.site} set-config ocr_password {b64(password)}")
    print(f"bench --site {a.site} set-config ocr_url {ocr_url}")
    print("===================================================================================")
    print("These are credentials: copy them straight into the ERP session. Do not paste them")
    print("into chat, tickets or email. Then restart the OCR service:")
    print("  sudo systemctl restart ocr_app.service && sudo systemctl status ocr_app.service --no-pager")


if __name__ == "__main__":
    main()

```

## B. Screenshot register

Figures from the Belimo setup, with sensitive identifiers masked, and screenshots still to capture for the SOP.

| Figure | Content | Status |
| --- | --- | --- |
| 1 | OCR-production-S3 saved with the BEL lines in the Allow statement (20–21) and the Deny/NotResource statement (40–41). | embedded |
| 2 | WorkingDirectory is /home/ubuntu/canofy_OCR, and gunicorn listens on 172.31.35.206:5000. | embedded |
| 3 | Copies found: the live file /home/ubuntu/canofy_OCR/property.json, plus backups and a stray /home/property.json. Don't edit those. | embedded |
| 4 | Permission denied as ssm-user; every following OCR-server step runs as ubuntu. | embedded |