Skip to content
GetProfitable
Search

Scheduling, logging and alerts

Lesson 26 · about 12 min

A daily strategy needs to run once a day, at the right moment, and tell you what it did. Each of those three requirements has a boring, reliable solution and several exciting, fragile ones. This lesson chooses the boring ones: the operating system's scheduler, Python's standard logging module writing structured lines to a file, and a single alert function that reaches you when something needs a human.

When to run

For a daily-bar strategy on US stocks with next-open execution, the bar closes at 16:00 New York time. The data vendor needs a few minutes to publish it. Running at 16:30 New York gives the bar time to settle and leaves the order sitting at the broker overnight for the open. For crypto on 00:00 UTC bars, run at 00:05 UTC. For forex, decide which vendor's daily boundary you are using (Module 2) and run shortly after it.

Two rules. Run after the bar you act on is complete; acting on a partial bar is a different strategy from the one you tested. And run in the market's timezone, not your laptop's, because your laptop's timezone changes when you travel and when daylight saving flips.

The scheduler: cron, not a Python loop

The tempting approach is a Python program that runs forever and sleeps until the next run. Do not. A process that must stay alive for months will eventually die (an update, a reboot, a memory leak, an unhandled exception at 3 a.m.), and then nothing runs and nothing tells you. The operating system's scheduler does not die.

On macOS and Linux, crontab -e and a line like:

# minute hour dom month dow   command
30 16 * * 1-5   cd /home/you/algo-course && TZ=America/New_York .venv/bin/python run_live.py >> logs/cron.out 2>&1

TZ=America/New_York makes cron interpret 16:30 in New York time on systems that support it; on others, set the system timezone or compute the UTC hour yourself and adjust twice a year. On Windows, Task Scheduler does the same job. On a cloud VM, the same cron line works; the VM has the advantage of not being asleep in a bag when the market closes.

The program itself runs once and exits. That is run_once from the previous lesson wrapped in a main function that sets up logging, checks the kill switch (lesson 3), runs, and reports.

Logging: structured lines to a file

print goes to a terminal you are not looking at. The logging module writes to files, rotates them, timestamps every line, and can be pointed at a different destination without changing the code that logs.

# src/logsetup.py
import logging
from logging.handlers import RotatingFileHandler
from pathlib import Path


def setup_logging(log_dir: str = "logs", name: str = "live") -> logging.Logger:
    Path(log_dir).mkdir(exist_ok=True)
    logger = logging.getLogger(name)
    logger.setLevel(logging.INFO)
    if logger.handlers:                                   # idempotent: safe to call twice
        return logger
    fmt = logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s", "%Y-%m-%dT%H:%M:%S%z")
    file_handler = RotatingFileHandler(f"{log_dir}/{name}.log", maxBytes=5_000_000, backupCount=10)
    file_handler.setFormatter(fmt)
    stream_handler = logging.StreamHandler()
    stream_handler.setFormatter(fmt)
    logger.addHandler(file_handler)
    logger.addHandler(stream_handler)
    return logger

What to log, every run, in this order:

log = setup_logging()
log.info("run start symbol=%s run_date=%s", "SYNTH", "2024-06-03")
log.info("bars fetched n=%d last_bar=%s last_close=%.2f", 60, "2024-06-03", 101.23)
log.info("state cash=%.2f held=%.0f", 100_000.0, 0)
log.info("decision signal=%s delta_qty=%.0f", "long", 950)
log.info("order submitted client_order_id=%s status=%s broker_id=%s", "2024-06-03-SYNTH-crossover", "accepted", "abc123")
log.info("run end ok=%s elapsed_s=%.1f", True, 2.3)

Every line has key=value pairs, which makes the log greppable: grep "order submitted" logs/live.log lists every order ever sent; grep "ok=False" lists every failed run. The last close is logged so that a month later you can check the signal by hand. Log the inputs to each decision, not just the decision; the decision you can recompute, the inputs you cannot.

Exceptions get the traceback:

from src.broker import PaperBroker, run_once
from src.data import synthetic_ohlcv

broker = PaperBroker({"SYNTH": synthetic_ohlcv(300, seed=42)})
try:
    ack = run_once(broker, "SYNTH", "2024-06-03")
except Exception:
    log.exception("run failed")          # logs the full traceback at ERROR level
    raise

log.exception inside an except block records the stack trace. The raise afterwards makes the process exit non-zero so cron's output shows a failure and the alert in the next section fires.

Key idea: Run once per bar from the operating system's scheduler, after the bar is complete, in the market's timezone. Log inputs, decisions and outcomes as key=value lines to a rotating file. The log is the only witness to what the program did.

Alerts: one function, two levels

You need to hear about two kinds of event: a run that failed or did something surprising (immediately), and a daily summary that the run happened at all (so that silence means failure, not success). Both go through one function so that switching from email to a chat webhook is a one-line change.

# src/alerts.py
import json
import logging
import os
import urllib.request

log = logging.getLogger("live")


def send_alert(level: str, text: str, sender=None) -> bool:
    """Deliver an alert. `sender` is injectable for tests; default posts JSON to ALERT_WEBHOOK_URL."""
    payload = {"level": level, "text": text}
    log.info("alert level=%s text=%s", level, text)
    try:
        if sender is not None:
            return bool(sender(payload))
        url = os.environ.get("ALERT_WEBHOOK_URL")
        if not url:
            log.warning("ALERT_WEBHOOK_URL not set; alert not delivered")
            return False
        req = urllib.request.Request(url, data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"})
        with urllib.request.urlopen(req, timeout=10) as resp:
            return 200 <= resp.status < 300
    except Exception:
        log.exception("alert delivery failed")
        return False

In tests, pass a sender that appends to a list. In production, ALERT_WEBHOOK_URL points at a Slack, Discord or similar incoming webhook, or at a small service that sends a text message. Alert delivery failing must never crash the run, which is why the function catches everything and returns False.

sent = []
send_alert("info", "run 2024-06-03 ok: no trade", sender=lambda p: sent.append(p) or True)
send_alert("error", "run 2024-06-03 FAILED: connection timeout", sender=lambda p: sent.append(p) or True)
print(sent)

The daily "ok" message is the important one. If you receive an "ok" every trading day at 16:31 and one day you do not, you know within minutes that something is wrong, without any monitoring infrastructure. Lesson 3 adds a heartbeat file for the same reason.

The main function

# run_live.py
import sys
import time
from src.alerts import send_alert
from src.broker import PaperBroker, run_once
from src.data import synthetic_ohlcv
from src.logsetup import setup_logging


def build_broker():
    """Return the broker for this run. Swap in AlpacaLikeBroker behind a config.LIVE flag when ready."""
    return PaperBroker({"SYNTH": synthetic_ohlcv(300, seed=42)})


def main(symbol: str, run_date: str) -> int:
    log = setup_logging()
    started = time.time()
    try:
        broker = build_broker()
        ack = run_once(broker, symbol, run_date)
        send_alert("info", f"run {run_date} ok: {ack.status if ack else 'no trade'}")
        log.info("run end ok=True elapsed_s=%.1f", time.time() - started)
        return 0
    except Exception as exc:
        log.exception("run failed")
        send_alert("error", f"run {run_date} FAILED: {exc}")
        return 1


if __name__ == "__main__":
    sys.exit(main(symbol="SYNTH", run_date=time.strftime("%Y-%m-%d")))

Exit code 0 on success, 1 on failure. Cron records it; your alert channel hears about it; the log has the details.

Try it: Wire main to the paper broker and run it from a cron entry two minutes in the future. Confirm the log file appears with the six expected lines and the alert sender was called. Then make get_bars raise an exception and run again: confirm the traceback is in the log, the error alert fires, and the exit code is 1.

Recap

  • Run after the bar completes, in the market's timezone, from cron or Task Scheduler, never from a long-lived Python loop.
  • Use logging with a rotating file handler; write key=value lines for inputs, decisions and outcomes.
  • log.exception inside except captures the traceback; re-raise or return non-zero so failures are visible.
  • One send_alert function with an injectable sender; failures to deliver never crash the run.
  • Send a daily "ok" so that silence means something is wrong.