Skip to content

Improve throttle notice emails and admin summary #592

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bpepple merged 2 commits into from
Aug 13, 2026
Merged

Improve throttle notice emails and admin summary #592

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ec55e49
Improve throttle notice wording for single-day triggers
bpepple Aug 13, 2026
3bebda3
Show prior notice count in notify_throttled_clients summary
bpepple Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions api/management/commands/notify_throttled_clients.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,15 +125,15 @@ def handle(self, *args, **options) -> None:

prior_notices = ThrottleNotice.objects.filter(user=user).count()
will_enforce = enforce and prior_notices >= options["enforce_after"]
candidates.append((user, offender, will_enforce))
candidates.append((user, offender, will_enforce, prior_notices))

self._print_summary(candidates, send)

if candidates:
send_pushover(self._pushover_message(candidates, send))

if send:
for user, offender, will_enforce in candidates:
for user, offender, will_enforce, _prior_notices in candidates:
if will_enforce:
self._enforce_and_notify(user, offender)
else:
Expand All @@ -144,16 +144,21 @@ def _print_summary(self, candidates, send: bool) -> None:
self.stdout.write("No candidates found.")
return

for user, offender, will_enforce in candidates:
for user, offender, will_enforce, prior_notices in candidates:
if will_enforce:
action = self._predict_enforcement_action(user)
verb = f"Enforcing ({action})" if send else f"Would enforce ({action})"
else:
verb = "Emailing" if send else "Would email"
prior_notices_msg = (
f"{prior_notices} prior notice{'' if prior_notices == 1 else 's'}"
if prior_notices
else "no prior notices"
)
self.stdout.write(
f"{verb} {user.username} <{user.email}>: throttled on "
f"{offender.days_throttled} day(s), {offender.total_count} total, "
f"worst day {offender.worst_day_count}"
f"worst day {offender.worst_day_count}, {prior_notices_msg}"
)
if not send:
self.stdout.write(
Expand All @@ -169,7 +174,7 @@ def _predict_enforcement_action(user: CustomUser) -> str:
@staticmethod
def _pushover_message(candidates, send: bool) -> str:
verb = "Emailed" if send else "Found"
names = ", ".join(user.username for user, _, _ in candidates)
names = ", ".join(user.username for user, _, _, _ in candidates)
return f"{verb} {len(candidates)} throttled API client(s) on Metron: {names}"

@staticmethod
Expand Down
20 changes: 16 additions & 4 deletions api/templates/api/throttle_enforcement_email.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,22 @@
font-size: 14px;
color: #4a4a4a;
font-family: Arial, Helvetica, sans-serif">
Most recently, your client received
<strong>{{ total_count }} rate-limited (429) response{{ total_count|pluralize }}</strong>
across <strong>{{ days_throttled }} day{{ days_throttled|pluralize }}</strong>,
including <strong>{{ worst_day_count }}</strong> in a single day.
{% if days_throttled == 1 %}
{% if total_count == worst_day_count %}
Most recently, your client received
<strong>{{ worst_day_count }} rate-limited (429) response{{ worst_day_count|pluralize }}</strong>
in a single day.
{% else %}
Most recently, your client received
<strong>{{ total_count }} rate-limited (429) response{{ total_count|pluralize }}</strong>
recently, including <strong>{{ worst_day_count }}</strong> in a single day.
{% endif %}
{% else %}
Most recently, your client received
<strong>{{ total_count }} rate-limited (429) response{{ total_count|pluralize }}</strong>
across <strong>{{ days_throttled }} day{{ days_throttled|pluralize }}</strong>,
including <strong>{{ worst_day_count }}</strong> in a single day.
{% endif %}
</p>
</td>
</tr>
Expand Down
6 changes: 4 additions & 2 deletions api/templates/api/throttle_enforcement_email.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@ We previously warned you that your Metron API client was repeatedly hitting our
rate limits without backing off. Since that behavior has continued, we've had
to take action on this account:

Most recently, your client received {{ total_count }} rate-limited (429) response{{ total_count|pluralize }}
{% if days_throttled == 1 %}{% if total_count == worst_day_count %}Most recently, your client received {{ worst_day_count }} rate-limited (429) response{{ worst_day_count|pluralize }}
in a single day.{% else %}Most recently, your client received {{ total_count }} rate-limited (429) response{{ total_count|pluralize }}
recently, including {{ worst_day_count }} in a single day.{% endif %}{% else %}Most recently, your client received {{ total_count }} rate-limited (429) response{{ total_count|pluralize }}
across {{ days_throttled }} day{{ days_throttled|pluralize }}, including {{ worst_day_count }}
in a single day.
in a single day.{% endif %}

{% if account_disabled %}This account has been disabled. You won't be able to log in or use the API
until an administrator reinstates it.{% else %}All API tokens for this account have been revoked. You'll need to generate a
Expand Down
36 changes: 28 additions & 8 deletions api/templates/api/throttle_notice_email.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,18 @@
font-size: 16px;
color: #4a4a4a;
font-family: Arial, Helvetica, sans-serif">
We've noticed that your Metron API client has repeatedly hit our rate
limits over the past several days. When a client keeps retrying
immediately after a 429 response instead of backing off, it adds
unnecessary load to the site for everyone.
{% if days_throttled == 1 %}
We've noticed that your Metron API client hit our rate limits hard,
with {{ worst_day_count }} rate-limited (429) responses in a single
day. When a client keeps retrying immediately after a 429 response
instead of backing off, it adds unnecessary load to the site for
everyone.
{% else %}
We've noticed that your Metron API client has repeatedly hit our rate
limits over the past several days. When a client keeps retrying
immediately after a 429 response instead of backing off, it adds
unnecessary load to the site for everyone.
{% endif %}
</p>
<!-- Stats Box -->
<table border="0"
Expand All @@ -70,10 +78,22 @@
font-size: 14px;
color: #4a4a4a;
font-family: Arial, Helvetica, sans-serif">
Your client received
<strong>{{ total_count }} rate-limited (429) response{{ total_count|pluralize }}</strong>
across <strong>{{ days_throttled }} day{{ days_throttled|pluralize }}</strong>,
including <strong>{{ worst_day_count }}</strong> in a single day.
{% if days_throttled == 1 %}
{% if total_count == worst_day_count %}
Your client received
<strong>{{ worst_day_count }} rate-limited (429) response{{ worst_day_count|pluralize }}</strong>
in a single day.
{% else %}
Your client received
<strong>{{ total_count }} rate-limited (429) response{{ total_count|pluralize }}</strong>
recently, including <strong>{{ worst_day_count }}</strong> in a single day.
{% endif %}
{% else %}
Your client received
<strong>{{ total_count }} rate-limited (429) response{{ total_count|pluralize }}</strong>
across <strong>{{ days_throttled }} day{{ days_throttled|pluralize }}</strong>,
including <strong>{{ worst_day_count }}</strong> in a single day.
{% endif %}
</p>
</td>
</tr>
Expand Down
13 changes: 9 additions & 4 deletions api/templates/api/throttle_notice_email.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
{% autoescape off %}Hi {{ user.username }},

We've noticed that your Metron API client has repeatedly hit our rate limits over
{% if days_throttled == 1 %}We've noticed that your Metron API client hit our rate limits hard, with
{{ worst_day_count }} rate-limited (429) responses in a single day. When a client keeps
retrying immediately after a 429 response instead of backing off, it adds
unnecessary load to the site for everyone.{% else %}We've noticed that your Metron API client has repeatedly hit our rate limits over
the past several days. When a client keeps retrying immediately after a 429
response instead of backing off, it adds unnecessary load to the site for
everyone.
everyone.{% endif %}

Your client received {{ total_count }} rate-limited (429) response{{ total_count|pluralize }}
{% if days_throttled == 1 %}{% if total_count == worst_day_count %}Your client received {{ worst_day_count }} rate-limited (429) response{{ worst_day_count|pluralize }}
in a single day.{% else %}Your client received {{ total_count }} rate-limited (429) response{{ total_count|pluralize }}
recently, including {{ worst_day_count }} in a single day.{% endif %}{% else %}Your client received {{ total_count }} rate-limited (429) response{{ total_count|pluralize }}
across {{ days_throttled }} day{{ days_throttled|pluralize }}, including {{ worst_day_count }}
in a single day.
in a single day.{% endif %}

Every API response includes X-RateLimit-* headers showing your current limit,
remaining requests, and when the window resets. A 429 response also means you've
Expand Down
Loading