Rate limit hit during full keycloak sync

albertskja commented

2026-03-09 22:23:38 +01:00

(Migrated from gitlab.ia.utwente.nl)

We are hitting some kind of rate limit during the periodic full keycloak sync. It's a matrix limit, we should figure out if we can let our bot ignore rate limits, or build in a pacing timer (sleep) after each request.

Workaround: Users can be manually synced in the keycloak log channel by typing !kc sync user @username:inter-actief.net

Stack trace:

Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/mautrix/client/syncer.py", line 241, in _catch_errors
    await handler(data)
  File "/opt/maubot/maubot/handlers/command.py", line 142, in __call__
    ok, res = await self.__call_subcommand__(evt, call_args, remaining_val)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/maubot/maubot/handlers/command.py", line 163, in __call_subcommand__
    return True, await subcommand(
                 ^^^^^^^^^^^^^^^^^
  File "/opt/maubot/maubot/handlers/command.py", line 142, in __call__
    ok, res = await self.__call_subcommand__(evt, call_args, remaining_val)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/maubot/maubot/handlers/command.py", line 163, in __call_subcommand__
    return True, await subcommand(
                 ^^^^^^^^^^^^^^^^^
  File "/opt/maubot/maubot/handlers/command.py", line 154, in __call__
    return await self.__mb_func__(self.__bound_instance__, evt, **call_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/keycloakbot.py", line 258, in sync_full
    raise e
  File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/keycloakbot.py", line 254, in sync_full
    await self.syncer.sync_full(status_msg_id=event_id, ignored_users=self.ignored_users)
  File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/syncer.py", line 82, in sync_full
    changes, errors = await self.sync_user(matrix_uid=member_uid, status_msg_id=status_msg_id, i=i, total=num_members, changes=changes, errors=errors)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/syncer.py", line 181, in sync_user
    await room.invite_user(
  File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/utils/matrix.py", line 72, in invite_user
    await self.client.invite_user(room_id=self.room_id, user_id=UserID(matrix_uid), reason=reason)
  File "/usr/lib/python3.12/site-packages/mautrix/client/store_updater.py", line 99, in invite_user
    await super().invite_user(room_id, user_id, reason, extra_content=extra_content)
  File "/usr/lib/python3.12/site-packages/mautrix/client/api/rooms.py", line 452, in invite_user
    await self.api.request(Method.POST, Path.v3.rooms[room_id].invite, content=data)
  File "/usr/lib/python3.12/site-packages/mautrix/api.py", line 425, in request
    resp_data, resp = await self._send(
                      ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/mautrix/api.py", line 265, in _send
    raise make_request_error(
mautrix.errors.request.MLimitExceeded: Too Many Requests

We are hitting some kind of rate limit during the periodic full keycloak sync. It's a matrix limit, we should figure out if we can let our bot ignore rate limits, or build in a pacing timer (sleep) after each request. Workaround: Users can be manually synced in the keycloak log channel by typing `!kc sync user @username:inter-actief.net` Stack trace: ``` Traceback (most recent call last): File "/usr/lib/python3.12/site-packages/mautrix/client/syncer.py", line 241, in _catch_errors await handler(data) File "/opt/maubot/maubot/handlers/command.py", line 142, in __call__ ok, res = await self.__call_subcommand__(evt, call_args, remaining_val) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/maubot/maubot/handlers/command.py", line 163, in __call_subcommand__ return True, await subcommand( ^^^^^^^^^^^^^^^^^ File "/opt/maubot/maubot/handlers/command.py", line 142, in __call__ ok, res = await self.__call_subcommand__(evt, call_args, remaining_val) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/maubot/maubot/handlers/command.py", line 163, in __call_subcommand__ return True, await subcommand( ^^^^^^^^^^^^^^^^^ File "/opt/maubot/maubot/handlers/command.py", line 154, in __call__ return await self.__mb_func__(self.__bound_instance__, evt, **call_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/keycloakbot.py", line 258, in sync_full raise e File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/keycloakbot.py", line 254, in sync_full await self.syncer.sync_full(status_msg_id=event_id, ignored_users=self.ignored_users) File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/syncer.py", line 82, in sync_full changes, errors = await self.sync_user(matrix_uid=member_uid, status_msg_id=status_msg_id, i=i, total=num_members, changes=changes, errors=errors) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/syncer.py", line 181, in sync_user await room.invite_user( File "/data/plugins/net.inter-actief.maubot.keycloaksync-v1.1.4.mbp/utils/matrix.py", line 72, in invite_user await self.client.invite_user(room_id=self.room_id, user_id=UserID(matrix_uid), reason=reason) File "/usr/lib/python3.12/site-packages/mautrix/client/store_updater.py", line 99, in invite_user await super().invite_user(room_id, user_id, reason, extra_content=extra_content) File "/usr/lib/python3.12/site-packages/mautrix/client/api/rooms.py", line 452, in invite_user await self.api.request(Method.POST, Path.v3.rooms[room_id].invite, content=data) File "/usr/lib/python3.12/site-packages/mautrix/api.py", line 425, in request resp_data, resp = await self._send( ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mautrix/api.py", line 265, in _send raise make_request_error( mautrix.errors.request.MLimitExceeded: Too Many Requests ```

albertskja commented

2026-03-09 23:24:53 +01:00

(Migrated from gitlab.ia.utwente.nl)

changed the description

albertskja commented

2026-03-09 23:34:59 +01:00

(Migrated from gitlab.ia.utwente.nl)

We can override the rate limit for a user:

https://element-hq.github.io/synapse/latest/admin_api/user_admin_api.html#set-ratelimit

Setting both to 0 will disable rate limits for the user:

{
  "messages_per_second": 0,
  "burst_count": 0
}

We can override the rate limit for a user: https://element-hq.github.io/synapse/latest/admin_api/user_admin_api.html#set-ratelimit Setting both to 0 will disable rate limits for the user: ```json { "messages_per_second": 0, "burst_count": 0 } ```

albertskja commented

2026-03-09 23:44:21 +01:00

(Migrated from gitlab.ia.utwente.nl)

I set the rate limit with these commands. Got the admin token from checking the session data when logged into https://admin.matrix.ia.utwente.nl

To check the current rate limit setting:

curl --header "Authorization: Bearer mat_XXXXXXX" -X GET https://synapse.matrix.ia.utwente.nl/_synapse/admin/v1/users/@claudia:inter-actief.net/override_ratelimit

# Response (for normal user): {}

To disable rate limiting:

curl --header "Authorization: Bearer mat_XXXXXXX" -X POST --json '{"messages_per_second": 0, "burst_count": 0}' https://synapse.matrix.ia.utwente.nl/_synapse/admin/v1/users/@claudia:inter-actief.net/override_ratelimit
# Response: {"messages_per_second":0,"burst_count":0}

# After checking GET again:
# Response: {"messages_per_second":0,"burst_count":0}

I set the rate limit with these commands. Got the admin token from checking the session data when logged into https://admin.matrix.ia.utwente.nl To check the current rate limit setting: ```bash curl --header "Authorization: Bearer mat_XXXXXXX" -X GET https://synapse.matrix.ia.utwente.nl/_synapse/admin/v1/users/@claudia:inter-actief.net/override_ratelimit # Response (for normal user): {} ``` To disable rate limiting: ```bash curl --header "Authorization: Bearer mat_XXXXXXX" -X POST --json '{"messages_per_second": 0, "burst_count": 0}' https://synapse.matrix.ia.utwente.nl/_synapse/admin/v1/users/@claudia:inter-actief.net/override_ratelimit # Response: {"messages_per_second":0,"burst_count":0} # After checking GET again: # Response: {"messages_per_second":0,"burst_count":0} ```

albertskja (Migrated from gitlab.ia.utwente.nl) closed this issue

2026-03-09 23:44:54 +01:00

Rows
Columns

Rate limit hit during full keycloak sync #1