sqlite3.OperationalError: no such table: event_txn_id_device_id when upgrading from Docker v1.89.0 to v1.92.2

[Anthial]

Description

Messages cannot be sent. When attempting to send a message the server logs a database error.

Steps to reproduce

Note, I do not currently have the capabilities to do this fresh, but assume a normal upgrade path from somewhere around v1.70.

1. Run Synapse Docker v1.89.0 with sqlite3
2. Upgrade to v1.92.2 via docker compose pull
3. Messages can no longer be sent

Homeserver

the.eldritch.gift

Synapse Version

v1.92.2

Installation Method

Docker (matrixdotorg/synapse)

Database

SQLite (Docker Default)

Workers

Single process

Platform

Distro: Pop OS 22.04
Hardware:

• CPU: Intel(R) Celeron(R) N4505 @ 2.00GHz
• 16GB DDR4 RAM
• 500GB Storage

Configuration

No response

Relevant log output

docker-synapse-1  | 2023-09-17 12:52:14,536 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 0/5
docker-synapse-1  | 2023-09-17 12:52:14,537 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 1/5
docker-synapse-1  | 2023-09-17 12:52:14,537 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 2/5
docker-synapse-1  | 2023-09-17 12:52:14,538 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 3/5
docker-synapse-1  | 2023-09-17 12:52:14,538 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 4/5
docker-synapse-1  | 2023-09-17 12:52:14,539 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 5/5
docker-synapse-1  | 2023-09-17 12:52:14,539 - synapse.http.server - 140 - ERROR - PUT-287 - Failed handle request via 'RoomSendEventRestServlet': <XForwardedForRequest at 0x7f35bb9654d0 method='PUT' uri='/_matrix/client/v3/rooms/!bMPYjpwMVArbXrMnrM%3Athe.eldritch.gift/send/m.room.encrypted/m1694955131659.2' clientproto='HTTP/1.1' site='8008'>
docker-synapse-1  | Traceback (most recent call last):
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
docker-synapse-1  |     result = context.run(
docker-synapse-1  |              ^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
docker-synapse-1  |     return g.throw(self.type, self.value, self.tb)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/rest/client/room.py", line 366, in _do
docker-synapse-1  |     ) = await self.event_creation_handler.create_and_send_nonmember_event(
docker-synapse-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 1012, in create_and_send_nonmember_event
docker-synapse-1  |     event = await self.get_event_from_transaction(
docker-synapse-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 931, in get_event_from_transaction
docker-synapse-1  |     existing_event_id = await self.get_event_id_from_transaction(
docker-synapse-1  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 901, in get_event_id_from_transaction
docker-synapse-1  |     await self.store.get_event_id_from_transaction_id_and_device_id(
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/events_worker.py", line 2031, in get_event_id_from_transaction_id_and_device_id
docker-synapse-1  |     return await self.db_pool.simple_select_one_onecol(
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1709, in simple_select_one_onecol
docker-synapse-1  |     return await self.runInteraction(
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 945, in runInteraction
docker-synapse-1  |     return await delay_cancellation(_runInteraction())
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
docker-synapse-1  |     result = context.run(
docker-synapse-1  |              ^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
docker-synapse-1  |     return g.throw(self.type, self.value, self.tb)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 911, in _runInteraction
docker-synapse-1  |     result = await self.runWithConnection(
docker-synapse-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1040, in runWithConnection
docker-synapse-1  |     return await make_deferred_yieldable(
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
docker-synapse-1  |     result = inContext.theWork()  # type: ignore[attr-defined]
docker-synapse-1  |              ^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
docker-synapse-1  |     inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
docker-synapse-1  |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
docker-synapse-1  |     return self.currentContext().callWithContext(ctx, func, *args, **kw)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
docker-synapse-1  |     return func(*args, **kw)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
docker-synapse-1  |     result = func(conn, *args, **kw)
docker-synapse-1  |              ^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1033, in inner_func
docker-synapse-1  |     return func(db_conn, *args, **kwargs)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 773, in new_transaction
docker-synapse-1  |     r = func(cursor, *args, **kwargs)
docker-synapse-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1752, in simple_select_one_onecol_txn
docker-synapse-1  |     ret = cls.simple_select_onecol_txn(
docker-synapse-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1775, in simple_select_onecol_txn
docker-synapse-1  |     txn.execute(sql, list(keyvalues.values()))
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 432, in execute
docker-synapse-1  |     self._do_execute(self.txn.execute, sql, parameters)
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 484, in _do_execute
docker-synapse-1  |     return func(sql, *args, **kwargs)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  | sqlite3.OperationalError: no such table: event_txn_id_device_id

Anything else that would be useful to know?

I do believe this was also the case for Synapse v1.90.0

[michaelschefczyk]

With postgresql, I did end up in a similar situation. I had to revert to 1.89.0, as I do not have docker images for the versions in between available. Before failing, the database was updated, so I was required to restore the previous version from a backup/snapshot.

[clokep]

Table was added in #15318, back in Synapse 1.83.0.

[michaelschefczyk]

Thank you very much @clokep.

Is there a practical way to add missing tables to the database? Would my homeserver otherwise be doomed and lost so that I would have to restart from zero under a different name - or use signal only instead of two alternatives?

[clokep]

Is there a practical way to add missing tables to the database? Would my homeserver otherwise be doomed and lost so that I would have to restart from zero under a different name - or use signal only instead of two alternatives?

I'm not sure how it missed the update to add it, but I haven't actually done any investigation.

Can you run SELECT * FROM applied_schema_deltas; and paste the results here?

[michaelschefczyk]

Sorry, we are tailing a different thread here. The similiarity is moving from v1.89.0 to v1.92.2 at least, but I am not absolutely certain about the rest. Should I file this as a separate topic??

The output is:

version | file
---------+----------------------------------------------------------------------------
55 | 55/access_token_expiry.sql
55 | 55/track_threepid_validations.sql
55 | 55/users_alter_deactivated.sql
56 | 56/add_spans_to_device_lists.sql
56 | 56/current_state_events_membership.sql
56 | 56/current_state_events_membership_mk2.sql
56 | 56/delete_keys_from_deleted_backups.sql
56 | 56/destinations_failure_ts.sql
56 | 56/destinations_retry_interval_type.sql.postgres
56 | 56/device_stream_id_insert.sql
56 | 56/devices_last_seen.sql
56 | 56/drop_unused_event_tables.sql
56 | 56/event_expiry.sql
56 | 56/event_labels.sql
56 | 56/event_labels_background_update.sql
56 | 56/fix_room_keys_index.sql
56 | 56/hidden_devices.sql
56 | 56/nuke_empty_communities_from_db.sql
56 | 56/public_room_list_idx.sql
56 | 56/redaction_censor.sql
56 | 56/redaction_censor2.sql
56 | 56/redaction_censor3_fix_update.sql.postgres
56 | 56/redaction_censor4.sql
56 | 56/remove_tombstoned_rooms_from_directory.sql
56 | 56/room_key_etag.sql
56 | 56/room_membership_idx.sql
56 | 56/room_retention.sql
56 | 56/signing_keys.sql
56 | 56/signing_keys_nonunique_signatures.sql
56 | 56/state_group_room_idx.sql
56 | 56/stats_separated.sql
56 | 56/unique_user_filter_index.py
56 | 56/user_external_ids.sql
56 | 56/users_in_public_rooms_idx.sql
57 | 57/delete_old_current_state_events.sql
57 | 57/device_list_remote_cache_stale.sql
57 | 57/local_current_membership.py
57 | 57/remove_sent_outbound_pokes.sql
57 | 57/rooms_version_column.sql
57 | 57/rooms_version_column_2.sql.postgres
57 | 57/rooms_version_column_3.sql.postgres
58 | 58/00background_update_ordering.sql
58 | 58/02remove_dup_outbound_pokes.sql
58 | 58/03persist_ui_auth.sql
58 | 58/05cache_instance.sql.postgres
58 | 58/06dlols_unique_idx.py
58 | 58/07add_method_to_thumbnail_constraint.sql.postgres
58 | 58/07persist_ui_auth_ips.sql
58 | 58/08_media_safe_from_quarantine.sql.postgres
58 | 58/09shadow_ban.sql
58 | 58/10_pushrules_enabled_delete_obsolete.sql
58 | 58/10drop_local_rejections_stream.sql
58 | 58/10federation_pos_instance_name.sql
58 | 58/11user_id_seq.py
58 | 58/12room_stats.sql
58 | 58/13remove_presence_allow_inbound.sql
58 | 58/14events_instance_name.sql
58 | 58/14events_instance_name.sql.postgres
58 | 58/15_catchup_destination_rooms.sql
58 | 58/15unread_count.sql
58 | 58/16populate_stats_process_rooms_fix.sql
58 | 58/17_catchup_last_successful.sql
58 | 58/18stream_positions.sql
58 | 58/11dehydration.sql
58 | 58/11fallback.sql
58 | 58/19instance_map.sql.postgres
58 | 58/19txn_id.sql
58 | 58/20instance_name_event_tables.sql
58 | 58/20user_daily_visits.sql
58 | 58/21as_device_stream.sql
58 | 58/21drop_device_max_stream_id.sql
58 | 58/22puppet_token.sql
58 | 58/22users_have_local_media.sql
58 | 58/23e2e_cross_signing_keys_idx.sql
58 | 58/24drop_event_json_index.sql
58 | 58/25user_external_ids_user_id_idx.sql
58 | 58/26access_token_last_validated.sql
58 | 58/27local_invites.sql
58 | 58/28drop_last_used_column.sql.postgres
58 | 58/28rejected_events_metadata.sql
59 | 59/01ignored_user.py
59 | 59/02shard_send_to_device.sql
59 | 59/03shard_send_to_device_sequence.sql.postgres
59 | 59/04_event_auth_chains.sql
59 | 59/04_event_auth_chains.sql.postgres
59 | 59/04drop_account_data.sql
59 | 59/05cache_invalidation.sql
59 | 59/06chain_cover_index.sql
59 | 59/06shard_account_data.sql
59 | 59/06shard_account_data.sql.postgres
59 | 59/07shard_account_data_fix.sql
59 | 59/08delete_pushers_for_deactivated_accounts.sql
59 | 59/08delete_stale_pushers.sql
59 | 59/09rejected_events_metadata.sql
59 | 59/10delete_purged_chain_cover.sql
59 | 59/11drop_thumbnail_constraint.sql.postgres
59 | 59/12account_validity_token_used_ts_ms.sql
59 | 59/12presence_stream_instance.sql
59 | 59/12presence_stream_instance_seq.sql.postgres
59 | 59/13users_to_send_full_presence_to.sql
59 | 59/11add_knock_members_to_stats.sql
59 | 59/15locks.sql
59 | 59/16federation_inbound_staging.sql
59 | 59/14refresh_tokens.sql
60 | 60/01recreate_stream_ordering.sql.postgres
60 | 60/02change_stream_ordering_columns.sql.postgres
61 | 61/01change_appservices_txns.sql.postgres
61 | 61/01insertion_event_lookups.sql
61 | 61/02drop_redundant_room_depth_index.sql
61 | 61/02state_groups_state_n_distinct.sql.postgres
61 | 61/03recreate_min_depth.py
62 | 62/01insertion_event_extremities.sql
63 | 63/01create_registration_tokens.sql
63 | 63/02delete_unlinked_email_pushers.sql
63 | 63/02populate-rooms-creator.sql
63 | 63/03session_store.sql
63 | 63/04add_presence_stream_not_offline_index.sql
64 | 64/01msc2716_chunk_to_batch_rename.sql.postgres
65 | 65/01msc2716_insertion_event_edges.sql
65 | 65/02_thread_relations.sql
65 | 65/03remove_hidden_devices_from_device_inbox.sql
65 | 65/04_local_group_updates.sql
65 | 65/06remove_deleted_devices_from_device_inbox.sql
65 | 65/05_remove_room_stats_historical_and_user_stats_historical.sql
65 | 65/07_arbitrary_relations.sql
65 | 65/08_device_inbox_background_updates.sql
65 | 65/10_expirable_refresh_tokens.sql
65 | 65/11_devices_auth_provider_session.sql
67 | 67/01drop_public_room_list_stream.sql
68 | 68/01event_columns.sql
68 | 68/02_msc2409_add_device_id_appservice_stream_type.sql
68 | 68/03_delete_account_data_for_deactivated_accounts.sql
68 | 68/04_refresh_tokens_index_next_token_id.sql
68 | 68/04partial_state_rooms.sql
68 | 68/05partial_state_rooms_triggers.py
68 | 68/06_msc3202_add_device_list_appservice_stream_type.sql
69 | 69/01as_txn_seq.py
69 | 69/01device_list_oubound_by_room.sql
69 | 69/02cache_invalidation_index.sql
70 | 70/08_state_group_edges_unique.sql
70 | 70/01clean_table_purged_rooms.sql
71 | 71/01rebuild_event_edges.sql.postgres
71 | 71/01remove_noop_background_updates.sql
71 | 71/02event_push_summary_unique.sql
72 | 72/01add_room_type_to_state_stats.sql
72 | 72/01event_push_summary_receipt.sql
72 | 72/02event_push_actions_index.sql
72 | 72/03bg_populate_events_columns.py
72 | 72/03drop_event_reference_hashes.sql
72 | 72/03remove_groups.sql
72 | 72/04drop_column_application_services_state_last_txn.sql.postgres
72 | 72/05remove_unstable_private_read_receipts.sql
72 | 72/05receipts_event_stream_ordering.sql
72 | 72/06add_consent_ts_to_users.sql
72 | 72/06thread_notifications.sql
72 | 72/07force_update_current_state_events_membership.py
72 | 72/07thread_receipts.sql.postgres
72 | 72/08begin_cache_invalidation_seq_at_2.sql.postgres
72 | 72/08thread_receipts.sql
73 | 73/01event_failed_pull_attempts.sql
73 | 73/02add_pusher_enabled.sql
73 | 73/02room_id_indexes_for_purging.sql
73 | 73/03pusher_device_id.sql
73 | 73/03users_approved_column.sql
73 | 73/04partial_join_details.sql
73 | 73/04pending_device_list_updates.sql
73 | 73/05old_push_actions.sql.postgres
73 | 73/06thread_notifications_thread_id_idx.sql
73 | 73/08thread_receipts_non_null.sql.postgres
73 | 73/09partial_joined_via_destination.sql
73 | 73/09threads_table.sql
73 | 73/10_update_sqlite_fts4_tokenizer.py
73 | 73/10login_tokens.sql
73 | 73/11event_search_room_id_n_distinct.sql.postgres
73 | 73/12refactor_device_list_outbound_pokes.sql
73 | 73/13add_device_lists_index.sql
73 | 73/20_un_partial_stated_room_stream.sql
73 | 73/21_un_partial_stated_room_stream_seq.sql.postgres
73 | 73/22_rebuild_user_dir_stats.sql
73 | 73/22_un_partial_stated_event_stream.sql
73 | 73/23_fix_thread_index.sql
73 | 73/23_un_partial_stated_room_stream_seq.sql.postgres
73 | 73/24_events_jump_to_date_index.sql
73 | 73/25drop_presence.sql
74 | 74/01_user_directory_stale_remote_users.sql
74 | 74/90COMMENTS_destinations.sql.postgres
74 | 74/02_set_device_id_for_pushers_bg_update.sql
74 | 74/03_membership_tables_event_stream_ordering.sql.postgres
74 | 74/03_room_membership_index.sql
74 | 74/04_delete_e2e_backup_keys_for_deactivated_users.sql
74 | 74/04_membership_tables_event_stream_ordering_triggers.py
74 | 74/05_events_txn_id_device_id.sql
76 | 76/01_add_profiles_full_user_id_column.sql
76 | 76/02_add_user_filters_full_user_id_column.sql
76 | 76/03_per_user_experimental_features.sql
76 | 76/04_add_room_forgetter.sql
77 | 77/01_add_profiles_not_valid_check.sql.postgres
77 | 77/02_add_user_filters_not_valid_check.sql.postgres
77 | 77/03bg_populate_full_user_id_profiles.sql
77 | 77/04bg_populate_full_user_id_user_filters.sql
77 | 77/05thread_notifications_backfill.sql
77 | 77/06thread_notifications_not_null_event_push_actions.sql.postgres
77 | 77/06thread_notifications_not_null_event_push_actions_staging.sql.postgres
77 | 77/06thread_notifications_not_null_event_push_summary.sql.postgres
77 | 77/14bg_indices_event_stream_ordering.sql
78 | 78/01_validate_and_update_profiles.py
78 | 78/02_validate_and_update_user_filters.py
78 | 78/03_remove_unused_indexes_user_filters.py
78 | 78/03event_extremities_constraints.py
78 | 78/04_add_full_user_id_index_user_filters.py
78 | 78/04_read_write_locks_triggers.sql.postgres
79 | 79/03_read_write_locks_triggers.sql.postgres
79 | 79/04_mitigate_stream_ordering_update_race.py
79 | 79/05_read_write_locks_triggers.sql.postgres
(214 lines)

What I wonder about is that after searching the huge log file, the beginning of troubles did look different, indeed. When starting the container after the upgrade, I did see output in the terminal which I took as an indication of the missing table. When I did roll back to 1.89.0, it said that the database version would be beyond 1.89.0, so that I had to scale back the database. When I look at the relevant time in the log file and then for the first exceptions, it does seem to start somewhat different - in an endless file:

2023-09-17 15:32:15,337 - synapse.storage.txn - 765 - WARNING - prune_old_user_ips-472439- [TXN OPERROR] {_prune_old_user_ips-2f58cd} server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
0/5
2023-09-17 15:32:15,340 - twisted - 275 - CRITICAL - prune_old_user_ips-472439- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback
self.conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:15,343 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:15,346 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback
self._pool.disconnect(self._connection)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect
raise Exception("wrong connection for thread")
builtins.Exception: wrong connection for thread

2023-09-17 15:32:15,350 - synapse.metrics.background_process_metrics - 244 - ERROR - prune_old_user_ips-472439- Background process 'prune_old_user_ips' threw an exception
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/client_ips.py", line 484, in _prune_old_user_ips_txn
txn.execute(sql, (timestamp,))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/metrics/background_process_metrics.py", line 242, in run
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/client_ips.py", line 486, in _prune_old_user_ips
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback
self.conn.rollback()
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 66, in rollback
raise ConnectionLost()
twisted.enterprise.adbapi.ConnectionLost
2023-09-17 15:32:16,147 - synapse.storage.txn - 765 - WARNING - StoreKeyFetcher-322756- [TXN OPERROR] {get_server_keys_json-2f58ce} server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
0/5
2023-09-17 15:32:16,150 - twisted - 275 - CRITICAL - StoreKeyFetcher-322756- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback
self.conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:16,153 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:16,156 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback
self._pool.disconnect(self._connection)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect
raise Exception("wrong connection for thread")
builtins.Exception: wrong connection for thread

2023-09-17 15:32:16,162 - synapse.crypto.keyring - 687 - INFO - PerspectivesKeyFetcher-80445- Requesting keys [_FetchKeyRequest(server_name='vac.systemec.nl', minimum_valid_until_ts=1694957536141, key_ids=['ed25519:a_MLcI'])] from notary server matrix.org
2023-09-17 15:32:17,194 - synapse.http.matrixfederationclient - 348 - INFO - PerspectivesKeyFetcher-80445- {POST-O-650059} [matrix.org] Completed request: 200 OK in 1.03 secs, got 450 bytes - POST matrix-federation://matrix.org/_matrix/key/v2/query
2023-09-17 15:32:17,201 - synapse.storage.txn - 765 - WARNING - PerspectivesKeyFetcher-80445- [TXN OPERROR] {store_server_keys_json-2f58cf} server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
0/5
2023-09-17 15:32:17,204 - twisted - 275 - CRITICAL - PerspectivesKeyFetcher-80445- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback
self.conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:17,208 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:17,210 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback
self._pool.disconnect(self._connection)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect
raise Exception("wrong connection for thread")
builtins.Exception: wrong connection for thread

2023-09-17 15:32:17,215 - synapse.crypto.keyring - 646 - ERROR - PerspectivesKeyFetcher-80445- Unable to get key from 'matrix.org': InterfaceError connection already closed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback
self.conn.rollback()
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 66, in rollback
raise ConnectionLost()
twisted.enterprise.adbapi.ConnectionLost

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 638, in get_key
return await self.get_server_verify_key_v2_indirect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 741, in get_server_verify_key_v2_indirect
processed_response = await self.process_v2_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 601, in process_v2_response
await make_deferred_yieldable(
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/keys.py", line 170, in store_server_keys_json
await self.db_pool.simple_upsert(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1243, in simple_upsert
return await self.runInteraction(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1021, in inner_func
self.engine.attempt_to_set_autocommit(conn, False)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/engines/postgres.py", line 220, in attempt_to_set_autocommit
return conn.set_session(autocommit=autocommit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.InterfaceError: connection already closed
2023-09-17 15:32:17,234 - synapse.http.federation.well_known_resolver - 253 - INFO - ServerKeyFetcher-112- Fetching https://vac.systemec.nl/.well-known/matrix/server
2023-09-17 15:32:17,380 - synapse.http.federation.well_known_resolver - 190 - INFO - ServerKeyFetcher-112- Response from .well-known: {'m.server': 'vac.systemec.nl:443'}
2023-09-17 15:32:17,543 - synapse.http.matrixfederationclient - 348 - INFO - ServerKeyFetcher-112- {GET-O-650060} [vac.systemec.nl] Completed request: 200 OK in 0.31 secs, got 313 bytes - GET matrix-federation://vac.systemec.nl/_matrix/key/v2/server
2023-09-17 15:32:17,550 - synapse.storage.txn - 765 - WARNING - ServerKeyFetcher-112- [TXN OPERROR] {store_server_keys_json-2f58d0} server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
0/5
2023-09-17 15:32:17,552 - twisted - 275 - CRITICAL - ServerKeyFetcher-112- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback
self.conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:17,556 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:17,558 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback
self._pool.disconnect(self._connection)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect
raise Exception("wrong connection for thread")
builtins.Exception: wrong connection for thread

2023-09-17 15:32:17,563 - synapse.crypto.keyring - 867 - ERROR - ServerKeyFetcher-112- Error getting keys from vac.systemec.nl
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
return self.simple_upsert_txn_native_upsert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
txn.execute(sql, list(allvalues.values()))
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback
self.conn.rollback()
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 66, in rollback
raise ConnectionLost()
twisted.enterprise.adbapi.ConnectionLost

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 862, in get_keys
keys = await self.get_server_verify_keys_v2_direct(server_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 919, in get_server_verify_keys_v2_direct
return await self.process_v2_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 601, in process_v2_response
await make_deferred_yieldable(
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/keys.py", line 170, in store_server_keys_json
await self.db_pool.simple_upsert(
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1243, in simple_upsert
return await self.runInteraction(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
result = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1021, in inner_func
self.engine.attempt_to_set_autocommit(conn, False)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/engines/postgres.py", line 220, in attempt_to_set_autocommit
return conn.set_session(autocommit=autocommit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.InterfaceError: connection already closed
2023-09-17 15:32:17,582 - synapse.federation.transport.server._base - 312 - WARNING - PUT-569862- authenticate_request failed: 401: Failed to find any key to satisfy: _FetchKeyRequest(server_name='vac.systemec.nl', minimum_valid_until_ts=1694957536141, key_ids=['ed25519:a_MLcI'])
2023-09-17 15:32:17,585 - synapse.http.server - 118 - INFO - PUT-569862- <XForwardedForRequest at 0x7fdb28f94650 method='PUT' uri='/_matrix/federation/v1/send/1680639381906' clientproto='HTTP/1.1' site='8008'> SynapseError: 401 - Failed to find any key to satisfy: _FetchKeyRequest(server_name='vac.systemec.nl', minimum_valid_until_ts=1694957536141, key_ids=['ed25519:a_MLcI'])
2023-09-17 15:32:17,596 - synapse.access.http.8008 - 465 - INFO - PUT-569862- 89.20.83.36 - 8008 - {None} Processed request: 1.448sec/0.008sec (0.007sec, 0.001sec) (0.000sec/0.000sec/0) 187B 401 "PUT /_matrix/federation/v1/send/1680639381906 HTTP/1.1" "Synapse/1.80.0" [0 dbevts]
2023-09-17 15:32:20,338 - synapse.storage.txn - 765 - WARNING - prune_old_user_ips-472440- [TXN OPERROR] {_prune_old_user_ips-2f58d1} server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
0/5
2023-09-17 15:32:20,341 - twisted - 275 - CRITICAL - prune_old_user_ips-472440- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
return func(db_conn, *args, **kwargs)
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback
self.conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:20,345 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:20,348 - twisted - 275 - CRITICAL - sentinel- Rollback failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback
self._pool.disconnect(self._connection)
File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect
raise Exception("wrong connection for thread")
builtins.Exception: wrong connection for thread

[clokep]

builtins.Exception: wrong connection for thread

This is a very unrelated error, see #16333. If that doesn't fix your issue please file a separate issue.

[clokep]

It looks like Synapse 1.83.0 did the following operations:

• Bumped the minimum schema compat version from 73 -> 74.
• Bumped the schema version from 74 -> 76.

I'm suspicious that bumping the schema version twice isn't safe as we have CI checks which compare if develop is +/- of the PR version.

---

@Anthial The SQL in #16335 (comment) would be interesting to see still.

[Anthial]

Yes! I'm on it! Just trying to figure out the best way of accessing the SQLite database! I figure I might just back it up once more, and use a DB browser to find the required information! (There seems to be a lack of documentation on how to access the SQLite database when utilising Docker)

[Anthial]

Something like?

Version | file
73 | 73/01event_failed_pull_attempts.sql
-- | --
73 | 73/02add_pusher_enabled.sql
73 | 73/02room_id_indexes_for_purging.sql
73 | 73/03pusher_device_id.sql
73 | 73/03users_approved_column.sql
73 | 73/04partial_join_details.sql
73 | 73/04pending_device_list_updates.sql
73 | 73/05old_push_actions.sql.sqlite
73 | 73/06thread_notifications_thread_id_idx.sql
73 | 73/08thread_receipts_non_null.sql.sqlite
73 | 73/09partial_joined_via_destination.sql
73 | 73/09threads_table.sql
73 | 73/10_update_sqlite_fts4_tokenizer.py
73 | 73/10login_tokens.sql
73 | 73/12refactor_device_list_outbound_pokes.sql
73 | 73/13add_device_lists_index.sql
73 | 73/20_un_partial_stated_room_stream.sql
73 | 73/22_rebuild_user_dir_stats.sql
73 | 73/22_un_partial_stated_event_stream.sql
73 | 73/23_fix_thread_index.sql
73 | 73/24_events_jump_to_date_index.sql
73 | 73/25drop_presence.sql
74 | 74/01_user_directory_stale_remote_users.sql
74 | 74/01membership_tables_event_stream_ordering.sql
74 | 74/02_set_device_id_for_pushers_bg_update.sql
74 | 74/02membership_tables_event_stream_ordering_triggers.py
76 | 76/01_add_profiles_full_user_id_column.sql
76 | 76/02_add_user_filters_full_user_id_column.sql
76 | 76/03_per_user_experimental_features.sql
76 | 76/04_add_room_forgetter.sql
77 | 77/03bg_populate_full_user_id_profiles.sql
77 | 77/04bg_populate_full_user_id_user_filters.sql
77 | 77/05thread_notifications_backfill.sql
77 | 77/06thread_notifications_not_null.sql.sqlite
77 | 77/14bg_indices_event_stream_ordering.sql
78 | 78/01_validate_and_update_profiles.py
78 | 78/02_validate_and_update_user_filters.py
78 | 78/03_remove_unused_indexes_user_filters.py
78 | 78/04_add_full_user_id_index_user_filters.py
78 | 78/03event_extremities_constraints.py
79 | 79/03_read_write_locks_triggers.sql.sqlite
79 | 79/04_mitigate_stream_ordering_update_race.py
79 | 79/05_read_write_locks_triggers.sql.sqlite
80 | 80/01_users_alter_locked.sql
80 | 80/02_scheduled_tasks.sql

[clokep]

Thanks! It looks like you're missing 3 deltas:

• 74/03_membership_tables_event_stream_ordering.sql.sqlite
• 74/04_membership_tables_event_stream_ordering_triggers.py
• 74/05_events_txn_id_device_id.sql

[Anthial]

Very strange! I wonder how that could have happened!

[Anthial]

Is there anyway to apply these deltas to the database or is it permanently broken?

[clokep]

Is there anyway to apply these deltas to the database or is it permanently broken?

You could try to apply them by hand, in order. (And hope no invalid data has occurred in the meantime.) I'd love to know how you database ended up in this state though.

[clokep]

I'm suspicious that bumping the schema version twice isn't safe as we have CI checks which compare if develop is +/- of the PR version.

#11559 seems to describe the concern I have.