SSH: refactor Jupyter init script to use better Spark session initialization flow

[ilia-db]

Changes

Why

Tests

[eng-dev-ecosystem-bot]

Commit: a753de0

Run: 22631818550

	Env	🪲​BUG	❌​FAIL	🟨​KNOWN	🔄​flaky	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
🪲​	aws linux	1		7	2	1	7	265	776	7:47
🪲​	aws windows	1		7	1	1	7	268	774	6:54
🪲​	aws-ucws linux	1			2	7	7	363	691	8:07
🪲​	aws-ucws windows	1			2	7	7	365	689	7:19
🪲​	azure linux	2	8	1		1	9	261	774	145:41
🪲​	azure windows	2	6	1	5		9	261	772	125:31
🪲​	azure-ucws linux	2	8	1	1	1	9	359	687	147:14
🪲​	azure-ucws windows	2	8	1	1	1	9	361	685	145:52
🪲​	gcp linux	2	8	1		1	9	257	777	144:55
🪲​	gcp windows	2	8	1		1	9	259	775	144:59

31 interesting tests: 8 FAIL, 7 KNOWN, 7 flaky, 7 SKIP, 2 BUG

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
🟨​	TestAccept	🟨​K	🟨​K	💚​R	🔄​f	🟨​K	🟨​K	🟨​K	🟨​K	🟨​K	🟨​K
🪲​	TestAccept/bundle/deployment/bind/alert	🙈​s	🙈​s	🙈​s	🙈​s	🪲​B	🪲​B	🪲​B	🪲​B	🪲​B	🪲​B
❌​	TestAccept/bundle/deployment/bind/alert/DATABRICKS_BUNDLE_ENGINE=direct					❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/deployment/bind/alert/DATABRICKS_BUNDLE_ENGINE=terraform					❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
🔄​	TestAccept/bundle/destroy/jobs-and-pipeline	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p	✅​p	✅​p	✅​p
🔄​	TestAccept/bundle/destroy/jobs-and-pipeline/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f	✅​p	✅​p	✅​p	✅​p
❌​	TestAccept/bundle/generate/alert	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/generate/alert/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/generate/alert/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	❌​F	🔄​f	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/resources/alerts/with_file	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=direct	✅​p	✅​p	✅​p	✅​p	❌​F	🔄​f	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=terraform	✅​p	✅​p	✅​p	✅​p	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
🙈​	TestAccept/bundle/resources/permissions	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🟨​	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions	🟨​K	🟨​K	💚​R	💚​R	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🟨​	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨​K	🟨​K	💚​R	💚​R
🟨​	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨​K	🟨​K	💚​R	💚​R
🟨​	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions	🟨​K	🟨​K	💚​R	💚​R	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🟨​	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨​K	🟨​K	💚​R	💚​R
🟨​	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨​K	🟨​K	💚​R	💚​R
🙈​	TestAccept/bundle/resources/postgres_branches/basic	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/postgres_branches/recreate	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/postgres_branches/update_protected	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/postgres_branches/without_branch_id	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/postgres_endpoints/recreate	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/synced_database_tables/basic	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🔄​	TestAccept/selftest/record_cloud/pipeline-crud/DATABRICKS_BUNDLE_ENGINE=direct	🔄​f	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p
🔄​	TestAccept/selftest/record_cloud/pipeline-crud/DATABRICKS_BUNDLE_ENGINE=terraform	🔄​f	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p
🔄​	TestAccept/ssh/connect-serverless-gpu	🙈​s	🙈​s	🔄​f	🔄​f	🙈​s	🙈​s	🔄​f	🔄​f	🙈​s	🙈​s
🔄​	TestAccept/ssh/connection	💚​R	💚​R	🔄​f	💚​R	💚​R	🔄​f	💚​R	💚​R	💚​R	💚​R
🔄​	TestGenerateFromExistingPipelineAndDeploy	✅​p	🔄​f	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p	✅​p
🪲​	TestImportFileFormatSource	🪲​B	🪲​B	🪲​B	🪲​B	🪲​B	🪲​B	🪲​B	🪲​B	🪲​B	🪲​B

Top 22 slowest tests (at least 2 minutes):

duration	env	testname
4:39	azure windows	TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=direct
4:35	azure windows	TestAccept/bundle/generate/alert/DATABRICKS_BUNDLE_ENGINE=terraform
4:10	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:49	azure windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:43	azure-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:20	azure-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:13	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:12	aws-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:07	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:56	aws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:47	aws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:44	aws-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:42	aws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:40	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:40	aws-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:38	aws-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:38	aws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:14	azure linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:13	azure linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:11	azure-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:08	azure windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:03	azure-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct