From 831710239c24cea657c05deb58ed04420927e9b5 Mon Sep 17 00:00:00 2001 From: Airton Lastori Date: Thu, 23 Jan 2025 19:51:54 -0500 Subject: [PATCH 1/9] docs: update sync-diff-inspector docs for move to tiflow repo - Update GitHub repository links from tidb-tools to tiflow - Add TiUP installation method for v9.0.0+ - Update Docker image references - Add version-specific installation instructions - Update download-ecosystem-tools.md with new package information --- download-ecosystem-tools.md | 6 +++--- .../sync-diff-inspector-overview.md | 21 ++++++++++++++++--- 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/download-ecosystem-tools.md b/download-ecosystem-tools.md index 90de221f3f004..99ae33cda5714 100644 --- a/download-ecosystem-tools.md +++ b/download-ecosystem-tools.md @@ -8,11 +8,11 @@ aliases: ['/docs/dev/download-ecosystem-tools/','/docs/dev/reference/tools/downl This document describes how to download the TiDB Toolkit. -TiDB Toolkit contains frequently used TiDB tools, such as data export tool Dumpling, data import tool TiDB Lightning, and backup and restore tool BR. +TiDB Toolkit contains frequently used TiDB tools, such as data export tool Dumpling, data import tool TiDB Lightning, backup and restore tool BR, and data consistency checker sync-diff-inspector. > **Tip:** > -> - If your deployment environment has internet access, you can deploy a TiDB tool using a single [TiUP command](/tiup/tiup-component-management.md), so there is no need to download the TiDB Toolkit separately. +> - For TiDB v9.0.0 and later, most tools including sync-diff-inspector are directly available through TiUP. If your deployment environment has internet access, you can deploy a TiDB tool using a single [TiUP command](/tiup/tiup-component-management.md), so there is no need to download the TiDB Toolkit separately. > - If you need to deploy and maintain TiDB on Kubernetes, instead of downloading the TiDB Toolkit, follow the steps in [TiDB Operator offline installation](https://docs.pingcap.com/tidb-in-kubernetes/stable/deploy-tidb-operator#offline-installation). ## Environment requirements @@ -46,7 +46,7 @@ Depending on which tools you want to use, you can install the corresponding offl | [TiDB Data Migration (DM)](/dm/dm-overview.md) | `dm-worker-{version}-linux-{arch}.tar.gz`
`dm-master-{version}-linux-{arch}.tar.gz`
`dmctl-{version}-linux-{arch}.tar.gz` | | [TiCDC](/ticdc/ticdc-overview.md) | `cdc-{version}-linux-{arch}.tar.gz` | | [Backup & Restore (BR)](/br/backup-and-restore-overview.md) | `br-{version}-linux-{arch}.tar.gz` | -| [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) | `sync_diff_inspector` | +| [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) | For TiDB v9.0.0 and later: included in TiUP and `tiflow-{version}-linux-{arch}.tar.gz`
For TiDB versions before v9.0.0: `sync_diff_inspector` from tidb-tools | | [PD Recover](/pd-recover.md) | `pd-recover-{version}-linux-{arch}.tar` | > **Note:** diff --git a/sync-diff-inspector/sync-diff-inspector-overview.md b/sync-diff-inspector/sync-diff-inspector-overview.md index 1a9e19ca90518..33e2bb915aea7 100644 --- a/sync-diff-inspector/sync-diff-inspector-overview.md +++ b/sync-diff-inspector/sync-diff-inspector-overview.md @@ -6,15 +6,30 @@ aliases: ['/docs/dev/sync-diff-inspector/sync-diff-inspector-overview/','/docs/d # sync-diff-inspector User Guide -[sync-diff-inspector](https://github.com/pingcap/tidb-tools/tree/master/sync_diff_inspector) is a tool used to compare data stored in the databases with the MySQL protocol. For example, it can compare the data in MySQL with that in TiDB, the data in MySQL with that in MySQL, or the data in TiDB with that in TiDB. In addition, you can also use this tool to repair data in the scenario where a small amount of data is inconsistent. +[sync-diff-inspector](https://github.com/pingcap/tiflow/tree/master/sync_diff_inspector) is a tool used to compare data stored in the databases with the MySQL protocol. For example, it can compare the data in MySQL with that in TiDB, the data in MySQL with that in MySQL, or the data in TiDB with that in TiDB. In addition, you can also use this tool to repair data in the scenario where a small amount of data is inconsistent. -This guide introduces the key features of sync-diff-inspector and describes how to configure and use this tool. To download sync-diff-inspector, use one of the following methods: +This guide introduces the key features of sync-diff-inspector and describes how to configure and use this tool. The installation method varies depending on your TiDB version: +For TiDB v9.0.0 and later: ++ Install using TiUP: + ```shell + tiup install sync-diff-inspector + ``` + Binary package. The sync-diff-inspector binary package is included in the TiDB Toolkit. To download the TiDB Toolkit, see [Download TiDB Tools](/download-ecosystem-tools.md). + Docker image. Execute the following command to download: {{< copyable "shell-regular" >}} + ```shell + docker pull pingcap/tiflow:latest + ``` + +For TiDB versions before v9.0.0: ++ Binary package from the legacy tidb-tools repository. The sync-diff-inspector binary package is included in the TiDB Toolkit. To download the TiDB Toolkit, see [Download TiDB Tools](/download-ecosystem-tools.md). ++ Docker image (legacy version). Execute the following command to download: + + {{< copyable "shell-regular" >}} + ```shell docker pull pingcap/tidb-tools:latest ``` @@ -106,7 +121,7 @@ skip-non-existing-table = false # (optional) Use the snapshot feature. If enabled, historical data is used for comparison. # snapshot = "386902609362944000" - # When "snapshot" is set to "auto", the last syncpoints generated by TiCDC in the upstream and downstream are used for comparison. For details, see . + # When "snapshot" is set to "auto", the last syncpoints generated by TiCDC in the upstream and downstream are used for comparison. For details, see . # snapshot = "auto" ########################### Routes ############################## From e6c4ca4222f7f1a52bd318aca7213a3553156644 Mon Sep 17 00:00:00 2001 From: Airton Lastori Date: Thu, 23 Jan 2025 20:18:34 -0500 Subject: [PATCH 2/9] style: fix markdown linting errors in sync-diff-inspector docs --- sync-diff-inspector/sync-diff-inspector-overview.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/sync-diff-inspector/sync-diff-inspector-overview.md b/sync-diff-inspector/sync-diff-inspector-overview.md index 33e2bb915aea7..d66b0196a0eac 100644 --- a/sync-diff-inspector/sync-diff-inspector-overview.md +++ b/sync-diff-inspector/sync-diff-inspector-overview.md @@ -11,11 +11,15 @@ aliases: ['/docs/dev/sync-diff-inspector/sync-diff-inspector-overview/','/docs/d This guide introduces the key features of sync-diff-inspector and describes how to configure and use this tool. The installation method varies depending on your TiDB version: For TiDB v9.0.0 and later: + + Install using TiUP: + ```shell tiup install sync-diff-inspector ``` + + Binary package. The sync-diff-inspector binary package is included in the TiDB Toolkit. To download the TiDB Toolkit, see [Download TiDB Tools](/download-ecosystem-tools.md). + + Docker image. Execute the following command to download: {{< copyable "shell-regular" >}} @@ -25,7 +29,9 @@ For TiDB v9.0.0 and later: ``` For TiDB versions before v9.0.0: + + Binary package from the legacy tidb-tools repository. The sync-diff-inspector binary package is included in the TiDB Toolkit. To download the TiDB Toolkit, see [Download TiDB Tools](/download-ecosystem-tools.md). + + Docker image (legacy version). Execute the following command to download: {{< copyable "shell-regular" >}} @@ -298,8 +304,6 @@ A SQL file contains the tale to which the chunk belong and the range information DIFF COLUMNS ╏ `K` ╏ `C` ╏ `PAD` ╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╋╍╍╍╍╍╍╍╍╍╋╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╋╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍ source data ╏ 2501808 ╏ 'hello' ╏ 'world' -╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╋╍╍╍╍╍╍╍╍╍╋╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╋╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍ - target data ╏ 5003616 ╏ '0709824117-9809973320-4456050422' ╏ '1714066100-7057807621-1425865505' ╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╋╍╍╍╍╍╍╍╍╍╋╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╋╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍ */ REPLACE INTO `sbtest`.`sbtest99`(`id`,`k`,`c`,`pad`) VALUES (3700000,2501808,'hello','world'); From 8032c475af4fdd09b173d65dc9666935081d79c8 Mon Sep 17 00:00:00 2001 From: Airton Lastori Date: Tue, 28 Jan 2025 12:28:06 -0500 Subject: [PATCH 3/9] Update sync-diff-inspector-overview.md Update Docker image for sync-diff-inspector The previous `docker pull pingcap/tiflow:latest` command was incorrect. After confirming with Flare Zuo (R&D Group - Engineering Productivity - Efficiency & Ops team), we should use `docker pull pingcap/sync-diff-inspector:latest` to align with TiCDC and DM, ensuring consistency across TiFlow-managed images. This update reflects the correct image name for better usability. --- sync-diff-inspector/sync-diff-inspector-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sync-diff-inspector/sync-diff-inspector-overview.md b/sync-diff-inspector/sync-diff-inspector-overview.md index d66b0196a0eac..b0518a798b2f4 100644 --- a/sync-diff-inspector/sync-diff-inspector-overview.md +++ b/sync-diff-inspector/sync-diff-inspector-overview.md @@ -25,7 +25,7 @@ For TiDB v9.0.0 and later: {{< copyable "shell-regular" >}} ```shell - docker pull pingcap/tiflow:latest + docker pull pingcap/sync-diff-inspector:latest ``` For TiDB versions before v9.0.0: From 8fc345108ea341cf58700543dff79f5fbaee0d66 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 26 Feb 2025 16:18:52 +0800 Subject: [PATCH 4/9] revise wording and remove copyable --- download-ecosystem-tools.md | 4 ++-- sync-diff-inspector/sync-diff-inspector-overview.md | 6 +----- 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/download-ecosystem-tools.md b/download-ecosystem-tools.md index 99ae33cda5714..41254cb8439e8 100644 --- a/download-ecosystem-tools.md +++ b/download-ecosystem-tools.md @@ -12,7 +12,7 @@ TiDB Toolkit contains frequently used TiDB tools, such as data export tool Dumpl > **Tip:** > -> - For TiDB v9.0.0 and later, most tools including sync-diff-inspector are directly available through TiUP. If your deployment environment has internet access, you can deploy a TiDB tool using a single [TiUP command](/tiup/tiup-component-management.md), so there is no need to download the TiDB Toolkit separately. +> - For TiDB v9.0.0 and later versions, most tools, including sync-diff-inspector, are directly available through TiUP. If your deployment environment has internet access, you can deploy a TiDB tool using a single [TiUP command](/tiup/tiup-component-management.md), so there is no need to download the TiDB Toolkit separately. > - If you need to deploy and maintain TiDB on Kubernetes, instead of downloading the TiDB Toolkit, follow the steps in [TiDB Operator offline installation](https://docs.pingcap.com/tidb-in-kubernetes/stable/deploy-tidb-operator#offline-installation). ## Environment requirements @@ -46,7 +46,7 @@ Depending on which tools you want to use, you can install the corresponding offl | [TiDB Data Migration (DM)](/dm/dm-overview.md) | `dm-worker-{version}-linux-{arch}.tar.gz`
`dm-master-{version}-linux-{arch}.tar.gz`
`dmctl-{version}-linux-{arch}.tar.gz` | | [TiCDC](/ticdc/ticdc-overview.md) | `cdc-{version}-linux-{arch}.tar.gz` | | [Backup & Restore (BR)](/br/backup-and-restore-overview.md) | `br-{version}-linux-{arch}.tar.gz` | -| [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) | For TiDB v9.0.0 and later: included in TiUP and `tiflow-{version}-linux-{arch}.tar.gz`
For TiDB versions before v9.0.0: `sync_diff_inspector` from tidb-tools | +| [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) | For TiDB v9.0.0 and later versions: included in TiUP and `tiflow-{version}-linux-{arch}.tar.gz`
For TiDB versions before v9.0.0: `sync_diff_inspector` from tidb-tools | | [PD Recover](/pd-recover.md) | `pd-recover-{version}-linux-{arch}.tar` | > **Note:** diff --git a/sync-diff-inspector/sync-diff-inspector-overview.md b/sync-diff-inspector/sync-diff-inspector-overview.md index b0518a798b2f4..e13df50c587ce 100644 --- a/sync-diff-inspector/sync-diff-inspector-overview.md +++ b/sync-diff-inspector/sync-diff-inspector-overview.md @@ -10,7 +10,7 @@ aliases: ['/docs/dev/sync-diff-inspector/sync-diff-inspector-overview/','/docs/d This guide introduces the key features of sync-diff-inspector and describes how to configure and use this tool. The installation method varies depending on your TiDB version: -For TiDB v9.0.0 and later: +For TiDB v9.0.0 and later versions: + Install using TiUP: @@ -22,8 +22,6 @@ For TiDB v9.0.0 and later: + Docker image. Execute the following command to download: - {{< copyable "shell-regular" >}} - ```shell docker pull pingcap/sync-diff-inspector:latest ``` @@ -34,8 +32,6 @@ For TiDB versions before v9.0.0: + Docker image (legacy version). Execute the following command to download: - {{< copyable "shell-regular" >}} - ```shell docker pull pingcap/tidb-tools:latest ``` From f02252d4429024578650ce48f9a92b13e1a97dc6 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 6 Mar 2025 15:29:52 +0800 Subject: [PATCH 5/9] add GitHub link --- sync-diff-inspector/sync-diff-inspector-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sync-diff-inspector/sync-diff-inspector-overview.md b/sync-diff-inspector/sync-diff-inspector-overview.md index e13df50c587ce..288443f55cbe8 100644 --- a/sync-diff-inspector/sync-diff-inspector-overview.md +++ b/sync-diff-inspector/sync-diff-inspector-overview.md @@ -28,7 +28,7 @@ For TiDB v9.0.0 and later versions: For TiDB versions before v9.0.0: -+ Binary package from the legacy tidb-tools repository. The sync-diff-inspector binary package is included in the TiDB Toolkit. To download the TiDB Toolkit, see [Download TiDB Tools](/download-ecosystem-tools.md). ++ Binary package from the legacy [`tidb-tools`](https://github.com/pingcap/tidb-tools) repository. The sync-diff-inspector binary package is included in the TiDB Toolkit. To download the TiDB Toolkit, see [Download TiDB Tools](/download-ecosystem-tools.md). + Docker image (legacy version). Execute the following command to download: From b288bc91a21b2b0b5230563923dbcda8b452ad55 Mon Sep 17 00:00:00 2001 From: Airton Lastori Date: Fri, 7 Mar 2025 19:52:39 -0500 Subject: [PATCH 6/9] Update download-ecosystem-tools.md Co-authored-by: Grace Cai --- download-ecosystem-tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/download-ecosystem-tools.md b/download-ecosystem-tools.md index 41254cb8439e8..c577e81004c48 100644 --- a/download-ecosystem-tools.md +++ b/download-ecosystem-tools.md @@ -46,7 +46,7 @@ Depending on which tools you want to use, you can install the corresponding offl | [TiDB Data Migration (DM)](/dm/dm-overview.md) | `dm-worker-{version}-linux-{arch}.tar.gz`
`dm-master-{version}-linux-{arch}.tar.gz`
`dmctl-{version}-linux-{arch}.tar.gz` | | [TiCDC](/ticdc/ticdc-overview.md) | `cdc-{version}-linux-{arch}.tar.gz` | | [Backup & Restore (BR)](/br/backup-and-restore-overview.md) | `br-{version}-linux-{arch}.tar.gz` | -| [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) | For TiDB v9.0.0 and later versions: included in TiUP and `tiflow-{version}-linux-{arch}.tar.gz`
For TiDB versions before v9.0.0: `sync_diff_inspector` from tidb-tools | +| [sync-diff-inspector](/sync-diff-inspector/sync-diff-inspector-overview.md) | For TiDB v9.0.0 and later versions: `tiflow-{version}-linux-{arch}.tar.gz`
For TiDB versions before v9.0.0: `sync_diff_inspector` | | [PD Recover](/pd-recover.md) | `pd-recover-{version}-linux-{arch}.tar` | > **Note:** From 7d7687d51485309ee406a2fcf53dcf70a08635b3 Mon Sep 17 00:00:00 2001 From: Airton Lastori Date: Mon, 17 Mar 2025 12:55:37 -0400 Subject: [PATCH 7/9] Update sync-diff-inspector/sync-diff-inspector-overview.md Co-authored-by: Grace Cai --- sync-diff-inspector/sync-diff-inspector-overview.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/sync-diff-inspector/sync-diff-inspector-overview.md b/sync-diff-inspector/sync-diff-inspector-overview.md index 288443f55cbe8..37eadf884a4ac 100644 --- a/sync-diff-inspector/sync-diff-inspector-overview.md +++ b/sync-diff-inspector/sync-diff-inspector-overview.md @@ -8,7 +8,11 @@ aliases: ['/docs/dev/sync-diff-inspector/sync-diff-inspector-overview/','/docs/d [sync-diff-inspector](https://github.com/pingcap/tiflow/tree/master/sync_diff_inspector) is a tool used to compare data stored in the databases with the MySQL protocol. For example, it can compare the data in MySQL with that in TiDB, the data in MySQL with that in MySQL, or the data in TiDB with that in TiDB. In addition, you can also use this tool to repair data in the scenario where a small amount of data is inconsistent. -This guide introduces the key features of sync-diff-inspector and describes how to configure and use this tool. The installation method varies depending on your TiDB version: +This guide introduces the key features of sync-diff-inspector and describes how to configure and use this tool. + +## Install sync-diff-inspector + +The installation method varies depending on your TiDB version: For TiDB v9.0.0 and later versions: From 2cb952b680820abe78b990be9e96951eca9f8bee Mon Sep 17 00:00:00 2001 From: Airton Lastori Date: Mon, 17 Mar 2025 13:02:47 -0400 Subject: [PATCH 8/9] Move Key features section before Install section to match introduction flow --- .../sync-diff-inspector-overview.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/sync-diff-inspector/sync-diff-inspector-overview.md b/sync-diff-inspector/sync-diff-inspector-overview.md index 37eadf884a4ac..033071e3a10cb 100644 --- a/sync-diff-inspector/sync-diff-inspector-overview.md +++ b/sync-diff-inspector/sync-diff-inspector-overview.md @@ -10,6 +10,15 @@ aliases: ['/docs/dev/sync-diff-inspector/sync-diff-inspector-overview/','/docs/d This guide introduces the key features of sync-diff-inspector and describes how to configure and use this tool. +## Key features + +* Compare the table schema and data +* Generate the SQL statements used to repair data if the data inconsistency exists +* Support [data check for tables with different schema or table names](/sync-diff-inspector/route-diff.md) +* Support [data check in the sharding scenario](/sync-diff-inspector/shard-diff.md) +* Support [data check for TiDB upstream-downstream clusters](/ticdc/ticdc-upstream-downstream-check.md) +* Support [data check in the DM replication scenario](/sync-diff-inspector/dm-diff.md) + ## Install sync-diff-inspector The installation method varies depending on your TiDB version: @@ -40,15 +49,6 @@ For TiDB versions before v9.0.0: docker pull pingcap/tidb-tools:latest ``` -## Key features - -* Compare the table schema and data -* Generate the SQL statements used to repair data if the data inconsistency exists -* Support [data check for tables with different schema or table names](/sync-diff-inspector/route-diff.md) -* Support [data check in the sharding scenario](/sync-diff-inspector/shard-diff.md) -* Support [data check for TiDB upstream-downstream clusters](/ticdc/ticdc-upstream-downstream-check.md) -* Support [data check in the DM replication scenario](/sync-diff-inspector/dm-diff.md) - ## Restrictions of sync-diff-inspector * Online check is not supported for data migration between MySQL and TiDB. Ensure that no data is written into the upstream-downstream checklist, and that data in a certain range is not changed. You can check data in this range by setting `range`. From c8580cbb5330a4088c5ec019ecf841013ead8fbd Mon Sep 17 00:00:00 2001 From: Airton Lastori Date: Wed, 26 Mar 2025 13:08:11 -0400 Subject: [PATCH 9/9] Update sync-diff-inspector/sync-diff-inspector-overview.md Co-authored-by: Aolin --- sync-diff-inspector/sync-diff-inspector-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sync-diff-inspector/sync-diff-inspector-overview.md b/sync-diff-inspector/sync-diff-inspector-overview.md index 033071e3a10cb..d5b79b507945c 100644 --- a/sync-diff-inspector/sync-diff-inspector-overview.md +++ b/sync-diff-inspector/sync-diff-inspector-overview.md @@ -127,7 +127,7 @@ skip-non-existing-table = false # (optional) Use the snapshot feature. If enabled, historical data is used for comparison. # snapshot = "386902609362944000" - # When "snapshot" is set to "auto", the last syncpoints generated by TiCDC in the upstream and downstream are used for comparison. For details, see . + # When "snapshot" is set to "auto", the last syncpoints generated by TiCDC in the upstream and downstream are used for comparison. For details, see . # snapshot = "auto" ########################### Routes ##############################