metrics: Enhance the o11y of TiFlash storage layer (#10275)

[ti-chi-bot]

This is an automated cherry-pick of #10275

What problem does this PR solve?

Issue Number: close #10272

Problem Summary:

What is changed and how it works?

* Add metrics about
  * Number of keyspace, Storage instance, Segment instance, MemTable instance
  * Bytes of MemTable and allocated bytes of MemTable
* System table
  * Add `delta_cache_alloc_size` to `system.dt_segments`
  * Add `column_count`,`delta_cache_alloc_size` to `system.dt_tables`

New Grafana panels on TiFlash-Summary:

• "Threads" - a copy of "Threads" from TiFlash-Proxy-Details. We can check whether specified thread request a large IO, etc
• "Imbalanced read/write" - Check whether the CPU usage/read throughput/write throughput is imbalanced between different TiFlash instances
• "Memory trace" - A summary status of objects in memory, check whether there are OOM risk. Number of tables, segments, memtable bytes, etc

With the "Memory trace" panel, we can get more insight about the issue #10253 (comment) that mem-table may take too much memory.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

* deploy a cluster with 2 tiflash
* load chbenchmark 1500
* run chbenchmark workload with 50 tp thread and 1 ap thread for 60 minutes
"tiup bench ch --host 10.2.12.81 -P 8020 --warehouses 1500 run -D chbenchmark -T 50 -t 1 --time 60m"
* collect the metrics below

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Enhance the observability for TiFlash OOM risk under wide-column scenario

[ti-chi-bot]

@JaySon-Huang This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jayson-huang. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment