Repository navigation
Use Node-IPs for external reachability and localities #641
Description
Scope
This issue involves the following topics:
- ensuring that HBase is reachable externally via listener endpoints (see parent issue)
- ensure that these endpoints (or a load balancer that uses them) is consistently used in the UI screens
- ensure the HBase advertises the nodes it is running on so that e.g. an accurate locality can be derived (see feat: HBase resolvable endpoints docker-images#1159)
UI endpoint resolution - previous notes
UI endpoint resolution
The endpoints exposed in the HBase UI (mainly the master) are not resolvable for end-users. This means that, while it is possible to use the listener classes to write discovery information to ConfigMaps (so that users can easily reach master/regionserver/rest-server UIs), links referenced in these webpages - most obviously the list of region servers - are dead.
Context
- Hbase uses the following internally for hostname and port resolution. If these settings are not in hbase-site.xml then a default mechanism is used.
hbase.{ master | regionserver| rest }.{ hostname | port }
- the default mechanism involves a DNS lookup (see here and here), which also requires that reverse DNS lookups not be disabled. See here.
- the endpoint information written to Zookeeper is what is used for listing the regionservers, and is also used for internal binding
- this causes problems as HBase does not have access to the host network
- tweaking internal details for the sake of user-facing endpoints runs the risk of breaking things, so a safer solution would be to patch HBase to use listener information only for the purpose of providing externally-reachable endpoints
Proposed solution
- extend proto structs and use where needed - see below Use Node-IPs for external reachability and localities #641 (comment)
Node advertisement
- the aim is to use the proto struct extensions when calculating localities etc. Not yet researched.
Metadata
Metadata
Assignees
Labels
No labels
Activity
adwk67 commentedon Apr 23, 2025
Notes
HBase master- (
HMaster extends HRegionServer) and region-servers (HRegionServer) do not have a direct connection: server data is persisted/shared in ZooKeeper and also in HBase meta tables. The data in ZooKeeper uses protobuf entities which serialize from / deserialize toServerNameobjects, which are used in a variety of places in the code to share server data (servlet rendering, RPC calls, ZooKeeper persistence etc. etc.).Listener data (hostname and port) can be written to custom config settings in
hbase-site.xmlon start-up, once the listener data has been written to the pods. These settings can then be retrieved for each server. However, the host/port are used for internal RPC calls as well as UI data, and node IPs (in the case ofexternal-stable/external-unstablelistener classes) are not reachable without giving access to the host network.Extending proto entities
Extending
ServerName(and the associated protobuf class) to add two new fields - externalHostname and externalPort - has the advantage of keeping that data with other server meta data: when the active master changes (or when region servers are added) the existing mechanisms for tracking these changes are unaffected. However, this is fairly invasive asServerNameis the object used in multiple places in the code. If we want to use these fields in a just-in-time manner, thenMasterStatusServletneeds to be patched to replace hostname/port with the "external" values. This is not that simple, though, as the wholeServerNameobject is used as the key for internal caching, and the link is thus broken between the server name and the metrics that should also be rendered in the UI.Changes to UI rendering
Alternatively, we can leave the
MasterStatusServletcode as-is (i.e. do not attempt to replace host with external host etc.) and make the changes in the.jamonand.jspfiles, where the host/port are ready out and displayed directly.Patch Status
There is a WIP-patch available in this branch (associated changes for hbase-operator changes are here).
adwk67 commentedon Apr 24, 2025
I've asked about this on the dev mailing list: https://lists.apache.org/thread/rx7box0f7b02pg0xnyhlh0sj5b7zp6td
adwk67 commentedon Apr 29, 2025
Moving this into Development:Track with the following comments:
RegionReplicaInfo: it is unclear where theServerName(which currently is missing theexternalHostnameandexternalPortfields in the patch) is being written for these objects and fixing this will probably require a further change to proto entities (i.e getting ever more invasive)Points 2. and 5. are the ones which make me doubt that this (patching HBase) is the correct/best approach.
[-]Make HBase UI endpoints resolvable[/-][+]Make HBase externally reachable[/+][-]Make HBase externally reachable[/-][+]Use Node-IPs for external reachability and localities[/+]