Services and State with Ambari REST API

  • 时间:
  • 浏览:0

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X DELETE

"Body":{"HostRoles":{"state":"INSTALLED"}}}' 

Storm REST ServerSTORM_REST_APISTORM1

// Delete service

18

INIT <-UNINSTALLED, INIT, WIPING_OUT

In this post we are going to look a little closer at the way the Ambari API can be used to manage Hadoop services. At the end of this post you will find a list of all the currently supported Hadoop services with all the needed master, slave and client components that can be manged and administrated within your HDP stack. Also this posts contains the possible states and state transitions a component might have which could become useful when facing problems like Host config is in invalid state.

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/YARN_CLIENT

Oozie ClientOOZIE_CLIENTOOZIE1+

-d '{"ServiceComponentInfo": {"state": "INSTALLED"}}'

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/hosts

Sqoop ClientSQOOPSQOOP1+

ZookeeperZOOKEEPER_SERVERZOOKEEPER1+ (odd #)

HBase ClientHBASE_CLIENTHBASE1+

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/hosts

Ganglia ServerGANGLIA_SERVERGANGLIA1

Mahout ClientMAHOUTMAHOUT0+

DISABLEDDisabled master’s backup state.

Zookeeper ClientZOOKEEPER_CLIENTZOOKEEPER1+

A Hadoop cluster contains of multiple services that individually have either master, slave or client components. Below you will find a list of all currently supported components as part of a HDP stack divided in master, slave, or client groups. The cardinality notates the number of components that can exist at the same time in one cluster.

"Body":{"HostRoles": {"state""INSTALLED"}}}' 

// Get service info

// Add component to host

/<NEW_HOST_FQDN>/host_components/OOZIE_SERVER

8

25

HiveServer2HIVE_SERVERHIVE1-2

Spark Job History ServerSPARK_JOBHISTORYSERVERSPARK1

Service

2

6

20

6

UPGRADINGIn the process of upgrading the deployed bits.

HDFS ClientHDFS_CLIENTHDFS1+

1

23

3

"Body":{"HostRoles": {"state""INSTALLED"}}}' 

7

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/services/KNOX/

8

NameNodeNAMENODEHDFS1-2

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST -d 

Oozie ServerOOZIE_SERVEROOZIE1

curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X GET

4

// Stop the old component by putting it back in state INSTALLED

3

Nimbus ServerNIMBUSSTORM1

StateDescription

1

17

/<NEW_HOST_FQDN>/host_components/OOZIE_SERVER

10

2

Ranger Admin ServerRANGER_ADMINRANGER1-3

Desired StateStart State

MapReduce ClientMAPREDUCE2_CLIENTMAPREDUCE21+

5

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X PUT 

-d '{"RequestInfo": {"context": "Install Oozie","query":"HostRoles/component_name.in('OOZIE_SERVER')"}, 

HBase RegionServerHBASE_REGIONSERVERHBASE1+

Ambari Metrics ServiceMETRICS_COLLECTORAMS1

INSTALLEDState when install completed successfully.

STOPPINGIn the process of stopping.

YARN ClientYARN_CLIENTYARN1+

Hive ClientHIVE_CLIENTHIVE1+

Falcon ServerFALCON_SERVERFALCON1

Secondary NameNodeNFS_GATEWAYHDFS0+

12

1

Among the need to change the state of a service component a very common issue is the need to actually move one service – most likely a master service – to different host. Users faced with this problems often need to issue the existing API due to a lack of support by the existing UI – at least with older releases of Ambari Web UI. This is an example of moving the OOZIE_SERVER to a new host.

11

13

The Ambari management tool for Hadoop offers among other handy tools a comprehensive REST API for cluster administration. Logically a cluster is divided into hosts, services and service components. While the UI might not always has support for all needed scenarios sure the REST API can be used to achieve it. For example moving a master component of a service from one host to another.

// Delete old component

5

STARTEDState when start completed successfully.

12

Node ManagerNODEMANAGERYARN1+

Zookeeper Failover ServiceZKFCHDFS0+

16

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/hosts

21

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/services/KNOX

NameAmbari Component Name

// Install component by achieving stated INSTALLED

STARTED <-INSTALLED, STARTING, STARTED

-d '{"RequestInfo": {"context": "Stop Oozie","query":"HostRoles/component_name.in('OOZIE_SERVER')"}, 

Cardinality

2

-d '{"ServiceInfo": {"state": "INSTALLED"}}'

Journale Nodes for NameNode HAJOURNALNODEHDFS0+ (odd #)

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X DELETE

Nagios ServerNAGIOS_SERVERNAGIOS1

-d '{"RequestInfo":{"context":"Install YARN_CLIENT"},

// Delete component

15

8

// Getting the state of the component

UNINSTALLINGIn the process of uninstalling.

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/YARN_CLIENT 

7

// Setting the state of the component to INSTALLED

4

DISABLED <-INSTALLED, INSTALL_FAILED, UNKNOWN

26

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT 

11

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/services/KNOX

Removing a complete service from the cluster would require all the removal of all it’s components together with the service itself. For example for removing the Knox service we would need to remove the KNOX_GATEWAY component from the cluster, but before let’s get their state:

-d '{"RequestInfo": {"context": "Start Oozie","query":"HostRoles/component_name.in('OOZIE_SERVER')"}, 

UNKNOWNState could not be determined.

Spark ClientSPARK_CLIENTSPARK1+

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/services/KNOX/components/KNOX_GATEWAY

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/hosts

3

NameAmbari Component Name

ServiceCardinality

DRPC ServerDRPC_SERVERSTORM1

14

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X GET

10

15

HistoryServerHISTORYSERVERMAPREDUCE2 1

WebHcat ServerWEBHCAT_SERVERHIVE1

Falcon ClientFALCON_CLIENTFALCON1+

"Body":{"HostRoles": {"state""STARTED"}}}' 

13

16

STARTINGIn the process of starting.

/<OLD_HOST_FQDN>/host_components/OOZIE_SERVER

7

9

Kafka BrokerKAFKA_BROKERKAFKA1+

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/hosts

NameAmbari Component Name

curl -u admin:admin -H "X-Requested-By:ambari" -i -X DELETE 

Ganglia Metrics CollectorGANGLIA_MONITORGANGLIAALL

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X PUT 

// Stop component KNOX_GATEWAY

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT 

INSTALL_FAILEDInstall failed.

Possible states of a component:

9

Secondary NameNodeSECONDARY_NAMENODEHDFS1

UNINSTALLEDState when uninstall completed successfully.

/<NEW_HOST_FQDN>/host_components/OOZIE_SERVER

DataNodeDATANODEHDFS1+

The possible states a service component can be in is listed below. To change or get the state of a service component the following API calls can be issued.

// Starting the new component

Ranger User SyncRANGER_USERSYNCRANGER1

/<OLD_HOST_FQDN>/host_components/OOZIE_SERVER

4

HBase MasterHBASE_MASTERHBASE1+

Application Timeline ServerAPP_TIMELINE_SERVERYARN1

5

INSTALLINGIn the process of installing.

INITInitial/Clean state.

Allowed State transitions of a component:

Ambari Metrics CollectorMETRICS_MONITORAMSALL

22

UNINSTALLED <-INSTALLED, UNINSTALLED, UNINSTALLING

http://<HOST>:500500/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/KNOX_GATEWAY

19

24

6

HCat ClientHCATHIVE1+

ResourceMangerRESOURCEMANAGERYARN1-2

// Stop service KNOX

Ranger Key Management ServerRANGER_KMS_SERVERRANGER_KMS1+

Zeppelin ServerZEPPELIN_MASTERSPARK / HIVE1

17

Knox GatewayKNOX_GATEWAYKNOX1+

Phoneix Query ServerPHOENIX_QUERY_SERVERHBASE0+

Storm UISTORM_UI_SERVERSTORM1

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT 

Hive MetastoreHIVE_METASTOREHIVE1-2

Storm SupervisorSUPERVISORSTORM1+

WIPING_OUTIn the process of wiping out the install.

INSTALLED <-INIT, UNINSTALLED, INSTALLED, INSTALLING, STARTED, INSTALL_FAILED, UPGRADING, STOPPING, UNKNOWN, DISABLED

ServiceCardinality

curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X PUT 

14