RPC Internals
Home |
---|
Contents[edit | edit source]
Overview[edit | edit source]
Gluster uses a transport independent RPC mechanism for communications between cluster processes. Within Gluster the two primary consumers of RPC services are:
- glusterd : used for communications between glusterd management processes.
- protocol/client and protocol/server translators. This translator pair is inserted in a translator stack to enable a a translator stack to span multiple nodes, such as when a brick is served remotely. The protocol/client and protocol/server translators implement the full translator API, including all allowable FOPS, enabling it to be transparently inserved within a volspec.
RPC Usage[edit | edit source]
Below RPC usage table is based on references to rpc_clnt_new() ` and `rpcsvc_init()
Module | Client | Server |
---|---|---|
api | Yes | No |
cli | Yes | No |
glusterfsd | Yes | Yes |
xlators/mgmt/glusterd | Yes | Yes |
xlators/nfs/server | Yes | Yes |
xlators/protocol/Client | Yes | No |
xlators/protocol/server | No | Yes |
protocol/client Translator[edit | edit source]
Client Translator volume specification[edit | edit source]
volume my_vol-client-0 type protocol/client option remote-host 192.168.122.125 option remote-subvolume /glusterdata/brick125 option transport-type tcp end-volume
protocol/client Internals[edit | edit source]
Source may be found at: xtalors/protocol/client
protocol/server Translator[edit | edit source]
Server Translator volume specification[edit | edit source]
volume my_vol-server type protocol/server option transport-type tcp option auth.login./glusterdata/brick125.allow 79431724-6ea3-482f-8d2e-7654d8934d83 option auth.login.79431724-6ea3-482f-8d2e-7654d8934d83.password be6b8ca2-89fb-4223-b521-d976e3c5f6cd option auth.addr./glusterdata/brick125.allow * subvolumes /glusterdata/brick125 end-volume
protocol/server Internals[edit | edit source]
Source may be found at: xlators/protocol/server
Auth[edit | edit source]
Authenticated Volspecs[edit | edit source]
Example of client-side authenticated connection:
volume my_vol-client-0 type protocol/client option remote-host 192.168.122.125 option remote-subvolume /glusterdata/brick125 option transport-type tcp option username 79431724-6ea3-482f-8d2e-7654d8934d83 option password be6b8ca2-89fb-4223-b521-d976e3c5f6cd end-volume
Example of server-side authenticated connection:
volume my_vol-server type protocol/server option transport-type tcp option auth.login./glusterdata/brick125.allow 79431724-6ea3-482f-8d2e-7654d8934d83 option auth.login.79431724-6ea3-482f-8d2e-7654d8934d83.password be6b8ca2-89fb-4223-b521-d976e3c5f6cd option auth.addr./glusterdata/brick125.allow * subvolumes /glusterdata/brick125 end-volume
Auth Internals[edit | edit source]
Source may be found at: `xlators/protocol/auth<code> `
glusterd use of RPC[edit | edit source]
RPC calls within glusterd are localized to the <code>gllusterd-rpc-p[s.c file, which includes references to shared RPC via the collowing includes:
- RPC:
rpc-clnt.h
andprotocol-common.h
- Wire encoding:
glusterd1-xhr.h
,cli1-xdr.h
andxdr-generic.h
.
RPC related volume options for glusterd:
struct volume_options options[] = { ... { .key = {"transport-type"}, .type = GF_OPTION_TYPE_ANY, }, { .key = {"transport.*"}, .type = GF_OPTION_TYPE_ANY, }, { .key = {"rpc-auth.*"}, .type = GF_OPTION_TYPE_ANY, }, { .key = {"rpc-auth-allow-insecure"}, .type = GF_OPTION_TYPE_BOOL, }, ... };
RPC initialization snippet in glusterd.c init() code:
ret = rpcsvc_register_notify (rpc, glusterd_rpcsvc_notify, this); ... ret = rpcsvc_create_listeners (rpc, this->options, this->name); ... ret = glusterd_program_register (this, rpc, &gd_svc_peer_prog); ... ret = glusterd_program_register (this, rpc, &gd_svc_cli_prog); ... ret = glusterd_program_register (this, rpc, &gd_svc_mgmt_prog); ... ret = glusterd_program_register (this, rpc, &gluster_pmap_prog); ... ret = glusterd_program_register (this, rpc, &gluster_handshake_prog); ...
Background on functions called:
rpcsvc_register_notify (rpc, glusterd_rpcsvc_notify, this)
: call to core RPC notify registration routine located in rpc/rpc-lib/src/rpcsvc.crpcsvc_create_listeners (rpc, this->options, this->name)
: call to core RPC listener registration routine located in rpc/rpc-lib/src/rpcsvc.cglusterd_program_register (this, rpc, &XXX_prog)
: calls core RPC program registration routine (rpcsvc_program_register
) located in rpc/rpc-lib/src/rpcsvc.c
Specific programs registered:
gd_svc_peer_prog
gd_svc_cli_prog
gd_svc_mgmt_prog
gluster_pmap_prog
gluster_handshake_prog
sample glusterd volspec[edit | edit source]
Sample volspec file for glusterd including reference to rpc. Can be found at `/etc/glusterfs/glusterd.vol `
volume management type mgmt/glusterd option working-directory /var/lib/glusterd option transport-type socket,rdma option transport.socket.keepalive-time 10 option transport.socket.keepalive-interval 2 option transport.socket.read-fail-log off end-volume
Other functions called:
- rpc_clnt_transport_unix_options_build (&options, socketpath);
- glusterd_submit_request (peerinfo→rpc, &req, frame, peerinfo→peer, GLUSTERD_PROBE_QUERY, NULL, this, glusterd_probe_cbk, (xdrproc_t)xdr_gd1_mgmt_probe_req);
- rpc_clnt_submit (rpc, prog, procnum, cbkfn, &iov, count, NULL, 0, iobref, frame, NULL, 0, NULL, 0, NULL);
- glusterd_submit_reply (req, cli_rsp, NULL, 0, NULL, xdrproc);
- glusterd_serialize_reply (req, arg, &rsp, xdrproc);
- rpcsvc_submit_generic (req, &rsp, 1, payload, payloadcount, iobref);
- xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gd1_mgmt_probe_rsp);
Libraries[edit | edit source]
rpc-lib[edit | edit source]
rpc-clnt.h[edit | edit source]
struct rpc_clnt *rpc_clnt_new (dict_t *options, glusterfs_ctx_t *ctx, char *name, uint32_t reqpool_size); int rpc_clnt_start (struct rpc_clnt *rpc); int rpc_clnt_register_notify (struct rpc_clnt *rpc, rpc_clnt_notify_t fn, void *mydata); int rpc_clnt_submit (struct rpc_clnt *rpc, rpc_clnt_prog_t *prog, int procnum, fop_cbk_fn_t cbkfn, struct iovec *proghdr, int proghdrcount, struct iovec *progpayload, int progpayloadcount, struct iobref *iobref, void *frame, struct iovec *rsphdr, int rsphdr_count, struct iovec *rsp_payload, int rsp_payload_count, struct iobref *rsp_iobref); struct rpc_clnt * rpc_clnt_ref (struct rpc_clnt *rpc); struct rpc_clnt * rpc_clnt_unref (struct rpc_clnt *rpc); int rpc_clnt_connection_cleanup (rpc_clnt_connection_t *conn); void rpc_clnt_set_connected (rpc_clnt_connection_t *conn); void rpc_clnt_unset_connected (rpc_clnt_connection_t *conn); void rpc_clnt_reconnect (void *trans_ptr); void rpc_clnt_reconfig (struct rpc_clnt *rpc, struct rpc_clnt_config *config); int rpcclnt_cbk_program_register (struct rpc_clnt *svc, rpcclnt_cb_program_t *program, void *mydata); int rpc_clnt_transport_unix_options_build (dict_t **options, char *filepath); void rpc_clnt_disable (struct rpc_clnt *rpc);
rpc-svc.h[edit | edit source]
extern int rpcsvc_program_register (rpcsvc_t *svc, rpcsvc_program_t *program); extern int rpcsvc_program_unregister (rpcsvc_t *svc, rpcsvc_program_t *program); extern int32_t rpcsvc_create_listeners (rpcsvc_t *svc, dict_t *options, char *name); void rpcsvc_listener_destroy (rpcsvc_listener_t *listener); extern int rpcsvc_program_register_portmap (rpcsvc_program_t *newprog, uint32_t port); extern int rpcsvc_register_portmap_enabled (rpcsvc_t *svc); extern rpcsvc_t * rpcsvc_init (xlator_t *xl, glusterfs_ctx_t *ctx, dict_t *options, uint32_t poolcount); int rpcsvc_register_notify (rpcsvc_t *svc, rpcsvc_notify_t notify, void *mydata); int rpcsvc_unregister_notify (rpcsvc_t *svc, rpcsvc_notify_t notify, void *mydata); int rpcsvc_submit_message (rpcsvc_request_t *req, struct iovec *proghdr, int hdrcount, struct iovec *payload, int payloadcount, struct iobref *iobref); int rpcsvc_submit_generic (rpcsvc_request_t *req, struct iovec *proghdr, int hdrcount, struct iovec *payload, int payloadcount, struct iobref *iobref); extern int rpcsvc_error_reply (rpcsvc_request_t *req); extern int rpcsvc_transport_peername (rpc_transport_t *trans, char *hostname, int hostlen); extern inline int rpcsvc_transport_peeraddr (rpc_transport_t *trans, char *addrstr, int addrlen, struct sockaddr_storage *returnsa, socklen_t sasize); extern int rpcsvc_transport_peer_check (dict_t *options, char *volname, rpc_transport_t *trans); extern int rpcsvc_transport_privport_check (rpcsvc_t *svc, char *volname, rpc_transport_t *trans); extern int rpcsvc_submit_vectors (rpcsvc_request_t *req); extern int rpcsvc_request_attach_vector (rpcsvc_request_t *req, struct iovec msgvec, struct iobuf *iob, struct iobref *ioref, int finalvector); extern int rpcsvc_auth_request_init (rpcsvc_request_t *req); extern int rpcsvc_auth_init (rpcsvc_t *svc, dict_t *options); extern int rpcsvc_auth_transport_init (rpc_transport_t *xprt); extern int rpcsvc_authenticate (rpcsvc_request_t *req); extern int rpcsvc_auth_array (rpcsvc_t *svc, char *volname, int *autharr, int arrlen); extern gid_t * rpcsvc_auth_unix_auxgids (rpcsvc_request_t *req, int *arrlen); extern int rpcsvc_combine_gen_spec_volume_checks (int gen, int spec); extern char * rpcsvc_volume_allowed (dict_t *options, char *volname); int rpcsvc_callback_submit (rpcsvc_t *rpc, rpc_transport_t *trans, rpcsvc_cbk_program_t *prog, int procnum, struct iovec *proghdr, int proghdrcount); int rpcsvc_transport_unix_options_build (dict_t **options, char *filepath); int rpcsvc_set_allow_insecure (rpcsvc_t *svc, dict_t *options); int rpcsvc_auth_array (rpcsvc_t *svc, char *volname, int *autharr, int arrlen); char * rpcsvc_volume_allowed (dict_t *options, char *volname); rpcsvc_vector_sizer rpcsvc_get_program_vector_sizer (rpcsvc_t *svc, uint32_t prognum, uint32_t progver, uint32_t procnum);
rpc-transport[edit | edit source]
socket[edit | edit source]
rdma[edit | edit source]
xdr[edit | edit source]
Home |
---|