clef: documentation generator + docs (#19020)

* clef: implement documentation generation + remove unused struct

* clef: formatting + spelling

* clef: updates to doc

cmd/clef/README.md

@@ -16,7 +16,8 @@ Check out
 
 * the [tutorial](tutorial.md) for some concrete examples on how the signer works.
 * the [setup docs](docs/setup.md) for some information on how to configure it to work on QubesOS or USBArmory. 
-
+* the [data types](datatypes.md) for detailed information on the json types used in the communication between
+  clef and an external UI 
 
 ## Command line flags
 Clef accepts the following command line options:
@@ -24,25 +25,31 @@ Clef accepts the following command line options:
 COMMANDS:
    init    Initialize the signer, generate secret storage
    attest  Attest that a js-file is to be used
-   addpw   Store a credential for a keystore file
+   setpw   Store a credential for a keystore file
+   gendoc  Generate documentation about json-rpc format
    help    Shows a list of commands or help for one command
    
 GLOBAL OPTIONS:
    --loglevel value        log level to emit to the screen (default: 4)
    --keystore value        Directory for the keystore (default: "$HOME/.ethereum/keystore")
-   --configdir value       Directory for clef configuration (default: "$HOME/.clef")
-   --networkid value       Network identifier (integer, 1=Frontier, 2=Morden (disused), 3=Ropsten, 4=Rinkeby) (default: 1)
+   --configdir value       Directory for Clef configuration (default: "$HOME/.clef")
+   --chainid value         Chain id to use for signing (1=mainnet, 3=ropsten, 4=rinkeby, 5=Goerli) (default: 1)
    --lightkdf              Reduce key-derivation RAM & CPU usage at some expense of KDF strength
    --nousb                 Disables monitoring for and managing USB hardware wallets
    --rpcaddr value         HTTP-RPC server listening interface (default: "localhost")
+   --rpcvhosts value       Comma separated list of virtual hostnames from which to accept requests (server enforced). Accepts '*' wildcard. (default: "localhost")
+   --ipcdisable            Disable the IPC-RPC server
+   --ipcpath               Filename for IPC socket/pipe within the datadir (explicit paths escape it)
+   --rpc                   Enable the HTTP-RPC server
    --rpcport value         HTTP-RPC server listening port (default: 8550)
-   --signersecret value    A file containing the password used to encrypt signer credentials, e.g. keystore credentials and ruleset hash
+   --signersecret value    A file containing the (encrypted) master seed to encrypt Clef data, e.g. keystore credentials and ruleset hash
    --4bytedb value         File containing 4byte-identifiers (default: "./4byte.json")
    --4bytedb-custom value  File used for writing new 4byte-identifiers submitted via API (default: "./4byte-custom.json")
    --auditlog value        File used to emit audit logs. Set to "" to disable (default: "audit.log")
    --rules value           Enable rule-engine (default: "rules.json")
-   --stdio-ui              Use STDIN/STDOUT as a channel for an external UI. This means that an STDIN/STDOUT is used for RPC-communication with a e.g. a graphical user interface, and can be used when the signer is started by an external process.
-   --stdio-ui-test         Mechanism to test interface between signer and UI. Requires 'stdio-ui'.
+   --stdio-ui              Use STDIN/STDOUT as a channel for an external UI. This means that an STDIN/STDOUT is used for RPC-communication with a e.g. a graphical user interface, and can be used when Clef is started by an external process.
+   --stdio-ui-test         Mechanism to test interface between Clef and UI. Requires 'stdio-ui'.
+   --advanced              If enabled, issues warnings instead of rejections for suspicious requests. Default off
    --help, -h              show help
    --version, -v           print the version
    

cmd/clef/datatypes.md

+## UI Client interface
+
+These data types are defined in the channel between clef and the UI
+### SignDataRequest
+
+SignDataRequest contains information about a pending request to sign some data. The data to be signed can be of various types, defined by content-type. Clef has done most of the work in canonicalizing and making sense of the data, and it's up to the UI to presentthe user with the contents of the `message`
+
+Example:
+```json
+{
+  "content_type": "text/plain",
+  "address": "0xDEADbEeF000000000000000000000000DeaDbeEf",
+  "raw_data": "GUV0aGVyZXVtIFNpZ25lZCBNZXNzYWdlOgoxMWhlbGxvIHdvcmxk",
+  "message": [
+    {
+      "name": "message",
+      "value": "\u0019Ethereum Signed Message:\n11hello world",
+      "type": "text/plain"
+    }
+  ],
+  "hash": "0xd9eba16ed0ecae432b71fe008c98cc872bb4cc214d3220a36f365326cf807d68",
+  "meta": {
+    "remote": "localhost:9999",
+    "local": "localhost:8545",
+    "scheme": "http",
+    "User-Agent": "Firefox 3.2",
+    "Origin": "www.malicious.ru"
+  }
+}
+```
+### SignDataResponse - approve
+
+Response to SignDataRequest
+
+Example:
+```json
+{
+  "approved": true,
+  "Password": "apassword"
+}
+```
+### SignDataResponse - deny
+
+Response to SignDataRequest
+
+Example:
+```json
+{
+  "approved": false,
+  "Password": ""
+}
+```
+### SignTxRequest
+
+SignTxRequest contains information about a pending request to sign a transaction. Aside from the transaction itself, there is also a `call_info`-struct. That struct contains messages of various types, that the user should be informed of.
+
+As in any request, it's important to consider that the `meta` info also contains untrusted data.
+
+The `transaction` (on input into clef) can have either `data` or `input` -- if both are set, they must be identical, otherwise an error is generated. However, Clef will always use `data` when passing this struct on (if Clef does otherwise, please file a ticket)
+
+Example:
+```json
+{
+  "transaction": {
+    "from": "0xDEADbEeF000000000000000000000000DeaDbeEf",
+    "to": null,
+    "gas": "0x3e8",
+    "gasPrice": "0x5",
+    "value": "0x6",
+    "nonce": "0x1",
+    "data": "0x01020304"
+  },
+  "call_info": [
+    {
+      "type": "Warning",
+      "message": "Something looks odd, show this message as a warning"
+    },
+    {
+      "type": "Info",
+      "message": "User should see this aswell"
+    }
+  ],
+  "meta": {
+    "remote": "localhost:9999",
+    "local": "localhost:8545",
+    "scheme": "http",
+    "User-Agent": "Firefox 3.2",
+    "Origin": "www.malicious.ru"
+  }
+}
+```
+### SignDataResponse - approve
+
+Response to SignDataRequest. This response needs to contain the `transaction`, because the UI is free to make modifications to the transaction.
+
+Example:
+```json
+{
+  "transaction": {
+    "from": "0xDEADbEeF000000000000000000000000DeaDbeEf",
+    "to": null,
+    "gas": "0x3e8",
+    "gasPrice": "0x5",
+    "value": "0x6",
+    "nonce": "0x4",
+    "data": "0x04030201"
+  },
+  "approved": true,
+  "password": "apassword"
+}
+```
+### SignDataResponse - deny
+
+Response to SignDataRequest. When denying a request, there's no need to provide the transaction in return
+
+Example:
+```json
+{
+  "approved": false,
+  "Password": ""
+}
+```
+### OnApproved - SignTransactionResult
+
+SignTransactionResult is used in the call `clef` -> `OnApprovedTx(result)`
+
+This occurs _after_ successful completion of the entire signing procedure, but right before the signed transaction is passed to the external caller. This method (and data) can be used by the UI to signal to the user that the transaction was signed, but it is primarily useful for ruleset implementations.
+
+A ruleset that implements a rate limitation needs to know what transactions are sent out to the external interface. By hooking into this methods, the ruleset can maintain track of that count.
+
+**OBS:** Note that if an attacker can restore your `clef` data to a previous point in time (e.g through a backup), the attacker can reset such windows, even if he/she is unable to decrypt the content. 
+
+The `OnApproved` method cannot be responded to, it's purely informative
+
+Example:
+```json
+{
+  "raw": "0xf85d640101948a8eafb1cf62bfbeb1741769dae1a9dd47996192018026a0716bd90515acb1e68e5ac5867aa11a1e65399c3349d479f5fb698554ebc6f293a04e8a4ebfff434e971e0ef12c5bf3a881b06fd04fc3f8b8a7291fb67a26a1d4ed",
+  "tx": {
+    "nonce": "0x64",
+    "gasPrice": "0x1",
+    "gas": "0x1",
+    "to": "0x8a8eafb1cf62bfbeb1741769dae1a9dd47996192",
+    "value": "0x1",
+    "input": "0x",
+    "v": "0x26",
+    "r": "0x716bd90515acb1e68e5ac5867aa11a1e65399c3349d479f5fb698554ebc6f293",
+    "s": "0x4e8a4ebfff434e971e0ef12c5bf3a881b06fd04fc3f8b8a7291fb67a26a1d4ed",
+    "hash": "0x662f6d772692dd692f1b5e8baa77a9ff95bbd909362df3fc3d301aafebde5441"
+  }
+}
+```
+### UserInputRequest
+
+Sent when clef needs the user to provide data. If 'password' is true, the input field should be treated accordingly (echo-free)
+
+Example:
+```json
+{
+  "prompt": "The question to ask the user",
+  "title": "The title here",
+  "isPassword": true
+}
+```
+### UserInputResponse
+
+Response to SignDataRequest
+
+Example:
+```json
+{
+  "text": "The textual response from user"
+}
+```
+### ListRequest
+
+Sent when a request has been made to list addresses. The UI is provided with the full `account`s, including local directory names. Note: this information is not passed back to the external caller, who only sees the `address`es. 
+
+Example:
+```json
+{
+  "accounts": [
+    {
+      "address": "0xdeadbeef000000000000000000000000deadbeef",
+      "url": "keystore:///path/to/keyfile/a"
+    },
+    {
+      "address": "0x1111111122222222222233333333334444444444",
+      "url": "keystore:///path/to/keyfile/b"
+    }
+  ],
+  "meta": {
+    "remote": "localhost:9999",
+    "local": "localhost:8545",
+    "scheme": "http",
+    "User-Agent": "Firefox 3.2",
+    "Origin": "www.malicious.ru"
+  }
+}
+```
+### UserInputResponse
+
+Response to list request. The response contains a list of all addresses to show to the caller. Note: the UI is free to respond with any address the caller, regardless of whether it exists or not
+
+Example:
+```json
+{
+  "accounts": [
+    {
+      "address": "0x0000000000000000000000000000000000000000",
+      "url": ".. ignored .."
+    },
+    {
+      "address": "0xffffffffffffffffffffffffffffffffffffffff",
+      "url": ""
+    }
+  ]
+}
+```

cmd/clef/main.go

@@ -36,12 +36,15 @@ import (
 	"runtime"
 	"strings"
 
+	"github.com/ethereum/go-ethereum/accounts"
 	"github.com/ethereum/go-ethereum/accounts/keystore"
 	"github.com/ethereum/go-ethereum/cmd/utils"
 	"github.com/ethereum/go-ethereum/common"
+	"github.com/ethereum/go-ethereum/common/hexutil"
 	"github.com/ethereum/go-ethereum/console"
 	"github.com/ethereum/go-ethereum/core/types"
 	"github.com/ethereum/go-ethereum/crypto"
+	"github.com/ethereum/go-ethereum/internal/ethapi"
 	"github.com/ethereum/go-ethereum/log"
 	"github.com/ethereum/go-ethereum/node"
 	"github.com/ethereum/go-ethereum/params"
@@ -171,10 +174,16 @@ Clef that the file is 'safe' to execute.`,
 			signerSecretFlag,
 		},
 		Description: `
-		The setpw command stores a password for a given address (keyfile). If you enter a blank passphrase, it will 
+The setpw command stores a password for a given address (keyfile). If you enter a blank passphrase, it will 
 remove any stored credential for that address (keyfile)
-`,
-	}
+`}
+	gendocCommand = cli.Command{
+		Action: GenDoc,
+		Name:   "gendoc",
+		Usage:  "Generate documentation about json-rpc format",
+		Description: `
+The gendoc generates example structures of the json-rpc communication types.
+`}
 )
 
 func init() {
@@ -203,7 +212,7 @@ func init() {
 		advancedMode,
 	}
 	app.Action = signer
-	app.Commands = []cli.Command{initCommand, attestCommand, setCredentialCommand}
+	app.Commands = []cli.Command{initCommand, attestCommand, setCredentialCommand, gendocCommand}
 
 }
 func main() {
@@ -743,6 +752,154 @@ func decryptSeed(keyjson []byte, auth string) ([]byte, error) {
 	return seed, err
 }
 
+// GenDoc outputs examples of all structures used in json-rpc communication
+func GenDoc(ctx *cli.Context) {
+
+	var (
+		a    = common.HexToAddress("0xdeadbeef000000000000000000000000deadbeef")
+		b    = common.HexToAddress("0x1111111122222222222233333333334444444444")
+		meta = core.Metadata{
+			Scheme:    "http",
+			Local:     "localhost:8545",
+			Origin:    "www.malicious.ru",
+			Remote:    "localhost:9999",
+			UserAgent: "Firefox 3.2",
+		}
+		output []string
+		add    = func(name, desc string, v interface{}) {
+			if data, err := json.MarshalIndent(v, "", "  "); err == nil {
+				output = append(output, fmt.Sprintf("### %s\n\n%s\n\nExample:\n```json\n%s\n```", name, desc, data))
+			} else {
+				log.Error("Error generating output", err)
+			}
+		}
+	)
+
+	{ // Sign plain text request
+		desc := "SignDataRequest contains information about a pending request to sign some data. " +
+			"The data to be signed can be of various types, defined by content-type. Clef has done most " +
+			"of the work in canonicalizing and making sense of the data, and it's up to the UI to present" +
+			"the user with the contents of the `message`"
+		sighash, msg := accounts.TextAndHash([]byte("hello world"))
+		message := []*core.NameValueType{{"message", msg, accounts.MimetypeTextPlain}}
+
+		add("SignDataRequest", desc, &core.SignDataRequest{
+			Address:     common.NewMixedcaseAddress(a),
+			Meta:        meta,
+			ContentType: accounts.MimetypeTextPlain,
+			Rawdata:     []byte(msg),
+			Message:     message,
+			Hash:        sighash})
+	}
+	{ // Sign plain text response
+		add("SignDataResponse - approve", "Response to SignDataRequest",
+			&core.SignDataResponse{Password: "apassword", Approved: true})
+		add("SignDataResponse - deny", "Response to SignDataRequest",
+			&core.SignDataResponse{})
+	}
+	{ // Sign transaction request
+		desc := "SignTxRequest contains information about a pending request to sign a transaction. " +
+			"Aside from the transaction itself, there is also a `call_info`-struct. That struct contains " +
+			"messages of various types, that the user should be informed of." +
+			"\n\n" +
+			"As in any request, it's important to consider that the `meta` info also contains untrusted data." +
+			"\n\n" +
+			"The `transaction` (on input into clef) can have either `data` or `input` -- if both are set, " +
+			"they must be identical, otherwise an error is generated. " +
+			"However, Clef will always use `data` when passing this struct on (if Clef does otherwise, please file a ticket)"
+
+		data := hexutil.Bytes([]byte{0x01, 0x02, 0x03, 0x04})
+		add("SignTxRequest", desc, &core.SignTxRequest{
+			Meta: meta,
+			Callinfo: []core.ValidationInfo{
+				{"Warning", "Something looks odd, show this message as a warning"},
+				{"Info", "User should see this aswell"},
+			},
+			Transaction: core.SendTxArgs{
+				Data:     &data,
+				Nonce:    0x1,
+				Value:    hexutil.Big(*big.NewInt(6)),
+				From:     common.NewMixedcaseAddress(a),
+				To:       nil,
+				GasPrice: hexutil.Big(*big.NewInt(5)),
+				Gas:      1000,
+				Input:    nil,
+			}})
+	}
+	{ // Sign tx response
+		data := hexutil.Bytes([]byte{0x04, 0x03, 0x02, 0x01})
+		add("SignDataResponse - approve", "Response to SignDataRequest. This response needs to contain the `transaction`"+
+			", because the UI is free to make modifications to the transaction.",
+			&core.SignTxResponse{Password: "apassword", Approved: true,
+				Transaction: core.SendTxArgs{
+					Data:     &data,
+					Nonce:    0x4,
+					Value:    hexutil.Big(*big.NewInt(6)),
+					From:     common.NewMixedcaseAddress(a),
+					To:       nil,
+					GasPrice: hexutil.Big(*big.NewInt(5)),
+					Gas:      1000,
+					Input:    nil,
+				}})
+		add("SignDataResponse - deny", "Response to SignDataRequest. When denying a request, there's no need to "+
+			"provide the transaction in return",
+			&core.SignDataResponse{})
+	}
+	{ // WHen a signed tx is ready to go out
+		desc := "SignTransactionResult is used in the call `clef` -> `OnApprovedTx(result)`" +
+			"\n\n" +
+			"This occurs _after_ successful completion of the entire signing procedure, but right before the signed " +
+			"transaction is passed to the external caller. This method (and data) can be used by the UI to signal " +
+			"to the user that the transaction was signed, but it is primarily useful for ruleset implementations." +
+			"\n\n" +
+			"A ruleset that implements a rate limitation needs to know what transactions are sent out to the external " +
+			"interface. By hooking into this methods, the ruleset can maintain track of that count." +
+			"\n\n" +
+			"**OBS:** Note that if an attacker can restore your `clef` data to a previous point in time" +
+			" (e.g through a backup), the attacker can reset such windows, even if he/she is unable to decrypt the content. " +
+			"\n\n" +
+			"The `OnApproved` method cannot be responded to, it's purely informative"
+
+		rlpdata := common.FromHex("0xf85d640101948a8eafb1cf62bfbeb1741769dae1a9dd47996192018026a0716bd90515acb1e68e5ac5867aa11a1e65399c3349d479f5fb698554ebc6f293a04e8a4ebfff434e971e0ef12c5bf3a881b06fd04fc3f8b8a7291fb67a26a1d4ed")
+		var tx types.Transaction
+		rlp.DecodeBytes(rlpdata, &tx)
+		add("OnApproved - SignTransactionResult", desc, &ethapi.SignTransactionResult{Raw: rlpdata, Tx: &tx})
+
+	}
+	{ // User input
+		add("UserInputRequest", "Sent when clef needs the user to provide data. If 'password' is true, the input field should be treated accordingly (echo-free)",
+			&core.UserInputRequest{IsPassword: true, Title: "The title here", Prompt: "The question to ask the user"})
+		add("UserInputResponse", "Response to SignDataRequest",
+			&core.UserInputResponse{Text: "The textual response from user"})
+	}
+	{ // List request
+		add("ListRequest", "Sent when a request has been made to list addresses. The UI is provided with the "+
+			"full `account`s, including local directory names. Note: this information is not passed back to the external caller, "+
+			"who only sees the `address`es. ",
+			&core.ListRequest{
+				Meta: meta,
+				Accounts: []accounts.Account{
+					{a, accounts.URL{Scheme: "keystore", Path: "/path/to/keyfile/a"}},
+					{b, accounts.URL{Scheme: "keystore", Path: "/path/to/keyfile/b"}}},
+			})
+
+		add("UserInputResponse", "Response to list request. The response contains a list of all addresses to show to the caller. "+
+			"Note: the UI is free to respond with any address the caller, regardless of whether it exists or not",
+			&core.ListResponse{
+				Accounts: []accounts.Account{
+					{common.HexToAddress("0xcowbeef000000cowbeef00000000000000000c0w"), accounts.URL{Path: ".. ignored .."}},
+					{common.HexToAddress("0xffffffffffffffffffffffffffffffffffffffff"), accounts.URL{}},
+				}})
+	}
+
+	fmt.Println(`## UI Client interface
+
+These data types are defined in the channel between clef and the UI`)
+	for _, elem := range output {
+		fmt.Println(elem)
+	}
+}
+
 /**
 //Create Account
 

signer/core/api.go

@@ -236,12 +236,6 @@ type (
 	Message struct {
 		Text string `json:"text"`
 	}
-	PasswordRequest struct {
-		Prompt string `json:"prompt"`
-	}
-	PasswordResponse struct {
-		Password string `json:"password"`
-	}
 	StartupInfo struct {
 		Info map[string]interface{} `json:"info"`
 	}

signer/core/signed_data.go

@@ -250,7 +250,7 @@ func (api *SignerAPI) determineSignatureFormat(ctx context.Context, contentType
 				message := []*NameValueType{
 					{
 						Name:  "message",
-						Typ:   "text/plain",
+						Typ:   accounts.MimetypeTextPlain,
 						Value: msg,
 					},
 				}

signer/core/types.go

@@ -75,7 +75,7 @@ type SendTxArgs struct {
 	Nonce    hexutil.Uint64           `json:"nonce"`
 	// We accept "data" and "input" for backwards-compatibility reasons.
 	Data  *hexutil.Bytes `json:"data"`
-	Input *hexutil.Bytes `json:"input"`
+	Input *hexutil.Bytes `json:"input,omitempty"`
 }
 
 func (args SendTxArgs) String() string {

signer/rules/rules_test.go

@@ -83,10 +83,6 @@ func (alwaysDenyUI) RegisterUIServer(api *core.UIServerAPI) {
 func (alwaysDenyUI) OnSignerStartup(info core.StartupInfo) {
 }
 
-func (alwaysDenyUI) OnMasterPassword(request *core.PasswordRequest) (core.PasswordResponse, error) {
-	return core.PasswordResponse{}, nil
-}
-
 func (alwaysDenyUI) ApproveTx(request *core.SignTxRequest) (core.SignTxResponse, error) {
 	return core.SignTxResponse{Transaction: request.Transaction, Approved: false, Password: ""}, nil
 }
@@ -261,10 +257,6 @@ func (d *dummyUI) OnApprovedTx(tx ethapi.SignTransactionResult) {
 	d.calls = append(d.calls, "OnApprovedTx")
 }
 
-func (d *dummyUI) OnMasterPassword(request *core.PasswordRequest) (core.PasswordResponse, error) {
-	return core.PasswordResponse{}, nil
-}
-
 func (d *dummyUI) OnSignerStartup(info core.StartupInfo) {
 }
 
@@ -543,10 +535,6 @@ func (d *dontCallMe) RegisterUIServer(api *core.UIServerAPI) {
 func (d *dontCallMe) OnSignerStartup(info core.StartupInfo) {
 }
 
-func (d *dontCallMe) OnMasterPassword(request *core.PasswordRequest) (core.PasswordResponse, error) {
-	return core.PasswordResponse{}, nil
-}
-
 func (d *dontCallMe) ApproveTx(request *core.SignTxRequest) (core.SignTxResponse, error) {
 	d.t.Fatalf("Did not expect next-handler to be called")
 	return core.SignTxResponse{}, core.ErrRequestDenied