# OpenWare OPEX

#OPEX installation and administration

This document aims to cover the OpenWare Platform EXchange (OPEX) installation and administration.

#Installation

# Prerequisites

Please install the following tools and read the documentations before starting.

Terraform

Please make sure to use Terraform 0.11 due to breaking changes in version 0.12

• What is Terraform

• Getting started with Terraform

• Terraform binaries

AWS Command Line Interface

• Install AWS CLI
• Configure AWS CLI

Kubernetes

• What is Kubernetes?
• Understanding Kubernetes objects
• Cluster management using kubectl

Helm

• Using Helm

Ruby

• apt (Debian or Ubuntu)

$ sudo apt-get install ruby-full

• yum (CentOS, Fedora, or RHEL)

$ sudo yum install ruby

# Create a base platform repository

#Install the Kite DevOps tool

Make sure you have a fresh Ruby version installed and run:

$ gem install kite

#Create your Infrastructure as Code base repository

Create a new kite project using:

$ kite new PROJECT_NAME

#Change configuration for cloud provider

Make sure to delete GCP block if you will be using AWS or vice versa

Please note that regions and zones differ for AWS and GCP

 default: &default
   # Note that region formats differ between AWS, GCP and AliCloud
   region:             "europe-west4"   # always change this for AWS
   zone:               "europe-west4-c" # always change this for AWS
   gcp:                                 # delete this block if you're using AWS
     project:          "helios-devel"
     service_account:  "~/safe/helios-devel.json" # if you want to use OPEX workspace this file should be located in the root of the kite project
     bucket_name:      "kite-state-helios-devel"
   aws:                                 # delete this block if you're using GCP
     access_key:       "********************" # make sure to put credentials here
     secret_key:       "********************" # make sure to put credentials here
     bucket_name:      "kite-state-helios-devel"
 ~

#Generate an environment(e.g. development/test/production)

Kite environments are separated workspaces with their own credentials, variables and modules.

Generate an environment

$ kite generate environment *env_name* --provider=aws|gcp

If you want to change the credentials for an environment, edit config/cloud.yml and regenerate environment with the same command.

Now the environment should be generated at config/environments/*env_name*

#Add a module to your environment

To add a Kite module to your environment, you should first fetch them:

  kite mod fetch https://url.for/your/module --env *env_name*

This should clone module's source files into modules/*module_name*.

Then you need to render modules:

  kite mod vars https://url.for/your/module --env *env_name*

This will create a vars.*module_name*.yml file with all variables needed by the module. If you add kite-opex modules your default configuration will look similar to this:

---
# Infrastructure configuration(ignore when using baremetal)
terraform:

  node_type:     "m4.large"
  cluster_name:  "test"
  max_nodes:     "5"
  vpc_name:      "peatio-platform"
  subnet_name:   "peatio-rds-subnet"
  vpc_cidr:      "10.0.0.0/16"
  subnet_cidr_1: "10.0.10.0/26"
  subnet_cidr_2: "10.0.20.0/26"
  az_1_name:          "eu-central-1a" # make sure that availability zones are within the region
  az_2_name:          "eu-central-1b" # make sure that availability zones are within the region
  registry_name: "test"

  rbac: true
profile: standard # small, standard and large available
email: [email protected]
infrastructure:
  load_balancer_ip: ''
# Sendgrid configuration
smtp:
  sender_name: "Kayen Admin"
  host: "smtp.sendgrid.net"
  port: "2525"
  user: "apikey"
  password: "*smtp_password*"
# Bitgo proxy configuration
bitgo:
  enabled: true
  environment: test
# Change tags if you want to use different imgaes
dependencies:
  redis: 1.0.2
  rabbitmq: 0.8.1
  gcloud-sqlproxy: 0.3.3
  kube-lego: 0.4.0
  nginx-ingress: 0.14.1
  peatio: 0.1.1
  barong: 0.2.3
  vault: 0.12.0
  enterprise-front: 0.1.1
tags:
  barong: 2.0.4-alpha
  coinhub: 0.2.4
  peatio: 2.0.2-alpha
  vault: 0.10.1
peatio:
  titleMarkets: KAYEN
  host: api.2-0-alpha.kayen.org
  daemons:
    blockchain: 1
    global_state: 1
    k: 1
    payment_transaction: 1
    withdraw_audit: 1
  amqpDaemons:
    deposit_collection: 1
    deposit_collection_fees: 1
    deposit_coin_address: 1
    slave_book: 1
    market_ticker: 1
    matching: 1
    order_processor: 1
    pusher_market: 1
    pusher_member: 1
    trade_executor: 1
    withdraw_coin: 1
  env:
    admin: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
# Here you can configure withdraw limits and fees
  fiats:
  - code: usd
    symbol: "$"
    min_deposit_amount: 0
    min_collection_amount: 0
    withdraw_limit_24h: 100
    withdraw_limit_72h: 200
    deposit_fee: 0
    withdraw_fee: 0
  coins:
  - code: btc
    symbol: "฿"
    base_factor: 100000000
    blockchain_key: btc-testnet
    precision: 8
    min_deposit_amount: 0
    min_collection_amount: 0
    withdraw_limit_24h: 100
    withdraw_limit_72h: 200
    deposit_fee: 0
    withdraw_fee: 0
  markets:
  - base:
      code: btc
      precision: 8
    quote:
      code: usd
      precision: 8
# Change this to use custom frontend
front:
  enabled: true
  host: www.kayen.io
  repository: gcr.io/front/frontend
  tag: latest
  authUrl: https://api.test.kayen.io/auth/barong
  blockchainInfoUrl: https://testnet.blockchain.info/tx/
  tradePageUrl: https://api.test.kayen.io/trading/btcusd
# Change this to use custom trading-ui
frontend:
  enabled: false
  host: trading.kayen.io
  repository: gcr.io/openware-production/frontend
  tag: latest
# Change this to configure twilio
twilio:
  account_sid: ACd944e3b2e1a4c5e37bfa8d9f11a245b6
  auth_token: 694e60aff87c5d1a66c272a62f931aab
  phone_number: "+18556312248"
# Change this to configure cryptonodes deployment
cryptonodes:
  # THIS CONFIGURATION IS USED BOTH FOR PACKER AND K8S CRYPTONODES DEPLOYMENT
  ethereum:
    enabled:         true      # Set to false if you don't want it to be deployed
    client:          "geth"    # Available options: geth, parity.
    network:         "rinkeby" # Available options: rinkeby, kovan, mainnet.
    disk_size:       "30"
    hostname:        "eth.helioscloud.com"
    rpcport:         "8545"
    port:            "30303"
    whitelisted_ips: ["0.0.0.0/0"] # Set this to empty array [] to whitelist just cluster IP

  bitcoind:
    enabled:         true      # Set to false if you don't want it to be deployed
    network:         "testnet"
    disk_size:       "30"
    hostname:        "btc.helioscloud.com"
    rpcuser:         "admin"
    rpcpassword:     "changeme"
    rpcport:         "18332"
    port:            "18333"
    whitelisted_ips: ["0.0.0.0/0"] # Set this to empty array [] to whitelist just cluster IP

  bitcoincash:
    enabled:         true      # Set to false if you don't want it to be deployed
    network:         "testnet"
    disk_size:       "30"
    hostname:        "bch.helioscloud.com"
    rpcuser:         "admin"
    rpcpassword:     "changeme"
    rpcport:         "18332"
    port:            "18333"
    whitelisted_ips: ["0.0.0.0/0"] # Set this to empty array [] to whitelist just cluster IP

  litecoind:
    enabled:         true      # Set to false if you don't want it to be deployed
    network:         "testnet"
    disk_size:       "30"
    hostname:        "ltc.helioscloud.com"
    rpcuser:         "admin"
    rpcpassword:     "changeme"
    rpcport:         "19332"
    port:            "19335"
    whitelisted_ips: ["0.0.0.0/0"]

  dashd:
    enabled:         true      # Set to false if you don't want it to be deployed
    network:         "testnet" # set to mainnet, testnet or regtest
    disk_size:       "30"
    hostname:        "dash.helioscloud.com"
    rpcuser:         "admin"
    rpcpassword:     "changeme"
    rpcport:         "19998"
    port:            "19999"
    whitelisted_ips: ["0.0.0.0/0"] # Set this to empty array [] to whitelist just cluster IP

# Change this to configure scout
scout:
  enabled: false
  key: changeme
  app: 2-0-alpha.kayen.org
  logLevel: warn
  env: production

Fill in vars.*module_name*.yml with correct values and render the module:

Note: the profile field determines the scale of your deployment, you can choose from small, standard and large depending on your load expectations

  kite mod render modules/*module_name* --env *env_name*

#[Optional] Build and run the OPEX Workspace

OPEX Workspace is a Docker image built from your local kite platform and tailor-made for all the deployment operations. It contains all the necessary tools and is ready to use out of the box.

To bring the workspace up and build it if needed, simply run the bin/workspace_up.sh script.

WARNING Please make sure that you service account json file is located in the root of your kite project, before building the docker image. Also in config/cloud.yml the path should be considering kite project root folder. e.g. google-sa.json, instead of ~/safe/google-sa.json

The built image an be accessed by the opex-workspace:*env_name*-*version* tag, which is generated using the VERSION file in the root of the platform.

#Apply Terraform configuration from the environment

Note: if you'd like to use a baremetal environment as your infrastructure, follow this document

Set your default gcloud credentials using

  aws configure

  kite terraform apply --env *env_name*

# Configure and apply the kite-opex module

#Step-By-Step Deployment

#0. Enable shell autocompletion

To enable shell autocompletion for the OPEX Ruby binary, simply add:

source <($PWD/bin/opex complete *bash or zsh*)

to your bashrc or zshrc

#1. Infrastructure

Apply Terraform configuration from the environment first

Set your default gcloud credentials using

  gcloud auth application-default login

To bring up infrastructure with Helm, Ingress and Kube Lego

./bin/opex infra install all

You can also install components separately, to see available options, type:

./bin/opex infra install

If you want to use an existing platform created by your own you need to dump infrastructure information in the file config/environments/tf-outputs.yml according to those specifications:

Here is an example of output format:

cluster_name: "test"
instance_connection_name: "helios-devel:europe-west4:master-1337"
...

#2. App Dependencies

Next you should deploy all the Peatio and Barong dependencies (Redis, RabbitMQ, SQL proxy, BitGo proxy, Bitcoind, Parity) Most components have their respective configs files located at config/environments/*env_name*/

You can either edit vars.*module_name*.yml file or edit a each file for a corresponding component respectively

Then install all components with the following command:

./bin/opex deps install all # This doesn't include Bitcoind and Parity

To check if everything is installed properly, run:

$ kubectl -n <deployment_id>-backend get pods
NAME                                      READY     STATUS    RESTARTS   AGE
gcloud-sqlproxy-xxxxx-xxx                 1/1       Running   0          1m
kube-lego-xxxxx-xxx                       1/1       Running   0          1m
nginx-ingress-controller-xxxxx-xxx        1/1       Running   0          1m
nginx-ingress-default-backend-xxxxx-xxx   1/1       Running   0          1m
rabbitmq-xxxxx-xxx                        1/1       Running   0          1m
redis-xxxxx-xxx                           1/1       Running   0          1m

Main dependency of Barong is Vault, so first check if the Vault configuration in config/environments/*env_name*/vault.yml is correct.

For a more detailed view on Vault usage and deployment , check its dedicated documentation.

Main dependency of Peatio is Cryptonodes, so first check if Cryptonodes configuration in config/environments/*env_name*/(bitcoind,parity).yml is correct. A more detailed instruction on cryptonodes deployment can be found here

#3.x Application Deployments

In case you'd like to deploy the whole application stack(i.e. initial OPEX deployment), simply run ./bin/opex app install all

The following chapters will cover each application's configs, deployment and administration.

#3. Barong Server

Configure your Barong deployment in config/environments/*env_name*/barong.yml:

• Twilio configuration

twilio:
    accountSid:  "ACd944e3b2e1a4c5e37bfa8d9f11a245b6"
    authToken:   "694e60aff87c5d1a66c272a62f931aab"
    phoneNumber: "+18556312248"

• Recaptcha configuration

    APP_NAME:     "Kayen"
    CAPTCHA_ENABLED: true
    RECAPTCHA_SITE_KEY: "changeme"
    RECAPTCHA_SECRET_KEY: "changeme"

• S3 configuration

storage:
    provider: "AWS"
    bucket: "1337-bucket-barong-docs"
    accessKey: "************"
    secretKey: "************"
    region: "eu-central-1" # make sure that the right region is used

• Change seeds and configure management api

You can add seeds in this block

seeds_yml:
    users:
      - email: "[email protected]"
        password: "tdj8Zwrr"
        role: "admin"
        state: "active"
        level: 3

    levels:
      - key: email
        value: verified
        description: "User clicked on the confirmation link"
      - key: phone
        value: verified
        description: "User entered a valid code from sms"
      - key: document
        value: verified
        description: "User personal documents have been verified"

Documentation for management api is available here

management_api_yml:
    keychain:
      applogic:
        algorithm: RS256
        value: *key*
    scopes:
      otp_sign:
        mandatory_signers:
          - applogic
        permitted_signers:
          - applogic

Next, run:

./bin/opex app install barong

You can also grant an existing user admin privilliges (Barong admin will not be Peatio admin by default):

# Retrieve a barong pod name
$ kubectl get pods

# Run the rails console on Barong
$ kubectl exec -it BARONG_POD -- rails console

# Find the user
irb(main):001:0> admin = User.find_by(email: "[email protected]")

# Grant the user admin
irb(main):002:0> admin.update(role: 'admin')

# Validate email manually
irb(main):003:0> admin.update(confirmed_at: Time.now)
irb(main):004:0> admin.level_set(:mail)

#4. Peatio deployment

Each cryptocurrency needs to either use a BitGo wallet or an RPC node. To use a BitGo wallet in the system, follow these steps:

• Create a wallet
• Get your cluster nodes' IPs by running kubectl get nodes -o template --template='{% raw %}{{range.items}}{{range.status.addresses}}{{if eq .type "ExternalIP"}}{{.address}}, {{end}}{{end}}{{end}}{% raw %}'
• Create a BitGo API access token for each cryptocurrency, whitelisting the nodes' IPs and setting a spending limit for only one currency at a time- Enter the wallet ID, address and access token for each currency in config/environments/*env_name*/wallets.yml

- name:             Ethereum Deposit Wallet
  blockchain_key:   eth-testnet
  currency_id:      eth
  address:          *wallet address*
  kind:             deposit
  nsig:             1
  max_balance:      0.0
  status:           active
  gateway:          bitgo   # For cryptonode it would be an RPC client, ex. geth
  settings:
    bitgo_test_net:              on
    bitgo_wallet_id:             *wallet id*
    bitgo_wallet_passphrase:     *wallet passphrase*
    bitgo_rest_api_root:         http://coinhub-coinhub.*deployment_id*-backend/api/v2
    bitgo_rest_api_access_token: *api token*

To use an RPC node follow these steps:

• Deploy the cryptocurrency node

• Generate the wallets as listed here

• Configure config/environments/*env_name*/wallets.yml the following way

  - name:             Ethereum Hot Wallet
    blockchain_key:   eth-rinkeby
    currency_id:      eth
    address:          '0x9999999'
    kind:             hot
    nsig:             1
    max_balance:      100.0
    status:           active
    gateway:          geth  # It can be any other supported RPC client
    settings:
    uri:            'http://node_addr:8545'
    secret:         'secret'
  

All the other Peatio config files are located at config/environments/*env_name*/peatio:

• application.yml - general settings including API CORS configuration, base customization(logo, color etc.), metadata and hostname

  You can also configure the timeout for peatio endpoints and /admin tab, to do so change the following block in application.yml

  service:
    timeout_ms: "10000" # This value is specified in milliseconds
    idle_timeout_ms:  "500000"
    connect_timeout_ms: "4000"
  

• blockchain.yml - blockchain configuration for each cryptonode

• wallets.yml - cryptocurrency wallet configuration

• currencies.yml - configs for initially provisioned cryptocurrencies

• markets.yml - configs for initially provisioned markets/trading pairs

• management_api_v1.yml - configs for the Peatio Management API.Documentation for Management API is available here

• plugins.yml - a list of Peatio plugins to enable(the respective source code must be bundled into the Docker image for it to work)

To finally deploy Peatio, run:

./bin/opex app install peatio

#5. Cryptonodes deployment

For a more detailed view on Cryptonodes usage and deployment , check dedicated documentation.

#6. Frontend deployment

To deploy the custom frontend component, see the instructions for it.

#7. Tower deployment

To deploy Tower, check if all the values from config/environments/*env_name*/tower.yml are correct, the default ones should be valid, then run the deployment script:

./bin/opex app install tower

#Maintenance

If you want to do some maintenance on k8s nodes or cryptonodes that you've deployed, you can do so via a bastion node. To be able to ssh to these VM's you have to follow this guide:

1.From your local machine, start ssh-agent with the following command to manage your keys for you:

  me@local:~$ eval `ssh-agent`

2.Call ssh-add to load the gcloud compute public keys from your local computer into the agent, and use them for all SSH commands for authentication:

  me@local:~$ ssh-add ~/.ssh/google_compute_engine

3.Log into an instance with an external IP(use private IP for k8s nodes) address while supplying the -A argument to enable authentication agent forwarding.

  gcloud compute ssh --ssh-flag="-A" INSTANCE

#Troubleshooting

• If you are getting "wrong certificate" errors in the browser, you probably should simply wait a few minutes before Let's Encrypt issue the certificate.

#Appendix

# Benchmarking the deployment with Toolbox

To benchmark your deployment, fill in config/environments/*env_name*/toolbox.yml with correct values and then simply run ./bin/opex toolbox benchmark. The benchmark report would be saved in config/environments/*env_name*/reports afterwards.

# Debugging the deployment with Toolbox console

If you'd like to debug your deployment, feel free to launch the Toolbox console by simply running ./bin/opex toolbox console. This will create a Kubernetes Pod with all the necessary debug tools already installed. To connect to the pod, run kubectl exec -it toolbox-*id* bash

# Using a private Docker registry

Store the credentials to login to the registry as kubernetes secret:

kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

Then configure the components values with the secret name and your private registry:

image:
  repository: <your-registry-server>
  tag: 1.2.3
  pullPolicy: Always
  pullSecret: regcred