How To Make A Cryptocurrency Trading Bot With Golang

A Quick Disclaimer: I'm a programmer and have very little experience trading. Please do your own research before using the algorithm I create for the cryptocurrency trading bot or any other algorithm. Only trade with money you can afford to lose.

In this guide I will show you how to create  a cryptocurrency trading bot with Go. I highly recommend an IDE such as Goland but you can use a simple text editor if you would like. Feel free to take a look at my guide on how to develop with Go on Linux if you don’t have Go installed.

With that out of the way we can move on to the good part. I will be creating my own index, a very simple index, which goes up or down depending on the quantity of sell and buy orders placed. Basically I will only look at orders that are 2% above or below the latest fill order price and use those to make an index. I will use Bittrex for trading as it offers a lot of trade pairs and will focus on BTC-VTC as I believe in both coins and would like to hold either no matter what my bot does.  My index is basically order rate relative to last price multiplied by the number of VTC the order is for. If the latest price is 0.000100 and a buy order for 10000 VTC is placed at 0.000102 my index would go up by 10200 points.

(0.000102/0.000100)*10000 = 10200

This is a really simple index and someone bidding 10000 VTC 2% higher than latest fill price should maybe have a larger impact? However, to keep things simple I decided to just go with this implementation. If someone was to sell 10000 for 2% less than latest price my index would decrease by 10200 in the same way.

Ah and did I mention that my index resets every minute? I wanted my bot to make minute by minute decisions so I reset the index to 0 every minute.  But enough details, let’s get started with some code.

I used a library called go-bittrex but the current version does not support websockets which will give us real time data. There is a fork of go-bittrex that implements a function for websockets and I have decided to go with that instead. In addition to this library you will also need an API key and secret which you will find on the Settings page under the menu API Keys. I activated everything except Withdraw capabilities just in case my keys get compromised. Once you have your keys we can take a look at the example code on the go-bittrex project page:

package main

import (
	"fmt"
	"github.com/toorop/go-bittrex"
)

const (
	API_KEY    = "YOUR_API_KEY"
	API_SECRET = "YOUR_API_SECRET"
)

func main() {
	// Bittrex client
	bittrex := bittrex.New(API_KEY, API_SECRET)

	// Get markets
	markets, err := bittrex.GetMarkets()
	fmt.Println(err, markets)
}

If you run the code above you should see a long list of all the key-pairs.

[{LTC BTC Litecoin Bitcoin 0.01435906 BTC-LTC true false https://bittrexblobstorage.blob.core.windows.net/public/6defbc41-582d-47a6-bb2e-d0fa88663524.png} {DOGE...

Now, it might look a bit weird and that’s because we are simply using print on our object called markets. If we actually took the time to loop through our markets we could take the pieces we want and format it in a better, more readable way. Let’s try to get our account balances and format it a bit better. Make sure you have replaced your api keys with YOUR_API_KEY and YOUR_API_SECRET in the script above. We will replace the code under //Get markets with our own code. My changed code looks like this:

// Get markets
// Get balances
balances, err := bittrex.GetBalances()
if err == nil {
   for _, b := range balances {
      fmt.Println(b.Currency, b.Available, "/", b.Balance, " Deposit:", b.CryptoAddress)
   }
} else {
   fmt.Println(err)
}

This code will loop through all the balances in our account and print the cryptocurrency, how much we have available for trading, how much we have in total and our deposit address if there is one. If you don’t see any balances it could be because you don’t have anything in your account or you might have the wrong api keys. The output for my account looks like the following:

ARK 0 / 0 Deposit: 
BTC 0.01160229 / 0.01160229 Deposit: 1KqNRbFyi9b2Ej248BMnskCcR82Vdenk7a
BTG 0 / 0 Deposit: GYi1VdAJ892Dwkx5JgRHeXUyMk9PYhBLWW
CVC 0 / 0 Deposit: 
LTC 0 / 0 Deposit: LKPnskMDSsKyPMyPibGDVcFgQiqJme1igR
NAV 0 / 0 Deposit: 
QRL 0 / 0 Deposit: 
RISE 0 / 0 Deposit: 
SC 0 / 0 Deposit: 
SNT 0 / 0 Deposit: 
SYS 0 / 0 Deposit: 
UBQ 0 / 0 Deposit: 
VTC 522.26812636 / 522.26812636 Deposit: 
XVG 0 / 0 Deposit:

You can take a look at the bittrex object to see what other methods are available, I doubt I have had a chance to explore them all.

Goland Hints for Cryptocurrency Trading Bot

The Code of a Cryptocurrency Trading Bot

When I set up my trading bot I added a few constants that work with my index to trigger sells or buys and also limits how much I need to gain or lose before selling, buying or cancelling an order. Here are my constants:

const (
   API_KEY       = ""
   API_SECRET    = ""
   BUY_STRING    = "BTC"
   SELL_STRING   = "VTC"
   MARKET_STRING = BUY_STRING + "-" + SELL_STRING
   MIN_GAIN      = 0.02
   MAX_LOSS      = 0.02
   ORDER_RANGE      = 0.02

   BUY_TRIGGER    = 10000.0
   SELL_TRIGGER   = -10000.0
   ORDER_VARIANCE = 0.02
)

I also need some variables that will keep track of balances, orders, prices, my index, if I have an open order and so on. I also added a bool called readyToRun which is set to false for the first 60 seconds as my script gathers data and it also disables all trading if I lose connection to Bittrex. I also added a highIndex and a lowIndex to get an idea for what the max and min are in one minute for my index.

var (
   balances     []bittrex.Balance
   orders       []bittrex.Order
   ticker       = bittrex.Ticker{}
   lastPrice    float64
   lastBuyPrice = 0.00
   buySellIndex = 0.00
   openOrder    = false
   readyToRun   = false

   highIndex = 0.00
   lowIndex  = 0.00
)

The main function is what is initially called when you start running my program. This function will call all other functions.

func main() {

   // Bittrex client
   bittrexClient := bittrex.New(API_KEY, API_SECRET)

   go updateStats(bittrexClient)

   // A Simple Trading Strategy
   // We create our own buy/sell index, this resets with every buy and sell
   // If we buy and incur a 2% loss we sell at current ask
   // If we buy and make at least a 2% profit and our index is sell we sell
   // If we place an order and it does not clear and market moves -+2% we cancel
   // Every trade has a 0.25% fee

   ch := make(chan bittrex.ExchangeState, 16)

   go subscribeMarket(bittrexClient, ch)

   for st := range ch {
      // Order placed
      for _, b := range st.Buys {
         //log.Println("Buy: ", b.Quantity, " for ", b.Rate, " as ", b.Type)
         quantity, _ := b.Quantity.Float64()
         rate, _ := b.Rate.Float64()
         calculateIndex(true, quantity, rate)
      }
      for _, s := range st.Sells {
         //log.Println("Sell: ", s.Quantity, " for ", s.Rate, " as ", s.Type)
         quantity, _ := s.Quantity.Float64()
         rate, _ := s.Rate.Float64()
         calculateIndex(false, quantity, rate)
      }
      // Order actually fills
      for _, f := range st.Fills {
         //log.Println("Fill: ", f.Quantity, " for ", f.Rate, " as ", f.OrderType)
         // We could say that lastPrice is technically the fill price
         lastPrice, _ = f.Rate.Float64()
      }
      log.Printf("BuySellIndex:     %.4f\n", buySellIndex)
      decideBuySell(bittrexClient)
   }
}

Here we create a new bittrex client with our api keys and then use this to perform further actions. We call updateStats as a goroutine, this means it will run asynchronously with the rest of our program. To learn more about goroutines I would recommend checking out Go By Example. The rest of the code will be executed no matter what happens with updateStats. The next thing we do is to make a channel to handle messages received from Bittrex over our websocket connection. The subscribeMarket function will set up a websocket connection that uses the channel that was created and our for loop at the end of the script will handle any incoming messages. The messages I care about are buys, sells and fills. Buys and sells are just orders, a fill is when one of these orders goes through making it “official” so to speak. I use the buys and sells to calculate my index using the calculateIndex function. Any fill is used to update or lastPrice so that we are always acting with the most recent price when deciding to buy or sell. Finally we decide if we should buy or sell anything.

func subscribeMarket(b *bittrex.Bittrex, ch chan bittrex.ExchangeState) {
      log.Println("Connecting to:", MARKET_STRING)
      err := b.SubscribeExchangeUpdate(MARKET_STRING, ch, nil)
      if err != nil {
         log.Println("Error:", err)
      }
      log.Println("Reconnecting....")
      go subscribeMarket(b, ch)
}

The above code handles our websocket connection via the bittrex-go library. If it should end or error for some reason we try reconnecting. This function is poorly written but works. I would sometimes lose the websocket connection and might need to come back and update this function if I figure out how to reconnect in a better way.

To buy or to Sell?

func decideBuySell(b *bittrex.Bittrex) {
   if openOrder {
      // Should we close the open order?
      for _, o := range orders {
         ppu, _ := o.PricePerUnit.Float64()
         log.Printf("Order percent: %.4f\n", ppu/lastPrice)
         if ppu/lastPrice > (1.00+ORDER_VARIANCE) || ppu/lastPrice < (1.00-ORDER_VARIANCE) {
            log.Println("Canceled order: ", o.OrderUuid)
            b.CancelOrder(o.OrderUuid)
            // We assume we only have one order at a time
         }
      }
   }
   // If we have no open order should we buy or sell?
   if !openOrder {
      if buySellIndex > BUY_TRIGGER {
         log.Println("BUY TRIGGER ACTIVE!")
         for _, bals := range balances {
            bal, _ := bals.Balance.Float64()
            if BUY_STRING == bals.Currency {
               //log.Printf("Bal: %.4f %s == %s\n", bal/lastPrice, SELL_STRING, bals.Currency)
            }
            if bal > 0.01 && BUY_STRING == bals.Currency && lastPrice > 0.00 {
               // Place buy order
               log.Printf("Placed buy order of %.4f %s at %.8f\n=================================================\n", (bal/lastPrice)-5, BUY_STRING, lastPrice)
               order, err := b.BuyLimit(MARKET_STRING, decimal.NewFromFloat((bal/lastPrice)-5), decimal.NewFromFloat(lastPrice))
               if err != nil {
                  log.Println("ERROR ", err)
               } else {
                  log.Println("Confirmed: ", order)
               }
               lastBuyPrice = lastPrice
               openOrder = true
            }
         }
      } else if buySellIndex < SELL_TRIGGER {
         log.Println("SELL TRIGGER ACTIVE!")
         for _, bals := range balances {
            bal, _ := bals.Balance.Float64()
            if SELL_STRING == bals.Currency {
               //allow := "false"
               //if allowSell() {
               // allow = "true"
               //}
               //log.Printf("Bal: %.4f %s == %s && %s\n", bal, BUY_STRING, bals.Currency, allow)
            }
            if bal > 0.01 && SELL_STRING == bals.Currency && lastPrice > 0.00 && allowSell() {
               // Place sell order
               log.Printf("Placed sell order of %.4f %s at %.8f\n=================================================\n", bal, BUY_STRING, lastPrice)
               order, err := b.SellLimit(MARKET_STRING, decimal.NewFromFloat(bal), decimal.NewFromFloat(lastPrice))
               if err != nil {
                  log.Println("ERROR ", err)
               } else {
                  log.Println("Confirmed: ", order)
               }
               openOrder = true
            }
         }
      }
   }
}
func allowSell() bool {
   if lastBuyPrice > 0 {
      gain := lastPrice / lastBuyPrice
      if gain < (1.00 - MAX_LOSS) {
         return true
      }
      if gain < (1.00 + MIN_GAIN) {
         return false
      }
   }
   return true
}

The functions above is a big part of the simple algorithm I created. The first part of the function checks if we have an open order. I decided that if I place an order and the market moves +- 2% then I want to cancel any order because it will most likely never fill. The 2% might be a bit high but it works well as a safety measure as most of my orders tend to fill quickly. Buying or selling is only triggered if my index hits 10000 or -10000 and then some additional checks are run. I implemented the allowSell function to make sure that my bot only sells if there is a gain or loss of more than the Bittrex trading fee of 0.25%. In my code I set the margin to be at least +-2%.

func calculateIndex(buy bool, q float64, r float64) {
   // q is quantity VTC
   // r is the rate
   percent := 0.00
   // Calculate percentage of rate
   if r > 0 && q > 0 && lastPrice > 0 && readyToRun {
      percent = lastPrice / r
      if buy {
         //log.Printf("Buy percent: %.4f\n", percent)
         //log.Printf("Buy quantity: %.4f\n", q)
         if percent > (1.00 - ORDER_RANGE) && percent < (1.00 + ORDER_RANGE) {
            buySellIndex = buySellIndex + (percent * q)
         }
      } else {
         //log.Printf("Sell percent: %.4f\n", percent)
         //log.Printf("Sell quantity: %.4f\n", q)
         if percent > (1.00 - ORDER_RANGE) && percent < (1.00 + ORDER_RANGE) {
            percent = percent - 2.00 // Reverse percent, lower is higher
            buySellIndex = buySellIndex + (percent * q)
         }
      }
   }
   if buySellIndex > highIndex {
      highIndex = buySellIndex
   }
   if buySellIndex < lowIndex {
      lowIndex = buySellIndex
   }
   // Reset really high or low numbers due to startup
   if highIndex > 5000000.00 || lowIndex < -5000000.00 {
      highIndex = 0.00
      lowIndex = 0.00
      buySellIndex = 0.00
   }
}

Here is the calculateIndex function which does exactly what it says. It is what my bot uses to decide to buy or sell. At the bottom I implemented a reset for my indexes as I would sometimes see very high or low values. This happend when launching my program as a lot of orders come through the websocket right away. This is also why my program waits 60 seconds to start. My index is calculated every time there is a message via the websocket. This can happen several times per second and on each update my program will decide to buy/sell. This allows the bot to be very responsive and act instantly.

func updateStats(b *bittrex.Bittrex) {
   var err error = nil
   for {
      go func(b *bittrex.Bittrex) {
         balances, err = b.GetBalances()
         orders, err = b.GetOpenOrders(MARKET_STRING)
         ticker, err = b.GetTicker(MARKET_STRING)
         if err != nil {
            log.Println("Error:", err)
            // Pause calculations in case of error
            readyToRun = false
         }

         log.Printf("====================================\n")
         log.Printf("Last price:       %v\n", ticker.Last)
         log.Printf("Index:           %.4f\n", buySellIndex)
         log.Printf("High Index:       %.4f\n", highIndex)
         log.Printf("Low Index:           %.4f\n", lowIndex)
         lastPrice, _ = ticker.Last.Float64()
         buySellIndex = 0.00

         log.Printf("Bid:         %v\n", ticker.Bid)
         log.Printf("Ask:         %v\n", ticker.Ask)

         // Do we have an open order?
         openOrder = len(orders) > 0

         for _, o := range orders {
            log.Println("Pending order: ", o.OrderType, " Quanitity: ", o.QuantityRemaining, "/", o.Quantity, " Price: ", o.PricePerUnit)
         }

         // Where do we have balances
         for _, b := range balances {
            bal, _ := b.Balance.Float64()
            if bal > 0.00 {
               log.Printf("%s:          %v %s %v\n", b.Currency, b.Available, "/", b.Balance)
            }
         }
         log.Printf("====================================\n")

      }(b)
      <-time.After(60 * time.Second)
      // Wait 60 to init and collect data
      readyToRun = true
   }
}

Finally we have the updateStats function. This function loops every 60 seconds and updates the orders, balances, last price and resets my index to 0. I also configured it to print out useful data for tweaking my variables or debugging the application.

All of the code I have shared with you above is available on Github and is MIT licensed so that you are free to take the code and modify it for your own needs. I also recommend testing some of the calculations in a scratch file, I made an example scratch file that I used to make sure my calculations did what I thought they would.

This seemed like a good topic to write about. I hope my example will help someone else learn. I would love to hear your feedback, criticism and hopefully improvements to the code in the comments below.

Thanks for reading!

Update: Turns out the first cryptocurrency trading bot I ever made was not profitable at all. Initially it made some good trades but then it kept losing money. I tried to tweak my current algorithm and ran it for a week loosing 30% or about 300 euro. At the moment I shut the bot down and have plans to make a new, better trading bot in the future.

Parsing Websites with Golang and Colly

I stumbled across a scraper and crawler framework written in Go called Colly. Colly makes it really easy to scrape content from web pages with it’s fast speed and easy interface. I have always been interested in web scrapers ever since I did a project for my university studies and you can read about that project here. Before continuing, please note that scraping of websites is not always allowed and sometimes even illegal. In the guide below we will be parsing this blog, GoPHP.io.

To begin let’s take a look at the Colly Github page and scroll down to the example code listed there. We will create a new project with a new main.go file that looks like this:

package main

import (
   "fmt"
   "github.com/gocolly/colly"
)

func main() {
   c := colly.NewCollector()

   // Find and visit all links
   c.OnHTML("a[href]", func(e *colly.HTMLElement) {
      e.Request.Visit(e.Attr("href"))
   })

   c.OnRequest(func(r *colly.Request) {
      fmt.Println("Visiting", r.URL)
   })

   c.Visit("http://go-colly.org/")
}

You may need to use go get -u github.com/gocolly/colly/... to download the framework into your go directory. Now let’s go ahead and change the url to the gophp.io website.

c.Visit("https://gophp.io/")

And then we can run the script by typing go run main.go in your terminal making sure you are in the project directory when you do this. You can use ctrl+c in your terminal to cancel as it may run for a long time. What do we get as our output? For me it looked like this:

Scraping the web with Colly

What we see here is exactly what you would expect. Our program parsed all the urls on the main gophp.io page and then proceeded to the first link. This first link is a post at gophp.io but the first link on that page is a link to Virtualbox and our program will keep looping until it stops finding links. That could be a long time and unless you want to make a search engine spider it won’t be the most efficent. What I want is a server that I can call on from a PHP script that just fetches and formats the data I need. Luckily Colly has a complete example of what we need, a scraper server.

package main

import (
   "encoding/json"
   "log"
   "net/http"

   "github.com/gocolly/colly"
)

type pageInfo struct {
   StatusCode int
   Links      map[string]int
}

func handler(w http.ResponseWriter, r *http.Request) {
   URL := r.URL.Query().Get("url")
   if URL == "" {
      log.Println("missing URL argument")
      return
   }
   log.Println("visiting", URL)

   c := colly.NewCollector()

   p := &pageInfo{Links: make(map[string]int)}

   // count links
   c.OnHTML("a[href]", func(e *colly.HTMLElement) {
      link := e.Request.AbsoluteURL(e.Attr("href"))
      if link != "" {
         p.Links[link]++
      }
   })

   // extract status code
   c.OnResponse(func(r *colly.Response) {
      log.Println("response received", r.StatusCode)
      p.StatusCode = r.StatusCode
   })
   c.OnError(func(r *colly.Response, err error) {
      log.Println("error:", r.StatusCode, err)
      p.StatusCode = r.StatusCode
   })

   c.Visit(URL)

   // dump results
   b, err := json.Marshal(p)
   if err != nil {
      log.Println("failed to serialize response:", err)
      return
   }
   w.Header().Add("Content-Type", "application/json")
   w.Write(b)
}

func main() {
   // example usage: curl -s 'http://127.0.0.1:7171/?url=http://go-colly.org/'
   addr := ":7171"

   http.HandleFunc("/", handler)

   log.Println("listening on", addr)
   log.Fatal(http.ListenAndServe(addr, nil))
}

What does the above code do? It will start a webserver running locally on your machine on port 7171. It takes a url parameter and returns all the links found on the url you input. Let’s give it a go by going to http://127.0.0.1:7171/?url=https://gophp.io/. Here is an example of the json encoded output we get:

{
  "StatusCode": 200,
  "Links": {
    "http://185.201.144.162:9181/static/": 1,
    "http://humanstxt.org/": 1,
    "http://pierrickcalvez.com/journal/a-five-minutes-guide-to-better-typography": 1,
    "http://www.gjermundbjaanes.com/understanding-ethereum-smart-contracts/": 1,
    "http://www.neopets.com/": 1,
    "http://www.zoon.cc/stupid/": 1,
    "https://archives.tenghamn.com": 1,
    "https://archives.tenghamn.com/2013/02/10/best-php-ide-jetbrains-phpstorm-review-2013.html": 1,
    "https://bcrypt.fun": 1,
    "https://bitinfocharts.com/vertcoin/address/VcMhEJrnYKNjTSrkazwJXgLHVRB5vKuouv": 1,
    "https://bittrex.com/": 1,
    "https://caddy.community/c/plugins": 1,
    "https://caddyserver.com/": 3,
    "https://code.tutsplus.com/tutorials/apache-vs-nginx-pros-cons-for-wordpress--cms-28540": 1,
    "https://coincall.io/": 1,
    "https://cryptozombies.io/": 1,
    "https://en.bitcoin.it/wiki/Hardware_wallet": 1,
    "https://etherscan.io/address/0xbbb2917f759a09299490d443b82d5324aefe8f9f": 1,
    "https://ferdinand-muetsch.de/caddy-a-modern-web-server-vs-nginx.html": 1,
    "https://github.com/Password-Fun/bcrypt_fun": 1,
    "https://github.com/bayandin/awesome-awesomeness": 1,
    "https://github.com/caddyserver/examples/blob/master/laravel/Caddyfile": 1,
    "https://github.com/egonelbre/gophers": 1,
    "https://github.com/joshbuchea/HEAD": 1,
    "https://github.com/markustenghamn/golang-cryptotracker": 1,
    "https://github.com/markustenghamn/golang-steem-cryptotracker": 1,
    "https://github.com/mholt/caddy": 1,
    "https://github.com/thedaviddias/Front-End-Checklist": 1,
    "https://github.com/vertcoin-project/One-Click-Miner/releases": 1,
    "https://godotengine.org/": 1,
    "https://golang.org/dl/": 1,
    "https://gophp.io/": 2,
    "https://gophp.io/a-simple-bcrypt-hash-generator-website/": 4,
    "https://gophp.io/author/markustenghamngophp/": 10,
    "https://gophp.io/category/caddy/": 2,
    "https://gophp.io/category/crypto/": 3,
    "https://gophp.io/category/crypto/games/": 1,
    "https://gophp.io/category/crypto/hardware-wallet/": 1,
    "https://gophp.io/category/crypto/vertcoin/": 1,
    "https://gophp.io/category/general/": 9,
    "https://gophp.io/category/go/": 5,
    "https://gophp.io/category/html/": 1,
    "https://gophp.io/category/notifications/": 1,
    "https://gophp.io/cli-crypto-portfolio-tracker-in-go/": 3,
    "https://gophp.io/cryptokitties-a-game-played-on-the-blockchain/": 4,
    "https://gophp.io/facebook-free-10-credit-will-applied/": 4,
    "https://gophp.io/go-html-head/": 4,
    "https://gophp.io/how-to-developing-with-go-on-linux/": 4,
    "https://gophp.io/how-to-make-a-simple-go-program-to-track-the-price-of-steem-via-an-api/": 3,
    "https://gophp.io/how-to-protect-your-cryptocurrencies/": 4,
    "https://gophp.io/page/2/": 2,
    "https://gophp.io/switching-from-nginx-to-caddy/": 3,
    "https://gophp.io/tag/ads/": 1,
    "https://gophp.io/tag/api/": 1,
    "https://gophp.io/tag/bcrypt/": 1,
    "https://gophp.io/tag/blockchain/": 1,
    "https://gophp.io/tag/caddy/": 2,
    "https://gophp.io/tag/cryptocurrencies/": 2,
    "https://gophp.io/tag/cryptokitties/": 1,
    "https://gophp.io/tag/description/": 1,
    "https://gophp.io/tag/development/": 1,
    "https://gophp.io/tag/ethereum/": 1,
    "https://gophp.io/tag/facebook/": 1,
    "https://gophp.io/tag/games/": 1,
    "https://gophp.io/tag/go/": 1,
    "https://gophp.io/tag/goland/": 1,
    "https://gophp.io/tag/golang-crypto-portfolio-bitcoin-steemit/": 1,
    "https://gophp.io/tag/golang/": 4,
    "https://gophp.io/tag/hardware-wallet/": 1,
    "https://gophp.io/tag/head/": 1,
    "https://gophp.io/tag/html/": 1,
    "https://gophp.io/tag/ledger-nano-s/": 1,
    "https://gophp.io/tag/linux/": 1,
    "https://gophp.io/tag/mining-crypto/": 1,
    "https://gophp.io/tag/mining-vertcoin/": 1,
    "https://gophp.io/tag/nginx/": 1,
    "https://gophp.io/tag/notifications/": 1,
    "https://gophp.io/tag/safety/": 1,
    "https://gophp.io/tag/steem/": 1,
    "https://gophp.io/tag/steemit/": 1,
    "https://gophp.io/tag/title/": 1,
    "https://gophp.io/tried-mining-vertcoin-1-month/": 4,
    "https://gophp.io/wp-content/uploads/2018/05/goland-screenshot.png": 1,
    "https://hackernoon.com/why-isnt-agile-working-d7127af1c552": 1,
    "https://jeiwan.cc/posts/building-blockchain-in-go-part-1/": 1,
    "https://keepass.info/": 1,
    "https://letsencrypt.org/": 1,
    "https://lightning.network/": 1,
    "https://ma.rkus.io/": 1,
    "https://metamask.io/": 1,
    "https://password.fun": 1,
    "https://revel.github.io/": 1,
    "https://steemit.com/@tenghamn": 1,
    "https://steemit.com/cryptocurrency/@tenghamn/how-to-make-a-simple-go-program-to-track-the-price-of-steem-via-an-api": 1,
    "https://steemit.com/cryptocurrency/@tenghamn/i-made-a-command-line-cryptocurrency-tracker-in-go": 1,
    "https://support.google.com/webmasters/answer/79812?hl=en": 1,
    "https://thousandetherhomepage.com/": 1,
    "https://tobsta.github.io/OpenSource/": 1,
    "https://trezor.io/": 2,
    "https://vtcpool.io/how-to-start-mining-vertcoin/": 1,
    "https://wordpress.org/": 1,
    "https://www.binance.com/": 2,
    "https://www.coinbase.com/": 1,
    "https://www.coinbase.com/join/56a40201e9f0bb14ae000072": 1,
    "https://www.cryptokitties.co/": 4,
    "https://www.cryptokitties.co/profile/0xbbb2917f759a09299490d443b82d5324aefe8f9f": 1,
    "https://www.facebook.com/ads/manager": 1,
    "https://www.facebook.com/ads/manager/account_settings/notification_preferences": 1,
    "https://www.google.com/webmasters/tools/home?hl=en": 1,
    "https://www.jetbrains.com/go/": 2,
    "https://www.jetbrains.com/store/?fromMenu": 1,
    "https://www.lastpass.com/": 1,
    "https://www.ledgerwallet.com/r/eb88": 2,
    "https://www.ledgerwallet.com/r/eb88?path=/products/ledger-nano-s": 3,
    "https://www.nginx.com/resources/wiki/start/topics/examples/full/": 1,
    "https://www.sublimetext.com/": 1,
    "https://www.theguardian.com/technology/2016/aug/31/dropbox-hack-passwords-68m-data-breach": 1,
    "https://www.travian.com/international": 1,
    "https://www.virtualbox.org/": 1,
    "https://www.yes-www.org/why-use-www/": 1
  }
}

The above json output is only 1 level deep. Notice that it does not keep finding links on the pages it finds. This is great because now we could use this program as a sort of microservice. A PHP application could make calls to this microservice and receive all links for the specified url which could later be processed by the PHP application. Now, links are good but we might want to parse other content on the page. Let’s customize our code for this purpose.

Queries For Specific Content With Colly

If we take a look at the source of gophp.io we can see that every title has the css class entry-title which we can use for our query. We will modify the handler function by adding another map for headings. I am only including the section of code that I have changed below:

...
type pageInfo struct {
   StatusCode int
   Links      map[string]int
   // Added headings
   Headings   map[string]int
}

func handler(w http.ResponseWriter, r *http.Request) {
   URL := r.URL.Query().Get("url")
   if URL == "" {
      log.Println("missing URL argument")
      return
   }
   log.Println("visiting", URL)

   c := colly.NewCollector()

   // We add Headings here
   p := &pageInfo{Links: make(map[string]int), Headings: make(map[string]int)}

   // count links
   c.OnHTML("a[href]", func(e *colly.HTMLElement) {
      link := e.Request.AbsoluteURL(e.Attr("href"))
      if link != "" {
         p.Links[link]++
      }
   })

   // count headings
   c.OnHTML(".entry-title", func(e *colly.HTMLElement) {
      // We are looping through the .entry-title elements and then getting the text of the a element
      heading := e.ChildText("a")
      if heading != "" {
         p.Headings[heading]++
      }
   })
...

Now if we restart our program and navigate to our page on port 7171 again we will see some additional output in our json response.

...
  "Headings": {
    "CLI Crypto Portfolio Tracker In Go": 1,
    "CryptoKitties – A Game Played On The Blockchain": 1,
    "Facebook – A Free $10 Credit Will Be Applied When …": 1,
    "How To Make A Simple Go Program To Track The Price Of Steem Via An API": 1,
    "How To Protect Your Cryptocurrencies": 1,
    "How to: Developing with Go on Linux": 1,
    "I made a simple Bcrypt hash generator website with Golang": 1,
    "I tried mining Vertcoin for 1 month": 1,
    "Switching From Nginx To Caddy": 1,
    "What should go in the HTML head?": 1
  }
}

As you can see we have now parsed all the titles on the page and added them to our json output. Using queries we can make very general or specific parsers for any kind of website.

I hope this guide helps someone get started with web scraping. There are several real world examples in the documentation if you would like to learn more. I would love to hear your feedback, questions and comments below!

How to: Developing with Go on Linux

This guide assumes you are using Ubuntu Linux 16.04 (Xenial) but will most likely work for setting up a Go development environment on other versions of linux. You can still follow this guide if you are using OSX but some small things will differ such as the binary to download. You could always run Go on Linux in a Virtual Machine with Virtual Box.

To begin we need to head over to the Go downloads page. The latest release when writing this article is go 1.10.1 and is what I will use. Your links and filenames will differ if you pick a different version. Before starting you should run sudo apt-get updateto make sure we get the latest version of packages we install.

Click on a link or use wget https://dl.google.com/go/go1.10.1.linux-amd64.tar.gz to download the version you would like to install. If you are not sure which version you need simply go with the stable version. It is fairly easy to switch versions later on.

Now let’s extract the files from the tar file we downloaded using the following command:sudo tar -xvf go1.10.1.linux-amd64.tar.gz. Now you can decide where you want to copy your go files. This will be the “program” that is go so to speak and needs to be in your path. I personally prefer to put the files in /usr/local/ and I recommend that you do this as well unless you have a reason to do otherwise. So we simply move our folder using sudo mv ./go /usr/local

You should now be able to type go version and it should print the version you just downloaded and moved. In my case the version shows as 1.10.1. We also need to set up our paths for go binaries so that they can be run from the terminal. Edit our profile file in the home directory, ~/.profile. Add the following to the bottom of the file if it is not already there: export PATH=$PATH:/usr/local/go/bin and then run source ~/.profile to reread this file and apply these new changes to your terminal session. In the future the .profile will be read automatically when you use a terminal.

Now we need to setup a working directory. You can call this whatever you would like or follow my example like so: mkdir $HOME/goprojects. Now we need to add this as our GOPATH so open your .profile file again and add export GOPATH=$HOME/goprojects at the end of the file. Any projects you work on should now be placed under the following structure goprojects/src/github.com/user/yourproject.

Recommended Go IDE

Now we are ready to start editing code. There is no requirement to use an IDE and I know many who prefer something simple such as Sublime Text. However, when it comes to larger Go projects I prefer something more robust and my go to choice is GoLand.

GoLand Go IDE

GoLand comes from Jetbrains and you may have read my previous review of PHPStorm which is another IDE they make for PHP. I really like their software and have a All Products Pack subscription for all of their software. This subscription starts at 25 euro and is reduced for each consecutive year you keep the subscription active. Please note that I am not affiliated with Jetbrains, I am not being paid to write this and there are no affiliate links in this post. I just like their software and would recommend it.

Now as to why I like to use GoLand. It really helps with the small things such as automatically letting you know when you are missing a package and downloading it. Coding assistance is really useful with hints for variables and functions. You might think you don’t need it, I probably don’t but it’s a quality of life thing that makes my work easier to do. There are also many community plugins available that can help you extend this IDE when needed.

That’s the end of my post. I hope this helps you get going with Go and that you find it interesting to use. It is one of my favorite programming languages and I am still learning. I will try to write another guide for Mac OS and possible for Windows if I can get go to work there! I would love to hear your comments and feedback.

Image credit: https://github.com/egonelbre/gophers

I tried mining Vertcoin for 1 month

…And I plan to keep mining.

It has been a fun experience and I would recommend it to anyone who is interested in learning how mining Vertcoin works. I have a gaming PC at home which has a pretty powerful graphics card, the Nvidia 1080Ti. Before I started mining I would turn this beast off to save money on my power bill. However, these days it runs 24/7 as I would save less money by turning it off.

Why start mining Vertcoin?

First of all, only try mining cryptocurrency if you either pay for your own electricity or have discussed this with however does. Mining can get expensive, especially if you have multiple graphics cards.

The reason why I started mining was pretty simple. Over the last year I had taken a greater interest in cryptocurrency and felt the itch to develop something. I had set up my own Bitcoin node and started making applications that would parse and analyse the blockchain. By the way, if you want to get more in depth there is a pretty awesome tutorial on how you can create your own blockchain and cryptocurrency with Go.

I moved from Bitcoin to Vertcoin as the blockchain is much smaller than the 160GB beast that Bitcoin has. It’s just easier to work with and more portable. Vertcoin also follows Bitcoin with implementations like Lightning that will speed up transactions and make them much cheaper. The main difference between Vertcoin and Bitcoin is Vertcoin’s ASIC resistance. Bitcoin is currently very hard to mine and requires an investment of at least a few thousands of dollars for specialized miners to get started. This is not something that the average person can buy. Vertcoin uses more memory heavy algorithms which makes it perfect for mining with a graphics card.

Of course this has also had some negative side effects as miners started buying every graphics card they could get their hands on. This has resulted in much higher prices and a lack of availability for PC gamers. I hope that this is a temporary side effect.

How much did I earn?

I have earned a total of 40 VTC as can be seen on my mining address here: VcMhEJrnYKNjTSrkazwJXgLHVRB5vKuouv. This is about $120 in todays value. Since I am living in Germany at the moment my electricity bill is a bit high but comes in at less than half of my rewards. I did this without mining full time, I paused my mining almost every day and played some games for an hour or two. I plan to keep my VTC to save them or use them to test future projects that I think of. I still have 3 days left before I have actually mined for a full month.

How do you mine Vertcoin?

I run a public node that can be seen here, anyone is free to mine on it http://185.201.144.162:9181/static/. I have also written a guide on how to mine VTC over at VTCPool.io.

The developers have made mining Vertcoin super easy, I run the One Click Miner on my Windows 10 PC without the need to configure anything.

I made a simple Bcrypt hash generator website with Golang

Sometimes when I am working with dummy database I might need to change the password of a generated user. The easiest way to do this is to got to google for a bcrypt generator and then paste the generated hash into the database in order to update the password. This is much quicker than having to write a function to do this for me or to create a new user from scratch. It is also useful if I need to reset a password to an account which I have totally lost access to. Turns out I don’t like many of the bcrypt generators that are out there and the domain bcrypt.fun was available so I figured, why not make my own generator since I already created password.fun.

So in one afternoon I sat down, copied most of the code I had used for password.fun and created this new website. The website uses the Revel framework which feels overkill for a small website like this. I also have no frontend design skills. I just went with the old standard bootstrap layout which looks alright. Now that the site is done you are able to paste in one or more strings (one per line) which will turn into a bcrypt hash when you hit generate. That’s all that it really does, so simple. It does have an API which is technically open to anyone to use but I skipped the documentation and FAQ sections for this project as it was probably overkill for the password generator anyways. There is also no mailchimp sign up form, why did I even add that to the password site?

Deploying Bcrypt.fun

Once again I would like to mention that this is all hosted with Caddy. Caddy makes the setup super simple compared to Nginx or Apache that I used to use in the past. The configuration is literally 4 lines which includes the SSL configuration, see below:

www.bcrypt.fun, bcrypt.fun {
    tls markus@tenghamn.com
    proxy / localhost:8005
}

As always, you can find the entire source code for Bcrypt.fun on Github.

CLI Crypto Portfolio Tracker In Go

From what I learned in earlier posts, mostly parsing json from an API endpoint, I created a simple command line program that tracks any cryptocoins and their respective values, you can input the price and amount you bought it for. Colors can be turned on or off, the update interval can be configured and you could have as many or as few coins as you would like. See it in action here:Golang Crypto Portfolio Tracker

Read my original Steemit post here: https://steemit.com/cryptocurrency/@tenghamn/i-made-a-command-line-cryptocurrency-tracker-in-go

See the code on github and download the release: https://github.com/markustenghamn/golang-cryptotracker

How To Make A Simple Go Program To Track The Price Of Steem Via An API

Earlier today I wrote a quick tutorial on how to use golang to track STEEM or any cryptocurrency for that matter. You can find the post on my Steemit account @tenghamn on steemit here https://steemit.com/cryptocurrency/@tenghamn/how-to-make-a-simple-go-program-to-track-the-price-of-steem-via-an-api

The current script is pretty simple and I plan to expand on it to make a tool that will allow you to track any cryptocurrencies along with a gained or lost amount or percentage based on at what price and how much you purchased these coins for.

You can find the entire script on github here: https://github.com/markustenghamn/golang-steem-cryptotracker

Learning Go

So I have spent the past few weeks trying to get better at Go. I wanted to share a really good resource that I found useful and it’s called Go By Example. I love this site because it provides easy to understand examples for pretty much any of the basic things in Go. As a PHP developer I know how I would like to solve a problem, perhaps I need a for loop and maybe I don’t feel 100% sure how to write this. In that case I can turn to go by example which has a great example of a for loop in go.

So lately I have been working on my side project tournify.io which is a site where you can create tournaments for free. The current site is written in Laravel and contains plenty of bugs. My current goal is to rewrite all the logic with a go backend and use ReactJS for the front end. I have my good friend Alx101 helping me with the frontent. Both Go and ReactJS is pretty new to us so we will see how it goes.