Is this an ok way to benchmark Azure VMs?

Message boards : Projects : Is this an ok way to benchmark Azure VMs?
Message board moderation

To post messages, you must log in.

AuthorMessage
Seriema

Send message
Joined: 2 Aug 20
Posts: 2
Message 100179 - Posted: 2 Aug 2020, 7:58:04 UTC

Hi,

I've set up a few Azure VM's and I'm trying to figure out what would give "most bang for the buck". I've never been deep in the benchmarking world so after some time scouring the Internet I came out empty (too many "it depends") and decided to do this:


  1. Use BOINC
  2. Connected to my World Community Grid (WCG) with an Account Key so they all run the same project (COVID-19).
  3. Let each VM run for 48 hours from "Date Installed & Registered", then shut it down
  4. Whatever's reported to WCG "Device Statistics History" during that time is what I'll use
    * This might mean some VM was seconds away from returning a new batch of results
  5. Wait half a day or so for WCG results ("Last Updated" message always says something around "6 hour(s) ago")
  6. Wait at least a day for Azure costs to be calculated
  7. Post results on this forum, see example table below



Example results table:

| VM   | vCPU | Memory | GPU? | Total Run Time | Points Generated | Results Returned | Cost Listed USD/month | Cost relative | Cost Actual 48h | Cost used Spot price? |
|------|------|--------|------|----------------|------------------|------------------|-----------------------|---------------|-----------------|-----------------------|
| B2ms | 2    | 8gb    | no   | 0:001:12:17:19 | 6,306            | 7                | 60.74                 | 2             | X               | Yes                   |
| B4ms | 4    | 16gb   | no   | 0:003:11:14:29 | 9,560            | 11               | 121.18                | 3.99          | Y               | No                    |
| B2s  | 2    | 4gb    | no   | 0:000:06:40:46 | 1,162            | 1                | 30.37                 | 1             | Z               | No                    |


Any suggestions on how to improve my approach? Anything I should change, add, or remove? Is there a good console command for getting some useful stats worth sharing? I'll be shutting off the first batch of VM's tonight, and the rest tomorrow.
ID: 100179 · Report as offensive
ProDigit

Send message
Joined: 8 Nov 19
Posts: 633
United States
Message 100188 - Posted: 2 Aug 2020, 23:43:25 UTC

On all systems, MS Azure, Google Cloud Computing, and Amazon AWS, a system gives less score than 1/3rd of the actual hardware.
I don't know why this is the case, other than them sometimes cutting performance to the CPU cores.
I removed all my cloud accounts, bought a few Atomic Pi units, and though they run at only 1,69Ghz, they still outperform the Intel Xeon 2,1Ghz quadcores by a margin of almost 3x.

What's worse, the shared cores are being cut sometimes for weeks on end running at below 100%.
Sometimes 80%, sometimes 50%, sometimes they run for weeks/months at 1% if unattended.
ID: 100188 · Report as offensive
Seriema

Send message
Joined: 2 Aug 20
Posts: 2
Message 100227 - Posted: 6 Aug 2020, 15:06:31 UTC - in response to Message 100188.  

@ProDigit That's pretty scary! The reason for me to do this is because I tend to get free credits on Azure.

Here are the results for what it's worth. They're all sorted by "Result/Price". The "Cost 48h" is my actual bill and 1 SEK is about $0.12.

Using free credits
One can get up to 150 USD/month, which only affords a few simple VM types as seen by the rightmost column. A bunch of "B1s" seems like the most cost effective way?

| VM                 | vCPU | Memory (GiB) | Total Run Time (y:d:h:m:s) | Points Generated | Results Returned | Cost 48h (SEK) | Result/Price | Price/Month (USD) |
|--------------------|------|--------------|:--------------------------:|:----------------:|:----------------:|----------------|--------------|-------------------|
| B1ls               |    1 |          0.5 |       -:---:--:--:--       |         0        |         0        |           2.05 |            - |                 4 |
| B1s                |    1 |            1 |       0:001:10:03:29       |       4,204      |         2        |           4.03 | 0.4962779156 |                 9 |
| B2s                |    2 |            4 |       0:002:13:35:10       |      10,231      |         7        |          16.84 | 0.4156769596 |                36 |
| B2ms               |    2 |            8 |       0:003:00:16:29       |      10,994      |        14        |          34.34 | 0.4076878276 |                72 |
| B4ms               |    4 |           16 |       0:007:12:44:33       |      16,711      |        23        |          68.02 | 0.3381358424 |               144 |


Paying yourself
Spot VM's are dirt cheap, but subject to availability and can't use free credits. The GPU enabled "NV4as v4" were not worth it, and Linux easily beats Windows (unsurprisingly, but one guide recommended it for the GPU drivers).

| VM                 | vCPU | Memory (GiB) | Total Run Time (y:d:h:m:s) | Points Generated | Results Returned | Cost 48h (SEK) | Result/Price (SEK) |
|--------------------|------|--------------|:--------------------------:|:----------------:|:----------------:|----------------|--------------|
| F4s_v2             |    4 |            8 |       0:007:21:57:31       |      54,881      |        96        |          14.11 |   6.80368533 |
| NV4as_v4           |    4 |           14 |       0:007:17:43:51       |      57,176      |        103       |          19.64 |  5.244399185 |
| E4as_v4            |    4 |           32 |       0:007:04:48:30       |      54,405      |        97        |          20.24 |  4.792490119 |
| NV4as_v4 (Windows) |    4 |           14 |       0:006:22:32:31       |      39,378      |        70        |          17.06 |  4.103165299 |
ID: 100227 · Report as offensive

Message boards : Projects : Is this an ok way to benchmark Azure VMs?

Copyright © 2021 University of California. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.