Buy new drives for osgeo8 and osgeo9

[robe]

We have control over osgeo8 and osgeo9 (donated by OSMF) now.

The specs are as follows:

Both are HP ProLiant DL360 G6 servers

osgeo8 has me a little concerned in that one of the drive bays (drive bay 7 shows as faulty)

It has following specs

Drives: 
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda            8:0    0  1.7T  0 disk
├─sda1         8:1    0    1M  0 part
├─sda2         8:2    0  488M  0 part /boot
└─sda3         8:3    0  1.7T  0 part
  ├─lvm-swap 253:0    0  1.9G  0 lvm  [SWAP]
  └─lvm-root 253:1    0   10G  0 lvm  /

So this gives us I think 800GB storage after we consider ZFS mirroring

2 physical processors (Intel(R) Xeon(R) CPU X5660  @ 2.80GHz, 12288 KB cache) - each with 12 cores it seems - cause reads 24 cpu from Ubuntu (but shows 2 processors from KVM)

Internal L1 Cache:	192 KB
Internal L2 Cache:	1536 KB
Internal L3 Cache:	12288 KB

osgeo9 has the following specs So I think we definitely should put in some new discs to make most of the cpu and memory.

Drives: 
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    0   410G  0 disk
├─sda1         8:1    0     1M  0 part
├─sda2         8:2    0   488M  0 part /boot
└─sda3         8:3    0 409.5G  0 part
  ├─lvm-swap 253:0    0   1.9G  0 lvm  [SWAP]
  └─lvm-root 253:1    0    10G  0 lvm  /


So this gives us I think 200GB storage after we consider ZFS mirroring

2 physical Processors (Intel(R) Xeon(R) CPU X5660  @ 2.80GHz, 12288 KB cache) - each with 12 cores it seems - cause reads 24 cpu from Ubuntu (but shows 2 processors from KVM)


Internal L1 Cache:	192 KB
Internal L2 Cache:	1536 KB
Internal L3 Cache:	12288 KB

[robe]

Grant Slater and Lance from OSUOSL commented:

> > I see that on osgeo8 it shows drive bay 7 is faulty?  Do you suggest we do
> > something about that before we start using it?
>
> If you want to purchase and send us a replacement, we can get that taken care of
> before you use it fully.
>

Model number of existing disks for reference:
HP EG0300FAWHV

G6 and G7 use the same disk caddies, but you can likely just reuse existing.
Gen8+ machines use a newer caddy.

The disk has reasonable reliability and is available cheaply on ebay.
10K rpm disks are more reliable than 15K rpm disks.

Kind regards,

Grant

[robe]

I'd really like to move forward with this purchase. I've started to use osgeo9 mostly for building images. I'm holding off using osgeo8 because of the concern of the bad disk.

[robe]

Thinking about this more, maybe we only need new drives for osgeo9, would be nice if we could put some ssds in that one.

For osgeo8, as Angelos mentioned, there will be a need for vms dedicated for ogc testing in cases where osgeo is the reference implementation (things like various demo sites and also maybe a tutorial). For these kinds of use cases, we need power, but if the disks go bad it's not as critical since these will most likely be rebuilt by ci/cds. The other thing is these kinds of use-cases would probably tend to wear down the disks more so I'd rather get the last bit of life out of the ones we have.

That said osgeo8 has a good chunk of disk -- true I think one might be bad, but maybe we should just wear it down, put non-mission critical stuff on it. I also want to experiment with full VMS. Linux containers work great for most things, but what they can't do is emulate hardware like the arm platforms (something that a true VM can do). So this might be a good place for those as well which would be dedicated to ci/cd testing.

[wildintellect]

What size and quantity of disks do you propose for OSGeo9? What's the intended use of OSGeo9?

[robe]

For OSGeo9 it depends if we can put in SSDs or not. If SSDs then probably best use of it for database intensivish stuff like wordpress (osgeo), trac.osgeo.org (gitea), weblate (which I already have hosted on there ) but can easily move since it's containerized.

Without SSDs it would just provide more redundancy for OSGeo7 / OSGeo3 and possibly set up a true failover system. But right now disk space of OSGeo9 is like 400GB so not enough to do much with.

​https://wiki.osgeo.org/wiki/Osgeo9 (it too has 8 drivebays I think but they are stuffed with much smaller drives than osgeo8). I think they accept the same kind of drives though.

[robe]

I would like to get this done before my contract expires so I can put more stuff on osgeo8 and osgeo9. Both these servers are slightly more powerful than osgeo3.

Also eventually I'd want to rebuild osgeo7, which requires moving everything to different server, but no server has enough disk space left to fully accommodate osgeo7 instances.

[wildintellect]

The same question stands from before, how many and which size disks are being requested? Someone also needs to then look at some vendor options and propose the purchase.

SSD seems possible but that might require that we also get brackets to make them compatible with the caddies and slots?

Looks like we have $2500 in hardware budget, so the proposal would preferably be less than that amount.

[robe]

osgeo9 has 8 drives of product id - DH0072FAQRD ( DH0072FAQRD HP 72-GB 3G 15K 2.5 SP SAS ) that are all filled each with -73 GB 15 RPM disks.

osgeo8 also has 8 days all filled drives of EG0300FAWHV (HP EG0300FAWHV HP 300-GB 6G 10K 2.5 DP SAS) but 6 and 7 drives are showing as faulty.

The pricing on these drives are pretty cheap.

But I'd like something bigger than 300GB. I think disk space is more important than SSD at this point. But I'll ask OSUOSL for their recommendations.

[pnorman]

I would strongly recommend against mechanical hard drives at this point unless you need >> 10 TB space. SSDs involve less administrative work because they fail much less often, and they're an order of magnitude faster.

For our new servers we've tend to purchase them with enough disk space to last the lifetime of the machine, so I haven't been involved in replacing disks on them. Grant has, so you're better off asking him./

The two sensible options are new OEM disks, or used HP-branded disks.

The cheapest way to get 3TB usable per machine is 3x 1.92TB SSD in RAID5. With your budget, you can afford to spend $416/drive.

For new SATA SSDs, it's about $400 per drive (Samsung PM883) For used HP SATA or SAS SSDs, it's about $325 per drive.

This does seem like a lot to spend on HP DL360 G6 machines, which are only worth about 100 bucks each.

For 2.5k you should be able to get a used HP DL360 Gen9 with a CPU launched in 2016, 128GB RAM, and 3x1.92TB drives instead of a machine with CPUs from 2010. The new machine would be far more powerful than the two old machines combined.

[darkblueb]

as a test setup

https://www.servermonkey.com/refurbished-hp-proliant-dl360-gen9-8-port-configure-to-order.html

Selected Items:

HP ProLiant DL360 Gen9 8-Port Chassisx 1
Intel Xeon E5-2683 v3 Processor (2.0 GHz, 14C, 35MB Cache)x 2
    alt. E5-2680v4 (look for a deal)
16GB DDR4 RDIMM 2400MT/sx 12
HP H240 12Gb Smart Host Bus Adapterx 1
HP 960GB 6Gbps SATA 2.5" Solid State Drivex 3
RAID 5x 1
On-Board Quad Port 1GbE LOMx 1
No Additional Cardx 1
HPE Ethernet 366T Quad Port 1GbE Network Adapterx 1
No Optical Drivex 1
(2) HP G9 800 Watt PSUx 1
No Railsx 1
No Cable Management Armx 1
No Bezelx 1
iLo Standardx 1
No VMwarex 1
TekShield 8x5 Support with Replacement for 1 Yearx 1

[robe]

We are not buying another server this year. and I don't want to throw out osgeo8 and osgeo9 cause they are more powerful than osgeo3 and osgeo4, which have been lifesavers.

Plus there is additional headache of throwing out hardware. I also like the idea of having 2 servers exactly the same so I can cluster them. I can't do that with one server. We already have plenty of power. What we lack is disk space capacity to match the amount of data currently on osgeo7 so I can move instances to osgeo9 and osgeo8.

With that in mind, I'm leaning toward getting 10 disks HP EG0300FAWHV 300GB 10000RPM SAS 6GBPS DUAL PORT SFF 2.5INCH HARD DISK DRIVE

2 would go into osgeo8 (to replace the 2 faulty disks)

8 will replace all the disks in osgeo9

making the two servers equivalent.

Looking at - ​https://www.serversupply.com

Each disk is $44.00, so for 10 $440
Sub Total:
    $440.00

Shipping:
    $0.00

Tax(0.00%)
    $0.00

Discount:
    $4.40

Total:
    $435.60 

It would be nice to have maybe something that is 600GB, but I think if osgeo9 had as much disk as osgeo8 we can live with that.

As Paul mentioned, probably not worth spending tons of money like $500/ SSD disk for these servers

[robe]

Oh ebay pricing for same:

​https://www.ebay.com/itm/144589257532

This one we can get 10 for $330, so a bit better deal (28 remaining)

[robe]

Scratch that, this one seems like a better deal, but would require swapping out all the drives. So would be a purchase of 16 drives.

​https://www.ebay.com/itm/223605842312

600GB 10K SAS 2.5" 6Gb/s SERVER HARD DRIVE for HP DL360 DL380 DL385 G5 G6 G7

Cost per drive is $18*16 ~ $300 .

Giving us a raw 4.8 TB on each server to work with. Even at RAID 10, we would get 2.4 TB usable on each.

[msmitherdc]

Let me know when you finalize and I can purchase with the OSGeo cc.

[robe]

This one is an SSD, and seems doable. For this might want to just put in one of the servers and if we like buy 8 more. Or just use 4 drive bays on each and split in two. Or take a risk and buy 16.

​https://www.ebay.com/itm/204064953402 ( Micron 1TB SATA SSD 2.5" 6GBPS Hard Drive Fits HP DL360 DL380 DL385 G5 G6 G7 )

Since the size is 1TB per drive, we could maybe buy 8 ($60 / drive) = $480

If we go with 16 would be 16*60 = $960 (which would cover both servers).

These are cheaper cause they are refurbished and not HP brand, but I think that's all fine. the servers are old so we don't need the drives to last longer than the server and from reading reviews, none HP drives work just as well in HP servers as HP drives

[robe]

Sadly I don't think SSDs are going to work since those require SATA and we have SAS.

So safest is just stick with HDD SAS, but get larger size.

​https://www.ebay.com/itm/224083213581 1.2TB 10K SAS 2.5" 6G Server Hard Drive for Fits HP DL360 DL380 DL385 G5 G6 G7

Cost: 4*$33 = $132

So planning to get 4 of these -- put 2 in osgeo8 to replace the 2 bad drives, and 2 in osgeo 9 (swapping with 2 of the existing 70GB ones)

If these look good, then we can get 12 more = $396

[robe]

This was put for vote, motion passed - ​https://lists.osgeo.org/pipermail/sac/2022-October/014711.html

4 drives have been purchased and onroute to OSUOSL. They are expected there tomorrow.

In prep for upgrade, I am moving grass and weblate to osgeo7 today just in case anything goes wrong and to minimize on downtime.

[robe]

I've moved both grass and weblate to osgeo7 for now until the drives are in place.

[robe]

New drives have been put into osgeo8 and osgeo9 and they all show okay and are rebuilding.

We won't be able to increase the raid array until we put in remaining 12 drives, but we know these work and are good. No more fault drive error on osgeo8

[robe]

Lance is done with the work. All 16 new drives are in place now and osgeo8 and osgeo9 now each have 6TB of space to work with after the RAID 6 reconfig.

I'm going to expand the ZFS to 4 TB on both, but I'll leave 2 TB since we may need to use a different partition type other than ZFS. ZFS isn't great for docker and might not be the best for full VMS either.

[robe]

Actually I'll just expand to 3TB since it's hard to reclaim but easy to expand.